Information Method: The cornerstones of one's material technique are key phrases. They serve as a blueprint for writing blog entries, articles, products descriptions, and other forms of fabric. txt file is then parsed and will instruct the robot as to which internet pages usually are not to generally be crawled. https://www.youtube.com/watch?v=aKirkq1uHGM