Content material Method: The cornerstones of the content material approach are search phrases. They function a blueprint for writing web site entries, articles, item descriptions, and other forms of material. txt file is then parsed and can instruct the robotic as to which webpages usually are not to get crawled. https://www.youtube.com/watch?v=CWAXJPBjGjE