Robots.txt is usually a file that exists on the basis directory of each Site and may be used to instruct search engines like google on which directories/data files of the website they will crawl and contain of their index. When you materialize to be the journalist who wrote The Guardian https://technik.forum