Robots.txt can be a file that exists on the foundation directory of each Web page and can be used to instruct search engines like google and yahoo on which directories/information of the website they will crawl and include inside their index. In case you are expecting them for being 100% https://news.rafeeg.ae