In order to avoid unwanted written content within the lookup indexes, site owners can instruct spiders not to crawl specified files or directories with the normal robots.txt file in the foundation Listing of your domain. In addition, a web site can be explicitly excluded from a internet search engine's databases https://news.rafeeg.ae