To stay away from unwanted content from the lookup indexes, website owners can instruct spiders to not crawl certain documents or directories with the conventional robots.txt file in the basis Listing on the domain. On top of that, a website page may be explicitly excluded from the search engine's databases https://news.rafeeg.ae