Quote:
Originally Posted by gurukid22
A robot and a spider is a program that crawls webpages looking for information. In this case the robots.txt file is a file which holds instructions for these robots normally search engines because it is the only robots that webmasters like to help crawl their site.
The robots file it not always needed for a website most common instructions can be placed in the header of webpage to the same thing. Normally people block crawling certain links and sub pages.
|
Thanks for the explanation.Why would people want to avoid crawling of ome links?