robots.txt

[ranilyn]

How robots.txt works? Why is it important?

[spinxwebdesign]

Most sites contain pages that should not be indexed by the search engines. Administrative pages, for example, ABC's "contact" page: "contact.html". There's no need to have it indexed, so we use the robots.txt file to tell the search engine spider (robot) to ignore it.
You can use robots.txt file for saving PR juice for that particular page.

[archana_sharma]

Hello guys,
Robots.txt is a simple text file, put on your site to tell search robots which pages you would like them not to visit.

[suzib]

The reason also is to avoid duplicate content. Dynamic pages often have different urls but end up at the same place ie website /pageid?xxx.asp or website /about-us.asp. Also some terms & privacy pages appear to use the same site meta data so by excluding those pages you avoid duplicate tags in the site.

[waynelapricon]

It is vital to explain that robots.txt is not a way from avoiding search engines from crawling your site that is it is not a firewall or a kind of password protection and the fact that you put a robots.txt file is something like an unlocked door.

[stephanrobert]

When you are doing some changes in your website and you want Google to not crawl your website then robots.txt tells the Google crawler which pages you want not to visit.