The robots.txt is a file that can be used to restrict access to your site or parts of your site by preventing access from the search engine robots as they crawl the web. If you have under construction pages, they will not be indexed by google as your robots.txt instructed google bot to do so.
Robots.txt is a text file that you place on website that show search engines which pages on your website should not be crawled. And if your robots.txt is not properly,search engines spider will fail to crawl your website or even public your private of your website. So robots.txt is very important for your website.
The robots.txt file is a set of instructions for visiting robots (spiders) that index content of your web site pages. For those spiders that follow the file, it provides a map for what they can, and cannot index.
The Robots.txt is a easy text file so as to tells the search engine not to throat or indexes a exacting web page, file or directory. It is stored in a website.
This is a file that keeps away search engines from indexing pages, robots.txt file tells search engines what they can and what they can not crawl and index in your site.
__________________ buy earrings the site is for sale.
A robots.txt is good if you want to limit the bots and what they index. I always disallow my images from the bots so they dont waste time indexing all images. Other good things to use a robots file for is to specify where your sitemap is located.
If you dont want to limit the bots you dont need a robots.txt really.
New 
Linear Mode
