What is the use of Robot.txt file?

[Robdale]

What is robot.txt? how it works and what is the exact use of it?

[jingwen]

Use it you can choose which search engine could crawled your site.

[handslicemarketing]

its like a record of your site that was indexed by google.. im not really sure...

[Robdale]

Thanks guys.... hoping for some more clear answers.

[sampatrick]

All search engines, or at least all the important ones, now look for a robots.txt file as soon their spiders or bots arrive on your site. So, even if you currently do not need to exclude the spiders from any part of your site, having a robots.txt file is still a good idea, it can act as a sort of invitation into your site.

[Robdale]

Thanks sam... that was something different. Thanks for your reply.

[tanja]

As I understand this file should be written in the sourse code of the page?

[Robdale]

Originally Posted by tanja

As I understand this file should be written in the sourse code of the page?

What do you mean by that? Please elaborate your sentence.

[kanchan31]

I think I am sure That, Its Used for crowler crawled your site.

[Hsekhar]

Robots.txt file is a text file with a couple of instructions to Search Engines that respect the file as to which links of the site they can crawl and index and which they should not.

An example is: My Wordpress Robots.txt file below:
------------------------------------------
# This is your robots.txt file. Visit Options->Robots.txt to change this text.
User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /category
Disallow: /tag
Disallow: /author
Disallow: /trackback
Disallow: /*trackback
Disallow: /*trackback*
Disallow: /*/trackback
Disallow: /*?*
Disallow: /*.html/$
Disallow: /*feed*


# Google Image
User-agent: Googlebot-Image
Disallow:
Allow: /*


# Google AdSense
User-agent: Mediapartners-Google*
Disallow:
Allow: /*

------------------------------------------------

By using robots.txt, you can prevent any personal pages on your site from getting indexed.

[Robdale]

I have noticed that many of the major sites disallow's everything in robot.txt except the content. Why?

[janicejan]

Robot text are most used by large websites to avoid unnecessary pages to get crawled by bots and getting indexed.

[pooja84]

It provides restriction to search engine robots that crawl the web. You can get a lot of information on sitemaps.blogspot.com/2006/02/using-robotstxt-file.html

[shruchi nagar]

Robots.txt is used for giving some instruction to search engine.
It’s basically having the list of links of the site they can crawl and index and which they should not. Our site is crawl quickly if it contains sitemap and robots.txt

[jimmy03]

Robots.txt is a file which let you to restrict or let you to decide which search engine you want to crawl your website, or the crawler of search engine which you don't want to crawl your website you can restrict that crawler by disallowing that search engine in search engine.
Also you can also decide which page of your website is to be crawled or not...

[betty]

Robots.txt is a text file which teaches the search engines about pages or directories to keep out from its database

[jhon12]

Robot.txt is very important in terms of stop any robot to crawl any data or page of websites.

[paradigmad]

Google is your best friend on this

Here is more info on robots.txt

http://www.javascriptkit.com/howto/robots.shtml