what is robots.txt

[Firsttender]

Robots.txt is a file which helps search engines know which pages to crawl and which not to do

[spyindia07]

The Most common usage of Robot.text is to ban crawls from visiting private folder or content that gives them to additional Information.

[AmieWilson]

Dear Friend,

Robots.txt is Text file.. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site and the location of robots.txt is very important...

Thanks & Regards
AmieWilson

[jaysh4922]

Robot.txt is an on-page SEO technique and it is basically used to allow for the web robots also known as the web wanderers, crawlers or spiders. It is a program that traverses the website automatically and this helps the popular search engine like Google to index the website and its content.

Just create a file named robots.txt, put it on your root directory of your website(where index.htm/index.php is), edit with a text editor and add these:

User-agent: * (means refer to all crawlers)
Disallow: *something*

[Firsttender]

Robot.txt is the process where we hide some page which we don't want to cache by any search engine.

[John Pitt]

Robots.txt is just an text file in which you mention the search engine crawler to crawl the website and you should mention the particular folder , file or image to which you dont want crawler to crawl while indexing.

[shopping wish]

Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to pay a visit.

[ccole]

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results.

[chugtairizwan001]

Robots text file are those which forced to search engine how to crawl and index the your site or web pages.
It is great when search engines frequently visit your site and index your content but often there are cases when indexing parts of your online content is not what you want.

[pankajspyindia]

It is a text file in root directory which tell the search engine whether it should be crawl or not.

[davisjay89]

Robots.txt acts as a guide to the crawlers it helps in Indexing of pages

[Sohail]

Robots.txt is a file which tell search engine crawlers that which pages should not be crawled. It is used to restrict the search engines crawl from admin folder, js or css folder etc. Some times you also want to restrict your images folder.

[indirataxi]

robot.txt helps the crawler for vising the website pages

[deelip]

Robot .txt is a text (not a html file) you put on your site which tell to search robots which page you like them not to visit my site that's called robot txt.

[vivekindia]

We given instruction to spider how to access and review my webpages through the help of robots.txt

[jammesscot]

The robots.txt is a simple text file in your web site that inform search engine bots how to crawl and index website or web pages.