what is a robot.txt file

adam0211 · #1 (**permalink**) 04-24-2009, 05:52 AM

pls Explain me robot.txt file

is it important from the google point of view

pooja84 · #2 (**permalink**) 04-29-2009, 11:25 AM

It is very important from google point of view. For more information refer to javascriptkit.com/howto/robots.shtml

mokmok69 · #3 (**permalink**) 05-01-2009, 10:26 AM

I think it will help you to crawl all your meta's description of your site,...if your site is crawlable by Google it will easily indexed.

Ben Trugde · #4 (**permalink**) 05-06-2009, 02:14 PM

yeh what's a robot and a spider?

gurukid22 · #5 (**permalink**) 06-16-2009, 10:30 AM

A robot and a spider is a program that crawls webpages looking for information. In this case the robots.txt file is a file which holds instructions for these robots normally search engines because it is the only robots that webmasters like to help crawl their site.

The robots file it not always needed for a website most common instructions can be placed in the header of webpage to the same thing. Normally people block crawling certain links and sub pages.

Patricia · #6 (**permalink**) 07-04-2009, 07:36 AM

Yes, It is very very important from Google point of view.

mark spenser · #7 (**permalink**) 08-12-2009, 01:01 PM

it commands search engine crawlers that which pages to crawl and which pages not.

josine · #8 (**permalink**) 01-08-2010, 01:13 PM

Quote:

Originally Posted by gurukid22

A robot and a spider is a program that crawls webpages looking for information. In this case the robots.txt file is a file which holds instructions for these robots normally search engines because it is the only robots that webmasters like to help crawl their site.

The robots file it not always needed for a website most common instructions can be placed in the header of webpage to the same thing. Normally people block crawling certain links and sub pages.

Thanks for the explanation.Why would people want to avoid crawling of ome links?

prakask · #9 (**permalink**) 02-13-2010, 12:00 PM

This is a file that keeps away search engines from indexing pages you do not want spidered. But every site cannot use it.

alina11 · #10 (**permalink**) 03-02-2010, 12:31 PM

The robots.txt file is just a very plain text file (or an ASCII file, as some like to say), with a very simple set of instructions that we give to a web robot, so the robot knows which pages we need scanned (or crawled, or spidered, or indexed - all terms refer to the same thing in this context) and which pages we would like to keep out of search engines.

yunadowney · #11 (**permalink**) 03-12-2010, 02:08 PM

Quote:

Originally Posted by adam0211

pls Explain me robot.txt file

is it important from the google point of view

It is a convention to prevent cooperating web spiders and other web robots from accessing all or part of a website which is otherwise publicly viewable. Robots are often used by search engines to categorize and archive web sites, or by webmasters to proofread source code.

source: wikipedia

Susan Davies · #12 (**permalink**) 03-29-2010, 12:13 PM

A "robots.txt" file tells search engines whether they can access and therefore crawl parts of your site.
This file, which must be named "robots.txt", is placed in the root directory of your site.

Hope this will help you.

BETTY2010 · #13 (**permalink**) 03-29-2010, 01:24 PM

Through “robots.txt" file you direct the search engines which page to index or not to index.

jason830 · #14 (**permalink**) 03-30-2010, 04:16 AM

it is a file that tell spider which page he could crawling,which he could not.

crazedpanda · #15 (**permalink**) 04-09-2010, 06:46 AM

A robots.txt file tells search engines what they can and what they can not crawl and index in your site.

qr8gul1 · #16 (**permalink**) 04-22-2010, 08:48 AM

The robots.txt file is also called the robots exclusion file. It is used to indicate to spiders which pages you do not want them to index.

Susan Davies · #17 (**permalink**) 04-26-2010, 07:10 AM

Hi,

"Robots.txt" is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all. For example, you may not want Google to crawl the /images directory of your site, as it's both meaningless to you and a waste of your site's bandwidth. "Robots.txt" lets you tell Google just that.

Regards.

tommydoo · #18 (**permalink**) 04-26-2010, 07:38 AM

Robot.txt file serving search engins for instructions if you do not want to have your index of the contents or images or something else. So if you have some duplicate content on the site thus showing that these search engines to bypass and does not appear in their search results.

dannissmith20 · #19 (**permalink**) 05-29-2010, 06:32 AM

The robots.txt is a very simple text file that is placed on your root directory. You can ONLY have one robots.txt on your site and ONLY in the root directory. All major search engine spiders respect this, and naturally most spambots such as email collectors for spammers do not.

robineparera · #20 (**permalink**) 06-03-2010, 06:51 AM

Robots.txt is a text file that you place on your website, that show search engines, which pages on your website should not be crawled. It is a very import file for seo,if you do not put such a file in your website,your whole content in your website will be crawled by search engines spider.