Robots.txt?

[jesicawillss]

What is robots.txt? Is it a part of on-page?

[spinxwebdesign]

Yes robots.txt is one part of on-page optimization. Most sites contain pages that should not be indexed by the search engines. Administrative pages, for example, ABC's "contact" page: "contact.html". There's no need to have it indexed, so we use the robots.txt file to tell the search engine spider to ignore it.

[Ancient]

It's used to restrict search engine crawler to access some files and directories

[IntegratedS]

Robots.txt is a useful file that keeps search engines away from indexing pages you do not want to be crawled.

[angelina12]

Robots.txt comes in on page and thats true, it is in the root of the website and informs search engines which files to crawl.

[iresh]

It is a file containing instructions to the crawling robots, it will inform them not to index some part or all of your site in there engine.

[seomelbourne]

robots.txt is simply a plain-text file that a Web publisher should put in the root directory of their website. The text files includes instructions that tell indexing spiders, or "robots," what content and directories on that website they may, or may not, look at.

[webseo]

yes, Robots.txt part of on page optimization but read for more details Robots.txt on Google..

[Siyal]

Yes , It is a part of onpage optimization.It can define on webmaster tool it will be easy to index your WebPages .

[jean may lee]

Originally Posted by spinxwebdesign

Yes robots.txt is one part of on-page optimization. Most sites contain pages that should not be indexed by the search engines. Administrative pages, for example, ABC's "contact" page: "contact.html". There's no need to have it indexed, so we use the robots.txt file to tell the search engine spider to ignore it.

Additional, it will make your site on a stronger base or structure once you design your sites with robots.txt,a wise doing from your optimizing task.

[jesicawillss]

Thanks Spinx. It is very nice info. I understand it. But, how can I do it in practical? Can I use it on blogger blog?

[indiarajasthan]

Yes this a part of on page seo . If you want to not crawl any page in search engine then use this

[Anthony696]

Originally Posted by jesicawillss

What is robots.txt? Is it a part of on-page?

Yes robots.txt is mainly considered as the technical SEO work but i think also comes into the on page SEO techniques because this file instruct search engine spider that which pages of the website need to index or which is not.

[Logicspice]

Robots.txt is very useful for dynamic website, because they auto generate pages. by robots.txt we can stop crawler to index extra page link admin, loging, or directory.

[michaelhomas83]

Right robots.txt is helpful for site who preventing crawler on extra pages ...

[C.Rebecca]

There is another very good use of robots.txt ... You can use robots.txt files with your Sitemap for auto discovery.
Accoding to Sitemap protocol, maor search engines are capable of auto discovery of Sitemaps with robot.txt files.

[ameeghey]

robots.txt is simply a plain-text file that a Web publisher should put in the root directory of their website.

[hetal]

in short robots can ignore your /robots.txt....rest you will know from above..

[hetal]

what is malware robots and malware issue ??