SEO & Robots txt file

seopeter · #1 (**permalink**) 11-11-2009, 06:03 PM

Robots txt file is very usefull to prevent any bot to index your web site, infact each web site should have because i do not think that many crawlers check and index your web site with having robots.txt

lexdino · #2 (**permalink**) 11-13-2009, 11:51 AM

You should only have a Robot.txt file in your site if you wish to block bots from indexing your site so not every site should require a Robot.txt file.

pmolds · #3 (**permalink**) 01-15-2010, 12:49 PM

You should write a robot.txt file and put it to your site host.

jason830 · #4 (**permalink**) 01-18-2010, 07:19 AM

yes,add a robots on your site is a good way to make spider crawl your site,it could tell spider which page he could crawl, which one he could not.

sarkarkumar09 · #5 (**permalink**) 01-28-2010, 01:14 PM

Quote:

Originally Posted by seopeter

Robots txt file is very usefull to prevent any bot to index your web site, infact each web site should have because i do not think that many crawlers check and index your web site with having robots.txt

You are right mate ...Still i think that happens..

But at one instance it happened that i used it and still my site got indexed ..i did some social bookmarking of that site ...it may be the whole sol reason...!!