robots.txt file

[annera26]

hi guys did any one of you going on smoothly without the robots.txt file for your site when i was new to this i didnt had it but it really helped me when i put a robots.txt file to the main directory.

[wesley]

What exactly did it help you to do? I have never ever used one and cannot imagine what I would use it for to make my site better?

Please let us know what it does and why we should use it. thank you

[VietXKosuke]

Yeah I agree with Wesley. I've never used one either.. to be honest, I never even read what it does lol. I just know that I've seen other sites with it and I can't tell the difference with -or- without it.

[WebBoy]

A robots.txt file basically gives any visiting search engine spider express permission to crawl every page on the site. Should specific areas of the site be protected from crawling, the folder or file location should appear after the ‘Disallow:’ instruction.
For example:

1. To exclude all robots from a server:
User-agent: *
Disallow: /

2. To allow all robots complete access:
User-agent: *
Disallow:

3. To exclude all robots from parts of a server:
User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow:/private/

4. To exclude a single robot:
User-agent: BadBot
Disallow: /

It now has the added benefit specifying the location of an XML sitemap which is a must if you want Google and the other search engines every opportunity to crawl and index more pages of your website.

[quikfind_jerry]

How it will help us? I have never used robot.txt before.

please elaborate me more pls.

regards
Jerry
================================================== =============================

PPC Search Engine Directory - Affiliates Make Money with Our PPC Search Engine!
Data, Design, Databases, Website Content and more!
Hosting From $2.99 a month!