Quote:
Originally Posted by adam0211
pls Explain me robot.txt file
is it important from the google point of view
|
It is a convention to prevent cooperating web spiders and other web robots from accessing all or part of a website which is otherwise publicly viewable. Robots are often used by search engines to categorize and archive web sites, or by webmasters to proofread source code.
source: wikipedia