Thread: what is a robot.txt file
View Single Post
  #14 (permalink)  
Old 03-30-2010, 04:16 AM
jason830 jason830 is offline
Senior Member
  
Join Date: Aug 2009
Posts: 237
jason830 is on a distinguished road
it is a file that tell spider which page he could crawling,which he could not.
Reply With Quote