|
Robot.txt file serving search engins for instructions if you do not want to have your index of the contents or images or something else. So if you have some duplicate content on the site thus showing that these search engines to bypass and does not appear in their search results.
|