robots.txt
The robots.txt is a file you should have in the root directory of your web server. This file is used by search engine robots as a guide to which files and pages they should access on your website.Here is an example of a robots.txt file.
# PARTIAL access (All Spiders)
User-agent: *
Disallow: /callme_generic.asp
Disallow: /cgi-bin/
Disallow: /new_style.css
Disallow: /images/
Disallow: /send_generic.asp
Disallow: /send_newsletter.asp
Disallow: /templates/
Disallow: /test.asp
As you can see there are two main sections to the file. These are:
User-agent: This is where you identify the search robot or robots you want to communicate with. The "*" means all robots.
Disallow: These are the files or directories that you do not wish the robot or robots to access or index.
There are two main uses for the robots.txt file, which are:
1. Telling the search engine robots which files or directories you do not wish them to index. This can be done because you want the robot to access your files in the most efficient way i.e. not waste time on the non-important pages/files. The other reason is because there are files that you do not want to appear in the search engine results for confidential or relevancy reasons.
2. It is a good and quick way of seeing how many times your site has been visited by search engine robots in any one period.
Of course, we would be happy to help you with this aspect of search engine optimisation, contact SearchPath today.
0 Comments:
Post a Comment
Links to this post:
Create a Link