SearchPath Blog

SearchPath Internet Marketing Blog - Thoughts, ideas, humour, information and more ...

Wednesday, August 27, 2008

Top 5 Reasons for Web Pages Not Being Indexed

There are many reasons why you may have problems with getting your web pages indexed, and most have issues to do with making it difficult for search engines 'robots' accessig your web pages.

Search engine robots are simple programs - if they can't find your site content immediately, they will skip our site and move on to the next site. It is therefore critical that your site is setup to be search engine friendly and 'crawable' by search engine robots.

Below are the top 5 reasons for robots not crawling your site:

1. Your robots.txt file is setup incorrectly or corrupted.
A robots.txt file tells robot which files/directories you don't want it to index. If you have any typos or errors in your robots.txt file, then a robot may completely ignore your entire site. Log into Google Webmaster Tools and use their robots.txt testing facility.

2. Your URLs contain too many variables
Long URL strings, URLs with question marks, that are dynamically generated, or that contain multiple symbols can be difficult for search engines to follow. Make your URLs as simple and descriptive as possible.

3. Session IDs in URLs
Session IDs cause duplicate problems issues and are a big no for Google. Session IDs should be avoided at all costs - look to use cookies instead.

4. Excessive onsite code
If you have too much code on your site and the real content of your site is difficult to find, robots may just give up. Make sure your CSS, javascript etc is moved off your pages and into separate files.

5. Poor website navigation
When we say poor website navigation, we're talking about code that Google can't follow - that means flash based menus, Ajax, some Javascript and DHTML.

Crucial to any chance of being ranked on Google is to ensure that its robots can find and index all of your web pages. Don't put barriers in the robots path, bt just try to make their job as easy as possible. Make sure you have god navigaiton, that your robots.txt file is working and that you have an on-page sitemap. Once your pages are indexed, you can then see what additional work is needed to get your pages the exposure you need, whether that's adding more content or working on developing more good quality links to those pages.