Spider-stoppers

How to tell if your site is locking out the search engines

by Bill Marshall
of http://www.oyster-web.co.uk

Last updated: 12 Sep 2007

If you have a website which is under-performing in the search engines then one of the first things you need to check is whether the site is being fully indexed. If it isn't then you may be suffering from spider-stoppers. Likewise when putting a new site together you need to know what techniques to avoid.

There are a number of web design techniques and other scenarios that will prevent a Search Engine spider from traversing your site.

Firstly it needs a way in - are there any links pointing at your site from reputable sources? If not then the search engines won't know about the site.
Once a spider has found your home page is there anything there to see? If it finds a Frameset it may not be able to see the content, or it may find that the content is being pulled in from another site. If all it finds is a Flash file then it won't see any useful content.
Can it go anywhere from the home page? There may be buttons that people can click on but if they are made in Flash, or they are operated by certain forms of JavaScript, then the Spiders either won't see them or won't be able to follow them. The same applies to jump menus.
Those jump menus are a basic type of form. Forms can be another block on spiders - if your content can only be accessed via a search form then it might as well be behind a brick wall.
If your site is database driven and there are long Query Strings then the spiders may have difficulties. Keep parameters as simple as possible.
If your site uses session ids for any reason and these are introduced as soon as the site loads rather than only when a purchase is being made then the spiders will reject the links because otherwise every time they visit, the same pages will appear to have new addresses.

All these scenarios are very common and will usually kill your chances of ranking stone dead. Our engineers are trained to spot these problems as a matter of routine and can quickly check the visible source code to identify and confirm them. But what can you do to tell if you may have a problem that requires professional help?

Make sure you know how big your site is and compare the number of pages with the results of a check on Google, MSN, or Yahoo. On all three you can run a site command - simply type site: followed by your web address - and you will see a list of all the pages that engine has indexed. If your site has been running for a while and few of the pages are appearing then there's a good chance you may have a problem with spider stoppers. While the big 3 search engines often disagree on the numbers of pages, if you're seeing a big difference then it may be an indication that one or more of them is having trouble accessing the site

If many of the pages which are listed in Google have "supplemental result" next to them then again you may have a problem. Check the last cached dates - if they are quite old then there may be a reason that they haven't been able to access the files even though they know about them.

Look closely at your page addresses and the links that show in your status bar when you hover over a link. If the addresses are really long and have lots of ? characters, long strings of numbers, or have words like "sessid" in them then the likelihood is that spiders will not find them friendly. If the links don't show up when you hover over them then the links may be unfollowable.

If you suspect there is something wrong but aren't sure what it is or what to do about it then give us a call.