Search Engine Spiders and Internet Crawlers

This page is currently under development

Every time a search engine spiders or indexes your website they will leave a note of this in you server logs. Through analysing these or by using an internet stats package you can discover which search engines are spidering your site. These can give you information on which pages are being included and which pages are not, which pages are the most important to the search engines and also the regularity with which different search engines are indexing your site.