What Are Search Engine Spiders? Part Three

Analysing Web Server Logs.

Web Server Logs are reasonably simple when you understand the basic conventions of Log files are understood. Note that your server company log files may differ from those that follow. We have tried to use the most commonly user styles of log files.

Example File : Information is colour coded to correspons in each section

mercurythread.co.uk : [08/Aug/2005:03:27:12 +0100] 66.249.71.9 - - "GET / HTTP/1.0 /seo-resources/internet-services.htm " 200 - "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"

Domain : [date] I.P adress of the Spider -- "Page Requested" Header Response - User Agent - Info Page

This information allows you to see which information each part of the log file string contains. From this information you are able to understand the movements of the search engines within your website. This will allow you to see if all the currently active spiders visiting your site are obeying the robots.txt file and/or are properly indexing all of your site.

:: Tutorial Index :: What is a Search Engine? :: What are Search Engine Spiders? :: Controlling Search Engine Spiders :: Analysing Log Files :: What Search Engines look for in Web Document Content :: Anatomy of Search Engine Results ::

Copyright Information

This article or tutorial relating to Search Engine Optimization is provided by conSEOquences. This article and the copyright relating to it remains the sole property of its owners. We do allow our articles to be reproduced by other web site owners but only when the article is produced as being from the conSEOquences site and that a link to the original article is provided.