What Are Search Engine Spiders?

Search Engine Spiders, or Search Engine Robots, are the programs and sub routines that are used by Search Engine Companies to navigate the Internet and locate new and updated web documents. The Search Engine Spider then relays all the information it discovers to the processing unit of the Search Engine.

Why would a Search Engine Spider visit your Web Pages?

1. The Search Engine Spider has been sent to visit your page as you have submitted your Web Site Address (URL) to that particular Search Engine.
2. The Search Engine Spider found a link to a document on your site whilst spidering another web site.
3. The Search Engine Spider has located a link to a document on your site whilst spidering your web site.
4. The Search Engine Spider knows that the page exists and is attempting to discover if the content on that page has changed.

What can Search Engine Spiders index?

Search Engine Spiders can place almost all file types into the index. This does not mean that they understand each particular file types. The Google ™ Search Engine has the ability to understand many file types including PDF, HTML, ASP, PHP and TXT. However it is able to index many type of flash movies, MP3's and Power Tab files without being able to understand and process their contents.

:: Tutorial Index :: What is a Search Engine? :: What are Search Engine Spiders? :: Controlling Search Engine Spiders :: Analysing Log Files :: What Search Engines look for in Web Document Content :: Anatomy of Search Engine Results ::

Copyright Information

This article or tutorial relating to Search Engine Optimization is provided by conSEOquences. This article and the copyright relating to it remains the sole property of its owners. We do allow our articles to be reproduced by other web site owners but only when the article is produced as being from the conSEOquences site and that a link to the original article is provided.