Search engines constantly index hundreds of millions of pages on the web.  The work is never done because the internet is constantly changing.  There are two basic methods of doing this, crawler-based search engines and human-powered directories.  Many search engines use both.  Human powered directories are directories built of individual submissions by people who would like their site included in a particular search engine directory. Crawler-based search engines use software to scan the web and create indexes.

Crawler-based search engines use special software robots called “spiders” to build huge word lists.  A spider might start out on a popular site and follow every link found within that site.  In this way the spider travels across the web which is often referred to as “crawling”.

The word lists that spiders build must be indexed in a useful way and then encoded and stored. Search engines use different methods to create their indexes.  Some index every single word on a page while another might only index significant words leaving out articles such as “the”.  Some methods of indexing give special attention to page titles and meta-tags or keywords inserted in the HTML code that describe the concepts and information contained within the page.  Search engines also employ diverse methods of ranking or assigning weight to pages.  Since these formulas for ranking vary widely between search engines you will often find very different results when you look up the same thing on more than one search engine. 

Basics of Searching