Searching the Internet: Types of Search Engines

Crawlers     |     Directories     |     Hybrids     |     Meta

Crawlers

These types of search engines use a "spider" or a "crawler" to search the Internet.  The crawler digs through individual web pages, pulls out keywords and then adds the pages to the search engine's database.  Google and Yahoo are examples of crawler search engines.

The advantage of crawlers are:

• They contain a huge amount of pages. 
   
• Ease of use. 
   
• Familiarity.  Most people who search the Internet are familiar with Google.

There are several disadvantages to crawlers:

• Sometimes, it's just too much information.
   
• It is easy to trick the crawler.  Websites have hidden data that can be manipulated to make the page appear like it's something it's not.  So, that search result for Descartes might actually take you to a porn site. 
   
• Page rank can be manipulated.  While search engine companies frown on the practice, there are ways to improve where your page appears on the list of results. 

Directories

Directories are human powered search engines.  A website is submitted to the directory and must be approved for inclusion by editorial staff.  Open Directory Project and the Internet Public Library are examples of directories.

Advantages:

• Each page is reviewed for relevance and content before being included.  This means no more surprise porn sites.
• Less results sometimes means finding what you need quicker.

Disadvantages:

• Unfamiliar design and format. 
   
• Delay in creation of a website and it's inclusion in the directory.
   
• May have trouble with more obscure searches.

Hybrids

Hybrids are a mix of crawlers and directories.  Sometimes, you have a choice when you search whether to search the Web or a directory.  Other times, you may receive both human powered results and crawler results for the same search.  In this case, the human results are usually listed first. 

Meta

Meta search engines are ones that search several other search engines at once and combines the results into one list.  While you get more results with meta search engines, the relevancy and quality of the results may sometimes suffer.  Dogpile and Clusty are examples of meta search engines.