The History of Web Search Engines

1
The History of Web Search Engines

• Archie, 1990
• Veronica, 1993
• World wide web wanderer and the ALIWEB
• Spiders
• Yahoo!, 1994
• Brains webcrawler, 1994
• Mellon-mania The birth of lycos
• Infoseek
• Altavista

2
Archie -- Grandfather of All Search Engines

• Created in 1990 by alan emtage, a student at
  mcgill university in montreal
• Archives was shorten for unix standard
• The primary method of storing and retrieving
  files was via the file transfer protocol (FTP)
• Many important files were scattered on small FTP
  servers.
• Archies gatherer scoured FTP sites across the
  internet and indexed them, and provided users
  with access to these files.

3
Veronica -- Grandmother of Search Engines

• Developed by the university of nevada system
  computing services group
• Veronica (very easy rodent-oriented netwide index
  to computerized archives)
• A searching device similar to archie but for
  gopher files
• Gopher servers contains plain-text documents (no
  images, no hypertext) that can be retrieved)
• Another one -- jughead (jonzys universal gopher
  hierarchy excavation and display)

4
World Wide Web Wanderer-- Mother of Search Engines

• Created by matthew gray
• First robot on the web
• Designed to track the webs growth
• Initially for web servers, later to capture urls
  too
• The database of captured urls became the wandex,
  the first web database.

5
What Is a Robot?

• Computer programs that automatically perform a
  repetitive task at speeds that would be
  impossible for humans to match, just like the
  task todays robots perform in factory.
• Programs that explore the internet for some sort
  of information. Web robots search the internet
  for web pages, usually for the purpose of
  compiling a large, searchable database.
• Web degradation robot access the same page
  hundreds of times of a day.

6
Invasion of the Spiders

• Mathew grays wanderer inspired a number of
  programmers to fellow on the idea of web robot.
• Jumpstation--title, header, search linearly,
  irrelevant result order
• WWW worm -- only titles and urls, irrelevant
  result order
• RSBE (repository-based software engineering)--
  first implementing a ranking system based on
  relevance to keyword string
• Excite (architext)-- search based on statistical
  analysis of word relationships

7
Yahoo! -- Searchable directory or Engine?

• Created by David Filo Jerry Yang (Stanford) in
  1994
• Searchable directory
• originally entries were entered and categorized
  mamually
• automated some aspects of the gathering and
  classification process
• contains additional descriptive information about
  the indexed sites.

8
Brians WebCrawler

• Created by Brian Pinkerton (U. of Washington) in
  1994
• Bought by America Online, then by Excite in 1997
• First full-text web search engine, be able to
  index the entire text of a web page

9
Mellon-Mania the birth of Lycos

• Created by Michael Mauldin in 1994 (Carnegie
  Mellon U.)
• provides ranked relevance retrieval
• provides prefix matching and word proximity bonus
• index first 20 lines of a document including
  http, gopher, ftp documents.

10
AltaVista

• Created by Digital Equipment Corporation in Dec.
  1995
• Speed -- handle millions of hits per day
• the first to use natural language queries
• first to implement advanced searching techniques,
  such as Boolean operators
• search newsgroup articles
• first search engine allows user add to or delete
  their own URLs from the index, placing them
  online within 24 hours
• ability to search for all of the sites that link
  to a particular URL

11
Hotbot

• Sponsored by Inktomi Corporation and HotWired in
  1996
• most powerful search engine, index 10 million
  pages per day
• use cookie technology to store personal search
  preference in formation
• a cookie is a small file that a site can store on
  you r own computer. It can be read only by the
  site that generate it.

12
What you have learned?

• Key players in the search engine area
• some of the issues that search engines face