Micro Miner

1
Micro Miner

• A Scalable Web Search Engine

A project by
Venu Kurugandhi (Project Manager) Viral
Masarani Shree Kumar Menon Chen, Yanjiao Vivek
Kodan
2
QUICK OVERVIEW

• System to search the web.
• Input user search key word.
• Output URL and possible keyword matches.

3
Google way
4
How

• Search Database
• Retrieve matching urls

• The Database consists of
• Descriptor
• URL

5
Objective

• Build Database for the Web Search from the
  database given by various Clients
• Use of algorithm to search the resulting database
• Use of KWIC (Key Word In Context ) Tool
• which is used to prepare the database

6
KWICModules

• The KWIC system comprises the following modules
• 1. Applet Provides the user interface. It
  provides the input string to the Input Manager
  and displays the alphabetized circularly shifted
  lines.
• 2. Input Manager It is the main control unit and
  invokes each of the modules sequentially. It is
  also responsible for getting input from Applet,
  storing it, passing input to various modules and
  displaying results at last.
• 3. Circular Shift Creates circular shifts of the
  stored lines.
• 4. Alphabetizer Creates alphabetized lines of
  the circular shifted lines.

7
KWICAbstract Data Type Architecture

• Abstract Data type is an Object Oriented
  approach. The main characteristic of this style
  is that data is encapsulated in objects,
  representing the actual components of the system.
  In ADT, Data is no longer directly shared by the
  process components. Instead, each module provides
  an interface. Other components access data only
  by invoking that interface. Objects have certain
  characteristics
• Each object
• Handles the representation of the data
• Provides procedural interface for accessing the
  data

8
KWICKWIC ADT Architecture
9
KWIC Pipe and Filter Style Architecture

• In a pipe and filter architecture, each component
  has a set of inputs and outputs. A component in
  the architecture reads streams of data on its
  inputs and produces a stream of data on its
  outputs. The components are termed filters, and
  the connecters which serve as the conduits for
  information transfer are called the pipes.
• Pipes move stream data from a filter output to a
  filter input, but the flow is uni-directional.
  The Filter transforms some amount of data from
  the inputs, and transmits that data to its
  outputs. Filters do not share their state with
  other filters. This architecture can be
  represented as follows.

10
KWICPipe and Filter Architecture Contd..
11
KWICPipe and Filter Architecture Contd..

• Advantages
• Pipe and filter architecture maintains the
  intuitive flow of processing.
• Reuse It supports reuse as each filter is
  independent and new filters can be added for new
  functions.
• Modifiability It is easy to modify the system.
  The change of one filter will not affect other
  filters since they are independent.
• Enhanceability The system can be easily
  enhanced, as new functions can be added by adding
  new filters and old filters can be replaced by
  improved ones
• Understandability. It is easy to understand the
  overall input/output behavior of the system as
  simple composition of the behavior of the
  individual filters. They naturally support
  concurrent execution.

12
Architecture Comparison
13
Example
Descriptor best email service provider free
encyclopedia in english enterprise library for
dot net
URL www.yahoo.com www.wikipedia.org www.msdn.com
KWIC
14
Contd..
Phase 1

• Circular Shift

15
Contd..

• Alphabetical Sorting

string service provider
16
Screen Shots
17
Contd
18
Screen Shots
19
Contd
20
Contd..
21
Merits and Demerits

• De-Merits
• Consumes large amount of memory
• Time consuming while building up database
• Merits
• Fast response
• Easy to modify
• High reusability

22

• Thank You.