Information Extraction

1
Information Extraction

• Adapted from slides by Junichi Tsujii, Ronen
  Feldman and others

2
Most Data are Unstructured (Text) or
Semi-Structured

• Email
• Insurance claims
• News articles
• Web pages
• Patent portfolios

• Customer complaint letters
• Contracts
• Transcripts of phone calls with customers
• Technical documents

Text data mining has become more and more
important
(Adapted from J. Dorre et al. Text Mining
Finding Nuggets in Mountains of Textual Data)
3
Application Tasks of NLP
(1)Information Retrieval/Detection
To search and retrieve documents in response to
queries for information
(2)Passage Retrieval
To search and retrieve part of documents in
response to queries for information
(3)Information Extraction
To extract information that fits pre-defined
database schemas or templates, specifying the
output formats
(4) Question/Answering Tasks
To answer general questions by using texts as
knowledge base Fact retrieval, combination of IR
and IE
(5)Text Understanding
To understand texts as people do Artificial
Intelligence
4
Information ExtractionA Pragmatic Approach

• Let application requirements drive semantic
  analysis
• Identify the types of entities that are relevant
  to a particular task
• Identify the range of facts that one is
  interested in for those entities
• Ignore everything else

5
IE definitions

• Entity an object of interest such as a person or
  organization
• Attribute A property of an entity such as name,
  alias, descriptor or type
• Fact A relationship held between two or more
  entities such as Position of Person in Company
• Event An activity involving several entities
  such as terrorist act, airline crash, product
  information

6
IE accuracy typical figures by information type

• Entity 90-98
• Attribute 80
• Fact 60-70
• Event 50-60

7
MUC conferences

• MUC 1 to MUC 7
• 1987 to 1997
• Topics
• Naval operations (2)
• Terrorist Activity (2)
• Joint venture and microelectronics
• Management changes
• Space Vehicles and Missile launches

8
MUC and Scenario Templates

• Define a set of interesting entities
• Persons, organizations, locations
• Define a complex scenario involving interesting
  events and relations over entities
• Example management succession persons,
  companies, positions, reasons for succession
• This collection of entities and relations is
  called a scenario template.

9
Problems with Scenario Template

• Encouraged development of highly domain specific
  ontologies, rule systems, heuristics, etc.
• Most of the effort expended on building a
  scenario template system was not directly
  applicable to a different scenario template.

10
Addressing the Problem

• Address a large number of smaller, more focused
  scenario templates (Event-99)
• Develop a more systematic ground-up approach to
  semantics by focusing on elementary entities,
  relations, and events (ACE)

11
The ACE Evaluation

• The ACE program challenge of extracting content
  from human language. Research effort directed to
  master
• first the extraction of entities
• Then the extraction of relations among these
  entities
• Finally the extraction of events that are
  causally related sets of relations
• After two years, top systems successfully capture
  well over 50 of the value at the entity level

12
The ACE Program

• Automated Content Extraction
• Develop core information extraction technology by
  focusing on extracting specific semantic entities
  and relations over a very wide range of texts.
• Corpora Newswire and broadcast transcripts, but
  broad range of topics and genres.
• Third person reports
• Interviews
• Editorials
• Topics foreign relations, significant events,
  human interest, sports, weather
• Discourage highly domain- and genre-dependent
  solutions

13
Applications of IE

• Routing of information
• Infrastructure for IR and categorization (higher
  level features)
• Event based summarization
• Automatic creation of databases and knowledge
  bases

14
Where would IE be useful?

• Semi-structured text
• Generic documents like news articles
• Most of the information in the doc is centered
  around a set of easily identifiable entities

15
The Problem
Date
Time Start - End
Location
Speaker
Person
16
What is Information Extraction
As a task
Filling slots in a database from sub-segments of
text.
October 14, 2002, 400 a.m. PT For years,
Microsoft Corporation CEO Bill Gates railed
against the economic philosophy of open-source
software with Orwellian fervor, denouncing its
communal licensing as a "cancer" that stifled
technological innovation. Today, Microsoft
claims to "love" the open-source concept, by
which software code is made public to encourage
improvement and development by outside
programmers. Gates himself says Microsoft will
gladly disclose its crown jewels--the coveted
code behind the Windows operating system--to
select customers. "We can be open source. We
love the concept of shared source," said Bill
Veghte, a Microsoft VP. "That's a super-important
shift for us in terms of code access. Richard
Stallman, founder of the Free Software
Foundation, countered saying
NAME TITLE ORGANIZATION
Courtesy of William W. Cohen
17
What is Information Extraction
As a task
Filling slots in a database from sub-segments of
text.
October 14, 2002, 400 a.m. PT For years,
Microsoft Corporation CEO Bill Gates railed
against the economic philosophy of open-source
software with Orwellian fervor, denouncing its
communal licensing as a "cancer" that stifled
technological innovation. Today, Microsoft
claims to "love" the open-source concept, by
which software code is made public to encourage
improvement and development by outside
programmers. Gates himself says Microsoft will
gladly disclose its crown jewels--the coveted
code behind the Windows operating system--to
select customers. "We can be open source. We
love the concept of shared source," said Bill
Veghte, a Microsoft VP. "That's a super-important
shift for us in terms of code access. Richard
Stallman, founder of the Free Software
Foundation, countered saying
IE
NAME TITLE ORGANIZATION Bill Gates
CEO Microsoft Bill Veghte VP
Microsoft Richard Stallman founder Free
Soft..
Courtesy of William W. Cohen
18
What is Information Extraction
Information Extraction segmentation
classification association clustering
October 14, 2002, 400 a.m. PT For years,
Microsoft Corporation CEO Bill Gates railed
against the economic philosophy of open-source
software with Orwellian fervor, denouncing its
communal licensing as a "cancer" that stifled
technological innovation. Today, Microsoft
claims to "love" the open-source concept, by
which software code is made public to encourage
improvement and development by outside
programmers. Gates himself says Microsoft will
gladly disclose its crown jewels--the coveted
code behind the Windows operating system--to
select customers. "We can be open source. We
love the concept of shared source," said Bill
Veghte, a Microsoft VP. "That's a super-important
shift for us in terms of code access. Richard
Stallman, founder of the Free Software
Foundation, countered saying
Microsoft Corporation CEO Bill Gates Microsoft Gat
es Microsoft Bill Veghte Microsoft VP Richard
Stallman founder Free Software Foundation
aka named entity extraction
Courtesy of William W. Cohen
19
What is Information Extraction
Information Extraction segmentation
classification association clustering
October 14, 2002, 400 a.m. PT For years,
Microsoft Corporation CEO Bill Gates railed
against the economic philosophy of open-source
software with Orwellian fervor, denouncing its
communal licensing as a "cancer" that stifled
technological innovation. Today, Microsoft
claims to "love" the open-source concept, by
which software code is made public to encourage
improvement and development by outside
programmers. Gates himself says Microsoft will
gladly disclose its crown jewels--the coveted
code behind the Windows operating system--to
select customers. "We can be open source. We
love the concept of shared source," said Bill
Veghte, a Microsoft VP. "That's a super-important
shift for us in terms of code access. Richard
Stallman, founder of the Free Software
Foundation, countered saying
Microsoft Corporation CEO Bill Gates Microsoft Gat
es Microsoft Bill Veghte Microsoft VP Richard
Stallman founder Free Software Foundation
Courtesy of William W. Cohen
20
What is Information Extraction
Information Extraction segmentation
classification association clustering
October 14, 2002, 400 a.m. PT For years,
Microsoft Corporation CEO Bill Gates railed
against the economic philosophy of open-source
software with Orwellian fervor, denouncing its
communal licensing as a "cancer" that stifled
technological innovation. Today, Microsoft
claims to "love" the open-source concept, by
which software code is made public to encourage
improvement and development by outside
programmers. Gates himself says Microsoft will
gladly disclose its crown jewels--the coveted
code behind the Windows operating system--to
select customers. "We can be open source. We
love the concept of shared source," said Bill
Veghte, a Microsoft VP. "That's a super-important
shift for us in terms of code access. Richard
Stallman, founder of the Free Software
Foundation, countered saying
Microsoft Corporation CEO Bill Gates Microsoft Gat
es Microsoft Bill Veghte Microsoft VP Richard
Stallman founder Free Software Foundation
Courtesy of William W. Cohen
21
What is Information Extraction
Information Extraction segmentation
classification association clustering
October 14, 2002, 400 a.m. PT For years,
Microsoft Corporation CEO Bill Gates railed
against the economic philosophy of open-source
software with Orwellian fervor, denouncing its
communal licensing as a "cancer" that stifled
technological innovation. Today, Microsoft
claims to "love" the open-source concept, by
which software code is made public to encourage
improvement and development by outside
programmers. Gates himself says Microsoft will
gladly disclose its crown jewels--the coveted
code behind the Windows operating system--to
select customers. "We can be open source. We
love the concept of shared source," said Bill
Veghte, a Microsoft VP. "That's a super-important
shift for us in terms of code access. Richard
Stallman, founder of the Free Software
Foundation, countered saying
Microsoft Corporation CEO Bill Gates Microsoft Gat
es Microsoft Bill Veghte Microsoft VP Richard
Stallman founder Free Software Foundation




Courtesy of William W. Cohen
22
Landscape of IE TasksSingle Field/Record
Jack Welch will retire as CEO of General Electric
tomorrow. The top role at the Connecticut
company will be filled by Jeffrey Immelt.
Single entity
Binary relationship
N-ary record
Person Jack Welch
Relation Person-Title Person Jack
Welch Title CEO
Relation Succession Company General
Electric Title CEO Out
Jack Welsh In Jeffrey Immelt
Person Jeffrey Immelt
Relation Company-Location Company General
Electric Location Connecticut
Location Connecticut
Named entity extraction
23
Landscape of IE Techniques
Lexicons
Abraham Lincoln was born in Kentucky.
member?
Alabama Alaska Wisconsin Wyoming
Courtesy of William W. Cohen
24
IE with Hidden Markov Models
Given a sequence of observations
Yesterday Pedro Domingos spoke this example
sentence.
and a trained HMM
person name
location name
background
Find the most likely state sequence (Viterbi)
Yesterday Pedro Domingos spoke this example
sentence.
Any words said to be generated by the designated
person name state extract as a person name
Person name Pedro Domingos
25
HMM for Segmentation

• Simplest Model One state per entity type

26
Discriminative Approaches
Yesterday Pedro Domingos spoke this example
sentence.
Is this phrase (X) a name? Y1 (yes) Y0
(no) Learn from many examples to predict Y from X
parameters
Maximum Entropy, Logistic Regression
Features (e.g., is the phrase capitalized?)
More sophisticated Consider dependency between
different labels (e.g. Conditional Random Fields)
27
Example of IE FASTUS(1993)
28
Example of IE FASTUS(1993)
29
Example of IE FASTUS(1993)
30
Example of IE FASTUS(1993)
31
Example of IE FASTUS(1993)
32
FASTUS
Based on finite states automata (FSA)
1.Complex Words Recognition of multi-words and
proper names
set up new Twaiwan dallors
2.Basic Phrases Simple noun groups, verb groups
and particles
a Japanese trading house had set up
3.Complex phrases Complex noun groups and verb
groups
4.Domain Events Patterns for events of interest
to the application Basic templates are to be
built.
5. Merging Structures Templates from different
parts of the texts are merged if they provide
information about the same entity or event.
33
Example of IE FASTUS(1993)
34
Information Extraction
. Jurgen Pfrang, 51, reportedly stumbled upon
the robbers on the second floor of his Nanjing
home early on Sunday. The deputy general manager
of Yaxing Benz, a Sino-German joint venture that
makes buses and bus chassis in nearby
Yangzhou, was hacked to death with 45 cm
watermelon knives. .
Name of the Venture Yaxing Benz Products
buses and bus chassis Location
Yangzhou,China Companies involved
(1)Name X?
Country German
(2)Name Y?
Country China

35
Information Extraction
A German vehicle-firm executive was stabbed to
death . . Jurgen Pfrang, 51, reportedly
stumbled upon the robbers on the second floor of
his Nanjing home early on Sunday. The deputy
general manager of Yaxing Benz, a Sino-German
joint venture that makes buses and bus chassis
in nearby Yangzhou, was hacked to death with 45
cm watermelon knives. .
Crime-Type Murder Type
Stabbing The killed Name Jurgen Pfrang
Age 51
Profession Deputy general
manager Location Nanjing, China

Different template for crimes
36
Interpretation of Texts
(1)Information Retrieval/Detection
(2)Passage Retrieval

(3)Information Extraction
(4) Question/Answering Tasks
(5)Text Understanding
37
IR System
Collection of Texts
38
IR System
Collection of Texts
39
Passage IR System
Collection of Texts
40
Passage IR System
IE System
Collection of Texts
Texts
41
IE System
Templates
Texts
42
IE as compromise NLP
Interpretation
IE System
Templates
Texts
Predefined
43
Performance Evaluation
(1)Information Retrieval/Detection
(2)Passage Retrieval

(3)Information Extraction
(4) Question/Answering Tasks
(5)Text Understanding
44
Collection of Documents
45
Collection of Documents
More complicated due to partially filled
templates
46
Framework of IE
IE as compromise NLP
47
Difficulties of NLP
General Framework of NLP
(1) Robustness Incomplete Knowledge
Morphological and Lexical Processing
Syntactic Analysis
Semantic Analysis
Incomplete Domain Knowledge Interpretation
Rules
Context processing Interpretation
48
Difficulties of NLP
General Framework of NLP
(1) Robustness Incomplete Knowledge
Morphological and Lexical Processing
Syntactic Analysis
Semantic Analysis
Incomplete Domain Knowledge Interpretation
Rules
Context processing Interpretation
49
Approaches for building IE systems

• Knowledge Engineering Approach
• Rules crafted by linguists in cooperation with
  domain experts
• Most of the work done by insoecting a set of
  relevant documents

50
Approaches for building IE systems

• Automatically trainable systems
• Techniques based on statistics and almost no
  linguistic knowledge
• Language independent
• Main input annotated corpus
• Small effort for creating rules, but crating
  annotated corpus laborious

51
Techniques in IE
(1) Domain Specific Partial Knowledge
Knowledge relevant to information to be extracted
(2) Ambiguities Ignoring irrelevant
ambiguities Simpler NLP techniques
(3) Robustness Coping with Incomplete
dictionaries (open
class words) Ignoring irrelevant parts of
sentences
(4) Adaptation Techniques Machine
Learning, Trainable systems
52
General Framework of NLP
Open class words Named entity recognition
(ex) Locations Persons
Companies Organizations
Position names
Morphological and Lexical Processing
Syntactic Analysis
Semantic Anaysis
Domain specific rules ltWordgtltWordgt, Inc.
Mr. ltCpt-Lgt. ltWordgt Machine Learning
HMM, Decision Trees Rules Machine Learning
Context processing Interpretation
53
FASTUS
General Framework of NLP
Based on finite states automata (FSA)
1.Complex Words Recognition of multi-words and
proper names
Morphological and Lexical Processing
2.Basic Phrases Simple noun groups, verb groups
and particles
Syntactic Analysis
3.Complex phrases Complex noun groups and verb
groups
4.Domain Events Patterns for events of interest
to the application Basic templates are to be
built.
Semantic Anaysis
Context processing Interpretation
5. Merging Structures Templates from different
parts of the texts are merged if they provide
information about the same entity or event.
54
FASTUS
General Framework of NLP
Based on finite states automata (FSA)
1.Complex Words Recognition of multi-words and
proper names
Morphological and Lexical Processing
2.Basic Phrases Simple noun groups, verb groups
and particles
Syntactic Analysis
3.Complex phrases Complex noun groups and verb
groups
4.Domain Events Patterns for events of interest
to the application Basic templates are to be
built.
Semantic Anaysis
Context processing Interpretation
5. Merging Structures Templates from different
parts of the texts are merged if they provide
information about the same entity or event.
55
FASTUS
General Framework of NLP
Based on finite states automata (FSA)
1.Complex Words Recognition of multi-words and
proper names
Morphological and Lexical Processing
2.Basic Phrases Simple noun groups, verb groups
and particles
Syntactic Analysis
3.Complex phrases Complex noun groups and verb
groups
4.Domain Events Patterns for events of interest
to the application Basic templates are to be
built.
Semantic Analysis
Context processing Interpretation
5. Merging Structures Templates from different
parts of the texts are merged if they provide
information about the same entity or event.
56
Chomsky Hierarchy Hierarchy of
Grammar of Automata Regular
Grammar Finite State
Automata Context Free Grammar
Push Down Automata Context Sensitive Grammar
Linear Bounded Automata Type 0
Grammar Turing
Machine
57
Chomsky Hierarchy Hierarchy of
Grammar of Automata Regular
Grammar Finite State
Automata Context Free Grammar Push
Down Automata Context Sensitive Grammar
Linear Bounded Automata Type 0 Grammar
Turing Machine
58
1
s
PN
Art
2
0
ADJ
N
Art
s
3
Johns interesting book with a nice cover
P
4
PN
59
1
s
PN
Art
2
0
ADJ
N
Art
s
3
Johns interesting book with a nice cover
P
4
PN
60
1
s
PN
Art
2
0
ADJ
N
Art
s
3
Johns interesting book with a nice cover
P
4
PN
61
1
s
PN
Art
2
0
ADJ
N
Art
s
3
Johns interesting book with a nice cover
P
4
PN
62
1
s
PN
Art
2
0
ADJ
N
Art
s
3
Johns interesting book with a nice cover
P
4
PN
63
1
s
PN
Art
2
0
ADJ
N
Art
s
3
Johns interesting book with a nice cover
P
4
PN
64
1
s
PN
Art
2
0
ADJ
N
Art
s
3
Johns interesting book with a nice cover
P
4
PN
65
1
s
PN
Art
2
0
ADJ
N
Art
s
3
Johns interesting book with a nice cover
P
4
PN
66
1
s
PN
Art
2
0
ADJ
N
Art
s
3
Johns interesting book with a nice cover
P
4
PN
67
1
s
PN
Art
2
0
ADJ
N
Art
s
3
Johns interesting book with a nice cover
P
4
PN
68
Pattern-maching PN s (ADJ) N P Art (ADJ) N
PN s/ Art(ADJ) N(P Art (ADJ) N)
1
s
PN
Art
2
0
ADJ
N
Art
s
3
Johns interesting book with a nice cover
P
4
PN
69
FASTUS
General Framework of NLP
Based on finite states automata (FSA)
1.Complex Words Recognition of multi-words and
proper names
Morphological and Lexical Processing
2.Basic Phrases Simple noun groups, verb groups
and particles
Syntactic Analysis
3.Complex phrases Complex noun groups and verb
groups
4.Domain Events Patterns for events of interest
to the application Basic templates are to be
built.
Semantic Analysis
Context processing Interpretation
5. Merging Structures Templates from different
parts of the texts are merged if they provide
information about the same entity or event.
70
Example of IE FASTUS(1993)
1.Complex words
2.Basic Phrases Bridgestone Sports Co.
Company name said
Verb Group Friday
Noun Group it
Noun Group had set up
Verb Group a joint venture
Noun Group in
Preposition Taiwan
Location
71
Example of IE FASTUS(1993)


1.Complex words
2.Basic Phrases Bridgestone Sports Co.
Company name said
Verb Group Friday
Noun Group it
Noun Group had set up
Verb Group a joint venture
Noun Group in
Preposition Taiwan
Location
a Japanese tea house a Japanese tea house a
Japanese tea house
72
Example of IE FASTUS(1993)
1.Complex words
2.Basic Phrases Bridgestone Sports Co.
Company name said
Verb Group Friday
Noun Group it
Noun Group had set up
Verb Group a joint venture
Noun Group in
Preposition Taiwan
Location
73
Example of IE FASTUS(1993)
3.Complex Phrases
2.Basic Phrases Bridgestone Sports Co.
Company name said
Verb Group Friday
Noun Group it
Noun Group had set up
Verb Group a joint venture
Noun Group in
Preposition Taiwan
Location
74
Example of IE FASTUS(1993)
3.Complex Phrases
2.Basic Phrases Bridgestone Sports Co.
Company name said
Verb Group Friday
Noun Group it
Noun Group had set up
Verb Group a joint venture
Noun Group in
Preposition Taiwan
Location
Some syntactic structures like
75
Example of IE FASTUS(1993)
3.Complex Phrases
2.Basic Phrases Bridgestone Sports Co.
Company name said
Verb Group Friday
Noun Group it
Noun Group had set up
Verb Group a joint venture
Noun Group in
Preposition Taiwan
Location
Syntactic structures relevant to information to
be extracted are dealt with.
76
Syntactic variations
GM set up a joint venture with Toyota. GM
announced it was setting up a joint venture with
Toyota. GM signed an agreement setting up a joint
venture with Toyota. GM announced it was signing
an agreement to set up a joint venture with
Toyota.
77
Syntactic variations
GM set up a joint venture with Toyota. GM
announced it was setting up a joint venture with
Toyota. GM signed an agreement setting up a joint
venture with Toyota. GM announced it was signing
an agreement to set up a joint venture with
Toyota.
GM plans to set up a joint venture with
Toyota. GM expects to set up a joint venture with
Toyota.
78
Syntactic variations
GM set up a joint venture with Toyota. GM
announced it was setting up a joint venture with
Toyota. GM signed an agreement setting up a joint
venture with Toyota. GM announced it was signing
an agreement to set up a joint venture with
Toyota.
S
NP
VP
GM
V
set up
GM plans to set up a joint venture with
Toyota. GM expects to set up a joint venture with
Toyota.
79
Example of IE FASTUS(1993)
3.Complex Phrases 4.Domain Events COMPANYSET-U
PJOINT-VENTUREwithCOMPNY COMPANYSET-UPJO
INT-VENTURE (others) withCOMPNY
80
Complications caused by syntactic variations
Relative clause The mayor, who was kidnapped
yesterday, was found dead today.
NG Relpro NG/others VG NG/othersVG N
G Relpro NG/others VG
81
Complications caused by syntactic variations
Relative clause The mayor, who was kidnapped
yesterday, was found dead today.
NG Relpro NG/others VG NG/othersVG N
G Relpro NG/others VG
82
Complications caused by syntactic variations
Relative clause The mayor, who was kidnapped
yesterday, was found dead today.
NG Relpro NG/others VG NG/othersVG N
G Relpro NG/others VG
83
FASTUS
Based on finite states automata (FSA)
NP, who was kidnapped, was found.
1.Complex Words
2.Basic Phrases
3.Complex phrases
4.Domain Events Patterns for events of interest
to the application Basic templates are to be
built.
5. Merging Structures Templates from different
parts of the texts are merged if they provide
information about the same entity or event.
84
FASTUS
Based on finite states automata (FSA)
NP, who was kidnapped, was found.
1.Complex Words
2.Basic Phrases
3.Complex phrases
4.Domain Events Patterns for events of interest
to the application Basic templates are to be
built.
Piece-wise recognition of basic templates
5. Merging Structures Templates from different
parts of the texts are merged if they provide
information about the same entity or event.
Reconstructing information carried via syntactic
structures by merging basic templates
85
FASTUS
Based on finite states automata (FSA)
NP, who was kidnapped, was found.
1.Complex Words
2.Basic Phrases
3.Complex phrases
4.Domain Events Patterns for events of interest
to the application Basic templates are to be
built.
Piece-wise recognition of basic templates
5. Merging Structures Templates from different
parts of the texts are merged if they provide
information about the same entity or event.
Reconstructing information carried via syntactic
structures by merging basic templates
86
FASTUS
Based on finite states automata (FSA)
NP, who was kidnapped, was found.
1.Complex Words
2.Basic Phrases
3.Complex phrases
4.Domain Events Patterns for events of interest
to the application Basic templates are to be
built.
Piece-wise recognition of basic templates
5. Merging Structures Templates from different
parts of the texts are merged if they provide
information about the same entity or event.
Reconstructing information carried via syntactic
structures by merging basic templates
87
Current state of the arts of IE

• Carefully constructed IE systems
• F-60 level (interannotater agreement
  60-80)
• Domain telegraphic messages about naval
  operation
• (MUC-187, MUC-289)
• news articles and
  transcriptions of radio broadcasts
• Latin American terrorism
  (MUC-391, MUC-41992)
• News articles about joint
  ventures (MUC-5, 93)
• News articles about
  management changes (MUC-6, 95)
• News articles about space
  vehicle (MUC-7, 97)
• Handcrafted rules (named entity recognition,
  domain events, etc)

Automatic learning from texts Supervised
learning corpus preparation
Non-supervised, or controlled learning