Using Information Extraction for Question Answering

1
Using Information Extraction for Question
Answering

• Done by
• Rani Qumsiyeh

2
Problem

• More Information added to the web everyday.
• Search engines exist but they have a problem
• This calls for a different kind of search engine.

3
History of QA

• QA can be dated back to the 1960s
• Two common approaches to design QA
• Information Extraction
• Information Retrieval
• Two conferences to evaluate QA systems
• TREC (Text REtrieval Conference)
• MUC (Message Understanding Conference)

4
Common Issues with QA systems

• Information retrieval deals with keywords.
• Information extraction learns the question.
• The question could have multiple variations which
  means
• Easier for IR but more broad results
• Harder for IE but more EXACT results

5
Message Understanding Conference (MUC)

• Sponsored by the Defense Advanced Research
  Projects Agency (DARPA) 1991-1998.
• Developed methods for formal evaluation of IE
  systems
• In the form of a competition, where the
  participants compare their results with each
  other and against human annotators key
  templates.
• Short system preparation time to stimulate
  portability to new extraction problems. Only 1
  month to adapt the system to the new scenario
  before the formal run.

6
Evaluation Metrics

• Precision and recall
• Precision correct answers/answers produced
• Recall correct answers/total possible answers
• F-measure
• Where is a parameter representing relative
  importance of P R
• E.g., 1, then PR equal weight, 0, then
  only P
• Current State-of-Art F.60 barrier

7
MUC Extraction Tasks

• Named Entity task (NE)
• Template Element task (TE)
• Template Relation task (TR)
• Scenario Template task (ST)
• Coreference task (CO)

8
Named Entity Task (NE)

• Mark into the text each string that represents, a
  person, organization, or location name, or a date
  or time, or a currency or percentage figure

9
Template Element Task (TE)

• Extract basic information related to
  organization, person, and artifact entities,
  drawing evidence from everywhere in the text.

10
Template Relation task (TR)

• Extract relational information on employee_of,
  manufacture_of, location_of relations etc. (TR
  expresses domain independent relationships
  between entities identified by TE)

11
Scenario Template task (ST)

• Extract prespecified event information and relate
  the event information to particular organization,
  person, or artifact entities (ST identifies
  domain and task specific entities and relations)

12
Coreference task (CO)

• Capture information on corefering expressions,
  i.e. all mentions of a given entity, including
  those marked in NE and TE (Nouns, Noun phrases,
  Pronouns)

13
An Example

• The shiny red rocket was fired on Tuesday. It is
  the brainchild of Dr. Big Head. Dr. Head is a
  staff scientist at We Build Rockets Inc.
• NE entities are rocket, Tuesday, Dr. Head and We
  Build Rockets
• CO it refers to the rocket Dr. Head and Dr. Big
  Head are the same
• TE the rocket is shiny red and Heads brainchild
• TR Dr. Head works for We Build Rockets Inc.
• ST a rocket launching event occurred with the
  various participants.

14
Scoring templates

• Templates are compared on a slot-by-slot basis
• Correct response key
• Partial response key
• Incorrect response ! key
• Spurious key is blank
• overgenspurious/actual
• Missing response is blank

15
Maximum Results Reported
16
KnowitAll, TextRunner, KnowitNow

• Differ in implementation, but do the same thing.

17
Using them as QA systems

• Able to handle questions that produce 1 relation
• Who is the president of the US? can handle
• Who was the president of the US in 1998? fails
• Produces a huge number of facts that the user
  still has to go through.

18
Textract

• Aims at solving ambiguity in text by introducing
  more named entities.
• What is Julian Werver Hill's wife's telephone
  number?
• equivalent to What is Polly's telephone number?
• Where is Werver Hill's affiliated company
  located?
• equivalent to Where is Microsoft located?

19
Proposed System

• Determine what named entity we are looking for
  using Textract.
• Use Part of Speech tagging.
• Use TextRunner as the basis for search.
• Use WordNet to find synonyms.
• Use extra entities in text as constraints

20
Example
21
Example

• (WP who) (VBD was) (DT the) (JJ first) (NN man)
  (TO to) (VB land) (IN on) (DT the) (NN moon)
• The verb (VB) is treated as the argument.
• The noun (NN) is treated as the predicate
• We make sure that position is maintained
• We keep prepositions if they have two nouns.
  (president of the US)
• Other non stop words are constraints, i.e.,
  first

22
Example
23
Anaphora Resolution

• Use anaphora resolution to determine that landed
  is not associated with landed but wrote instead.

24
Use Synonyms

• We use word net to find possible synonyms for
  verbs and nouns to produce more facts.
• We only consider 3 synonyms as it takes more time
  the more fact retrievals we have to do.

25
Using constraints
26
Delimitations

• Works well with Who, When, Where questions as
  named entity is easily determined.
• Achieves about 90 accuracy on all
• Works less well with What, How questions
• Achieves about 70 accuracy
• Takes about 13 seconds to answer question.

27
Future Work

• Build an ontology to determine named entity and
  parse question (faster)
• Handle combinations of questions.
• When and where did the holocaust happen?