Two Related Approaches to the Problem of Textual Inference

1
Two Related Approachesto the Problem of Textual
Inference

• Bill MacCartney
• NLP Group
• Stanford University
• 6 March 2008

2
The textual inference task

• Does premise P justify an inference to hypothesis
  H?
• An informal, intuitive notion of inference not
  strict logic
• Focus on local inference steps, not long chains
  of deduction
• Emphasis on variability of linguistic expression
• Robust, accurate textual inference could enable
• Semantic search
• H lobbyists attempting to bribe U.S.
  legislatorsP The A.P. named two more senators
  who received contributions engineered by
  lobbyist Jack Abramoff in return for political
  favors
• Question answering Harabagiu Hickl 06
• H Who bought JDE? P Thanks to its recent
  acquisition of JDE, Oracle will soon
• Relation extraction (database building)
• Document summarization

3
A two-part talk

• The Stanford RTE system
• Describes a system to which I am one of many
  contributors
• Starts by aligning dependency trees of premise
  hypothesis
• Then extracts global, semantic features and
  classifies entailment
• A talk presented at NAACL-06 (with updated
  results)
• The NatLog system natural logic for textual
  inference
• Describes a system of which I am the principal
  author
• Assumes an alignment, but interprets as an edit
  sequence
• Classifies entailments across each edit
  composes results
• A talk presented at WTEP-07 (ACL workshop),
  updated

4
The Stanford RTE system
Introduction Foundations of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion

• to other slide deck

5
Containment, Exclusion, and ImplicativityA
Model of Natural Logic for Textual Inference

• Bill MacCartney and Christopher D. Manning
• NLP Group
• Stanford University
• 6 March 2008

6
Some simple inferences
Introduction Foundations of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion
No state completely forbids casino gambling.
What kind of textual inference system could
predict this?
7
Textual inferencea spectrum of approaches
Introduction Foundations of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion
deep,but brittle
naturallogic
robust,but shallow
8
What is natural logic?
Introduction Foundations of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion

• (natural logic ? natural deduction)
• Lakoff (1970) defines natural logic as a goal
  (not a system)
• to characterize valid patterns of reasoning via
  surface forms (syntactic forms as close as
  possible to natural language)
• without translation to formal notation ? ? ? ? ?
  ?
• A long history
• traditional logic Aristotles syllogisms,
  scholastics, Leibniz,
• van Benthem Sánchez Valencia (1986-91)
  monotonicity calculus
• Precise, yet sidesteps difficulties of
  translating to FOL
• idioms, intensionality and propositional
  attitudes, modalities, indexicals,
  reciprocals,scope ambiguities, quantifiers such
  as most, reciprocals, anaphoric adjectives,
  temporal and causal relations, aspect,
  unselective quantifiers, adverbs of
  quantification, donkey sentences, generic
  determiners,

9
Monotonicity calculus (Sánchez Valencia 1991)
Introduction Foundations of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion

• Entailment as semantic containment
• rat lt rodent, eat lt consume, this morning lt
  today, most lt some
• Monotonicity classes for semantic functions
• Upward monotone some rats dream lt some rodents
  dream
• Downward monotone no rats dream gt no rodents
  dream
• Non-monotone most rats dream most rodents dream

• Handles even nested inversions of monotonicity
• Every state forbids shooting game without a
  hunting license

• But lacks any representation of exclusion
  (negation, antonymy, )
• Garfield is a cat lt Garfield is not a dog

10
Implicatives factives
Introduction Foundations of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion

• Work at PARC, esp. Nairn et al. 2006
• Explains inversions nestings of implicatives
  factives
• Ed did not forget to force Dave to leave ? Dave
  left
• Defines 9 implication signatures
• Implication projection algorithm
• Bears some resemblance to monotonicity calculus
• But, fails to connect to containment or
  monotonicity
• John refused to dance ? John didnt tango

11
Outline
Introduction Foundations of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion

• Introduction
• Foundations of Natural Logic
• The NatLog System
• Experiments with FraCaS
• Experiments with RTE
• Conclusion

12
A new theory of natural logic
Introduction Foundations of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion

• Three elements

• an inventory of entailment relations
• semantic containment relations of Sánchez
  Valencia
• plus semantic exclusion relations
• a concept of projectivity
• explains entailments compositionally
• generalizes Sánchez Valencias monotonicity
  classes
• generalizes Nairn et al.s implication signatures
• a weak proof procedure
• composes entailment relations across chains of
  edits

13
Entailment relations in past work
Introduction Foundations of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion
14
16 elementary set relations
Introduction Foundations of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion
Q
?Q
?P
P
P and Q can representsets of entities (i.e.,
predicates)or of possible worlds
(propositions)cf. Tarskis relation algebra
15
16 elementary set relations
Introduction Foundations of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion
Q
?Q
?P
P
P and Q can representsets of entities (i.e.,
predicates)or of possible worlds
(propositions)cf. Tarskis relation algebra
16
7 basic entailment relations
Introduction Foundations of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion
Relations are defined for all semantic types
tiny lt small, hover lt fly, kick lt strike,this
morning lt today, in Beijing lt in China, everyone
lt someone, all lt most lt some
17
Projectivity ( monotonicity)
Introduction Foundations of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion

• How do the entailments of a compound expression
  depend on the entailments of its parts?
• How does the entailment relation between (f x)
  and (f y) depend on the entailment relation
  between x and y(and the properties of f)?
• Monotonicity gives partial answer (for , lt, gt,
  )
• But what about the other relations (, , _)?
• Well categorize semantic functions based on how
  they project the basic entailment relations

18
Example projectivity of not
Introduction Foundations of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion
19
Example projectivity of refuse
Introduction Foundations of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion
20
Projecting entailment relations upward
Introduction Foundations of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion
Nobody can enter without a shirt lt Nobody can
enter without clothes

• Assume idealized semantic composition trees

• Propagate lexical entailment relations upward,
  according to projectivity class of each node on
  path to root

21
A weak proof procedure
Introduction Foundations of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion

• Find sequence of edits connecting P and H
• Insertions, deletions, substitutions,
• Determine lexical entailment relation for each
  edit
• Substitutions depends on meaning of
  substituends cat dog
• Deletions lt by default red socks lt socks
• But some deletions are special not hungry
  hungry
• Insertions are symmetric to deletions gt by
  default
• Project up to find entailment relation across
  each edit
• Compose entailment relations across sequence of
  edits

22
Composing entailment relations
Introduction Foundations of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion

• Relation composition if a R b and b S c, then a
  ? c
• cf. Tarskis relation algebra
• Many compositions are intuitive
• º ? lt º lt ? lt lt º ? lt
  º ?
• Some less obvious, but still accessible
• º ? lt fish human, human nonhuman,
  fish lt nonhuman
• But some yield unions of basic entailment
  relations!
• º ? , lt, gt, , (i.e. the
  non-exhaustive relations)
• Larger unions convey less information (can
  approx. with )
• This limits power of proof procedure described

?
23
Implicatives factives
Introduction Foundations of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion

• Nairn et al. 2006 define nine implication
  signatures
• These encode implications (, , o) in and
  contexts
• Refuse has signature /orefuse to dance implies
  didnt dancedidnt refuse to dance implies
  neither danced nor didnt dance
• Signatures generate different relations when
  deleted
• Deleting /o generates Jim refused to dance
  Jim dancedJim didnt refuse to dance _ Jim
  didnt dance
• Deleting o/ generates ltJim attempted to dance lt
  Jim dancedJim didnt attempt to dance gt Jim
  didnt dance
• (Factives are only partly explained by this
  account)

24
Outline
Introduction Foundations of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion

• Introduction
• Foundations of Natural Logic
• The NatLog System
• Experiments with FraCaS
• Experiments with RTE
• Conclusion

25
The NatLog system
Introduction Foundations of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion
textual inference problem
linguistic analysis
1
alignment
2
lexical entailment classification
3
entailment projection
4
entailment composition
5
prediction
26
Step 1 Linguistic analysis
Introduction Foundations of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion

• Tokenize parse input sentences (future NER
  coref )
• Identify items w/ special projectivity
  determine scope
• Problem PTB-style parse tree ? semantic
  structure!

no pattern DT lt /Nno/ arg1 ?M on dominating
NP __ gt(NP) (NPproj !gt NP) arg2 ?M on
dominating S __ gt (Sproj !gt S)
No state completely forbids casino gambling

• Solution specify scope in PTB trees using Tregex
  Levy Andrew 06

27
Step 2 Alignment
Introduction Foundations of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion

• Phrase-based alignments symmetric, many-to-many
• Can view as sequence of atomic edits DEL, INS,
  SUB, MAT

• Ordering of edits defines path through
  intermediate forms
• Need not correspond to sentence order
• Decomposes problem into atomic entailment
  problems
• We havent (yet) invested much effort here
• Experimental results use alignments from other
  sources

28
Running example
Introduction Foundations of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion
OK, the example is contrived, but it compactly
exhibits containment, exclusion, and implicativity
29
Step 3 Lexical entailment classification
Introduction Foundations of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion

• Predict basic entailment relation for each edit,
  based solely on lexical features, independent of
  context
• Feature representation
• WordNet features synonymy, hyponymy, antonymy
• Other relatedness features Jiang-Conrath
  (WN-based), NomBank
• String and lemma similarity, based on Levenshtein
  edit distance
• Lexical category features prep, poss, art, aux,
  pron, pn, etc.
• Quantifier category features
• Implication signatures (for DEL edits only)
• Decision tree classifier
• Trained on 2,449 hand-annotated lexical
  entailment problems
• gt99 accuracy on training data captures
  relevant distinctions

30
Running example
Introduction Foundations of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion
31
Step 4 Entailment projection
Introduction Foundations of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion
32
Step 5 Entailment composition
Introduction Foundations of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion
33
Outline
Introduction Foundations of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion

• Introduction
• Foundations of Natural Logic
• The NatLog System
• Experiments with FraCaS
• Experiments with RTE
• Conclusion

34
The FraCaS test suite
Introduction Foundations of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion

• FraCaS mid-90s project in computational
  semantics
• 346 textbook examples of textual inference
  problems
• examples on next slide
• 9 sections quantifiers, plurals, anaphora,
  ellipsis,
• 3 possible answers yes, no, unknown (not
  balanced!)
• 55 single-premise, 45 multi-premise (excluded)

35
FraCaS examples
Introduction Foundations of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion
36
Results on FraCaS
Introduction Foundations of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion
37
Results on FraCaS
Introduction Foundations of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion
38
FraCaS confusion matrix
Introduction Foundations of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion
guess
gold
39
Outline
Introduction Foundations of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion

• Introduction
• Foundations of Natural Logic
• The NatLog System
• Experiments with FraCaS
• Experiments with RTE
• Conclusion

40
The RTE3 test suite
Introduction Foundations of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion

• RTE more natural textual inference problems
• Much longer premises average 35 words (vs. 11)
• Binary classification yes and no
• RTE problems not ideal for NatLog
• Many kinds of inference not addressed by NatLog
• paraphrase, temporal reasoning, relation
  extraction,
• Big edit distance ? propagation of errors from
  atomic model

41
RTE3 examples
Introduction Foundations of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion
42
Results on RTE3 data
Introduction Foundations of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion
(each data set contains 800 problems)

• Accuracy is unimpressive, but precision is
  relatively high
• Maybe we can achieve high precision on a subset?
• Strategy hybridize with broad-coverage RTE
  system
• As in Bos Markert 2006

43
A simple bag-of-words model
Introduction Foundations of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion
H
P
similarity scores on 0, 1for each pair of
words (I used a really simple-mindedsimilarity
function based onLevenshtein string-edit
distance)
max sim for each hyp word
how rare each word is
(max sim)IDF
?h P(hP)
44
Results on RTE3 data
Introduction Foundations of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion
(each data set contains 800 problems)
45
Combining BoW NatLog
Introduction Foundations of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion

• MaxEnt classifier
• BoW features P(HP), P(PH)
• NatLog features7 boolean features encoding
  predicted entailment relation

46
Results on RTE3 data
Introduction Foundations of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion
(each data set contains 800 problems)
47
Problem NatLog is too precise?
Introduction Foundations of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion

• Error analysis reveals a characteristic pattern
  of mistakes
• Correct answer is yes
• Number of edits is large (gt5) (this is typical
  for RTE)
• NatLog predicts lt or for all but one or two
  edits
• But NatLog predicts some other relation for
  remaining edits!
• Most commonly, it predicts gt for an insertion
  (e.g., RTE3_dev.71)
• Result of relation composition is thus , i.e. no
• Idea make it more forgiving, by adding features
• Number of edits
• Proportion of edits for which predicted relation
  is not lt or

48
Results on RTE3 data
Introduction Foundations of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion
(each data set contains 800 problems)
49
Outline
Introduction Foundations of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion

• Introduction
• Foundations of Natural Logic
• The NatLog System
• Experiments with FraCaS
• Experiments with RTE
• Conclusion

50
What natural logic cant do
Introduction Foundations of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion

• Not a universal solution for textual inference
• Many types of inference not amenable to natural
  logic
• Paraphrase Eve was let go Eve lost her job
• Verb/frame alternation he drained the oil lt the
  oil drained
• Relation extraction Aho, a trader at UBS lt Aho
  works for UBS
• Common-sense reasoning the sink overflowed lt the
  floor got wet
• etc.
• Also, has a weaker proof theory than FOL
• Cant explain, e.g., de Morgans laws for
  quantifiers
• Not all birds fly Some birds dont fly

51
What natural logic can do
Introduction Foundations of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion

• Natural logic enables precise reasoning about
  containment, exclusion, and implicativity, while
  sidestepping the difficulties of translating to
  FOL.
• The NatLog system successfully handles a broad
  range of such inferences, as demonstrated on the
  FraCaS test suite.
• Ultimately, open-domain textual inference is
  likely to require combining disparate reasoners,
  and a facility for natural logic is a good
  candidate to be a component of such a system.
• Future work phrase-based alignment for textual
  inference