Modeling Semantic Containment and Exclusion in Natural Language Inference

1
Modeling Semantic Containment and Exclusion in
Natural Language Inference

• Bill MacCartney and Christopher D. Manning
• NLP Group
• Stanford University
• 22 August 2008

2
Natural language inference (NLI)
Introduction A Theory of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion

• Aka recognizing textual entailment (RTE)
• Does premise P justify an inference to hypothesis
  H?
• An informal, intuitive notion of inference not
  strict logic
• Emphasis on variability of linguistic expression

P Every firm polled saw costs grow more than
expected,even after adjusting for
inflation. H Every big company in the poll
reported cost increases. yes

• Necessary to goal of natural language
  understanding (NLU)
• Can also enable semantic search, question
  answering,

3
NLI a spectrum of approaches
Introduction A Theory of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion
Solution?
Problemhard to translate NL to FOL idioms,
anaphora, ellipsis, intensionality, tense,
aspect, vagueness, modals, indexicals,
reciprocals, propositional attitudes, scope
ambiguities, anaphoric adjectives,
non-intersective adjectives, temporal causal
relations, unselective quantifiers, adverbs of
quantification, donkey sentences, generic
determiners, comparatives, phrasal verbs,
Problemimprecise ? easily confounded by
negation, quantifiers, conditionals, factive
implicative verbs, etc.
4
Outline
Introduction A Theory of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion

• Introduction
• A Theory of Natural Logic
• The NatLog System
• Experiments with FraCaS
• Experiments with RTE
• Conclusion

5
What is natural logic? (? natural deduction)
Introduction A Theory of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion

• Characterizes valid patterns of inference via
  surface forms
• precise, yet sidesteps difficulties of
  translating to FOL
• A long history
• traditional logic Aristotles syllogisms,
  scholastics, Leibniz,
• modern natural logic begins with Lakoff (1970)
• van Benthem Sánchez Valencia (1986-91)
  monotonicity calculus
• Nairn et al. (2006) an account of implicatives
  factives
• We introduce a new theory of natural logic
• extends monotonicity calculus to account for
  negation exclusion
• incorporates elements of Nairn et al.s model of
  implicatives

6
7 basic entailment relations
Introduction A Theory of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion
Venn symbol name example
P Q equivalence couch sofa
P ? Q forward entailment (strict) crow ? bird
P ? Q reverse entailment (strict) European ? French
P Q negation (exhaustive exclusion) human nonhuman
P Q alternation (non-exhaustive exclusion) cat dog
P _ Q cover (exhaustive non-exclusion) animal _ nonhuman
P Q independence hungry hippo
Relations are defined for all semantic types
tiny ? small, hover ? fly, kick ? strike,this
morning ? today, in Beijing ? in China, everyone
? someone, all ? most ? some
7
Entailment semantic composition
Introduction A Theory of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion

• Ordinarily, semantic composition preserves
  entailment relations eat pork ? eat meat, big
  bird big fish
• But many semantic functions behave
  differentlytango ? dance ? refuse to
  tango ? refuse to danceFrench German ? not
  French _ not German
• We categorize functions by how they project
  entailment
• a generalization of monotonicity classes,
  implication signatures
• e.g., not has projectivity , ??, ??, ,
  _, _,
• e.g., refuse has projectivity , ??, ??,
  , , _,

8
Projecting entailment relations upward
Introduction A Theory of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion

• If two compound expressions differ by a single
  atom, their entailment relation can be determined
  compositionally
• Assume idealized semantic composition trees
• Propagate entailment relation between atoms
  upward, according to projectivity class of each
  node on path to root

9
A (weak) inference procedure
Introduction A Theory of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion

• Find sequence of edits connecting P and H
• Insertions, deletions, substitutions,
• Determine lexical entailment relation for each
  edit
• Substitutions depends on meaning of
  substituends cat dog
• Deletions ? by default red socks ? socks
• But some deletions are special not ill ill,
  refuse to go go
• Insertions are symmetric to deletions ? by
  default
• Project up to find entailment relation across
  each edit
• Compose entailment relations across sequence of
  edits
• à la Tarskis relation algebra

10
The NatLog system
Introduction A Theory of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion
NLI problem
linguistic analysis
1
alignment
2
lexical entailment classification
3
entailment projection
4
entailment composition
5
prediction
11
Running example
Introduction A Theory of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion
P Jimmy Dean refused to move without blue
jeans. H James Dean didnt dance without pantsyes
OK, the example is contrived, but it compactly
exhibits containment, exclusion, and implicativity
12
Step 1 Linguistic analysis
Introduction A Theory of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion

• Tokenize parse input sentences (future NER
  coref )
• Identify items w/ special projectivity
  determine scope
• Problem PTB-style parse tree ? semantic
  structure!

S
category /o implicatives examples refuse,
forbid, prohibit, scope S complement pattern
__ gt (/VB./ gt VP . Sarg) projectivity ,
??, ??, , , _,
VP

S

VP

VP

PP
NP
NP
NNP NNP VBD TO VB IN JJ
NNS
Jimmy Dean refused to move without blue
jeans

• Solution specify scope in PTB trees using Tregex
  Levy Andrew 06

13
Step 2 Alignment
Introduction A Theory of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion
P Jimmy Dean refused to move without blue jeans
H James Dean did nt dance without pants
editindex 1 2 3 4 5 6 7 8
edittype SUB DEL INS INS SUB MAT DEL SUB

• Alignment as sequence of atomic phrase edits
• Ordering of edits defines path through
  intermediate forms
• Need not correspond to sentence order
• Decomposes problem into atomic inference problems
• We havent (yet) invested much effort here
• Experimental results use alignments from other
  sources

14
Step 3 Lexical entailment classification
Introduction A Theory of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion

• Goal predict entailment relation for each edit,
  based solely on lexical features, independent of
  context
• Approach use lexical resources machine
  learning
• Feature representation
• WordNet features synonymy (), hyponymy (?/?),
  antonymy ()
• Other relatedness features Jiang-Conrath
  (WN-based), NomBank
• Fallback string similarity (based on Levenshtein
  edit distance)
• Also lexical category, quantifier category,
  implication signature
• Decision tree classifier
• Trained on 2,449 hand-annotated lexical
  entailment problems
• E.g., SUB(gun, weapon) ?, SUB(big, small) ,
  DEL(often) ?

15
Step 3 Lexical entailment classification
Introduction A Theory of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion
P Jimmy Dean refused to move without blue jeans
H James Dean did nt dance without pants
editindex 1 2 3 4 5 6 7 8
edittype SUB DEL INS INS SUB MAT DEL SUB
lexfeats strsim0.67 implic/o cataux catneg hypo hyper
lexentrel ? ? ?
16
Step 4 Entailment projection
Introduction A Theory of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion
P Jimmy Dean refused to move without blue jeans
H James Dean did nt dance without pants
editindex 1 2 3 4 5 6 7 8
edittype SUB DEL INS INS SUB MAT DEL SUB
lexfeats strsim0.67 implic/o cataux catneg hypo hyper
lexentrel ? ? ?
projec-tivity ? ? ? ? ? ? ? ?
atomicentrel ? ? ?








17
Step 5 Entailment composition
Introduction A Theory of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion
P Jimmy Dean refused to move without blue jeans
H James Dean did nt dance without pants
editindex 1 2 3 4 5 6 7 8
edittype SUB DEL INS INS SUB MAT DEL SUB
lexfeats strsim0.67 implic/o cataux catneg hypo hyper
lexentrel ? ? ?
projec-tivity ? ? ? ? ? ? ? ?
atomicentrel ? ? ?
compo-sition ? ? ? ? ?
18
The FraCaS test suite
Introduction A Theory of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion

• FraCaS a project in computational semantics
  Cooper et al. 96
• 346 textbook examples of NLI problems
• 3 possible answers yes, no, unknown (not
  balanced!)
• 55 single-premise, 45 multi-premise (excluded)

P At most ten commissioners spend time at home.
H At most ten commissioners spend a lot of time at home. yes

P Dumbo is a large animal.
H Dumbo is a small animal. no

P Smith believed that ITEL had won the contract in 1992.
H ITEL won the contract in 1992. unk
19
Results on FraCaS
Introduction A Theory of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion
System prec rec acc
most common class 183 55.7 100.0 55.7
MacCartney Manning 07 183 68.9 60.8 59.6
this work 183 89.3 65.7 70.5
20
Results on FraCaS
Introduction A Theory of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion
System System prec rec acc
most common class most common class 183 55.7 100.0 55.7
MacCartney Manning 07 MacCartney Manning 07 183 68.9 60.8 59.6
this work this work 183 89.3 65.7 70.5

Category prec rec acc
1 Quantifiers 44 95.2 100.0 97.7
2 Plurals 24 90.0 64.3 75.0
3 Anaphora 6 100.0 60.0 50.0
4 Ellipsis 25 100.0 5.3 24.0
5 Adjectives 15 71.4 83.3 80.0
6 Comparatives 16 88.9 88.9 81.3
7 Temporal 36 85.7 70.6 58.3
8 Verbs 8 80.0 66.7 62.5
9 Attitudes 9 100.0 83.3 88.9
1, 2, 5, 6, 9 1, 2, 5, 6, 9 108 90.4 85.5 87.0
21
The RTE3 test suite
Introduction A Theory of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion

• Somewhat more natural, but not ideal for NatLog
• Many kinds of inference not addressed by
  NatLogparaphrase, temporal reasoning, relation
  extraction,
• Big edit distance ? propagation of errors from
  atomic model

P As leaders gather in Argentina ahead of this weekends regional talks, Hugo Chávez, Venezuelas populist president is using an energy windfall to win friends and promote his vision of 21st-century socialism.
H Hugo Chávez acts as Venezuelas president. yes

P Democrat members of the Ways and Means Committee, where tax bills are written and advanced, do not have strong small business voting records.
H Democrat members had strong small business voting records. no
22
Results on RTE3 NatLog
Introduction A Theory of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion
System Data Yes Prec Rec Acc
Stanford RTE dev 50.2 68.7 67.0 67.2
test 50.0 61.8 60.2 60.5
NatLog dev 22.5 73.9 32.4 59.2
test 26.4 70.1 36.1 59.4
(each data set contains 800 problems)

• Accuracy is unimpressive, but precision is
  relatively high
• Strategy hybridize with Stanford RTE system
• As in Bos Markert 2006
• But NatLog makes positive prediction far more
  often (25 vs. 4)

23
Results on RTE3 hybrid system
Introduction A Theory of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion
System Data Yes Prec Rec Acc
Stanford RTE dev 50.2 68.7 67.0 67.2
test 50.0 61.8 60.2 60.5
NatLog dev 22.5 73.9 32.4 59.2
test 26.4 70.1 36.1 59.4
Hybrid dev 56.0 69.2 75.2 70.0
test 54.5 64.4 68.5 64.5
(each data set contains 800 problems)
24
Conclusion what natural logic cant do
Introduction A Theory of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion

• Not a universal solution for NLI
• Many types of inference not amenable to natural
  logic
• Paraphrase Eve was let go Eve lost her job
• Verb/frame alternation he drained the oil ? the
  oil drained
• Relation extraction Aho, a trader at UBS ? Aho
  works for UBS
• Common-sense reasoning the sink overflowed ? the
  floor got wet
• etc.
• Also, has a weaker proof theory than FOL
• Cant explain, e.g., de Morgans laws for
  quantifiers
• Not all birds fly Some birds dont fly

25
Conclusion what natural logic can do
Introduction A Theory of Natural Logic
The NatLog System Experiments with FraCaS
Experiments with RTE Conclusion

• Natural logic enables precise reasoning about
  containment, exclusion, and implicativity, while
  sidestepping the difficulties of translating to
  FOL.
• The NatLog system successfully handles a broad
  range of such inferences, as demonstrated on the
  FraCaS test suite.
• Ultimately, open-domain NLI is likely to require
  combining disparate reasoners, and a facility for
  natural logic is a good candidate to be a
  component of such a system.