Natural Language Inference

1
Natural Language Inference

• Bill MacCartney
• NLP Group
• Stanford University
• 8 May 2009

2
Natural language inference (NLI)
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion

• Aka recognizing textual entailment (RTE)
• Does premise P justify an inference to hypothesis
  H?
• An informal, intuitive notion of inference not
  strict logic
• Emphasis on variability of linguistic expression

P Several airlines polled saw costs grow more
than expected,even after adjusting for
inflation. H Some of the companies in the poll
reported cost increases. yes

• Necessary to goal of natural language
  understanding (NLU)
• Many more immediate applications

3
Applications of NLI
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion
semantic search
question answering
Harabagiu Hickl 06
King et al. 07
Q How much did Georgias gas price increase?
A In 2006, Gazprom doubled Georgias gas bill.
A Georgias main imports are natural gas,
machinery, ...
A Tbilisi is the capital and largest city of
Georgia.
A Natural gas is a gas consisting primarily of
methane.
Georgias gas bill doubled
Search
summarization
MT evaluation
Pado et al. 09
Tatar et al. 08
input Gazprom va doubler le prix du gaz pour la
Géorgie.
machine translation
X
output Gazprom will double the price of gas for
Georgia.
X
evaluation does output paraphrase target?
target Gazprom will double Georgias gas Bill.
4
NLI problem sets
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion

• RTE (Recognizing Textual Entailment)
• 4 years, each with dev test sets, each 800 NLI
  problems
• Longish premises taken from (e.g.) newswire
  short hypotheses
• Balanced 2-way classification entailment vs.
  non-entailment

P As leaders gather in Argentina ahead of this weekends regional talks, Hugo Chávez, Venezuelas populist president is using an energy windfall to win friends and promote his vision of 21st-century socialism.
H Hugo Chávez acts as Venezuelas president. yes

• FraCaS test suite
• 346 NLI problems, constructed by semanticists in
  mid-90s
• 55 have single premise remainder have 2 or more
  premises
• 3-way classification entailment, contradiction,
  compatibility

P Smith wrote a report in two hours.
H Smith spend more than two hours writing the report. no
5
NLI a spectrum of approaches
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion
Solution?
Problemhard to translate NL to FOL idioms,
anaphora, ellipsis, intensionality, tense,
aspect, vagueness, modals, indexicals,
reciprocals, propositional attitudes, scope
ambiguities, anaphoric adjectives,
non-intersective adjectives, temporal causal
relations, unselective quantifiers, adverbs of
quantification, donkey sentences, generic
determiners, comparatives, phrasal verbs,
Problemimprecise ? easily confounded by
negation, quantifiers, conditionals, factive
implicative verbs, etc.
6
Shallow approaches to NLI
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion

• Example the bag-of-words approach Glickman et
  al. 2005
• Measures approximate lexical similarity of H to
  (part of) P

P Several airlines polled
saw costs grow H Some of the companies
in the poll reported cost increases .
more than expected, even after adjusting for
inflation.

• Robust, and surprisingly effective for many NLI
  problems
• But imprecise, and hence easily confounded
• Ignores predicate-argument structure this can
  be remedied
• Struggles with antonymy, negation, verb-frame
  alternation
• Crucially, depends on assumption of upward
  monotonicity

• Non-upward-monotone constructions are rife!
  Danescu et al. 2009 not, all, most, few,
  rarely, if, tallest, without, doubt, avoid,
  regardless, unable,

7
The formal approach to NLI
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion
Relies on full semantic interpretation of P H

• Translate to formal representation apply
  automated reasoner
• Can succeed in restricted domains, but not in
  open-domain NLI!

P Several airlines polled saw costs grow more
than expected,even after adjusting for inflation.
(exists p (and (poll-event p)
(several x (and (airline x) (obj p x)
(exists c (and (cost c) (has x c)
(exists g (and (grow-event g) (subj g c)
(greater-than (magnitude g)
..... ?

• Need background axioms to complete proofs but
  from where?
• Besides, NLI task based on informal definition of
  inferability
• Bos Markert 06 found FOL proof for just 4 of
  RTE problems

8
Solution? Natural logic! (? natural deduction)
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion

• Characterizes valid patterns of inference via
  surface forms
• precise, yet sidesteps difficulties of
  translating to FOL
• A long history
• traditional logic Aristotles syllogisms,
  scholastics, Leibniz,
• modern natural logic begins with Lakoff (1970)
• van Benthem Sánchez Valencia (1986-91)
  monotonicity calculus
• Nairn et al. (2006) an account of implicatives
  factives
• We introduce a new theory of natural logic
• extends monotonicity calculus to account for
  negation exclusion
• incorporates elements of Nairn et al.s model of
  implicatives
• and implement evaluate a computational model
  of it

9
Outline
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion

• Introduction
• Alignment for NLI
• A theory of entailment relations
• A theory of compositional entailment
• The NatLog system
• Conclusions
• Not covered today the bag-of-words model, the
  Stanford RTE system

10
Alignment for NLI
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion

• Most approaches to NLI depends on a facility for
  alignment

P Gazprom today confirmed a two-fold increase in
its gas price for Georgia, beginning next
Monday. H Gazprom will double Georgias gas
bill. yes

• Linking corresponding words phrases in two
  sentences
• Alignment problem is familiar in machine
  translation (MT)

11
Alignment example
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion
H (hypothesis)
P (premise)
12
Approaches to NLI alignment
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion

• Alignment addressed variously by current NLI
  systems
• In some approaches to NLI, alignments are
  implicit
• NLI via lexical overlap Glickman et al. 05,
  Jijkoun de Rijke 05
• NLI as proof search Tatu Moldovan 07, Bar-Haim
  et al. 07
• Other NLI systems make alignment step explicit
• Align first, then determine inferential validity
  Marsi Kramer 05, MacCartney et al. 06
• What about using an MT aligner?
• Alignment is familiar in MT, with extensive
  literatureBrown et al. 93, Vogel et al. 96, Och
  Ney 03, Marcu Wong 02, DeNero et al. 06,
  Birch et al. 06, DeNero Klein 08
• Can tools techniques of MT alignment transfer
  to NLI?
• Dissertation argues not very well

13
The MANLI aligner
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion

• A model of alignment for NLI consisting of four
  components

1. Phrase-based representation
2. Feature-based scoring function
3. Decoding using simulated annealing
4. Perceptron learning

14
Phrase-based alignment representation
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion
Represent alignments by sequence of phrase edits
EQ, SUB, DEL, INS
EQ(Gazprom1, Gazprom1) INS(will2) DEL(today2) DEL(
confirmed3) DEL(a4) SUB(two-fold5 increase6,
double3) DEL(in7) DEL(its8)

• One-to-one at phrase level (but many-to-many at
  token level)
• Avoids arbitrary alignment choices can use
  phrase-based resources

15
A feature-based scoring function
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion

• Score edits as linear combination of features,
  then sum

• Edit type features EQ, SUB, DEL, INS
• Phrase features phrase sizes, non-constituents
• Lexical similarity feature max over similarity
  scores
• WordNet synonymy, hyponymy, antonymy,
  Jiang-Conrath
• Distributional similarity à la Dekang Lin
• Various measures of string/lemma similarity
• Contextual features distortion, matching
  neighbors

16
Decoding using simulated annealing
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion
100 times
17
Perceptron learning of feature weights
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion

• We use a variant of averaged perceptron Collins
  2002

Initialize weight vector w 0, learning rate R0
1 For training epoch i 1 to 50 For each
problem ?Pj, Hj? with gold alignment Ej Set Êj
ALIGN(Pj, Hj, w) Set w w Ri ? (?(Ej)
?(Êj)) Set w w / ?w?2 (L2 normalization) Set
wi w (store weight vector for this
epoch) Set Ri 0.8 ? Ri1 (reduce learning
rate) Throw away weight vectors from first 20 of
epochs Return average weight vector
Training runs require about 20 hours (on 800 RTE
problems)
18
The MSR RTE2 alignment data
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion

• Previously, little supervised data
• Now, MSR gold alignments for RTE2
• Brockett 2007
• dev test sets, 800 problems each
• Token-based, but many-to-many
• allows implicit alignment of phrases
• 3 independent annotators
• 3 of 3 agreed on 70 of proposed links
• 2 of 3 agreed on 99.7 of proposed links
• merged using majority rule

19
Evaluation on MSR data
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion

• We evaluate several alignment models on MSR data
• Baseline a simple bag-of-words aligner
• Matches each token in H to most string-similar
  token in P
• Two well-known MT aligners GIZA Cross-EM
• Supplemented with lexicon tried various
  symmetrization heuristics
• A representative NLI aligner the Stanford RTE
  aligner
• Cant do phrase alignments, but can exploit
  syntactic features
• The MANLI aligner just presented
• How well do they recover gold-standard
  alignments?
• Assess per-link precision, recall, and F1 and
  exact match rate

20
Aligner evaluation results
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion
RTE2 dev RTE2 dev RTE2 dev RTE2 dev RTE2 test RTE2 test RTE2 test RTE2 test
System P R F1 E P R F1 E
Bag-of-words 57.8 81.2 67.5 3.5 62.1 82.6 70.9 5.3
GIZA 83.0 66.4 72.1 9.4 85.1 69.1 74.8 11.3
Cross-EM 67.6 80.1 72.1 1.3 70.3 81.0 74.1 0.8
Stanford RTE 81.1 75.8 78.4 0.5 82.7 75.8 79.1 0.3
MANLI 83.4 85.5 84.4 21.7 85.4 85.3 85.3 21.3

• Bag-of-words aligner good recall, but poor
  precision
• MT aligners fail to learn word-word
  correspondences
• Stanford RTE aligner struggles with function
  words
• MANLI outperforms all others on every measure
• F1 10.5 higher than GIZA, 6.2 higher than
  Stanford
• Good balance of precision recall matched gt20
  exactly

21
MANLI results discussion
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion

• Three factors contribute to success
• Lexical resources jail prison, prevent stop
  , injured wounded
• Contextual features enable matching function
  words
• Phrases death penalty capital punishment,
  abdicate give up
• But phrases help less than expected!
• If we set max phrase size 1, we lose just 0.2
  in F1
• Recall errors room to improve
• 40 need better lexical resources conservation
  protecting, organization agencies, bone
  fragility osteoporosis
• Precision errors harder to reduce
• equal function words (49), forms of be (21),
  punctuation (7)

22
Alignment for NLI conclusions
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion

• MT aligners not directly applicable to NLI
• They rely on unsupervised learning from massive
  amounts of bitext
• They assume semantic equivalence of P H
• MANLI succeeds by
• Exploiting (manually automatically constructed)
  lexical resources
• Accommodating frequent unaligned phrases
• Using contextual features to align function words
• Phrase-based representation shows potential
• But not yet proven need better phrase-based
  lexical resources

23
Outline
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion

• Introduction
• Alignment for NLI
• A theory of entailment relations
• A theory of compositional entailment
• The NatLog system
• Conclusion

24
Entailment relations in past work
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion

X is a man
X is a woman
X is a hippo
X is hungry
X is a fish
X is a carp
X is a crow
X is a bird
X is a couch
X is a sofa
25
16 elementary set relations
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion
Assign sets ?x, y? to one of 16 relations,
depending on emptiness or non-emptiness of each
of four partitions








?y
y








? ?
? ?
?x
x








empty








non-empty
26
16 elementary set relations
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion
But 9 of 16 are degenerate either x or y is
either empty or universal. I.e., they correspond
to semantically vacuous expressions, which are
rare outside logic textbooks. We therefore focus
on the remaining seven relations.
































27
The set of basic entailment relations
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion
diagram symbol name example
x ? y equivalence couch ? sofa
x ? y forward entailment (strict) crow ? bird
x ? y reverse entailment (strict) European ? French
x y negation (exhaustive exclusion) human nonhuman
x y alternation (non-exhaustive exclusion) cat dog
x ??y cover (exhaustive non-exclusion) animal ? nonhuman
x y independence hungry hippo
Relations are defined for all semantic types
tiny ? small, hover ? fly, kick ? strike,this
morning ? today, in Beijing ? in China,
everyone ? someone, all ? most ? some
28
Joining entailment relations
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion
x
y
y
z
? ? ? ? ?
? ? ? ? ?
? ? ? ? ?
? ? ?
R ? ? ? R
? ? R ? R
?


29
Some joins yield unions of relations!
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion
x y y z x ? z
couch table table sofa couch ? sofa
pistol knife knife gun pistol ? gun
dog cat cat terrier dog ? terrier
rose orchid orchid daisy rose daisy
woman frog frog Eskimo woman Eskimo
30
The complete join table
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion

• Of 49 join pairs, 32 yield relations in 17
  yield unions

Larger unions convey less information limits
power of inference
In practice, any union which contains can be
approximated by so, in practice, we can
avoid the complexity of unions
31
Outline
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion

• Introduction
• Alignment for NLI
• A theory of entailment relations
• A theory of compositional entailment
• The NatLog system
• Conclusion

32
Lexical entailment relations
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion
x
e(x)

• ????????????? will depend on
• the lexical entailment relation generated by e
  ?(e)
• other properties of the context x in which e is
  applied

?( , )
Example suppose x is red car If e is SUB(car,
convertible), then ?(e) is ? If e is DEL(red),
then ?(e) is ? Crucially, ?(e) depends solely on
lexical items in e, independent of context x But
how are lexical entailment relations determined?
33
Lexical entailment relations SUBs
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion

• ?(SUB(x, y)) ?(x, y)
• For open-class terms, use lexical resource (e.g.
  WordNet)
• ? for synonyms sofa ? couch, forbid ? prohibit
• ? for hypo-/hypernyms crow ? bird, frigid ?
  cold, soar ? rise
• for antonyms and coordinate terms hot cold,
  cat dog
• ? or for proper nouns USA ? United States,
  JFK FDR
• for most other pairs hungry hippo
• Closed-class terms may require special handling
• Quantifiers all ? some, some no, no all,
  at least 4 ? at most 6
• See dissertation for discussion of pronouns,
  prepositions,

34
Lexical entailment relations DEL INS
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion

• Generic (default) case ?(DEL()) ?, ?(INS())
  ?
• Examples red car ? car, sing ? sing off-key
• Even quite long phrases car parked outside since
  last week ? car
• Applies to intersective modifiers, conjuncts,
  independent clauses,
• This heuristic underlies most approaches to RTE!
• Does P subsume H? Deletions OK insertions
  penalized.
• Special cases
• Negation didnt sleep did sleep
• Implicatives factives (e.g. refuse to, admit
  that) discussed later
• Non-intersective adjectives former spy spy,
  alleged spy spy
• Auxiliaries etc. is sleeping ? sleeps, did
  sleep ? slept

35
The impact of semantic composition
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion

• How are entailment relations affected by semantic
  composition?

The monotonicity calculus provides a partial
answer
If f has monotonicity
UP UP UP
? ? ?
? ? ?
? ? ?
?
DOWN DOWN DOWN
? ? ?
? ? ?
? ? ?
?
NON NON NON
? ? ?
? ?
? ?
?
But how are other relations (, , ?) projected?
36
A typology of projectivity
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion

• Projectivity signatures a generalization of
  monotonicity classes

In principle, 77 possible signatures, but few
actually realized
negation negation negation
? ? ?
? ? ?
? ? ?
?
? ?
? ?
?
not happy ? not glad





isnt swimming isnt hungry
didnt kiss ? didnt touch
not ill ? not seasick
not human not nonhuman
not French ? not German
not more than 4 not less than 6
37
A typology of projectivity
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion

• Projectivity signatures a generalization of
  monotonicity classes
• Each projectivity signature is a map
• In principle, 77 possible signatures, but few
  actually realized

intersectivemodification intersectivemodification intersectivemodification
? ? ?
? ? ?
? ? ?
?
?
? ?
?
negation negation negation
? ? ?
? ? ?
? ? ?
?
? ?
? ?
?
live human live nonhuman
French wine Spanish wine

metallic pipe nonferrous pipe
See dissertation for projectivity of connectives,
quantifiers, verbs
38
Projecting through multiple levels
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion
Propagate entailment relation between atoms
upward, according to projectivity class of each
node on path to root
nobody can enter with a shirt ? nobody can enter
with clothes
39
Implicatives factives Nairn et al. 06
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion
9 signatures, per implications (, , or o) in
positive and negative contexts
signature example
implicatives / he managed to escape
/ o he was forced to sell
o / he was permitted to live
implicatives / he forgot to pay
/ o he refused to fight
o / he hesitated to ask
factives / he admitted that he knew
/ he pretended he was sick
o / o he wanted to fly
40
Implicatives factives
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion
We can specify relation generated by DEL or INS
of each signature
signature example example example ?(DEL) ?(INS)
implicatives / he managed to escape ? he escaped ? ?
/ o he was forced to sell ? he sold ? ?
o / he was permitted to live ? he lived ? ?
implicatives / he forgot to pay he paid
/ o he refused to fight he fought
o / he hesitated to ask ? he asked ? ?
nonfactives o / o he wanted to fly he flew
Room for variation w.r.t. infinitives,
complementizers, passivation, etc.
Some more intuitive when negated he didnt
hesitate to ask he didnt ask
Doesnt cover factives, which involve
presuppositions see dissertation
41
Putting it all together
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion

• Find a sequence of edits ?e1, , en? which
  transforms p into h. Define x0 p, xn h, and
  xi ei(xi1) for i ? 1, n.
• For each atomic edit ei
• Determine the lexical entailment relation ?(ei).
• Project ?(ei) upward through the semantic
  composition tree of expression xi1 to find the
  atomic entailment relation ?(xi1, xi)
• Join atomic entailment relations across the
  sequence of edits?(p, h) ?(x0, xn) ?(x0,
  x1) ? ? ?(xi1, xi) ? ? ?(xn1, xn)

Limitations need to find appropriate edit
sequence connecting p and htendency of ?
operation toward less-informative entailment
relations lack of general mechanism for
combining multiple premises Less deductive power
than FOL. Cant handle e.g. de Morgans Laws.
42
An example
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion
P The doctor didnt hesitate to recommend
Prozac. H The doctor recommended
medication. yes
i ei xi lex atom join
The doctor didnt hesitate to recommend Prozac. The doctor didnt hesitate to recommend Prozac. The doctor didnt hesitate to recommend Prozac. The doctor didnt hesitate to recommend Prozac.
1 DEL(hesitate to) DEL(hesitate to)
The doctor didnt recommend Prozac. The doctor didnt recommend Prozac. The doctor didnt recommend Prozac. The doctor didnt recommend Prozac.
2 DEL(didnt) DEL(didnt)
The doctor recommended Prozac. The doctor recommended Prozac. The doctor recommended Prozac. The doctor recommended Prozac.
3 SUB(Prozac, medication) SUB(Prozac, medication)
The doctor recommended medication. The doctor recommended medication. The doctor recommended medication. The doctor recommended medication.
?




?
?
?
?
yes
43
Different edit orders?
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion
i ei lex atom join
1 DEL(hesitate to) ?
2 DEL(didnt) ?
3 SUB(Prozac, medication) ? ? ?
i ei lex atom join
1 DEL(hesitate to) ?
2 SUB(Prozac, medication) ? ?
3 DEL(didnt) ?
i ei lex atom join
1 DEL(didnt)
2 DEL(hesitate to) ? ? ?
3 SUB(Prozac, medication) ? ? ?
i ei lex atom join
1 DEL(didnt)
2 SUB(Prozac, medication) ? ?
3 DEL(hesitate to) ? ? ?
i ei lex atom join
1 SUB(Prozac, medication) ? ? ?
2 DEL(hesitate to) ?
3 DEL(didnt) ?
i ei lex atom join
1 SUB(Prozac, medication) ? ? ?
2 DEL(didnt)
3 DEL(hesitate to) ? ? ?
Intermediate steps may vary final result is
typically (though not necessarily) the same
44
Outline
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion

• Introduction
• Alignment for NLI
• A theory of entailment relations
• A theory of compositional entailment
• The NatLog system
• Conclusion

45
The NatLog system
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion
NLI problem
next slide
linguistic analysis
1
from outside sources
alignment
2
core of systemcovered shortly
lexical entailment classification
3
entailment projection
4
straightforwardnot covered further
straightforwardnot covered further
entailment joining
5
prediction
46
Stage 1 Linguistic analysis
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion

• Tokenize parse input sentences (future NER
  coref )
• Identify items w/ special projectivity
  determine scope
• Problem PTB-style parse tree ? semantic
  structure!

S
category /o implicatives examples refuse,
forbid, prohibit, scope S complement pattern
__ gt (/VB./ gt VP . Sarg) projectivity ??,
??, ??, , , _,
VP

S

VP

VP

PP
NP
NP
NNP NNP VBD TO VB IN JJ
NNS
Jimmy Dean refused to move without blue
jeans

• Solution specify scope in PTB trees using Tregex
  Levy Andrew 06

47
Stage 3 Lexical entailment classification
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion

• Goal predict entailment relation for each edit,
  based solely on lexical features, independent of
  context
• Approach use lexical resources machine
  learning
• Feature representation
• WordNet features synonymy (?), hyponymy (?/?),
  antonymy ()
• Other relatedness features Jiang-Conrath
  (WN-based), NomBank
• Fallback string similarity (based on Levenshtein
  edit distance)
• Also lexical category, quantifier category,
  implication signature
• Decision tree classifier
• Trained on 2,449 hand-annotated lexical
  entailment problems
• E.g., SUB(gun, weapon) ?, SUB(big, small) ,
  DEL(often) ?

48
The FraCaS test suite
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion

• FraCaS a project in computational semantics
  Cooper et al. 96
• 346 textbook examples of NLI problems
• 3 possible answers yes, no, unknown (not
  balanced!)
• 55 single-premise, 45 multi-premise (excluded)

P At most ten commissioners spend time at home.
H At most ten commissioners spend a lot of time at home. yes

P Dumbo is a large animal.
H Dumbo is a small animal. no

P Smith believed that ITEL had won the contract in 1992.
H ITEL won the contract in 1992. unk
49
Results on FraCaS
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion
System prec rec acc
most common class 183 55.7 100.0 55.7
MacCartney Manning 07 183 68.9 60.8 59.6
this work 183 89.3 65.7 70.5
50
Results on FraCaS
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion
System System prec rec acc
most common class most common class 183 55.7 100.0 55.7
MacCartney Manning 07 MacCartney Manning 07 183 68.9 60.8 59.6
this work this work 183 89.3 65.7 70.5

Category prec rec acc
1 Quantifiers 44 95.2 100.0 97.7
2 Plurals 24 90.0 64.3 75.0
3 Anaphora 6 100.0 60.0 50.0
4 Ellipsis 25 100.0 5.3 24.0
5 Adjectives 15 71.4 83.3 80.0
6 Comparatives 16 88.9 88.9 81.3
7 Temporal 36 85.7 70.6 58.3
8 Verbs 8 80.0 66.7 62.5
9 Attitudes 9 100.0 83.3 88.9
1, 2, 5, 6, 9 1, 2, 5, 6, 9 108 90.4 85.5 87.0
51
The RTE3 test suite
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion

• Somewhat more natural, but not ideal for NatLog
• Many kinds of inference not addressed by
  NatLogparaphrase, temporal reasoning, relation
  extraction,
• Big edit distance ? propagation of errors from
  atomic model

P As leaders gather in Argentina ahead of this weekends regional talks, Hugo Chávez, Venezuelas populist president is using an energy windfall to win friends and promote his vision of 21st-century socialism.
H Hugo Chávez acts as Venezuelas president. yes

P Democrat members of the Ways and Means Committee, where tax bills are written and advanced, do not have strong small business voting records.
H Democrat members had strong small business voting records. no
52
Results on RTE3 NatLog
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion
System Data Yes Prec Rec Acc
Stanford RTE dev 50.2 68.7 67.0 67.2
test 50.0 61.8 60.2 60.5
NatLog dev 22.5 73.9 32.4 59.2
test 26.4 70.1 36.1 59.4
(each data set contains 800 problems)

• Accuracy is unimpressive, but precision is
  relatively high
• Strategy hybridize with Stanford RTE system
• As in Bos Markert 2006
• But NatLog makes positive prediction far more
  often (25 vs. 4)

53
Results on RTE3 hybrid system
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion
System Data Yes Prec Rec Acc
Stanford RTE dev 50.2 68.7 67.0 67.2
test 50.0 61.8 60.2 60.5
NatLog dev 22.5 73.9 32.4 59.2
test 26.4 70.1 36.1 59.4
Hybrid dev 56.0 69.2 75.2 70.0
test 54.5 64.4 68.5 64.5
(each data set contains 800 problems)
54
Outline
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion

• Introduction
• Alignment for NLI
• A theory of entailment relations
• A theory of compositional entailment
• The NatLog system
• Conclusion

55
What natural logic cant do
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion

• Not a universal solution for NLI
• Many types of inference not amenable to natural
  logic
• Paraphrase Eve was let go ? Eve lost her job
• Verb/frame alternation he drained the oil ? the
  oil drained
• Relation extraction Aho, a trader at UBS ? Aho
  works for UBS
• Common-sense reasoning the sink overflowed ? the
  floor got wet
• etc.
• Also, has a weaker proof theory than FOL
• Cant explain, e.g., de Morgans laws for
  quantifiers
• Not all birds fly ? Some birds dont fly

56
What natural logic can do
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion

• Enables precise reasoning about semantic
  containment
• hypernymy hyponymy in nouns, verbs, adjectives,
  adverbs
• containment between temporal locative
  expressions
• quantifier containment
• adding dropping of intersective modifiers,
  adjuncts
• and semantic exclusion
• antonyms coordinate terms mutually exclusive
  nouns, adjectives
• mutually exclusive temporal locative
  expressions
• negation, negative restrictive quantifiers,
  verbs, adverbs, nouns
• and implicatives and nonfactives
• Sidesteps myriad difficulties of full semantic
  interpretation

57
Contributions of this dissertation
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion

• Undertook first systematic study of alignment for
  NLI
• Examined the relation between alignment in NLI
  and MT
• Evaluated bag-of-words, MT, and NLI aligners for
  NLI alignment
• Proposed a new model of alignment for NLI MANLI
• Extended natural logic to incorporate semantic
  exclusion
• Defined expressive set of entailment relations (
  join algebra)
• Introduced projectivity signatures a
  generalization of monotonicity
• Unified account of implicativity under same
  framework
• Implemented a robust system for natural logic
  inference
• Demonstrated practical value on FraCaS RTE test
  suites

58
The future of NLI
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion

• No silver bullet for NLI problems are too
  diverse
• A full solution will need to combine disparate
  reasoners
• simple lexical similarity (e.g., bag-of-words)
• relation extraction
• natural logic related forms of semantic
  reasoning
• temporal, spatial, simple mathematical
  reasoning
• commonsense reasoning
• Key question how can they best be combined?
• Apply in parallel, then combine predictions?
  How?
• Fine-grained interleaving? Collaborative proof
  search?

59
Thanks!
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion

• My heartfelt appreciation to
• My committee Profs. Genesereth, Jurafsky,
  Manning, Peters, and van Benthem
• My collaborators Marie-Catherine de Marneffe,
  Michel Galley, Teg Grenager, and many others
• My advisor Prof. Chris Manning
• My girlfriend Destiny Man Li Zhao

60
Backup slides follow
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion
61
NLI alignment vs. MT alignment
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion

• Doubtful NLI alignment differs in several
  respects
• Monolingual can exploit resources like WordNet
• Asymmetric P often longer has content
  unrelated to H
• Cannot assume semantic equivalence
• NLI aligner must accommodate frequent unaligned
  content
• Little training data available
• MT aligners use unsupervised training on huge
  amounts of bitext
• NLI aligners must rely on supervised training
  much less data

62
Projectivity of connectives
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion
negation (not)
conjunction (and) / intersective modification
? ? ?
? ? ?
? ? ?

?
?

63
Projectivity of connectives
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion
negation (not)
conjunction (and) / intersective modification
disjunction (or)

waltzed or sang ? danced or sang

human or equine ? nonhuman or equine
red or yellow blue or yellow


? ? ? ?
? ? ? ?
? ? ? ?
?
?
? ?

64
Projectivity of connectives
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion
negation (not)
conjunction (and) / intersective modification
disjunction (or)
conditional (if) (antecedent)

If he drinks tequila,he feels nauseous ? If he drinks liquor,he feels nauseous

If its sunny, we surf If its not sunny, we surf
If its sunny, we surf If its rainy, we surf


? ? ? ? ?
? ? ? ? ?
? ? ? ? ?
?
?
? ?

65
Projectivity of connectives
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion
negation (not)
conjunction (and) / intersective modification
disjunction (or)
conditional (if) (antecedent)
conditional (if) (consequent)

If he drinks tequila,he feels nauseous ? If he drinks tequila,he feels sick

If its sunny, we surf If its sunny, we dont surf
If its sunny, we surf If its sunny, we ski


? ? ? ? ? ?
? ? ? ? ? ?
? ? ? ? ? ?
?
?
? ?

66
Projectivity of connectives
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion
negation (not)
conjunction (and) / intersective modification
disjunction (or)
conditional (if) (antecedent)
conditional (if) (consequent)
biconditional (if and only if)
? ? ? ? ? ? ?
? ? ? ? ? ?
? ? ? ? ? ?
?
?
? ?

67
Projectivity of quantifiers
Introduction Alignment for NLI Entailment
relations Compositional entailment The
NatLog system Conclusion