Test Suites for Textual Inference

1
Test Suites for Textual Inference

• Johan BosUniversity of Rome La Sapienza
• Websitehttp//www.cogsci.ed.ac.uk/jbos/ti/

2
Textual Inference Data Sets

• Pascal data sets
• RTE 1 (1.376 pairs)
• RTE 2 (1.600 pairs)
• Other less known data sets
• FRACAS (346 pairs) Cooper et al. 1996
• PARC (76 pairs) Zaenen, Karttunen Crouch 2005
  
• Specific phenomena
• Adjectives (ca. 1.000 pairs) Amoia Gardent
  2006

3
The FRACAS test suite

• European Project on Computational Semantics, in
  the mid 1990s
• Test suite published in D16, but since then
  forgotten?
• Cooper et al. (1996) Using the Framework. Fracas
  deliverable D16, section 3
• Aim of test suite measure semantic competence of
  NLP system

4
The FRACAS test suite

• Grouped on linguistic and semantic phenomena
• Generalised quantifiers
• Plurals
• Nominal Anaphora
• Ellipsis
• Adjectives
• Comparatives
• Temporal Reference
• Verbs
• Attitudes

5
FRACAS example pairs

• 3.209 Mickey is a small animal.
  Dumbo is a large animal.
  Is Mickey
  smaller than Dumbo? YES
• 3.205 Dumbo is a large animal.
  .. ... Is Dumbo
  a small animal? NO
• 3.206 Fido is not a small animal.
  .. Is Fido a
  large animal? DONT KNOW

6
Differences

• Textual inference, not textual entailment
• Instead of a two valued classification TRUE and
  FALSE, we have three values YES, NO, DONT KNOW
• No or very little world knowledge
• Focussing on specific phenomena
• Not real text, but artificially constructed

7
Discussion

• Advantages
• Useful for debugging NLP systems
• Better control over whether a system understands
  certain phenomena
• Similar proposal
• Zaenen, Karttunen Crouch 2005
• Opponents of this proposal
• Manning 2006

8
The way forward

• Not one single task, but
• Introduce different classes of RTI
• Linguistic/Semantic
• World knowledge
• Robust
• Limit artificiality of examples
• Construct T from real text
• Construct H from template rules

9
Example

• Text, from real data
• The Osaka World Trade Center is the highest
  building in Japan
• Generated Hypotheses
• The Osaka World Trade Center is the third highest
  building in Japan NO
• The Osaka World Trade Center is a building in
  Japan YES
• The Osaka World Trade Center is one of the
  highest buildings in Japan YES
• The Osaka World Trade Center is the highest
  building in Western Japan MAYBE