Whats syntax got to do with it

1
Whats syntax got to do with it?

• Lucy Vanderwende
• Deborah Coughlin
• Bill Dolan
• April 12, 2005

2
Examples of Data Set

• Text Eyeing the huge market potential,
  currently led by Google, Yahoo took over search
  company Overture Services Inc last year.
• Hypothesis Yahoo bought Overture.
• True
• Text Microsoft's rival Sun Microsystems Inc.
  bought Star Office last month and plans to boost
  its development as a Web-based device running
  over the Net on personal computers and Internet
  appliances.
• Hypothesis Microsoft bought Star Office.
• False
• Our goal for participation provide a baseline
• What can be handled by syntax/logical form,
  including basic syntactic alternations?

3
Annotating the Test Set

• Important we annotated the test set manually
• To provide a baseline for an ideal parser
• Partial submission
• Annotation tool
• Three way choice
• Syntax Entailment (True/False) can be decided
  based on syntactic cues
• Not-Syntax Entailment cannot be decided based on
  syntactic cues
• Cant decide
• Optional needs thesaurus
• Examples

4
Decidable with syntax alone
5
Decidable with syntax thesaurus
6
Findings

• Notes
• Agreement with test set 96
• Inter-annotator agreement 72
• 60 of disagreements between Syntax
  False/not-syntax
• 25 of disagreements between Syntax
  True/not-syntax
• Remainder, cant decide

7
Requirements for a parsing system

• Apposition / copula construction by far the
  most frequent pattern
• ltTgt The Alameda Central, west of the Zocalo, was
  created in 1592.
• ltHgt The Alameda Central is west of the Zocalo.
• Predicate-argument assignment
• ltTgt Microsoft's rival Sun Microsystems Inc.
  bought Star Office last month and plans to boost
  its development as a Web-based device running
  over the Net on personal computers and Internet
  appliances.
• ltHgt Microsoft bought Star Office.

8
Requirements for a parsing system

• Active / passive alternation
• ltTgt Latvia, for instance, is the lowest-ranked
  team in the field but defeated World Cup
  semifinalist Turkey in a playoff to qualify for
  the final 16 of Euro 2004.
• ltHgt Turkey is defeated by Latvia.
• Nominalization / tensed clause
• ltTgt The debacle marked a new low in the erosion
  of the SPDs popularity, which began shortly
  after Mr Schroeders election in 1998.
• ltHgt Schroeder was elected in 1998.
• Finite / non-finite construction
• ltTgt A 30-year-old man has been killed in a shark
  attack at a surfing beach near Perth in West
  Australia where he was surfing with four other
  people.
• ltHgt A 30-year-old man was killed in a shark
  attack while surfing.

9
Identifying Cases where Syntax signals False

• False, violation of syntactic analysis
• Shared Subject and Predicate, mismatch on Object
• ltTgt The White House ignores Zinnis opposition to
  the Iraq War.
• ltHgt White House ignores the threat of attack.
• False, no shared syntactic structure
• Note we do not know whether this can be
  operationalized
• ltTgt Scott and Lance Hacking talked with their
  younger brother at the hospital July 24.
• ltHgtDouglas and Scott Hacking are physicians.

10
Interesting questions

• What sentences that can be handled by syntactic
  analysis could also have been captured by string
  level analyses?
• How many more sentences can be handled by
  bringing in paraphrase?
• If not lexical, syntactic, or paraphrase what
  else is required to predict entailment?

11
Paraphrase needed in entailment task

• Three of its five permanent members, France,
  Russia and China, are pushing the U.N. Security
  Council to lift the embargo on Iraq.
• France, Russia and China want to put an end to
  sanctions against Iraq.
• Satomi Mitarai died of blood loss.
• Satomi Mitarai bled to death.

12
Paraphrase needed in entailment task

• The Clark County medical examiner's office said
  the man who was killed was 33 years old.
• The Clark County medical examiner's office put
  the dead man's age at 33.