Linguistics 187287 Week 5

1
Linguistics 187/287 Week 5
Robustness and Ambiguity

• Ron Kaplan and Tracy King

2
HW4 Issues

• Getting PERS feature in NP coordination

NP-CONJ "if any conjunct is first
person," "the whole NP is first person"
(! PERS) 1 ( PERS) 1 "if any
conjunct is second person," "the whole NP is
second person" "unless there was a first
person conjunct" (! PERS) 2 (
PERS)c 1 ( PERS) 2 "third
person if both conjuncts are" (! PERS) 3
( PERS)c 1 ( PERS)c 2
(PERS) 3 .
3
Ambiguity and Robustness

• Large-scale grammars are massively ambiguous
• Grammars parsing real text need to be robust
• "loosening" rules to allow robustness increases
  ambiguity even more
• Need a way to control the ambiguity
• version of Optimality Theory (OT)

4
Theoretical OT

• Grammar has a set of violable constraints
• Constraints are ranked by each language
• This gives cross-linguistic variation
• Candidates (analyses) compete
• John waited for Mary. vs. John waited for 3
  hours.
• Constraint ranking determines winning candidate
• Issues for XLE
• Candidates can be very ungrammatical
• we have a grammar to produce grammatical analyses
• even with robust, ungrammatical analyses, these
  are controlled
• Generation, not parsing direction
• we know what the string is already
• for generation we have a very specified analysis

5
XLE OT

• Incorporate idea of ranking and (dis)preference
• Filter syntactic and lexical ambiguity
• Reconcile robustness and accuracy
• Allow parsing grammar to be used for generation

6
XLE OT Implementation

• OT marks in
• grammar rules
• templates
• lexical entries
• CONFIG states
• preference vs. dispreference
• ranking
• parsing vs. generation orders

7
The o projection

• OT marks are not f-structure features
• OT marks are in their own projection

f-structure
c-structure
o-structure (set of OT marks)
8
The o projection

• The o-structure is just a set of marks
• PPadj GuessedN
• Instead of and !, have o (NB !?f)
• PP ( ADJUNCT)!
• PPadj o
• the f-structure is exactly the same
• there is now an additional o-structure

9
Ranking analyses

• Specify relative importance of OT marks in the
  CONFIG
• OPTIMALITYORDER Mark3 Mark2 Mark1.
• Comparing analyses
• Find most important mark where the analyses
  differ
• Prefer the analysis with the
• Least number of dispreference marks (no )
• Most number of preference marks ()

10
Ranking analyses (continued)

• an analysis with Mark2 is preferred over an
  analysis with Mark3
• an analysis with no mark is preferred over an
  analysis with Mark2 or Mark3
• an analysis with one Mark2 is preferred over one
  with two Mark2
• an analysis with Mark1 is preferred over an
  analysis with no mark
• an analysis with two Mark1 is preferred over an
  analysis with one Mark1

11
Difference with Theoretical OT

• Theoretical OT only dispreference marks
• XLE OT
• dispreference marks Mark1
• preference marks Mark1
• NOTE is only indicated in the CONFIG
• only the name (Mark1) appears in
  the
• grammar
• Deciding which to use can be difficult

12
Example PP ambiguities

• John waited for Mary.
• John waited for 3 hours.
• Rule with OT marks Using template
  OT(_mark)_mark o.
• VP --gt V
• (NP ( OBJ)!)
• PP ( OBL)!
• _at_(OT PPobl)
• ! ( ADJUNCT)
• _at_(OT PPadj).

13
Basic Structures
John waited for Mary f-str PRED 'waitltSUBJgt'
SUBJ PRED 'John' ADJ PRED 'forltOBJgt'
OBJ PRED 'Mary' o-str
PPadj
John waited for Mary f-str PRED 'waitltSUBJ
OBLgt' SUBJ PRED 'John' OBL PRED
'forltOBJgt' OBJ PRED 'Mary'
o-str PPobl
14
Ranking for Example

• Disprefer ADJUNCTs
• OPTIMALITYORDER PPadj.
• Problem will disprefer adjuncts even when no OBL
  analysis is possible
• Prefer OBLs
• OPTIMALITYORDER PPobl.
• Problem will prefer OBL even when the other
  analysis was not an ADJUNCT
• Still probably better than dispreferring ADJUNCTs
• Solution local OT marks (not discussed here)

15
Special OT marks in XLE

• Separate other marks into fields
• Marks preceding
• NOGOOD remove parts of the grammar
• for debugging or specializing
• STOPPOINT apply on a second pass
• for extending grammar on failure
• CSTRUCTURE filter when the c-structure is built
• for speed
• There is lots of discussion in the XLE
  documentation the reading on the web is a bit
  out of date for these marks

16
The NOGOOD Mark

• OT marks can be used to remove parts of the
  grammar
• rules or rule parts
• templates or template parts
• lexical items or parts of them
• Use for
• grammar adaptation/sharing
• grammar development
• Example
• OPTIMALITYORDER FrontMatter NOGOOD.

17
NOGOOD Example

• ROOT rule allows for front matter for special
  corpus
• ROOT --gt (FR-MAT ( ID)!
• _at_(OT
  FrontMatter))
• S.
• FR-MAT --gt NUMBER
• (PERIOD).
• 1. The light flashes.

18
FR-MAT

• Grammars for corpora with front matter will not
  rank the OT mark FrontMatter
• (unranked marks are neutral)
• Grammars for corpora without front matter will
  make the OT mark a NOGOOD
• OPTIMALITYORDER FrontMatter NOGOOD.
• Effective ROOT rule ROOT --gt S.
• Allows rule sharing across grammars
• Can also be used for debugging

19
Robustness

• What to do if the grammar doesn't provide an
  analysis?
• Graceful failure
• FRAGMENTs (next week)
• Specific relaxations
• Ungrammatical analysis only if no grammatical one
• Avoid ungrammatical analyses in generation

20
Robustness STOPPOINT

• On first pass, STOPPOINT is treated as NOGOOD
• Small, fast grammar for standard constructions
• If first pass fails, ignore STOPPOINT and extend
  grammar
• Relaxation possibilities precede STOPPOINT
• OPTIMALITYORDER BadDetNAgr STOPPOINT.

21
STOPPOINT Mark example

• Example NP this boy NP this boys
• Template call with OT mark
• DEMON(_P _N) ( SPEC PRED)'_P'
• ( NUM)c _N
• ( NUM) _N
• _at_(OT
  BadDetNAgr).
• Lexical entry
• this DET XLE _at_(DEMON stem sg).
• Ranking
• OPTIMALITYORDER BadDetNAgr STOPPOINT.

22
Structures for STOPOINT example
NP this boys f-str PRED 'boy' NUM pl
SPEC PRED 'this' o-str BadDetNAgr
NP this boy f-str PRED 'boy' NUM sg
SPEC PRED 'this' o-str

• Parsing this boys will be slow the grammar
• has to parse a second time
• But the ungrammatical input gets a parse
• Only put OT marks behind the STOPPOINT
• if they will be rarely triggered

23
Preference marks and STOPPOINT

• Preference marks behind the STOPPOINT are tried
  first (counter to intuitition)
• OPTIMALITYORDER MWE STOPPOINT.
• Use MWE readings if at all possible
• If fail, do a second pass with the analytic
  (non-MWE) structure (inefficient if fail)
• Example
• print quality N _at_(NOUN STEM) _at_(OT MWE).
• The N print quality is excellent.
• I want to V print NP quality documents.

24
CSTRUCTURE Marks

• Apply marks before f-structure constraints are
  processed
• OPTIMALITYORDER NoCloseQuote Guessed CSTRUCTURE.
• Improve performance by filtering early
• May loose some analyses
• coverage/efficiency tradeoff

25
CSTRUCTURE example Guessed

• Only use guessed form if another form is not
  found in the morphology/lexicon
• OPTIMALITYORDER Guessed CSTRUCTURE.
• Trade-off lose some parses, but much faster
• The foobar is good.
• no entry for foobar gt parse with guessed N
  
• The audio is good.
• audio only A in morphology gt no parse

26
CSTRUCTURE example Quote

• Only allow unbalanced quote marks if there is no
  other quote mark
• Then I left." vs. He said, "they
  appeared."
• METARULEMACRO
• _CAT QT _at_(OT NoCloseQt)
• XLE only tries balanced version, not double
  unbalanced version
• failure when really needed two unbalanced quotes

27
Combining the OT marks

• All the types of OT marks can be used in one
  grammar
• ordering of NOGOOD, CSTRUCTURE, STOPPOINT are
  important
• Example
• OPTIMALITYORDER
• Verbmobil NOGOOD
• Guessed CSTRUCTURE
• MWE Fragment STOPPOINT
• RareForm StrandedP Obl.

28
Other Features

• Grouping have marks treated as being of equal
  importance
• OPTIMALITYORDER (Paren Appositive) Adjunct.
• Ungrammatical markup have XLE report analyses
  with this mark with a
• these are treated like any dispreference mark for
  determining the optimal analyses
• OPTIMALITYORDER NoDetAgr STOPPOINT.

29
Generation

• XLE uses the same basic grammar to parse and
  generate
• Do not always want to generate all the
  possibilities that can be parsed
• Put in special OT marks for generation to block
  or prefer certain strings
• fix up bad subject-verb agreement
• only allow certain adverb placements
• control punctuation options
• GENOPTIMALITYORDER
• Details week after next

30
Other fun grammar engineering tricks

• Diagnose sources of efficiency problems (Kuhn and
  Rohrer 1997)
• systematically move marks from neutral/absent to
  dispreferred to NOGOOD
• test for speed and coverage
• Grammar-based semi-automatic lexicon acquisition
  (Kuhn, Eckle-Kohler, and Rohrer 1998)
• try possible V subcategorization frames on corpus
  sentences
• use OT marks to filter out known problems (1st
  and 2nd person objects, stative passive, V
  coordination)

31
OT Marks Main points

• Ambiguity broad coverage results in ambiguity
  OT marks allow preferences
• Robustness want fall back parses only when
  regular parses fail OT marks allow multipass
  grammar
• XLE provides for complex orderings of OT marks
• NOGOOD, CSTRUCTURE, STOPPOINT
• preference, dispreference, ungrammatical
• see the XLE documentation for details

32
FRAGMENT grammar

• What to do when the grammar does not get a parse
• always want some type of output
• want the output to be maximally useful
• Why might it fail
• construction not covered yet
• "bad" input
• took too long (XLE parsing parameters)

33
Grammar engineering approach

• First try to get a complete parse
• If fail, build up chunks that get complete parses
  (c-str and f-str)
• Have a fall back for things without even chunk
  parses
• Link these chunks and fall backs together in a
  single f-structure

34
Basic idea

• XLE has a REPARSECAT which it tries if there is
  no complete parse
• Grammar writer specifies what category the
  possible chunks are
• OT marks are used to
• build the fewest chunks possible
• disprefer using the fall back over the chunks

35
Sample output

• the the dog appears.
• Split into
• "token" the
• sentence "the dog appears"
• ignore the period

36
C-structure
37
F-structure
38
How to get this
FRAGMENTS --gt NP ( FIRST)!
_at_(OT-MARK Fragment) S ( FIRST)!
_at_(OT-MARK Fragment) TOKEN (
FIRST)! _at_(OT-MARK Fragment)
(FRAGMENTS ( REST)! ).
Lexicon -token TOKEN ( TOKEN)stem
_at_(OT-MARK Token).
39
Why First-Rest?

• FIRST-REST
• FIRST PRED
• REST FIRST PRED
• REST
• Efficient
• Encodes order
• Possible alternative set
• PRED
• PRED
• Not as efficient (copying)
• Even less efficient if mark scope facts

40
Accuracy?

• Evaluation against gold standard
• PARC 700 f-structure bank for Wall Street
  Journal
• Measure F-score on dependency triples
• F-score average of precision and recall
• Dependency triples separate f-structure
  features
• Subj(run, dog) Tense(run, past)
• Results for best-matching f-structure
• Full parses F88.5
• Fragment parses F76.7

(Riezler et al, 2002)
41
Fragments summary

• XLE has a chunking strategy for when the grammar
  does not provide a full analysis
• Each chunk gets full c-str and f-str
• The grammar writer defines the chunks based on
  what will be best for that grammar and
  application
• Quality
• Fragments have reasonable but degraded f-scores
• Usefulness in applications is being tested

42
Resource limitations Time and space
43
Exceeding available resources
44
Hard limits Time and storage

• For some applications
• No output on a few hard sentences is better than
  getting hung up, never getting to easy ones
• E.g.
• Search applications you never find everything
  anyway
• Grammar testing/debugging no surprise, move on
• XLE commands
• set timeout 60 abort after 60
  second
• set max_xle_scratch_storage 50 abort after 50
  megabytes

45
Soft limits Skimming

• Bound the f-structure effort per subtree
• Compute normally until a threshhold is reached
• set start_skimming_when_scratch_storage_exceeds
  700 (megabytes)
• set start_skimming_when_total_events_exceed XX
  (some number)
• (XX estimated from timeouts in test runs)
• Then limit the number of solutions per edge
• set max_new_events_per_graph_when_skimming XX
• Bounded computation/edge ? cubic
• Result in reasonable time/space
• At least one solution for every sentence
• But some solutions will be missed
• Suppress weighty constituents
• Limit length of medial constituents
• set max_medial_constituent_weight 20
• Dont allow medial edges that span more than 20
  terminals
• (approximation to avoiding center embedding)

46
Accuracy?

• Again, evaluation against gold standard
• PARC 700 f-structure bank for Wall Street
  Journal
• Results for best-matching f-structure
• Full parses F88.5
• Fragment parses F76.7
• Skimmed parses F70.3
• Skimmed/Fragments F61.3

(Riezler et al, 2002)
47
Introduction to Integrating Shallow Mark up (more
in two weeks) Part of speech tags Named
entities Syntactic brackets
48
Shallow mark-up of input strings

• Part-of-speech tags (tagger?)
• I/PRP saw/VBD her/PRP duck/VB.
• I/PRP saw/VBD her/PRP duck/NN.
• Named entities (named-entity recognizer)
• ltpersongtGeneral Millslt/persongt bought it.
• ltcompanygtGeneral Millslt/companygt bought it
• Syntactic brackets (chunk parser?)
• NP-S I saw NP-O the girl with the
  telescope.
• NP-S I saw NP-O the girl with the
  telescope.

49
Hypothesis

• Shallow mark-up
• Reduces ambiguity
• Increases speed
• Without decreasing accuracy
• (Helps development)

• Issues
• Markup errors may eliminate correct analyses
• Markup process may be slow
• Markup may interfere with existing robustness
  mechanisms (optimality, fragments, guessers)
• Backoff may restore robustness but decrease speed
  in 2-pass system (STOPPOINT)

50
Implementation in XLE
How to integrate with minimal changes to existing
system/grammar?
51
XLE String Processing
lexical forms
Multiwords
Modify sequences
token morphemes
Morph,Guess, Tok
Analyze
tokens
Tthe TB oil TB filter TB s TB gone TB
Decap, split, commas
Tokenize
string
The oil filters gone
52
Part of speech tags
lexical forms
Multiwords
token morphemes
Analyze

• How do tags pass thru Tokenize/Analyze?
• Which tags constrain which morphemes?
• How?

tokens
Tokenize
string
The/DET_ oil/NN_ filter/NN_s/VBZ_
gone/VBN_
53
Named entities Example input

• parse ltpersongtMr. Thejskt Thejslt/persongt
  arrived.
• tokenized string
• Mr. Thejskt Thejs TB NEperson Mr(TB). TB
  Thejskt TB Thejs

. (.) TB (, TB) .
TB arrived
TB
54
Resulting C-structure
55
Resulting F-structure
56
Syntactic brackets

• Chunker labelled bracketing
• NP-SBJ Mary and John saw NP-OBJ the girl with
  the telescope.
• They V pushed and pulled the cart.
• Implementation
• Tokenizing FST identifies, tokenizes labels
  without interrupting other patterns
• Bracketing constraints enforced by Metarulemacro
• METARULEMACRO(_CAT _BASECAT _RHS)
• _RHS
• LSB
• CAT-LB_BASECAT
• _CAT
• RSB.

57
Syntactic brackets

• NP-SBJ Mary appeared.
• Lexicon NP-SBJ CAT-LBNP (SUBJ ).

S
VP
NP
V appeared
LSB
CAT-LBNP
NP
RSB


NP-SBJ
N Mary
58
Experimental test

• Again, F-scores on PARC 700 f-structure bank
• Upper bound Sentences with best-available
  markup
• POS tags from Penn Tree Bank
• Some noise from incompatible coding
• Werner is president of the
  parent/JJ company/NN. Adj-Noun
  vs. our Noun-Noun
• Some noise from multi-word treatment
• Kleinword/NNP
  Benson/NNP /CC Co./NNP
• vs.
  Kleinword_Benson__Co./NNP
• Named entities hand-coded by us
• Labeled brackets also approximated by Penn Tree
  Bank
• Keep core-GF brackets S, NP, VP-under-VP
• Others are incompatible or unreliable discarded

59
Results
60
(No Transcript)