Tomitas Parser Tihomira Panayotova Paolina Teneva

1
Tomitas ParserTihomira PanayotovaPaolina
Teneva

• Seminar für Sprachwissenschaft
• 31.01.2007

2
A simple overview

• LR(0) conflicts
• Tomitas method summarized
• Complications
• Two optimiziations
• A moderately ambiguous grammar
• Stack duplication
• Combinig equal states
• Combinig equal stack prefixes
• Discussion
• Summary

3
LR(0) Conflicts

• LR Parser
• 1.handle recognizing FS automaton
• 2. no inadequate states
• There exist grammars for which the automaton has
  some inadequate states

4
LR(0) Conflicts
5
Tomitas method summarized

• Simple definition
• A breadth-first search over those parsing
  decisions that are not solved by the LR automaton
• It gives an efficient and very effective approach
  to grammars for which the automaton has some
  inadequate states

6
Tomitas method summarized

• How does the parser act when it encounters an
  inadequate state on the top of the stack?
• Step1. It duplicates the stack and splits the
  parse into a different process for each copy
• One copy is reserved for the REDUCE step
• The other copy is reserved for the SHIFT step
• Step2. Stacks that have a right-most state
  that does not allow a shift on the next input
  token are DISCARDED

7
Tomitas method summarized

• SHIFT step push a new symbol and the new state
  onto the stack
• REDUCE step removes part of the right end of the
  stack and replaces it with a non-terminal using
  this non-terminal as a move in the automaton, we
  find a new state to put on the top of the stack
• Conclusion Every time we encounter an inadequate
  state on the top of the stack, the duplication
  process is repeated untill all reduces have been
  treated.

8
Complications

• The repetition of the duplication process can
  cause a proliferation of stacks. A great number
  will be copied and subsequently discarded
• If all stacks are discarded in Step2 gt
• the input was in error
• Grammars with loops A-gtB
• B-gtA
• - the process may not terminate

9
Complications

• Some ideas to cope with the complications
• 1. Use of look-ahead to decide which reduces
  can be made in Step1
• 2.Grammar with loops
• 2.1. upon creating a stack, check if it is
  already
• there (and then ignore it)
• 2.2 check the grammar in advance for loops
• (and then reject it).

10
Two optimizations

• Combining equal states
• Combibing equal stack prefixes

11
A moderately ambiguous grammar

• SS -gt E
• E -gt E E
• E -gt d
• Figure 9.38 A moderately ambiguous grammar

12
LR(0) automaton for the grammar
13
Stack Duplication

• a. 1
  ddd shift
• b. 1 d 2
  dd reduce
• c. 1 E 3
  dd shift
• d. 1 E 3 4
  dd shift
• e. 1 E 3 4 d 2
  d reduce
• f. 1 E 3 4 E 5
  d duplicate to g1 and g2
  
• g1. 1 E 3 4 E 5
  d REDUCE reduce to g1.1
• g2. 1 E 3 4 E 5
  d SHIFT shift to g1.2
• g1.1 1 E 3
  d shift to h1
• g1.2 1 E 3 4 E 5 4
  d shift to h2
• h1 1 E 3 4
  d shift to h1.1
• h2 1 E 3 4 E 5 4 d 2
  reduce to h1.2
• h1.1 1 E 3 4 d 2
  reduce to i
• h1.2 1 E 3 4 E 5 4 E 5
  duplicate to i1 and i2
• i 1 E 3 4 E 5
  duplicate to j1 and j2

14
Stack Duplication

• i1 1 E 3 4 E 5 4 E 5
  REDUCE, reduce to k1
• I2 1 E 3 4 E 5 4 E 5
  SHIFT - DISCARDED
• j1 1 E 3 4 E 5
  REDUCE, reduce to k2
• j2 1 E 3 4 E 5
  SHIFT - DISCARDED
• k1 1 E 3 4 E 5
  reduce to l1
• k2 1 E 3
  shift to l2
• l1 1 E 3
  shift to m1
• l2 1 E 3 6
  reduce to m2
• m1 1 E 3 6
  reduce to n
• m2 1 S
  ACCEPT
• n 1 S
  ACCEPT

15
Parse trees
16
Combining equal states

• Examine the following
• Both stacks have the same state on topgtfurther
  actions on both stacks will be identical
• Combine the two stacks to avoid duplicate work

17
Combining equal states

• f) 1. 1 E 3 4 d 2
  both
• 2. 1 E 3 4 E 5 4 d 2
  REDUCE to g

18
Combining equal states

• f) 1. 1 E 3 4 d 2
  both
• 2. 1 E 3 4 E 5 4 d 2
  REDUCE to g
• g) 1. 1 E 3 4 E 5
  duplicate to
• 2. 1 E 3 4 E 5 4 E 5 g
  and g

19
Combining equal states

• f) 1. 1 E 3 4 d 2
  both
• 2. 1 E 3 4 E 5 4 d 2
  REDUCE to g
• g) 1. 1 E 3 4 E 5
  duplicate to
• 2. 1 E 3 4 E 5 4 E 5 g
  and g
• g) 1. 1 E 3 4 E 5
  for REDUCE
• 2. 1 E 3 4 E 5 4 E 5

20
Combining equal states

• f) 1. 1 E 3 4 d 2
  both
• 2. 1 E 3 4 E 5 4 d 2
  REDUCE to g
• g) 1. 1 E 3 4 E 5
  duplicate to
• 2. 1 E 3 4 E 5 4 E 5 g
  and g
• g) 1. 1 E 3 4 E 5
  for REDUCE
• 2. 1 E 3 4 E 5 4 E 5
• g) 1. 1 E 3 4 E 5
  copy to h3)
• 2. 1 E 3 4 E 5 4 E 5
  for SHIFT

21
Combining equal states

• g.1) 1 E 3 4 E 5
  REDUCE to h.1)

22
Combining equal states

• g.1) 1 E 3 4 E 5
  REDUCE to h.1)
• g.2) 1 E 3 4 E 5 4 E 5
  REDUCE to h.2)

23
Combining equal states

• g.1) 1 E 3 4 E 5
  REDUCE to h.1)
• g.2) 1 E 3 4 E 5 4 E 5
  REDUCE to h.2)
• h.1 ) 1 E 5
  SHIFT

24
Combining equal states

• g.1) 1 E 3 4 E 5
  REDUCE to h.1)
• g.2) 1 E 3 4 E 5 4 E 5
  REDUCE to h.2)
• h.1 ) 1 E 5
  SHIFT
• h2 ) 1 E 3 4 E 5
  REDUCE to h2.1) and h2.2)

25
Combining equal states

• g.1) 1 E 3 4 E 5
  REDUCE to h.1)
• g.2) 1 E 3 4 E 5 4 E 5
  REDUCE to h.2)
• h.1 ) 1 E 5
  SHIFT
• h2 ) 1 E 3 4 E 5
  REDUCE to h2.1) and h2.2)
• h2.1) 1 E 5
  SHIFT

26
Combining equal states

• g.1) 1 E 3 4 E 5
  REDUCE to h.1)
• g.2) 1 E 3 4 E 5 4 E 5
  REDUCE to h.2)
• h.1 ) 1 E 5
  SHIFT
• h2 ) 1 E 3 4 E 5
  REDUCE to h2.1) and h2.2)
• h2.1) 1 E 5
  SHIFT
• h2.2) 1 E 3 4 E 5
  SHIFT

27
Combining equal states

• g.1) 1 E 3 4 E 5
  REDUCE to h.1)
• g.2) 1 E 3 4 E 5 4 E 5
  REDUCE to h.2)
• h.1 ) 1 E 5
  SHIFT
• h2 ) 1 E 3 4 E 5
  REDUCE to h2.1) and h2.2)
• h2.1) 1 E 5
  SHIFT
• h2.2) 1 E 3 4 E 5
  SHIFT
• h3) 1. 1 E 3 4 E 5
  SHIFT
• 2. 1 E 3 4 E 5 4 E 5

28
Combining equal states

• Now we have five stacks (h1, h2.1, h2.2, h3).
• h1) and h2.1) carry state (3) on top
• h2.2) and h3) carry state (5) on top
• h1 ) 1 E
  3 SHIFT
• h2 ) 1 E 3 4 E 5
  REDUCE to h2.1), copy to h2.2)
• h2.1) 1 E
  3 SHIFT
• h2.2) 1 E 3 4 E 5
  SHIFT
• h3) 1. 1 E 3 4 E 5
  SHIFT
• 2. 1 E 3 4 E 5 4 E 5

29
Combining equal states

• We combine the stacks with identical states on
  top into two bundles h and h.
• h) h1) 1 E
  3 copy to i)
• h2.1) 1 E
  3

30
Combining equal states

• We combine the stacks with identical states on
  top into two bundles h and h.
• h) h1) 1 E
  3 copy to i)
• h2.1) 1 E
  3
• h) h3) 1. 1 E 3 4 E 5
• 2. 1 E 3 4 E 5 4 E 5
  discard
• h2) 1 E 3 4 E 5
  

31
Combining equal states

• i) 1) 1 E 3
• 
  3 6
• 2) 1 E 3

32
Combining equal states

• i) 1) 1 E 3
• 
  3 6
• 2) 1 E 3
• i) 1 E 3
  6 REDUCE to j1)

33
Combining equal states

• i) 1) 1 E 3
• 
  3 6
• 2) 1 E 3
• i) 1 E 3
  6 REDUCE to j1)
• i) 1 E 3
  6 REDUCE to j2)

34
Combining equal states

• i) 1) 1 E 3
• 
  3 6
• 2) 1 E 3
• i) 1 E 3
  6 REDUCE to j1)
• i) 1 E 3
  6 REDUCE to j2)
• j1) 1 S
  ACCEPT

35
Combining equal states

• i) 1) 1 E 3
• 
  3 6
• 2) 1 E 3
• i) 1 E 3
  6 REDUCE to j1)
• i) 1 E 3
  6 REDUCE to j2)
• j1) 1 S
  ACCEPT
• j2) 1 S
  ACCEPT

36
Combining equal stack prefixes

• When the parser makes the call for the stack
• to be copied,there is no actual need to copy
  the entire stack!
• It is enough to copy the top state suffixes

37
Combining Equal Stack-Prefixes

• If we observe the example
• e) 1 E 3 4 E 5 d

38
Combining Equal Stack-Prefixes

• If we observe the example
• e) 1 E 3 4 E 5 d
• When we duplicate the stack we have two copies of
  
• It and REDUCE is applied only to one of the
  copies and
• only so much of the stack is copied

39
Combining Equal Stack-Prefixes

• If we observe the example
• e) 1 E 3 4 E 5 d
• When we duplicate the stack we have two copies of
  
• it and REDUCE is applied only to one of the
  copies and
• only so much of the stack is copied
• e) 1 E 3 d SHIFT
• e) 1 E 3 4 E 5 d SHIFT

40
Discussion

• Table characteristics
• -the method can work with every bottom-up
  table
• -the weaker the table, the more
  non-determinism will have to be resolved by
  breadth-first search
• Time requirements
• - in theory exponential
• - in practice - linear or slightly more than
  linear

41
SUMMARY

• Breadth-first search over those parsing decisions
  that are not solved by the LR automaton
• Important notions that should be memorized
• stack duplication (inadequate states, reduce,
  shift, discarded)
• combining equal states
• combining equal stack prefixes

42
References

• Dick Grune Ceriel Jacobs (1990). Parsing
  Techniques