A computational model of sentence and story comprehension'

1
A computational model of sentence and story
comprehension.

• Stefan Frank
• Nijmegen Institute for Cognition and Information
• In cooperation with
• Mathieu Koppen (NICI)
• Leo Noordman (UvT)
• Wietske Vonk (MPI for Psycholinguistics and KUN)

2
Story comprehension

• The first lines of the Jip en Janneke (Bob and
  Jilly) story Going to the shoemaker
• My shoe has a hole, Bob says.
• My foot is wet.
• In order to comprehend this pair of sentences,
  inferences need to be made
• Bobs foot is wet because his shoe has a hole and
  he walked through a puddle.

3
Some discourse-comprehension theory

• Kintsch Van Dijk (1978) The comprehension of a
  text leads to three distinct levels of mental
  representation

• The surface text (literal wording)
• The textbase (propositional structure)
• The situation model (what happens in the text)

4
An experiment

• Fletcher Chrysler (1990)
• Subjects read short stories.
• They were given a sentence of each story, without
  the last word.
• They had to recognize which of two given words
  was the original last word of the sentence.

5
Results
6
My shoe has a hole, Bob says.
My foot is wet.
Surface
7
The Distributed Situation Space model Frank,
Koppen, Noordman, Vonk, 2003

• The DSS model simulates how readers make
  knowledge-based inferences during story
  comprehension.
• It represents the story at the situational level
  only.
• Stories take place in a microworld.

8
Bob and Jillys microworld
Basic events 1. the sun shines2. it
rains 3. Bob is outside4. Jilly is
outside 5. Bob and Jilly play soccer6. Bob and
Jilly play hide-and-seek7. Bob plays a computer
game8. Jilly plays a computer game9. Bob plays
with the dog10. Jilly plays with the dog 11. Bob
is tired12. Jilly is tired 13. Bob
wins14. Jilly wins
9
Microworld rules

• For instance
• Soccer can only be played outside, computer games
  only inside.
• Someone can only win if Bob and Jilly play
  soccer, hide-and-seek, or both play a computer
  game.
• Whoever is tired, is less likely to win.

10
The microworld description

• A temporal sequence of 250 situations, obeying
  the mircroworld rules.
• In all situations, each basic event is either
  true or false.
• World knowledge and representations of microworld
  events are extracted from the microworld
  description .

11
Representing situations
A Self-Organizing Map (SOM) was trained on the
microworld description
bob is outside
jilly is outside
bj play soccer
bj play hide-and-seek
12
Combining situations

bob is outside
not
bob is inside

?
jilly is outside
and
jilly plays with the dog outside
jilly plays with the dog

?
bj play soccer
bj play hide-and-seek
or
13
An inference example
Inferences result from applying world knowledge
about causal/temporal relations.
It rains. Bob and Jilly are playing hide-and-seek.
bob and jilly play hide-and-seek
it rains
14
Another example

• 1. The sun shines and Bob and Jilly play soccer.
• 2. Bob is tired, but Jilly isnt.
• 3. Next, one of them wins.
• Who wins?
• What are they playingat that moment?

15
General results

• An inference takes place if
• it is likely given the story and world knowledge
• it leads to increased coherence
• processing is sufficiently deep

which is in accordance with Vonk Noordmans
(1990) theory that inferencing depends on

• the availability of relevant knowledge
• the contribution to the texts coherence
• the readers goals.

16
Model results and experimental data

• Processing a less related statement leads to
  longer processing times and more inference.
• Increasing depth-of-processing results in longer
  processing times and more inference.
• Story retention recall percentages and
  intrusions
• Ambiguous pronoun resolution reading times and
  error rates

17
Conclusions

• The DSS model
• simulates the making of knowledge-based
  inferences
• predicts experimental data
• supports Vonk Noordmans theory of inference
• uses only a situation-level representation.

But how about the other two levels surface text
and textbase?
18
Text input to the DSS model

• Situations in the microworld can be described
  using a microlanguage.
• A neural network is trained to transform
  microlanguage sentences into the corresponding
  situational representation.
• The existence of a textbase-level representation
  is not assumed.

19
A simple model of sentence comprehension
Bob, and, Jilly, play, soccer
Surface
.
During training, intermediate representations of
sentences develop in the hidden layer.
hidden layer
Textbase?
Situation
bob and jilly play soccer
20
Comparing the intermediate representations

• The intermediate representation of a sentence is
  a vector of activation values.
• The euclidean distance between two vectors is a
  measure for the dissimilarity between the
  corresponding sentences
• Surface text difference Bob and Jilly play
  soccer vs. Jilly and Bob play soccer
• Textbase differenceBob plays soccer vs. Jilly
  plays soccer
• Situation differenceBob plays with the
  dog vs. Jilly plays with the dog

21
Fletcher Chryslers (1990) results
22
Model results
The model predicts Fletcher Chryslers results
within a single level of representation.
23
Conclusions

• The textbase may be just an intermediate level
  that is useful for constructing the situational
  representation.
• It does not consist of propositional structures.
• Intermediate representations combine properties
  of surface text, proposition, and situation.
• But
• This simple model cannot predict reading times.
• And it lacks an effect of world knowledge on
  sentence processing.

24
A more complex model of sentence comprehension
Bob, and, Jilly, play, soccer
.
Surface
Fully recurrent network
Textbase?
Interactive connections
Situation