The%20Neural%20Basis%20of%20Thought%20and%20Language

1
The Neural Basis ofThought and Language

• Final
• Review Session

2
Administrivia

• Final in class next Tuesday, May 9th
• Be there on time!
• Format
• closed books, closed notes
• short answers, no blue books
• And then youre done with the course!

3
The Second Half
Cognition and Language
Computation
Structured Connectionism
abstraction
Computational Neurobiology
Biology
Midterm
Final
4
Overview

• Bailey Model
• feature structures
• Bayesian model merging
• recruitment learning
• KARMA
• X-schema, frames
• aspect
• event-structure metaphor
• inference

• Grammar Learning
• parsing
• construction grammar
• learning algorithm
• SHRUTI
• FrameNet
• Bayesian Model of Human Sentence Processing

5
Full Circle
Embodied Representation
Structured Connectionism
Probabilistic algorithms
ConvergingConstraints
6
Q A
7
How can we capture the difference between

• Harry walked into the cafe.
• Harry is walking into the cafe.
• Harry walked into the wall.

8
Harry walked into the café.
Utterance
Constructions
Analysis Process
Semantic Specification
General Knowledge
Belief State
Simulation
9
The INTO construction
construction INTO subcase of Spatial-Relation f
orm selff .orth ? into meaning
Trajector-Landmark evokes Container as
cont evokes Source-Path-Goal as spg trajector
? spg.trajector landmark ? cont cont.interior
? spg.goal cont.exterior ? spg.source
10
The Spatial-Phrase construction
construction SPATIAL-PHRASE constructional cons
tituents sr Spatial-Relation lm
Ref-Expr form srf before lmf meaning srm.lan
dmark ? lmm
11
The Directed-Motion construction
construction DIRECTED-MOTION constructional con
stituents a Ref-Exp m Motion-Verb p
Spatial-Phrase form af before mf mf
before pf meaning evokes Directed-Motion as
dm selfm.scene ? dm dm.agent ? am dm.motion
? mm dm.path ? pm
schema Directed-Motion roles agent
Entity motion Motion path SPG
12
What exactly is simulation?

• Belief update plus X-schema execution

13
Harry walked into the café.
walk
ready
done
14
Harry is walking to the café.
Utterance
Constructions
Analysis Process
Semantic Specification
General Knowledge
Belief State
Simulation
15
Harry is walking to the café.
WALK
16
Harry has walked into the wall.
Utterance
Constructions
Analysis Process
Semantic Specification
General Knowledge
Belief State
Simulation
17
Perhaps a different sense of INTO?
construction INTO subcase of spatial-prep form
selff .orth ? into meaning evokes
Trajector-Landmark as tl evokes Container as
cont evokes Source-Path-Goal as
spg tl.trajector ? spg.trajector tl.landmark
? cont cont.interior ? spg.goal cont.exterior
? spg.source
construction INTO subcase of spatial-prep form
selff .orth ? into meaning evokes
Trajector-Landmark as tl evokes Impact as
im evokes Source-Path-Goal as
spg tl.trajector ? spg.trajector tl.landmark
? spg.goal im.obj1 ? tl.trajector im.obj2 ?
tl.landmark
18
Harry has walked into the wall.
WALK
19
Map down to timeline
consequence
20
further questions?
21
What about

• Harry walked into trouble
• or for stronger emphasis,
• Harry walked into trouble, eyes wide open.

22
Metaphors

• metaphors are mappings from a source domain to a
  target domain
• metaphor maps specify the correlation between
  source domain entities / relation and target
  domain entities / relation
• they also allow inference to transfer from source
  domain to target domain (possibly, but less
  frequently, vice versa)

ltTARGETgt is ltSOURCEgt
23
Event Structure Metaphor

• Target Domain event structure
• Source Domain physical space
• States are Locations
• Changes are Movements
• Causes are Forces
• Causation is Forced Movement
• Actions are Self-propelled Movements
• Purposes are Destinations
• Means are Paths
• Difficulties are Impediments to Motion
• External Events are Large, Moving Objects
• Long-term Purposeful Activities are Journeys

24
KARMA

• DBN to represent target domain knowledge
• Metaphor maps link target and source domain
• X-schema to represent source domain knowledge

25
Metaphor Maps

• map entities and objects between embodied and
  abstract domains
• invariantly map the aspect of the embodied domain
  event onto the target domain
• by setting the evidence for the status variable
  based on controller state (event structure
  metaphor)
• project x-schema parameters onto the target domain

26
further questions?
27
How do you learn

• the meanings of spatial relations,
• the meanings of verbs,
• the metaphors, and
• the constructions?

28
How do you learn

• the meanings of spatial relations,
• the meanings of verbs,
• the metaphors, and
• the constructions?

Thats the Regier model. (first half of semester)
29
How do you learn

• the meanings of spatial relations,
• the meanings of verbs,
• the metaphors, and
• the constructions?

VerbLearn
30
(No Transcript)
31
schema elbow jnt posture accel
slide 0.9 extend 0.9 palm 0.9 6
schema elbow jnt posture accel
slide 0.9 extend 0.9 palm 0.9 6 - 8
schema elbow jnt posture
slide 0.9 extend 0.9 palm 0.7 grasp 0.3
schema elbow jnt posture accel
depress 0.9 fixed 0.9 index 0.9 2
schema elbow jnt posture accel
slide extend palm 6
data 1
schema elbow jnt posture accel
slide extend palm 8
data 2
schema elbow jnt posture accel
depress fixed index 2
data 3
schema elbow jnt posture accel
slide extend grasp 2
data 4
32
Computational Details

• complexity of model ability to explain data
• maximum a posteriori (MAP) hypothesis

33
How do you learn

• the meanings of spatial relations,
• the meanings of verbs,
• the metaphors, and
• the constructions?

conflation hypothesis (primary metaphors)
34
How do you learn

• the meanings of spatial relations,
• the meanings of verbs,
• the metaphors, and
• the constructions?

construction learning
35
Usage-based Language Learning
Reorganize
Hypothesize
Partial
Acquisition
36
Main Learning Loop

• while ltutterance, situationgt available and cost gt
  stoppingCriterion
• analysis analyzeAndResolve(utterance,
  situation, currentGrammar)
• newCxns hypothesize(analysis)
• if cost(currentGrammar newCxns) lt
  cost(currentGrammar)
• addNewCxns(newCxns)
• if (re-oganize true) // frequency depends on
  learning parameter
• reorganizeCxns()

37
Three ways to get new constructions

• Relational mapping
• throw the ball
• Merging
• throw the block
• throwing the ball
• Composing
• throw the ball
• ball off
• you throw the ball off

38
Minimum Description Length

• Choose grammar G to minimize cost(GD)
• cost(GD) a size(G) ß complexity(DG)
• Approximates Bayesian learning cost(GD)
  posterior probability P(GD)
• Size of grammar size(G) 1/prior P(G)
• favor fewer/smaller constructions/roles
  isomorphic mappings
• Complexity of data given grammar 1/likelihood
  P(DG)
• favor simpler analyses(fewer, more likely
  constructions)
• based on derivation length score of derivation

39
further questions?
40
Connectionist Representation

• How can entities and relations be represented at
  the structured connectionist level?
• or
• How can we represent
• Harry walked to the café
• in a connectionist model?

41
SHRUTI

• entity, type, and predicate focal clusters
• An entity is a phase in the rhythmic activity.
• Bindings are synchronous firings of role and
  entity cells
• Rules are interconnection patterns mediated by
  coincidence detector circuits that allow
  selective propagation of activity
• An episode of reflexive processing is a transient
  propagation of rhythmic activity

42
Harry walked to the café.

• asserting that walk(Harry, café)
• Harry fires in phase with agent role
• cafe fires in phase with goal role

entity
type
predicate
43
Harry walked to the café.

• asserting that walk(Harry, café)
• Harry fires in phase with agent role
• cafe fires in phase with goal role

entity
type
predicate
44
Activation Trace for walk(Harry, café)
45
further questions?
46
Human Sentence Processing

• Can we use any of the mechanisms we just
  discussed
• to predict reaction time / behavior
• when human subjects read sentences?

47
Good and Bad News

• Bad news
• No, not as it is.
• ECG, the analysis process and simulation process
  are represented at a higher computational level
  of abstraction than human sentence processing
  (lacks timing information, requirement on
  cognitive capacity, etc)
• Good news
• we can construct bayesian model of human sentence
  processing behavior borrowing the same insights

48
Bayesian Model of Sentence Processing

• Do you wait for sentence boundaries to interpret
  the meaning of a sentence? No!
• As words come in, we construct
• partial meaning representation
• some candidate interpretations if ambiguous
• expectation for the next words
• Model
• Probability of each interpretation given words
  seen
• Stochastic CFGs, N-Grams, Lexical valence
  probabilities

49
SCFG N-gram
Reduced Relative
Stochastic CFG
50
SCFG N-gram
Reduced Relative
Main Verb
N-Gram
51
SCFG N-gram
Different Interpretations
Reduced Relative
Main Verb
52
Predicting effects on reading time

• Probability predicts human disambiguation
• Increase in reading time because of...
• Limited Parallelism
• Memory limitations cause correct interpretation
  to be pruned
• The horse raced past the barn fell
• Attention
• Demotion of interpretation in attentional focus
• Expectation
• Unexpected words

53
Open for questions