Introduction to Cognitive Science

1
16th International Summer School in Cognitive
Science, New Bulgarian University, Sofia,
Bulgaria, 2009
The Embodiment of Mind as a Natural Result of
Interactive Dynamics
Michael J. Spivey Department of Cognitive
Science University of California, Merced
2
Embodied Cognition
COURSE OVERVIEW
Day 1 Embodied cognition is unavoidable in
an organism that allows its subsystems to richly
interact
Day 2 Findings in the Embodied Cognition
literature
Day 3 How classical symbolic cognitive science
would like to interpret embodied cognition
findings
Day 4 Findings of the motor system directly
constraining cognitive processes
Day 5 Computational models that exploit
the embodiment of cognition
3
The Continuity of Mind
OUTLINE
The Embodiment of Mind
Temporal Dynamics in Neuronal Population Coding
Temporal Dynamics in Spoken Word Recognition
Temporal Dynamics in Sentence Processing
Temporal Dynamics in Question Answering
Mental Trajectories in Neuronal State Space
4
The Continuity of Mind
OUTLINE
The Embodiment of Mind
Temporal Dynamics in Neuronal Population Coding
Temporal Dynamics in Spoken Word Recognition
Temporal Dynamics in Sentence Processing
Temporal Dynamics in Question Answering
Mental Trajectories in Neuronal State Space
5
Theories of Embodied Mind
Embodiment of Language (e.g., Lakoff
Johnson, 1980, 1999 Sweetser, 1998 Gibbs,
2006 Langacker, 2008 Talmy, 2006) Embodiment
of Cognition (e.g., Barsalou, 1999 MacWhinney,
1999 Glenberg, 1997 Spivey, 2007 Varela,
Thompson, Rosch, 1992) Embodiment of
Perception (e.g., Gibson, 1979 Turvey, 1992
Hommel et al., 2001) Embodiment of Cognitive
Development (e.g., J. Mandler, 1992 L. Smith,
2005) Embodiment of Artificial Intelligence
(e.g., Brooks, 1991 Ballard et al., 1997 Roy,
2005 Steels, 2003)
6
Findings of Embodied Mind
Page 1 of 2 Motor areas are active when
hearing action verbs (Pulvermüller, 1999)
Sentences evoke perceptual simulations (Stanfield
Zwaan, 2001) Sentences evoke motor
simulations (Glenberg Kaschak, 2002) Real
and fictive motion priming influences reasoning
about language (Boroditsky Ramscar, 2002
Matlock, Ramscar, Boroditsky, 2005) Image
schemas activated by language interfere with
perceptual input (Richardson, Spivey, Barsalou,
McRae, 2003 Bergen et al., 2007) Summaries
of much of this work can be found in Pecher
Zwaan (2005).
7
Findings of Embodied Mind
Page 2 of 2 Sync in postural sway emerges
during mutual conversation (Shockley, Santana,
Fowler, 2003) Sync in eye movements at
shared scene emerges during conversation
(Richardson Dale, 2005) Fictive motion
sentences induce more directional scanning of
scene (Richardson Matlock, 2007) Motion
words influence visual perception of motion, and
vice versa (Meteyard, Bahrami, Vigliocco,
2007 Meteyard et al. 2008) Direction words
heard during a reach cause curvature in the
trajectory (Boulenger et al., 2006 Nazir et
al., 2008)
8
The Continuity of Mind
OUTLINE
The Embodiment of Language
Temporal Dynamics in Neuronal Population Coding
Temporal Dynamics in Spoken Word Recognition
Temporal Dynamics in Sentence Processing
Temporal Dynamics in Question Answering
Mental Trajectories in Neuronal State Space
9
Neural Dynamics
(Tyler Spivey, 2005)
10
Neural Dynamics
11
Neural Dynamics
12
Neural Dynamics
13
Neural Dynamics
14
Neural Dynamics
15
Neural Dynamics
16
Neural Dynamics
17
Neural Dynamics
18
Neural Dynamics
19
Neural Dynamics
20
Neural Dynamics
21
Neural Dynamics
Population Code for candy
22
Neural Dynamics
Population Code for candle
23
Neural Dynamics
0 ms
24
Neural Dynamics
100 ms
25
Neural Dynamics
200 ms
26
Neural Dynamics
300 ms
27
Neural Dynamics
400 ms
28
Neural Dynamics
Population code for recognizing candy
29
Neural Dynamics
Population code for recognizing candle
30
Neural Dynamics
0ms
31
Neural Dynamics
100ms
32
Neural Dynamics
200ms
33
Neural Dynamics
300ms
34
Neural Dynamics
Population code for recognizing candle
400ms
35
Visual Processing
Perrett, Oram, Ashbridge (1998)
600
frontal
Cumulative Response (spikes)
300
3/4 profile
profile
1/4 profile
back-of-head
0
0
500
1000
After Stimulus Onset (ms)
36
Visual Processing
Temporal dynamics of population coding (Rolls
Tovee, 1995)
Cumulative Information (bits)
(ms)
If normal eye movements occur 3-4 times per
second, then a population code rarely gets enough
time to asymptote!
37
Visual Processing
Multi-dimensional scaling of the 14 cells (Rolls
Tovee, 1995)
38
Visual Processing
Trajectories in state space for recognizing faces
and objects
39
Visual Processing
Trajectories in state space as a result of
free-viewing of a scene
40
Artificial Neural Dynamics
McClelland Elmans (1986) TRACE Model of Speech
Processing
41
Artificial Neural Dynamics
TRACE Model of Speech Perception
(McClelland Elman, 1986)
1-sum candle candy pencil penny
Normalized Activation
42
Dynamical Systems
Energy Landscape
43
Dynamical Systems
Vector Field
44
Dynamical Systems
Vector Field
45
Dynamical Systems
Vector Field
46
Quantitative Equivalence
On the equivalence between dynamic neural
patterns and a trajectory through mental state
space
(Onnis Spivey, submitted)
47
The Continuity of Mind
Fodors (1983) modularity of mind has been
giving way to a continuity of mind.
Visual Perception (Motter, 1993 Sekuler,
Sekuler, Lau, 1997 Spivey Spirn, 2000)
Spoken Word Recognition (Elman McClelland,
1988 Allopenna et al., 1998) Sentence
Processing (MacDonald Seidenberg, 1993
Tanenhaus Trueswell, 1995 Chambers,
Tanenhaus, Magnuson, 2004) Conceptual
Integration and Blending (Fauconnier Turner,
2002 Coulson, 2001) Construction Grammar,
eschewing the distinction between core grammar
and periphery (Fillmore, 1988 Goldberg, 2003)
48
The Continuity of Mind
OUTLINE
The Embodiment of Language
Temporal Dynamics in Neuronal Population Coding
Temporal Dynamics in Spoken Word Recognition
Temporal Dynamics in Sentence Processing
Temporal Dynamics in Question Answering
Mental Trajectories in Neuronal State Space
49
Dynamics and Eye Movements
Headband-mounted Eyetracking
50
Dynamics and Eye Movements
Headband-mounted Eyetracking
51
Dynamics and Eye Movements
Why headband-mounted eyetracking?
Eye position provides a continuous measure
without interrupting processing.
Eye movements are largely resistant to
strategic control.
Saccades are frequent 3-4 times per second.
Headband allows ecologically valid continuous
interaction between human and environment
Saccades are promiscuous (low threshold for
execution), and thus sensitive to subtle
probabilistic biases.
52
Dynamics and Eye Movements
Vector Field
thresholds for a saccade
53
Spoken Word Recognition
Pick up the candle.
(Tanenhaus, Spivey-Knowlton, Eberhard, Sedivy,
1996)
54
Spoken Word Recognition
Pick up the candle.
(Tanenhaus, Spivey-Knowlton, Eberhard, Sedivy,
1996)
55
Spoken Word Recognition
1.0
.8
.6
Probability of Fixation
.4
.2
0
(ms)
(Spivey-Knowlton, 1996)
56
Spoken Word Recognition
McClelland Elmans (1986) TRACE Model of Speech
Processing
57
Spoken Word Recognition
TRACE Model of Speech Processing
(Allopenna, Magnuson Tanenhaus, 1998)
58
Spoken Word Recognition
Eye-movement data
(Spivey-Knowlton, 1996)
1.0
.8
.6
Probability of Fixation
.4
.2
0
(ms)
59
Spoken Word Recognition
Saliency Map
Pick up the
(Reali, Spivey, Tyler, Terranova, 2006)
60
Spoken Word Recognition
Saliency Map
Pick up the ca
(Reali, Spivey, Tyler, Terranova, 2006)
61
Spoken Word Recognition
Saliency Map
Pick up the cand
(Reali, Spivey, Tyler, Terranova, 2006)
62
Spoken Word Recognition
Saliency Map
Pick up the candle.
(Reali, Spivey, Tyler, Terranova, 2006)
63
Spoken Word Recognition
Continuous Non-ballistic Movements
(Spivey, Grosjean, Knoblich, 2005)
64
Spoken Word Recognition
Mouse-click start box at bottom
(Spivey, Grosjean, Knoblich, 2005)
65
Spoken Word Recognition
Click the ladle
(Spivey, Grosjean, Knoblich, 2005)
66
Spoken Word Recognition
900
800
700
600
y
500
400
300
200
0
200
400
600
800
1000
x
(Spivey, Grosjean, Knoblich, 2005)
67
Spoken Word Recognition
(Spivey, Grosjean, Knoblich, 2005)
68
Spoken Word Recognition
Click the beetle
(Spivey, Grosjean, Knoblich, 2005)
69
Spoken Word Recognition
(Spivey, Grosjean, Knoblich, 2005)
70
Spoken Word Recognition
Graded spatial attraction toward phonological
competitors visible in averaged trajectories
(Spivey, Grosjean, Knoblich, 2005)
71
The Continuity of Mind
72
The Continuity of Mind
OUTLINE
The Embodiment of Language
Temporal Dynamics in Neuronal Population Coding
Temporal Dynamics in Spoken Word Recognition
Temporal Dynamics in Sentence Processing
Temporal Dynamics in Question Answering
Mental Trajectories in Neuronal State Space
73
Sentence Processing
Incrementality in Syntax
Sentence
Verb Phrase
Syntactic Ambiguity
Verb
Noun Phrase
Prep. Phrase
...
Preposition
Put
the
apple
on...
74
Sentence Processing
In a Visual Context
(Tanenhaus, Spivey-Knowlton, Eberhard Sedivy,
1995)
Put the apple on the towel in the box
Ambiguous
Put the apple thats on the towel in the box
Unambiguous
One-Referent Context
Two-Referent Context
75
Sentence Processing
One-Referent Context
Tanenhaus, Spivey-Knowlton, Eberhard Sedivy,
1995)
76
Sentence Processing
Two-Referent Context
Tanenhaus, Spivey-Knowlton, Eberhard Sedivy,
1995)
77
Sentence Processing
In a Visual Context
((Tanenhaus, Spivey-Knowlton, Eberhard Sedivy,
1995)
78
Sentence Processing
Continuous Non-ballistic Reaching Movements
79
Sentence Processing
Put the apple on the towel in the box
Ambiguous
Put the apple thats on the towel in the box
Unambiguous
Two-Referent Context
One-Referent Context
(Farmer, Anderson, Spivey, 2007)
80
Sentence Processing
(Farmer, Anderson, Spivey, 2007)
81
Sentence Processing
(Farmer, Anderson, Spivey, 2007)
82
Sentence Processing
(Farmer, Anderson, Spivey, 2007)
83
Sentence Processing
(Farmer, Anderson, Spivey, 2007)
84
Sentence Processing
One-Referent Context, Ambiguous Sentence
(Farmer, Anderson, Spivey, 2007)
85
Sentence Processing
(Farmer, Anderson, Spivey, 2007)
86
Sentence Processing
(Farmer, Anderson, Spivey, 2007)
87
Sentence Processing
(Farmer, Anderson, Spivey, 2007)
88
Sentence Processing
(Farmer, Anderson, Spivey, 2007)
89
Sentence Processing
(Farmer, Anderson, Spivey, 2007)
90
Sentence Processing
(Farmer, Anderson, Spivey, 2007)
91
Sentence Processing
(Farmer, Anderson, Spivey, 2007)
92
The Continuity of Mind
OUTLINE
The Embodiment of Language
Temporal Dynamics in Neuronal Population Coding
Temporal Dynamics in Spoken Word Recognition
Temporal Dynamics in Sentence Processing
Temporal Dynamics in Question Answering
Mental Trajectories in Neuronal State Space
93
Decision Making
YES
NO
Is a thousand more than a million?
start
(McKinstry, Dale, Spivey, 2008)
94
Decision Making
YES
NO
Should you brush your teeth every day?
start
(McKinstry, Dale, Spivey, 2008)
95
Decision Making
YES
NO
Is murder sometimes justifiable?
start
(McKinstry, Dale, Spivey, 2008)
96
Decision Making
(McKinstry, Dale, Spivey, 2008)
97
Decision Making
(McKinstry, Dale, Spivey, 2008)
98
Decision Making
(McKinstry, Dale, Spivey, 2008)
99
Decision Making
(McKinstry, Dale, Spivey, 2008)
100
Continuity Among Mind and Body
OUTLINE
The Embodiment of Language
Temporal Dynamics in Neuronal Population Coding
Temporal Dynamics in Spoken Word Recognition
Temporal Dynamics in Sentence Processing
Temporal Dynamics in Question Answering
Mental Trajectories in Neuronal State Space
101
Continuity Among Mind and Body
Toward an equivalence between dynamic neural
patterns and a trajectory through mental state
space
(Onnis Spivey, submitted)
102
Continuity Among Mind and Body
103
Continuity Among Mind and Body
The new form of insight can perhaps best be
called Undivided Wholenesss in Flowing Movement.
This view implies that flow is, in some sense,
prior to that of the things that can be seen to
form and dissolve in the flow. -David
Bohm, Wholeness and the Implicate Order (1980)
104
Slides for questions
105
A. Weights over time for alternative reaches
control condition
cohort condition
candy
candy
Relative Activation
candle foil
book foil
(Spivey, Dale, Grosjean, Knoblich, in press)
Time steps
B. Simulated reach movements
control

cohort
reach to foil
reach to target
Y (cm)
X (cm)
106
The Continuity of Mind
Quasilinguistic thought while learning racquetball
107
Visual Processing
Multi-dimensional scaling of the 14 cells (Rolls
Tovee, 1995)
108
Visual Processing
Temporal dynamics of population coding (Rolls
Tovee, 1995)
109
Visual Processing
Trajectories in state space for recognizing faces
and objects
110
Visual Processing
Trajectories in state space as a result of
free-viewing of a scene
111
The Continuity of Mind
Symbolic Dynamics