Abstract Neuron

1
Abstract Neuron
2
Link to Vision The Necker Cube
3
(No Transcript)
4
Constrained Best Fit in Nature
inanimate animate
physics lowest energy state
chemistry molecular minima
biology fitness, MEU Neuroeconomics
vision threats, friends
language errors, NTL
5
Computing other relations

• The 2/3 node is a useful function that activates
  its outputs (3) if any (2) of its 3 inputs are
  active
• Such a node is also called a triangle node and
  will be useful for lots of representations.

6
Triangle Nodes Encoding relational information
with abstract neurons

• The triangle node (aka 2/3 node) is a useful
  function that activates its outputs (3) if any
  (2) of its 3 inputs are active
• Such a node will be useful for lots of
  representations.

7
Triangle nodes and McCullough-Pitts Neurons
Relation (A)
Object (B)
Value (C)
A
B
C
8
Basic Ideas

• Parallel activation streams.
• Top down and bottom up activation combine to
  determine the best matching structure.
• Triangle nodes bind features of objects to values
• Mutual inhibition and competition between
  structures
• Mental connections are active neural connections

9
5 levels of Neural Theory of Language
Spatial Relation
Motor Control
Pyscholinguistic experiments
Metaphor
Grammar
Cognition and Language
Computation
Structured Connectionism
abstraction
Neural Net
SHRUTI
Computational Neurobiology
Triangle Nodes
Biology
Neural Development
Midterm
Quiz
Finals
10

• Behavioral Experiments
• Identity Mental activity is Structured Neural
  Activity
• Spreading Activation Psychological model/theory
  behind priming and interference experiments
• Simulation Necessary for meaningfulness and
  contextual inference
• Parameters Govern simulation, strict inference,
  link to language

11
Bottom-up vs. Top-down Processes

• Bottom-up When processing is driven by the
  stimulus
• Top-down When knowledge and context are used to
  assist and drive processing
• Interaction The stimulus is the basis of
  processing but almost immediately top-down
  processes are initiated

12
Stroop Effect

• Interference between form and meaning

13
Name the words

• Book Car Table Box Trash Man Bed
• Corn Sit Paper Coin Glass House Jar
• Key Rug Cat Doll Letter Baby Tomato
• Check Phone Soda Dish Lamp Woman

14
Name the print color of the words

• Blue Green Red Yellow Orange Black Red
• Purple Green Red Blue Yellow Black Red
• Green White Blue Yellow Red Black Blue
• White Red Yellow Green Black Purple

15
Body-Specificity Hypothesis
If concepts and word meanings are constituted, in
part, by mental simulations our own perceptions
and actions then their neurocognitive
representations should differ for people with
different kinds of bodies, who perceive and act
upon the environment in systematically different
ways.
(Casasanto, in review, link on course page)
16
Testing body-specificity
pinch
chew
Manual Action
Non-Manual Action
17
Testing body-specificity
Left handed movements for one block of trials
Right handed movements for the other block of
trials
18
Design

• 96 Words (48 manual verbs, 48 non-manual verbs)
• 2 blocks (LH movements, RH movements)
• 2 groups of Ss (Left handers, Right handers)
• Dependent measures RT Surprise recognition
  (Old/New)

19
Move marbles by word color
caress jab fling grip pound tap yank erase dial
sigh cough growl watch crawl peek tumble dance say
Manual Action
Non-Manual Action
20

• Predictions
• If action word meanings are constituted, in part,
  by mental simulation of perceptuo-motor
  experiences, then
• Both online and offline effects of congruity
  should be found between manual motor actions and
  the meanings of manual action verbs (but not
  non-manual action verbs).
• Right and left handed participants should show
  opposite effects of using their right and left
  hands to move marbles during incidental encoding
  of manual action verbs.

(Casasanto, in review)
21
Reaction Time Results


Marble movement time in ms
Manual Action Verb
(Casasanto, in review)
22
Handedness-ness predicts congruity effect
23
Recognition Memory Results
0.80
LeftHand
RightHand
0.75


0.70
Proportion correct recognition
0.65
0.60
0.55
0.50
LEFTIES
RIGHTIES
Manual Action Verb
(Casasanto, in review)
24
Procedure for experiment that demonstrates the
word-superiority effect. First the word is
presented, then the mask XXXXs, then the letters.
25
Word-Superiority Effect Reicher (1969)

• Which condition resulted in faster more
  accurate recognition of the letter?
• The word condition
• Letters are recognized faster when they are part
  of a word then when they are alone
• This rejects the completely bottom-up feature
  model
• Also a challenge for serial processing

26
Connectionist ModelMcClelland Rumelhart (1981)

• Knowledge is distributed and processing occurs in
  parallel, with both bottom-up and top-down
  influences
• This model can explain the Word-Superiority
  Effect because it can account for context effects

27
Connectionist Model of Word Recognition
28
Basic Ideas

• Parallel activation streams.
• Top down and bottom up activation combine to
  determine the best matching structure.
• Triangle nodes bind features of objects to values
• Mutual inhibition and competition between
  structures
• Mental connections are active neural connections

29
Interaction in language processing Pragmatic
constraints on lexical access

• Jim Magnuson
• Columbia University

30
Information integration

• A central issue in psycholinguistics and
  cognitive science
• When/how are such sources integrated?
• Two views
• Interaction
• Use information as soon as it is available
• Free flow between levels of representation
• Modularity
• Protect and optimize levels by encapsulation
• Staged serial processing
• Reanalyze / appeal to top-down information only
  when needed

31
(No Transcript)
32
Reaction Times in Milliseconds after They all
rose
0 delay 200ms. delay
flower 685 659
stood 677 623
desk 711 652
33
Example Modularity and word recognition

• Tanenhaus et al. (1979) also Swinney, 1979
• Given a homophone like rose, and a context biased
  towards one sense, when is context integrated?
• Spoken sentence primes ending in homophones
• They all rose vs. They bought a rose
• Secondary task name a displayed orthographic
  word
• Probe at offset of ambiguous word priming for
  both stood and flower
• 200 ms later only priming for appropriate sense
• Suggests encapsulation followed by rapid
  integration
• But the constraint here is weak -- overestimates
  modularity?
• How could we examine strong constraints in
  natural contexts?

34
(No Transcript)
35
They all rose

• triangle nodes
• when two of the abstract neurons fire, the third
  also fires
• model of spreading activation

36
Allopenna, Magnuson Tanenhaus (1998)
Eye
Eye camera
tracking
computer
Scene camera
Pick up the beaker
37
Do rhymes compete?

• Cohort (Marlsen-Wilson) onset similarity is
  primary because of the incremental nature of
  speech
• (serial/staged Shortlist/Merge)
• Cat activates cap, cast, cattle, camera, etc.
• Rhymes wont compete
• NAM (Neighborhood Activation Model Luce) global
  similarity is primary
• Cat activates bat, rat, cot, cast, etc.
• Rhymes among set of strong competitors
• TRACE (McClelland Elman) global similarity
  constrained by incremental nature of speech
• Cohorts and rhymes compete, but with different
  time course

38
Allopenna et al. Results
39
Study 1 Conclusions

• As predicted by interactive models, cohorts and
  rhymes are activated, with different time courses
• Eye movement paradigm
• More sensitive than conventional paradigms
• More naturalistic
• Simultaneous measures of multiple items
• Transparently linkable to computational model
• Time locked to speech at a fine grain

40
Theoretical conclusions

• Natural contexts provide strong constraints that
  are used
• When those constraints are extremely predictive,
  they are integrated as quickly as we can measure
• Suggests rapid, continuous interaction among
• Linguistic levels
• Nonlinguistic context
• Even for processes assumed to be low-level and
  automatic
• Constrains processing theories, also has
  implications for, e.g., learnability

41
Producing words from pictures or from other
words A comparison of aphasic lexical access
from two different input modalities

Gary Dell
with Myrna Schwartz, Dan Foygel, Nadine Martin,
Eleanor Saffran, Deborah Gagnon, Rick Hanley,
Janice Kay, Susanne Gahl, Rachel Baron, Stefanie
Abel, Walter Huber
42
Boxes and arrows in the linguistic system
Semantics
Syntax
Lexicon
Output Phonology
Input Phonology
43
Picture Naming Task
Semantics
Syntax
Say cat
Lexicon
Output Phonology
Input Phonology
44
A 2-step Interactive Model of Lexical Access in
Production
Semantic Features
FOG
DOG
CAT
RAT
MAT
f
r
d
k
m
ae
o
t
g
Onsets
Vowels
Codas
45
Step 1 Lemma Access
Activate semantic features of CAT
FOG
DOG
CAT
RAT
MAT
f
r
d
k
m
ae
o
t
g
Onsets
Vowels
Codas
46
Step 1 Lemma Access
Activation spreads through network
FOG
DOG
CAT
RAT
MAT
f
r
d
k
m
ae
o
t
g
Onsets
Vowels
Codas
47
Step 1 Lemma Access
Most active word from proper category is selected
and linked to syntactic frame
NP
N
FOG
DOG
CAT
RAT
MAT
f
r
d
k
m
ae
o
t
g
Onsets
Vowels
Codas
48
Step 2 Phonological Access
Jolt of activation is sent to selected word
NP
N
FOG
DOG
CAT
RAT
MAT
f
r
d
k
m
ae
o
t
g
Onsets
Vowels
Codas
49
Step 2 Phonological Access
Activation spreads through network
NP
N
FOG
DOG
CAT
RAT
MAT
f
r
d
k
m
ae
o
t
g
Onsets
Vowels
Codas
50
Step 2 Phonological Access
Most activated phonemes are selected
FOG
DOG
CAT
RAT
MAT
Syl
On Vo Co
f
r
d
k
m
ae
o
t
g
Onsets
Vowels
Codas
51
Semantic Error dog
Shared features activate semantic neighbors
NP
N
FOG
DOG
CAT
RAT
MAT
f
r
d
k
m
ae
o
t
g
Onsets
Vowels
Codas
52
Formal Error mat
Phoneme-word feedback activates formal neighbors
NP
N
FOG
DOG
CAT
RAT
MAT
f
r
d
k
m
ae
o
t
g
Onsets
Vowels
Codas
53
Mixed Error rat
Mixed semantic-formal neighbors gain activation
from both top-down and bottom-up sources
NP
N
FOG
DOG
CAT
RAT
MAT
f
r
d
k
m
ae
o
t
g
Onsets
Vowels
Codas
54
Errors of Phonological Access- dat mat
Selection of incorrect phonemes
FOG
DOG
CAT
RAT
MAT
Syl
On Vo Co
f
r
d
k
m
ae
o
t
g
Onsets
Vowels
Codas
55
A Test of the ModelPicture-naming Errors in
Aphasia
cat
175 pictures of concrete nounsPhiladelphia
Naming Test 94 patients (Broca,Wernicke, anomic,
conduction) 60 normal controls
56
Response Categories
Correct Semantic Formal Mixed Unrelated
Nonword CAT DOG MAT RAT
LOG DAT
Continuity Thesis
Normal Error Pattern 97 Correct
Random Error Pattern 80 Nonwords
cat dog mat rat log dat
cat dog mat rat log dat
57
Implementing the Continuity Thesis
2. Set processing parameters of the model so that
its error pattern matches the normal controls.
Random Pattern Model Random Pattern
cat dog mat rat log dat
Normal Controls Model Normal Pattern
1.Set up the model lexicon so that when noise is
very large, it creates an error pattern similar
to the random pattern.
cat dog mat rat log dat
58
Lesioning the model The semantic-phonological
weight hypothesis
Semantic Features
Semantic-word weight S
FOG
DOG
CAT
RAT
MAT
Phonological- word weight P
f
r
d
k
m
ae
o
t
g
Onsets
Vowels
Codas
59
Patient CAT DOG MAT RAT LOG
DAT Correct Semantic Formal
Mixed Unrelated Nonword
LH .71 .03 .07 .01
.02 .15
s.024 p.018 .69 .06 .06 .01
.02 .17
IG .77 .10 .06 .03
.01 .03
s.019 p.032 .77 .09 .06 .01
.04 .03
GL .29 .04 .22 .03
.10 .32
s.010 p.016 .31 .10 .15 .01
.13 .30
60
Representing Model-Patient Deviations
Root Mean Square Deviation (RMSD) LH .016 IG
.016 GL .043
61
94 new patientsno exclusions
94.5 of variance accounted for
62
Conclusions
The logic underlying box-and-arrow- models is
perfectly compatible with connectionist
models. Connectionist principles augment the
boxes and arrows with -- a mechanism for
quantifying degree of damage -- mechanisms for
error types and hence an explanation of the
error patterns Implications for recovery and
rehabilitation

63
Behavioral and Imaging Experiments Ben Bergen
and Shweta Narayan

• Do Words and Images Match?
• Behavioral Image First
• Does shared effector slow negative
  response?
• Imaging Simple sentence using verb first
• Does verb evoke activity in motor effector
  area?

64
Structured Neural Computation in NTL The theory
we are outlining uses the computational modeling
mechanisms of the Neural Theory of Language
(NTL). NTL makes use of structured
connectionism (Not PDP connectionism!). NTL is
localist, with functional clusters as units.
Localism allows NTL to characterize precise
computations, as needed in actions and in
inferences.
65
Simulation To understand the meaning
of the concept grasp, one must at least be able
to imagine oneself or someone else grasping an
object. Imagination is mental simulation,
carried out by the same functional clusters used
in acting and perceiving. The conceptualization
of grasping via simulation therefore requires the
use of the same functional clusters used in the
action and perception of grasping.
66
Parameters All actions,
perceptions, and simulations make use of
parameters and their values. Such neural
parameterization is pervasive. E.g., the action
of reaching for an object makes use of the
parameter of direction the action of grasping an
object makes use of the parameter of force. The
same parameter values that characterize the
internal structure of actions and simulations of
actions also characterize the internal structure
of action concepts.
67
Advantages of Structured Connectionism Struct
ured connectionism operates on structures of the
sort found in real brains. From the structured
connectionism perspective, the inferential
structure of concepts is a consequence of the
network structure of the brain and its
organization in terms of functional clusters.
68
Multi-Modal Integration Cortical premotor
areas are endowed with sensory properties. They
contain neurons that respond to visual,
somatosensory, and auditory stimuli. Posterior
parietal areas, traditionally considered to
process and associate purely sensory information,
alsos play a major role in motor control.
69
Somatotopy of Action Observation
Foot Action
Hand Action
Mouth Action
Buccino et al. Eur J Neurosci 2001
70
The Simulation Hypothesis
How do mirror neurons work? By simulation.
When the subject observes another individual
doing an action, the subject is simulating the
same action. Since action and simulation use
some of the same neural substrate, that would
explain why the same neurons are firing during
action-observation as during action-execution.
71
Mirror Neurons Achieve Partial
Universality, since they code an action
regardless of agent, patient, modality
(action/observation/hearing), manner,
location. Partial Role Structure, since they
code an agent role and a purpose role. The
Agent Role In acting, the Subject is an agent
of that action. In observing, the Subject
identifies the agent of the action as having the
same role as he has when he is acting namely,
the agent role. The Purpose Role Mirror neurons
fire only for purposeful actions.
72
Conclusion 1 The Sensory-Motor System Is
Sufficient For at least one concept, grasp,
functional clusters, as characterized in the
sensory-motor system and as modeled using
structured connectionist binding and inference
mechanisms, have all the necessary conceptual
properties.
73
Conclusion 2 The Neural Version of Ockhams
Razor Under the traditional theory, action
concepts have to be disembodied, that is, to be
characterized neurally entirely outside the
sensory motor system. If true, that would
duplicate all the apparatus for characterizing
conceptual properties that we have discussed.
Unnecessary duplication of this sort is highly
unlikely in a brain that works by neural
optimization.
74
Behavioral ExperimentsBen Bergen and Shweta
Narayan

• Do Words and Images Match?
• Does shared effector slow negative
  response?
• Behavioral Image First
• Does verb evoke activity in motor effector
  area?

75
(No Transcript)
76
WALK
77
(No Transcript)
78
GRASP
79
(No Transcript)
80
WALK
81
Preliminary Behavior Results
Same Action Other Effector
Same Effector
788 804 871
767 785 825
40 Native Speakers
Eliminate RT gt 2 sec.
82
5 levels of Neural Theory of Language
Spatial Relation
Motor Control
Metaphor
Grammar
Cognition and Language
Computation
Structured Connectionism
abstraction
Neural Net
SHRUTI
Computational Neurobiology
Triangle Nodes
Biology
Neural Development
Midterm
Quiz
Finals