Internal models in sensorimotor control

1
Internal models in sensorimotor control
Philip N. Sabes Neuroscience and Bioengineering
Graduate Programs Keck Center for Integrative
Neuroscience UCSF sabes_at_phy.ucsf.edu
2
Internal models in sensorimotor control
3
Basic anatomy of motor systems
4
Basic model of the plant three-joint arm
5
Definition of Kinematics
6
Definition of Dynamics
In general the Dynamics of a system is composed
of two equations
next state equation
output equation (kinematics, sensory feedback)
7
Internal models

• We consider four putative internal models

Why have internal forward models? Do we have them?
8
Inverse ModelsFeedforward Controllers

• Possible Input
• desired extrinsic location of .
• desired perceptual state
• desired intrinsic state (x)

• Possible Output
• motor unit activity
• signals to Brainstem (eye)
• signals to spinal cord (arm)

9
Example 1 the three-joint arm
Kinematics
Jacobian
10
Redundancy of the three-joint arm
11
Forward ModelsInternal Prediction of Plant,
World,

• Possible Output
• predicted extrinsic location of .
• predicted perceptual state
• predicted intrinsic state (x)

• Possible Input
• copy of motor unit activity
• copy of signals to Brainstem (eye)
• copy of signals to spinal cord (arm)

12
Dynamics of the arm
Analytic form of the dynamics for the two-joint
arm
Scheidt and Rymer, 2000
The two-joint arm is highly non-linear
13
Muscles are the controllers
Grays Anatomy, Gramercy Press (1977)
14
and their effect is not simple
Buneo, Soechting, and Flanders, 1997
15
and even motor units are complex
Figure 12. a, 3D best direction of each unit
versus their locations in deltoid, for each
subject. The best direction is given by the
direction of the unit vector from the origin. The
color of the vector codes for location. As
location changes from posterior to anterior (and
color changes from the blue to the red end of the
spectrum), best direction changes from
downoutbackward to upoutforward. b, Pooled 3D
best directions of all units versus their
normalized locations in deltoid, for all
subjects. Best directions are coded for location
as in a.
Herrmann and Flanders, 1998
16
Example 2 At least the eye is simple!

• The intrinsic coordinates of the eye match the
  extrinsic, right?

Kandel, Schwartz and Jessell, Principles of
Neuroscience, 4th ed.
17
The eyes non-linear kinematics
Kandel, Schwartz and Jessell, 4th ed.
18
Internal models

• Is there evidence for the existence of internal
  models in the brain?
• Is the notion of an internal model even useful?
  After all, dont inverse models have to exist?

• Perhaps not .

19
Equilibrium Point Hypothesis (EPH)
Bizzi et al (1976), A.G. Feldman (1966), etc
Solves inverse dynamics in terms of a sort of
inverse kinematics much easier!
20
Bizzi stimulates the frog ( rat) spine
Fairly easy solution to the inverse kinematics
21
Equilibrium Point Hypothesis is wrong
Deafferented monkey with no visual feedback
What happens if you bring arm to target before GO
signal?
Bizzi et al (1984)
22
Modified model equilibrium trajectory
Flash (1987)
23
Better stiffness measurements
Gomi and Kawato (1996)
24
Joint Interaction Torques
Hollerbach and Flash (1982)
25
Joint Interaction Torques
Hollerbach and Flash (1982)
26
Joint Interaction Torques
No velocity terms
Original Trajectory
No interaction terms
Hollerbach and Flash (1982)
27
Do we account for Interaction Torques in
feedforward manner (inverse model)?
Gribble and Ostry (1999)
28
Interaction Torque compensation before movement
initiation
Gribble and Ostry (1999)
29
FEEDFORWARD CONTROL

• So a real inverse model exists, mapping desired
  movement (state trajectory) into motor commands
• What is the desired movement?

30
Motor Equivalance
Right hand (dom) R hand, wrist fixed L hand Pen
between teeth Pen attached to foot
Rosenbaum, Human Motor Control (1991)
31
Path and Velocity Invariance
Gordon, Ghilardi, and Ghez (1994)
32
across the workspace
Raw velocity profiles Normalized time, aligned
to peak of mean profiles Align before averaging
Atkeson and Hollerbach (1985)
33
and lots of other things
v
Speeds Movements
Loads Subjects
Atkeson and Hollerbach (1985)
34
paths always sort of straight
Atkeson and Hollerbach (1985)
35
Invariance and Optimality

• Argument invariances reflect underlying planning
  criteria
• Optimal Control models make the criteria explicit
• Given
• a set of possible trajectories,
• A cost function(al)
• Then

36
Early optimal control planning models
Limited usefulness for multi-joint movement
Nelson (1983)
37
Minimum Jerk
38
Minimum Jerk predictions
Flash and Hogan (1985)
39
Why would planner (brain) care about how the
movement looks surely its what you do that
matters!
Wolpert, Ghahramani, and Jordan (1995)
40
Reach planning cares about the extrinsic path.
Cost function must depend in part on extrinsic
characteristics of the movement. Similar finding
if you put subject in an altered dynamics
environment (lots of literature)
Wolpert, Ghahramani, and Jordan (1995)
41
A more extreme design
Flanagan and Rao (1995)
42
Results
Flanagan and Rao (1995)
43
Statistical Optimality

• Up to now, optimality is in terms of smoothness.
  Why???
• Observation in neural signals and in motor
  output (e.g. force production) noise scales with
  the mean (signal dependent noise, SDP)
• In this case, large commands are noisier!
• Suggests an optimality principle minimum task
  variance in the face of SDP

44
Minimum Variance

• minimize task variance in the face of SDP

Hamilton and Wolpert (2002) Sabes and Jordan
(1996)
Harris and Wolpert (1998)
45
Minimum Variance

• minimize task variance in the face of SDP

Note the poor fits at the end of the movements
Hamilton and Wolpert (2002) Sabes and Jordan
(1996)
Harris and Wolpert (1998)
46
FEEDBACK CONTROL

• Feedforward optimal control models give nice fits
  to mean trajectory, but cant handle statistics
  of movements properly
• Not surprising since we know that movements are
  not feedforward

47
Evidence for feedback control
Single step paradigm
Prablanc, Pelisson, and Goodale (1986)
48
The double step experiment
Prablanc and Martin (1992)
49
Single Step (no target jump) vs Double Step
Two lines are open loop and closed loop
Prablanc and Martin (1992)
50
Double Step
Acceleration angle used to determine reaction time
RT 155msec open or closed loop!
Prablanc and Martin (1992)
51
Feedback correction in the PPC?
TMS (transcranial magnetic stimulation) applied
to PPC at movement onset
Desmurget et al (1992)
52
Optimal Feedback Control

• What would optimal feedback control look like?
• Example linear quadratic regulator (LQR)
• Todorov (2005) has devised some nice tricks to
  solve the optimal feedback control model with
• non-linear plant and sensors
• non additive noise
• Create local linear LQRs
• Allows one to model a feedback version of minimum
  variance

53
Deeper rule from optimal feedback control?
Assuming signal dependent noise, one arrives at
the Minimum intervention principle
Todorov and Jordan (2002)
54
Deeper rule from optimal feedback control?
Minimum intervention principle
Todorov and Jordan (2002)
55
Deeper rule from optimal feedback control?
Minimum intervention principle
Note the details of this principle rely
heavily on the form of the noise model!
Todorov and Jordan (2002)
56
State estimation and sensory signals

• Feedback control requires feedback (and state
  estimation)!
• Remember the Double Step experiment reaction
  time results were the same for open and
  closed loop.
• If its open loop, how can you have feedback
  control???
• Two possibilities
• Proprioceptive feedback
• Efference copy (forward model)

57
The role of proprioceptive feedback
The effect of Large Sensory Fiber Neuropathy
Gordon, Ghilardi and Ghez (1995)
58
Effects of inertia on initial movement
Normals compensate for inertia with movement time
Gordon, Ghilardi and Ghez (1995)
59
Why are deafferented patients impaired?
60
How are vision and proprioception combined in
normal movement?
Rossetti et al (1995)
61
How are vision and proprioception combined in
normal movement?
proprioception-only prediction
data
vision-only prediction
Rossetti et al (1995)
62
Sensory combination evident at movement onset


Tempting to compute a weighting of each sensory
signal but cant quantify the integration of
vision and proprioception without knowing how the
state estimate contributes to movement
real position proprioception
visual and proprioception combined
visual position
Rossetti et al (1995)
63
How should you combine multiple sensory inputs?
Minimum Variance model predicts cue combination
in perceptual tasks Vision and haptics (e.g.
Banks et al, Nature and Science, 2002) Vision
and vision (e.g. Jacobs et al., Vision
Research, 1999)
We can test this idea in the context of reaching,
where simple models are available.
64
Two sources of errors
Movement Vector Error (MV)
Inverse Error (INV)
Sober and Sabes (2003)
65
Two sources of errors
Sober and Sabes (2003)
66
Model reach initiation
67
Model predictions
By comparing these predictions to experimental
data we infer aMV, aINV.
68
Results Sample Subject
Subject HA
69
Results Weighting parameters
70
Why two estimators?
While Minimum Variance says two estimates should
be the same, this doesnt account for the
cost of computing coordinate transformations
How to test this idea change the coordinate
frame of the the task demands (estimator output)
or sensory feedback (estimator input)
71
Manipulating target coordinate frame
72
Reaching to visual and proprioceptive targets
Subject CO
73
Weighting parameters
Visual Targets Proprioceptive Targets
74
Vision, Proprioception, and State Estimation

• Remember the case of patients with Large Sensory
  Fiber Neuropathy
• Trouble reaching because they dont know where
  their hands are
• What about when the have visual feedback?

75
Vision helps, and the benefit persists
Ghez, Gordon, and Ghilardi (1995)
76
Vision helps, and the benefit persists
Ghez, Gordon, and Ghilardi (1995)
77
Why does prior vision help?
78
Decay of state estimate?
Ghez, Gordon, and Ghilardi (1995)
79
Kalman Filter
80
Sensory integration as Kalman Filter
Wolpert et al, 1995
81
Sensory integration as Kalman Filter
To get this to work, had to assume that force
output is underestimated (??)
Wolpert et al, 1995
82
Kalman Filter
The many roles of the Kalman Gain
83
Kalman Filter
Evidence for forward (predictive) models and the
use of efference copy
84
A nice example of predictive behavior modulation
of grip force with load force
Flanagan and Wing, 1993 Flangan, Tresilian and
Wing, 1993
85
A nice example of predictive behavior modulation
of grip force with load force
Flanagan and Wing, 1993 Flangan, Tresilian and
Wing, 1993
86
Does grip force modulation require predictive
(forward) models?Cant inverse models do the
trick?

• Have subjects carry a load while learning to
  reach in a novel environment
• An inverse model is needed to make an accurate
  movement
• Prediction (might be / is) needed for accurate
  grip force modulation

Flanagan et al 2003
87
Predictive (forward) models vs inverse models for
grip force modulation
Flanagan et al 2003
88
Predictive (forward) models vs inverse models for
grip force modulation
Flanagan et al 2003
89
Efference copy and perception?
Are the predictable sensory consequences of
movement discounted?
Bays, Wolpert, Flanagan, 2005 Bays, Flanagan,
Wolpert, 2005
90
Efference copy and perception?
T(test) T(active)
Relative felt amplitudes of the active tap vs
test tap
Bays, Wolpert, Flanagan, 2005 Bays, Flanagan,
Wolpert, 2005
91
Efference copy and perception?
Relative felt amplitudes of the active tap vs
test tap
Group A Subjects typically contact button
Group B Subjects never contact button
Bays, Wolpert, Flanagan, 2005 Bays, Flanagan,
Wolpert, 2005
So the effect is predictive!
92
Efference copy can be largely central

• Tactile stimulus on the arm during movement
  compare magnitude to stimulus at rest
• PSE point of subject equality
• Repeat when movement is delayed due to TMS
  stimulation over motor cortex (presumably
  blocking descending signals)

Voss et al 2006
93
Cortical Control of Saccades
Ill end with an example of efference copy at
high level from physiology
94
Lateral Intraparietal Area (LIP)
Delayed memory guided saccade task Mays and
Sparks (1980)
Pare and Wurtz (1997)
95
Stimulation in LIP
Avg latency 32 ms Threshold 25-150 mA
Their and Andersen (1996)
96
Reversible Lesions in LIP
Normal
Muscimol (R LIP)
Memory
Visual
Li et al (1998)
97
Sensory Remapping
Activity remaps 80 msec before the beginning of
the saccade!