ACQ and the Basal Ganglia

1
ACQ and the Basal Ganglia

• Jimmy Bonaiuto
• USC Brain Project
• 2/12/2007

2
Outline

• Alstermarks Cat
• ACQ
• ACQ ? Basal Ganglia
• Basal Ganglia Model Implementations (NSL)
• The Search for Executability

3
Alstermarks Cat
4
ACQ
5
ACQ
6
ACQ - Executability

• 2D Gaussian kernel populations
• Food location relative to mouth
• Food location relative to paw
• Food location relative to tube opening

7
Learning Executability
- Success or failure is signaled by the match or
mismatch between efferent signals and mirror
system output
8
Learning Desirability
- Eligibility signal computed from - Internal
state - Mirror system output - Efferent
signal
9
Priority
Simplified form priority executability
desirability
Leaky integrator form
10
Action Selection
- Winner declared when max CC layer element
firing rate is greater or equal to e1 (0.9) and
all other element firing rates are less than or
equal to e2 (0.1).
11
ACQ
12
ACQ Selection Properties
Contrast-Dependent Latency
13
ACQ Selection Properties

• Approximation to
• Boltzmann equation

Ttemperature
14
ACQ TD Learning
Effects of Desirability Weight Initialization on
Mean Trial Length During TD Learning
Eat initialized
No initialized weights
Reach-grasp initialized
15
ACQ Simulation Results
Mean Trial Length
Final Desirability Weights
Mean Unsuccessful Action Attempts
16
ACQ Simulation Results
MF - Eat
MF Grasp Jaw
PF Reach Food
PF Reach Tube
17
Where in the Brain is ACQ?

• Affordances
• Posterior parietal cortex
• Object-directed motor schemas
• Premotor cortex
• Winner-Take-All
• Basal ganglia (Winner-Lose-All)
• Desirability Learning
• Striatum with TD error signal from midbrain
  dopaminergic system (SNc, VTA)
• What about Executability?

18
Basal Ganglia Model Implementations (NSL)

• The following models are implemented in NSL and
  available for extension or experimentation
• GPR
• Brown, Bullock, Grossberg
• RDDR

19
Gurney, Prescott, Redgrave (GPR)

• Interlayer winner-lose-all
• Control signal calculated from the sum of the
  cortical signal provides a gain signal to the
  competition

20
GPR
Str-D1
Cortex
Str-D2
STN
GPi/SNr
GPe
21
GPR

• What does a consideration of the GPR model bring
  to ACQ?
• Intralayer WTA ? Interlayer WTA
• WTA ? WLA
• Do we need a control (gain) signal?
• We may want to explore the possibility of
  chunking when two actions are activated to
  similar levels

22
Brown, Bullock, Grossberg

• Ventral striatum ? ventral pallidum ? PPTN
• Learns to activate SNc given secondary
  reinforcer
• Cortex ? Striosomes
• Learns to inhibit SNc response to primary
  reinforcer
• Learns timing between primary and secondary
  reinforcers

23
Brown, Bullock, Grossberg
24
Brown, Bullock, Grossberg
25
Brown, Bullock, Grossberg
26
Brown, Bullock, Grossberg

• What does a consideration of the Brown, Bullock,
  Grossberg model bring to ACQ?
• A neural method of computing the TD error signal
• Can we extend it to have multiple primary
  reinforcers (dimensions of reinforcement)?

27
Reinforcement Driven Dimensionality Reduction
(RDDR)

• Extension of PCA neural network methods to
  include reinforcement

• Feedforward connections normalized
  multi-Hebbian with reinforcement

• Lateral connections normalized anti-Hebbian

28
RDDR - Pretraining
29
RDDR Mid-training
30
RDDR - Trained
31
RDDR - Retraining
32
RDDR - Retrained
33
RDDR

• What does a consideration of the Brown, Bullock,
  Grossberg model bring to ACQ?
• Maybe nothing, but it may be useful in chunking
  actions in hACQ

34
Where is Executability?

• We can map ACQ onto the basic BG architecture by
  modeling an interlayer WLA network with
  cortico-striatal connection weights encoding
  desirability and modified via TD learning
• How does executability fit in?

35
Parietal / Basal Ganglia Projections

• Petras (1971) Projections from the inferior and
  superior parietal lobules to the striatum and
  thalamus
• Cavada Goldman (1991) Subregions of parietal
  area 7 project to portions of the striatum
  bilaterally
• Flaherty Graybiel (1991) Somatotopic
  projections from S1 to the striatum
• Only innervates matrix not striosomes
• Graziano Gross (1993) Bimodal somatotopic map
  in putamen
• Lawrence et al. (2000) Dorsal stream projects to
  the anterodorsal striatum

36
ACQ Basal Ganglia

• Could executability and desirability be
  represented in segregated regions of the striatum
  and be combined in the globus pallidus?
• Or perhaps they are combined in the striatum?

37
References

• Bar-Gad, I., Morris, G., Bergman, H. (2003)
  Information processing, dimensionality reduction
  and reinforcement learning in the basal ganglia.
  Progress in Neurobiology, 71 439473.
• Brown, J., Bullock, D., Grossberg, S. (1999) How
  the Basal Ganglia Use Parallel Excitatory and
  Inhibitory Learning Pathways to Selectively
  Respond to Unexpected Rewarding Cues. J.
  Neurosci., 19(23) 10502-10511.
• Cavada, C., Goldman-Rakic, P.S. (1991)
  Topographic Segregation of Corticostriatal
  Projections from Posterior Parietal Subdivisions
  in the Macaque Monkey. Neuroscience, 42(3)
  683-696.
• Flaherty, A.W., Graybiel, A.M. (1991)
  Corticostriatal Transformations in the Primate
  Somatosensory System. Projections from
  Physiologically Mapped Body-Part Representations.
  J. Neurophys. 66(4) 1249-1263.
• Graziano, M.S.A., Gross, C.G. (1993) A bimodal
  map of space Somatosensory receptive fields in
  the macaque putamen with corresponding visual
  receptive fields. Exp Brain Res, 97 96-109.
• Gurney, K., Prescott, T.J., Redgrave, P. (2001) A
  computational model of action selection in the
  basal ganglia. I. A new functional anatomy. Biol.
  Cybern. 84 401-410.
• Lawrence, A.D., Watkins, L.H.A., Sahakian, B.J.,
  Hodges, J.R., Robbins, T.W. (2000) Visual object
  and visuospatial cognition in Huntingtons
  disease implications for information processing
  in corticostriatal circuits. Brain, 123
  1349-1364.
• Petras, J.M. (1971) Connections of the Parietal
  Lobe. J. Psychiat. Res., 8 189-201.