Characteristics of Neuronal Prediction Error Signals

1
Characteristics of Neuronal Prediction Error
Signals
2
A Neural Substrate of Prediction and Reward

• An adaptive organism must be able to predict
  future events such as the presence of mates,
  food, and danger.
• Predictions give an animal time to prepare
  behavioral reactions and can be used to improve
  the choices an animal makes in the future.

3
A Neural Substrate of Prediction and Reward

• This anticipatory capacity is crucial for
  deciding between alternative courses of action
  because some choices may lead to food whereas
  others may result in injury or loss of resources.

4
A Neural Substrate of Prediction and Reward

• Experiments show that animals can predict many
  different aspects of their environments,
  including complex properties such as the spatial
  locations and physical characteristics of stimuli
  . One simple, yet useful prediction that animals
  make is the probable time and magnitude of future
  rewarding events.

5
A Neural Substrate of Prediction and Reward

• Reward is an operational concept for describing
  the positive value that a creature ascribes to an
  object, a behavioral act, or an internal physical
  state. The function of reward can be described
  according to the behavior elicited.

6
A Neural Substrate of Prediction and Reward

• The reward value associated with a stimulus is
  not a static, intrinsic property of the stimulus.
  Animals can assign different appetitive values to
  a stimulus as a function of their internal states
  at the time the stimulus is encountered and as a
  function of their experience with the stimulus.

7
A Neural Substrate of Prediction and Reward

• One clear connection between reward and
  prediction derives from a wide variety of
  conditioning experiments. In these experiments,
  arbitrary stimuli will function as rewarding
  stimuli after being repeatedly associated in time
  with rewarding objects.

8
A Neural Substrate of Prediction and Reward

• Some theories of reward-dependent learning
  suggest that learning is driven by the
  unpredictability of the reward. One of the main
  ideas is that no further learning takes place
  when the reward is entirely predicted.

9
A Neural Substrate of Prediction and Reward

• For example, if presentation of a light is
  consistently followed by food, a rat will learn
  that the light predicts the future arrival of
  food. If, after such training, the light is
  paired with a sound and this pair is consistently
  followed by food, then something unusual
  happensthe rats behavior indicates that the
  light continues to predict food, but the sound
  predicts nothing.

10
A Neural Substrate of Prediction and Reward

• This phenomenon is called blocking. The
  prediction-based explanation is that the light
  fully predicts the food that arrives and the
  presence of the sound adds no new predictive
  (useful) information therefore, no association
  developed to the sound.

11
A Neural Substrate of Prediction and Reward

• It appears therefore that learning is driven by
  deviations or errors between the predicted time
  and amount of rewards and their actual
  experienced times and magnitudes

12
A Neural Substrate of Prediction and Reward

• The question is
• Are there neuronal systems whose
  electrophysiological pro-file encodes prediction
  errors by reflecting the unpredictability of
  outcomes? In other words, are there systems that
  respond differentially to predicted and
  unpredicted outcomes and to the unexpected
  omission of a predicted outcome?
• The following sections evaluate various candidate
  neuronalsystems.

13
Dopamine Neurons

• Dopamine Neurons
• Dopamine neurons show homogeneous, short
  latency responses to two classes of events,
  certain attention-inducing stimuli and
  reward-related stimuli.

14
Dopamine Neurons

• Attention-inducing stimuli, such as novel, elicit
  an activation-depression sequence.
• Reward-related stimuli, such as primary liquid
  and food rewards, and visual and auditory stimuli
  predicting such rewards elicit pure activations.
  Events that physically resemble reward-predicting
  stimuli induce smaller, generalizing activations
  followed by depressions.

15
Dopamine Neurons

• The dopamine neurons code an error in the
  prediction of reward .
• Primary rewards are unpredictable during initial
  behavioral reactions and reliably elicit neuronal
  activations.
• With continuing experience, reward becomes
  predicted by conditioned stimuli, and the
  activations elicited by reward decrease.

16
Dopamine Neurons

• If, however, a predicted reward fails to occur
  because the animal makes an incorrect response,
  dopamine neurons are depressed at the time the
  reward would have occurred.

17
Dopamine Neurons
18
Dopamine Neurons

• The depression in the activity of the dopamine
  neuron at the expected time of the omitted reward
  shows that this activity encodes not only the
  simple expected occurrence of the reward but also
  the specific predicted time of the reward.

19
Dopamine Neurons

• A reward occurring earlier than predicted induces
  an activation, but no depression occurs at the
  original time of reward, as if the precocious
  reward has cancelled the reward prediction.

20
Norepinephrine Neurons

• Most norepinephrine neurons in locus coeruleus in
  rats, cats, and monkeys show homogeneous,
  biphasic activating-depressant responses to
  visual, auditory, and somatosensory stimuli
  eliciting orienting reactions .

21
Norepinephrine Neurons

• In relation to prediction errors, it appears that
  norepinephrine neurons respond to unpredicted but
  not predicted rewards, probably as part of their
  responses to attention-inducing stimuli.

22
Nucleus Basalis Meynert

• Primate basal forebrain neurons are phasically
  activated by a variety of behavioral events,
  including conditioned, reward-predicting stimuli
  and primary rewards.

23
Nucleus Basalis Meynert

• Activations
• reflect the familiarity of stimuli
• become more important with stimuli and movements
  occurring closer to the time of reward
• differentiate well between visual stimuli on the
  basis of appetitive and aversive associations
• change within a few trials during reversal

24
Nucleus Basalis Meynert

• In relation to prediction errors, it appears that
  some nucleus basalis neurons respond particularly
  well to unpredicted rewards .

25
Cerebellar Climbing Fibers

• Movement Climbing fiber inputs to Purkinje
  neurons are particularly activated when loads for
  wrist movements or gains between arm movements
  and visual feedback are changed.
• In a model of predictive tracking of moving
  targets by the eyes, climbing fibers carry
  prediction errors between eye and target
  positions . These data suggest that climbing
  fiber activity is compatible in several instances
  with a role for a prediction error in motor
  performance.

26
Cerebellar Climbing Fibers

• Aversive Conditioning A second argument for a
  role of climbing fibers in learning is derived
  from the study of aversive classical
  conditioning. A fraction of climbing fibers is
  activated by aversive air puffs to the cornea.
  These responses are lost when the air puff
  becomes predicted .

27
Cerebellar Climbing Fibers

• Relation to Prediction Error The increased
  climbing fiber activity with motor performance
  errors may be related to error magnitude but
  possibly not error direction. Climbing fibers
  responding to aversive events may code a
  punishment prediction error, as they
• activated by unpredicted aversive events
• do not respond to fully predicted aversive events
• possibly depressed by omitted aversive events.

28
Superior Colliculus

• Neurons in the intermediate layer of superior
  colliculus are activated in association with a
  predicted visual stimulus brought into their
  receptive fields by a future saccadic eye
  movement.
• These neurons appear to code the difference
  between the current and future eye position and
  not specific retinal target positions.
• In relation of prediction errors, it appears that
  intermediate and deep-layer neurons code errors
  in current eye positions relative to future eye
  positions and targets.