Brain Mechanisms of Reward

1
Brain Mechanisms of Reward
Psychobiology of Addiction Lecture 12 Dai
Stephens Quick overview Carlson 8th edition,
Chapter 18
2
What is reward?

• Websters dictionary Something given or done in
  return especially a gift, prize or recompense
  for merit, service or achievement.
• Instrumental conditioning
• Thorndikes Law of Effect If an animal makes a
  motor movement that results in food, that
  response will be strengthened. Thorndike named
  such objects satisfiers.
• Skinner argued that a term was needed that did
  not have implications of subjective hedonic
  states (such as satisfier or reward) and settled
  on the term reinforcement.
• In recent use, consequences that strengthen
  behaviour are called reinforcers, but these are
  frequently also called rewards.
• Pavlovian (classical) conditioning
• Pavlov (1903) reinforcement is the
  strengthening of the association between an
  unconditioned stimulus and a conditioned stimulus
  when the two are paired.

3
What should a reward system be capable of doing?

• Detect the presence of a reinforcing stimulus
• Strengthen the connections between the neurones
  that detect the stimulus, and the neurones that
  produce the instrumental response
• Enhance the storage of information about
  situations in which they are encountered
  (Thorndike, Hull)
• Give rise to subjective feelings of pleasure.
• Are all of these managed by a single system?

4
Incentive properties of reinforcers (e.g.Tolman)

• Reinforcers are events that elicit behaviours,
  i.e. they are motivators
• An encounter with a reinforcer leads to the
  formation of of a representation of the event,
  and of its relationship to other events and
  objects. As representations of reinforcers can
  also act as motivators of behaviour, when the
  representation of the reinforcer is aroused by
  one of these associations, it results in
  behaviour appropriate to the situation, even in
  the absence of the reinforcers itself-
  conditioned motivation.

5
Is it worthwhile seeking in the brain mechanisms
that map on to the constructs derived from
psychological theory?

• If two behaviours have a common neural
  mechanism, they are two reflections of a single
  phenomenon. That the phenomena of reward and
  reinforcement are the same thing does not derive
  from either Pavlovian or Skinnerian psychology,
  it derives from findings that the same central
  manipulations alter behaviour in both
  reinforcement and reward paradigms. It is the
  fact that the same site of opiate action seems to
  be involved in place preference
  conditioning,....in direct rewarding
  actions,....in facilitating actions on brain
  stimulation reward,....in locomotor enhancing
  actions,.....and in feeding enhancing actions
  that makes it attractive to view all these as
  different labels for the same phenomenon it is
  the apparent biological homology of these
  behaviours, not their theoretical compatibility,
  that suggests a way that they may be meaningfully
  integrated. R.A.Wise (1989).

6
Criteria for Neuronal Pathways being a Component
of a Reinforcement System

•      Activation of the pathway by experimental
  means should result in a behavioural change
  comparable to that produced by the delivery of
  natural reinforcers
•       Behavioural reinforcement produced by
  activation of the pathway and by delivery of
  natural reinforcers should show similar
  sensitivity to paradigmatic manipulations (e.g.
  extinction, schedule effects)
•       Presentation of natural reinforcers should
  give rise to changes in physiological activity
  within the pathway
•       Disruption of the functioning of the
  pathway should result in disruption of
  behavioural control by natural reinforcers

7
Intracranial self-stimulationBrain stimulation
reward

• Olds Milner (1954)
• electrical stimulation of the Medial Forebrain
  bundle results in rats returning to the place
  where they receive the stimulation
• Most early studies used septal or MFB because the
  rates of stimulation from such sites are
  spectacularly high - a few thousand presses per
  hour, the rats stopping only when they became
  exhausted.
• No satiation mechanism. Olds and Milner suggested
  that they were directly activating a brain
  mechanism subserving the signalling of reward.
  Subsequently a similar phenomenon has been
  demonstrated for many species, including humans. 
• Early studies were based on the assumption that
  ICSS is a unitary phenomenon, that the
  fundamental properties are the same, regardless
  of the site of stimulation. (diagram page 449 in
  Pinel).
•  

8
(No Transcript)
9
Does ICSS mimic natural reward?

• Soon after its discovery, two issues were
  identified that questioned ICSS as a model of
  natural reward
•        despite the high rates of stimulation,
  rats stop pressing very quickly if the current is
  turned off (More rapid extinction than with
  natural rewards).
• the high rates of bar pressing suggested the
  stimulation was a particularly strong reinforcer
  (a la Skinner)
• the rapid extinction suggests it is not.
•        experienced self-stimulators often do not
  recommence pressing when returned to the
  apparatus
• the experimenter then gives a few free
  stimulations, and they resume pressing at a high
  rate. This phenomenon is known as priming.

10
Does ICSS mimic natural reward?

• Brain stimulation through ICSS electrodes often
  gives rise to natural motivated behaviours such
  as feeding, drinking, maternal behaviour,
  copulation, in the presence of appropriate goal
  objects
• Increases levels of natural motivations (drives)
  by food or water deprivation, by hormone
  injections, etc, leads to an increase in the
  rates of ICSS
• Although MFB ICSS looks different from responding
  for natural rewards, ICSS at other sites has slow
  acquisition, slow rates of responding, slow
  extinction, and priming is not necessary

11
Properties of ICSS

• Trowill, Panksepp and their colleagues (1967)
  pointed out that responding for Brain Stimulation
  Reward differs from responding for food or water
  in that the reinforcement is immediately
  contingent on the operant in the case of natural
  reinforcers, the animal must perform a chain of
  responses - press the lever, move to food cup,
  pick up pellet. Responding is also maintained by
  secondary reinforcers such as the noise of the
  pellet dispenser.
• Trowill and Panksepp showed that if non-deprived
  rats responding for food reinforcement receive
  the reinforcers directly injected into the mouth
  (a drop of chocolate milk, through an implanted
  tube), they also show rapid extinction On the
  other hand, if rats responding for BSR have to
  lever press, and then go to the food cup to get
  the stimulation, they do not show the rapid
  extinction. Shows importance of conditioned cues
  in maintaining behaviour.

12
Crow (1972) found that sites in the mesencephalon
positive for ICSS are located in the substantia
nigra and VTA and other sites\ map on to dopamine
projections
13
ICSS is related to dopamine

• Corbett and Wise (1980) used movable electrodes
  to map positive and negative sites along the
  tracks of the electrodes. The brains were then
  sectioned and exposed to formaldehyde vapour for
  fluorescence histochemistry to reveal location of
  dopamine neurones. The current thresholds for
  ICSS were lower and response rates higher in
  areas where the DA neurones are most dense.
• Shizgal and colleagues showed that ICSS of other
  mesencephalic sites (e.g. lateral hypothalamus)
  also activates non-DA neurones which project to
  the VTA

14
ICSS is related to dopamine

• ICSS in VTA or lateral hypothalamus leads to an
  increased turnover of DA in terminal areas of the
  dopamine systems, on the side ipsilateral to the
  stimulation that contralateral metabolism is not
  increased is an important control for the effects
  not being due to, say, locomotor activity.
• Drugs like cocaine and amphetamine facilitate
  ICSS.
• 6-hydroxydopamine lesions of DA neuronal systems
  ipsilateral to side of stimulating electrode
  reduces rates of ICSS, whereas contralateral
  lesions do not. (Fibiger, Phillips et al 1987).
• Neuroleptics block ICSS, the pattern of blockade
  has been argued to resemble extinction.

15
Problems in Interpreting Effects of Lesions and
of Neuroleptics

• Since dopamine is involved in functions other
  than reward, it is difficult to rule out effects
  e.g on impairment of motor function, or
  sensory-motor deficits. e.g. rats with lateral
  hypothalamic lesions (or s. nigra 6-OHDA lesions)
  show reduced food intake.
• Originally thought to be due to destruction of
  "hunger centre". Later demonstrated to be due to
  sensory motor deficits on side ipsilateral to
  lesion, so that food presented on that side fails
  to elicit feeding. A stress, like a forced cold
  water swim results in a temporary reinstatement
  of feeding. c.f. parkinsonism
• Use of curve-shift paradigm (Franklin, 1978) to
  study pimozide suggests a clear effect on
  reducing reward, though later studies suggest
  effects on reward and performance..
• Use of double extinction method

16
How do neuroleptics affect ICSS?
17
Double extinction test of pimozides effects on
ICSS (Gallistel, 1982)
18
Effect of dopamine receptor blockade on
lever-pressing for brain stimulation reward.   In
this experiment, pimozide (0.5 mg kg-1, 4 hours
before testing) failed to affect latency to leave
the start box (a), speed to traverse a 1.8-m
runway (b), or the rate of lever-pressing for
fifteen 500-ms trains of lateral hypothalamic
stimulation (c). Within 10 trials, however, the
animals stopped running and lever-pressing. Note
that the animals did not stop running until the
trial after they stopped lever-pressing, and the
animals did not stop leaving the start-box
quickly until the trial after they stopped
running. So, pimozide impaired the ability of the
stimulation to sustain responding rather than the
ability of the animals to initiate responding.
Indeed, after a 10-min time-out, the animals
again initiated responding normally, and ran
normally but responded minimally on the first of
a new set of trials4
19
Effect of dopamine receptor blockade on
lever-pressing for food reward.   a Animals
were trained for 23 weeks to lever-press for
food under conditions of 20-h deprivation. bf
The effects of dopamine blockade (b,c) or
non-reward (d,e) were then assessed on four
occasions, which were separated by two days each
of retraining. On the first day of such testing,
both non-rewarded and pimozide-treated animals
(0.5 or 1.0 mg kg-1 4 hours before the start of
testing) persisted in responding at almost normal
levels. Such responding in the non-reward
condition demonstrated the strength of
conditioned responding such responding in the
pimozide conditions shows that the drug did not
severely attenuate performance capacity. On
subsequent days, responding decreased
progressively in both conditions. Weak responding
on the fourth day of testing in the high-dose
pimozide condition was not a consequence of drug
accumulation, as animals given the first three
pimozide injections without the opportunity to
taste food in this condition (f, home cage (HC)
transfer condition) responded as much after their
fourth injection as did the animals that were
given their first injection in the test box.
20
Diagram illustrating the proposed mechanism for
the intersubject variability in the reinforcing
responses to MP. We postulate that, for
equivalent levels of DAT blockade, the increases
in DA induced by MP will be larger in subjects
with a high DA tone than in those with a low DA
tone this will result in a larger DA occupancy
of DA D2 receptors (DA D2-R) and a more intense
high
Nora D. Volkow, JPET 291 (1999)
21
(No Transcript)
22
Refractory period of neurones underlying ICSS
C T
C T
C T
Current
Time
23
T-pulse response
No T-pulse response
24
Collision Experiment(Bielajew Shizgal, 1986)
Current
VTA
MFB
Electrode T
Electrode C
Antidromic
Orthodromic
25
Summary of Psychophysical data

• Refractory period of DA neurones in VTA (ca. 2.5
  msec) is too long for neurones maintaining ICSS
• Conduction velocity estimates for neurones
  supporting ICSS in MFB is about 3.9 m/sec that
  of DA neurones from ventral tegmental area is ca.
  5m/sec.

26
Alternative accounts of MFB ICSS
VTA
VTA
27
NMDA antagonist infused into VTA blocks MFB ICSS
(Herberg Rose, 1990)
Accumbens
VTA
28
Accumbens circuitry
Hippocampus
opioid
Ventral pallidum
Glutamate
GABA
Prefrontal Cortex
DA
Ventral Tegmental Area
Amygdala
29
mPFC
cingulate
MDThal
Core
VP
BLA
Sensory Information
CeN
Shell
VTA
LC
Brainstem effectors
Raphe
30
VTA circuitry
Glutamate

• VTA receives inputs from many brain areas
  including accumbens, lateral hypothalamus raphe
  nucleus, locus coeruleus, central amygdala,
  prefrontal cortex.

GABA
GABA
DA
opioid
31
What is role of dopamine?
32
Wises Anhedonia Hypothesis

• All forms of reward, whether natural (food, sex)
  or drug (morphine, alcohol, cannabis, cocaine)
  are mediated by the mesolimbic dopamine system.
• Thus, even drugs which do not as their primary
  site of action interact with DA function will do
  so indirectly.
• Still influential, though thought by many
  psychopharmacologists as too simplistic

33
Some relevant evidence

• Heroin self-admin is attenuated by DA depletion
  from mesolimbic system
• Morphine is self-administered directly into the
  region of the cell bodies of VTA dopamine
  neurones, probably by increasing accumbens
  release of dopamine

34
Testing Wises Hypothesis

• Do all abused drugs increase accumbens dopamine
  release?
• Psychostimulants do, as do opiates
• Nicotine does, probably through an action at
  nicotinic cholinergic receptors on DA cell bodies
  in VTA
• Benzodiazepines do NOT, though DA depletion in
  accumbens blocks benzodiazepine-induced place
  preference

35
In support of the dopamine hypothesis

• Rats will self-administer amphetamine directly
  into accumbens (Wise and Bozarth, 1987) If
  dopamine systems disrupted using 6OH-DA, then
  animals no longer self-administer amphetamine.
• Morphine (and alcohol) is self-administered
  directly into VTA (Bozarth Wise, 1984).
• During withdrawal from abused drugs, DA levels in
  accumbens are reduced, and ICSS thresholds
  increased
• Volkow et al (1997) image blood flow changes in
  striatum that correlate with subjective responses
  such as euphoria and craving in cocaine addicts
• Do differences in D2 receptors (allelic forms)
  differ between individuals who are sensitive to
  drugs of abuse (Uhl et al, Neuropsychopharmacol
  1999).
• Mice lacking D2 receptor consume less alcohol
  than normal mice (Tamara Phillips, Nature
  Neurosci 1 610 (1999)
• BUT Mice lacking the DAT continue to
  self-administer cocaine (Rocha Nature 393 175
  (1998)

36
Some contradictory evidence

• Ettenberg et al have reported that DA receptor
  blockade does not lead to compensatory increases
  in heroin self-administration
• Koob et al report that heroin self-admin is
  relatively unaffected by DA depletion from
  accumbens, but is affected by methylnalaxonium
  administration into accumbens, and by damage to
  the ventral pallidal outflow from the accumbens
• Thus although ventral striatal pathways may be
  important in both stimulant and opiate reward,
  dopamine may have only a minor role in the latter

37
Testing Wises hypothesis

• Dopamine depletion from ventral striatum does not
  impair consummatory behaviour (food, sex) in the
  presence of a primary reinforcer or goal
  food, a sexual partner.
• Incentive properties of the same objects is
  reduced investigative and appetitive behaviour
  decreased by DA blockade.
• Thus, lever pressing to obtain oestrus female
  reduced by DA lesions in accumbens, but
  copulation isnt.
• Locomotor excitement in anticipation of food
  decreased, but consumption isnt.
• Dopamine released in accumbens by stimuli
  predicting food, sex, etc.

38
Testing Wises hypothesis

• Electrophysiological studies by Schultz show that
  DA neurones initially increase their rates of
  firing when primary rewards are experienced, but
  then only in response to stimuli predictive of
  primary rewards (conditioned incentives), or to
  unexpected rewards.
• Propose that dopamine provides a teaching
  signal to striatal and cortical neurones involved
  in reward-related learning

39
Conditioned Reinforcement

• The importance of dopamine systems in conditioned
  reward is demonstrated by conditioned
  reinforcement (Robbins et al, 1989)
• Pair food with light in a pavlovian conditioning
  paradigm
• Give rats opportunity to learn a new response to
  obtain the conditioned stimulus light

40
Conditioned Reinforcement

• Infusion of amphetamine into ventral, but nor
  dorsal striatum, increases responding for
  secondary reinforcer
• Responding is not increased for a randomly-paired
  stimulus (I.e. not predictive of primary reward)
• Amphetamine-induced facilitation is blocked by DA
  depletion in ventral, but not dorsal striatum

41
Amygdala and reward          Neurotoxic lesions
of the basolateral amygdala attenuate lever
pressing for a light conditioned to the
appearance of an oestrous female, though the
lesion did not affect sexual behaviour (Everitt
et al, 1989). Effect reversed by amphetamine
injections into the accumbens        Rats
trained to associate a flashing light with water
reinforcement, and then tested in two-lever
operant chamber, in which activation of one lever
gave rise to light. Amygdala lesions attenuate
preference for lever giving secondary reinforcer
(Cador et al, 1989). Reinstated by amphetamine in
accumbens.        Lesions of the lateral nucleus
of the amygdala attenuate a conditioned place
preference produced by administration of
d-amphetamine (Hiroi and White, 1991). Also
blocked by neuroleptic flupenthixol into
accumbens.        White and Milner (1992)
suggest that ventral striatum is an interface
between the limbic system and motor systems.
42
Neural Circuitry of Conditioned Reinforcement

• Lesions of the basolateral amygdala also block
  the acquisition of conditioned reinforcement .
• Can be overcome by amphetamine injected into
  accumbens.
• Resemblance between conditioned reinforcement and
  reinstatement models of relapse.

43
Relevance of Conditioned Reinforcement to Craving
and Relapse

• Craving is initiated by stimuli conditioned to
  drug
• Such stimuli elicit drug-seeking behaviour
• In animal models of relapse, rats are trained to
  lever-press for drug, whose delivery is paired
  with a light CS
• Subsequently, responding for drug is
  extinguished.
• When response levels are low, the light is
  reinstated, so that lever presses give rise to
  light presentation (but not drug).
• Lever-pressing (drug seeking) is reinstated,
  which is interpreted as relapse

44
Relevance of Conditioned Reinforcement to Craving
and Relapse

• Note similarities in this model between relapse
  and conditioned reinforcement
• Note also that psychostimulants facilitate
  responding for conditioned reinforcers
• Thus psychostimulants like cocaine not only have
  rewarding properties in their own right, but will
  potentiate the likelihood of conditioned
  reinforcers inducing relapse a vicious circle.

45
Dopamine receptors in cocaine addictsVolkow et
al, Nature 386 1997)
Positron Emission Tomography (PET) to visualise
11C-raclopride, a ligand for D2 receptors Cocaine
abusers (4g/week) vs nondrug users. Controls show
higher raclopride binding in striatumTreat with
methylphenidate to increase DA in cleft MP
reduces R binding in all brain areas indicating
DA increase induced by MP competes with R for D2
receptors MP-induced changes in striatum were
greater in controls than in cocaine users
suggesting impaired DA release in users, that may
make cocaine less effective in addicts Opposite
for thalamus. Changes in thalamus associated with
craving
46
Methylphenidate significantly increased whole
brain glucose metabolism compared with the
placebo condition, but that increase was
augmented by approximately 50 when
methylphenidate was expected relative to when it
was not. The self-reports of "drug high" were
greater for methylphenidate when it was expected
than when unexpected, by approximately 50.
Expectation enhances the pharmacologic effects
of methylphenidate, through amplifying dopamine
and norepinephrine signals generated by blocking
dopamine and norepinephrine transporters. The
notion that expectation can emulate drug-induced
brain effects may explain a "placebo" effect.
Thus, the reinforcing effects of drugs are not
just a function of their pharmacological effects
but also of the expectation of their actions.
In drug abusers, the subjective responses to a
drug are more pleasurable when the person expects
to receive the drug than when they do not.
Examined the response of brain glucose
utilization to intravenous methylphenidate in
cocaine abusers under distinct conditions of
expectation and no expectation.
Volkow, Am J Psychiatry 161621, April 2004
47

• Wolfram Schulz (2000) Nature Reviews Neurosci 1
  199-207
• Primate dopamine neurones fire when animal
  touches apiece of food in the absence of stimuli
  that predict food
• Touching the food has this effect, but touching
  the wire on which food is impaled does not
• Firing rate is depressed when the food reward is
  delayed beyond the expected time point, but
  enhanced when the food is subsequently presented

48
Addiction as aberrant learning (e.g. Kelley
Psychobiology 27 198 (1999) Robbins and
Everitt, Nature 398 567 (1999)

•       Activity in dopaminergic neurones serves
  as a predictor of reward or stimulus salience.
•       Dopaminergic pathways alter firing
  patterns during learning. Initially they are
  activated by primary rewards, but following many
  presentations they are no longer activated by the
  primary reward but by CS associated with that
  reward.
•       Accumbens receives (glutamatergic) inputs
  from hippocampus, prefrontal cortex and amygdala,
  which converge on to dendritic spines of
  GABAergic median spiny neurones which are also
  innervated by dopamine.
•       The strengthening of connections between
  the glutamatergic (sensory/cognitive) inputs and
  GABAergic (motor) outputs will result in the
  strengthening of stimulus-response habits.
•      Dopamine facilitates the strengthening of
  such connections, through activity at D1
  receptors.
• Thus, stimulus-response connections which
  occur in the presence of highly salient
  (DA-releasing) stimuli will be strengthened to a
  greater extent than those occurring in the
  absence of DA release.

49
(No Transcript)
50
(No Transcript)
51
(No Transcript)
52
(No Transcript)
53
(No Transcript)
54
(No Transcript)
55
(No Transcript)
56
(No Transcript)
57
Lower Striatal Dopamine D2 Receptor Binding
inDrug Users During Withdrawal From Cocaine,
Methamphetamine,and Alcohol Than in Normal
ComparisonSubjects