Novelty bias on decision making

1
Novelty bias on decision making

• Summer School St. Johann, September 2009
• Workshop 1
• Irrationale Entscheidungs- und
  Gedächtnisprozesse im Gehirn
• Christian Clemm
• 09/09/09 (!)

2
Overview

• - Exploration vs. Exploitation and Novelty
• - Practical example
• - Introduction Exploration vs. Exploitation
• - One Experiment
• - (If we have time!!!) Some more results from
  other experiments
• - Incentive salience What's the difference
  between liking and wanting?

3
(No Transcript)
4
Exploration vs. Exploitation

• conflict between opposing needs of gathering and
  exploiting information Daw, 2006

5

• A fully rational trade-off...
• ...would have to take into account many future
  decisions which exploration will have a benefit
  on.

6

• A fully rational trade-off...
• ...would have to take into account many future
  decisions which exploration will have a benefit
  on.
• ...which is too difficult even for computers (to
  this day)

7

• A fully rational trade-off...
• ...would have to take into account many future
  decisions which exploration will have a benefit
  on.
• ...which is too difficult even for computers (to
  this day)
• ...and therefore probably also for our brain.

8

• The way out...
• ... a short cut?

9

• Remember Emrah's talk yesterday
• Novelty to our brain is like a reward!
• Much less complicated than calculating benefits
  on future choices...

10

• Is that clear?

11
Experiment Wittmann et al., 2008

• one-armed bandit task

12
...slighty variated
13

• The rules
• - 100 different pictures

14

• The rules
• - 100 different pictures
• - each stands for a specific probability of the
  subject winning one Pound (mean .33) Unknown to
  participants!

15

• The rules
• - 100 different pictures
• - each stands for a specific probability of the
  subject winning one Pound (mean .33). Unknown to
  participants!
• - on each trial subjects choose between four
  images

16

• The rules
• - 64 different pictures
• - each stands for a specific probability of the
  subject winning one Pound (mean .33). Unknown to
  participants!
• - on each trial subjects choose between four
  images
• - subjects find out a picture's reward
  probability only by repeatedly choosing it. 100
  Trials in total.

17

• Important additional rule
• Half of the 64 images were presented to
  participants before the actual experiment!

18

• Note Two types of novelty not to be mixed up
• 1 Images presented to subjects before the actual
  experiment or not familiarized vs novel images.
• Note Familiarization does not tell subjects
  anything about the reward probability.
• 2 Images appearing for the first time in the
  bandit task reward is unknown, only to be found
  out by repeatedly choosing the image.

19
What do we want to find out?

• How do subjects calculate an optimum trade-off
  between
• Not in this experiment!!!!!

Choosing new images for the sake of
better-informed decisions in the future
Choosing images that proved rewarding in
previous choices
20

• The calculation is too complicated...

21

• This calculation would be too complicated...
• ...therefore the investigators looked for the
  short cut (novelty bonus)

22

• This calculation would be too complicated...
• ...therefore the investigators looked for the
  short cut (novelty bonus)
• ...and therefore images were divided into novel
  and familiarized images.

23
Remember?

• Two types of novelty not to be mixed up
• 1 Images presented to subjects before the actual
  experiment or not familiarized vs novel images.
• Note Familiarization does not tell subjects
  anything about the reward probability.
• 2 Images appearing for the first time in the
  bandit task reward is unknown, only to be found
  out by repeatedly choosing the image.

24

• Note Novel and familiarized images have the same
  reward probability on average.

25

• Note Novel and familiarized images have the same
  reward probability on average.
• ...So preferring novel images would be
  irrational!!!
• (As we hypothesize the short-cut to be, because,
  again, the fully rational calculation is too
  complicated.)

26

• How to calculate whether novel images are
  preferred?

27

• How to calculate whether novel images are
  preferred?
• Again The temporal difference learning model.

28

• (some formulae)

29

• - Novelty influences decision making even if no
  exploration is entailed!
• - In this experiment, novelty was worth 4 pence.
• 2 2 4

30

• What about underlying brain activity?

31
(No Transcript)
32

• A the component of theprediction error signal
  in-
• dependent of novelty

33

• A the component of theprediction error signal
  in-
• dependent of novelty
• B the component of the additional PE signal due
  tonovelty bonus

34

• A the component of theprediction error signal
  in-
• dependent of novelty
• B the component of the PE signal that is due
  tonovelty bonus
• C the overlap between the two

35

• To take home
• novelty constitutes a reward-like bonus ...and
  therefore evokes an additional prediction error
  signal,
• ...which is located in the striatum.
• To think about 2 possibilities
• - Novelty acts as a bonus in itself and therefore
  makes the prediction error more positive,
  irrespective of reward outcome.
• - Novelty is irrationally taken to predict
  additional reward, which increases Q. In case of
  no reward obtained, the prediction error would
  become more negative.

36
New topic Incentive salience
37
Incentive salience Berridge, Robinson, 1998

• Ikemoto Panksepp, 1996

38

• What does it tell us?
• (your turn)

39

• Conclusion from the experiment
• DA does not encode pleasure itself,

40

• Conclusion from the experiment
• DA does not encode pleasure itself,
• ...but underlies reward-seeking behaviour.

41

• Conclusion from the experiment
• DA does not encode pleasure itself,
• ...but underlies reward-seeking behaviour.

42
(No Transcript)
43

• How can this finding/hypothesis be integrated
  into the concepts of dopamine function we have
  got to know so far?

44

• How can this finding/hypothesis be integrated
  into the concepts of dopamine function we have
  got to know so far?
• Discussion!

45

• Two points to take home
• About novelty bias on decision making
• - a novelty bonus is used to push our decision
  making towards exploration, because...

46

• Two points to take home
• About novelty bias on decision making
• - a novelty bonus is used to push our decision
  making towards exploration, because...
• - calculating future advantages of exploration
  would be too complicated
• - therefore...

47

• A few points to take home
• About novelty bias on decision making
• - a novelty bonus is used to push our decision
  making towards exploration, because...
• - calculating future advantages of exploration
  would be too complicated
• - therefore decisions favouring novelty are often
  irrational

48

• A few points to take home
• About novelty bias on decision making
• - a novelty bonus is used to push our decision
  making towards exploration, because...
• - calculating future advantages of exploration
  would be too complicated
• - therefore decisions favouring novelty are often
  irrational
• About incentive salience
• There is evidence for the notion that dopamine
  does not encode pleasure but drives subjects to
  pleasure-seeking behaviour

49
References

• Wittmann, B.C., Daw, N.D., Seymour, B. Dolan
  R.J. Striatal activity underlies novelty-based
  choice in humans. Neuron, 58, 967-973 (2008)
• McClure, S.M., Daw, N.D. Montague, P.R. A
  computational substrate for incentive salience.
  Trends Neursci 26. 423-8 (2003)
• Ikemoto S., Panksepp, J. Dissociations between
  appetitive and comsumatory responses by
  pharmacological manipulations of reward-relevant
  brain regions. Behav. Neurosci. 110. 331-45 (1996)