Representative design and the problem of generalization

1
Representative design and the problem of
generalization

• Daniel Read
• PRELIMINARY draft 17/06/2008

2
Essence of representative design

• A study is a sample of behavior
• A representative study is one that contains a
  representative sample of subjects, of situations,
  and of tasks.
• The concept of representative design comes from
  Egon Brunswik.

3
Brunswiks ideas were based on his theory

• Humans as intuitive statisticians
• Environment conveys unreliable and redundant cues
  to underlying reality
• Humans learn to interpret these cues to infer
  that reality.
• Led to lens model. Picture here

4
(No Transcript)
5
Brunswick design
Measured photographic size of images correlates
with estimated bodily size much less than with
measured bodily size.
The agent is using more information than
photographic size (and maybe not even using it at
all). Representative design can help us assess
precisely what is going on.
6
But representative design does not need the theory

• Theories/claims about human behavior are claims
  about people/behaviors/tasks (and how they will
  interact with underlying constructs).
• Underlying the concept of representative design
  are two critiques concerning when
  non-representative outcomes can occur
• unfamiliar situations or combinations of
  situations produce atypical responses.
• selecting situations increases likelihood of
  extreme or unlikely observations

7
Some misleading questions
Questions from Piatelli-Palmarini Inevitable
Illusions Tests for confidence in textbooks
inevitably contain many questions like
these. Correct answers are Vegetation and
Peru.

• Adonis was the god of
• (a) Love
• (b) Vegetation
• The potato originated in
• (a) Ireland
• (b) Peru

8
Comparing Representative with Non-R choice of
stimuli
Almanac questions e.g., Which city is farther
north? Non-representative People asked to
choose good general knowledge items Representati
ve Random items chosen from same
source. Source Juslin, 1994
Representative
Non-representative
9
Why stimuli only?

• Is random sampling of methods possible?
• Not meaningful in many cases, but we can ask
• Does changing the method change the result?
• If so, in what way?
• Parameter change? Qualitative shift?
• What method(s) give us the most information about
  the domain we care about?

10
Effect of varying method

• Claim People are hyperbolic discounters
• The rate at which we devalue
• Method Choose between 100 today, or 120 in
  one year. Return given in nominal amounts.
• Alternative Choose between 100, or save the
  100 for one year at 20 interest rate.

10
11
Results Inferred discount function can take
any form, and a wide range of values, depending
on how you ask. Conventional theoretical
interpretation arises from use of Nominal
amount condition.
Maximum borrowing rate
Realistic lending rate
Constant
Hypobolic
Hyperbolic
11
12
Lesson

• We cannot generalise from one method designed to
  measure one construct, to other methods designed
  to measure the same construct.
• Essence of multi-trait, multi-method matrix.

13
The key questions

• (a) How general are the claims do you wish to
  make?
• How general are the results you have (or plan to
  have) obtained.
• Set the goal of making the two levels of
  generality equivalent.

14
What does people are overconfident mean?

• For these people, at all times, for all
  questions, and for all tasks, (inferred)
  subjective probability is greater than objective
  probability?

• Something in between?

• For these people, at this time, for these
  questions, and for this task subjective
  probability is greater than objective probability?

15
Minimal generality

• If theory T states that in situation X a person
  (agent) will do A, and the person in fact does B
  (and does so regularly enough that chance can be
  ruled out) then theory T is ruled out.
• No need to show that the agent will do B in the
  real world
• No need to show that the agent will do B in other
  situations, even other experimental arrangements.

c.f. Mook, 1983
16
Harlows monkeys
Harlow found that baby monkeys hung out with a
cloth mother that did not feed them instead of
with a wire mother that did. But this is
sufficient to test the hunger-reduction
interpretation of mother love. Babies do not
lover their mothers only because she feeds them.
See Mook, 1983
17

• "Representative design in its full scope requires
  not only a basic theoretical and methodological
  restructuring but is a formidable task in
  practice as well. Ideally, it would take
  concerted research projects of a magnitude
  hitherto unheard of in experimental psychology
• Brunswik, 1956

18
References

• Mook, D. G. (1983). In defense of external
  invalidity. American Psychologist, 379-387.
• Juslin, P. (1994). The overconfidence phenomenon
  as a consequence of informal experimenter-guided
  selection of almanac items. Organizational
  Behavior and Human Decision Processes, 57,
  226-246.