Experimental Design

1
Experimental Design Analysis

• Design
• February 13, 2007

2
Designs

• Research goals
• Test a causal relationship
• Examine boundary conditions for causal
  relationship
• Tease apart competing explanations
• Demonstrate convergent, discriminant validity

3
Testing a Causal Relationship

• Does advertising increase sales?
• What is the independent variable?
• How do we operationalize it?
• Levels?
• How do we measure the dependent variable?
• Units?
• Setting?

4
Examining Boundary Conditions

• When does advertising fail to increase sales?
• What are possible independent variables?
• How to operationalize them?
• Levels?
• What is the inference we can make?

5
Competing Explanations

• Perhaps advertising doesnt increase sales, but
  manufacturer investment does
• Manufacturer investment is confounded with
  advertising expenditures
• Does advertising have an effect independent of
  manufacturer investment?
• What are the constructs being tested?

6
Convergent Validity

• Convergent validity
• Measures of constructs that theoretically should
  be related to each other are, in fact, observed
  to be related to each other
• When a variable produces a pattern of results
  similar to another variable known to represent a
  construct, convergence is established

7
Convergent Validity

• Example of convergent validity
• How to represent cognitive resources
• Expert vs. novice
• High motivation vs. low motivation
• Effortful vs. automatic
• No cognitive load vs. cognitive load

8
Discriminant Validity

• Discriminant validity
• Measures of constructs that theoretically should
  not be related to each other are, in fact,
  observed not to be related to each other
• When two separate processes are hypothesized,
  different effects are observed

9
Discriminant Validity

• Example of discriminant validity
• Promotion vs. prevention as motivational states
• Approach positive outcome
• Avoid negative outcome

10
Convergent Discriminant Validity
11
Within-Subjects Designs

• Also known as repeated measures because design
  repeats manipulation for participants
• Effect of training program on organizational
  leadership
• Program vs. no program
• After 1 week and after 1 month
• Therapeutic effect of varying dosages of drug A
• 1ml, 5ml 10ml

12
Sources of Confounds

• Sometimes confounds are introduced by the way the
  experiment is executed
• Order of manipulations
• Experimenter (status, style, sex)
• Time of day or setting

13
Variable Selection

• Adequacy of dependent variable to represent
  construct of interest
• How to represent depth of processing?
• How to represent persuasion?
• How to represent leadership? Intelligence?
  Effectiveness?
• One item vs. multiple items?
• Precedent

14
Necessity of Pretests?

• Advantages
• Calibration of manipulations

• Disadvantages
• Calibration on one sample doesnt necessarily
  transfer to another
• Reactivity may introduce bias
• Confers false sense of security

15
Necessity of Manipulation Checks?

• Manipulation checks, process measures
• Does absence negate hypothesized effect?
• Does presence rule out rival explanations?
• According to Sigall and Mills (1998), what are
  advantages of manipulation checks?