Experimental Design

1
Experimental Design
2

• The Big Questions . . .
• How is the effectiveness of a manipulation
  verified?
• Why care?
• What mistakes are commonly made?

3
How do we knowmanipulations really work?
4
A critical ingredient of an experiment is the
manipulation of one or more variables. By
harnessing the timing and variability of one
variable, the experiment eliminates the
directionality problem. Clear inference requires
more than the mere manipulation of a variable.
The manipulation holds the same status as any
other measure in the design and so its
reliability and validity are important to
consider.
5
The reliability of a manipulation affects the
statistical conclusion validity of the relation
between the manipulation and an outcome
variable. The validity of the manipulation
affects the construct validity of that
operational definition and the external validity
of any causal relations involving that
operational definition.
6
A manipulation will be coded quantitatively and
used in statistical analyses. In a simple
two-group experiment, participants might be given
codes of 1 for Control and 2 for Treatment. A
statistical analysis would then take the form of
either a simple t-test for the difference between
means for the outcome variable, or, a
point-biserial correlation that relates the
treatment code to the outcome variable.
7
More often than not, a manipulation is simply
accepted at face value and is assumed to be a
perfectly reliable variable. After all, everyone
within a particular condition gets the same
score. But, that score may not be an accurate
representation of the manipulation that any
particular person actually received. As in
traditional measurement, participants may not
get the scores they deserve based on the
manipulation to which they were actually
exposed. The consequence is a manipulation with
less than perfect reliability.
8
As is true in traditional measurement theory, an
unreliable treatment implementation will
attenuate the correlation between treatment and
outcome.
9
If ZOutcome bZTreatment e and Treatment is
implemented with perfect reliability, then the
only source of variability within groups will be
due to random error because within groups,
ZTreatment will be a constant. But, if the
treatment is implemented unreliably, then
ZTreatment is not a constant within groups and
ZOutcome will have an additional source of
variability.
10
(No Transcript)
11
Careful training can usually impose the kind of
consistency in how a manipulation is delivered
that its reliability will be substantial and not
a source of error in a study. A potentially more
important problem is the validity of the
manipulation. Validity here, as in traditional
measurement, refers to the accuracy of the label
that is attached to the manipulation.
12

• The label for a manipulation refers to the state
  that the manipulation is intended to induce in
  participants. This suggests that manipulations
  have two different features
• Structural features How the manipulation is
  implemented (reliability)
• Psychological features The state or experience
  that the manipulation is intended to induce
  (validity)

13
Attempts to verify that the manipulation produced
the intended state are called manipulation
checks. They represent attempts to measure and
verify that experimental groups are different in
ways that should be true if the manipulation is
working. Like traditional measurement, some
manipulations are much more distant from their
constructs and so demonstrating their validity is
much more important and difficult.
14

• Examples
• Cold pressor test as a manipulation of pain
• Medication as a manipulation of physiology
• Task difficulty as a manipulation of frustration
• Priming as a manipulation of accessibility
• Uncontrollable failure as a manipulation of
  learned helplessness

15
All have a common underlying model
In this model, the psychological state is a
mediator variable. It transfers the effect of the
manipulation to the outcome.
16
Example In a learned helplessness study,
uncontrollable failure supposedly induces
perceived loss of control, which in turn produces
the deficits associated with learned helplessness
(e.g., poor task performance on a solvable
follow-up task).
UncontrollableFailure
Perceived Loss of Control
Task Performance
17
To verify this model, a researcher would need to
measure control perceptions as well as task
performance. The usual tactic is to then test for
differences on these measures between treatment
and control conditions in separate ANOVAs. The
joint outcomes of those two tests produce four
possible outcome patterns.
18
(No Transcript)
19
Case 1 is usually considered sufficient evidence
for claiming that the manipulation worked, that
is, that Manipulation Psychological State
Outcome
20
Case 2 indicates that the manipulation altered
the measure of psychological state, but did not
produce significant changes on the outcome
measure. Why might this happen? What implications
does it have for research?
21
Case 3 occurs when the manipulation affects the
outcome as intended but does not have an
influence on the psychological state measure.
This suggests that the underlying process is
unclear. Why might this happen? Implications?
22
Case 4 occurs when the manipulation affects
neither the outcome measure nor the manipulation
check.
23
One major problem is lack of reliability.
Manipulation checks are often constructed from
few items (perhaps only 1). The lack of
reliability in this critical measure, and in the
outcome, can make a real Case 1 appear to be an
instance of Case 2, Case 3, or Case 4.
24
The traditional two-test approach to testing
the validity of the manipulation is also
flawed It does not demonstrate the discriminant
validity of the manipulation. Perhaps the
manipulation created other unintended
psychological states that change the outcome in
addition to, perhaps instead of, the intended
psychological state.
25
Anger
Poor Performance on a Follow-up Task
Perceived Loss of Control
Failure
Suspicion
26
A good manipulation check considers convergent
and discriminant validity. It is as important to
rule out plausible alternative processes as it is
to confirm the predicted mediating process.
27
The second problem with the traditional two-test
procedure is that it does not adequately test the
presumed underlying model
(B)
(A)
1
Manipulation Psychological
State
Outcome
2
3
An effective manipulation should act
throughPaths 1 and 2, transferring all of its
effect on the outcome through the mediator. Baron
and Kenny (1986) proposed requirements for
establishing mediation.
(C)
28

• A complete test for mediation requires the
  following
• The manipulation must produce changes in the
  outcome. This is demonstrated as part of the
  usual two-test approach. In the path model, it
  requires that rAC be different from zero. This
  test establishes that there is an effect that
  might be mediated.

(B)
1
(A)
Manipulation Psychological
State
Outcome
2
rAC gt 0
(C)
29

• A complete test for mediation requires the
  following
• The manipulation must produce changes in the
  mediator. This is also demonstrated in the usual
  two-test approach. In the path model, it requires
  that rAB be different from zero. This test
  establishes that the manipulation transfers its
  effect to the mediator.

(B)
rAB gt 0
(A)
Manipulation Psychological
State
Outcome
2
rAC gt 0
(C)
30

• The mediating variable and the outcome variable
  must be related independent of the manipulation.
  This requires that the regression of the outcome
  on the mediator holding the manipulation constant
  be greater than 0.

(B)
rAB gt 0
(A)
Manipulation Psychological
State
Outcome
BBC.A gt 0
rAC gt 0
(C)
If this condition is satisfied, then the
psychological state is a mediator but it may not
be a complete mediator.
31

• The manipulation and the outcome variable are not
  related when the mediating variable is
  statistically controlled. This requires that the
  regression of the outcome on the manipulation
  holding the manipulation check constant be 0.

(B)
rAB gt 0
(A)
Manipulation Psychological
State
Outcome
rAC gt 0
BBC.A gt 0
BAC.B 0
(C)
If this condition is satisfied, then the
psychological state is a complete mediator.
32
A single significance test for mediation is
provided by the Sobel test
There are actually three different versions of
the Sobel test. The original version, proposed
by Sobel, eliminates this last term. Adding the
last term represents the Aroian version of the
test. This one is recommended by Baron
Kenny. Subtracting the last term represents the
Goodman version.
33
CORRELATIONS /VARIABLEStx state outcome
/PRINTTWOTAIL NOSIG /MISSINGPAIRWISE .
34
GLM state outcome BY tx /METHOD SSTYPE(3)
/INTERCEPT INCLUDE /EMMEANS TABLES(tx)
/PRINT DESCRIPTIVE ETASQ PARAMETER RSSCP
/CRITERIA ALPHA(.05) /DESIGN tx .
35
(No Transcript)
36
(No Transcript)
37
(No Transcript)
38
(No Transcript)
39
REGRESSION /DESCRIPTIVES MEAN STDDEV CORR SIG
N /MISSING LISTWISE /STATISTICS COEFF OUTS CI
R ANOVA CHANGE /CRITERIAPIN(.05) POUT(.10)
/NOORIGIN /DEPENDENT outcome /METHODENTER
state tx .
40
(No Transcript)
41
(No Transcript)
42
(No Transcript)
43
(No Transcript)
44
(No Transcript)
45
UNIANOVA outcome BY tx WITH state /METHOD
SSTYPE(3) /INTERCEPT INCLUDE /PRINT
PARAMETER /CRITERIA ALPHA(.05) /DESIGN
state tx .
46
(No Transcript)
47
(No Transcript)
48
rAB .78 BAB 3.53 (.54)
(B)
(A)
Manipulation Psychological
State
Outcome
rAC .75
BBC.A .84 (.1)
BAC.B .22
(C)
49
Two excellent websites for mediation and the
Sobel test
http//www.psych.ku.edu/preacher/
http//davidakenny.net/cm/mediate.htm
50
Requirements 1 and 2
Outcome
Control
Treatment
Manipulation Check
51
Requirement 3
Outcome
Control
Treatment
Manipulation Check
52
Requirement 4
Adjusted difference
Outcome
Control
Treatment
Manipulation Check
53
The problem with Case 1 is that it is consistent
with a non-mediational model and does not
distinguish between the two forms of mediation
Psychological State
Manipulation
Outcome
Manipulation
Psychological State
Outcome
Manipulation
Psychological State
Outcome
54
With partial mediation, an additional mediator
may have been missed. It is important to
anticipate, measure, and rule this out.
Psychological State 1
Outcome
Manipulation
Psychological State 2
What else might explain partial mediation?
55
The manipulation check could also moderate the
effect of the manipulation
Psychological State 1
Manipulation
Outcome
56

• Summary
• To make a convincing claim for the validity of a
  manipulation requires
• Presumption of reliability
• Measurement of presumed mediator
• Measurement of possible alternative mediators
• Complete statistical tests that establish the
  degree of mediation present.

57

• Some Additional Considerations
• Scale level versus scale differences.
• Differences on the manipulation check tell only
  part of the story. Finding, for example, that on
  a 7-point scale the treatment group mean (M
  5.5) is significantly lower than the control
  group mean (M 6.5) for perceived control is
  important, but the overall levels of control can
  be important to consider as well.
• To what extent has learned helplessness really
  been created if the overall levels of control are
  so high?

58
Some Additional Considerations 2. Objective
versus subjective assessment. (a) Participant
reports versus judges or raters (e.g., to
avoid sensitizing subjects or disrupting the
process) (b) Nonverbal behavior (e.g.,
emotion) (c) Other unobtrusive markers (e.g.,
self- reference) (d) The participants
ability to report on the process
59
Inaccessible Psychological State
Outcome
Manipulation
Verbal Explanatory System
60
3. Reactivity (a) Measurement may instigate
processing that would not ordinarily
occur (b) Measurement might induce distinctions
not ordinarily made (c) Repeated assessment
can change the subject
61

• Solutions
• Assess the psychological state at the end. Why
  not?
• Indirect assessment. Perhaps, as an adjunct.
• Two-stage procedure or a four-group design
• Manipulation Psychological State Outcome
• Psychological State Outcome
• Manipulation
  Outcome
• Outcome

62
When Manipulations Fail When the manipulation
fails to have the predicted effect on the
outcome, the manipulation check can be used as a
proxy for the manipulation and its relation to
the outcome used as a first approximation to the
effect the manipulation might have were it to be
implemented successfully. This internal analysis
is no longer an experiment, but it may reveal
information useful for designing a more
convincing future study.
63

• The Big Questions . . .
• How is the effectiveness of a manipulation
  verified?
• Why care?
• What mistakes are commonly made?