Title: Basic experimental design, control, and context
1Psychology 242, Dr. McKirnan
Right click for full Screen or end show.
Left click to proceed,
1/17/09
- Basic experimental design, control, and context
- Overall research strategy
- Validity
- Internal Validity
- External Validity
- Ecological Validity
- Researcher biases
- Participant biases
- Social cultural context bias
2Research flow What is the core research question?
- What is being studied? Why?
- What is the contrast space?
- What is compared to what?
- What needs explaining / what is given
- What is known about the core hypothetical
constructs? - How do you propose they relate to each other?
- How will the study expand or clarify theory?
- Use existing theory to explain a new phenomenon
(Divergent use of theory)? - Test contrasting theories of one phenomenon
(Convergent use of theory)? - New / expanded theory?
3Research flow
- What key variables best represent the constructs?
- What is your prediction about how they are
related? - Measurement design?
- Quasi-experiment?
- True experiment?
- How have the variables been operationally
defined? - Alternative operationalizations?
- Implications of this operationalization?
- Is the predictor best measured or manipulated?
- Virtues / limitations of each approach?
- Sampling?
4Basic Designs Methods cont.
- Who are your participants?
- What is your sampling method Probability or
Non-Probability sample? - Where do you recruit participants? -- what is
your sampling frame? - Is you study externally valid -- does your sample
represent the population? - How will your control group be formed?
- Can you practically ethically have one?
- Are you using existing groups?
- Can participants self-select into a group?
- Is random assignment or matching feasible?
- How is the independent variable presented?
- Simple presence v. absence?
- Different doses?
5Overview Basic Designs
- Pre-experimental designs no control group
Post-Test Only Design
Pre- Post- Test Design
True (or Quasi-)experimental designs with a
control group
After only Control group design
Pre- Post- Group Comparisons
Multiple group comparison
6Pre-experimental designs
Post-Test Only Design
Treatment
Measure
Group
Only 1 group - typically an existing group no
selection or assignment occurs.
Experimental intervention (Treatment) may or
may not be controlled by the researcher. Use
for naturally occurring or system-wide events
(e.g., group trauma, government policy change,
etc.).
Measurement may or may not be controlled by the
researcher.
Pre- Post- Test Design
Measure1
Treatment
Measure1
Group
- Only one group
- only group available?
- naturally occurring intervention?
Measurements given to all participants at
baseline follow-up
All participants get the same treatment, which
may or may not be controlled by the researcher.
7Pre-experimental Designs (2)
Advantage of Post- Pre- Post- Designs
- Study naturally occurring intervention,
- e.g., test scores before and after some school
change, - rime rates after a policy change, etc.
- Having both Pre- and Post measures allows us to
examine change.
- Disadvantage no control group many threats to
internal validity - Maturation Participants may be older / wiser by
the post-test - History Cultural or historical events may occur
between pre- and post-test that change the
participants - Mortality Participants may non-randomly drop out
of the study - Regression to baseline Participants who are more
extreme at baseline look less extreme over time
as a statistical confound. - Reactive Measurement Participants may change
their scores due to being measured twice, not the
experimental manipulation.
8Experiments
- After only Control group design
Treatment
Measure
Group 1
Measure
Group 2
Control
- Adds a control group. Either
- Observed Groups
- Naturally occurring (e.g., Class 1. v. Class 2)
or - self-selected (sought therapy v. did not).
- Assigned Groups Randomly assign participants to
experimental v. control group, or match
participants to create equivalent groups.
Measure Dependent Variables(s) only at
follow-up. Use experimental or standard measures
(e.g., grades, census data, crime reports).
- Advantage Control group lessens confounds /
threats to internal validity. - Random assignment decreases threats to internal
validity. - Disadvantage Existing or self-selected groups
may have confounds. - No baseline or pre- measure available
- assess change?
- ceiling (or floor) effects?
- cannot assess equivalence of groups at baseline.
9Basic Designs True experiments (2)
- Pre- Post- Group Comparisons (most common study
design)
Group 1
Measure 1
Treatment
Measure 2
Control
Group 2
Measure 1
Measure2
Two groups Observed (quasi-experiment)
or Assigned (true experiment).
Only one group receives experimental intervention.
- Post-test follow-up of dependent variable(s)
- Simple outcome
- Change from baseline.
Baseline (pre-test) measure of study variables
and possible confounds.
Advantages Pre-measure assesses baseline level
of Dependent Variable -- allows researcher to
assess change -- can detect ceiling (or floor)
effects -- can use to assign participants to
groups via matching -- can assess baseline
equivalence of groups Disadvantage Highly
susceptible to confounds if using observed or
self-selected groups.
10More Complex Experimental Designs
- Multiple group comparison
Measure2
Treatment 1
Treatment 2
Measure2
Control
Control
- 3 (or more) groups
- typically formed by Random assignment.
- 2 experimental groups, e.g.
- low v. high dose,
- exp. situation 1 v. 2, etc.,
- plus the control group.
- Compare
- Level 1 of independent variable from Level 2
- Either / both experimental groups from control
grp.
- Advantage Test dose or context effects
- Drug doses, amounts of psychotherapy, levels of
anxiety, etc. Increasing dose effect can be
tested against no dose. - Diverse conditions to test 2nd hypotheses or
confounds, e.g., therapy delivered by same sex v.
opposite sex therapist. - Disadvantage
- More costly and complex.
- Potential ethical problem with a no dose (or
very high dose) condition.
11Overview of true experimental designs
Representative of the larger population? -
Selection - Size of sample
- Groups equal at baseline?
- Existing groups or self-selection
- v.
- - Random assignment
Equality of procedures? - Information -
Expectancies - Quality of blinding
- Faithfulness of treatment?
- - Operational def.
- - Correct dose?
- Manipulation check
Groups really different at outcome? -
Statistical significance
Internal Validity Likelihood of chance results
External validity Random selection of sample
Internal validity Random Assignment
Internal validity Lack of confounds
External Validity Correct independent variable?
12Basics of Design Internal Validity
Internal Validity Can we validly determine
what is causing the results of the experiment?
- General Research Hypothesis the experimental
outcome (values of the Dependent Variable) is
caused only by the experiment itself (Independent
Variable). - Confound a 3rd variable (unmeasured variable
other than the Independent Variable) actually led
to the results. - Core Design Issue Eliminate confounds in..
- Assigning participants to experimental v. control
groups - Procedures in each group.
13Key threats to internal validity
- Lack of control group
- Non-equivalent groups
Maturation Participants may be older / wiser by
the post-test History Cultural or historical
events may occur between pre- and post-test that
change the participants Mortality Participants
may non-randomly drop out of the study Regression
to baseline Participants who are more extreme at
baseline look less extreme over time as a
statistical confound. Reactive Measurement
Participants may change their scores due to being
measured twice, not the experimental manipulation.
Group differences in any of these represents a
core confound.
14Internal validity, 2
Ensuring Internal Validity 1. Group Assignment
- Self-selection people may join or drop out of
groups for reasons other than the independent
variable. - Self-selection in rare, substantial confound if
present. - Self-selection out common confound in behavioral
studies, e.g., Project EXPLORE and differential
drop-out of risky MSM from experimental group. - Existing groups may differ on variables besides
independent variable. - Naturally occurring convenience samples (e.g.,
9am class v. 11am class, NYC v. Chicago) may
differ in subtle ? variables that are difficult
to assess - Naturally occurring groups that express
phenomenon (those who seek therapy v. not, more /
less extreme scores at baseline) may differ in
crucial variables, some of which may be
measurable. - Cures
- Random assignment to experimental v. control
groups. - Matching participants on key confounding measures
(e.g., education, age) and systematically
assigning to groups. - Assessment of potential confounds (demographics,
? variables).
15Internal validity procedures
Ensuring Internal Validity 2. Procedures
- Equality of procedures across experimental
control group all conditions must be held
constant except the IV. - Participants blind
- Equalize (control) expectations motivations x
group - Control drop-out, loss to follow-up
- Experimenter blind
- Control explicit bias
- Control self-fulfilling expectations
- Standardization / automation of experimental
process - All procedures must be independent of the
participants group assignment - Equality of procedures across pre-test and
post-test
16Summary Internal validity
- Internal Validity overview
- Are results due to something other than the
Independent Variable? - ? Confounds within the experiment
- Procedural differences x group
- Biased assignment to group.
- ? Confounds from outside the experiment
- History, maturation, cultural change etc.
- within single-group study
- differences x group in multi-group study
17Generalizability general research results
External Validity Can we generalize from this
study to the larger world?
The larger population
Other settings
External Validity How well can we generalize to
18External validity The larger population
The larger population
- How well does your research sample represent the
larger population you want to generalize to.
- Volunteerism bias people who volunteer for
research may be unlike the general population. - attitudes, motivations
- responses to financial incentives
- Convenience sampling of existing groups
- College class, specific shopping mall, bar or
other venue - Bias by self-selection.
- Random selection maximizes external validity by
best representing the population.
We will spend several lectures on sampling later
on.
Cure
19Random selection v. assignment
- Key distinction
- Random selection from a larger population to the
research sample. - Random assignment from the sample to
experimental v. control groups.
20External validity social cultural context.
Other settings
How representative (or realistic) is the social
cultural setting of the research?
- Context is the research setting similar to real
life settings, or are the results specific to
this laboratory, this questionnaire, etc.? - Procedures are results an artifact of a
particular procedure, experimenter, or place or
setting? - Replication of study by different researchers, in
different setting(s), with different samples. - Converging studies that test the same hypotheses
with substantially different methods - Field v. lab studies
- Experimental v. non- (or quasi-) experimental
methods. - Qualitative v. quantitative approaches
Cures
21External validity the conditions or model
How representative is the Independent Variable
(experimental manipulation)?
- Modeling the phenomenon
- does the experimental condition or manipulation
create the state you want it to? - e.g., stress, mood, information, motivation
manipulation - Dose of the IV e.g.
- drug dose
- Psychotherapy intensity
Cures
- Manipulation check
- Dose-response studies
22External validity the outcome
Representativeness of the Dependent Variable
- Operationalization
- does the assessment of the DV reflect how the
process works outside of the lab? - Construct validity
- Are you modeling the hypothetical construct you
intended? - How well have you captured a specific ? process?
Cures
- Standardized measures or assessments e.g.,
depression, stress - Psychometric studies
- Reliability does the measure consistently yield
similar scores? - Validity does an instrument measure what it is
intended to?
23External validity summary
Is the sample typical of the larger population?
The research Sample
Is this typical of real world settings where
the phenomenon occurs?
The research Setting
Is the outcome measure represen-tative, valid
reliable?
The Dependent Variable
The study structure context
Does the experimental manipulation (or measured
predictor) actually create (validly assess) the
phenomenon you are interested in?
24Generalizability example
- Core design elements (external validity areas)
- Sample UIC students
- Setting Classroom situation
25Generalizability population and context
How well do the results generalize to
Larger population(s)
People in general
Across contexts
Other Americans
Other University settings
Other structured settings
Other social situations
Other young people
UIC students tested in class
26Generalizability Independent dependent
variables.
External validity Do the results generalize to
Across forms of anxiety (the IV)
Natural anxiety
Across outcomes (DV)
Other forms of stress
Other cognitive tasks
Less structured leaning tasks
Job or other performance
Other instructions
I.Q. instructions abstract memory task
27Generalizability of student experiment
- How well do these data generalize to.
The larger population?
Sample UIC Students
Setting Classroom
Dep. Var. Abstract memory task
Other social or learning settings?
Other cognitive skills or tasks?
The study structure context
Other anxiety conditions?
28Generalizability general research results
Each element of external validity helps determine
how meaningful research results are.
29Ecological Validity
The larger population
Other settings
Ecological Validity specifically addresses the
context of research
Other out-comes
Key elements of the research process
Other conditions
- The researcher
- The research participant
- The physical, social and cultural, setting
research takes place within.
30Research ecology Researcher Participant
- Expectations biases
- Motivations
- Personal characteristics
- Cultural background
- Expectations biases
- Motivations
- Personal characteristics
- Cultural background
- Similarity or conflict
- Personal / cultural expectations
- Social roles
- Motivations
31Context effects
- Time place (field v. lab, medical v. academic)
- Familiarity or comfort
- Expectations (e.g., medical setting, formal v.
informal) - Information available
- Reactive measurement
32Context effects
- Complex interaction of Researcher by Participant
by Context - May create very specific conditions under which
data are collected - Can limit External Validity
33Ecological Validity Researcher Effects
Personal attributes of the researcher
The researcher
- Biosocial age, race, gender, status...
- inherent social conflicts?
- Representative context for all participants?
- Psychosocial attitudes, warmth, skills...
- degree of cooperation
- participants' understanding of tasks
- Situational e.g., physician, teacher as
researcher - prior relationship or 'dual role' situation
34 Ecological Validity, 2 Researcher Effects
- Researchers biases or expectations
- Knowledge of hypothesis or experimental condition
- Response to participants attributes
The researcher
- Self-fulfilling expectations (verbal or
non-verbal). - Rosenthal experiment smart v. dumb rats
maze learning. - Education research powerful effects of teacher
expectations on student performance. - Biased procedures or handling of participants.
- Clinical research differential handling of
cases. - Mental health research more extreme diagnosis
treatment recommendations for minorities / lower
SES pts. - Biased data recording quantitative and
qualitative - Non-random errors in data coding or entry
- Confirmatory biases in recall.
35Researcher effects
Cures
The researcher
- Randomize experimenters across condition
- match or stratify experimenter x participant
- 'unknown' experimenters
- Blinding of experimenter(s)
- double blind
- Not informing 'hands on' researcher (when
blinding impossible) - Aggressive standardization automation
36Ecological validity participant effects
- Participant expectations
- Motivation to be a 'good' (or bad) subject
- Social desirability responding
- Primarily for personal information
- Cultural personal differences in what is
considered personal - Face to face v. computer assessment
- Changes in response over time (Doll et al., risk
disclosure in brief v. full interviews). - Infer hypothesis or enrollment criteria
(correctly or incorrectly) - HIV vaccine research
- Risky men lied to get into low risk vaccine
cohorts, then showed HIV infections. Did the
vaccine itself cause infections? - Reactive risk behavior concentrated in men who
believed they received the vaccine.
The participant
37Participant effects, 2
- Participants personal characteristics
- Biosocial similar to experimenter issues
- age, race, gender, status
- Possible conflicts
- Values life experience College students in the
laboratory - knowledge and sophistication
- Participants ability to understand research
protocol (also ethical issue). - Variables such as psychological mindedness
behavioral intervention research.
The participant
38participant effects, 3.
The participant
Cures
- Blinding participants
- Constancy of procedures
- automation or structured protocol
- training researchers
- Deception or concealment of hypothesis
- Diverse sampling of participants
- Computer assessment
39Ecological Validity Context and people
Demand characteristics of the research setting
- Social context powerfully affects individual
behavior - Zimbardo prison experiment, Rosenthal
psychiatric settings - Medical context and health measures e.g, "white
coat" effect - Self-awareness ? norm following
- Context and informational availability
- Minimalist social psychology experiments social
judgments. - Survey / interview measures and uni-dimensional
responding - Political / economic demands and simple bias or
fraud - Drug Co. research and cherry picking positive
results - Political pressure for No Child Left Behind
Houston Miracle
40Ecological validity setting effects, 2.
Reactive measurement
- Simple learning of research measures
- Simple sophistication in self-reporting, test
taking skills. - Awareness of research context and response biases
- Social desirability response set.
- Book example responses to erotic stimulus.
- due to the stimulus itself?
- self-generated imagery via exp. demand?
- simulated response?
- Attributing origins of responses
- Research measures or procedures can create
attitude change - E.g., Survey questions normalizing
When do you feel it is O.K. to cheat on
an exam?
..when I really do not know the material ..
when others are doing it .. when I think the
exam is unfair
41Setting effects Cures
- Cures
- Clear description of research context to aid
interpretation - Replication of research in other settings / labs
/ researchers - Converging studies that test the same hypothesis
with different methods / contexts, sources of
participants, measures.
42Design validity overview
- Overall research questions
- Internal validity (confounds)
- Group assignments
- Procedures
- External validity
- Sample ? population
- Context Research lab ? real settings
contexts - Conditions Independent variable ? real
conditions - Outcomes Dependent variable ? adequate model
of phenomenon? - Ecological validity (context conditions)
- Researcher effects
- Participant effects
- Setting effects
- Operational Definitions, hypothetical constructs,
confounds, etc.