Internal and External Validity

1
Internal and External Validity

• Gitendra Uswatte
• Department of Psychology, UAB

2
Types of Validity
3
Internal Validity

• History or Background Effects
• Changes in subjects environments that might
  explain study results
• either in or outside of the lab
• 9/11 attacks taking place during a study of
  anxiety treatment based in New York City
• Advent of summer smog for study of asthma
  treatment based in Birmingham
• Fire drill during study of startle responses in
  patients with GAD

4
Internal Validity

• Maturation Effects
• Changes in physiology or psychology of subjects
• really a type of background effect but refers to
  internal as opposed to external changes
• developmental changes
• Changes in Kohlbergs stages of moral reasoning
  that take place spontaneously during testing of
  intervention for enhancing kindness in
  adolescents
• changes in hormone levels
• Menstrual cycle changes in estrogen during a
  study that tests an intervention that improves
  motor function by stimulating brain plasticity

5
Internal Validity

• Practice/Order Effects
• Changes in scores on testing instruments due to
  familiarity with the tests or practice on the
  tests, i.e., order in which they are taken
• especially salient for within-groups designs
• Contrast Effects
• Changes in scores due to order relative to
  another condition

6
Internal Validity

• Instrumentation Effects
• Changes in testing instruments that might affect
  test scores or their interpretation
• Might be caused by change in the composition of
  the test items (e.g., large scale surveys of
  sexual behavior across time)
• Wear and tear on instrument if measure is
  physical
• Response shift, i.e., change in how subject
  interprets questions or scales

7
Internal Validity

• Regression to the Mean
• Tendency for a group with extreme test scores to
  have scores closer to the mean when tested again
• For example, only subjects who score above a
  certain threshold on the BDI (i.e., have extreme
  scores) might be selected for a study evaluating
  the efficacy of an antidepressant.
• These subjects would be expected to score lower
  on their second assessment simply because of the
  operation of this effect.

8
Internal Validity

• Selection Bias
• Systematic pre-treatment differences between
  study groups that might explain results
• In developmental and clinical studies, random
  assignment is not always possible. You might
  have to assign pre-existing groups to control and
  experimental conditions
• When testing school-based anti-violence programs,
  for example, one might have to assign one school
  to the experimental condition and another to the
  control
• Can only subjects who can wait for treatment to
  control
• Failure of random assignment when cell size lt 20

9
Internal Validity

• Attrition
• Uneven or high dropout
• Subjects who do worse are more likely to drop out
  inflating your treatment effect
• Introduces a post-hoc selection bias
• Threatens generalizability of result, i.e., if
  dropout rate is high results may valid for only
  that subset of subjects who can tolerate the
  experimental treatment or who are responsive to it

10
Internal Validity

• Diffusion or Imitation of Treatment
• Control procedure contains many or important
  elements of the therapeutic factors in the
  experimental procedure
• Basically, a failure to manipulate the IV
• Can be due to flaws in design or implementation,
  e.g.,
• study testing efficacy of TM for reducing blood
  pressure in which the control procedure is guided
  visual imagery for relaxation
• Failure to maintain adequate training intensity
  in motor controls study where intensity of
  training is an important therapeutic factor for
  the experimental treatment

11
Internal Validity

• Special Treatment/Reaction of Controls
• Hawthorne Effect
• Subjects change their behavior simply because
  they are being studied
• Attention over and above what is standard given
  to control group produces an effect
• Assignment to control group inspires subjects to
  compete with experimental group
• For example, if the control condition is a
  standard teaching method and the experimental
  group is a new method, teachers in the control
  group exert extra effort when teaching to protect
  the status quo

12
List of Effects that ThreatenInternal Validity
13
External Validity

• Sample Characteristics
• Study sample is not representative of population
  about which conclusion is made
• Do results apply to populations or subpopulation
  than the one studied?
• Data often show generality even across species so
  this concern is often expressed excessively
• Findings from college student samples often
  generalize to others
• Sometimes point is not generality but whether the
  phenomenon can be demonstrated at all in any
  population
• Of course, there are important exceptions

14
External Validity

• Stimulus Characteristics and Settings
• Features associated with experimental
  intervention and study setting, e.g., room in
  which study is conducted, experimental
  personality, etc
• For example, white coat effect on blood pressure
  has pointed to importance of ambulatory blood
  pressure monitoring
• Or, using one particular video vignette as a
  stimulus for a study

15
External Validity

• Reactivity of Experimental Arrangements
• Same as Hawthorne effect
• Multiple-treatment Inferences
• Same as contrast effect

16
External Validity

• Novelty Effect
• Effect is due to something different or new being
  done rather intervention per se
• For example, effectiveness of yellow vs. red fire
  trucks in avoiding accidents
• Similar to Hawthorne effect
• As with Hawthorne effect, Multiple-treatment
  inferences this can also be a threat to internal
  validity

17
External Validity

• Reactivity of Assessment
• Are subjects changing their behavior or responses
  because they know they are being tested?
• Do test scores reflect real-world behavior?
• For example, scores of rehabilitation patients on
  laboratory motor performance tests often do not
  reflect actual use of the impaired extremity in
  daily life
• Similar to Hawthorne, Stimulus Characteristics,
  and Instrumentation effects

18
External Validity

• Test Sensitization
• Does pre-test sensitive subjects to construct
  being tested, and change their responses?
• Do particular way that questions are posed on
  post-test sensitize subjects to the construct
  under study in a way that biases the test of your
  hypothesis?
• Similar to Practice/Order and Instrumentation
  effects

19
External Validity

• Timing of measurement
• When particular timing of measurement biases
  findings in favor of a particular hypothesis
• For example, only evaluating the efficacy of an
  antidepressant only immediately after finishing
  the course of drug therapy
• For cross-sectional studies, need to consider
  cohort effects, i.e., do findings only apply to
  the particular generation studied.

20