Chapter 7: Reliability and Validity in Experimental Research

1
Chapter 7 Reliability and Validity in
Experimental Research
2
Reliability

• Definition _____________________________________
  ___
• Requirements for valid experimental results
• __________________
• Consistency, stability, or repeatability of a
  behavior
• ___________________
• The correctness or truthfulness of an inference
  that is made from the results of an experiment

3
Relationship Between Reliability and Validity

• Experimental reliability ________ exist without
  experimental validity
• Experimental validity _______ exist without
  experimental reliability

4
Experimental Reliability

• Factors influencing experimental reliability
• Number of research participants
• __________________
• Reliability of identifying the ______________
  needed for the study
• Identify indicators of who should be included
• _____________ of the researchers
• gender, attractiveness, anything else???
• _____________of the researcher
• Skills, training

5
Reliability and the Independent and Dependent
Variables

• ___________________ Concrete operations must
  produce the same effect in research participants
  for reliability to exist
• ___________________ -The effect measured should
  be about the same regardless of when it is
  measured

6
Measurement Error

• Definition __________________________________
• Factors causing measurement error
• _______________________
• Relationship between measurement error and
  reliability
• _____________________________

7
Methods of Assessing Reliability

• Successive measurement
• Test-retest
• Equivalent forms
• Simultaneous measurement
• Interrater reliability
• Interobserver agreement
• Internal consistency
• Split-half
• Cronbachs alpha

8
Experimental Validity

• Definition ________________________________
• Types of validity that maximize obtaining
  experimental validity
• Statistical conclusion validity
• Internal validity
• _______________
• ________________

9
Statistical Conclusion Validity

• Refers to the accuracy of an inference that the
  independent and dependent variables covary
• Threats
• ______________________
• Many others but beyond this course to discuss

10
Internal Validity

• Definition-- accuracy of the inference that the
  independent variable caused the effect observed
  in the dependent variable
• Primary threat
• ___________________
• How to eliminate the confounding influence of
  extraneous variables
• __________________________

11
Threats to Internal Validity

• History
• Maturation
• Instrumentation
• Testing
• Regression Artifact
• Attrition
• Selection
• Additive and Interactive effects

12
Factors that Threaten Internal Validity

• There are many threats to internal validity
• L.C., He Made The IRS MIDday
• 1. L.C. ______________________
• Pre-test post-test designs without control
  group
• Pretest ----- Treatment ----- Posttest

13
Factors that Threaten Internal Validity

• Ex
• Does test anxiety program reduce test anxiety in
  freshmen?
• All freshmen complete test anxiety questionnaire
  (high scores indicate high levels of test
  anxiety)
• Students scoring high on questionnaire are
  selected for study

14
L.C. Lack of Control Group

• Pretest -------------- Treatment
  --------------------Posttest____
• test anxiety test anxiety test anxiety
• measure program measure__
• 90 70
• Did the test anxiety program reduce test
  anxiety? Cant tell!! Something else may have
  reduced test anxiety scores (see next
  slides)must have control group as shown below in
  order to determine if anxiety program had an
  effect.
• Pretest ----- Treatment ----- Posttest____
• Experimental 90 70
• Control 90 90

15
Threats to Internal Validity (continued)

• L.C., He Made The IRS MIDday
• 2. H ________________
• Meaningful events change people b/w repeated
  measures.
• An event may occur b/w pre post-test that is
  responsible for the results
• Ex Job attitudes --gt strike --gt job attitudes
• Ex announcing that grades will be replaced by
  Pass/Fail will reduce anxiety

16
Threats to Internal Validity (continued)

• L.C., He Made The IRS MIDday
• 3. M __________________
• changes in people during a study
• Ex
• Getting accustomed (1st vs. last semester in
  college)
• Bored, tired, fatigued (physical ability
  measures, long survey)
• If control group is used, the effects of history
  maturation can be explained
• Pretest ----Treatment---- Posttest____
• Experimental 90 70
• Control 90 90

17
Threats to Internal Validity (continued)

• L.C., He Made The IRS MIDday
• 4. T ___________________
• Practice effects people may score better on a
  measure, the more practice they have with it
• Ex SAT scores improve by simply taking it over
• 5. I ________________-
• The measurement itself changes from pre- to
  posttest
• Ex using different forms one may be easier
• Also refers to wear and tear or to judges
  abilities
• Ex computer breakdown

18
Threats to Internal Validity (continued)

• L.C., He Made The IRS MIDday
• 6. R _________________________
• People selected on the basis of test scores tend
  to have retest scores closer to the average
  (mean).
• If score 1 is an extreme score, then score 2 will
  be closer to the average of the large set of
  scores

19

• Pretest----Treatment--Posttest
• Exper. 90 70
• Control 90 80
• Although some regression is involved (90 vs.
  80), the treatment still had an effect beyond
  regression (90 vs. 70)

20
Threats to Internal Validity (continued)

• L.C., He Made The IRS MIDday
• 7. S ____________________
• groups are not equivalent to each other from the
  very start (Group composition unequal at start of
  study).
• Ex volunteers for effects of beer on driving.
  Effects of exercise with more athletes in one
  group.
• Ex participants choosing the two courses may be
  different (one gives multiple choice tests, the
  other gives only essay tests)

21
Threats to Internal Validity (continued)

• L.C., He Made The IRS MIDday
• 8. M ______________________
• People die/quit/drop out of the study
• Sometimes certain types of people tend to drop
  out creating a biased sample
• Ex smoking cessation, exercise
• _____________________
• one group responds to something other than the IV
• Ex male female attitudes on AIDS before and
  after Magic Johnson announcement (selection
  history).
• 10. D ________________
• people tell each other the treatment.
• Ex corporate training, visualization in sport
  website for class.

22
Exam Review