Reliability

1
Reliability

• PSY 614
• Fall 2007
• Instructor Emily Bullock, Ph.D.

2
Reliability Outline

• Sources of Error
• Types of Reliability
• Factors effecting reliability
• Nature of the Test

3
Sources of Error

• Observed Score True Score Error
• Test Construction-item or content sampling
• Test Administration-test environment, test taker
  variables, examiner related variables
• Test Scoring and Interpretation

4
Types of Reliability

• Test Retest
• Parallel-Forms and Alternate-Forms
• Internal Consistency
• Inter-Rater

5
Types of Reliability-Test Retest

• Test-Retest Reliability estimate of reliability
  obtained by correlating pairs of scores from the
  same people on two different administrations of
  the same test
• Only appropriately used to measure reliability of
  stable constructs (ex. personality)
• Consider intervening variables due to time
• Interval time must be reported with test-retest

6
Types of Reliability-Parallel Forms and
Alternate Forms

• True Parallel form indicates that each form of
  the test produces equivalent means and variances
  of test scores
• Alternate forms does not meet the above standard
  but attempts at two versions of a test that have
  been constructed to be equivalent with respect to
  content and difficulty
• Often termed the coefficient of equivalence

7
Types of Reliability-Internal Consistency

• Split-Half Reliability obtained by correlating
  two pairs of scores obtained from equivalent
  halves of a single test administered once
• Steps of Split-Half
• Administer the entire test to participants
• Divide the test item scores into equivalent
  halves
• Compute a correlation (Pearsons r) between the
  two halves
• Adjust the half-test reliability using
  Spearman-Brown formula
• How do you split a test?

8
Types of Reliability-Internal Consistency

• Inter-item consistency degree of correlation
  among all the items on a scale
• Only appropriate for homogeneous scales or tests

9
Types of Reliability-Inter-Rater

• Inter-Rater or Inter-scorer reliability degree
  of agreement or consistency that exists between
  two or more scorers

10
Purpose of the Reliability Coefficient
11
Factors Effecting Reliability

• Some legitimate factors contribute to variability
  of test scores (see Murphy Davidshofer p. 118)
• Characteristics of the people taking the test
• What is your group of test takers like?
• Characteristics of the test itself
• Is the test worth making longer?
• The intended uses of the test scores
• Are your differences in test takers big or small?
• The method used to estimate reliability
• How long do you plan to make decisions based on
  this test?

12
What is good reliability?

• Commonly used standard
• .75 standard for decision-making tests
• .70 standard for research instruments
• Charter (2003) Article
• Less than .70 unacceptable
• .70-.80 fair
• .80 to .90 good
• Greater than .90 excellent
• Text
• .80 high reliability
• .70 moderate reliability
• .60 unacceptable reliability
• This information is most applicable to internal
  consistency reliability

13
The Nature of the Test

• Homogeneous or Heterogeneous test items
• Characteristics, ability, or trait being measured
  is presume to be dynamic or static
• Range of test scores is or is not restricted
• Test is a speed or a power test
• Test is or is not criterion-referenced

14
Using Reliability and Error in Interpretation

• Standard Error of Measurement
• an index of the extent to which one individuals
  scores vary over tests presumed to be parallel
• Indicates how much variability can be expected in
  test scores as a result of measurement error
• Can be used to form confidence intervals or
  bandwidths around scores