Introduction to Reliability and Validity

1
Introduction to Reliability and Validity
2
A Few Preliminary Definitions

• Test Variance represents variability on how
  people perform on a test

(a)
(b)
random measurement error
test variance (all of it)

• systematic
• test variance

3
More Definitions

• Systematic Variance essence is repeatability
• can be measured again (at least immediately)
• Random Variance reflects unrepeatable influences
  on test performance
• cannot be measured again
• cannot correlate with anything else
• is called random measurement error

4
Reliability

• Reliability is the proportion of a tests
  variance that is systematic variance
• If all variance is systematic, reliability 1
• If all variance is not systematic, reliability
  0
• Reliability .60 means that 60 of the variance
  is systematic remaining 40 is random
  measurement error
• Reliability is an index of whether test scores
  are reasonably free of random measurement error.

5
Another Kind of Error

• Systematic measurement error repeatable test
  variance that taps the wrong stuff

(c)
random measurement error

• non-error
• test variance

systematic measurement error
6
Validity

• Both random errors of measurement (unrepeatable
  stuff) and systematic errors of measurement
  (wrong stuff) hamper the usefulness of inferences
  that might be drawn from test scores
• Both limit the validity of a test
• Validity refers to whether a test can be shown to
  measure what it is supposed to measure (and not
  other stuff instead)

7
Subdividing Systematic Variance

• Non-error test variance may have several
  components systematic measurement error may also
  have several components

(d)
random measurement error
Three components of non-error test variance
two sources of systematic measurement error
8
Dimensionality of a Test

• Can investigate the number of components of
  systematic variance in a test by using factor
  analysis
• Number of distinguishable components underlying
  systematic test variance have to do with the
  dimensionality of the test
• Unidimensional versus multidimensional tests
• Dimensionality of test may change for different
  populations

9
Random Measurement Error
(e)
random measurement error
Three components of non-error test variance
two sources of systematic measurement error
10
Reliability Sets Maximum Correlation

• The reliability of a test sets an upper limit on
  the tests correlation with any other measure

Maximum test x can correlate with another measure

Reliability of test x
11
Example

• For a test with a reliability of .80, 80 of the
  variance is systematic, and therefore,
  predictable
• Thus, the maximum possible correlation of this
  test with another variable is

.80
.89
12
Correlation of Two Tests

• The correlation of two tests is limited by both
  of their reliabilities
• The maximum correlation they could attain is the
  product of the square roots of the respective
  reliabilities
• For example, for two tests with reliabilities of
  .6, the estimate of their maximum possible
  correlation would be

.60
.60
.60

13
Reliability of Alternate Forms of a Test

• Systematic variance would represent the same
  stuff
• Maximum correlation is equal to the product of
  the square roots of the respective reliabilities
• Two equally good tests their reliabilities are
  equal
• The square root of a value times the square root
  of the same value equals that value so it
  follows that the correlation between Form A and
  Form B is the estimate of either Forms
  reliability
• Thus, if we want to know the reliability of
  alternate forms of a test, we simply compute
  their correlation

14
Reliability and correlation

• rxx reliability of test x
• rxy correlation of test x with test y