Lesson%20Seven

1
Lesson Seven

• Reliability

2
Contents

• Definition of reliability
• Indication of reliability Reliability
  coefficient
• Ways of obtaining reliability coefficient
• Alternate/Parallel forms
• Test, retest
• Split-half
• (Inter-) rater (or scorer) reliability
• Two ways of testing reliability
• How to make sure the test is reliable

3
Definition of Reliability

• The consistency of measures across different
  times, test forms, raters, and other
  characteristics of the measurement context
  (Bachman, 1990, p. 24).
• The accuracy or precision with which a test
  measures something consistency, dependability,
  or stability of test results.

4
Reliability coefficient (r)

• To quantify the reliability of a test ? allow us
  to compare the reliability of different tests.
• 0 r 1 (ideal r 1, which means the test gives
  precisely the same results for particular testees
  regardless of when it happened to be
  administered).
• If r 1 100 reliable
• A good achievement test rgt .90

5
Alternate/Parallel forms the most stringent form

• Two forms, two administrations
• Equivalent forms (i.e., different items testing
  the same topic) taken by the same test taker on
  different days
• If r is high, this test is said to have good
  reliability.

6
Test, retest

• One form, two administrations

• The same test is administered to the same testees
  with a short time lag, and then calculate r.
• Appropriate for highly speeded test

7
Split-half (Spearman-Brown Procedure)

• One test, one administration
• Split the test into halves (i.e., odd questions
  vs even questions) to form two sets of scores.
• Internal consistency (also KR-21, KR-20)

Q1 Q2 Q3 Q4 Q5 Q6
First Half
Second Half
8
(Inter-) rater (or scorer) reliability

• Needed for subjective tests (e.g., writing, oral
  tests) when two or more independent raters are
  involved in scoring.
• Raters should be trained before scoring.
• Compare the scores of the same testee given by
  different raters. If r high, theres
  inter-rater reliability.

9
Ways of testing reliability

• Examine the amount of variation
• Standard Error of Measurement (SEM)
• The smaller the better
• Calculate reliability coefficient
• r
• The bigger the better

10
How to make sure the test is reliablefor teachers

• Take enough samples of behavior
• Try to avoid ambiguous items
• Provide clear and explicit instructions
• Well layout
• Provide uniform and undistracted condition
• Try to use objective tests
• Try to use direct tests
• Have independent, trained raters
• Try to identify the test takers by number, not by
  names
• Try to have more multiple independent scoring in
  subjective tests (Hughes, 1989, pp. 36-41).