Reliability

1
(No Transcript)
2
Reliability

• Using instruments of unknown reliability is
  analogous to running an instrumental analysis
  without creating a working curve with known
  standards. You get numbers you might even be
  able to generate some pattern from low to high.
• Beyond that, you just have a lot of numbers that
  you wont know the meaning of.

3
Reliability

• Lets say you have a reliable analytical
  instrument. You run a sample that you know is 50
  ppb Mn. It reads 50 ppb. You run another split
  from that same sample. It reads 49 ppb. You have
  a good idea how consistent the measures are that
  the instrument is furnishing.

4
Reliability

• Lets say you have an unreliable analytical
  instrument. You run a sample that you know is 50
  ppb Mn. It reads 76 ppb. You run another split
  from that same sample. It reads 28 ppb. You have
  a good idea that the measures that the instrument
  is furnishing wont be able to resolve any
  differences smaller than about 50 ppb.

5
Reliability is Fundamental

• Spearman-Brown reliability(R) measure (Jacobs and
  Chase, 1992)
• derives from the correlation coefficient (r)
  obtained from individuals' scores on two halves
  of a single test
• and is applicable to tests, properly constructed
  student ratings instruments, and knowledge
  surveys.

6
Reliability from split halves

• Treats the instrument like an analytical
  instrument
• Treats a single student or faculty as a sample
• Treats a score on half the item bank and a score
  on the other half of the bank as two splits from
  the same sample.
• The line-fit for split halves of all students
  provides a working curve, which is related
  closely to reliability of your survey or test.

7
Use Excel to estimate reliability. Split test
into two halfs even items and odd items
8
Next, compute the average score from each half.
9
Use Excels stat package to calculate the Pearson
Correlation Coefficient for pairs of scores for
odds evens This is r (lower case)
From r calculate the Pearson-Brown Reliability
Coefficient This is R (upper case) R 2r/(1r)

(20.969)/(10.969)
R 0.98
10
Reliability of 40 pt Formative Survey
11
You might think that you would get this
12
"Hey! My tests and grades don't correlate with
student ratings. That means student ratings are
worthless--right?"

• Wrong.

13
Horror show 1

• Most faculty-made tests have horrid reliability.

14
On a really good test you will get this
15
But it is more likely youll get something like
this
Most faculty tests have reliabilities of 0.2 to
0.3.
16
Horror show 2

• About the only thing worse than faculty made
  tests are grades derived from scores on multiple
  faculty-made tests.

17
Grade reliability is even lower than test
reliability.
18
Why We Need Knowledge Surveys! Faculty place too
much faith in the unexamined reliability of
faculty-made Tests Note r-values from a class
below
To estimate grade reliability, use tests and
quizzes as split halves
19
Validity

• Measures what the instrument is supposed to
  measure

20
Classic Validity

• Multisection
• Multisource
• Multibias
• Experimenter controlled
• Conceptual structure

21
New Validation Framework

• Content aspects
• Relationship relevance of form to construct
• Substantive aspects
• Relationships of process to construct
• Structural aspects
• Relationships of items to construct
• External aspects
• Relationships of achievement to construct
• Generalizability aspects
• Relationships hold true in general for construct
• Consequential aspects
• Relationships of consequences of use to construct