CAN INSTRUCTIONALLY INSENSITIVE ACCOUNTABILITY TESTS EVER EVALUATE EDUCATORS FAIRLY

1
CAN INSTRUCTIONALLY INSENSITIVE ACCOUNTABILITY
TESTS EVER EVALUATE EDUCATORS FAIRLY?

• W. James Popham
• University of California, Los Angeles
• Winter Conference
• Washington Educational Research Association and
  Office of Superintendent of Public Instruction,
  Seattle December 4,2008

2

• In nations where students scores on
  accountability tests play a pivotal role in the
  evaluation of schools, it is assumed that
  students performances on such tests accurately
  reflect instructional quality.

3

• But what if students scores on educational
  accountability tests did not accurately reflect
  instructional quality?

4
A DEFINITION OF INSTRUCTIONAL SENSITIVITY

• The degree to which students performances on a
  test accurately reflect the quality of
  instruction specifically provided to promote
  students mastery of what is being assessed.

5
(No Transcript)
6
WHY MIGHT A TEST ITEM BE INSTRUCTIONALLY
INSENSITIVE?

• Alignment Leniency
• Excessive Easiness
• Excessive Difficulty
• Confusion-Engendering Item Flaws
• Socioeconomic Status (SES) Links
• Academic Aptitude Links

7
ALIGNMENT LENIENCY

• Many items on accountability tests, when judged
  as to their alignment with the curricular aims
  they are supposed to be measuring, will be
  regarded as aligned with those aims (skills
  and/or knowledge) even if the items are only
  tangentially related to the curricular aim being
  assessed.

8
An Example of Lenient Alignment

• Item 23
• Using the bus schedule on the adjacent page, if
  your purpose was to determine the shortest time
  to reach Boston from Denver on a Monday, on which
  bus should you begin your journey?
• A. Bus 214
• B. Bus 197
• C. Bus 110
• D. Bus 202

9
Was the item aligned?

• If the curricular aim had been for students to be
  able to use appropriate functional texts such as
  train or bus schedules.
• If the curricular aim had been for students to be
  able to determine whether given functional texts
  would fulfill their purpose for using such texts.

10
EXCESSIVE EASINESS

• If an item is so easy that even completely
  untaught students would answer it correctly, then
  the item cant distinguish between well taught
  and poorly taught students.
• E.g., How many letters are there in the word
  seven?

11
EXCESSIVE DIFFICULTY

• If an item is so difficult that even marvelously
  instructed students might not answer it
  correctly, then the item cant distinguish
  between well taught and poorly taught students.
• E.g., Without using your computer, what is the
  square root of 1,522,756?

12
ITEM FLAWS

• Items embodying serious deficits (e.g.,
  ambiguities, garbled syntax, more than one
  correct answer, or no correct answer) will
  prevent well taught students from answering the
  item correctly, hence make it impossible for the
  item to accurately distinguish between
  effectively and ineffectively taught students.

13
SOCIOECONOMIC STATUS (SES) LINKS

• If an item gives a meaningful advantage to
  students from higher SES families, then the item
  will tend to measure what students bring to
  school rather than how well they are taught once
  they get there.

14
A 6th-Grade Science Item

• A plants fruit always contains seeds. Which of
  the items below is not a fruit?
• A. orange
• B. pumpkin
• C. apple
• D. celery

15
A 4th-Grade Reading Item
My fathers field is computer graphics.

• In which of the sentences below does the word
  field mean the same thing as in the sentence
  above?
• A. The shortstop knew how to field his
• position.
• B. We prepared the field by plowing it.
• C. What field do you plan to enter when you
• graduate?
• D. The nurse examined my field of vision.

16
ACADEMIC APTITUDE LINKS

• If an item gives a meaningful advantage to
  students who possess greater inherited
  quantitative, verbal, or spatial aptitudes, then
  the item will tend to measure what students bring
  to school rather than how well they are taught
  once they get there.

17
A 6th-Grade Social Studies Item

• If someone really wants to conserve resources,
  one good way to do so is to
• A. leave lights on even if they are not
• needed.
• B. wash small loads instead of large loads
• in a clothes-washing machine.
• C. write on both sides of a piece of paper.
• D. place used newspapers in the garbage.

18
A 3rd-Grade Mathematics Item

• The secret number is inside the circle. It is
  also inside the square. It is NOT inside the
  triangle. Which of these is the secret number?
• A. 2 B. 3 C. 5 D. 7

3
5
2
4
1
7
6
19
A 4th-Grade Mathematics Item

• Which of the letters below, when folded in half,
  will have two parts that match exactly?

20
WHY MIGHT A TEST ITEM BE INSTRUCTIONALLY
INSENSITIVE?

• Alignment Leniency
• Excessive Easiness
• Excessive Difficulty
• Confusion-Engendering Item Flaws
• Socioeconomic Status (SES) Links
• Academic Aptitude Links

21
A LESSON TO BE LEARNED

• When the measurement community became convinced
  that assessment bias in our high-stakes tests was
  threatening validity, we set out to (1) detect
  assessment bias and (2) reduce it. We were
  successful. We can be equally successful in
  coping with instructional insensitivity.

22
TWO STRATEGIES FOR DETERMINING INSTRUCTIONAL
SENSITIVITY

• A Judgmental Strategy whereby seasoned, well
  trained educators supply item-by-item ratings
  using a rigorous item-evaluation rubric
• An Empirical Strategy contrasting per-item
  performances of (1) taught versus untaught
  students or (2) effectively taught versus
  ineffectively taught students

23
JUDGMENTAL DETERMINATION OF AN ITEMS
INSTRUCTIONAL SENSITIVITY

• An Illustrative Review Question
• If a teacher has provided reasonably
  effective instruction related to the objective
  measured by this item, is it likely a substantial
  majority of the teachers students will respond
  correctly to the item?

24
EMPIRICAL DETERMINATION OF AN ITEMS
INSTRUCTIONAL SENSITIVITY

• Contrasting per-item performances of taught
  versus untaught students
• Contrasting per-item performances of effectively
  taught versus ineffectively taught students

25
INSTRUCTIONAL INSENSITIVITY UNFAIR AND HARMFUL

• When we allow educators quality to be determined
  on the basis of accountability tests incapable of
  performing that task, we are being profoundly
  unfair to those educators.
• Far worse, some wrongly evaluated,
  desperation-driven educators will engage in
  classroom practices that are educationally
  harmful to children.

26

• Presenters e-mail address
• wpopham_at_ucla.edu
• Reactions or suggestions regarding this topic
  will be welcomed.