ASSESSMENT LITERACY: ANTIDOTE TO PROFESSIONAL SUICIDE?

1
ASSESSMENT LITERACY ANTIDOTE TO PROFESSIONAL
SUICIDE?

• W. James Popham
• University of California, Los Angeles
• Ohio Innovative Learning Environments Bridging
  the Knowing-Doing Gap
• Hilliard, Ohio July 31, 2012

2
AN ANCIENT CHINESE CURSE

• May you live in an interesting time.

3
A WIDELY HELD VIEW

• American public schools are in much worse
  shape today than when I was a high school teacher
  in eastern Oregon.
• WHY?

4
ONE ANSWER (MINE)

• I believe the chief reason for todays
  public-school problems is a parade of profound
  mistakes made because of American educators
  pervasive assessment illiteracy.
• Lets briefly look at five of those mistakes!

5
MISTAKE NO.1

• We have often made educational decisions based
  on the supposed numerical accuracy of educational
  tests when, in fact, such tests, and the scores
  they yield, are far less precise than is usually
  thought.

6
MISTAKE NO. 2

• We almost never measure students attitudes,
  interests, and values when, in truth, those
  powerful and influential affective variables can
  be inexpensively assessed for groups of
  studentsnot individual studentsthen used to
  guide instruction.

7
MISTAKE NO. 3

• We have often set out to teach (and test)
  students mastery of far too many curricular
  aims, even though pursuing a plethora of
  curricular targets destines the assessment of
  those targets to be superficial and
  instructionally feckless.

8
HELP STAMP OUT NON-BEHAVIORAL OBJECTIVES!
HELP STAMP OUT NON-BEHAVIORAL OBJECTIVES

• HELP STAMP OUT SOME NON-BEHAVIORAL OBJECTIVES

9
MISTAKE NO. 4

• We currently spend oodles of dollars and
  substantial lumps of our classroom time on the
  use of interim assessments although there is
  essentially zero research evidence supporting the
  effectiveness of these assessment instruments.

10
AND THE MOST CATACLYSMIC OF ALL MISTAKE NO. 5

• We have allowed the success of American
  educators to be determined by educational tests
  that are altogether incapable of accomplishing
  this profoundly important measurement mission.

11
HOW CAN WE REMEDY THIS REALLY RAUNCHY SITUATION?

• We can take a gigantic step toward improving
  this simply by getting greater numbers of
  educators to become more knowledgeable about
  educational measurement, that is, to become more
  assessment literate.

12
WHAT IS ASSESSMENT LITERACY?

• Assessment literacy consists of an
  individuals understandings of the fundamental
  assessment concepts and procedures deemed likely
  to influence educational decisions.

13
THREE DEFINITIONAL EMPHASES

• Assessment literacy consists of an
  individuals understandings of the fundamental
  assessment concepts and procedures deemed likely
  to influence educational decisions.

14
WHAT ASSESSMENT LITERACY ISNT

• It is not a persons ability to carry out the
  abstruse, technical procedures associated with
  major operations in the field of educational
  measurement.
• It is not off-puttingly quantitative. That is,
  most of the significant understandings needed by
  an assessment-literate person arent
  mind-numbingly numerical.

15
Which of these would be possessed by an
assessment-literate person?

1. Knowledge about the measurement accuracy of
   standardized tests
2. The ability to perform a factor analysis to
   detect the variables a test measures
3. Comprehension of whats needed for a test to be
   truly diagnostic
4. Knowledge of effective lesson-design

16
AND NOW, A BIT OF PAIR-FARE

• TURN AND TALK
• BEND AND BANTER
• ROTATE AND RAMBLE
• (Alliteration always trumps clarity.)

17
PAIR-FARE DEFINITION DABBLING

• Please turn to one or two persons nearby, then
  take turns explaining to each other what is meant
  by the expression assessment literacy. Assume
  you are trotting out this definition for
  educator-colleagues, but still try to keep your
  definition easy to understand.

18
AND NOW, A PAIR-FARE SPECULATION

• Please turn to one or two persons nearby (the
  same ones or different ones, depending on how the
  previous pair-fare activity panned out), then
  collaboratively try to come up with at least two
  reasons why educators today need to be more
  assessment literate than educators of 25 years
  ago.

19
MY REASON 1 ASSESSMENT LITERACY IS NEEDED TODAY
BECAUSE

• We now possess sufficient research evidence to
  show that the formative-assessment process can
  lead to dramatic improvements in students
  learning. Educators need to understand what this
  assessment-rooted process is and how to get it
  used with students.

20
MY REASON 2 ASSESSMENT LITERACY IS NEEDED TODAY
BECAUSE

• Educators, collectively and individually, are
  now being evaluated chiefly by students scores
  on large-scale accountability assessments. But
  most of those assessments are totally
  inappropriate for this function. Educators must
  understand why.

21
HERES A DEFINITION (MINE) OF INSTRUCTIONAL
SENSITIVITY

• The degree to which students performances on
  a test accurately reflect the quality of
  instruction specifically provided to promote
  students mastery of what is being assessed.

22

• THE TRADITIONAL FUNCTION OF AMERICAS
  EDUCATIONAL TESTS, EVER SINCE WORLD WAR IIS ARMY
  ALPHA, HAS BEEN TO PERMIT COMPARATIVE SCORE-BASED
  INTERPRETATIONS FOR TEST-TAKERS.

23

• COMMITTTEE ON THE EXAMINATION OF RECRUITS,
  VINELAND (1917)

24
RECRUITS TAKING EXAMINATION AT CAMP LEE, 1917
25
WHY MIGHT A TEST ITEM BE INSTRUCTIONALLY
INSENSITIVE?

• Alignment Leniency
• Excessive Easiness
• Excessive Difficulty
• Confusion-Engendering Item Flaws
• Socioeconomic Status (SES) Links
• Academic Aptitude Links

26
SOCIOECONOMIC STATUS (SES) LINKS

• If an item gives a meaningful advantage to
  students from higher SES families, then the item
  will tend to measure what students bring to
  school rather than how well they are taught once
  they get there.

27
A 6th-Grade Science Item

• A plants fruit always contains seeds. Which of
  the items below is not a fruit?
• A. orange
• B. pumpkin
• C. apple
• D. celery

28
A 4th-Grade Reading Item
My fathers field is computer graphics.

• In which of the sentences below does the word
  field mean the same thing as in the sentence
  above?
• A. The shortstop knew how to field his
• position.
• B. We prepared the field by plowing it.
• C. What field do you plan to enter when you
• graduate?
• D. The nurse examined my field of vision.

29
ACADEMIC APTITUDE LINKS

• If an item gives a meaningful advantage to
  students who possess greater inherited
  quantitative, verbal, or spatial aptitudes, then
  the item will tend to measure what students bring
  to school rather than how well they are taught
  once they get there.

30
A 4th-Grade Mathematics Item

• Which of the letters below, when folded in half,
  will have two parts that match exactly?

F (A) Z (B)
S (C) B (D)
31
A 3rd-Grade Mathematics Item

• The secret number is inside the circle. It is
  also inside the square. It is NOT inside the
  triangle. Which of these is the secret number?
• A. 2 B. 3 C. 5 D. 7

3
5
2
4
1
7
6
32
A PAIR-FARE FORAY

• Please imagine that you are a teacher who is
  trying to explain the meaning of a tests
  instructional sensitivity to a layperson-member
  of your districts school board who has asked you
  to do so. Trying not to use language thats too
  technical, please take turns describing what
  instructional sensitivity is to a nearby
  colleague.

33
A FOLLOW-UP PAIR-FARE FORAY

• Recently, to secure federal waiver-relief from
  increasingly stringent NCLB demands, many states
  are installing teacher-evaluation programs in
  which students test scores play a prominent
  role. If those tests are instructionally
  insensitive, who is most harmed? Choose one (A)
  teachers, (B) students, (C) society. Discuss with
  a colleague.

34
ASSESSMENT LITERACY AN ANTIDOTE TO PROFESSIONAL
SUICIDE?

• Professional stupidity comes in many forms.
  Assessment literacy wont correct them all.
  However, given todays preoccupation with the
  role of students test scores,
  assessment-literate educators are apt to have
  substantially longer careers than their
  assessment-oafish colleagues.

35
JIMS YELLOW-BRICK ROAD TO ASSESSMENT LITERACY

• Read one or more of the following
• Mastering Assessment (2011) Pearson (15 brief
  booklets aimed at promoting measurement moxie)
• Everything School Leaders Need to Know About
  Assessment (2010) Corwin
• Classroom Assessment What Teachers Need to Know
  (2011) Pearson

36

• JUNGLE JIMS E-MAIL ADDRESS
• wpopham_at_ucla.edu