Current Research on Constructing Objective Assessments

1
Current Research on Constructing Objective
Assessments

• Presented November 1, 2006
• Liz Freeman

2

• How long has it been
• since you have thought about
• the way you write tests?

3
Definitions
psycho
4
Definitions
metrics
5
Definitions
psychometrics
The branch of psychology that deals with the
design, administration and interpretation of
quantitative tests that measure aptitude,
personality traits or intelligence. Also called
psychometry.
6
Why do we assess our students?

• Think about this from
• both sides of the desk
• --as a student
• --as a teacher

7
Why do we assess our students?

• What gets measured gets done
• External influences (NCLB)
• Internal influences
• Determine student growth and increased knowledge
• Self-evaluate teaching strategies
• Personal validation of the differences we make!

8
What questions might tests answer for us as
teachers?

• Did I take long enough to teach
• the topic well?
• Was the instructional approach I used a good one?
• Are there any gaps that I need to address before
  moving on?
• What did the students learn?
• Was my teaching effective?
• How can I change what I did to improve student
  achievement?

9
Your test should reflect

• what you teach in the classroom and how much
  importance it has.
• local, state or national standards.
• goals addressed in the appropriate course
  Algebra I, Algebra II.

10
Other modes of assessment NOT discussed today

• Portfolios and authentic assessment
• Performance based i.e. band audition
• Exhibitions
• Rubric based evaluations
• Observations
• Interviews
• Student conferences
• Self and peer assessments

11
What are some common types of objective questions?
12
Types of Objective Questions Commonly Asked of
Students

• Multiple-choice
• True/False
• Completion and Short answer
• Matching
• Essays with highly specific expectations

13
Some concepts that should be common amongst
question type

• Start with the curriculum guide to determine
  depth and breadth of content.
• The test should be a representative sample of the
  course
• (amount of time devoted to topic of
  questions)
• Students can demonstrate varied skills
  knowledge, comprehension, application,
  evaluation, etc.

14

• Each possible answer should be listed on
  separate lines underneath one another.
• A. 12 B. 15 A. 12 C. 20
• C. 20 D. 24 B. 15 D. 24
• A. 12
• B. 15
• C. 20
• D. 24

15

• Space should be between the questions and the
  possible answers.
• Correct grammar and spelling is important to the
  test taker.
• Typed tests are generally
• easier to read.

16

• Letter choices should be consistent within the
  test.
• Dont try to outsmart the students or trick them
  of course you can! We want to know what they
  know.
• The whole item should be

17

• on one page.

18

• What are the advantages to objective tests?

19
Advantages to Objective Test Items

• No bluffing
• Writing and spelling dont get in the way
• Scoring is
• unbiased
• fast
• Feedback is clear
• Answers can be analyzed
• with statistics

20

• What are the disadvantages to objective tests?

21
Disadvantages of Objective Test Items

• Usually limited to factual recall
• Does not allow the students to practice writing
  skills
• Cheating/leaking of information can occur
• Guessing is promoted
• Does not show thinking
• process (math other multi-step processes).

22
Question arrangement on a test

• Even though it is fun, avoid unrelated cartoons
  visuals
• as they may confuse and
• distract some students
• Pattern your test
• (ie chronologically in history, one work by
  one author and then a second work by the same
  author)

23
Question arrangement on a test

• Group similar types of questions
• --true false with other true false
• --matching with other matching
• Order the questions
• by difficulty

24
Multiple Choice Questions
25
Common multiple choice vocabulary to know

• Stem The leading part of the multiple choice
  question
• Alternatives all possible answers
• Distracters options that are incorrect
• Key the correct answer

26
Alternatives

• Should be as brief and simple as possible to
  reduce the reading load
• (this can help our IEP and ELL students)
• Should all be topical and possible
• Should be in ascending or descending order
  order by length if nothing else. Research
  indicates that items are easier if this rule is
  violated (Haladyna Downing, 1989).

27
Alternatives

• Eliminate all of the above
• Judiciously use none of the above
• Keep the same number of alternatives so that
  there is consistency within the test

28
Alternatives

• Write distracters that are common errors students
  make.
• positive integer instead of
• negative integer
• Write distracters that are attractive and
  plausible.

29
Alternatives
You (The Teacher)
The Attractive Distracter (part of the test
question)
Your Students
30
Keys

• should clearly be the best answer
• use dice to determine where they will be placed
  if you are not able to order them sequentially

31
True/False Yes/No Questions
32
True/False Yes/No Positives

• measure facts
• work well for definitions, dates, names
• generally good for struggling readers
• (IEPs and ELLs)

33
True/False Yes/No Negatives

• encourages memorization instead of comprehension
• coin-toss mentality
• can be trivial

34

• Avoid specific determiners such as always, never,
  only
• Be certain that you are basing the question on a
  statement that is absolute.
• Write statements in your own words try not to
  use quotes from the text.

35

• If you are using negative statements bold,
  underline, italicize, (or all three) the
  negative.
• Statements of opinion should be attributed to
  some source (Cornell University).
• You should be able to write the opposite question
  as well.

36
Matching Questions
37
Matching Questions

• Indicate if the choices can be used more than
  once
• Use homogeneous items all dates, all places
• Give them a context by labeling the columns

38
Matching Questions

• Place shorter phrases on the right.
• Order the choices in a logical fashion
  (numerically, chronologically, etc).
• Ideal between 4 and 10 choices with remainders

39
Completion and Short Answer
40
Completion and Short Answer

• The first president of the United States was
  ___________.
• Who was the first president of the United States?
  ____________

Completion Question
Short Answer
41
Positives and negatives of fill in the blank
prompts

• No guessing
• See where the students go wrong
• Easy to develop
• Hard to read for kids and teachers
• Must be graded by hand
• Item analysis will be completed by hand

42
Completion and Short Answer Questions

• Should specify in advance if spelling is to be
  graded.
• Blanks should be near or at the end of the
  statement.
• Preferably there should be only one blank per
  item or, at best, a related
• series.
• Italy primarily exports__ and ___.

43

• All blanks should be the same length.
• Try not to give clues such as a or an or
  singular/plurals.
• When an answer is to be in specific units be sure
  to indicate what you want.
• When was the French Revolution? _____ (a long
  time ago!)

44
Essay Questions
45
Essay Questions

• Define the task
• Set limits words, length of time, etc.
• Provide context, purpose, audience, type of
  writing

46
Essay Questions

• Try to NOT provide a choice
• If you are testing writing process then students
  should have essay prompt choices.

47
Essay Questions

• Give information (series of documents DBQ
  document based questions) to analyze and
  synthesize.
• Let students see grading criteria this gives
  them a focus.

48
How to increase objectivity when grading essay
exams

• Remove names
• Read same question to same question
• Read all and then grade

49
What can make these test questions better?
50

• Rabbits always have soft, short fur.
• A. True
• B. False

51

• An ___________ is a long sea creature that does
  not see well.
• A. Squid
• B. Porpoise
• C. Eel
• D. All of the above
• E. None of the above

52

• Match the author with the titles of the stories.
  You will use each answer once and only once.
• Name Author
• ___ Tale of Two Cities A. The Raven
• ___ The Color Purple B. Alice Walker
• ___ Red Bage of Couradge C. Charles Dickinson
• ___ The Grapes of Wrath D. Stephan Crane
• ___ Edgar Allen Poe F. John Steinbeck

53

• ____________ signed the Declaration of
  Independence.
• _______, ________ and _______ were imports.
• The best spreadsheet software is ________.

54

• Which is the symbol for iron?
• A. Fi B. Fo
• C. Fe D. Fa E. none

55
Choose one of the following topics to write about.

• Write a five-paragraph, persuasive essay in
  favor of or against school uniforms.
• OR
• Write a five-paragraph story about your most
  recent vacation.
• OR
• Make up a story about an alien who lands in
  Rochelle. What would they see and where would
  they go?

56
How might you modify your departments common
semester exam?
57
Think-Pair-ShareDeveloped by Frank Lyman (1981)
Try this technique sometime with your
students! STEP 1 Think about the
question STEP 2 Find a partner or two.
Today, this is who you are sitting with. STEP
3 Everyone share one or two answers to the
question. (everyone)
How might you modify your departments common
semester exam?
58
References

• Cornell University. Center for teaching and
  learning test construction manual.
  http//www.clt.cornell.edu/campus/teach/faculty/Ma
  terials/TestConstructionManual.pdf Retrieved
  October 18, 2006.
• Haladyna, T. M. Downing, S.M. (1989). Validity
  of a taxonomy of multiple-choice item-writing
  rules. Applied Measurement in Education, 2(1),
  51-78.
• http//www.utexas.edu/research/eureka/faculty/vie
  w.php?pid706 Retrieved October 18, 2006.
• Lyman, F. (1981). "The responsive classroom
  discussion." In Anderson, A. S. (Ed.),
  Mainstreaming Digest, College Park, MD
  University of Maryland College of Education.
• Knoll, M. K. (2002). Administrators guide to
  student achievement higher test scores.
• Popham, W. J. (2003). Test better, teach better
  The Instructional role of assessment.