Multiple Choice Item Construction

1
Multiple Choice Item Construction

• D. Michele Jacobsen
• Assistant Professor
• dmjacobs_at_acs.ucalgary.ca

2
Test Development Process

• 1. Statement of Goals
• 2. Content Outline
• 3. Table of Specifications
• 4. Item Selection
• 5. Item Construction
• 6. Composition of Instructions
• 7. Development of Answer Sheets
• 8. Construction of Answer Keys
• 9. Test Administration
• 10. Test Revision

3
Todays Focus

• 3. Table of Specifications
• 4. Item Selection
• 5. Item Construction

4
Two Important Characteristics of Tests

• Reliability
• consistency
• free from extraneous sources of error
• Validity
• how well a test measures what it is supposed to
  measure

5
Formative vs. Summative Tests

• Formative
• monitor progress toward goals within a course of
  study
• Summative
• assess overall achievement of course goals

6
A. Table of Specifications

• Blue print for test
• Purpose
• ensure proper emphasis given to all elements of a
  course of study
• content validity
• Guide for writing items

7
Table of Specifications (2)

• Start with Instructional Objectives

• Biology 30 Circulation Unit Test

1. Identifies parts of circulation system. 2.
Explains function of parts in relation to whole
system. 3. Distinguishes between circulation
system and other bodily systems.
8
Biology 30 Circulation
9
Need to Consider and Decide

• Length of Test
• Weight to be given to each objective
• Weight to be given to each level of taxonomy
• Estimate number of items in each cell

10
Biology 30 Circulation
11
B. Item Selection

• Types of Items
• Objectively Scored (Selection)
• true/false
• completion
• matching
• multiple choice
• Subjectively Scored (Supply)
• interpretive exercises
• essay

12
C. Item Construction

• Selected Response Test Items
• Item construction skills for valid and reliable
  measures of student achievement.
• Guidelines which apply to all types
• Specific Suggestions for writing each type
• Advantages and Disadvantages of Each

13
Guidelines for Writing Objective Items

• 1. Construct at appropriate level of
  difficulty for examinees
• 2. Include Items at appropriate level of
  difficulty for purpose of test.
• 3. Test significant elements of a course.
• 4. Write independent items.

14
II. Guidelines for Writing Objective Items

• 5. Construct questions free from extraneous
  reasons for problems.
• 6. Communicate the question in clear, concise
  language.
• 7. In the correct alternatives, paraphrase
  statements from the text.
• 8. Exclude clues to correct answer.

15
III. Guidelines for Writing Objective Items

• 9. Provide one correct answer.
• 10. Edit the Items.

16
Alternate Response Items

• Involves the selection of one of two alternatives
• true / false
• yes / no
• right / wrong
• fact / opinion
• Mainly for Knowledge Comprehension
• Can be written at higher levels

17
True / False

• Word statements clearly. Vague or ambiguous
  wording will confuse students.
• Avoid overgeneralizing.
• Poor Heavy smoking causes lung cancer. T F
• Better Heavy smoking often causes lung cancer. T
  F
• Avoid Trick questions.
• i.e., General Wolseley led Canadian troops to
  Manitoba in 1870.
• 3 tricks Colonel, British, Not yet a province
• Do not use trivial statements to pad out the
  number of questions and marks to arrive at a
  predetermined level.

18
True / False

• Statements should be entirely true, or entirely
  false
• Unacceptable
• In King Lear, Regan ordered Gloucesters eyes to
  be plucked out and Gloucester died when he jumped
  off the cliff of Dover.
• Acceptable
• In King Lear, Regan ordered Gloucesters eyes to
  be plucked out. (T)
• In King Lear, Gloucester died when he jumped off
  the cliff at Dover. (F)

19
True / False

• Avoid using universal descriptors such as
  never, none, always, and all.
• Testwise students will recognize that there are
  few absolutes.
• Avoid negative words, as they are often
  overlooked by students.
• Poor It was not unheard of for Henry VIII to
  close monasteries in England. T F
• Better Henry VIII closed some monasteries in
  England. (T)

20
True / False

• Do not include two ideas in one statement unless
  you are evaluating students understanding of
  cause and effect relationships.
• Poor Porpoises are able to communicate because
  they are mammals. T F
• Better Porpoises are mammals. T F Porpoises
  are able to communicate. T F
• Provide a T and F beside each statement and
  ask students to circle correct answer.
• Avoids problem of students writing illegible
  letters.

21
True / False Variations

• Include more false than true statments in any
  given test and vary the number of false
  statements from test to test.
• tendency to mark more statements true than false.
• discrimination between those who know the content
  and those who do not is greater for false
  expressions.
• Avoid using negative statements.
• Under the demands of the testing situation,
  students may fail to see the negative qualifier.

22
Matching Items

• Consist of
• a column of premises
• a column of responses
• directions for matching the two.
• Similar to multiple choice, but easier and more
  efficient to construct
• Can be written to assess Knowledge,
  Comprehension, Application, Analysis level
  behaviors

23
Guidelines for Writing Matching Items

• Provide clear instructions on how to indicate the
  correct answers.
• Indicate whether the same response can be used
  more than once.
• Maintain grammatical consistency within and
  between columns.
• within a column either sentence or point form
• between columns one or the other
• Ensure that any matching question appears
  entirely on one page.

24
2. Guidelines for Writing Matching Items

• Provide an unequal number of premises and
  responses
• reduces guessing and elimination
• increases measure of comprehension
• Avoid designing questions which require students
  to draw lines between premise and response.
• confusing for student and marker
• provide space for letter or number answers

25
3. Guidelines for Writing Matching Items

• Make sure lists are homogeneous.
• i.e., do not include items testing names, dates,
  and events.
• Instead, make every response plausible
• Make the wording of the premises longer than the
  wording of the responses.
• Identify the items in one list with numbers and
  those in the second list with letters.

26
Example

• Directions 1. On the line to the right of each
  phrase in Column I, write the letter for the
  word in Column II that best matches the
  phrase.
• 2. Each word in Column II may be used once,
  more than once, or not at all.

Column I
Column II
1. Name of the answer in addition problems. 2.
Name of the answer in subtraction problems. 3.
Name of the answer in multiplication problems. 4.
Name of the answer in division problems.
A. Difference B. Dividend C. Multiplicand D.
Product E. Quotient F. Subtrahend G. Sum
27
Short Answer Test Items

• Typically, the student is asked to reply with a
  word, phrase, name, or sentence, rather than a
  more extended response.
• Direct Questions / Short Answer
• Who is the current Prime Minister of Canada?
• Incomplete Sentences / Fill In the Blanks
• The current Prime Minister of Canada is _____?
• Items are fairly easy to construct and mark
• Assess mainly knowledge, comprehension, and some
  application.

28
Guidelines for Writing Short Answer Items

• Questions must be carefully worded so that all
  students understand the specific nature of the
  question asked and the answer required.
• Poor Wellington defeated Napoleon in _____
  ?
• Better In what battle fought in 1815 did
  Wellington defeat Napoleon? OR In what
  year did Wellington defeat Napoleon at
  Waterloo?

29
II. Guidelines for Writing Short Answer Items

• Word completion or fillin questions so that
  missing information is at, or near the end of,
  the sentence. Makes reading and responding
  easier.
• Poor In the year ______ , Canada turned 100
  years old.
• Better Canada turned 100 years old in the year
  _____ .
• Instructions and teachers expectations about
  filling in blanks should be made clear. Indicate
  whether each blank of equal length represents one
  word or several words, whether long blanks
  require sentences or phrases, and whether
  synonymous terms are accepted.

30
III. Guidelines for Writing Short Answer Items

• When an answer is to be expressed in numerical
  units, the unit should be stated.
• Poor If a room measures 7 metres by 4 metres,
  the perimeter is _____ .
• Better If a room measures 7 metres by 4 metres,
  the perimeter is _____ metres (or m).
• Do not use too many blanks in completion items.
  The emphasis should be on knowledge and
  comprehension, not mind reading!
• Consider In the year _____ , Prime Minister
  _________ signed the __________ , which led to a
  ___________ which was __________ .

31
Multiple Choice ItemsInterpretive Exercise

• Guidelines for Writing
• Advantages Disadvantages

32
Terminology Multiple Choice
1. The capital city of Canada is a.
Vancouver b. Montreal c. Toronto d. Ottawa

• 1. Stem presents the problem
• 2. Keyed Response correct or best answer
• 3. Distracters appear to be reasonable answers
  to the examinee who does not know the content
• 4. Options include the distracters and the keyed
  response.

33
Ambiguity

• Extrinsic
• Desirable quality in multiple choice items
• Outside the item
• Allows discrimination between those who know
  material and those who do not.

• Intrinsic
• Undesirable
• Should be avoided
• Inside the item
• poor wording
• more than one answer
• Even those who know content have difficulty
  choosing correct answer

34
Types of Multiple Choice Items

• Correct Answer
• Only one correct response
• Best Answer
• requires examinee to select alternative closest
  to being correct
• fine distinctions
• Multiple Answer
• More than one correct or best answer

35
Interpretive Exercise

• Usually begins with verbal, tabular or graphic
  information which is the basis for 1 or more
  multiple choice questions.
• map, passage from a story, a poem, a cartoon
• Can challenge students at various levels of
  understanding
• application, analysis, synthesis, evaluation
• Exercise contains all information needed to
  answer questions
• Readily adaptive to the more important outcomes
  of disciplines.

36
Interpretive Exercises (cont)

• Examples
• If student answers incorrectly it is because they
  have not mastered the thinking or reasoning
  required by the question, NOT because they failed
  to memorize background information.
• Math questions give students the formulas, test
  ability to apply concepts, rather than ability to
  memorize formulas.

37
Guidelines for Writing Multiple Choice Items

• State stem in the form of a question.
• Weak
• Canada is
• a) a country
• b) where you live
• c) between the Pacific and Atlantic oceans
• d) all of the above
• Better
• Between which two oceans is Canada located?
• a) Atlantic and Indian
• b) Atlantic and Pacific
• c) Pacific and Indian

38
Guidelines for Writing Stems

• Place most of the subject matter in the Stem
• ensures full statement of problem
• Eliminate extraneous material from the Stem
• goal is to measure student achievement, not to
  present new material
• maximize use of time for demonstrating
  understanding, not reading ability
• Avoid Negatively phrased Stems
• students may miss the qualifier
• use only when learning outcome requires this type
  of differentiation

39
I. Guidelines for Writing MC

• Ensure similarity among alternatives with regard
  to
• grammatical structure
• length
• mode of expression
• Grammatical errors provide unintentional clues to
  the answer
• When in doubt, students will select the longest
  alternative as the correct answer

40
Example Length of Alternatives

• Neurotics are more likely than psychotics to
• a. be dangerous to society
• b. have delusional symptoms
• c. be dangerous to themselves
• d. have insight into their own inappropriate
  behavior but nevertheless feel rather helpless
  in terms of dealing with their difficulties

41
II. Guidelines for Writing MC

• Make one of the alternatives the most clearly
  correct or best answer
• exception multiple answer form
• reduces intrinsic ambiguity
• reduces frustration during test
• Make distracters plausible
• desire to attract students who really do NOT know
  the answer to the question
• create distracters from elements of the correct
  response
• improves reliability of item

42
Example Undemanding Distracters

• Mickey Mouses two nephews are
• a) Huey, Dewey and Louie
• b) Clarabelle Cow
• c) Morty and Ferdy
• d) Abbott and Costello
• The Role of Humour?

43
III. Guidelines for Writing MC

• Avoid parallel language between the Stem and the
  Correct Response
• gives clues to keyed response
• emphasizes testwiseness, not knowledge
• Randomly distribute answers across the
  alternative positions
• inexperienced test writers emphasize b and c
  alternatives (hide the answer!!)
• do NOT use an interpretable order of keyed
  responses

44
IV. Guidelines for Writing MC

• Use qualifiers such as all of the above and
  none of the above sparingly
• testwise students will use process of elimination
  to select answer
• do NOT use to pad out the distracters because
  you cannot think of another one.

45
Advantages of Multiple Choice Items

• allow more adequate sampling of content.
• tend to more effectively structure the problem to
  be addressed
• items can be more efficiently and reliably scored
  than supply items
• different response alternatives can provide
  diagnostic feedback (item analysis)
• items can be constructed to address various
  levels of cognitive complexity

46
Disadvantages of Mult.Choice Items

• difficult time consuming to construct good
  items
• leads to emphasis on other selected response item
  types
• can lead the instructor to favour simple recall
  of facts
• high degree of dependence on students reading
  and instructors writing ability
• can be difficult to achieve clarity of expression
• measuring synthesis and evaluation can be
  difficult
• inappropriate for measuring outcomes that require
  skilled performance

47
Wrap-Up