The Design and Use of Multiple Choice Tests

1
"The Design and Use of Multiple Choice Tests"
2
Use of Multiple Choice Exams

• Not limited to Factual Knowledge
• Making inferences
• Application of models
• Evaluation

3
Multiple Choice vs. Descriptive Examinations
Descriptive Questions Multiple Choice
Setting the exam Easy Difficult
Grading Task Difficult Easy
Grading Time Long Short
Grade Consistency Varies High
4
Structure of a Multiple Choice Question

• Stem
• Answers

5
Formulating GoodMultiple Choice Questions

• Wording the stem so that there is a single
  correct answer
• Finding plausible wrong answers
• Sensitivity to ESL students
• Negative voice in the stem
• Conditional if would structures
• Cultural contexts for questions

6
Stems

• Statements
• A river that flows through a desert is
  called..
• Questions
• Which of the following descriptions best fits
• the character of Alsatian wines?
• Negative voice
• Which of the following grapes is not used to
• make Amarone?

7
Stems

• Conditional Voice
• If you want a VQA Niagara red wine that
• was very fruity and required only a short
• ageing period you would select

8
Answer Options

• Single correct answer options
• Collective answer options
• (a) and (c) are both correct
• all of the above are correct
• none of the above are correct

9
Plausible Answer Strategies

• Good questions contain several plausible answer
  options
• Grey area between designing tricky questions
  and providing answer options that test knowledge
  thoroughly.
• Inadvertently giving away the answers

10
Adding Uncertainty..

• The term "fortified wine" means
• a) the bottle had been reinforced
• b) the wine has had tannins and acid
• added
• c) the wine has had sugar added
• d) the wine has had SO2 added
• e) none of the foregoing

11
Adding difficulty..

• Choose the best answer.
• Roothams, a well-known local manufacturer of
  brandied fruit preserves sells their product
  through gourmet food shops and at craft fairs
  throughout Canada. Their recipes use genuine
  brandy and all natural ingredients. They are
  able to sell these legally because
• (a) the alcohol used is produced by fermentation
• (b) the product is made with ethyl alcohol
• (c) the alcohol used is purchased from the LCBO
• (d) the alcohol concentration is less than 1.2
• (e) the label says made with genuine brandy

12
Making Multiple Choice Questions Interesting

• A gracious, but parsimonious hostess wants to
  receive guests in the private dining room of your
  restaurant with a glass of champagne just to
  tantalise the taste buds and set the mood for the
  evening.
• 1.You suggest that if the wine is being poured at
  the bar, she should consider serving a Canadian
  sparkling wine that is very similar to Champagne
  in quality but considerably less expensive. Such
  a wine would be a

13

• 2. The cheapest sparkling wine would be obvious
  to sophisticated guests because they would know
  that in the best sparkling wines
• 3. If the gracious hostess insists on having
  French Champagne and you are left to
  rationalise costs, the most cost effective
  solution is to serve Champagne from..

14

• 4. Occasionally, a patron asks you to do
  something that will distinguish their event from
  others. One client wants to obtain a large
  bottle of Champagne to celebrate a particular
  event. Which is the largest bottle of champagne
  on the list below?
• 5. If someone wanted to serve a "glass of bubbly"
  during a reception, which of the following would
  be the most inappropriate type of wine to select
  as the pre-prandial (before dinner) libation?

15
Evaluating Multiple Choice Questions

• Difficulty Factor R/N
• R of times the correct answer is
• selected
• N of people taking the exam

Too Easy
Too Difficult
0.5
1
0
16
Difficulty Factor

• Values greater than 0.8
• The question is too easy
• Stem wording gives away the answer
• Not enough plausible incorrect answer options
• Values less than 0.2
• Guessing effect can be a factor
• The question is defective
• Material wasnt covered
• Stem wording is unclear, confusing
• Answer options need work

17
Evaluating Multiple Choice Questions

• Discrimination Index
• Proportion of capable students that select an
  answer option proportion of weak students
  that select the answer option
• i.e. D Ftop quarter Fbottom quarter

Good Discrimination
Reverse Discrimination
0
1
- 1
18
Discrimination Index

• For the Correct Answer Option
• Values 0.5 give good discrimination
• lt 0.2 consider dropping
• negative drop the question
• For Incorrect Answer Options
• Values should be negative.

19
Teaching WithMultiple Choice Questions

• Prepare lectures incorporating all material
  covered in the questions to be used
• After each lecture, check coverage by reviewing
  questions
• Provide students with multiple choice questions
  for self-evaluation at the end of readings

20
Teaching WithMultiple Choice Examinations

• Use stem questions to teach additional or
  ancillary terms not covered in the course
• i.e. . For post-prandial (after dinner) drinks
  you want to stock a supply of fortified wines and
  brandy. The term "fortified wine" means.

21
Provide Students the Stem Questions Prior to the
Exam!

• Exam questions should cover all the course
  material being tested
• Provides more detail than a course outline
• Students focus their study effort on the topics
  they understand the least
• Increases learning
• Reduces stress

22

• .
• Adapted from a presentation by J.E. (Joe) Barth
• School of Hospitality and Tourism Management
• University of Guelph, Ontario CANADA