Constructing Multiple Choice Exams

1
Constructing Multiple Choice Exams

• Abu Dhabi Campus
• January 7, 2007

Kevin Schoepp- CTL
2
Multiple Choice Exams
3
KWL
Constructing Multiple Choice Exams
Think about Strengths and Weaknesses
4
Learning Outcomes

• Learners will
• Identify the components of a MCQ
• Explain the strengths and weaknesses inherent in
  MCQS
• Justify creation decisions of a MCQ
• Construct sound MCQs

5
Samples

• MC exam analysis a) is a time consuming process
  b) eliminating bad questions c) betterexams d)
  all of the above
• Which of the following is the best explanation as
  to why one should analyze MC exams?
• To eliminate bad questions
• To identify poor distracters
• To create better exams

6
A Few Types of MCQs

• Alternate choice
• True/False
• Matching
• Context Dependent
• Conventional

7
Alternate Choice

• Which of the following would most effectively
  burn fat?
• Aerobic exercise
• Anaerobic exercise

8
True/False

• The capital of Alberta is Calgary.
• True
• False

9
Matching

• Match each exercise on the left with the body
  part on the right.

• Incline flys
• Concentration curls
• Military Press
• Squat

• Rear deltoids
• Upper Pecs
• Biceps
• Lats
• Quads

10
Context Dependent
http//depts.washington.edu/currmang/Toolsforteach
ing/MCItemWritingGuidelinesJAME.pdf
11
Conventional MCQ

• Which of the following most clearly defines the
  process of pollination?
• The joining of egg and sperm cells
• The transfer of pollen grains to the pistil.
• Food is broken down and energy is released.

http//depts.washington.edu/currmang/Toolsforteach
ing/MCItemWritingGuidelinesJAME.pdf
12
2 Types of Conventional MCQs

• Question
• What is the most common type of assessment at ZU?

• Statement
• The most common type of assessment at ZU is the

13
Construction

• Quality multiple-choice items are difficult to
  construct. The more experienced you are, the more
  you learn how hard it is to create fair and
  accurate questions. McKeachie, 1999

• Education is learning what you didn't even know
  you didn't know.-Daniel J, Boorstin

14
Item Components

• Which of the following question components
  describes the incorrect responses?
• Stem
• Correct response
• Distracter

Stem
Distracters
Options
Correct Response
15
Test Plan

• Need to have a clear set of student learning
  outcomes
• Develop a test plan or even a table of
  specifications to guide your exam creation

16
Strengths Weaknesses

• Strengths
• Less time to answer
• Easily accurately marked
• Analysis to improve exam
• Allow more adequate sampling

• Weaknesses
• Can often focus on recall of facts
• Cannot measure skilled performance
• Takes more time to write questions
• Difficult to construct

17
Blooms Taxonomy
Old
New

• Evaluation
• Synthesis
• Analysis
• Application
• Comprehension
• Knowledge

• Creating
• Evaluating
• Analysing
• Applying
• Understanding
• Remembering

(Based on Pohl, 2000, Learning to Think, Thinking
to Learn, p. 8)
18
CreatingGenerating new ideas, products, or
ways of viewing thingsDesigning, constructing,
planning, producing, inventing. EvaluatingJusti
fying a decision or course of actionChecking,
hypothesising, critiquing, experimenting,
judging  AnalysingBreaking information into
parts to explore understandings and
relationshipsComparing, organising,
deconstructing, interrogating, finding Applying
Using information in another familiar
situationImplementing, carrying out, using,
executing UnderstandingExplaining ideas or
conceptsInterpreting, summarising, paraphrasing,
classifying, explaining RememberingRecalling
informationRecognising, listing, describing,
retrieving, naming, finding 
BLOOMS REVISED TAXONOMY
Higher-order thinking
(Based on Pohl, 2000, Learning to Think, Thinking
to Learn, p. 8)
19
MCQ Writing Guidelines

• Content concerns
• Style concerns
• Writing the stem
• Writing the choices

KWL
20
Content Concerns

• Should reflect single content
• Important non-trivial content
• Use novel material, paraphrase
• Keep items independent
• Avoid opinion-based items
• Avoid trick items
• Keep vocabulary simple

21
Novel Material
http//testing.byu.edu/info/handbooks/betteritems.
pdf
22
Avoid Trick Items

• A trick item is
• Trivial content
• Too difficult discrimination
• Multiple correct answers
• High ambiguity
• Ammunition for students who did poorly on a test

23
Simple Vocabulary

• Why was the staff member made redundant?
• He was consistently tardy
• He was repeatedly truant
• He was an ineffectual employee

• Why did the man lose his job?
• He was often late
• He was often absent
• He was a bad worker

24
Style Concerns

• Edit and proof
• Minimize the amount of reading

25
Writing the Stem

• Clear directions
• Central idea in stem not choices
• Avoid window dressing
• Word stem positively

26
Central Idea in Stem
http//testing.byu.edu/info/handbooks/betteritems.
pdf
27
Writing the Choices

• Write as many plausible choices as you can (3 is
  enough)
• Only one correct answer
• Vary the location of correct answer
• Keep choices independent
• Homogenous options (grammar and content)
• Place choices in logical or numerical order
• Keep choices length equal
• Avoid All of the above
• Use None of the above carefully
• Avoid giving clues
• Make all distracters plausible

28
Homogenous
http//testing.byu.edu/info/handbooks/betteritems.
pdf
29
Logical or Numeric Order

• Discrimination increases when options are random-
  punishes lower ability students

• The students on the test scored 8, 6, 7, 5, 8 out
  of 10 on the exam. What is the SD for this exam?

• 1.28
• 1.45
• 1.30
• 1.03

30
Equal Length Choices

• First language students are more likely to
• Remain quiet in class
• Edit all written work
• Actively participate in class discussions with no
  fear of making a mistake

31
Avoid Clues
http//testing.byu.edu/info/handbooks/betteritems.
pdf
32
Plausible Distracters

• In which year was Zayed University founded?
• 2008
• 1999
• 1998
• 1997

33
KWL
Constructing Multiple Choice Exams
Think about Strengths and Weaknesses
34
Questions

• Any issues/ comments?

35
Resources

• A Review of multiple-choice item-writing
  guidelines for classroom assessment
• Designing Managing MCQ's
• How to Prepare Better MC Test Items
• Writing multiple-choice test items
• Improving multiple-choice tests
• Multiple choice item construction
• Techniques for Writing Multiple-Choice Items that
  Demand Critical Thinking
• Bloom's taxonomy Original and revised
• Blooms Revised Taxonomy
• McKeachie, W. J. (1999). McKeachies teaching
  tips. Boston Houghton Mifflin.

36
Remembering

• Which of the following persons is the author of
  the Zone of Proximal Development concept?
• Vygotsky
• Bruner
• Krashen
• Cummins

37
Understanding

• Insert Story here
• What is the main idea of the story?
• The lone wolf is able to succeed
• Trying is as important as succeeding
• Teams are better than individuals

38
Applying
Student Learning Outcome
http//testing.byu.edu/info/handbooks/betteritems.
pdf
39
Analyzing
Student Learning Outcome
http//testing.byu.edu/info/handbooks/betteritems.
pdf
40
Evaluating

• "The creative process is believed to take place
  in five stages, in the following order
  ORIENTATION, when the problem must be identified
  and defined, PREPARATION, when all the possible
  information about the problem is collected,
  INCUBATION, when there is a period where no
  solution seems in sight and the person is often
  busy with other tasks, ILLUMINATION, when the
  person experiences a general idea of how to
  arrive at a solution to the problem, and finally
  VERIFICATION, when the person determines whether
  the solution is the right one for the problem."
• How would you judge this student' s answer?
• EXCELLENT (all stages correct in the right order
  with clear and correct explanations)
• GOOD (all stages correct in the right order, but
  the explanations are not as clear as they should
  be).
• MEDIOCRE (one or two stages are missing OR the
  stages are in the wrong order, OR the
  explanations are not clear OR the explanations
  are irrelevant)
• UNACCEPTABLE (more than two stages are missing
  AND the order is incorrect AND the explanations
  are not clear AND/OR they are irrelevant)

http//www.le.ac.uk/castle/resources/mcqman/mcqapp
c.html