Designing and Using Multiple Choice Questions

1
Designing and Using Multiple Choice Questions

• Dr Rod CullenSenior Lecturer in Learning and
  Teaching TechnologyLearning and Teaching Unit
• Dr Alan FieldingSenior Learning and Teaching
  FellowSchool of Biology, Chemistry Health
  Science

2
Aim

• Writing well formed MCQs
• Designing automated feedback for formative
  assessments
• Assessing higher order skills with MCQs
• Appropriateness of MCQs
• Practicalities of online/computer-based MCQ

3
Session Outline
4
Background

• Rod Cullen
• Staff development and training at MMU
• Online and classroom based training and support
  for staff interested in using MCQs, where
  appropriate, to assess students.
• Previously worked at UoM
• Alan Fielding
• Using MCQs to assess own students
• Training School staff in design and use of MCQs
• This session will demonstrate issues raised and
  training activities we have used

5
An example MCQ

• Who is credited as discovering the antibacterial
  medicine penicillin? (STEM)
• Louis Pastuer (DISTRACTER)
• Edward Jenner (DISTRACTER)
• Alexander Fleming (CORRECT ANSWER)
• Jonas Salk (DISTRACTER)

6
Questions of design

• If MQCs are poorly designed it is easy to end up
  with questions containing clues to the correct
  answers
• Evidence to suggest that student who are
  regularly tested with MCQs can identify and use
  these cues to predict the correct answers

7
Activity 1 (5-10 minutes)

• The Swerglehopper quiz
• Possible to work out the correct answer for all
  of the questions even though you cant possibly
  know anything about Swerglehoppers.
• In pairs work through the 8 questions and try to
  spot the design errors.

8
Feedback for formative MCQs
9
Principles of feedback for MCQs

• Incorrect response from student
• What is the correct answer
• Reference to course text
• Reference to specific materials outside course
  text
• Correct response from student
• Same level/type of feedback

10
Testing higher order skills with MCQs
11
Testing Higher Order Skills

• A major criticism of MCQs is that they often
  focus on testing factual information (knowledge
  reproduction) and fail to test higher levels of
  cognitive thinking.
• What do we mean by higher order skills?

12
Blooms Taxonomy of Learning - Cognitive Domain
5
Learners are able to apply prior knowledge in
original ways to produce things that are new and
different and evaluate methods used
4
Learners are able to analyse problems to discover
component parts and interactions
3
Learners are able to apply their knowledge in new
situations to solve relatively simple problems
2
Learners gain increased comprehension of the
material and are able to explain the material and
interpret what they have learned
1
Learners know the terminology and specific facts
about a subject
Adapted from 2004
13
Intended Learning Outcomes

• A learning outcome is a statement of what a
  learner is expected to know or be able to do as a
  result of a learning process
• Centre for the Advancement of Teaching and
  Learning (2003)
• Questions need to be aligned with learning
  outcomes

14
Knowledge Reproduction

• Which one of the following persons is the author
  of "Das Kapital"?
• 1. Mannheim
• 2. Marx
• 3. Weber
• 4. Engels
• 5. Michels

15
Comprehension

• Which one of the following describes what takes
  place in the so-called PREPARATION stage of the
  creative process, as applied to the solution of a
  particular problem?
• 1. The problem is clearly identified and
  specifically defined.
• 2. All available information about the problem is
  collected.
• 3. An attempt is made to see if the proposed
  solution to the problem is acceptable.
• 4. The person goes through some experience
  leading to a general idea of how the problem can
  be solved.
• 5. The person sets the problem aside, and gets
  involved with some other unrelated activity.

16
Application

• You are the sole owner and manager of a small
  enterprise employing 15 workers. One of these,
  Alfred, (who has been working for you for the
  past year and has somewhat of a history of
  absenteeism), arrives late for work one Wednesday
  morning, noticeably intoxicated. Which one of the
  following actions is the most appropriate in the
  circumstances?
• 1. You terminate Alfred's employment on the spot,
  paying him the wages still due to him.
• 2. You parade Alfred in front of the other
  workers, to teach them all a lesson.
• 3. You give Alfred three weeks' wages in lieu of
  notice, and sack him.
• 4. You wait until Alfred is sober, discuss his
  problem, and give him a final written warning,
  should it be required.
• 5. You call Alfred's wife to take him home and
  warn her that this must not happen again.

17
Analysis

• The famous Chemist Auguste Kékulé was struggling
  with the problem of how the six carbon atoms of
  benzene were linked together. One day fell asleep
  in front of the fireplace while he was pondering
  on it. He dreamt of molecules twisting and
  turning around like snakes. Suddenly, one of the
  snakes swallowed its own tail and rolled around
  like a hoop. Kékulé woke up with a start, and
  realized that his problem could be solved if the
  six carbon atoms of benzene were attached to each
  other to form a ring. Further work showed that
  this was entirely correct."
• The above passage illustrates a particular phase
  of the creative process. Which one is it?
• 1. preparation
• 2. incubation
• 3. orientation
• 4. illumination
• 5. verification

18
Evaluation

• A student was asked the following question
• "Briefly list and explain the various stages of
  the creative process".
• As an answer, this student wrote the following
• "The creative process is believed to take place
  in five stages, in the following order
  ORIENTATION, when the problem must be identified
  and defined, PREPARATION, when all the possible
  information about the problem is collected,
  INCUBATION, when there is a period where no
  solution seems in sight and the person is often
  busy with other tasks, ILLUMINATION, when the
  person experiences a general idea of how to
  arrive at a solution to the problem, and finally
  VERIFICATION, when the person determines whether
  the solution is the right one for the problem."
• How would you judge this student' s answer?
• 1. EXCELLENT (all stages correct in the right
  order with clear and correct explanations)
• 2. GOOD (all stages correct in the right order,
  but the explanations are not as clear as they
  should be).
• 3. MEDIOCRE (one or two stages are missing OR the
  stages are in the wrong order, OR the
  explanations are not clear OR the explanations
  are irrelevant) 4. UNACCEPTABLE (more than two
  stages are missing AND the order is incorrect AND
  the explanations are not clear AND/OR they are
  irrelevant)

19
Example 1

• One approach to assessing whether a given
  enrichment initiative has achieved its intended
  behavioural goal is via the use of rating scales.
  Following the introduction of a new nesting
  material, several responses are possible. Arrange
  these in order with the least 'desirable'
  (0-rated) option firsta. Partial nests
  built.b. Ate substantial amounts of nesting
  material.c. Nests built and used by 50 of
  group.d. No response to nesting material.e. One
  nest built but not used.f. Dragged nesting
  material all over enclosure.ANSWERS1. b, d,
  a, f, e, c2. d, b, f, a, e, c3. f, a, e, b, d,
  c4. c, e, a, f, d, b5. b, d, f, a, e, c

20
Example 2

• "When the proximate factors governing the
  expression of particular behaviour patterns are
  considered, it is clear that some are triggered
  primarily from within, some from without and
  others by complex interactions between the two."
  (Hinde)
• An example of a complex relationship between
  internal and external factors is provided by
  dustbathing in birds. Identify which of the
  following statements support the dominance of
  endogenous factors in the control of this
  behaviour patterna. Dustbathing is stimulated
  by high lipid content in feathers.b. Genetically
  featherless birds dustbathe as much as equivalent
  groups of normal birds with feathers.c. Most
  dustbathing occurs in the middle of the day.d.
  Birds will perform the motor patterns of
  dustbathing in the absence of a substrate.

21
Example 3

• Note students were told to read Morisse et al.
  (1999) in advance of the assessment.
• The results described by Morisse et al. (1999)
  appear to demonstrate that rabbits kept under
  intensive conditions prefer a wire floor to a
  straw deep litter. From the following list,
  identify the aspects of experimental design that
  may have introduced bias into this study.a.
  Position of feeding hoppers within pensb. The
  effective stocking density in non-choice pens.c.
  Use of instantaneous scan sampling for the
  collection of behavioural data.d. The effective
  stocking density in choice pens. e. Sex ratio in
  sample groups.f. Frequency of litter changes in
  choice pens.g. Age structure of sample groups.
• h. Orientation of pens.

22
Example 4

• This is one of several questions relating to some
  printed material with Tables.
• The remaining questions relate to Table 1
  (printed material).
• There may be more than one correct hypothesis for
  each statement.
• Statement 1 Mass of females should be recorded
  at intervals throughout the observation
  period.a. Hypothesis 1. Males are eating less
  than the amount required for maintenance and are
  losing condition.b. Hypothesis 2. Females are
  eating more than requirements for maintaining
  condition.c. Hypothesis 3. Females are eating
  poorer quality food and so have to eat more.d.
  Hypothesis 4. Females have specific nutrient
  requirements and so have to eat more.e.
  Hypothesis 5. Sampling error.

23
Activity 1

• In Groups of three
• Match the questions on the sheet to levels of
  Blooms taxonomy

24
Developing MCQs

• Best done in teams
• Writer(s)
• Checker(s)
• Tester(s)
• Checklists can be helpful many examples

25
Summary

• MCQs not easy to do well
• Should be aligned with learning outcomes
• Matched to appropriate level of Blooms taxonomy
• Feedback is and essential part of formative
  process
• Best to develop in teams with QA process.