Computer Aided Multiple Choice Assessment and Level 3: Unlikely Bedfellows

1
Computer Aided Multiple Choice Assessment and
Level 3 Unlikely Bedfellows?

• Dr Dominic Upton
• Centre for Psychology.

2
Overview of presentation.

• Use of CAA/MCQs.
• Blooms taxonomy.
• Projects initial aims and developments.
• Projects status.
• Project development.

3
Why use CAA?

• Improved efficiency
• Improved speed
• Wider coverage of topics
• Analysis of individual questions to provide
  fuller feedback
• Use of on-line capabilities
• Elimination of subjective marking
• Develop questions based on academic level.

4
Why not CAA?

• Equipment failure.
• Students fear of IT.
• Time to develop.
• Co-ordination of services required.
• Question types are limited.
• Supports question types for lower academic
  levels only.

5
UWIC Previous work.

• Considerable work on CAA.
• Useful guidelines and pilot studies.
• This project- explores from a pedagogical
  perspective.

6
Question types and CAA.
CAA is good for objective tests.
Or can it?
Multiple choice tests are good objective tests.
But MCQs can only be used for certain academic
levels.
7
Blooms Taxonomy (1956) A reminder.

• Knowledge Recall of information, knowledge of
  facts.
• Comprehension Interpretation of information.
  Grasping meaning.
• Application Application of methods, theories,
  concepts to new situations.
• Analysis Identification of patterns.
• Synthesis Generalise from given knowledge.
  Organise and relate knowledge from several
  areas.
• Evaluations Make judgements. Evaluate data.

8
Blooms Taxonomy (1956) Question words.

• Knowledge List, define, label, describe, name.
• Comprehension Interpret, discuss, predict,
  summarise, classify.
• Application Apply, demonstrate, show, relate.
• Analysis Analyse, arrange, order, explain,
  connect, infer, compare, categorise.
• Synthesis Integrate, modify, invent, design,
  compose, plan.
• Evaluations Appraise, judge, evaluate, defend,
  rank, conclude, discriminate.

9
Blooms Taxonomy (1956) and academic level.
Level 1
Easy with MCQ

• Knowledge
• Comprehension
• Application
• Analysis
• Synthesis
• Evaluation

Level 3
Difficult with MCQ.
10
Previous work.

• Bloom (1956) Found that 95 of all MCQs assessed
  knowledge.
• Bacon (2004) MCQs used just to assess knowledge.
  
• Wilson and Case (1993) MCQs test recall of
  isolated facts.
• Lambert et al (1996) Growing dissatisfaction
  with MCQ since only assess recall.
• Bacon (2004) Should be able to use MCQs if
  construct questions properly

11
Question Can CAA/MCQs be used at Level 3 Health
Psychology?

• No, dont be so silly.
• Yes, of course they can.
• Somewhere between the two.
• Not yet

12
Elements of MCQs.
Options
Stem
Who developed the Gate Control Theory of pain?

• Folkman and Lazaus
• Fordyce
• Melzak and Wall
• Marks and Willig
• Selye

Distractors
Key
13
Benefits of CAA.

• Ability to add other media. For example What
  model of pain does the image represent?
• A. Social model
• B. Gate Control model
• C. Interactional model
• D. Linear model

14
Development of L3 questions.

• Projects aims
• Development of MCQs at Level 3.
• Development of database of options, stems,
  distractors, key answers and options for range of
  academic levels/courses.
• Introduction of sophisticated assessment
  exploiting full range of on-line and computer
  based material.
• Student evaluation of CAA/MCQs.

15
Health Psychology MCQs Knowledge

• Who developed the Gate Control Theory of pain?
• Folkman and Lazaus
• Fordyce
• Melzak and Wall
• Hobfoll
• Selye

16
Health Psychology MCQs Comprehension

• Whose model of stress can be classified as
  primarily physiological?
• Folkman and Lazaus
• Fordyce
• Melzak and Wall
• Hobfoll
• Selye

17
Health Psychology MCQs Application.

• Bill believes that his friends and family will
  support him while he tries to quit smoking. This
  believe about his available social support is an
  example of what aspect of the Theory of Reasoned
  Action?
• Perceived behavioural control
• Intentions
• Subjective norms
• Attitudes
• Self-efficacy

18
Health Psychology MCQs Analysis.

• You can conclude from the data
• Stress is higher in the lower social classes
• Stress is lower in the higher social classes
• Stress is NOT higher in the higher social
  classes
• Stress is NOT higher in the lower social classes
• Stress is equivalent amongst the social classes

19
Health Psychology MCQs Synthesis.

• The work of Leviatan and Cohen (1985) would
  suggest that the best way to reduce gender
  inequalities would be
• Ensure males and females continue with their
  current occupations.
• Get females more involved in masculine
  occupations.
• Get males more involved in feminine occupations.
• Get males more involved in masculine occupations.
• Get females more involved in feminine occupations.

Comprehension
Knowledge
Analysis
20
Health Psychology MCQs Evaluation

• The data from Grossarth-Maticek et al (1986) are
  suspect because
• The report was not subject to peer review.
• The psychometric properties of the Type C measure
  were poor.
• The group allocation was not randomised.
• It was not clear how the terms were
  operationalised.
• All of the above.

21
Database construction.

• Derivation of items and distractors.
• Topics selected for questions- ensuring CAA is
  embedded in learning outcomes of module.
• Derived Item shells.

22
Item shells.

• Knowledge/Comprehension
• Which best defines?
• Which is characteristic of?
• Which is the following example of?

• Synthesis/Evaluation
• Which is most effective?
• Which is better?
• What is the most effective method for?

• Application/Analysis
• What would happen if?
• If this happens, what should you do?
• Given what should you do?

• Synthesis/Evaluation/Problem solving
• What is a possible solution?
• Which is the most efficient solution?
• Why is the most efficientsolution?

23
The story so far

• Level 3 health psychology can be assessed by
  MCQs.
• Easy to devise 1 essay, but 100 MCQs?
• Time taken

24
Unanswered questions

• What form of assessment would students prefer?
• Can the MCQs be fully integrated into a Level 3
  module?
• What do students consider the positives and
  negatives of both CAA and MCQs?

25
Evaluation of CAA.

• Some have suggested CAA leads to drop in student
  performance (Ricketts and Wilks, 2001) whilst
  others not (OHare, 2001).
• Students studying IT subjects prefer CAA, other
  studies have equivocal results.
• But do students like any assessment?

26
Evaluation.

• Review students views and performance on CAA.
• Questionnaire developed for project.
• Based on Assessment Experience Questionnaire
  (AEQ, Gibbs and Simpson, 2003) and the work of
  OHare.

27
Some current concerns.

• Are MCQs the best way of assessing?
• Marking of MCQs- negative marking?
• Isolated areas of knowledge tested by MCQs, not
  depth?
• Answered by elimination.
• Difficult to write good one.

28
Project findings.

• Yes, it can be done.
• Takes time (too much?).
• Students views on MCQ are negative.
• Resistance amongst others (course members,
  external examiners).
• Is it worth it?

• Bloom (1956)/Bacon (2004) Perhaps a reason why
  so many just use MCQs for recall...
• Bacon (2004) Should be able to use MCQs if
  questions constructed properly

29
Is it worth it?
Students Course team External Examiners Most
others!
Me.
30
Developments for next year.

• Vignette, plus responses with different vignettes
  at different levels.
• Assessed through Blackboard.
• Running for real next year

31
Extended matching questions.

• Now being used in medical field.
• Scenario/vignette plus 20 possible responses from
  which to choose best answer(s).
• More difficult to write.
• Can use same responses with different scenarios.
  

32
Sample question.

• Lucy smokes 20 cigarettes a day, but is thinking
  about quitting.
• How could Lucys cognitions be described?
• What would be the best form of intervention for
  Lucy?
• Health promotion campaigns are usually useless
  for people in stage
• Health promotion campaigns are usually designed
  for people in stage
• Health promotion campaigns usually use
• Health promotion campaigns could use

• Responses
• Pre-contemplation
• Contemplation
• Preparation
• Action
• Maintenance
• Consciousness Raising
• Environmental Control
• Sell Reevaluation
• Commitment
• Helping Relationships
• Reward
• Countering

B
F
A
A
F
F, H, K, L
33
Conclusion.
It can be done.
Tutors views on whether it should be done.
Student views on whether it should be done.