Computerised Free-text Assessment of Key Feature Problems

1
Computerised Free-text Assessment of Key Feature
Problems

• Adrian Molyneux
• Steve Capey
• School of Medicine

2
Project aims

• Fulfil student requests for more formative
  assessment
• Fulfil need for practice of summative exam
  formats
• Provide quality feedback
• Achieve this without a heavy marking load placed
  on staff

3
Key-feature problems (KFPs)

• an innovative assessment format that tests
  clinical decision making skills by focussing only
  on the critical steps - or 'key features' - of
  each decision, thereby increasing the number of
  problems that can be answered in an examination.
• - Liz Farmer, Royal Australian College of GPs

4
SCENARIO - Jennifer Evans, a 38-year-old
statistician, presents to the accident and
emergency department with a two hour history of
sudden onset of leftsided pleuritic chest pain
shortness of breath and an episode of
haemoptysis. She had returned from a conference
in Malaga on a flight 48 hours earlier. On
examination she was found to be slightly febrile,
tachycardic and in atrial fibrillation. Her
blood pressure is 98/64mmHg. Examination of the
chest showed decreased expansion on the left and
localised decreasing breath sounds with a pleural
rub. A plain radiograph of the chest shows a
very small trace of fluid in the left
costophrenic angle to the lungs are clear. A
presumptive diagnosis of pulmonary embolism is
made.
5
Example questions from scenario

• 1. What initial investigations should be
  requested? Name only two (2)
• 2. Long distance travel is considered a minor
  risk factor for venous thromboembolism what are
  the major risk factors? Name only four (4)
• 3. What is the mechanism of action of heparin?
• 4. Shortly after the heparin was commenced,
  Jennifer was started on warfarin. What
  endogenous compound does warfarin inhibit to
  produce its anticoagulant effect? List only one
  (1)
• 5. How is warfarin therapy monitored?

6
Research questions

• How do we train a computer system to mark an
  open-ended freetext response?
• How to deliver the assessment
• How to feed back the results to students
• How to author succinct KFPs
• Can we develop a method for the most efficient
  creation of a computer-marked KFP?

7
Methods

• Generate 10 KFPs and marking guidelines
• Generate computer markscheme
• Build web interface
• Recruit students and run exam
• Computer-mark responses
• Tutor-mark responses
• Refine markschemes (iterative process)
• Statistical analysis

8
(No Transcript)
9
Exam delivery system and results

• Exam delivery
• Results

10
Computer Tutor comparison
11
Markscheme improvement

• Make first attempt at markscheme and run exam
• Examine student answers for further valid
  responses
• Include these in computer markscheme and mark a
  second time
• Examine computer markscheme for badly-performing
  questions and adjust (maybe loosen). Mark a
  third time.

12
Run Number Mean Mark of Human Mark
1 0.9 39
2 1.1 47
3 2.0 86
13
Conclusions practical limitations

• Succinct answers should be encouraged perhaps
  limit text entry
• Expect unexpected answers from students and
  markscheme modification
• Standardisation of human marking
• Adherence of humans to marking guidelines
• Evolution of computer markschemes to cover the
  whole range of student answers

14
Conclusions - positives

• Computer marking ideal for questions with narrow
  range of possible responses
• Reduce marking hours spent on formative
  questions, whilst providing quality feedback in a
  realistic examination format
• Perfect for low-stakes, on-demand assessment

15
Future direction of work

• Run exams with full year groups
• Refine question authoring
• Evaluate other freetext question formats
  currently used within the medical school
• Investigate the possibility of distributing tests
  asynchronously throughout the curriculum