INTRODUCTION TO DUNDEE

1
The Role of Multiple Mini Interviews (MMI) in
selection Dr Jon Dowell, Admissions Convenor,
Dundee Medical School
2
Overview

• Rationale behind MMI for selection
• A brief history
• What do MMIs assess
• How do MMIs work
• Performance in UK context
• Feedback from applicants and assessors
• The future?

3
Medicine has previous
Primus
Alexander Monros 1720 - 1846
Secundus
"I dislike him his lectures so much that I
cannot speak with decency about them. He is so
dirty in person actions."
Tertious
4
Rationale

• Personal qualities considered important but
  not reliably assessed by interview and without
  much predictive validity.
• (Goho Blackman 2006. cognitive r 0.06,
  clinical r 0.17)
• Introduced for medicine at McMaster, Canada
  (Reiter, Eva et al).
• Piloted 2002 12 x 8 min. Generalisability
  since /- 0.7
• Predicted OSCE performance ß 0.44 (interview ns)
• Clerkship ratings ß 0.57 (interview ns)
• MCC Part 1 (selected components) ß 0.3-0.4
  (interview ns)

5
Increasing detail emerging

• A cost efficiency comparison between the multiple
  mini-interview and traditional admissions
  interviews. Set up costs, running
  equates
• Rosenfeld JM, Reiter HI, Trinh K, Eva KW.
• Adv Health Sci Educ Theory Pract. 2008
  Mar13(1)43-58.
• The multiple mini-interview how long is long
  enough? 10 8 5 mins
• Dodson M, et al. Med Educ. 2009 Feb43(2)168-74
• Factors affecting the utility of the multiple
  mini-interview in selecting candidates for
  graduate-entry medical school. 8Q reliability
  0.7
• Roberts et al. Med Educ. 2008 Apr42(4)396-404
  applicant 22 variance
• Should candidate scores be adjusted for
  interviewer stringency or leniency in the
  multiple mini-interview? Rasch Modeling
• Roberts C, Rothnie I, Zoanetti N, Crossley J.
  Med Educ. 2010 Jul44(7)690-8

6
Increasingly widespread

• Countries
• Canada,
• Australia
• UK
• US

• Disciplines
• UG Medicine
• UG Veterinary
• UG Dentistry

7
MMI in Dundee 7 minute x 10 stations

• Reliability
• 2008/9 n 450 0.7
• 2009/10 n 500 0.69
• 2010/11 n 600 0.69/.67 (0.88)
• 2011/12 Dental school collaboration

8
MMI in St Georges 5 minute stations x 8
StationsMuriel Shannon Aileen OBrien

• Reliability
• 2009/10 n 1078 0.69
• 2010/11 n 1333

9
MMI in Belfast 5 minute x 9 stationsKeith
Steele

• Reliability
• 2010/11 n 500 0.56

10
MMIs typically aim to assess

• Interpersonal and communication skills (inc
  empathy)
• Teamwork and leadership
• Preparation and motivation
• Critical thinking, problem solving
• Ethical/moral reasoning
• Integrity

11
How?
12
Practicalities

• Capacity (n 600)
• 20 applicants per Run per session
• Two teams in parallel (80/day)
• 8 days total
• Staffing (2 academic leaders 2 admin)
• For each run
• 5 faculty (interview)
• 5 student or Simulated Patients (interactive)
• 2 actors (employed junior students)
• 3 role players (students or SP)

13
Preparation big job

• Stations - Drafted and piloted with students
• Assessor instructions
• Candidate instructions
• Resources (script, shapes, materials, UCAS forms
  etc)
• Actor training
• Training at start of every session
• 30 min general
• 20 min station specific

14

Station/Assessor/ Candidate Information
Domain Scores
Overall Judgement
Red Flag Narrative
15

16
Performance of Dundee MMI

17
Score Distribution
18
Station Reliability
     
Overall Alpha 0.69 Item-Total Correl Alpha if Item Deleted
1 0.34 0.67
2 0.31 0.67
3 0.17 0.70
4 0.37 0.66
5 0.43 0.65
6 0.47 0.64
7 0.34 0.67
8 0.28 0.67
9 0.38 0.66
10 0.39 0.66
19
Domain Reliability
Number of Stations Number of Scores Cronbachs alpha
Communication 9 12 .767
Critical Thinking 6 6 .453
Moral Reasoning 4 4 .170
Prep Motivation 3 4 .515
Teamwork 2 3 .395
Integrity 1 1 -
20
Gender difference - YesInteractive One - one
Females performed significantly better than
males. 2009 Female average 109 /150 vs 105 p
lt .01 Effect size 0.14
21
Applicant Group difference?
2009 2010 2010 - 2011
22
Apologies but we are intending to eventually
publish from this data so have removed it from
circulated slides.
23
Assessors Survey 2008-9

• 116 /201 (58) assessors completed online survey.
• 91 thought the process fair.
• 88 thought applicant stress moderate or less.
• All stations rated highly relevant, except for
  the Question of Ethics where opinions divided.

24
Candidate Survey 2008-9

• 324 /433 (75) completed online survey.
• 94 felt MMI was fair.
• 90 felt it is a valid way to assess
  candidates.
• 71 preferred MMI to traditional interview.
• Student Counselor (role play) most enjoyable

25
We are happybut where next?
26
Hawk-Dove Effect
STATIONS
1 2 3 4
5 6 7
8 9 10
A
B
C
D
E
27
Dundee 2011 Rasch vertical ruler
MFRM creates a common scale for candidate
ability, examiner stringency, item difficulty and
...... Comparing raw and fair scores for 350
offers 34 (9.7) different candidates selected.
28
Conclusions and Next Steps

• MMI appear very promising
• Set up costs considerable
• Running costs comparable (if students used)
• Student involvement positive
• Applicants are positive
• Predictive validity emerging and encouraging.
• Outstanding issues
• Rotating content.
• Application of IRT (DIFF, Fair scores etc)
• Oh yes and the bloopers are great.

29
Thank youDiscussion?