Multiple Mini Interviews in the UK

1
Multiple Mini Interviews in the UK

• Jon Dowell - University of Dundee
• Aileen OBrien - St George's University of London

2
Rationale

• Personal qualities considered important but not
  very reliably assessed by interview and without
  much predictive validity. (Goho Blackman 2006.
  cognitive r 0.06, clinical r 0.17)
• Introduced for assessing medical school
  applicants at McMaster University, Canada.
  (Reiter, Eva et al)
• Piloted 2002 12 x 8 min. Generalisability
  since /- 0.7
• Predicted OSCE performance ß 0.44 (interview nil)
• Clerkship ratings ß 0.57 (interview nil)
• MCC Part 1 (selected components) ß 0.3-0.4
  (interview nil or negative)
• Supporting evidence from
• Canberra - Harris and Owen 2007
• Calgary - Brownell et al 2007
• Dundee - Ponnamperuma 2008 / 4 station trial
  2007-8

3
Domains assessed

• Dundee
• Interpersonal and communication skills
• Critical thinking, problem solving
• Team work
• Robustness against stress
• Ethical/moral reasoning
• Integrity/honesty
• Preparation and motivation for medical school

• Georges
• Empathy
• Communication
• Organisation and problem solving
• Team work
• Initiative and resilience
• Insight and integrity
• Effective learning style
• Academic ability intellect
• ability intellect
• (already tested)

4
Dundee Stations
2010 2010
Role play OCD discussion
11 Professional Integrity
Task Sort it out - cards
11 Issues in Medicine
Task Puzzle
Role play Fitness to Practice
11 Medicine as a career
11 About you - UCAS
Role play Medications Muddle
11 Moral Dilemmas
5
How it runs
6
Marking
Training 30 min general 20 min station
specific Optimal criteria for each domain
defined
Name Label
7
Stations St Georges

• Standard interview questions
• Prioritising
• Scenario/ breaking bad news
• Presentation
• Observation
• Communication/ giving instructions

8
Georges Marking Scheme
Candidates name Motivation
to do medicine What are you looking forward to
about being a doctor? Points for
guidance Personal satisfaction from helping
people Satisfaction from achieving something
worthwhile Working with a diverse range of
people Combining an interest in science with
caring for people Variety of career opportunities
once qualified Job status and security Good
income The opportunity to work abroad Intellectual
rigour Lifelong learning and clear professional
career structure Camaraderie from clinical teams
Research collaboration Please mark out of
5 Poor motivation 1 2 3 4 5
good motivation Please also assign an overall
impression of the candidates suitability to be
offered a place at medical school based on their
performance at this station (please circle and
comment as appropriate). EXCELLENT ACCEPTABLE C
AUSE FOR CONCERN Red card Comments
9
Georges results

• Graduates and non graduates did equally well
• No correlation with
• Gender
• Age
• 4 year and 5 year course
• UKCAT and GAMSAT scores
• Reliable (kappa 0.68)

10
Georges Conclusions

• Successful pilot 2009
• Ran this year with 1000 candidates
• Smooth (two breakdowns!)
• Graduates and school leavers same cut off score
  for admission

11
Dundee results

• 2009 n 452 Cronbachs Alpha .70
• 2010 n 477 Cronbachs Alpha .69
• No correlation in 2009 with
• Gender
• Age
• Overseas candidates
• School type
• Small age gender interaction

12
2010 MMIs Gender

• Females performed significantly better than
  males.
• t(475) 3.19, p lt .01
• Effect size 0.14

13
2010 MMIs Applicant Group

• Significant differences between groups.
• F(4) 2.50, p lt .05
• Graduates/Mature perform significantly better
  than all other groups

14
Domain Reliability
Number of Stations Number of Scores Cronbachs alpha
Communication 9 12 .767
Critical Thinking 6 6 .453
Integrity 1 1 -
Moral Reasoning 4 4 .170
Motivation 3 4 .515
Teamwork 2 3 .395
15
Correlations between scales

• No correlation between UKCAT and MMI scores
• No correlation between our UCAS form and MMI
  scores

16
Dundee MMI - Conclusions

• Good reliability in last two years.
• All stations were reliable enough. No negative
  Item-total correlations no reason to reject any
  station.
• Successfully ran for over 900 candidates with no
  significant problems (no breakdowns, tears etc!)
• Putting more weight on MMI may increase female
  and graduate/mature candidates compared to the
  Pre-Interview Score

17
MMI/Pre-Interview
  40/60 60/40 80/20
Male 129 122 121
Female 201 208 209
A-Level 81 79 81
Scottish 189 182 179
G/M 27 35 38
EU 16 17 17
OS 17 17 18
48
40
Selection Difference
88
18
So what do the punters think?
19
Assessors Survey Dundee

• 116 /157 assessors (88 of students and 50 of
  staff) completed online survey to find out how
  they viewed MMI.
• 91 thought the process fair.
• 88 thought stress on applicants moderate or
  less.

20
Candidates Survey Dundee

• 324 /433 (75) of Candidates completed online
  survey.
• 94 felt MMI was fair.
• 90 felt it is a valid way to assess
  candidates.
• 71 preferred MMI to traditional interview.
• Student Counselor (role play) cited as the most
  enjoyable (153/172).

21
Assessors Survey Georges

• Equally fair
• Equally accurate
• Equally able to pick best candidate
• Interviews coached
• MMIs can be boring
• MMIs good distinguishing tool

22
Candidates survey Georges

• 5 years- MMI more fair
• -MMI more accurate
• -interview more difficult
• 4 years- MMI more difficult
• -equally fair and accurate

23
Conclusions and next steps

• MMIs becoming an interview tool at many schools.
  Is it justifiable not to use them?
• How to develop new stations? Collectively?
• Reliability potentially increased (above 0.8)
  using Rasch analysis alters decision for 10
• Predictive validity in UK? could UKCAT help?
• Questions...

24
Options to improve?

• ? Rasch Modeling

25
Variation in outcomes can be enhanced by using
multiple tracks
Examiner
5
6
7
8
9
10
4
3
2
1
Track
26
Yardstick
27
Effect of using Rasch

• Reliability based on fair scores 0.84
• 10 different candidates would be given offers.
  ( 10 not)
• Only the top 169 stable.
• Could not be 100 confident in process this year,
  so not applied. (complex black box stuff)