MULTI-STATION CLINICAL EXAMINATIONS

1

• MULTI-STATION CLINICAL EXAMINATIONS
• Win May

2
MSCE - Advantages

• Standardized administration
• Potentially good reliability
• Can provide excellent individual feedback from
  expert clinicians/trained raters

3
MSCE - Disadvantages

• Expensive to prepare and administer
• Educational and clinical experts need to work
  together
• Can be logistically difficult to administer

4
Basic Decisions

• Why Evaluate?
• Who to Evaluate?
• When and Where Evaluate?
• What to Evaluate?
• What Criteria for Pass/Fail?
• Data Collection and Analysis?
• What Feedback and Action?

5
Why Evaluate?

• Determine how results to be used
• Individual learner evaluation
• Formative - diagnostic
• Summative - part of grade
• Summative - certification
• Program evaluation
• Decide precision of results needed
• Decide who will use results

6
Who to Evaluate?

• Select level of learner and the point in training
  for the evaluation
• Determine how many learners to evaluate
• Individual decisions - all learners at selected
  level
• Program decisions - representative sample

7
When and Where Evaluate?

• Time
• Amount of testing time per student
• Total time needed for testing of all students
• Resources
• Personnel (faculty, staff, patients)
• Monetary resources
• Space
• How much available
• When available

8
What to Evaluate?

• Selection of Tasks
• List general skill components and specific
  content which could be tested
• Make station/content decisions
• Develop test plan

9
Number of stations per learner

• Depends on
• Purpose of test
• Time available per testing session
• Amount of time per station task
• Time set aside for feedback
• Who is to rate
• Physical layout of testing site
• Break needed - raters, students

10
Example

• Time / Learner - 4 Hours (240 minutes)
• Time Per Station Task - 15 Minutes
• Time for Feedback - 5 Minutes per station
• Who Will Rate - Faculty
• Physical Layout - 2 minutes in between stations
• Break Needed - 15 Minutes
• Result 240 Min - 15 Min for break 225 Min /
  22 Min per Station 10 stations

11
Other station/content decisions

• Stations static or changing
• Static - content can become known
• Changing - differential difficulty
• Content and skills- should be based on
• objectives
• common / important problems seen by your learner
  on graduation/end of residency

12
Grid for Case Selection
Case Name Ethnicity Age/ Gender Acuity System History Physical Exam
John Smith African-American 57 Male Chronic Cardio-vascular Yes Yes
Maria Luna Latina 36 Female Acute Abdomen Yes Yes
13
Individual StationDecisions

• Determine station content and task(s)
• Make sure tasks fit in the time limit
• Develop case materials
• Locate needed supplies
• Train standardized patient
• Determine how performance in station will be
  rated/scored
• Dichotomous - Yes/No
• Rating Scale

14
Individual StationDecisions

• Determine who will evaluate - faculty or
  standardized patient
• Train evaluators
• If feedback is to be provided, train to provide
  effective feedback

15
Pass/Fail Decisions

• Normative - compared to a group - cut point
  generally set between 1.5 and 2.0 standard
  deviations below the mean
• Criterion Referenced - compared to a standard

16
Some criteria to consider

• Petrusa (2004) - Minimal acceptable clinical
  practice
• Payne (2008) - Critical items - elements critical
  for optimal clinical outcomes

17
Pass/Fail Decisions

• Essential Items
• A set criteria using a list of essential action
  items identified by experts
• Must pass 4 out of 6 cases

18
Example Abdominal Pain Case Summary
A 22-year-old female who has been brought to the
emergency room by her sister. She has been
experiencing stomach pains since yesterday
afternoon. The pain started while she was at
work and it has been constant ever since. The
pain has become progressively more severe.
19
ExampleAbdominal Pain Essential Items
Students are allowed to miss any 1 of the 6
items to pass the case.
asked when the pain started
asked where the pain is located
asked when last menstrual period began.
palpated on abdomen
asked to do a pelvic exam or pelvic rectal exam or a bimanual exam or pap smear
explained she has an acute gyne infection, appendicitis, ectopic pregnancy or ovarian torsion.
20
Data Collection - Logistics

• Coordinator in charge of station development and
  administration for all stations
• Administrative support needs to be obtained to
  ensure that the test happens as intended

21
Data Analysis

• Data analysis should include
• Results by skill scores per station as well as
  overall score to facilitate effective feedback to
  learner
• Item analysis for each station to facilitate
  post hoc review

22
Feedback and Action

• Feedback - timely and meaningful feedback to
  students, administrators, test and station
  coordinators, and raters.
• Action - What happens to students who perform
  poorly needs to be decided in advance.

23
Remediation

• Video Review
• Self-Reflection
• Standardized Patient practice
• Work with Faculty
• Repeat MSCE

24
Post-Hoc Review

• Post-administration review should include
• Reflection on examination process
• Subjective and objective review of examination
  results for purposes of
• Individual student performance
• Overall group performance
• Station by station results

25
In summary

• Why Evaluate?
• Who to Evaluate?
• When and Where Evaluate?
• What to Evaluate?
• What Criteria for Pass/Fail?
• Data Collection and Analysis?
• What Feedback and Action?

26
Thank you!