Subjects

1
360 BENCHMARKS A NEW RESIDENCY EVALUATION TOOL
L Moraski, J Nowak, DS Bragg, D Brown, DE Simpson
Data Analysis Results
360o Instrument
Purpose/Problem

• Differences by Rater Groups
• No significant differences between rater groups
  on overall clinical competence item
• Significant differences by group obtained on
• - 3 patient care items (p lt.05)
• - 1 practice-based learning item (p lt.05)
• - 2 systems-based practice items (p lt.01)
• ? Follow-up analysis revealed significant
• differences between all others and the following
  groups (6 items above)
• - Resident self-assess 26 of items (6/6)
• - Students 22 of items (5/6)
• - Peers and Staff 4 of items (1/6)
• - Faculty 0 of items (0/6)
• Rank Order of Raters (Easy to Most Critical)
• - Students? Faculty/ Peers
• Lowest overall ratings in systems-based practice
  and practice-based learning

? New ACGME competency-based outcome standards
mandate use of multiple assessment methods and
assessors to evaluate residents progress ?
ACGME Toolkit 360 evaluation ? Multiple
assessors Self, peer, ancillary personnel,
faculty, nurses, medical students ? Different
categories of raters evaluate same competencies
on same scale
Evaluation Questions
Utility Is resident performance differentially
assessed by different rater groups using
360º? Feasibility Can you do a 360º for entire
residency program?
Limitations Discussion
Methods

• Subjects Administration
• 12 Med-Peds residents (PGY1-4s)
• After orientation consent, each submitted rater
  list
• Guidelines Self eval, 3-5 peers, 3-5 faculty,
  3-5 ancillary support staff (includes nurses,
  techs, RTs)
• All raters oriented evaluation sent via email
  paper
• 12 residents generated 191 unique raters
• MANOVA conducted to determine significant
  differences by rater category with Tukey Post Hoc
  to identify specific rater group differences
• Residents debriefed on individual results during
  semi-annual evaluation with program director

• 360o Instrument1
• 13 Specialty-specific RRC requirements analyzed
  to identify key competencies associated with each
  6 ACGME Outcomes.
• Competence included in 360o if item cited by gt
  45 (5/11) of RRCs.
• Each outcome yielded 3-5 common competencies for
  total of 23 items including overall competency.

• Feasibility
• Data collection no patient data data sources
  limited by sites 3 inpt 8 amb
• E-mail or residency tracking system required for
  adequate response rate
• Utility
• Limited variability by raters other than resident
  self-assessment.
• Time/effort for results yields limited return on
  investment
• Individual resident review well received and able
  to correlate findings with behaviors and
  performance.

1. Brown D, SIMPSON D, et al. Identifying the
Common Elements Across Residency Review
Committees for the ACGME Outcomes Competencies.
ACGME Annual Education Conference. Chicago, IL,
March 5-7, 2003.