The Objective Structured Assessment of Technical Skills (OSATS)

1
The Objective Structured Assessment of Technical
Skills (OSATS)

• Helen M. MacRae, M.D., FRCSC,
• D.H. Gales Director
• Technical Skills Centre, University of Toronto

2

• Certifying competence of new trainees

The Need for a Reliable and Valid Measure
Certifying Competence in New Technology (for
example laparoscopic surgery or robotic surgery)
Assessing the impact of interventions
3
The OSATS

• A multi-station bell-ringer examination
• Segments of 6 to 8 procedures
• Performed on bench models simulating relevant
  anatomical structures
• Knowledgeable OR nurses assisting

4
The OSATS (continued)

• 15 minutes per procedure
• Candidates move from station to station
• Each station marked by qualified surgeon
• Examiner uses two marking systems
• Task-specific checklists
• Global rating scales

5
Sample of Checklist Scoring (5 of 24)
Vascular Anastamosis Station
0
1

1. Selects appropriate suture
2. Avoids excessive trauma / handling of artery
   with instruments
3. Guides suture down to desired position following
   bites
4. Uses correct needle angle gt80 time.
5. Follows curve of needle on entrance / exit gt 80
   time

0
1
0
1
0
1
0
1
6
Sample of Global Rating (3 of 5)
Vascular Anastamosis Station
Respect for tissue 1 2 3 4 5
Consistently handled tissue appropriately with
minimal damage to tissue
Frequently caused damage by inappropriate use of
instruments
Careful handling of tissue but occasionally
caused inadvertent damage
Time and motion 1 2 3 4 5
Many unnecessary moves
Efficient time/motion but some unnecessary moves
Clear economy of movement and maximum efficiency
Flow of Operation 1 2 3 4 5
Frequently stopped operating and seemed unsure
of next move
Demonstrated some forward planning with
reasonable progression of procedure
Obviously planned course of operation with
effortless flow from one move to the next
7
OSATS - Reliability

• Internal consistency (across stations)

• Inter-rater reliability (within stations)

8
Validity Measures

• Clinical PGY level
• are more senior trainees scoring higher?
• Live animal version of exam
• are better live model operators scoring higher?
• Evaluation of final product
• are those with better outcomes scoring higher?
• Faculty rankings of clinical performance
• are better ranked trainees scoring higher?

9
OSATS - Validity

• Correlation with PGY level

• Correlation with other measures

10
Summary of OSATS

• A non-patient lab-based platform for testing
  technical skills which shows
• Good Reliability
• Good Validity
• Widespread Feasibility
• Widespread Applicability

11
Conclusions

• We can reliably and validly test some of the
  procedural skills of medicine in a formal,
  structured manner outside of the clinical setting