Messages from the data and next steps

1
Trainee Survey 2006

• Messages from the data and next steps

2
Objectives of the Trainee Survey

• Providing a UK wide picture of training provision
  as perceived by trainees themselves against PMETB
  Training Standards
• A key source of evidence for ongoing quality
  assurance work

3
Data collection processes

• Data was collected between 15th May and 28th July
  (injury time until 4th August)
• Deaneries nominated a survey contact for their
  deanery who worked with PMETB/CoPMED team.
• Deaneries were free to choose from one or more of
  the following routes of administration that were
  kindly supported by PGMECs
• Black box
• Web email
• Web letter
• Scannable paper form
• Deanerys own route

4
What the survey could add to QA processes that
used visits alone

• A standardised data set allowing comparison
  across specialties, grades and locations
• Greater reliability we are asking more people
  the same questions
• The ability to provide information on training to
  trainees themselves

5
Total responses

• 29,141 responses received
• Cases that didnt meet the inclusion criteria
  were then excluded, leaving 24,882 for analysis
• The excluded included 2,170 foundation trainees
  that some deaneries chose to survey
• This gives a UK wide response rate of 63

6
What the survey sought to measure

• Trainees perceptions of their providers
  adherence to PMETB standards
• our example today supervision
• Outcome measures
• overall satisfaction
• medical errors

7
Overall across the four nations trainees give
their posts a positive rating..
8
Overall satisfaction

• A set of questions taken together designed to
  measure satisfaction generally, not whether a
  specific facet of training has taken place
• Questions asking for an overall evaluation such
  as
• How would you the rate the quality of teaching in
  this post?

9
Overall satisfaction

• Questions scored from 0 100
• Mean taken (so long as four answered)
• Scale shown to be reliable
• Mean so long as 4 were answered (coded from 0
  100)
• Scale should be interpreted relatively (trainees
  at hospital/centre A more or less satisfied than
  trainees at hospital/centre B)
• Ceiling effect - e.g. 45 (N 24,882) said the
  post would be very useful to their future
  career
• 25th percentile is 76 so with generally high
  scores, ratings of fair are not particularly
  favourable

10
Satisfaction by specialty group - 1
11
Satisfaction by specialty group - 2

• Overall reported satisfaction is high
• GP trainees expressed greater satisfaction than
  those from other specialties
• Obs Gynae trainees expressed the least
  satisfaction

12
Satisfaction GPRs and Obs Gynae

• How would you describe this post to a friend who
  was thinking of applying for it?
• GPRs 2 (N 1,929) said Poor or Very Poor
• Obs Gynae 7 (N 1,815) said Poor or Very
  Poor

13
Which survey variables are associated with
overall satisfaction? - 1

• Multiple regression analysis using the surveys
  other scale scores was used to predict overall
  satisfaction
• The analysis statistically controlled for
• Method variance
• Specialty group, grade, sex, year of
  qualification and working in a different
  specialty group to the respondents intended
  specialty group
• Although strongly correlated with Overall
  Satisfaction, the Adequate Experience Score was
  not included in this analysis as the items did
  not demonstrate discriminant validity from
  Overall Satisfaction
• This may be because it is such an integral part
  of trainees evaluation of their post
• Or it could be because out of necessity the items
  were global ratings not specialty specific
  questions looking at facets of whether the
  experience is adequate

14
Which survey variables are associated with
overall satisfaction? - 2

• Supervision Score is most strongly related to
  Overall Satisfaction
• Workload Score is not related to Overall
  Satisfaction
• (R2 0.39, P lt 0.001, N 22,548)

15
Satisfaction and nations

• Including nation in the multiple regression model
  after controlling for
• Specialty group, grade, sex, year of
  qualification and working in a different
  specialty group to the respondents intended
  specialty group
• adds little (R2 change lt0.01)

16
Medical errors

• Used to demonstrate validity of other measures
  (not something used to compare training
  locations)
• Answer Yes or Yes more than one to either one
  or both of these questions (H07 and H08) to get a
  score of 1.
• In the last month have you made a serious medical
  error?
• In the last month, have you made a potentially
  serious medical error?

17
Medical errors

• Reporting of medical errors on this survey is
  related to how anonymous the respondent perceives
  the method of data collection to be

18
Medical errors by specialty
19
Which factors are associated with reporting
medical errors?

• Of the measures of trainees perceptions of
  adherence to PMETB standards Supervision was most
  strongly related to reporting making medical
  errors trainees reporting lower quality
  Supervision were more likely to report medical
  errors
• Workload was also strongly related those
  reporting a heavier workload were also more
  likely to report making medical errors
• Trainees qualifying in the UK were more likely to
  report medical errors

20
Supervision scale score

• These questions all relate to the trainees
  perception of their clinical supervision and
  relate to the patient safety domain of PMETBs
  Generic Standards, for instance
• How often, if ever, have you been supervised by
  someone who you feel isnt competent to do so?
• As before, items scaled from 0 to 100 and a mean
  calculated. A high score indicating good
  supervision
• Analysis has shown this is important as it is
  associated with overall satisfaction and
  reporting medical errors

21
Supervision by specialty group - 1
22
Supervision by specialty group - 2

• Pathology trainees are reporting higher quality
  supervision
• Surgical trainees are reporting the lowest
  quality supervision
• Surgical trainees also reported lower levels of
  Overall Satisfaction

23
Supervision and medical Error
24
Supervision and medical Error
25
Screening tool scores

• Adjusted for method variance first
• By specialty group
• More than three respondents at each location
  required for the mean to be calculated

26
Overall Satisfaction score and the screening tool
- using the data
27
Low on more than one Score. an example from
surgery

• Nationally two locations were lower than national
  mean on Overall Satisfaction and Supervision
  Scores
• Looking at one of these locations suggests
  trainees are expressing discontent
• 33 (4 out 12) would describe it as poor to a
  friend
• 33 are supervised by someone they feel isnt
  competent to do so (monthly and weekly combined)

28
European working time directive

• Some scores such Overall Satisfaction and
  Supervision are relative (i.e. their meaning only
  comes from comparisons) other scores derived
  from the survey can stand alone and be
  interpreted without comparison, for instance
• Have you been asked to submit hours that are
  compliant with the European Working Time
  Directive, when the hours you actually work are
  NOT compliant?

29
European working time directive

• 17 (N 21, 856, dont knows were excluded)
  answered yes to this question

30
Next steps - 1

• Each deanery has received their screening tool
  report (more reports and their own data to follow
  in the next few weeks)
• Deaneries have been asked to provide a written
  response to their report within 3 months
• The postgraduate deans will monitor progress on
  any actions proposed in the response
• PMETB will monitor progress on these actions on
  deanery wide visits

31
Next steps - 2

• PMETB will report on trainees stated intended
  specialties
• Initial analysis comparing the of SHOs
  intending to apply for a specialty against the
  of available posts (provisional figures only)
• GP under-subscribed
• Surgery and anaesthetics over-subscribed

32
Next steps - 3

• PMETB will publish a full report on this first
  survey, including the deaneries responses
• PMETB regard the survey data as one key source of
  information for and about the deaneries quality
  management work. Before taking action the
  following will need to be considered
• Further examination of the survey data for the
  given location
• Data from other sources
• Visits planned and trigger

33
The next trainee survey

• Specialty specific questions would enhance and
  refine our measurement of trainees perceptions
  of adherence to standards
• Deaneries will manage access to the web-survey to
  preserve trainee confidentiality
• The survey will be linked to data from trainer
  and manager surveys
• Information on the population of trainees
  (numbers per approved programme) will be
  requested from deaneries in advance of the survey
  itself

34
All surveys

• Web-based only for all three
• Access to the web-survey, including clarity in
  relation to appropriate sources of data, needs
  further consideration
• To achieve the links between the data from the 3
  surveys, they should be co-terminous
• PMETB and COPMeD will continue to respond to
  feedback