EvidenceBased Education: Problems and Solutions

1
Evidence-Based EducationProblems and Solutions

• Yonah Yaphe MD
• Department of Family Medicine
• Rabin Medical Centre

2
Evidence-Based Education Network

• Evidence as a basis for policy and practice in
  education
• Evidence that tell us what works comes from
  trying it out
• From well-designed, well-evaluated, controlled
  trials.

3
The Learning PyramidRetention of material by
teaching methodNational Training Laboratories,
Bethel, Maine, USA
Lecture 5
Reading 10
Audio-visual 20
Demonstration 30
Discussion group 50
Practice by doing 75
Teaching others 80
4
Improving prescribing(from Soumerai, 1990)

• printed material (-)
• patient specific drug lists (-)
• repeated feedback ()
• face to face outreach ()

5
Clinicians need Information

• If asked
• we need it twice a week,
• we get it from our text books journals.

6
Clinicians really need information!

• If shadowed
• we need it up to 60 times per week (twice per
  every three patients), and it could affect eight
  decisions per day.
• but we get only 30 of it,
• and that comes from passers-by
• my textbooks are out of date
• my journals too disorganised

7
Our textbooks are out-of-date

• Fail to recommend Rx up to ten years after its
  been shown to be efficacious.
• Continue to recommend therapy up to ten years
  after its been shown to be useless.

8
Time spent reading around ones patients is slim

• Self-reports from Oxford (medians)
• Medical Students 60 minutes per week
• House Officers none
• S.H.O.s 10 minutes
• Registrars 90 minutes
• Senior Registrars 45 minutes
• Consultants
• Post 1975 60 minutes
• Pre 1975 30 minutes

9
The inevitable consequence

• On average, the clinically-important knowledge of
  physicians deteriorates rapidly after we complete
  our training.

10
No wonder, then, that CME is mushrooming

• Big, and getting huge.
• Usually instructionally (fact) oriented.

11
Study 1 An RCT of CME

• We gave the list of clinical conditions to a
  random sample of physicians, and asked them to
  identify
• ones for which they really DID want to receive
  CME (high preference).
• ones for which they really did NOT want to
  receive CME (low preference).
• hid a few away for later use.

12
An RCT of CME

• Clinicians with similar preferences were
  randomised into
• an Experimental Group (who would receive CME now
  for high preference conditions if they agreed
  to study low preference conditions, too).
• a Control Group (who would receive CME later).

13
An RCT of CME

• State-of-the-art CME instructional packages
  were developed
• full of objectives.
• in both written and audio formats.
• individualised, carry-away packages.
• frequent examinations for mastery
• with substantial gains in knowledge

14
An RCT of CME

• Then measured the quality-of-care provided
  high-preference, low-preference, and hidden
  indicator conditions
• in both experimental and control practices
• both before and after the former group received
  their CME

15
An RCT of CME High Preference Conditions

• Quality of care rose slightly (statistically, but
  not clinically significantly) in the Experimental
  Practices
• An identical rise was observed in Control
  Practices !
• (If you want CME, you dont need it!)

16
An RCT of CME Low Preference Conditions

• Quality of care rose substantially in
  Experimental Practices.
• Quality of care declined slightly in Control
  Practices.
• (CME only works if you dont want it!)

17
An RCT of CME Hidden Conditions

• Quality of care deteriorated slightly in both
  Experimental and Control practices.
• CME does not cause general improvements in the
  quality of care.

18
Running Summary

• Clinicians need information, but most of our
  needs are never met
• Our textbooks are out of date.
• Our journals are disorganised.
• Consequently, our knowledge and performance
  deteriorate.
• And traditional instructional CME doesnt improve
  our performance.

19
Three solutions

• Clinical performance can keep up to date
• by learning how to practice evidence-based
  medicine ourselves.
• by seeking and applying evidence-based medical
  summaries generated by others.
• by accepting evidence-based practice protocols
  developed by our colleagues.

20
Study 2 Short term Evidence among Clinical
Clerks A Trial

• Experimental clerks worked with Clinical Tutors
  whod taken a crash course in critical appraisal
  and had worked up diagnostic tests and treatments
  bound to arise in their clerkship.
• Control clerks worked with usual Clinical Tutors.

21
Short term Evidence among Clinical Clerks A Trial

• Before and after the clerkship, both sets of
  clerks were given patient scenarios
• describing the patients clinical problem
• calling for diagnostic and treatment decisions
• accompanied by a clinical article advocating a
  specific diagnostic test or treatment for such
  patients.

22
Short term Evidence among Clinical Clerks A Trial

• After an evidence-based clerkship, Experimental
  Clerks made more correct decisions, and were
  better able to justify them.
• Control Clerks deteriorated, and were more likely
  to be wrong after their clerkship than before it!
• they had become more accepting of recommendations
  from authority figures.

23
Study 3 Does PBL make medical students more
competent after they're out in practice for a few
years?

• Random samples (N48 for each) of GPs who
  graduated from a PBL school (McMaster University,
  Hamilton, Ontario) and a traditional school (U of
  Toronto, Toronto, Ontario),
• Stratified for year of graduation (1974-85) and
  sex
• Sent questionnaires with 52 questions about how
  to measure blood pressure, which levels to treat,
  pharmacological and non-pharmacological
  treatment, detecting and managing low compliance,
  and follow-up.
• Response rates 87 in both cohorts.
• Result Problem-based learning graduates are
  more up to date 15 years later (in managing
  hypertension) than graduates of traditional
  medical schools

24
Study 4 Printed educational materials effects on
professional practice and health care outcomes

• The effects of printed educational materials
  compared with no active intervention appear small
  and of uncertain clinical significance.
• Eleven studies involving more than 1848
  physicians
• (Cochrane Review)

25
Study 5 Review of CME

• Davis D, Thomson O'Brien MA, Freemantle N, et al.
  Impact of formal continuing medical education.
• Do conferences, workshops, rounds, and other
  traditional continuing education activities
  change physician behavior or health care
  outcomes?
• JAMA. 1999 Sep 1282867-74.

26
Main results

• 64 studies.
• 14 studies met selection criteria.
• A meta-analysis that included only interactive
  and mixed CME interventions showed an effect on
  physician performance (standardized WMD 0.67, CI
  0.01 to 1.45).
• None of the 4 didactic CME interventions altered
  physician performance.
• No association between intervention group size
  and positive outcomes.
• Interactive, but not didactic, continuing medical
  education is effective in changing physician
  performance

27
Study 6Adolescent Health Care

• Evaluation of the effectiveness of an educational
  intervention for general practitioners in
  adolescent health care randomised controlled
  trial
• L A Sanci et al. BMJ 2000320224-230
• Sustainable improvements in knowledge, skill, and
  self perceived competency.

28
Instructional design

• Needs analysis
• From previous surveys and informally at start of
  workshops
• Primary educational strategy
• Workshops for 2.5 hours weekly for six weeks
  
• Course book
• Resource book
• Reading material expanding on workshop sessions
• Practice reinforcing and enabling strategies

29
Workshops

• Debriefing from previous session
• Brief didactic overviews
• Group problem based activities and discussion
  
• Modelling of interview skills on video
• Role play and feedback practice sessions with
  adolescent actors
• Activities set to practise in intervening week
  
• Individual feedback on pre-course evaluation
  video

30
Course book

• Goals, objectives, course requirements, and notes
  
• Suggested further reading
• Class or home activities with rationale for each

31
Practice reinforcing and enabling strategies

• Adolescent assessment chart for patient audit
• Logbook for reflection on experience with the
  patients audited
• Self assembled list of adolescent health
  services in local community
• Availabilty of tutor (LS) by phone for
  professional support between workshops
• Refresher session for group discussion of
  experiences in practice (six weeks after course)

32
Conclusions

• Active learning methods work
• Tailor programs to learning needs
• Keep lectures to a minimum
• Reinforce learned skills

33
Conclusion

• Our educational methods can and should be as
  evidence-based as our diagnostic and therapeutic
  methods.