SelfAssessment within a LearnerPrescribed CMECE Curriculum

1
Self-Assessment within a Learner-Prescribed
CME/CE Curriculum

• Destry Sulkes M.D., Vice President, Educational
  Services
• Medscape LLC

2
Acknowledgements
We wish to acknowledge the following for their
contributions and hard work

• From We Move
• Judy Blazer
• From Outcomes, Inc.
• Mazi Abdolrasulnia, PhD
• Linda Casebeer, PhD
• Hamid Doroodchi, MD
• Nancy Roepke, MBA
• From Medscape LLC
• Martin Irvine, PhD

3
Disclosure

• This activity was made possible by an
  unrestricted educational grant from
  Boehringer-Ingelheim Pharmaceuticals, Inc.

4
Sometimes things arent quite what they seem
5
Starting Point

• CME has been shown to improve knowledge, skills
  (cognitive) and attitude1
• Self-assessments have been shown to be inaccurate
  when not externally informed2
• Feedback from externally informed
  self-assessments, provided to both clinicians and
  Faculty, can be used to increase effectiveness of
  CME3

• Evid Rep Technol Assess (Full Rep). 2007
  Jan(149)1-69. Effectiveness of continuing
  medical education. Marinopoulos SS, Dorman T,
  Ratanawongsa N, Wilson LM, Ashar BH, Magaziner
  JL, Miller RG, Thomas PA, Prokopowicz GP, Qayyum
  R, Bass EB.
• JAMA. 2006 Sep 6296(9)1094-102. Accuracy of
  physician self-assessment compared with observed
  measures of competence a systematic review.
  Davis DA, Mazmanian PE, Fordis M, Van Harrison R,
  Thorpe KE, Perrier L.
• Med Teach. 2008 Mar30(2)124-45. The
  effectiveness of self-assessment on the
  identification of learner needs, learner
  activity, and impact on clinical practice BEME
  Guide no. 10. Colthart I, Bagnall G, Evans A,
  Allbutt H, Haig A, Illing J, McKinstry B. NHS
  Education for Scotland.

6
Current Model of Online Medical Education

• Medical education programs for healthcare
  professionals are designed using broad needs
  assessments across target audiences
• Individual activities within courses are not
  typically developed for specific segments of
  learners
• The courses clinicians select to maintain their
  professional licensure may or may not be
  appropriate to address gaps in specific
  individuals clinical knowledge, skills and
  attitude (K/S/A)

7
Hypothesis for an Externally Informed Self
Assessment

• Learners who participate in an externally
  informed self-assessment and engage in the
  corresponding prescribed educational activities
  are more likely to
• Engage in the educational activities
• Enhance their K/S/A relative to those who find
  the activities in an undirected fashion

8
Methodology

• To address K/S/A gaps identified through a needs
  assessment of Medscape members, an online CME/CE
  curriculum of activities was developed in
  collaboration with We Move and Outcomes Inc
• One part of the curriculum is a CME/CE-certified
  self-assessment activity that measures K/S/A
  related to recognition, diagnosis, and
  non-pharmacologic and pharmacologic management of
  RLS
• The online self-assessment provides automated
  activity recommendations based on the learners
  responses to K/S/A questions
• Post-assessments follow each activity to assess
  the relative efficacy of this new educational
  approach vs the standard, undirected approach, to
  improving clinical management of RLS

9
Course Formats/Titles/Focus
prescribed courses
10
Curriculum Design
11
Technical Platform Supporting Individualized
Feedback

• Feedback
• What
• Each learner receives an email listing the
  recommended activities based on the question
  responses provided in the self-assessment.
• How
• Each night, a script that queries the database
  for any new self-assessment participants.
• The script evaluates each participants answer
  choices and maps the choices to the recommended
  CME activities.
• The data is fed to an email application where it
  is formatted and sent to each participant with
  individualized activity recommendations, plus a
  link to the curriculum page.

12
Sample immediate feedback / email message
13
Process Schematic
Prescribed Learning Learner sent
recommendation from Externally Informed Self
Assessment
Automated response directing learner to specific
K, S, or A course
Post Test A1 Includes mapped Attitude questions
from Self Assessment
Attitude Course 1
Knowledge Course 1
Post Test K1 Includes mapped Knowledge
questions from Self Assessment
Learner
Undirected Learning Learner finds curriculum
and/or individual activities by standard search
criteria
Skill Course 1
Post Test S1 Includes mapped Skills questions
from Self Assessment
14
Datasets and Minimums Required for Analysis
Prescribed Post Assessment A1 N100
Undirected Post Assessment A N 1400 over 7
activities
Undirected Post Assessment A1 N100
Non-Medscape Membership Baseline
Survey N400 (Identical to E-I self-assessment)
Externally Informed Self-Assessment on
Medscape N400 (Identical to baseline survey)
Undirected Post Assessment K N 1400 over 7
activities
Prescribed Post Assessment K1 N100
Undirected Post Assessment K1 N100
Prescribed Post Assessment S1 N100
Undirected Post Assessment S1 N100
Undirected Post Assessment S N 1400 over 7
activities
Outcomes 1 Measures the courses prescribed vs.
the courses actually completed in a three month
time period, click-thru rate of A1/K1/S1
prescriptions that resulted in accessing the
activities
Outcomes 2 n100 Measures the Prescribed Post
Assessment to the E. I. Self Assessment by
Knowledge, Skills and Attitude
Outcomes 3 n100 Measures the Prescribed Post
Assessment to the Baseline Survey by Knowledge,
Skills and Attitude
Outcomes 4 n100 Measures the Prescribed Post
Assessment to the Undirected Post Assessment by
Knowledge, Skills and Attitude
Outcomes 5 n100 Measures the Undirected Post
Assessment (A1, K1, S1) to the Baseline survey by
Knowledge, Skills and Attitude
Outcomes 6 n200 Measures the Undirected Post
Assessment (4-10) to the Baseline Survey by
Knowledge, Skills, Attitude
15
Learnings to date

• K/S/A model helpful and could benefit from
  further refinement incorporating established
  competencies
• Email feedback is somewhat effective, but
  Medscape learners access activities primarily
  from their specialty homepage
• 65 specialty homepage
• 25 outbound email with editorial focus (note
  different than the prescriptive emails in this
  initiative)
• 10 external search
• Recognition that current CME/CE activity
  tagging for site-location and rotation purposes
  can be enhanced to include competencies, thus
  allowing all activities (gt3000 in 2007) to be
  targeted as needed in the future, perhaps on
  learners specialty homepages, and perhaps
  without explicitly communicating the targeting