Innovations in SelfAssessment:

1
Innovations in Self-Assessment
A Cycle of Practice-Based Self-Improvement

• Bob Galbraith
• Steve Clyman
• Center for Innovation

2
These slides are for information only and are
not to be copied.
3
Center for Innovation

• Formed in 2001 to develop and test feasibility of
  new assessment approaches
• Needs assessment
• Scouting and Research
• Pilots (professionalism, psychomotor skills,
  real-time credentialling, clinical judgment,
  portfolios, work-based assessment)

4
The House of Assessment 2005
Amount/Intensity of Assessment
High Fidelity Simulation
Medical school Residency Practice
5
2010?
Amount/Intensity of Assessment
High Fidelity Simulation
Low Fidelity Simulation
Med school Residency Practice
6
Self-assessment

• Definition
• A proposed Cycle
• Why
• Why Now
• Goals
• Key Innovations
• Scope

7
Definition The View From Google

• Web Results 1 - 10 of about 1,650,000 for
  self-assessment medicine. (0.30 seconds)    
   Self Assessment CenterKnow yourself through
  online testsand gain success in life and
  work.www.selfassessmentcenter.com
• McGraw-Hill's AccessMedicine ... LANGE basic and
  clinical science resources plus the free LANGE
  self-assessment tool powered by ... -Atlas of
  Emergency Medicine -Clinical Anesthesiology
  ...www.accessmedicine.com/ - 20k - Cached -
  Similar pages
• Institute for Safe Medication Practices -
  Preventing Medication Errors 2005 ISMP Medication
  Safety Self Assessment for Antithrombotic
  Therapy in Hospitals is ... Watch your mail for
  the Self Assessment booklet and password.
  www.ismp.org/ - 29k - Nov 13, 2005 - Cached -
  Similar pages
• Self Assessment Colour Review of Hepatobiliary
  Medicine - Book ... Self Assessment Colour Review
  of Hepatobiliary Medicine ... The
  problem-oriented, randomised self-assessment
  format, lends itself to revision and continuing
  ...www.blackwellpublishing.com/book.asp?ref18745
  45480 - 15k - Cached - Similar pages

8
Definition Current Uses

• Test Preparation
• MCAT
• USMLE
• Board certification
• Mastery
• E-learning

9
Definition of Self-Assessment Emerging Uses

• CME
• Mastery
• Changes in practice process and outcomes
• Self-directed measurement of proficiency
• Self-paced
• Mastery or adaptive

10
Definition Essential Features of Self-Assessment

• Tailored to the needs of the individual
• Predicated on the practice profile
  Just-for-You
• Just-in-Time, point-of-service
• Targeted towards quality improvement
• Supported by learning material that is
  authoritative and accessible adaptively (SCORM)

11
A Proposed Self-Assessment Cycle
Self-Assessment
12
A Simplistic Example

• A physician pulls up monthly Practice Profile
• 20 increase in new type II diabetics over year
• Takes self-assessment based on Practice Profile
• Many questions on diabetes subscores show
  deficiency
• Undertakes a CME activity in type II diabetes
• Also reads a release on new medications
• Repeats self-assessment
• Improves score (noted in personal CME log)
• Reports results
• Medical Society, Specialty Board, State Medical
  Board

13
Why Do Self-Assessment?

• Identifies areas of weakness
• Is directly relevant to patients seen
• Enhances self-efficacy
• Supports self-improvement
• Promotes reflection
• Stimulates self-accountability
• Is not primarily punitive
• Consistent with other emerging assessments

14
Why Self-Assessment Now?

• Previously, comprehensive practice data not
  available. Now, relevant data becoming available
  electronically
• Previously, major methodological difficulties in
  building tests based on the practice profile.
  Now, technical innovations dissolving barriers
• The biggest hindrance to development of effective
  self-assessment may be a lack of purpose, and of
  innovation in the realization

15
What if Self-Assessment were Just-in-Time,
Just-for-You?

• Immediacy and Timeliness
• Physicians might seek relevant CME/CPD in real
  time at point-of-service
• Salience and Utility
• Could link actual patients seen with relevant
  advances in medical knowledge and practice
• Buy-in
• Physicians might move from passive to active
  consumption of CME/CPD and from reactive to
  proactive driving of quality improvement

16
Key Innovations Needed

• Content
• Format
• Usage

17
Key Innovations in Content

• Driven by Practice Profile
• Adapts to changing profile of patients seen
• Better tied to improving healthcare delivery
• Defines deficiencies, targets learning needs
• Tied closely to CME/CPD activities
• Tracks changes in knowledge and practice
• Results are key measure of CME efficacy

18
Key Innovations in Format

• Ability to score formats beyond MCQ
• e.g. fill in the blank and short essay items
• Flexible delivery
• Web-based - available any time, anywhere
• Self-paced and done in pieces
• Feedback available
• Expert Consults (through Medical Societies,
  Specialty Boards, Professional Organizations

19
Key Innovations in Usage

• Increasing use for credit
• For CME/CPD activities
• Partial satisfaction of MOC/MOL
• In assessment of Practice-Based Improvement
• For performance planning and improvement
• To inform pay-for-performance
• To assist in assessment of impaired physicians
• Enabling discounts on malpractice insurance

20
Necessary Steps to Implement the Self-Assessment
Cycle

• Systems to provide usable practice profiles
• Expanded availability of up-to-date MCQ
• Timely creation of MCQ for new CME content
• Tying tests to practice profiles
• Access to physician-friendly CME logs
• Assistance to physicians in finding new CME
• Web-based testing software
• Flexible (device-independent) delivery

21
Expanded Availability of Up-to -Date
Self-Assessment Content

• Increased item harvesting
• Item pooling across organizations

22
More Timely Creation of Items

• Development of fill in the blank and short
  answer item formats
• Possible computer scoring of free text
• Technically possible
• Computer requires training on each item
• Time-consuming and expensive

23
More Timely Creation of Items

• Progress towards computer generation through
  Natural language processing
• Find key concepts
• Rearrange narrative sentences into interrogatives
  (truefalse)
• Add of distractors yields MCQs

24
Computer-Generated Item Example
Source sentence The drugs most commonly
associated with the induction of acute episodes
of asthma are aspirin, coloring agents e.g.
tartrazine, b-adrenergic antagonists, and
sulfiting agents. Generated stem Which drug is
most commonly associated with the induction of
acute episodes of asthma? A) aspirin B)
atropine sulfate C) epinephrine D) mefenamic
acid E) spironolactone
25
Systems to Provide Practice Profiles

• Concatenation of data from several sources
• Billing data from payors
• Diagnostic codes from EHR systems
• Self-reported data
• Show to physicians for validation

26
Tying Tests to Practice ProfileIndividually-Tail
ored Tests

• Defining appropriate blueprints
• Software to generate tests from Practice Profiles
  
• Modular approach to building tests
• Aggregation of small, modular testlets
• Fully flexible blueprinting of tests
• Architect? (Professional Societies, Specialty
  Boards, individual physician)
• Systems

27
Discipline Specifications
28
Maximum Minimum Specifications
29
Results of Automated Test Assembly
30
Web-based testing software
31
Linkage to Other Data

• Practice Profile
• Personal work portfolios
• CME logs

32
Flexible (Device-Independent) Delivery

• Cell phones
• PDAs
• Response to short answer and fill in the blank
  options by voice recognition

33
House of Assessment Other Individualized
Measurement Thrusts

• Portfolios for learning and assessment
• Practice profiles and procedure logs
• Competency-based assessment
• Maintenance of Certification (MOC)
• Work-based assessment (e.g. pay-for-performance)
• Broadening the base of assessment
• Integration of education and assessment
• Doing assessment continuously
• Assessment of Individual versus team or
  microsystem
• High-fidelity simulations

34
Where Does Self-Assessment Fit?
Self-Assessment

• IndividualizedAssessment

Standardized Assessment
35
The Scope of Self-Assessment Broadening The Base
of Assessment
36
Scope of Self-Assessment Climbing Millers
Pyramid
DOES (in the real setting)
High fidelity simulations Standardized patients,
Psychomotor procedural skills, Assessment
portfolios
SHOWS HOW (in an observed, realistic setting)
KNOWS HOW (integrates information into
functional latticework)
Low fidelity simulations MCQs, oral
examinations, reflective portfolios
KNOWS (pieces of information)

37
Climbing Millers Pyramid - 2
Real-life observation 360 multi-source
feedback, medical records audit
DOES (in the real setting)
High fidelity simulations Standardized patients,
Psychomotor procedural skills, Assessment
portfolios
SHOWS HOW (in an observed, realistic setting)
KNOWS HOW (integrates information into
functional latticework)
Low fidelity simulations MCQs, oral
examinations, reflective portfolios
KNOWS (pieces of information)

38
Climbing Above Millers Pyramid
Process Measures
PHYSICIAN BEHAVIOR
AND KNOWLEDGE

Structural Measures

39
Summary

• The self-assessment cycle should be leveraged to
  measure what is relevant to day-to-day practice.
  This requires
• gearing to individual practice profiles
• relevance to CME/CPD
• A broader base than just MCQ e.g. adding process
  and/or outcome measures
• Inviting and supportive, not primarily punitive
• Non-intrusive and low-cost

40
Other Important Issues?

• Agreement among organizations in the House of
  Medicine to collectively develop and support such
  an approach
• Joint vision
• Collaborative pilots testing feasibility
• Interoperability, standards

41
Possible Next Steps

• Progressive incorporation of process and outcome
  measures
• More active role for physician in designing
  self-assessment, and in interpreting results
• Results might be considered in performance
  evaluations
• Results might contribute to new episodic
  summations (eg Practice-Based Improvement)
• Possible involvement of patients in
  self-assessment of behaviors and skills

42
Putative 2010 Headlines

• Desirable
• How Voluntary Self-Assessment Engaged Physicians
  and Improved Healthcare Delivery

Less Desirable Regulatory Coalition Enacts
Sweeping Expansion of Physician Assessment
Physicians Vote With Feet
43
Responsive Questions - Test

• Regardless of accuracy, which of the following
  associations with Chicago is the most enduring?
• City of Broad shoulders
• Vote early, vote often
• Remember Ms. OLearys cow
• Poor skating in summer
• Heres the Beef
• It is said that the great Chicago fire was
  started by Ms. OLearys cow kicking over a
  kerosene lamp?

44
Responsive Questions

• Self-assessment should be self-directed
• Self-assessment should be based on practice
  profiles
• The self-assessment cycle is an important concept
• The self-assessment cycle defines individual
  needs
• Quality self-assessment should improve CME
• We are using self-assessment in relation to CME
• We are already using practice-based
  self-assessment
• We plan to use self-assessment in relation to CME
• We plan to use practice-based self-assessment
• Physicians will seek practice-based
  self-assessment
• Physicians will pay for practice-based
  self-assessment