A Process Approach to Outcome Measurement - PowerPoint PPT Presentation

1 / 29
About This Presentation
Title:

A Process Approach to Outcome Measurement

Description:

Learning style ... status delivered on the Learning Style questionnaire ... employed in this study was a modified version of Kolb's learning style inventory. ... – PowerPoint PPT presentation

Number of Views:21
Avg rating:3.0/5.0
Slides: 30
Provided by: sacm
Category:

less

Transcript and Presenter's Notes

Title: A Process Approach to Outcome Measurement


1
A Process Approach to Outcome Measurement
  • This presentation will probably involve audience
    discussion, which will create action items. Use
    PowerPoint to keep track of these action items
    during your presentation
  • In Slide Show, click on the right mouse button
  • Select Meeting Minder
  • Select the Action Items tab
  • Type in action items as they come up
  • Click OK to dismiss this box
  • This will automatically create an Action Item
    slide at the end of your presentation with your
    points entered.

2
A Process Approach to Outcome Measurement
  • Topics covered in this presentation
  • Design of the evaluation
  • Description of the participants
  • Knowledge findings
  • Participant learning styles
  • Instructor teaching styles
  • Intention to change practice findings
  • Motivation to change findings
  • Lessons learned

3
The Study
  • These data were developed at the 2006 Optimal
    Management of HIV Conference
  • My collaborators, Dr. Harold Kessler and Michael
    Saag, and I are in the process of analyzing these
    data for publication
  • The analyses I report here are preliminary are
    completed only for the purpose of these
    discussions

4
Two Elements Participant Descriptors and
Process Measurement
5
Data gathered
  • Learning style
  • Preference indicated by degree of endorsement of
    one or the other end of a dichotomy
  • Example
  • I prefer
  • Hands-on learning experience ...Learn
    ing through thinking and reasoning
  • Learning through simulation Learning
    through lectures

6
Data gathered (continued)
  • Demographics
  • Questions on age specialty and Board status
    delivered on the Learning Style questionnaire
  • Motivation to change
  • Motivation indicated by a modified standard scale
    of motivation to change
  • Example
  • Its important to use the new approaches Ive
    learned Strongly Agree - Strongly
    Disagree

7
Data gathered (continued)
  • Pre-Post Tests
  • Knowledge
  • Simple multiple choice knowledge based tests, one
    question per presenter per test, rotated to
    prevent order bias
  • Example
  • Which of the following antiretrovirals is the
    least likely to cause DSPN
  • indinavir
  • zalcitabine
  • stavudine
  • didanosine
  • Laminvudine

8
Data gathered (continued)
  • Pre-Post Tests (continued)
  • Intent to change
  • Procedural intent questions provided prior and
    post the session
  • Example
  • Pre
  • When treating HIV patients with PCP I use
    Strongly Agree - Strongly Disagree
  • TMP-SMX or TMP-dapsone
  • Post
  • When treating HIV patients with PCP I
    Strongly Agree-Strongly Disagree
  • intend to use.

9
Data gathered (continued)
  • Teaching style
  • 3-5 observers rated each session using simple
    descriptors of the presentation using agreement
    or disagreement with the description
  • Example
  • The Presentation focused on clinical application
    Strongly Agree - Strongly
    Disagree
  • The presenter focused on the underlying science
    of medicine Strongly Agree - Strongly
    Disagree

10
Description of the participants
  • Of the total participants in the program,
    analysis is focused on the 62 complete datasets,
    these participants provided data for all three
    days both in the morning and afternoon
  • While it is possible that they differed in some
    systematic way from the rest of the participants,
    complete data will be necessary for the ultimate
    analysis

11
Results Participants were generally middle aged
  • The participants had an average age of 49
  • The youngest was 29 and the oldest 73
  • The median age was 47 indicating a slight skew to
    the younger side of 49.

12
The average participant graduated from medical
school in 1984
  • The youngest participant graduated in 2003
  • The oldest participant graduated in 1958

13
Most participants were in infectious disease or
general internal medicine
Specialty Percentage
Family Medicine 36
Infectious Disease 24
Internal Medicine 28
Missing 6
Pediatrics 4
Psychiatry 2
14
Most of the participants were ABMS Board
Certificants
  • 62 of participants reported Certificant status
    for a Board
  • 37 were not members of a Board or did not
    respond.

15
The most frequent Boards of participants were
Infectious Disease or Internal medicine
Category Relative frequency per category ()
AAHIVS 3.3
Family Medicine 33.3
Infectious Disease 26.6
Internal Medicine 30.0
Pediatrics 3.3
Psychiatry 3.3
16
The participants improved their performance
  • Participants improved their performance from pre
    to post measures
  • The improvement was approximately 10.
  • The improvement was significant p lt .001.

17
17
Participants also showed an expectation of
changing practice patterns
  • Intention or expectation of changing practice
    patterns was measured by asking the likelihood of
    a particular practice being adopted
  • The likelihood was measured prior to the session
    (Morning) and after the session (Afternoon).
  • The stated likelihood increased from 3.4 to 4 on
    a 5 point scale.
  • The effect was assessed and the increase in
    likelihood is significant plt.001.

18
Learning style was assessed a modification of a
scale developed by Kolb
  • The approach to learning style that was employed
    in this study was a modified version of Kolbs
    learning style inventory.
  • We employed his core scale, adapting it from a
    dichotomy to a numeric format with a Kolb
    descriptor on each end of the scale.

19
Teaching style was assessed by an observer
assessment scale
  • The items were selected to reflect the teaching
    characteristics for the Kolb learning styles
  • During each presentation, 3-5 observers assessed
    the style of each presenter.

20
Motivation to change
  • Participants were asked to endorse 30 statements
    of change commitment
  • The statements were modified from the original
    focus to reflect motivation and readiness to
    learn new information and make changes in
    clinical practice patterns.

21
A Process Approach to Outcome Measurement
  • Outcome measurement
  • Direct assessment/measurement of actual practice
    change or improvement in patient health status is
    difficult
  • Focus is on valid secondary measures such as
  • Self-report or,
  • Vignettes.
  • The core question for CME measurement
    professionals is the validity of such secondary
    measures.
  • Assessment validity is typically based on a
    combination of
  • Face validity (is it reasonable)
  • Construct validity (does it render a measurement
    correlated with some other measure deemed valid)
    and,
  • Predictive validity.

22
First we looked at the acquisition of knowledge
  • We used the teaching style judgment questions as
    indicators of an underlying set of predominate
    teaching styles
  • We then used the averaged teaching style scores
    for each instructor as predictors of knowledge
    acquisition of the participants.

23
Teaching and Knowledge Acquisition
  • We looked at the relationship between Teaching
    Style and Knowledge
  • We found a significant relationship between the
    two measures
  • The unobserved variable Teaching Style explains
    approximately 25 of the variance in Knowledge.

24
We looked at the formation of an intent to change
  • First, we used the learning style questions as
    indicators of an underlying set of learning
    preferences
  • We then used those preferences as predictors of
    intent to change practice patterns

25
Learning Style, Knowledge and Intent to Change
Practice
  • We looked at the relationship between Learning
    Style, Knowledge and Intent to change practice
    patterns
  • We found a significant relationship between the
    two causal measures and Intent
  • The unobserved variable Learning Style explains
    approximately 13 of the variance in Intent
  • The variable Knowledge explains approximately 55
    of the variance in Intent.

26
Self report of practice change
  • First we looked at the relationship between
    Motivation to change and reported practice
    change
  • Motivation to change was assessed using a series
    of readiness for change items
  • Change items were taken as indicators of an
    underlying state of change readiness.

27
Motivation and Intent to Change Practice and
Self-reported Practice
  • We looked at the relationship between Motivation
    to change, Intent to change practice patterns and
    self-report of practice patterns at a 12 month
    follow-up
  • We found a significant relationship between the
    two causal measures and self-report practice
    change
  • We also found a relationship between motivation
    and intent to change.

28
A Path Model of Cognitive Elements of CME Efficacy
  • We looked at the relationships among a number of
    variables as they relate to practice change. We
    found
  • Teaching Style has a path to Knowledge
  • Knowledge and Learning Style have paths to Intent
    to Change
  • Motivation to Change and Intent to Change have
    paths to Self-reported Practice Change.

29
Discussion
  • All of these variables are related in a causal
    manner with self-reported practice change
  • Each variable has a theoretical reason to be
    thought of as causally related to practice
    change
  • These data suggest that there are multiple
    variables that could provide a meaningful
    estimate of the degree of efficacy of the CME
    program.
Write a Comment
User Comments (0)
About PowerShow.com