Title: A Process Approach to Outcome Measurement
1A Process Approach to Outcome Measurement
- This presentation will probably involve audience
discussion, which will create action items. Use
PowerPoint to keep track of these action items
during your presentation - In Slide Show, click on the right mouse button
- Select Meeting Minder
- Select the Action Items tab
- Type in action items as they come up
- Click OK to dismiss this box
- This will automatically create an Action Item
slide at the end of your presentation with your
points entered.
2A Process Approach to Outcome Measurement
- Topics covered in this presentation
- Design of the evaluation
- Description of the participants
- Knowledge findings
- Participant learning styles
- Instructor teaching styles
- Intention to change practice findings
- Motivation to change findings
- Lessons learned
3The Study
- These data were developed at the 2006 Optimal
Management of HIV Conference - My collaborators, Dr. Harold Kessler and Michael
Saag, and I are in the process of analyzing these
data for publication - The analyses I report here are preliminary are
completed only for the purpose of these
discussions
4Two Elements Participant Descriptors and
Process Measurement
5Data gathered
- Learning style
- Preference indicated by degree of endorsement of
one or the other end of a dichotomy - Example
- I prefer
- Hands-on learning experience ...Learn
ing through thinking and reasoning - Learning through simulation Learning
through lectures
6Data gathered (continued)
- Demographics
- Questions on age specialty and Board status
delivered on the Learning Style questionnaire - Motivation to change
- Motivation indicated by a modified standard scale
of motivation to change - Example
- Its important to use the new approaches Ive
learned Strongly Agree - Strongly
Disagree
7Data gathered (continued)
- Pre-Post Tests
- Knowledge
- Simple multiple choice knowledge based tests, one
question per presenter per test, rotated to
prevent order bias - Example
- Which of the following antiretrovirals is the
least likely to cause DSPN - indinavir
- zalcitabine
- stavudine
- didanosine
- Laminvudine
8Data gathered (continued)
- Pre-Post Tests (continued)
- Intent to change
- Procedural intent questions provided prior and
post the session - Example
- Pre
- When treating HIV patients with PCP I use
Strongly Agree - Strongly Disagree - TMP-SMX or TMP-dapsone
- Post
- When treating HIV patients with PCP I
Strongly Agree-Strongly Disagree - intend to use.
9Data gathered (continued)
- Teaching style
- 3-5 observers rated each session using simple
descriptors of the presentation using agreement
or disagreement with the description - Example
- The Presentation focused on clinical application
Strongly Agree - Strongly
Disagree - The presenter focused on the underlying science
of medicine Strongly Agree - Strongly
Disagree
10Description of the participants
- Of the total participants in the program,
analysis is focused on the 62 complete datasets,
these participants provided data for all three
days both in the morning and afternoon - While it is possible that they differed in some
systematic way from the rest of the participants,
complete data will be necessary for the ultimate
analysis
11Results Participants were generally middle aged
- The participants had an average age of 49
- The youngest was 29 and the oldest 73
- The median age was 47 indicating a slight skew to
the younger side of 49.
12The average participant graduated from medical
school in 1984
- The youngest participant graduated in 2003
- The oldest participant graduated in 1958
13Most participants were in infectious disease or
general internal medicine
Specialty Percentage
Family Medicine 36
Infectious Disease 24
Internal Medicine 28
Missing 6
Pediatrics 4
Psychiatry 2
14Most of the participants were ABMS Board
Certificants
- 62 of participants reported Certificant status
for a Board - 37 were not members of a Board or did not
respond.
15The most frequent Boards of participants were
Infectious Disease or Internal medicine
Category Relative frequency per category ()
AAHIVS 3.3
Family Medicine 33.3
Infectious Disease 26.6
Internal Medicine 30.0
Pediatrics 3.3
Psychiatry 3.3
16The participants improved their performance
- Participants improved their performance from pre
to post measures - The improvement was approximately 10.
- The improvement was significant p lt .001.
17
17Participants also showed an expectation of
changing practice patterns
- Intention or expectation of changing practice
patterns was measured by asking the likelihood of
a particular practice being adopted - The likelihood was measured prior to the session
(Morning) and after the session (Afternoon). - The stated likelihood increased from 3.4 to 4 on
a 5 point scale. - The effect was assessed and the increase in
likelihood is significant plt.001.
18Learning style was assessed a modification of a
scale developed by Kolb
- The approach to learning style that was employed
in this study was a modified version of Kolbs
learning style inventory. - We employed his core scale, adapting it from a
dichotomy to a numeric format with a Kolb
descriptor on each end of the scale.
19Teaching style was assessed by an observer
assessment scale
- The items were selected to reflect the teaching
characteristics for the Kolb learning styles - During each presentation, 3-5 observers assessed
the style of each presenter.
20Motivation to change
- Participants were asked to endorse 30 statements
of change commitment - The statements were modified from the original
focus to reflect motivation and readiness to
learn new information and make changes in
clinical practice patterns.
21A Process Approach to Outcome Measurement
- Outcome measurement
- Direct assessment/measurement of actual practice
change or improvement in patient health status is
difficult - Focus is on valid secondary measures such as
- Self-report or,
- Vignettes.
- The core question for CME measurement
professionals is the validity of such secondary
measures. - Assessment validity is typically based on a
combination of - Face validity (is it reasonable)
- Construct validity (does it render a measurement
correlated with some other measure deemed valid)
and, - Predictive validity.
22First we looked at the acquisition of knowledge
- We used the teaching style judgment questions as
indicators of an underlying set of predominate
teaching styles - We then used the averaged teaching style scores
for each instructor as predictors of knowledge
acquisition of the participants.
23Teaching and Knowledge Acquisition
- We looked at the relationship between Teaching
Style and Knowledge - We found a significant relationship between the
two measures - The unobserved variable Teaching Style explains
approximately 25 of the variance in Knowledge.
24We looked at the formation of an intent to change
- First, we used the learning style questions as
indicators of an underlying set of learning
preferences - We then used those preferences as predictors of
intent to change practice patterns
25Learning Style, Knowledge and Intent to Change
Practice
- We looked at the relationship between Learning
Style, Knowledge and Intent to change practice
patterns - We found a significant relationship between the
two causal measures and Intent - The unobserved variable Learning Style explains
approximately 13 of the variance in Intent - The variable Knowledge explains approximately 55
of the variance in Intent.
26Self report of practice change
- First we looked at the relationship between
Motivation to change and reported practice
change - Motivation to change was assessed using a series
of readiness for change items - Change items were taken as indicators of an
underlying state of change readiness.
27Motivation and Intent to Change Practice and
Self-reported Practice
- We looked at the relationship between Motivation
to change, Intent to change practice patterns and
self-report of practice patterns at a 12 month
follow-up - We found a significant relationship between the
two causal measures and self-report practice
change - We also found a relationship between motivation
and intent to change.
28A Path Model of Cognitive Elements of CME Efficacy
- We looked at the relationships among a number of
variables as they relate to practice change. We
found - Teaching Style has a path to Knowledge
- Knowledge and Learning Style have paths to Intent
to Change - Motivation to Change and Intent to Change have
paths to Self-reported Practice Change.
29Discussion
- All of these variables are related in a causal
manner with self-reported practice change - Each variable has a theoretical reason to be
thought of as causally related to practice
change - These data suggest that there are multiple
variables that could provide a meaningful
estimate of the degree of efficacy of the CME
program.