Measurement theory and provider profiling

1
Measurement theory and provider profiling

• Timothy P. Hofer MD

2
The measurement problem
3
Levels of care
4
Implications of the measurement model

• The indicator is a fallible measure of the
  construct
• Some indicators are less precise than others
• Quality indicators are very imprecise for a
  variety of reasons
• You need to account for the measurement error
• The location of the construct variability can
  suggest different causes, interventions and
  measurement procedures

5
Intra-class correlation(reliability)

• Ability to distinguish between physicians (or
  sites)
• single observation under a specified set of
  conditions of measurement.

6

Vol. 281 No. 22, pp. 2065-2160, June 9, 1999
7
MD laboratory utilization profiles
8
(No Transcript)
9
VA Network 11 Diabetes Care Project
Health Services Research Volume 37 Issue 5 Page 1159  - October 2002 doi10.1111/1475-6773.01102
 

Whom Should We Profile? Examining Diabetes Care Practice Variation among Primary Care Providers, Provider Groups, and Health Care Facilities
Sarah L. Krein, Timothy P. Hofer, Eve A. Kerr, and Rodney A. Hayward

10
Resources available

• VA Diabetes Registry Project (1998-2001)
• Automated Clinical Databases
• Data warehouse (VA Healthcare and analysis group)
• Database Components
• Encounter records (OPC/PTF )
• Outpatient Pharmacy
• Lab
• primary care provider database (PCMM ()
• Vitals
• Cohort identification procedure
• Data quality and measure validation
• Kerr EA , et al. Journal on Quality Improvement
  2002 28(10)555-65.

11
Selected MeasuresResource Use

• Cost of hypoglycemic medications
• Cost of home glucose monitoring for patients not
  on insulin
• Cost of calcium channel blockers

Processes
Outcomes
Intermediate Outcomes
12
Selected Measures Intermediate Outcomes

• Last A1c value
• A1c ? 9.5
• Last LDL value
• LDL ? 3.6 mmol/L (140mg/dl)

Processes
Outcomes
Intermediate Outcomes
13
Selected MeasuresProcess Measures

• Hemoglobin A1c obtained
• LDL-C obtained
• Lipid profile obtained

Processes
Outcomes
Intermediate Outcomes
14
Selected MeasuresMixed or Linked Measure

• LDL ? 3.6 mmol/L (140mg/dl) oron a statin

15
Are there differences between physicians?

• What are the sources of variation?
• Noise
• Unmeasured differences
• Physician effects
• Clinic or group effects
• Health System/payor effects

16
Outcomes
17
Intermediate outcomes
18
Process measures
19
Physician effect size
20
Physician effect size
Negligible
Small
Moderate
PCP Effect


200

Cost of homeglucose monitoring for patients not
on Insulin
Last LDL-C Value (1)
150

Last LDL-C value lt3.6 mmol/L or on a statin
(5)

100

Panel size
Hemoglobin A1c obtained (8)
50

Median PCP Panel size in study sample

0

.08

.10

.02
.04


Variance attributable to level of care

21
Implicit chart review site level

• Trained physician reviewers
• 621 records
• 26 clinical sites

22
Conclusions

• Measurement models are fundamentally important to
  measuring and profiling quality.
• There is often little reason or capability to
  profile at the physician level.
• Profiles that ignore measurement error
• Misrepresent the variability in quality
• Are difficult (or impossible) to validate

23
(No Transcript)
24
Example the imprecise thermometer

• Budget cuts inspire innovation in the clinic

25
Observed temperature
26
Observed vs. true temperature
27
Strength in numbers
105
100
Body temperature(F)
95
90
85
true
observed
average
28
Scale transformation
29
Reliability

• A person with one watch knows what time it is
• A person with two watches is never quite sure

30
Effect of gaming
31
(No Transcript)