Performance Monitoring

1
Performance Monitoring

• Day 4 EM Analyst Training Course 02.10.12

2
Learning Objectives

• At the end of the course, participants will be
  able to ...
• explain what is meant by performance monitoring
• identify the drivers, purpose and applications of
  performance monitoring
• illustrate how performance monitoring is
  undertaken in practice

3
Performance monitoring

• Executives worth their salt will be asking ...
• are we achieving our objectives?
• are we meeting our targets?
• what should our priorities be?
• whats our strategy for ....?
• what changes do we need to make to local
  services?
• where and how should we target our resources?
• are our initiatives and service changes having
  the intended effect?

4
Drivers and purpose

• Demand for accountability of public services
  Clinical governance.
• Quantifying and qualifying the results of
  surveillance.
• Encourage best practice identify opportunities
  for development and shared learning.
• Clarify an organisations objectives and
  priorities.
• Measuring progress against targets.

5
ASSESSING PERFORMANCE
6
Monitoring and evaluating play an important role
within the NHS

• Definition?
• Measuring and comparing observed data with a
  benchmark or baseline
• Comparisons can be
• over time (e.g. one year to the next)
• between areas (e.g. local authorities)
• against a target (e.g. 98)
• against an expected value

7
Choosing an indicatorThe good indicators guide
A poorly designed indicator with reliable
data, or a well designed indicator with
unreliable and/or untimely data, has very little
value (and is sometimes dangerous).
8
The good indicators guide10 key questions

1. What is being measured?
2. Why is it being measured?
3. How is the indicator defined?
4. Who does it measure?
5. When does it measure it?
6. Will it measure absolute numbers or proportions?
7. Where do the data actually come from?
8. How accurate and complete will the data be?
9. Are there any caveats/warnings/problems?
10. Are particular tests needed (e.g.
    standardisation, significance tests or SPC) to
    test the meaning of the data and the variation
    they show?

9
The good indicators guide

• Selection of indicators and the data that feed
  into them will often involve a trade off between
  what is convenient (and possible) and what is
  ideal.
• The right indicator at the right time and in the
  right context can stimulate huge change where it
  might have no effect in other contexts and at
  other times. Change hearts and minds
• Try not just to present the data look for the
  story it is telling.

10
Does this tell you enough?
Sources Office for National Statistics and
Teenage Pregnancy Unit
11
Comparators give context
Sources Office for National Statistics and
Teenage Pregnancy Unit
12
Ways of evaluating performance

• Spine chart
• Tartan rug
• Benchmarking
• Statistical Process Control
• Control and run charts
• Funnel plots

13
Ways of evaluating performance
Number of indicators Many/one Number of organisations Many/one Over time? Yes/no
Spine chart
Tartan rug
Benchmarking
Statistical Process Control (SPC) Control chart / Run chart
SPC funnel plot
14
Spine Chart
So you want to make a spine chart? http//www.wmph
o.org.uk/tools/ ERPHO Spine chart
video http//www.youtube.com/watch?v480Mswgcg8Mf
eatureplcp
15
Tartan rug
16
Benchmarking

• Contextualise performance
• Encourages best practice
• Shared learning and areas for development

17
Benchmarking

• Which local authority would you say had an issue
  with perinatal mortality? Can you think of any
  issues regarding presenting data in a league
  table such as this?

East of England average
Confidence interval
Perinatal mortality rate by local authority, East
of England 20012005, INpho 26
18
Statistical Process Control

• common cause variation
• the normal, everyday, inevitable (and usually
  unimportant), variation which is intrinsic and
  natural to any system
• special cause variation
• the more important variation which is indicative
  of something special happening and which calls
  for a fuller understanding and often action
• For more info see APHO technical briefing
• http//www.apho.org.uk/resource/view.aspx?RID3930
  6

19
Types of control chart

• Run chart displays any data over time against
  its median
• Control chart looks similar to a run chart but
  plots data against its mean and the appropriate
  control limits
• Good for monitoring trends over time
• Good for monitoring data against a specific
  target
• AE 4 hour waits
• Childhood obesity
• Control charts explained to music
• http//www.youtube.com/watch?v1XM3DpyBHss

20
Example of a control chart

• Control charts are plots of the data with lines
  indicating the target value (mean, median) and
  control limits superimposed.

21
Types of control chart con..

• Funnel plots
• A type of chart where the indicator of interest
  is plotted against the denominator or sample
  size.
• This gives it the characteristic funnel shape
• ERPHO funnel plot video http//www.youtube.com/wat
  ch?v480Mswgcg8Mfeatureplcp
• So you want to create a funnel plot?
• APHO provides tools for creating funnel plots for
  different kinds of data
• http//www.apho.org.uk/default.aspx?RID39403

22
Example of funnel plot
23
Using control charts and SPC methods

• Control charts can help us to present and
  interpret our information more intelligently.
• They can be used
• To detect unusual or outlying patterns, e.g. poor
  performance, outbreaks or unusual patterns of
  disease
• In health profiling and assessing levels of
  performance
• To decide whether or not targets are being met
• In assessing health inequalities (i.e. funnel
  plots)

24
Control chart AE 4 hour waitsChange in target
Green line target Red lines CIs
25
Funnel plotPerinatal Mortality
26
Breast cancer mortality
APHO technical briefing 2 Statistical process
control methods in public health intelligence
27
Lung cancer mortality
APHO technical briefing 2 Statistical process
control methods in public health intelligence
28
Summary

• Good monitoring and benchmarking requires use of
  comparators
• Good data visualisation is helpful
• Statistical process control allows you to
  distinguish between common cause variation and
  special cause variation, which might indicate a
  problem or an exceptional performance within the
  system.

29
Further information

• The Good Indicator Guide how to use and choose
  performance indicators http//www.apho.org.uk/reso
  urce/item.aspx?RID44584
• APHO Technical Briefings
• http//www.apho.org.uk/resource/view.aspx?RID3930
  6
• No. 2 Statistical process control methods in
  public health intelligence
• No. 3 Commonly used public health statistics
  their confidence intervals
• No. 4 Target setting in a multi-agency
  environment
• Health Profiles http//www.healthprofiles.info
• WMPHO spine chart generator http//www.wmpho.org.
  uk/resources/WMPHO_Spine_chart.xlsm
• ERPHO tool to produce run charts (you may need to
  log onto the ERPHO website) http//www.erpho.org.u
  k/viewResource.aspx?id14762
• Create a p chart in excel
• http//www.ehow.com/how_8039012_create-chart-using
  -samples.html

30
Learning Objectives

• At the end of the course, participants will be
  able to ...
• explain what is meant by performance monitoring
• identify the drivers, purpose and applications of
  performance monitoring
• illustrate how performance monitoring is
  undertaken in practice