Measuring the outputs of low level services

1
Measuring the outputs of low level services

• Julien Forder

2
Outcomes in social care

• How we might define and measure outcomes for
  people using social care
• Some early results for day care services
• How outcome information can be used
• Draw on
• Treasury funded project with Office of National
  Statistics Quality Measurement Framework -
  QMF
• DH funded project Outcomes of social care for
  adults - OSCA

3
QMF projectContext and overall aims

• Methodologies for assessing the value added of
  public services
• Mechanisms for efficient measurement of the
  provision of public services
• A toolkit usable by authorities and providers to
  assess and monitor performance, and guide
  commissioning

4
General Approach

• How do we assess the quality of services?
• ... By measuring the improvement in outcomes they
  confer
• How to measure outcomes for service users?
• Ask people to rate their current situation in a
  number of relevant outcome attributes i.e.
  activities and opportunities regarding our
  quality of life
• e.g. Enough good food, feeling safe, social
  life..
• Ask people to rate one of 3 levels for each
  good, adequate, poor..
• Apply relative importance weights (from
  preference analysis)

5
General Approach (cont.)

• How to measure the effect of services and
  support. What is the counter-factual?
• What would peoples situation be in outcome terms
  in the absence of the service?
• Ask this directly... by interview
• it is more difficult and time-consuming than
  asking about current outcomes
• ... But interviewing cannot be done routinely
• Instead find important markers of expected
  outcomes that we can use to infer those outcomes
  in self-completion questionnaire

6
Application and limitations

• Application
• Day care services for people over 65
• Care homes for older people and adults with LD
• Information and advice services
• Limitations
• The quality of services may be greater than the
  impact on service user outcomes alone
• E.g. Outcomes for staff, for the wider community
  etc.

7
Outcomes - What to measure?

• The impact of services on outcomes
• fundamental motives
• Quality of life, Utility, happiness or
  well-being
• But directly measurable? Instead, inferred
• peoples functioning states
• being clean, being fed, being safe
• but also feeling in control, meaningful social
  life, being occupied, dignity
• Capability to achieve improved functioning
• not just processes
• Whose values?

Resources opportunities
Capability
Service choices
Functioning
Utility/well-being
8
Capabilities or functioning?

• Functionings tend to reflect what people actually
  do
• E.g. actual level of socialising
• Capabilities ask whether people can do the things
  they want to do
• E.g. whether socialise as much as they want
• Problems
• Capabilities may reflect expectations/ adaptation
  to poor circumstances
• Functioning does not pick up respondents views
  may not be sensitive enough

9
Methods

• Interviews and focus groups with service users
• ... to capture users views on what outcomes we
  should measure
• Cognitive testing with service users
• ... to test the understanding of questions about
  outcomes
• Self Completion Tool (SCT)
• Questions encompassing outcome domains as
  identified in focus groups/interviews and
  previous work
• ... to collect information on
• (a) current outcomes
• (b) markers of outcomes in the absence of
  services
• Face to face interviews with users
• More comprehensive bank of measures (e.g. EQ5D)
• ... for validation/comparison of outcome measures
• ... to ascertain the best markers of outcomes in
  the absence of services

10
Self Completion Tool

• Comprised of 22 questions
• Age, Sex, Living circumstances
• Need indicators
• ADLs
• IADL dealing finances/paperwork
• Informal care and equipment
• Services used
• Nine outcome domain...
• Format of outcomes questions....

Personal cleanliness Safety Meals and
nutrition Activities/occupation Control over
daily life Social participation Home
cleanliness and comfort Anxiety Dignity and
respect
11
Self Completion Tool

• Example outcome domain questions
• Thinking about your home, which of the following
  statements best describes your present situation?
  
• Please cross one box only ?
• My home is as clean and comfortable as I want ?
• My home is less clean and comfortable than I
  want ?
• My home is not at all as clean or comfortable as
  I want ?
• Which of these statements best describes your
  present situation?
• Please cross one box only ?
•  
• I have as much control over my daily life as I
  want ?
• Sometimes I dont feel I have as much control
  over my daily life as I want ?
• I have no control over my daily life ?
• Which of the following statements best describes
  your social situation?
• Please cross one box only ?
• By social situation we mean keeping in touch with
  people and spending time with people that you
  want to be with.
•  

12
Fieldwork

• 961 SCT questionnaires from sample of 135
  providers
• Asked providers to hand out SCTs
• 224 Face to face interviews
• Sub-sample of consenting SCT respondents
• Total of 29 local authorities took part
• Who provided a list of day care providers

13
Study population

• SCT Data 961 cases
• Age
• 40 aged 75 to 84
• 33 aged 85
• Sex
• 31 Male, 69 female
• Living arrangement
• 64 living on their own
• 36 living with someone else

14
Study population

• Face to Face Data 224 cases
• Age
• 81 aged 75 or over (n181)
• Sex
• 32 Male (n71) 68 female (n153)
• Ethnicity
• 88 White British or White other (n198)
• 11 Black or minority ethnic groups (n25)
• Self-reported health in general
• 23 reported their general health as either bad
  or very bad (n53)
• 36 reported this as either good or very good
  (n77)

15
Need - ADLs

• Getting dressed and undressed
• Getting in and out of bed or a chair
• Washing your face and hands
• Preparing hot meals
• Walking at least 10 minutes

16
Outcomes

• Estimated from the interview data
• 92 completed whole task
• Current outcomes (on a 0-worst to 1-best scale)
  averaged 0.74
• 11 had outcome of 1
• 14 had outcome of less than 0.5
• Outcome gain Current outcomes - expected
  outcomes

17
Outcome gain measured

• Outcome gain due to Day Care
• Mean 0.18

18
What relates to the size of outcome gains?
19
Adaptation

• Some evidence that people adapt
• Peoples subjective views about their ability to
  achieve the level of functioning they want..
• .. is greater than a more objective measure of
  their functioning
• .. When their capability is low
• E.g. Older people adapt to a lower level of
  social contact with people they are fond of.

20
Why measure outcomes?

• Knowing outcomes, rather than just outputs, tells
  us about the value of social care services
• a measure of value is needed for
• Providers to best target resources
• Focus efforts on services that are most
  cost-effective
• Outcomes-based commissioning
• Moving away from needs based or historical
  allocations
• Regulation
• Moving away form a focus on inputs and processes
• National Accounting
• Moving away from cost-weighted measures
• Value information to guide deployment of
  services to give people what they really want
  from those services

21
Aims of this work...

• To develop an Adult Social Care Outcomes Toolkit
  (ASCOT)
• Able to measure the outcome improvement produced
  by services
• Components
• Validated and robust self-completion, interview
  and observational techniques
• Surveying techniques
• Data interpretation tools
• Draw-down of existing data

22
Why prefer ASCOT?

• Most quality toolkits about process
• A common metric
• Allows comparison across services
• Potential for continuity when changing routine
  data
• Variety of users and purposes
• Distinguishes what services could do and are
  doing
• Validated approach
• Incorporates population preferences
• Low burden?
• Designed to be easy to use/answer. Uses markers
  existing data where possible e.g. Inspection data

23
Risks

• ASCOT will not be perfect! Expectations?
• How good will relationships be?
• Evolution of methodology
• Measure has moved on e.g. Better ways to assess
  adaptation
• Changing context
• Standards/quality rating basis changing
• CSCI -gt CQC
• Personalisation agenda and commissioning
• Used for performance indicators
• But what are incentives for exaggerated responses

24
Future work needed..

• Incorporating carer outcomes
• Dealing with proxy respondents aided
  self-completion
• Validation of final measure(s)
• Making this practical and useable
• Making people aware of this work

25
Next steps

• Find markers of expected outcomes in the data
• Continue with validation of outcome measures
• Refine SCT in light of findings
• Assess best method for administering SCT in
  practice
• Elaborate on how this outcomes information can be
  used