Evidence of what, exactly Outcome data and decision making

1
Evidence of what, exactly? Outcome data and
decision making

• Certainty infinite need and finite resources
• Probability new legislation, new government
  enthusiasms, new entitlements, and expectation of
  more being done
• Possibility the desire to understand what works
  best, or what works at all, reflection and
  improvement
• Choice the consequence of trying to balance
  certainties, probabilities and possibilities
• Can outcome data help us make choices?

2
Some familiar statements?

• I cant prove to you that it works, but I know
  it does. You only have to see their faces to
  know that it must work.
• The project works with young people at risk of
  social exclusion. The activities promote
  self-esteem, confidence and peer relationships
  and reduce the likelihood of the young people
  becoming involved in anti-social behaviour and
  crime. During the first year car crime has
  fallen by 18 and burglary has by 22.
• Since the programme began in April 2003 it has
  supported fifteen community development projects
  in the borough. Funding for the programme will
  end in March 2006. The evaluation will support
  the case for continuation funding of the
  programme through Communities First.

3
Some common misunderstandings

• Hypothesis and evidence
• Definition and validation, conception and
  measurement
• Presumption of effect and the expectation that
  evaluation will prove it
• Proportionality time, money, methods,
  independence and expertise

4
An approach based on previous research

• best features of other peoples output
• inferred benefits
• condition 1 adequacy
• condition 2 proximity
• NICE and NHS Centre for Reviews and Dissemination
• but what to do when the evidence is indicative
  rather than decisive?

5
Hierarchy of study designs

• Experimental studies (e.g. RCT with concealed
  allocation
• Quasi-experimental studies (e.g. without
  randomisation)
• Controlled observational studies cohort studies
  and case studies
• Observational studies without control groups
• Expert opinion and consensus
• Inexpert opinion

6
Some limitations

• feasibility, capability and resources
• 4 per cent of budget is relatively little
• compensate for local limitations by borrowing
  from the accumulated research base
• Sure Start no need to replicate, but good
  reason to become familiar
• complementary data
• think beyond your own programmes

7
Does it work?

• What sort of change are we expecting?
• Can we measure it?
• If we know the first and can do the second, can
  we rule out other explanations?
• Immeasurables social exclusion and respect
• Lots of counting but limited opportunity for
  interpretation

8
What do we need to complement data from other
studies

• a plausible relationship between things being
  done and the problems to be solved
• a demonstrable theoretical or evidence base for
  the activity, from which inferences of effect can
  reasonably be drawn
• compatibility between activities of known or
  probable effect and activities of neither

9
Is it quite that simple?

• Unfortunately not, because there are also
  considerations of
• values
• beliefs
• resistance
• the provisional nature of many findings
• Which is why the next stage experiment might
  give us an opportunity to see how these play
  against the simple application of research.