Evidencing outcomes keeping it real

1

• Evidencing outcomes keeping it real

2
Introduction

• Aberlour has survived and thrived for over 130
  years because it has changed and adapted as
  Scottish society has changed
• Today, the environment is as challenging as at
  any time in Aberlours history
• Aberlour works within 46 services across Scotland
  with and for
• Children, young people and families affected by
  disability
• Children, young people and families affected by
  drugs and alcohol
• Young people at risk
• Parent and child development in the Early Years

3
External Drivers

• Children should be valued and provided for within
  communities
• the importance of strong sensitive relationships
  with parents and carers
• the right to a high quality of life and to access
  play
• the need to put children at the centre of service
  delivery
• to provide more support through universal
  services when children need it
• and that children should be able to achieve
  positive outcomes irrespective of race,
  disability or social background
• (Early Years Framework, Scottish Government,
  2008)

4
External Drivers

• National Outcomes
• Within the Scottish Government Guidance on
  Single Outcome Agreements for Local Authorities
• Outcome 5 suggests Our children have the best
  start in life and are ready to succeed.
• Outcome 8 states We have improved the life
  chances for children, young people and families
  at risk.
• Outcome 15 highlights our public services are
  high quality, continually improving, efficient
  and responsive to local peoples needs.

5
Internal Drivers

• Aberlour Planning structure, from high level
  Strategy to Service plans links to SOAs
• Need for evidence to support business case for
  continued delivery
• Soft indicators, service reviews, self evaluation
  reports, annual reports, ESS
• Ambition to develop evaluation which is
  meaningful to commissioners, services and users
  and reflects Aberlours mission To improve the
  lives of Scotlands children and young people

6
Getting it Right for Every Child, how will we
know?

• As part of every initial planning meeting six
  standard questions could be considered which will
  help us to define an outcome
• What does the young person/family want to
  achieve?
• What do we need to work on to get there?
• What is the context in which he/ she is trying to
  do this? (E.g. what are the relevant
  circumstances of the young person/ family/
  employment situation etc.)
• What are the strengths and/ or abilities that he
  /she/ they can build upon? SHANARI
• What are the circumstances or obstacles which may
  prevent him/ her/ them from getting there?
• What evidence can we use to show that he/ she is
  getting there?

7
The GIRFEC Practice Model
8

• Realist Evaluation What works, for whom, under
  which circumstances?
• Formative evaluation information is fed back
  into service planning throughout the intervention
• Systematic tracking progress, evaluating efficacy
  of intervention
• Quantitative as well as qualitative outcomes
• Measurement of users target problem over time,
  using appropriate outcome measures
  (Realist Evaluation
  in Practice, Dr MAF Kazi, SAGE publications, ISBN
  0-7619-6996-9)

9
Measurement tools

• Existing validated tools (DoH framework- well
  being indicators)
• Adapted validated tools
• Developing tools with users and validating them
  partnership with academia
• Linking assessment, input and evidence of
  outcomes in partnership with users and
  commissioners of the service

10
One I prepared earlier.

• Aberlours Moray Youth Action Housing support
  service for young people
• Domain model
• Outcomes per individual, per area, per service
• Leading to continuous evaluation, adapting of
  service model and input

11
The Ambition

• Research application to develop measurement
  tools with users and commissioners GIRFEC
  impact, not process outcomes
• Practitioner based research within Aberlour,
  which we want to disseminate across the voluntary
  sector
• Linzi listening to children..
• Relationships make results, not visions,
  strategies, or targets. And the work has to be
  done, again and again.

12
Moray Realtime Evaluation

• Are we.
• Getting it Right?

13
2005-6

• We were developing a performance management
  framework for the service we wanted it to
• Have a strong focus on effectiveness in terms of
  outcomes for service users
• Connect with the challenges and issues actually
  facing practitioners
• Support continuous improvement

14
We already had

• Two services Young Peoples Housing Support and
  Youth Justice Team were using REALIST
  Evaluation
• Evidence of positive and negative outcomes over a
  range of relevant issues
• Evidence of outputs from REALIST Evaluation being
  used to improve aspects of the service
• A partnership with Aberlour Child Care Trust and
  Dr Kazi

15
We ran a pilot with social work teams

• A holistic, client self-report measure, based on
  the Scottish My World Assessment Model
• A one page form to capture aspects of each
  clients circumstances
• Training and information pack
• Temporary support posts, to advise practitioners,
  enter data, provide feedback

16
Outputs from the pilot

• Out of nearly 90 clients started on the
  programme, we had initial and subsequent measures
  for 34
• The youngest child to use self-report measures
  was 6, the oldest 16
• 62.5 improved, 12.5 stayed the same, 25 got
  worse
• No improvement re Learning to be responsible

17
Learning outcomes

• We had underestimated the learning and cultural
  transition for practitioners
• Using self-report measures could play a valuable
  part in direct work between children, parents and
  practitioners
• We needed to adjust key processes and provide
  training in setting objectives and measuring
  progress

18
Where are we now? - 1

• We have made the changes to key assessment,
  planning and review processes
• We have developed a further holistic measure, for
  use by practitioners
• Training was very positively received
• We have made the use of reliable outcome measures
  a requirement in child protection cases

19
Where are we now? - 2

• We can provide up to date outcome charts and
  interpretation for individual cases to
  practitioners and case conferences
• We can report globally to managers on
• Issues affecting parenting
• Needs profile
• Common case objectives
• Outcomes for children and families
• We have some key champions throughout the service
  statutory and voluntary sectors

20
(No Transcript)
21
More learning

• Social service organisations are data rich but
  poor on analysis
• Every clear answer to a question raises another
  question and sends you back to the evidence
• The next step is using evaluation outcomes to
  design more effective interventions