Research in Practice Wales College Development Groups

1
Research in PracticeWales College Development
Groups

• NIFTY evaluation
• Conducting an Evaluation Getting started

Elizabeth Cooke June 2005
2
What is NIFTY evaluation?

• Neat
• Informative
• Feasible
• Timely
• Yours

3
Further information about NIFTY evaluation

• Partnership project between Research in Practice
  (RiP) and Research Department of National
  Childrens Bureau (NCB)
• Supporting agencies to do their own evaluations
  over a 12 month period
• Project outputs
• NIFTY evaluation an introductory handbook for
  social care staff (2003)
• Guide to evaluation resources (2005)
• All the material is on the web
• Further information from sal_at_rip.org.uk

4
Why evaluate? internal drivers

• Organisational culture
• Evidence-informed working
• Service improvement
• User involvement
• Need to produce evidence of effectiveness

5
Why evaluate? external drivers

• Inspection
• Standards
• Quality assurance systems
• Requirement of funders
• Cymorth (SureStart) focus on outcomes
• National Service Framework for Children, Young
  People and Maternity Services focus on outcomes

6
What do we mean by evaluation?

• A working definition
• Evaluation consists of a set of systematic,
  planned activities designed to assess the
  effectiveness or worth of a service or programme.

7
Evaluation in 10 stages (1-5)

• Make a firm commitment to evaluate
• Clarify the purpose of the evaluation
• Frame evaluation questions
• Design the evaluation what evidence will be
  needed?
• Select appropriate methods for collecting data

8
Evaluation in 10 stages (6- 10)

• Detailed planning of the evaluation
• Data collection period
• Analysis
• Reporting and dissemination
• Reflection and action

9
Groundwork

• Clarify purpose of the evaluation
• Identify stakeholders and their involvement
• Key evaluation questions
• Consider practicalities
• Revisit and refocus how NIFTY is the plan?

10
Clarify the purpose of the evaluation

• Key questions to consider
• Why is this service being evaluated?
• Who is the evaluation for?
• How will the findings be used?

11
Numerous political and ethical issues

• Whose interests are being served by the
  evaluation?
• Who is the real client (the person funding the
  study /
• or those who the service is intended to benefit)
• How are vested interests taken into account?
• (the evaluation may effect peoples
  jobs/education/
• health)
• Type/style of evaluation may mean the
• perspectives, values, goals of some parties or
• participants being selected rather than those of
  others.

12
Identify stakeholders

• Anybody with an interest in either the process of
  the evaluation, or its findings eg
• Service users
• Practitioners, managers, support staff
• Funders, strategic managers, elected members
• Some possible roles for stakeholders
• Active involvement in conducting the evaluation
• Representation on a steering or reference group
• Being informed and updated about the evaluation
• Being informed of the findings

13
Agree key evaluation questions

• Questions that the evaluation sets out to answer
• Just one or two key questions for a NIFTY
  evaluation
• SMART questions (specific, measurable,
  achievable, realistic, time-based)

14
Design an Evaluation Plan

• Rationale
• Aims and objectives
• Design (include methods of data collection)
• Ethical approval
• Evaluation team
• Costs of evaluation/resources needed
• Time scales
• Dissemination
• References

15
Consider the practicalities

• How will the data be collected?
• Who is going to do the work involved?
• When?
• Are there any resource or budgetary implications?

16
Revisit and refocus

• Most initial plans are far too ambitious!
• Concentrate on the absolute essentials
• Remember the purpose of the evaluation
• Adhere to NIFTY principles

17
Keeping it NIFTY

• Neat
• Informative
• Feasible
• Timely
• Yours

18
Keep it NEAT

• Make a detailed plan think things before
  getting started
• Use or adapt existing data sources eg databases
• Draw upon skills of colleagues
• Above all, keep it simple!

19
Ensure its INFORMATIVE

• Be clear about the purpose of the evaluation
• Focus on one or two key questions
• Collect appropriate types of evidence
• Make sure the right people are informed about
  your findings
• Use the findings to make recommendations and
  changes

20
Make it FEASIBLE

• Establish what support is available before
  finalising your plans
• Get people on board
• Anticipate how you are going to process and
  analyse the data you plan to collect
• Be realistic about what can be achieved

21
Keep an eye on the TIME

• Wherever possible, build in evaluation to new
  projects and services from the start
• Try and ensure data collection doesnt coincide
  with other major disruptive events
• Recognise that it may be difficult to make time
  for evaluation
• Make sure findings are available when they are
  needed

22
Own it, its YOURS

• Dont go it alone get an evaluation team
  together, or failing that, a critical friend
• Work at establishing wider ownership
• Begin to develop a culture of evaluation within
  the organisation

23
What has NIFTY evaluation done for others?

• Greater knowledge and understanding about the
  service including some surprises
• An opportunity to reflect in a focused way on the
  service and what it is trying to do
• New skills and ways of thinking
• More useful monitoring and recording systems
• Increased funding as a result of evaluation
  evidence