An Evaluation Framework

1
An Evaluation Framework

• By
• Tom Murphy
• Western Research Institute
• Wednesday 2nd March 2005

2
Outline

• Evaluation Background
• Service Delivery Plans
• Types of Evaluation Activities
• Where to from here

3
What is Evaluation?

• Measuring the effectiveness of service delivery
• Gathering evidence
• Analysing evidence
• Reporting and acting on results

Evaluation is not about you. It is about
monitoring and improving service delivery
4
Why Evaluate

• Meet reporting requirements of Commonwealth
• Keeps the program on track in meeting needs
• Helps improve service delivery and individual
  staff skills
• Evaluation need an evolving framework

5
Where Does Evaluation Fit In

• Planning what you intend to do
• Implementation what you do what happens
• Evaluation measures what happened (outcomes)
• Note evaluation occurs throughout the program

6
Service Delivery Plans

• Goals
• Objectives, Target groups, Priority Health Issue
• Strategies/ Activities
• Performance Indicators
• Reporting
• Service Outcome

7
(No Transcript)
8
Goals

• Enhanced access
• Community participation
• Integrated approaches
• Quality improvements

9
Objectives

• What you want to achieve (Priority Health Issue-
  Target Group)
• Must be realistic and measurable
• Focus on change to target community
• Object for each goal

10
Strategies/ Activities

• Actions to achieve objectives
• Number of steps
• Various inputs
• Strategies for each goal

11
Performance Indicators

• What change had occurred (outcome)?
• What is the best measure of the outcome?
• Good measure of outcome
• Easy to obtain evidence
• Timely
• What variations have occurred? Why?
• Recommendations

12
Performance IndicatorsEnhanced Access

• Recruitment retention of staff
• Occasions of service
• Number range accessing service
• Waiting lists
• Client feedback

13
Performance IndicatorsCommunity Participation

• RHS steering committee/ management group
• Attendance at meetings
• Issues discussed
• Directions or changes as result
• Community participants feedback

14
Performance IndicatorsIntegrated Approaches

• Other service providers integrated
• Interagency network
• Occasions of service in program by other
  providers
• Other provider feedback
• Client feedback
• Duplication

15
Performance IndicatorsQuality Improvements

• Action on evaluations
• Skill gaps identified addressed
• Training sessions held/attendances
• Evaluation of training sessions

16
Simple Reporting

• Simple
• Readable
• Logical
• Evidence
• Referenced
• Refer to PIs

17
Evaluation Activities

• Types of evaluation activities
• Choosing evaluation tools
• Analysing data
• Report writing
• Dissemination
• Acting on Results

18
Types of Evaluation Activities

• Basic information
• Client community feedback
• Staffing (including recruitment, training,
  supervision)
• Meetings
• Program development (including protocols,
  information resources)
• Health promotion event

19
Basic Information

• Waiting lists, referral rates, occasions of
  service proposed v actual days of service, number
  of pap tests, number of people accessing service.
• Tables to collect this information have already
  been designed for you in FISCH

20
Examples

• Proposed Visits V Actual Days of Services

21
Client Community Feedback

• Survey list of questions to measure attitudes,
  knowledge behaviour of people in project
• Can be conducted at beginning and end or even
  middle
• Simple What did you like? What didnt you like?
  What would you change?
• Tick boxes or open-ended questions or Likert scale

22
Likert Scale
23
Issues to Consider with Surveys

• Distribution- timing, response rate, coverage
• Literacy
• Ethics- confidentiality
• Analysis
• Presentation

24
Final Training Session Bathurst
25
StaffingReporting Table
26
StaffingService Delivery Plan
27
StaffingService Delivery Plan
28
MeetingsReporting Table
29
MeetingsService delivery Plan
30
Reporting Table
31
Reporting Table
Action
Responsible
Deadline
Evidence
Establish Steering Committee
Organise venue and catering
Presenters organised
Evaluation Survey developed
32
Where to From Here?

• Start small work up to big
• Remember evaluation on the run
• Decide how things to be done

33
Get a Diary Use it

• Record everything in diary
• Highlight post it notes coloured pens
• Carry reporting tables in diary

34
Schedule Evaluation

• Design first questionnaire and analyse results
• Update FISCH analyse results
• Review staffing
• Develop distribution log

35
Timetable

• Tomorrow- buy diary
• Next week- enter data FISCH, record meetings
• Next month- Develop skill log and training
  schedule
• Next 3 months- Conduct survey