Introduction to program evaluation

1
Introduction to program evaluation

• Prof Hal Swerissen
• Director

2
Outline

• Evaluation concepts
• Types of evaluation
• Measurement
• Utility
• Issues

3
Evaluation criteria

• Truth
• Justice
• Independence
• Utility

4
Major evaluation questions

• What happened?
• Was it implemented as planned?
• How could it be improved?
• How much service was provided?
• What did the users think?
• Was it fair and equitable?
• Did the users benefit?
• Were the needs met?
• How much did it cost?
• Were the outcomes worth the cost?
• Did it cause the outcomes?
• Is it sustainable?

5
Purposes

• To inform or learn
• To decide
• To improve
• To reward and punish

6
Conceptual issues

• Program logic
• Internal validity
• External validity

7
Evaluation logic
Needs
Inputs
Activities
Outputs
Impact
Outcome
Cost
Impact
Process -implementation
Process - output
Outcome
8
Equity efficiency
Equity
Needs
Inputs
Activities
Outputs
Impact
Outcome
Cost
Impact
Outcome
Process-implementation
Process-output
Technical efficiency
Allocative efficiency
9
Types of evaluation

• Process evaluation strategy implementation
  quality
• Impact evaluation achievement of objectives
• Outcome evaluation achievement of goals/meeting
  needs
• Cost effectiveness/benefit comparative value of
  program

10
Process evaluation
Causative factors
Population Needs
Factors mediating service seeking
Resource Allocation
Expressed Need
Latent Need
Service delivery
Waiting
Outcome
Substitution
Outcome
Direct demand management
Outcome
Prevention program
Outcome
11
Impact/outcome evaluation
Causative factors
Population Needs
Factors mediating service seeking
Resource Allocation
Expressed Need
Latent Need
Service delivery
Waiting
Outcome
Substitution
Outcome
Direct demand management
Outcome
Prevention program
Outcome
12
Technical efficiency
Causative factors
Population Needs
Factors mediating service seeking
Resource Allocation
Expressed Need
Latent Need
Service delivery
Waiting
Outcome
Substitution
Outcome
Direct demand management
Outcome
Prevention program
Outcome
13
Allocative efficiency
Causative factors
Population Needs
Factors mediating service seeking
Resource Allocation
Expressed Need
Latent Need
Service delivery
Waiting
Outcome
Substitution
Outcome
Direct demand management
Outcome
Prevention program
Outcome
14
History

• Post war emphasis on welfare state and social
  programs
• Emphasis on rational program planning and the
  allocation of scare resources
• Early debate about science versus pragmatism

15
Against method
16
Epistemologies

• Expert review
• Scientific
• Qualitative
• Mixed

17
Expert review

• Judicial enquiries, value for money audits, major
  enquiries etc
• Interviews, document review, secondary data and
  analysis
• Expert review and judgment the critical role of
  the evaluator

18
Scientific

• Experiments, quasi experiments, time series
  analysis, single subject designs, correlational
  studies
• Measurement reliability, objectivity empiricism
• Internal external validity
• Rational analysis
• The critical role of method

19
Qualitative

• Ethnographic, phenomenological, semiotic,
  hermeneutic, sociological, historical
• interpretation of meaning, relationships action
  in context
• Observation, discussion, documentary review
• Interpretative analysis
• The critical role of theory interpretation

20
In praise of theory
21
Intervention attribution
Problems/ needs
22
Program Logic
Causative factors
Population Needs
Factors mediating service seeking
Resource Allocation
Expressed Need
Latent Need
Service output/utilisation
Waiting
Outcome
Substitution
Outcome
Direct demand management
Outcome
Prevention program
Outcome
23
Measurement
24
Resource allocation measurement

• Staff
• Consumables
• On costs
• Capital and equipment

25
Program measurement

• Staff competence
• Management support competence
• Consistency with evidence based practice
• Consistency with regulations standards
• Consumer stakeholder experience

26
Service seeking factors measurement

• Community consumer program knowledge
• Community consumer perception of needs
• Structural barriers to service access
• Program responsiveness
• Personal preferences

27
Output measurement

• Units of service provided
• Numbers of consumers and level of service
• Episodes of care (consumers by levels of service)
• Cases
• Resource intensity
• Program plan based outputs

28
Impact/outcome measurement

• Program/service effect on consumer need or
  causative factors
• Biological
• Psychological
• Behavioural
• Organisational
• Social
• Environmental

29
The centrality of utility
30
The politics of influence

• Decision makers
• Stakeholders
• Community publicThe evaluation relationship
• Whose perspectives and interests matter?

31
Proportional evaluation

• To the resource base and complexity of the
  program
• To the sophistication of the audience decision
  makers
• To what is already known
• To the purpose of the evaluation
• How important is the evaluation?

32
Understandable evaluation

• The truth, justice independence of argument
• The clarity of the presentation
• The utility of the findings and recommendations
• The importance of the evaluation relationship
• Do you really need that 200 page report?

33
Issues

• Evaluations no-one want
• Unintended effects (good bad)
• Biased evaluations
• Unjust evaluations
• Unsound evaluations
• Impractical untimely evaluations