Using logic modelling to improve planning and evaluation

1
Using logic modelling to improve planning and
evaluation

• Clare Beeston - Public Health Adviser
• Policy Evaluation and Appraisal team

2
Outline

• What are logic models?
• Why use them?
• How do you develop them?
• An example of using logic models
• For strategy development
• For strategy evaluation

3
(No Transcript)
4
What are logic models?

• Diagram of how a strategy/policy/programme/servic
  e/ project is supposed to work to achieve
  intended changes and outcomes - convincing
  picture (but NOT reality)
• Visual of road map linking inputs,
  activities, outputs, and what you want to achieve
  (outcomes)

Resources / Inputs
Activities
Outputs
Short-term Outcomes
Long term Impact
Medium term Outcomes
ASSUMPTIONS That affect delivery
EXTERNAL FACTORS That affect results/impacts
5
Why use them?

• Provides a convincing story explicit
  evidence informed
• Helps facilitate thinking developing shared
  understanding
• Communicates your (agreed) vision and plans
• Provides clarity re resources needed, activities
  outcomes Highlights assumptions and risks
• Helps planning, focuses and improves
  implementation
• Provides a framework for integrating planning,
  delivery, performance management and evaluation
• Helps identify what to measure, and when

6
How do you develop a model?
Engage stakeholders and adapt and
refine Plausibility do-ability testability
Look for logic in plans and draft a model
Agree intended use(s) of model(s)
Use, revisit and refine it regularly
Share model more widely
Workshops, discussions, interviews, focus groups
7
An example

• Evidence based logic modelling to inform the new
  strategic approach to alcohol
• Developing a monitoring and evaluation framework
  to track implementation progress of, and outcomes
  from, the Licensing (Scotland) Act 2005 and the
  new strategic approach to alcohol

8
Communities
MODEL A Mapping the stakeholder outcomes
Individuals
Communities without fear of alcohol related
disorder

• Less likely to be a victim of crime
• live in cleaner, quieter streets and communities

Excessive alcohol consumption culturally
unacceptable
Safer, cleaner communities
Less likely to commit crime or ASB
Less damaged relationships with family, friends
and others
Individuals live longer and enjoy better health
and wellbeing as a result of fewer alcohol
related problems
1
Individuals make more informed choices about
alcohol consumption and its potential consequences
Less alcohol-related problems within families
2
Healthy productive population, economy health
services free from the negative effects of
alcohol consumption
Individuals businesses make full contribution
to community Scotland
In work, productive at work
Not consuming alcohol seen as unremarkable
A S P I R A T I O N S
Scotland has healthy culture of alcohol
consumption
Healthy vibrant country. Drunkenness and
inappropriate consumption of alcohol no longer
socially acceptable or tolerated
Scotland
9
Model 1 Model for improving long-term alcohol
related outcomes in Scotland

V3 021008
Intermediate outcomes
Long term outcomes
A culture in which low alcohol consumption is
valued and accepted as the norm
Children in need receive timely and appropriate
support
Model 2
Less absenteeism presenteeism in educational
establishments
Increased educational attainment
Safer drinking and wider environments
Model 3
Better parenting
Reduced acceptability of hazardous drinking and
drunkenness
Increased productivity in the workplace
Reduced excessive drinking and drunkenness
Less absenteeism presenteeism the workplace.
Less alcohol related incapacity
Increased knowledge and changed attitudes to
alcohol and drinking
Model 4
Safer happier families and communities
National outcomes
Reduced consumption per head of population over 16
Less alcohol related violence/abuse offences and
ASB
Reduced availability of alcohol
Model 5
Reduced health, social care, justice costs
Reduced individual consumption beyond
recommended limits (for life stage)
Reduced alcohol related injuries, physical and
psychological morbidity and mortality
Healthier individuals and populations
Reduced affordability of alcohol
Model 6
Individuals in need receive timely, sensitive
appropriate support
Fewer women drinking during pregnancy
Fewer brain-damaged children
Model 7
10
Model 5 Reduced availability



V2 020908
Short term outcomes
Action
Intermediate outcome
Reduced availability of alcohol
Increased time and effort required to buy alcohol
Alcohol only check-out(s) in off-sales
Decreased normalisation of alcohol (no ordinary
product)
Increased awareness of price of alcohol, and its
contribution to shopping bill
Bold action for consultation and therefore not
certain
11
Is it plausible?

• Is it plausible that the activities will achieve
  the outcomes?
• Are the activities supported by evidence-base,
  experience, logic? evidence informed
• Are the correct assumptions being made about the
  problem?
• Are the assumptions being made about the
  feasibility of the activities to make a
  difference plausible?
• Are there potential unintended consequences?

12
Is it doable?

• Are there the staff, skills, financial, IT, time
  resources necessary?
• Will there be enough resources for enforcement,
  training etc?
• Are we able to do enough of the activity to make
  a difference? RE-AIM Will the activity be
  adopted by enough staff in organisations,
  implemented in way intended and reaching enough
  of our target group

13
Is it testable?

• Can we use the model for evaluation? Is it
  specific enough to be measurable?
• Identify outcomes of most interest key outcome,
  important to the overall story where there is
  uncertainty possible differential impact
  potential for unintended consequences
• Identify when might expect to see changes in
  outcomes
• Identify indicators and data sources existing
  or new
• Prioritised and focused evaluation questions
  plans
• Monitor implementation (outputs and reach) and
  outcomes

14
FRAMEWORK FOR ASSESSING SCOTLANDS ALCOHOL
STRATEGY PROGRESS AND IMPLEMENTATION
15
Framework