Evidence-Informed

1
Evidence-Informed Program Improvement Using
principles of effectiveness to enhance the
quality and impact of youth and family programs
STEPHEN SMALL Professor of Human Development
Family Studies, University of Wisconsin-Madison a
nd Extension Human Development Family Relations
Specialist University of Wisconsin-Extension
Iowa State Webinar November 17, 2009
2
Agenda

• What are evidence-based programs, practices, and
  organizations?
• What are some common principles of effective
  evidence-based youth and family programs?
• How can Evidence-Based Program Improvement (EIPI)
  be used to improve existing programs?

3
Evidence-based Programs and Practices

• Years of research have demonstrated that specific
  approaches, practices and strategies can reduce
  problem behaviors and enhance positive
  developmental outcomes
• The most effective programs and practices are
  termed evidence-based these are the gold
  standard

4
Why the growing interest in evidence-based
programs and practices?

• Accountability Increasingly required by funders
  within political climate
• Efficiency Reduces development costs associated
  with creating a new program
• Availability of good science Body of scientific
  evidence has reached a critical mass
• Cost-effectiveness Increased likelihood that
  limited resources will be used wisely
• Marketability Documented evidence can help sell
  and justify program
• Ethics Responsibility to use what is known to be
  effective

5
The Evidence-Based Programs and Practices
Continuum
Evidence-based Programs
Evidence-based Kernels and core elements
Evidence-based Practices
Evidence-based Principles
Evidence-based Organizational Systems
6
Evidence-Based Programs

• A new class of programs that
• Are based on a sound scientific evidence and
  theory
• Have been carefully implemented, evaluated and
  replicated using rigorous scientific methods
• Have evaluation findings that have been subjected
  to critical review
• Often endorsed as evidence-based by a federal
  agencies or respected research organizations
  (e.g., Campbell Collaboration, SAMHSA, CDC)

7
Terminology

• Evidence-based ? Research-based
• Evidence-based Research-based Rigorously
  tested

8
Examples of Evidence-based Programs

• Family/Parent Education Programs
• Nurse Family Partnership 15-year follow-up
  showed ? arrests (54) among adolescents 3.59
  return on the dollar
• Strengthening Families Program for Parents and
  Youth 10-14 ? aggression, alcohol, tobacco,
  marijuana, meth 7.82 return

9
Examples of Evidence-based Programs

• School-based programs
• Life Skills Training 6-year follow-up showed ?
  alcohol, tobacco, marijuana, narcotics,
  hallucinogens 25.61 return on the dollar
• Olweus Bullying Prevention Program ? in bullying
  and victimization, vandalism, fighting, truancy
  ? in school attachment

10
Examples of Evidence-based Programs

• Mentoring
• Big Brothers Big Sisters ? in violence, alcohol
  and drug use ? in school performance 1.01
  return

11
The effect of the Strengthening Families 10-14
Program on teen aggressive and hostile behaviors
Aggression and Hostility Index
SOURCE Spoth, R., Redmond, C., Shin, C. (2000)
Reducing adolescents' aggressive and hostile
behaviors Randomized trial effects of a brief
family intervention four years past baseline.
Archives of Pediatrics and Adolescent Medicine
154, 1248-1257
12
What are some limitations of evidence-based
programs?

• Not always easy to find
• Can be costly to implement in both time
• May not address particular issues or audiences
• Tend to be problem focused
• May downplay local knowledge and community
  ownership
• Overlooks value of promising grassroots programs
• Often difficult to transport to new settings
• Other?

13
The Evidence-Based Programs and Practices
Continuum
Evidence-based Programs
Evidence-based Kernels and core elements
Evidence-based Practices
Evidence-based Principles
Evidence-based Organizational Systems
14
Evidence-Based Kernels

• Kernels units of behavioral influence and
  practice that have been shown through
  experimental evaluation to produce reliable
  effects on behavior
• Kernels Active ingredients of effective programs

15
The Evidence-Based Programs and Practices
Continuum
Evidence-based Programs
Evidence-based Kernels
Evidence-based Practices
Evidence-based Principles
Evidence-based Organizational Systems
16
Definitions of Evidence-based practices

• The integration of the best available research
  with clinical expertise in the context of patient
  characteristics, culture and preferences.
  (American Psychological Association)
• The integration of best-researched evidence and
  clinical expertise with patient values.
  (Institute of Medicine of the National Academies)

17
Evidence-Based Practice
Practitioner Expertise Experience
Evidence-Based Practices
Research Evidence
18
The Evidence-Based Programs and Practices
Continuum
Evidence-based Programs
Evidence-based Kernels
Evidence-based Practices
Evidence-based Principles
Evidence-based Organizational Systems
19
(No Transcript)
20
EVIDENCE-INFORMED PROGRAM IMPROVEMENT Using
principles of effectiveness to enhance the
quality and impact of family-based prevention
programs February 2009, Family Relations, 58,
1-13.
http//whatworks.uwex.edu
21
Principles of evidence-based youth and family
programs

• Design Content
• Relevance
• Delivery/Implementation
• Assessment Quality Assurance

22
Program design and content

• Effective programs
• Target relevant assets and/or risk and protective
  factors
• Are theory-driven
• Are of sufficient dosage and intensity
• Are comprehensive
• Use active learning approaches appropriate
  delivery formats

23
Theory-driven

• Based on strong, scientific theory
• Target relevant risk, protective factors and
  assets to accomplish objectives
• Logical, well-developed program theory

A well thought out recipeis essential
24
Theory driven - Example

• Nurse-Family Partnership (early childhood/parent
  ed) - grounded in theories of human ecology
  (Bronfenbrenner, 1979, 1992), self-efficacy
  (Bandura, 1977, 1982), and human attachment
  (Bowlby, 1969)

25
Theory-driven

• Based on strong, scientific theory
• Target relevant risk, protective factors and
  assets to accomplish objectives
• Logical, well-developed program theory

A well thought out recipeis essential
26
Have sufficient dosage and intensity
Enduring change takes effort
27
Thinking about sufficient dosage and intensity

• For a mentoring program to be effective, the
  relationship on average has to exist for at
  least
• 2 weeks
• 1 month
• 6 months
• 9 months
• 1 year

28
Comprehensive
Simple solutions rarely work
29
Use active learning techniques and appropriate
delivery format
30
Program relevance

• Effective programs
• Are developmentally appropriate
• Are socio-culturally relevant
• Are appropriately timed
• Are sensitive to local context
• Are responsive to individuals

One size does not fit all
31
Program delivery

• Effective programs
• Foster good relationships
• Are delivered by well-trained and committed staff
• Address environmental barriers to program
  delivery and behavior change
• -Provide a safe setting for the intervention
• Are embedded in a supportive organization
  infrastructure

32
Well trained and committed staff - Examples

• Big Brothers, Big Sisters (mentoring program)
  Orientation, volunteer screening, youth
  assessments, matches, supervision
• Promoting Alternative Thinking Strategies (PATHS,
  classroom program) - Teachers receive training in
  a two- to three-day workshop and in bi-weekly
  meetings with the curriculum consultant.

33
Program assessment and quality assurance

• Effective programs
• Are well-documented
• Focus on evaluation and refinement
• Provide opportunities for ongoing feedback

34
Make sure the program is well-documented

• Its important to understand the program in order
  to implement it well
• Maintaining program fidelity requires that people
  know what the program is about
• Program replication and evaluation is difficult
  if the program isnt well documented

35
Focus onevaluation refinement

• See evaluation as your friend
• Evaluation takes many forms
• Evaluation isnt always easy
• Consider EIPI as a first step

36
Evidence Informed Program Improvement
http//whatworks.uwex.edu/attachment/whatworks_man
ual.pdf
37
SUMMARYPrinciples of Effective Youth and
Family Prevention Programs

• Program design and content
• Program relevance
• Program delivery
• Program assessment and quality assurance

38
The Evidence-Based Programs and Practices
Continuum
Evidence-based Programs
Evidence-based Kernels
Evidence-based Practices
Evidence-based Principles
Evidence-based Organizational Systems
39
Evidence-based organization

• Promotes an organizational culture that values
  science and evidence-based approaches
• Creates an infrastructure that supports effective
  practices and programs
• Uses existing evidence-based programs when
  available and appropriate
• Employs a strong theoretical/science approach to
  new program design
• Provides regular opportunities to reflect on
  programs and effective practices
• Uses scientific tools to assess needs/assets,
  understand local context and adapt programs
• Provides ongoing technical support for selecting,
  implementing and evaluating effective programs
  and practices
• Builds organizational and community capacity to
  identify relevant goals and implement appropriate
  and effective approaches to address these goals

40
(No Transcript)
41
Questions/Discussion
42
(No Transcript)