EvidenceBased Programs and Practices: What, Why and How

1
Evidence-Based Programs and PracticesWhat, Why
and How?

• Janet Chiancone
• Research Coordinator, OJJDP

2
Today - 3 presentationsThemes which build upon
one another

• What, why, how OJJDP promotes EBPs
• Janet Chiancone, OJJDP
• Understanding more on the needs and strengths,
  and where to focus also what we still dont
  know
• Monica Robbers, Risk, Protective Factors of AI/NA
  Youth
• Taking the EBP concept to scale Project
  Venture, a success story
• McClellan Hall, National Indian Youth Leadership
  Project

3
What are Evidence Based Practices/Programs?

• A program or differentiated set of program
  elements, many combinations of which are
  associated with positive outcomes
• Also called
• Best practices
• Effective programs
• Research based programs
• Model programs

Source Lipsey, 1995 Lipsey Wilson, 2000
4
OJJDPs Definition of EBP

• Programs and practices that have been shown,
  through rigorous evaluation and replication, to
  be effective at preventing or reducing juvenile
  delinquency or victimization, or related risk
  factors. Evidence based programs or practices can
  come from many valid sources (OJJDPs MPG Guide).
  
• Also includes practices generally recognized as
  best practice based on research literature
  and/or the degree to which it is based on a
  clear, well-articulated theory or conceptual
  framework for delinquency or victimization
  prevention and/or intervention.

5
Reasons to Support/Promote/Increase EBPs

• OJJDPs authorizing legislation
• State level funders increasingly requiring it
• Reinvigorated federal focus on what works
• Current economic realities

6
Juvenile Justice and Delinquency Prevention Act
of 2002

• . . . to the extent practicable give priority
  in funding to programs and activities that are
  based on rigorous, systematic, and objective
  research that is scientifically based.

42 U.S.C. 5633 Sec. 223. (a)
7
State Level Funders that Require/Encourage EBPs

• Oregon
• Washington
• South Carolina
• Pennsylvania
• Many others

8
Oregons SB 267

• Requires 5 state agencies to increasingly spend
  funds on evidence based programs (EBP) so that
  by 2010, 75 of funds are to be spent on EBPs.
• Agencies include
• Commission on Children and Families
• Department of Corrections
• Department of Human Services/Office of Mental
  Health and Addictions Services
• Criminal Justice Commission
• Oregon Youth Authority

9
South Carolina Department of Alcohol and Other
Drug Abuse Services (DAODAS)

• South Carolina Toolkit for Evidence-Based
  Prevention Programs and Strategies
• (http//www.daodas.state.sc.us/toolkit.asp)
• July 2006- Required prevention providers to
  implement at least one evidence-based program or
  strategy selected from the Tool Kit.

10
PA Resource Center for Evidence-Based Prevention
and Intervention Programs and Practices
.

• Provide technical assistance for start-up,
  operation, sustainability of prevention or
  intervention programs proven to be effective.
• Ensure that PCCD- and DPW-funded Evidence-Based
  Program (EBP) models are implemented with
  fidelity
• Serve as a resource to the field related to the
  implementation of evidence-based programming.
• Promote proliferation of EBPs by educating
  practitioners and providers about practical and
  economic benefits of EBPs

11
Renewed Federal Focus on What Works
11

• Performance/Accountability a priority
• Economic realities make this necessary
• WH Position Chief Performance Officer
• . . . we must scour this budget, line-by-line,
  eliminating what we dont need, or what doesnt
  work, and improving the things that do.
• President Barack Obama

12
Evidence based Programs and Practices Save Money

• Cost to provide every student in U.S. a model
  drug prevention program like LST is 550 million
  per year
• Current national drug control spending is
  approximately 40 billion per year
• This represents 1.5 of our current drug control
  spending

13
Applying Evidence Based Practices Should you .
. .

• Replicate an existing Model Program
• or
• 2. Extract program principles or guidelines for
  effective interventions

14
Option 1 Replicating Model Programs

• Model programs must be
• Based on a sound theory or model
• Evidence based
• Well defined
• Well documented
• Replicated with fidelity

15
Things to Look for When Reviewing Programs

• Risk and protective factors targeted by program
• Population(s) targeted in the demonstration/test
• Level of effectiveness
• Guidelines for implementation do they exist?
• Cost
• Adapted for Native American/Alaska Native youth

16
EBPs That Show Effectiveness for American Indian
and Alaska Native Youth and Families

• Project Venture (Details later today!)
• American Indian Life Skills Development/Zuni
  Life Skills Development
• Strengthening Families Program
• Families and Schools Together (FAST)
• Not-On-Tobacco (N-O-T)
• Native American Prevention Project Against AIDS
  and Substance Abuse

17
American Indian Life Skills Development/Zuni
Life Skills Development

• 13-17 years old, boys and girls
• Reduced feelings of hopelessness
• Increased suicide prevention skills
• Zuni Life Skills Development curriculum
  originally for high school students in Zuni
  Pueblo. Has been adapted and replicated with
  several other tribes (American Indian Life Skills
  Development curriculum)
• Stanford University

18
Strengthening Families Program

• Ages 3-17, boys and girls
• Improved parenting skills
• Reduced conduct disorders among children
• Improved social competencies for children
• Extensively replicated and found effective for
  families of diverse backgrounds (including
  AI/NA).
• University of Utah

19
Families and Schools Together (FAST)

• Ages 4-12, boys and girls
• Displayed more positive social skills
• Lower aggressive behavior
• Higher academic performance
• One random control experiment focused on AI/NA
  youth and families. Results consistent.
• University of WI-Madison

20
Not-On-Tobacco (N-O-T) Smoking Cessation Program

• Ages 14-19, boys and girls
• More likely to quit smoking
• Increase in positive attitudes, behaviors
  (managing stress, feel better about themselves)
• Curriculum was adapted for Native American youth
• West Virginia University

21
Native American Prevention Project Against AIDS
and Substance Abuse

• 12-18 years old, boys and girls
• Reduced incidence/support of risky sexual
  behavior
• Less favorable attitudes about alcohol, drugs
• Stronger refusal skills
• National Institute of Child Health and Development

22
Fidelity vs Adaptation

• Some say the need for local adaptation is over
  estimated
• Adaptations must fit with program rationale
• Language/cultural adaptations most justified
• Little evidence for race/ethnicity, gender, or
  class differences in school program effects
• Most frequent threats to fidelity
• Frontline implementers
• Disseminating Agency

Source Elliot, D. 2008, Blueprints Conference
23
Fidelity vs Adaptation

• Adaptation is as likely to reduce effects as
  enhance them
• Local adaptation may increase buy in but also
  creates uncertainty about program effects
• Program success must be judged by real changes in
  behavior, not number of adaptions or survival

Source Elliot, D. 2008, Blueprints Conference
24
Option 2 Implementing Effective Program
Principles

• see checklist for example
• Targeting the highest risk youth
• Using risk and needs assessment instruments
• Targeting risk and protective factors
• Treatment approaches based on strengths and
  needs of targeted youth
• Multiple services
• Sustained services (more than 3 months)

25
Program Fidelity Principles

• Ensure the program is based on a specific,
  theoretical model
• Ensure program is delivered as designed
• Train workers in program delivery
• Have staff who are well-educated, qualified, and
  experienced
• Provide trained supervision
• Develop printed program materials describing
  program goals and content
• Select staff on key interpersonal or skill
  factors

26
Update/Refresh Your Logic Model Periodically

• Are goals and objectives still relevant?
• Are activities occurring as intended?
• Do you need to revisit outcomes or activities?
• Where are you experiencing challenges?
• Has something new entered the mix?
• Other changes?
• Use this process as a way to keep staff and
  partners engaged

27
Make Performance Measures a Priority
27

• Performance measures and data collection are
  building blocks of evaluation
• Hard proof of what/how/when/why your program is
  doing
• Documentation supports sustainability

28
Examine Experiences of EBPs(Lessons Learned)

• Conduct a needs assessment before implementation
• Have strong administrative support and program
  champion
• Have a stable agency/shared vision
• Obtain support, motivation, buy-in of staff
• Ensure implementation fidelity
• Use paid staff, not volunteer

Source Mihalic, et al., 2004. Successful
Program Implementation Lessons From Blueprints
(OJJDP Bulletin)
29
(No Transcript)
30
Commit to (Eventually) Evaluating Your Program

• Most (but not all) EBPs were established in a
  University setting or with University partners
• Make sure you are internally tracking the data
  needed to determine the programs effect
  focusing on those who receive the services
• Make linkages to existing resources in evaluation
  design and assistance (see handout)

31
Resources EPB Websites

• Model Programs Guide (MPG), Office of Juvenile
  Justice and Delinquency Prevention
• Blueprints for Violence Prevention, OJJDP and
  Centers for Disease Control and Prevention
• What Works Clearinghouse (WWC), U.S. Department
  of Educations Institute of Education Sciences
• National Registry of Evidence-based Programs and
  Practices (NREPP), Substance Abuse and Mental
  Health Services Administration
• Promising Practices Network on Children, Families
  and Communities (PPN), RAND corporation with many
  partners
• Find Youth Info web site, Federal Interagency
  Working Group on Youth Programs

32
OJJDP Model Programs Guide
www.ojjdp.ncjrs.gov/programs/mpg.html
An easy-to-use tool to search for evidence-based
programs State-of-the-art literature reviews
in dozens of program areas Over 200 program
descriptions
33
Moving Forward Becoming an EBP?

• Start by focusing on Best Practices Principles
• Continue to update/refresh your logic model
• Make performance measures a priority
• Learn more about those programs that have been
  through the EBP process
• What have been their successes and challenges,
  both expected and not
• Seek partners in evaluation (including OJJDP)
• Use the Tools and Resources available

34
Additional Resources

• OJJDPs Contractors
• Education Development Center (EDC)
• CSR, Inc.
• National Training and Technical Assistance Center
  (NTTAC)
• Each other share your experiences, successes,
  challenges
• Tell us what else you need training, TA
  materials, resources, assistance?

35
Thank You!

• Janet Chiancone, Research Coordinator
• Janet.Chiancone_at_usdoj.gov
• [phone]