The Continuing Adventures of Fidelity

1
The Continuing Adventures of Fidelity Adaptation
Implementing Science-based Programs Effectively
A Forum on Fidelity and Adaptation Issues Paul
Brounstein April 16, 2003
2
Fidelity and Adaptation

• Two sides of the same coin or 2 cents to rub
  together?
• Best thought of as complementary concepts
• Implementers should strive for balance between
  the two when adopting or adapting effective
  prevention programs

3
Fidelity

• Fidelity is high when the number and nature of
  activities remain the same over implementations.
• In RD efforts, when implemented as
  plannedoptimally when activities are clearly
  linked to need and desired outcomes and dose is
  sufficient to yield effects

4
Adaptations

• Adaptations may be made to number and/or
  regularity of activities, time spent in specific
  activities, personnel implementing, specific
  activities used in order to enhance effectiveness
  by
• Allowing implementers to use resources
  efficiently
• Adjusting for specific needs of the client
  population
• The benefits of adaptation include greater
  inclusion of local contextual information in the
  implementation process and increased local buy-in

5
Why You Should Give a Sou

• Adaptations can also attenuate program
  effectiveness
• Insufficient dosage
• Changing time schedules to compress program
• Inappropriate changes in activities given
  theoretical model of change
• Poor translation in attempts to make
  interventions culturally, age, or gender
  appropriate.

6
How do we know Adaptations are Advantageous?
7
HRY X-Site Design Characteristics

• Large Multi-Site Sample (48 sites, over 10,000
  youth)
• Comparison/Control Groups
• Common Instrument
• Four Points In Time
• Exposure to Prevention Services - (Dosage)
• Program Level Variables

8
Internal Risk and Protective Factors by Age in
the Baseline Sample
(N 10,473)
More Protection
More Risk
9
Average 30-Day Substance Use Over Time for Males
(N3,307)
0
22
8
29
Low comparison group exposure to prevention
services sites only. Averages include covariate
adjustments.
10
Average 30-Day Substance Use Over Time for
Females
(N 1,888)
9
6
3
1
Low comparison group exposure to prevention
services sites only. Averages include covariate
adjustments.
11
Finding

• Although female adolescent programs tended to
  emphasize affective content, the girls who
  attended programs focusing on behavioral- or
  life-skills demonstrated significantly greater
  positive outcomes for substance use over the four
  time points than did girls in other programs
  (female or coeducational).

12
Finding

• For boys, programs that emphasize interactive
  styles of delivering program content are more
  effective the four time points of the study.
  Emphasis on different types of content (e.g.,
  affective or life-skills) did not make a
  significant difference in sustaining effects.

13
Findings

• Culturally adapted programs produced effect sizes
  twice that of programs that were not so adapted.
• Cultural adaptations served to increase
  satisfaction, interest and exposure to the
  programs, effecting outcomes as a mediating
  variable

14
Trajectory of Innovations

• Classic RD Model
• Innovation is developed
• Prototyped technology is implemented and tested
• If successful, information is disseminated
• Those receiving the information and predisposed
  to change/improvement adopt the technology
• Diffusion

15
Reality Intervenes Again

• Some people adopt fads that resonate with their
  values and/or their belief system but that just
  dont work
• Some people make the decision to adopt an
  innovation but fail to comprehend the model or
  become overwhelmed and never implement it
  properly
• Research on adoption suggests adoption is
  facilitated by compatibility, perceived relative
  advantage, and complexity (negatively) of the
  innovation

16
One question Can programs be replicated with
Fidelity CSAP Replication Initiative Goals

• Identified Innovative, Promising Programs
• GAO selection criteria for exemplary programs
• FY 94 and FY 95
• 17 models of which 9 were selected by grantees
  for replication
• 16 5-year awards
• 6 self-replications

17
Fidelity Instruments

• Fidelity instruments were developed in
  consultation with the program developer to
  quantify the degree to which the new projects
  replicated the original project models in a
  manner that closely resembled the original

18
(No Transcript)
19
Interpreting Multi-Site Evaluation Results

• In general, the pattern of results observed
  replicated those of the original programs, not
  all of whom had consistent positive outcomes
• e.g., where results were mixed or flat, they
  tended to mirror the results achieved by the
  original program

20
Fidelity-Effectiveness Relation

• Some replicants provided data indicating a
  positive relationship between fidelity and
  outcomes
• One replicant implemented with greater fidelity
  to the program plan than did the original user
  and had better outcomes

21
Post Mortem w/Replicators

• all said they would replicate again
• self-replicants
• opportunity to increase what we know about
  particular program
• makes good sense once we know original works
• new replication grantees
• good use of resources
• great way to introduce new services to community
• provided mechanism to network with other programs
• original proof of effectiveness was invaluable

22
Core Components Analysis

• Reviewed 21 model and effective programs
• Teams of prevention researchers independently
  reviewed and coded programs.
• Developers interviewed.
• Program materials reviewed included
• Fidelity instruments
• Findings papers, publications
• Final reports
• Program curricula

23
Rating and Coding

• Teams of raters independently reviewed programs
  to answer the question
• What elements are fundamental to this program?
• Focused on participant and program
  characteristics, facets of evaluation design and
  implementation, and factors that enhance program
  adaptation and/or replication under differing
  contextual conditions.
• Detailed coding protocol.
• Inter-rater reliability assessed and improved
  with consensus meetings.

24
Results

• Besides imparting new knowledge and skills,
  effective prevention programs provide
  participants with opportunities to use this
  information.
• Effective programs move beyond change at the
  individual level. Emphasis is placed on creating
  lasting changes within individual, family, and
  school domains in an effort to create caring
  communities that share accountability for
  change.

25
Results

• Program developers consider parental involvement
  a critical factor for success.
• Efforts to include parents focus toward two
  interrelated goals enhancing parenting skills
  and self-efficacy, and increasing the level of
  parental involvement in childrens lives.

26
Results

• Successful programs promote a consistent message
  sent through multiple channels (e.g., parents,
  teachers, and peers).
• Effective programs attend to characteristics of
  the target population that place them at risk for
  alcohol, tobacco, and other drug use.

27
Results

• Successful programs emphasize relationship
  building as a precursor to delivery of program
  content. Number of sessions provided and
  activities that comprise the intervention vary
  but the first step of gaining influence is
  common.
• Importance of entering into the world of the
  client and integrating services into it are
  stressed.

28
Results

• Successful programs use naturally occurring
  social networks. Services are delivered via the
  school, community-based agencies or through other
  networks already in place (e.g., sports team
  setting).
• Process evaluation data reveal that successful
  programs enjoy high fidelity to the program plan
  or curriculum, dosage adequacy, and consistency.

29
Results

• Educational attainments and experience levels of
  persons delivering intervention are highly
  variable yet programs consistently require the
  training of delivery agents (self-instructional,
  curriculum-based or in-person) prior to
  implementation.

30
Conclusion

• Knowledge of the active ingredients in a
  science-based program provides guidance on where
  and how to adapt a program without losing
  fidelity.
• Programs adapted in line with CCA may in fact
  prove more effective than those unaltered from
  their original form.

31
Prevention Programs Outcomes Monitoring System
(PPOMS)

• Goals

• Quantify extent of science-based program
  implementation

• Measure fidelity and adaptation levels
• Document outcome results

• Relate fidelity and adaptation to outcomes

32
Accountability--The Prevention Programs Outcomes
Monitoring System (PPOMS)in Pursuit of Science
Prevention Programs Implemented
What was done? (Adaptations)
How Much? (Dosage)
What results? (Outcomes)
33
What You Can Do To Enhance Effectiveness

• Maximize fidelity to program model whenever
  possible
• Add adaptations if at all possible, dont
  substitute for regular activities
• Review program philosophy and logic model make
  sure changes made are consonant
• Talk to the developer and/or certified trainers
  about modifications, discuss core
  components/active ingredients

34
Final Recommendations, Cont.

• Perform assessments
• Pre, post and at intervals along the way
• Process and outcome
• Feedback Results to further improve program
• Cooperate with National Surveys (PPOMs)