Implementation Research

1
(No Transcript)
2
Factors Contributing to Successful Implementation
of EBPs

• The success of the intervention strategies
  themselves and
• The success of the implementation processes
• Too often, evaluations focus more on changes in
  client outcomes without due consideration of
  fidelity to the intervention model or the
  effectiveness of the implementation process
  itself.

3
Implementation Defined

• Implementation is a specified set of activities
  designed to put into practice an activity or
  program of known dimensions.
• The National Implementation Research Network at
  the University of Southern Florida conducted an
  exhaustive review of research on implementation
  and is the primary source for this presentation.

4
A Conceptual Framework for Understanding
Implementation of Evidence-Based Practices

• The essential implementation outcomes are
• Changes in adult professional behavior
• Changes in organizational structures and cultures
• Changes in relationships to consumers,
  stakeholders, and system partners

5
Implementation Framework
6
Stages of Implementation

• Exploration Adoption
• Program Installation
• Initial Implementation
• Full Operation
• Innovation
• Sustainability

7
Stages of Implementation

• Exploration Adoption
• Identify the need for an intervention considering
  existing conditions
• Assess the fit between the intervention and
  program and community needs
• Prepare organization, staff, and resources for
  mobilizing information and support.
• This was essentially accomplished in the work of
  the Work Groups.

8
Program Installation

• Program installation begins after an adoption
  has occurred and involves
• Ensuring availability of funding
• Human resource strategies (recruitment, hiring,
  training, etc.)
• Policy development to support change (e.g.
  referral mechanisms, reporting frameworks and
  outcome expectations)
• Facility requirements

9
Initial Implementation

• Initial implementation begins once conditions
  identified in the installation phase have been
  sufficiently completed.
• Initial implementation must NOT be confused with
  full operation, as during initial implementation,
  many factors contributing to fidelity to the
  model may not be fully or even partially in place.

10
Barriers to Full Operation

• Organizational Barriers
• Personnel rules
• Social stressors
• Union stewards
• Anxious administrators
• Political pressures
• Interpersonal rivalries
• Staff turnover

11
Barriers to Full Operation

• Human Barriers
• Fear of change
• Inertia
• Investment in the status quo
• Inherently difficult work of doing something
  differently

12
Full Operation

• Full Operation ONLY occurs when
• New learning becomes integrated into
  practitioner, organizational and community
  practices, policies, and procedures.
• Full case loads
• Full staffing
• Fidelity to the model being implemented
• Only once fidelity measures are above criterion
  levels most of the time, can the effectiveness of
  an evidence-based practice be evaluated. This is
  a very important point.

13
Dangers of Premature Evaluation

• Outcome evaluations should not be attempted
  until well after quality and participation have
  been maximized and documented in a process
  evaluation. Although outcome data can determine
  the effectiveness of a program, process data
  determine if a program exists at all.
• Gilliam, Ripple, Zigler Leiter (2000).

14
Innovation or Drift

• When evidence-based practice meets new local
  conditions. Two results can occur
• Innovation When desirable changes in the
  standard model are identified.
• Drift Undesirable changes in the standard model.
• It is critical to implement the model with
  fidelity before considering innovation and to
  make changes only after due consideration of
  client benefit.

15
Innovation or Drift Illustrated
16
Sustainability

• Throughout the two to four-year implementation
  process, the forces that led to and supported
  reform will change
• Staff and funding changes
• New social problems emerge
• Partnerships and political alliances change
• Champions and advocates move on to other causes
• Throughout the implementation process, it is
  essential that leadership maintain focus on
  sustaining the core components of reform.

17
Factors Contributing to Effective Implementation
of EBP

• Objective decision-making strategies that
  involved staff, good information about the reform
  and organizational leadership support during the
  exploration stage
• Evidence of a learning culture within the
  organization implementing the reform
• A system in place for monitoring implementation
• Access to technical assistance throughout
  implementation
• The perceived ability of the organization to
  manage risks
• Belief in the validity of the reform and most of
  all.
• Adherence to the core components of BOTH the EBP
  and effective implementation (discussion
  follows).

18
Core Intervention Components

• An essential first step to implementing EBP or
  system reform is to identify what are the core
  components of the intervention itself. Leadership
  must
• Carefully research alternatives
• Consider well-evaluated experiential learning
  from a number of replications and
• Achieve a clear understanding of what of the
  model must be maintained to achieve fidelity and
  effectiveness at the consumer level.
• In other words, before considering the core
  components of any intervention, leadership must
  be clear about what is to be implemented.

19
Core Components for Implementation

• The following Implementation Drivers influence
  staff behavior and organizational culture
• Practitioner Selection
• Preservice and inservice training
• A consultant coach
• Staff program evaluation
• Facilitative administration and
• System Interventions
• This is illustrated in the graphic that follows.

20
Implementation Drivers
21
Implementation Drivers

• These drivers are integrated in their influence,
  in that strengths in one area can compensate for
  weaknesses in other areas.
• The relative effectiveness of the implementation
  drivers is as important as the demonstrated
  effectiveness of the EBP itself.

22
Practitioner Selection

• How and who are selected to carry out the EBP
• Who is qualified to carry out the practices and
  programs?
• What are the methods for recruiting selecting
  practitioners?
• How important are individuals skills, experience
  and personal characteristics to effective
  implementation of EBP?

23
Research on Practitioner Selection

• There has been relatively little research to
  isolate the factors involved in selection of
  practitioners for EBP. Factors identified in
  effective practitioner selection for a national
  MST program included
• Responses to behavioral vignettes
• Responses to role playing situations related to
  the therapeutic environment for which they were
  being considered and
• Responses to mini-training that requires behavior
  change.
• Responses are rated with a rubric along several
  dimensions including collaborative and strength
  focused, efforts to overcome barriers, ability to
  use behavioral language, uses of logical thinking
  and openness to feedback.

24
Research on Practitioner Selection

• Morris Stuart, 2002 are attempting to identify
  the generic skills needed by frontline
  practitioners in a transformed behavioral health
  field. Among those qualities considered
• Assessment skills
• Family and support system involvement
• Social and cultural engagement skills
• Treatment skills
• Methods to optimize recovery and empowerment
• Consumer relationship skills and
• Community resource and coordination skills.
• It is interesting to note here and throughout
  this research, cultural competence was never
  discussed as an issue or factor.

25
Research on Staff Selection

• A Housing Urban Development study by Wanberg
  Banas (2000) examined practitioner
  characteristics in the context of organizational
  change and found that personal resilience and
  self-efficacy were associated with greater
  acceptance of change in the work place.
• This study would seem particularly important in
  SMCMHs system transformation.

26
Research on Staff Selection

• From the research we might conclude that as part
  of the implementation planning process, SMCMH
  leadership should
• Construct a rubric of ideal staff characteristics
  particularly as relates to operating in a
  transformed organization
• Design an interview process that includes
  vignettes and role plays that require candidates
  to demonstrate those characteristics

27
Preservice and Inservice Training

• Training is an efficient way to provide
• Background knowledge, background information,
  theory, philosophy, and values
• Demonstration of new skills (through video or
  role play)
• Opportunities to practice new skills through role
  plays and behavioral rehearsals and
• Feedback in a safe training environment.

28
Preservice and inservice training

• Role playing behavior rehearsals are critical
  for practicing new skills in training.
• Role playing asks you to pretend you are someone
  else and try this which builds empathy
• Behavioral rehearsals asks you to be in your
  practitioner role and you are asked to confront a
  specific situation and perform your practitioner
  role in reaction to that situation .which serves
  as direct preparation for the real thing.

29
Recommendations for Training

• Model or demonstrate new skills using role play,
  behavior rehearsals and video tape.
• Emphasize practice and use feedback on practice
  to teach the finer points of mapping.
• Use practice sessions to help trainees integrate
  thinking and doing.
• Provide guidance with respect to the boundaries
  of using the technique, describing when it may be
  useful and when it may not be useful.
• Provide guidance on the flexible use of the core
  components.
• Encourage peer and administrative support.

30
Limitations of Training

• Numerous evaluations on the impact of training
  in health and human services have found that
  training alone rarely impacts practice.
• The train-and-hope approach (Stokes Baer,
  1977) to implementation does not appear to work.
• Kelly et al (2000) in a study of HIV service
  organizations reported the largest increase in
  adoptions of HIV service guidelines occurred when
  consultation was added to training.
• A meta-analysis (Davis, 1995) found similar
  results in medicine. Davis concluded that
  formal CME conferences and activities without
  enabling or practice reinforcing strategies, had
  little impact. (page 700)

31
Limitations of Training Spray Pray

• While training may introduce knowledge,
  philosophy, and new approaches, there is no
  research study that has demonstrated significant
  changes in practice resulting from an
  intervention that provided only training.
• The limitations of training are directly related
  to the way in which adult learning occurs.

32
Stages of Adult Learning

• Generally adult learning progresses in stages
• Orientation new learning
• Mechanical use
• Routine use
• Refinement
• Integration
• Innovation

33
Challenges of Learning New Behaviors

• Based upon decades of research, Joyce Showers
  (2002) concluded that
• The newly-learned behavior is crude (i.e.
  mechanical) compared to performance by a master
  practitioner.
• Newly-learned behavior is fragile and needs to be
  supported in the face of reactions from consumers
  and others in the service setting.
• Newly-learned behavior is incomplete and will
  need to be shaped to be most functional in a
  service setting.

34
The Importance of Coaching

• While most skills can be introduced in training,
  newly-learned behaviors are only really learned
  on the job with help of a consultant or coach
  precisely because
• The challenges of adopting new behaviors cant be
  replicated in a training.
• Single-point-in-time training needs to be
  reinforced continuously in the work setting for
  behavior changes to be sustained.

35
Consultant or Coach Support

• Implementation of EBP requires changes in
  behavior at the practitioner, supervisory and
  administrative levels.
• Training and coaching are the primary strategies
  in which behavior change is brought about.
• In addition to training, substantial hands-on
  coaching and practice may be necessary before a
  counselor feels comfortable with a new strategy.
  
• Dansereau Dess (2002)

36
Components of Effective Coaching

• Coaching must be work-based, opportunistic,
  readily available, and reflective (Spouse 2001).
  Spouse described four roles for the coach
• Supervision
• Teaching while engaged in practice activities
• Assessment and feedback
• Provision of emotional support.

37
Factors that Impact Coaching

• Amount of time devoted to coaching (MST coaching
  occurs once or twice a week for 90 minutes)
  (Schoenwald et al., 2000)
• Includes direct observation of provision of
  direct services (Smart et al., 1979)
• Utilizes coaches who are expert in the content,
  techniques, and rationales of the program
  (Denton, Vaughn Fletcher 2003)
• Coaching relationships established during the
  training experience (Smart et al., 1979)

38
Other Factors Contributing to Effective Coaching

• Walker, Koroloff Schutte (2002) identified
  four additional factors that accounted for 62 of
  the variance in the perceived impact of
  supervision and coaching on practice. The
  Supervisor (coach)
• Taught new skills
• Strengthened confidence
• Offered safety in sessions
• Devoted time to discipline-specific skills

39
Personal Qualities of a Coach

• Encouraging enthusiastic
• Supportive
• Committed
• Sensitive
• Flexible
• Respectful diplomatic
• Willing to share information, credit and
  recognition.

40
Barriers to Effective Coaching

• Inadequate time allotted to coaching scheduling
  conflicts
• Role confusion due to the dual role of supervisor
  and coach
• Feelings of inadequacy on the part of coaches
  (hence the importance of selecting skilled
  coaches)
• Poor match between coach and practitioner
• Labor relationships that dont support
  observation and feedback
• Resistance in the organizational culture
• Absence of strong leadership commitment to
  implementation
• Focus on paperwork compliance as opposed to
  changes in practitioner behaviors
• Cost

41
Research on the Impact of Coaching

• The study below is based on an meta-analysis of
  hundreds of studies in education. Note the
  dramatic differences in impact upon practice.

42
Staff Evaluation

• Evaluation provides critical feedback to
  practitioners, trainers and managers related to
• Fidelity to the model being implemented
• Effectiveness of training and coaching strategies
• Impact of intervention upon clients
• Progress of implementation itself

43
Components of Evaluation

• Staff program evaluation and fidelity seem to
  consist of some combination of measures of
• Contextprerequisites that must be in place to
  operate (staffing, qualifications, ratios)
• Complianceextent to which the practitioner uses
  the core intervention components as prescribed by
  the EBP and avoids practices proscribed by the
  EBP
• Competence the level of skill shown by the
  practitioner in using the core intervention
  components

44
Effective Staff Evaluation Systems

• Huber et al., (2003) described a highly
  effective hospital management system for staff
  recruitment and evaluation that included
• Ongoing training and education focusing on
  specific skills
• Cross-training on related roles, and in-services
• Monthly dinners for discussion
• Performance evaluations based on direct
  observation to assess practice knowledge,
  communication skills, and use of time
• Prompt verbal feedback followed by a write up
  with recommendations and
• Quality improvement systems to keep the system on
  track.

45
Effective Staff Evaluation

• In other words, in a highly effective system,
  staff evaluation is part of a sequence of
  supports designed to have good people well
  prepared to do an effective job.

46
Staff Evaluation Fidelity to a Model

• One of the most critical purposes of staff
  evaluation in the context of implementing EBP is
  to test practitioner fidelity to the EBP core
  components.

47
Challenges to Measuring Fidelity

• McGrew et al., (1994) noted that development of
  fidelity measures is hampered by three factors
• Most treatment models are not well defined
  conceptually making it difficult to identify core
  intervention components
• When intervention components are identified, they
  are not operationally defined with agreed-upon
  criteria and
• Only a few models have been around long enough to
  study planned and unplanned variations.

48
Staff Evaluation Fidelity to a Model

• Fortunately the SMCMH transformation relies upon
  implementation of EBPs with effective fidelity
  instruments already in place.
• Multisystemic Treatment (MST) utilizes the
  Therapist Adherence Measure, a 27-item measure
  used in phone interviews with parents.
• The Wraparound Fidelity Index (WFI) consists of
  asking facilitators, parents and youth to rate 11
  dimensions of services.
• Assertive Community Treatment (ACT) utilizes a
  73-item tool with 17-item subset used to
  construct a fidelity index with three subscales
  staffing, organization, and service.

49
Fidelity to System Transformation

• A major challenge to SMCMH is to go beyond
  program level (MST, Wraparound, ACT) fidelity and
  to construct a fidelity index that reflects the
  qualities of a transformed system as defined and
  described in our plan.

50
Theory of Change Fidelity Models

• Possible resources to explore for developing a
  fidelity model for the transformation
• Hodges, Hernandez, Nesman Lipien (2002)
  demonstrated how a theory of change exercise can
  help programs clarify their strategies to develop
  fidelity measures.
• Shern, Trochim LaComb (1995) used concept
  mapping to develop fidelity measures for a mental
  health program.

51
Program Evaluation

• Program Evaluation assesses key elements of the
  overall performance of the organization in
  facilitating the implementation of the program or
  practice.

52
Facilitative Administration

• Facilitative administration
• Provides leadership
• Makes use of data inputs to inform decision
  making
• Supports overall processes and
• Keeps staff organized and focused on the desired
  clinical outcomes.

53
Systems Interventions

• Systems interventions are strategies to work
  with external systems to ensure the availability
  of the financial, organizational, and human
  resources required to support the work of the
  practitioners.

54
Relationship between Core Components External
Factors
55
Relationship between Core Components External
Factors
56
Key Elements to Organizational Change

• Implementation of EBP almost always requires
  organizational change. Research points to 8 key
  elements to organizational change
• Commitment of Leadership to the Implementation
  Process (Hunter, Hunter Rogers, 1993).
  Research indicates that leadership takes many
  forms, including to
• Initiate and shepherd the organization through
  the complex change process
• Set explicit goals, communicate them clearly
  throughout the organization, resolve conflicts
  with other goals, and reinforce persistence
• Help create the details of activities, processes,
  and tasks in order to operationalize
  implementation policies
• Inspire, guide, and provide direction
• Recruit, select, train, locate, advance, promote,
  or dismiss employees to further the aims of
  implementation policies

57
Other Key Elements to Organizational Change

• Involvement of stakeholders in planning and
  selection of programs to implement (Bierman et
  al., 2002)
• Creation of an implementation task force made of
  consumers, stakeholders (including unions), etc.
  (Joyce Showers, 2002)
• Suggestions for unfreezing current
  organizational practices (Cheung Cheng, 1997)
• Resources for extra costs, effort, equipment,
  manuals, materials recruiting, access to
  expertise, re-training for new organizational
  roles (Phillips, et al., 1978)

58
Other Key Elements to Organizational Change

• Alignment of organizational structures to
  integrate staff selection, training, performance
  evaluation, and on-going training (Blasé et al.,
  1984)
• Alignment of organizational structures to achieve
  horizontal and vertical integration (Unger et
  al., 2000)
• Commitment of on-going resources and support for
  providing time and scheduling for coaching,
  participatory planning, exercise of leadership,
  evolution of teamwork and for generating and
  using local data (Park Han, 2002)

59
Considerations for Effective Implementation

• At a system level and with staff and consumer
  input, develop a rubric that depicts what a
  transformed system would look like at the
  system and program levels.

60
Considerations for Effective Implementation

• Construct site-based highly structured exercises
  for sites working with their consumers to
  identify how focusing on this rubric would
  require changes in daily operations.
• How would you change staff meetings, staff
  evaluation, hours of operation, charts, reports,
  referrals, office operations, schedules, the
  waiting room?

61
Considerations for Effective Implementation

• Develop program level rubric that describe the
  extent to which a program is achieving fidelity
  to operating in conformance with the new
  transformed system.
• Build in personnel and program evaluation
  components that incorporate how staff performance
  and program structures are supporting the
  transformation.

62
Considerations for Effective Implementation

• At a structural level, consider a wide variety
  strategies that reinforce fidelity to
  transformation and the various program EBPs.
• Examine what specific practitioner and program
  changes are important and build a system of
  reinforcements that extend the impact of training.

63
Considerations for Effective Implementation

• Be mindful that changing behaviors is challenging
  and threatening.
• Training alone will not generate change in
  practitioner behavior or program priorities.
• Since implementation drivers are integrative and
  compensatory, everything you do administratively
  to facilitate and support changes, will reinforce
  the other drivers and compensate for where
  drivers are not as strong.

64
Considerations for Effective Implementation

• In so many ways, implementation is about the
  small things you do
• Small changes in intake forms, assessment tools,
  client engagement protocols, and client charts,
  changes that subtly cause practitioners to
  operate slightly differently
• Notes at the top of agenda and specific standing
  agenda items that focus on transformation issues
  and allow staff dialogue and input into
  implementation
• E-reminders to managers to observe practitioner
  groups
• Changes in personnel review tools to reinforce
  transformation
• Use of funds to foster individual staff research
  and presentations to teams about issues related
  to implementing transformation strategies.

65
Considerations for Effective Implementation

• In closing, Effective implementation of
  transformation is about
• Selection of the most effective, high-leverage
  programs
• Providing training and work-based support to
  reinforce the values and practices inherent in a
  transformed system and
• Building structures, procedures and policies that
  reinforce the priorities of the transformation
• And the devil is in the details. Implementation
  is All.