Assessing and Ensuring Treatment Integrity of SchoolBased Interventions

1
Assessing and Ensuring Treatment Integrity of
School-Based Interventions

• Lisa M. Hagermoser Sanetti, Ph.D.
• University of Connecticut
• Center for Behavior Education Research (CBER)

2
Advanced Organizer

• Treatment integrity defined
• Review conceptualizations of treatment integrity
• Creating treatment integrity assessments
• Research-supported strategies to promote
  treatment integrity

3
Treatment Integrity
The degree to which an intervention is
implemented as planned
4
Why care about treatment integrity?
1. Referral/ assessment data suggest intervention
is warranted
2. Evidence-based intervention selected
implemented
3. Student outcomes (SO) assessed
3. Treatment integrity (TI) assessed
Data reviewed
5a. Continue intervention
SO TI
4. Data-based decisions
5b. Implement strategies to promote treatment
integrity
- SO - TI
- SO TI
5c. Change intervention
5
Treatment integrity in research
6
Treatment integrity in practice

• Teachers treatment integrity decreases to low
  levels within 1-10 days.
• (Hagermoser Sanetti Kratochwill, 2007
  Mortenson Witt, 1998 Noell, et al., 1997,
  2005 Witt et al., 1997)
• Low levels of treatment integrity, in general,
  are associated with worse intervention outcomes.
• (Gansle McMahon, 1997 Greenwood et al., 1992
  Holcombe et al., 1994 McEvoy et al., 1990 Noell
  et al., 2002 Vollmer et al., 1999)

7
Conceptualizations of Treatment Integrity

• Yeaton Sechrest (1981)
• Gresham (1989)
• Moncher Prinz (1991)
• Power and colleagues (2005)

8
Time required to implement treatment (Gresham,
1989)
Treatment strength (Yeaton Sechrest, 1981)
Materials/ resources required to
implement (Gresham, 1989)
Perceived effectiveness of treatment (Gresham,
1989)
Components hypothesized to be related to
treatment integrity
Treatment agents motivation (Gresham, 1989)
Professional competence (Gresham 1989 ProYeaton
Sechrest)
Intervention complexity (Gresham, 1989 Yeaton
Sechrest, 1981)
Essential treatment components (Yeaton
Sechrest, 1981)
Number of treatment agents (Gresham, 1989)
9
Hypothesized dimensions of treatment integrity

• How much of the intervention was implemented?

• How well was the intervention implemented?

• Quality of delivery
• Participant responsiveness

• Adherence
• Exposure
• Program differentiation

(Dane Schneider,1998 Power et al., 2005)
10
Guidelines for treatment integrity assessment

• Direct or indirect methods may be used
• Operationally define intervention steps
• Written analysis of intervention plan
• Address deviations
• Monitor using multiple methods/sources
• Daily estimates component estimates
• Review data regularly

Consider using a partnership model
11
Direct or indirect methods

• Direct observation
• Permanent products
• Teacher self-report
• Behavioral interview
• Intervention scripts or manuals

12
Operationally define intervention steps

• Define each step in observable, behavioral
  terms.
• You should be able to implement the
  intervention from this definition.

13
Response Cost Lottery Example
Adapted from Gresham (1989)
14
Written analysis

• Taking operationally defined intervention steps
  and creating a written treatment integrity
  assessment.

15
Response Cost Lottery Example
Adapted from Gresham (1989)
16
Positive Reinforcement of Accurate Math
Performance Example
17
Behavior Support Plan Example
18
Address deviations

• Deviations may make the intervention
• More effective
• Less effective
• As effective
• Addressing deviations may allow you to
  proactively identify barriers to implementation.

19
Free response
20
Free response
21
Free response
22
Category of level of deviation and free response
23
Multiple methods sources

• Permanent products
• Direct observation
• Teacher self-report

Collect the completed academic work.
24
We have datanow what?

• Graph data to provide
• Daily/ session estimates of treatment integrity
• Intervention step / component estimates of
  treatment integrity

25
Daily estimates of treatment integrity
Source Hagermoser Sanetti Kratochwill (2007)
26
Component estimates of treatment integrity
Source Hagermoser Sanetti Kratochwill (2008)
27
Review data regularly!!!

• Collecting treatment integrity data is necessary
  but not sufficient for data-based decision making
  regarding intervention effectiveness!

28
Conceptual Models
Assessment Guidelines
Treatment Integrity Planning Protocol (TIPP)
Hagermoser Sanetti Kratochwill (2007)
29
TIPP

• STAGE 1. Defining the intervention
• STAGE 2. Planning the treatment integrity
  assessment
• STAGE 3. Constructing and planning the procedure
  for the treatment integrity assessment.

30
STAGE 1. Defining the intervention

• Operationally define each intervention step in
  sequential order
• Specify
• who will receive the intervention
• who will implement the intervention
• where the intervention will occur
• when the intervention will occur
• how often the intervention will occur
• how long the intervention will occur

31
STAGE 2. Planning the treatment integrity
assessment

• Specify
• how implementation will be assessed
• how covert implementation will be assessed (if
  applicable)
• For each intervention component
• Specify
• the response format
• how often the frequency of implementation will be
  assessed
• how implementation deviations will be assessed

32
STAGE 3. Constructing and planning the procedure
for the treatment integrity assessment.

• Specify
• the schedule for assessing treatment integrity
• the time period that will be covered by the
  assessment.
• who will assess treatment integrity.
• when treatment integrity will be assessed in
  relation to intervention implementation.

33
Why care about treatment integrity?
1. Referral/ assessment data suggest intervention
is warranted
2. Evidence-based intervention selected
implemented
3. Student outcomes (SO) assessed
3. Treatment integrity (TI) assessed
Data reviewed
5a. Continue intervention
SO TI
4. Data-based decisions
5b. Implement strategies to promote treatment
integrity
- SO - TI
- SO TI
5c. Change intervention
34
Research supported strategies to promote
treatment integrity

• Performance feedback
• Systematic line of research
• Effectively increases teachers level of
  treatment integrity
• Negative reinforcement
• 2 studies demonstrated positive effects when used
  in combination with performance feedback.
• (DiGenarro et al., 2005, 2007)

(Hagermoser Sanetti et al., 2007 Mortenson
Witt, 1997 Noell et al., 1997, 2005 Witt et al.
1997)
35
Implementing Performance Feedback

• Consultant collects treatment integrity data
• Consultant presents the teacher with a graph of
  the teachers treatment integrity (may also
  present student outcome data)
• Consultant consultee discuss missed
  intervention steps

36
Performance feedback with negative reinforcement

• Consultant meets with teacher and provides
  teachers with written and graphic performance
  feedback.

If treatment integrity is 100 no meeting
required
If treatment integrity is lt 100, meet with
consultant next day to review and practice missed
intervention steps
37
Review

• Lack of attention to treatment integrity in
  research practice
• Guidelines for assessing treatment integrity are
  available
• Research-based strategies for increasing
  treatment integrity are available.

38
Questions? Comments?

• CONTACT INFORMATION
• Lisa M. Hagermoser Sanetti, Ph.D.
• University of Connecticut
• E-mail lisa.sanetti_at_uconn.edu

39
References

• Dane, A. V., Schneider, B. H. (1998). Program
  integrity in primary and early secondary
  prevention Are implementation effects out of
  control? Clinical Psychology Review, 18, 23-45.
• DiGennaro, F. D., Martens, B. K., Kleinmann, A.
  E. (2007). A comparison of performance feedback
  procedures on teachers treatment implementation
  integrity and students inappropriate behavior in
  special education classrooms. Journal of Applied
  Behavior Analysis, 40, 447-461.
• DiGennaro, F. D., Martens, B. K., McIntyre, L.
  L. (2005). Increasing treatment integrity through
  negative reinforcement Effects on teacher and
  student behavior. School Psychology Review, 34,
  220-231.
• Gansle, K. A., McMahon, C. M. (1997). Component
  integrity of teacher intervention management
  behavior using a student self-monitoring
  treatment An experimental analysis. Journal of
  Behavioral Education, 7, 405-419.
• Greenwood, C. R., Terry, B., Arreaga-Mayer, C.,
  Finney, R. (1992). The classwide peer tutoring
  program Implementation factors moderating
  students' achievement. Journal of Applied
  Behavior Analysis, 25, 101-116.

40

• Gresham, F. M. (1989). Assessment of treatment
  integrity in school consultation and prereferral
  intervention. School Psychology Review, 18,
  37-50.
• Gresham, F. M., Gansle, K. A., Noell, G. H.
  (1993). Treatment integrity in applied behavior
  analysis with children. Journal of Applied
  Behavior Analysis, 26, 257-263.
• Hagermoser Sanetti, L. M., Kratochwill, T. R.
  (2007). Treatment integrity assessment in the
  schools An evaluation of the Treatment Integrity
  Planning Protocol (TIPP). Manuscript submitted
  for publication.
• Hagermoser Sanetti, L. M. Kratochwill, T. R.
  (2008). An assessment and analysis of treatment
  integrity of the good behavior game in elementary
  school classrooms. Manuscript in preparation.
• Hagermoser Sanetti, L., Luiselli, J. K.,
  Handler, M. W. (2007). Effects of verbal and
  graphic performance feedback on behavior support
  plan implementation in a public elementary
  school. Behavior Modification, 31, 454-465.

41

• Holcombe, A., Wolery, M., Snyder, E. (1994).
  Effects of two levels of procedural fidelity with
  constant time delay on children's learning.
  Journal of Behavioral Education, 4, 49-73
• McEvoy, M. A., Shores, R. E., Wehby, J. H.,
  Johnson, S. M., Fox, J. J. (1990). Special
  education teachers' implementation of procedures
  to promote social interaction among children in
  integrated settings. Education and Training in
  Mental Retardation, 25, 267-276.
• McIntyre, L. L., Gresham, F. M., DiGennaro, F.
  D., Reed, D. D. (2007). Treatment integrity of
  school-based interventions with children in
  Journal of Applied Behavior Analysis studies from
  1991 to 2005. Journal of Applied Behavior
  Analysis, 40, 659-672.
• Moncher, F. J., Prinz, R. J. (1991). Treatment
  fidelity in outcome studies. Clinical Psychology
  Review, 11, 247-266.
• Mortenson, B. P., Witt, J. C. (1998). The use
  of weekly performance feedback to increase
  teacher implementation of a prereferral academic
  intervention. School Psychology Review, 27,
  613-627.

42

• Noell, G. H., Gresham, F. M., Gansle, K. A.
  (2002). Does treatment integrity matter? A
  preliminary investigation of instructional
  implementation and mathematics performance.
  Journal of Behavioral Education, 11, 51-67.
• Noell, G. H., Witt, J. C., Gilbertson, D. N.,
  Ranier, D. D., Freeland, J. T. (1997).
  Increasing teacher intervention implementation in
  general education settings through consultation
  and performance feedback. School Psychology
  Quarterly, 12, 77-88.
• Noell, G. H., Witt, J. C., Slider, N. J.,
  Connell, J. E., Gatti, S. L., Williams, K. L.,
  Koenig, J. L., Resetar, J. L. (2005). Treatment
  implementation following behavioral consultation
  in schools A comparison of three follow-up
  strategies. School Psychology Review, 34, 87-106.
• Peterson, L., Homer, A., Wonderlich, S. (1982).
  The integrity of independent variables in
  behavior analysis. Journal of Applied Behavior
  Analysis, 15, 477-492.

43

• Power, T. J., Blom-Hoffman, J., Clarke, A. T.,
  Riley-Tillman, T. C., Kellerher, C., Manz, P.
  (2005). Reconceptualizing intervention integrity
  A partnership-based framework for linking
  research with practice. Psychology in the
  Schools, 42, 495-507.
• Vollmer, T. R., Roane, H. S., Ringdahl, J. E.,
  Marcus, B. A. (1999). Evaluating treatment
  challenges with differential reinforcement of
  alternative behavior. Journal of Applied Behavior
  Analysis, 32, 9-23.
• Witt, J. C., Noell, G. H., LaFleur, L. H.,
  Mortenson, B. P. (1997). Teacher use of
  interventions in general education settings
  Measurement and analysis of the independent
  variable. Journal of Applied Behavior Analysis,
  30, 693-696.
• Yeaton, W. H., Sechrest, L. (1981). Critical
  dimensions in the choice and maintenance of
  successful treatments Strength, integrity, and
  effectiveness. Journal of Consulting and Clinical
  Psychology, 49, 156-167.