Developing and Evaluating Interventions: DecisionMaking for Effectiveness and Eligibility

1
Developing and Evaluating Interventions
Decision-Making for Effectiveness and Eligibility

• G-CASE
• Savannah, GA
• November 9, 2006
• Dr. George M. Batsche
• Co-Director
• Institute for School Reform
• Florida PSM/RtI State Project
• School Psychology Program
• University of South Florida

2
Guiding Principles

• Effective instruction in general education is
  foundation for all decision-making
• Data guide decisions regarding core, supplemental
  and intensive interventions
• Therefore, good data must be available
• Infrastructure for core, supplemental and
  intensive instruction must be
• Evidence-based
• Integrated
• Aggregated

3
What Are the Desired Outcomes of a Successful
District Plan?

• End User (District) Outcomes
• Infrastructure for a 3-Tiered Model
• Problem-Solving Model Implemented with Integrity
• Effective Collection and Use of Data
• Decision Rules for Intervention Evaluation and
  Eligibility Determination
• Technology to Manage and Document Data-Based
  Decision Making
• Improved Academic and Behavior Outcomes for All
  Students
• Consumer Confidence and Satisfaction

4
(No Transcript)
5
What is a Good Response to Intervention?

• Good Response
• Significant improvement
• Gap is closing
• Can extrapolate point at which target student
  will come in range of peers--even if this is
  long range
• Questionable Response
• Improvement
• Rate at which gap is widening slows considerably,
  but gap is still widening
• Gap stops widening but closure does not occur
• Poor Response
• Slight improvement or NO improvement
• Gap continues to widen with no significant change
  in rate.

6
Intervention Decisions Based on RtI

• Good Response
• Continue existing intervention
• Move to next intervention level
• Questionable Response
• Increase exposure to the intervention--More Time
  or Focus
• Monitor at least weekly
• Poor Response
• Go back to problem-solving and develop new
  intervention or change existing one significantly

7
Planning AheadPredicting Who Will Be Referred

• Code referrals (reasons) for past 2-3 years
• Identifies problems teachers feel they do not
  have the skills/support to handle
• Referral pattern reflects skill pattern of the
  staff, the resources currently in place and the
  history of what constitutes a referral in that
  building
• Identifies likely referral types for next 2 years
• Identifies focus of Professional Development
  Activities AND potential Tier II and III
  interventions
• Present data to staff. Reinforces Need concept

8
Data-Driven InfrastructureIdentifying Needed
Interventions

• Assess current Supplemental Interventions
• Identify all students receiving supplemental
  interventions
• For those interventions, identify
• Type and Focus (academic, direct instruction,
  etc)
• Duration (minutes/week)
• Provider
• Aggregate
• Identifies instructional support types in
  building
• This constitutes Tier II and III intervention
  needs

9
Steps in the Problem-Solving Process

• PROBLEM IDENTIFICATION
• Identify replacement behavior
• Data- current level of performance
• Data- benchmark level(s)
• Data- peer performance
• Data- GAP analysis
• PROBLEM ANALYSIS
• Develop hypotheses( brainstorming)
• Develop predictions/assessment
• INTERVENTION DEVELOPMENT
• Develop interventions in those areas for
  which data are available and hypotheses
  verified
• Proximal/Distal
• Implementation support
• Response to Intervention (RtI)
• Frequently collected data
• Type of Response- good, questionable, poor

10
Example- ORF

• Current Level of Performance
• 40 WCPM
• Benchmark
• 92 WCPM
• Peer Performance
• 88 WCPM
• GAP Analysis 92/40 2X difference
  SIGNIFICANT GAP
• Is instruction effective? Yes, peer performance
  is at benchmark.

11
Example- Behavior

• Current Level of Performance
• Complies 35 of time
• Benchmark (set by teacher)
• 75
• Peer Performance
• 40
• GAP Analysis 40/35 1.1X difference NO
  SIGNIFICANT GAP
• Is behavior program effective? No, peers have
  significant gap from benchmark as well.

12
Data-Based Determination of Expectations

• Data- Current Level of Performance
• Data- Benchmark Level
• Date- of Weeks to Benchmark
• Calculate-
• Difference between current and benchmark level
• Divide by Weeks
• Result Rate per week of growth required
• REALISTIC? Compare to Peer Group Rate

13
Data-Based Determination of Expectations
Academic

• Benchmark Level 90 WCPM
• Current Level 40 WCPM
• Difference 50 WCPM
• Time to Benchmark 20 Weeks
• Rate of Growth Required
• 50/20 2.5 WCPM
• Peer Group Rate 2.0 wcpm growth
• REALISTIC? Not unless you increase AET

14
Data-Based Determination of Expectations
Behavior

• Same as academic calculations, EXCEPT
• Benchmark is fixed so you do not have peer rate
• Level of reality must await RtI to initial
  interventions.
• Research support for rates of improvement for the
  type of replacement behavior desired.
• Dont forget to consider ecological variables
  when using research
• Number of students in class
• Level of support for intervention implementation
• Frequency of progress monitoring

15
Problem Analysis

• Why is problem occurring?
• Facilitate Problem Analysis
• Skill vs performance
• Develop Hypotheses
• Which ones supported by data?
• Prioritize
• Note Specific Hypotheses Important-must lead to
  interventions. Reinforce data link

16
Integrated Data System

• Nine Characteristics
• Directly assess the specific skills within state
  and local academic standards.
• Assess marker variables that lead to the ultimate
  instructional target.
• Are sensitive to small increments of growth over
  time.
• Can be administered efficiently over short
  periods.

17
Integrated Data System

• May be administered repeatedly.
• Can readily be summarized in teacher-friendly
  formats/displays.
• Can be used to make comparisons across students.
• Can be used to monitor an IEP over time.
• Have direct relevance to the development of
  instructional strategies related to need.

18
PSM/RtI and LDRegulations to Practice

• New Federal regulations and current FLDOE
  regulations support 3-Tiered model of service
  delivery and decision-making using
  Problem-Solving
• Different types of data will be necessary to
  implement this model
• Decision rules will be required to evaluate
  interventions and to make eligibility decisions
• Intervention fidelity is critical

19
Tier I Regulations

• NCLB
• AYP
• Disaggregated Data by Target Groups
• Instruction is equally effective for all target
  groups- NO Child Left Behind

20
Tier I Regulations

• IDEIA
• Child does not achieve adequately for the childs
  age or to meet state-approved grade-level
  standards (discrepancy eliminated)
• (5) SPECIAL RULE FOR ELIBIGILITY DETERMINATION-
  In making a determination of eligibility under
  paragraph (4)(A), a child shall not be determined
  to be a child with a disability if the
  determinant factor for such determination
  is (A) lack of appropriate instruction in
  reading, including in the essential components of
  reading instruction (as defined in section
  1208(3) of the ESEA of 1965) (B) lack of
  appropriate instruction in math or (C) limited
  English proficiency.

21
Tier I Practice

• PROBLEM IDENTIFICATION (Step 1 of Problem
  Solving)
• Define desired academic and/or social behavior
• Current Level of Student Performance
• Curriculum-Based Assessment
• Observation for behavior referrals
• Observation for LD requirement
• Expected Performance for Age or State
  Standards/Benchmarks
• Establish that a significant GAP exists between
  current and expected levels

22
Tier I Practice

• Document that instruction is effective for the
  peer group
• AYP Data
• District-Wide Assessment Data
• AYP levels or higher?
• Document that the student has had access to the
  instruction
• Mobility is not a disability
• Effects of attendance problems
• Effects of LEP

23
Tier I Practice

• Decisions
• Instruction is Effective
• Yes
• Student had Access
• Yes Proceed to Tier II
• No Develop Interventions to Achieve Access (PSM)
• No
• Improve Core Instruction

24
Tier I Practice

• Staff involved in decisions that occur at Tier 1
• Teacher
• Principal
• Individual with expertise in diagnostic
  procedures/decisions, such as
• School psychologist
• Speech/language pathologist
• Remedial Reading Teacher

25
Example- ORF

• Current Level of Performance
• 40 WCPM
• Benchmark
• 92 WCPM
• Peer Performance
• 88 WCPM
• GAP Analysis 92/40 2X difference
  SIGNIFICANT GAP
• Is instruction effective? Yes, peer performance
  is at benchmark.
• DECISION MOVE TO TIER 2

26
Example- Behavior

• Current Level of Performance
• Complies 35 of time
• Benchmark (set by teacher)
• 75
• Peer Performance
• 40
• GAP Analysis 40/35 1.1X difference NO
  SIGNIFICANT GAP
• Is behavior program effective? No, peers have
  significant gap from benchmark as well. DEVELOP
  CORE INTERVENTION FOR ENTIRE GROUP

27
Tier II Regulations

• IDEIA
• Data-based documentation of repeated assessment
  of achievement at reasonable intervals,
  reflecting formal assessment of student progress
  during instruction
• Use information from an observation in routine
  classroom instruction and monitoring of the
  childs performance that was done BEFORE the
  child was referred for an evaluation
• OR
• After the child was referred for evaluation
  conducted by a required member of the
  determination group (teacher, individual
  qualified to conduct individual diagnostic
  examinations)

28
Tier II Regulations

• IDEIA
• If the child has participated in a process that
  assesses the childs response to scientific,
  research-based intervention
• The instructional strategies used and the
  student-centered data collected
• Strategies for increasing the childs rate of
  learning

29
Tier II Practice

• Problem Analysis/Intervention (Steps 2 and 3 of
  Problem Solving)
• Identify Supplemental Interventions-Standard
  Protocol
• Increase time
• Focus instruction/intervention
• Identify Supplemental Interventions-Flexible
  Protocol
• Based on available diagnostic data
• Continue data collection from Tier I to evaluate
  response to intervention

30
Tier II Practice

• Response to Intervention Determination
• Positive Response
• Improvement in Rate of Response
• GAP between current and desired is closing
• Decision
• Remain at Tier 2
• Initiate Fading of Intervention
• Monitor Response to Fading Intervention

31
Aimline 1.50 words/week
32
Tier II Practice

• Response to Intervention
• Questionable Response
• Improvement in Rate of Progress
• GAP not closing or closing too slowly
• Decision Options
• Continue intervention
• Increase intensity
• Go Back to Problem Solving
• Decision Remain at Tier 2

33
Aimline 1.50 words/week
Trendline 1.2 words/week
34
Tier II Practice

• Response to Intervention
• Poor Response
• No Improvement
• Improvement but GAP Continues to Widen
• Decision
• New Intervention
• Move to Tier 3

35
Aimline 1.50 words/week
Trendline 0.55 words/week
36
Tier III Practice

• Individual Problem-Solving Process
• Answer Questions
• Why is additional time and focus insufficient?
• What learner characteristics explain poor
  response to intervention?
• What instructional strategies will improve rate?
• Diagnostic/Prescriptive Process

37
Tier III Practice

• Goals of Tier 3 Problem-solving
• Identify the interventions that significantly
  increase the rate of learning
• Identify the resources necessary to implement
  those interventions
• Determine if the student has a disability

38
Aimline 1.50 words/week
Trendline 0.95 words/week
39
Eligibility Determination-SLD

• RtI Focus
• Dual Discrepancy
• Significant GAP Exists
• RATE of learning is less than peer rate
• RATE is not closing
• Functional Independence
• Can he student progress successfully without
  intensive interventions?

40
Aimline 1.50 words/week
Trendline 0.2.32 words/week
41
Aimline 1.50 words/week
Trendline 0.95 words/week
42
IDEIA Comprehensive Evaluation

• Problem Identification
• Oral Expression
• Listening Comprehension
• Written Expression
• Basic Reading Skill
• Reading Fluency Skills
• Reading Comprehension
• Mathematics Calculation
• Mathematics Problem-Solving

43
IDEIA Comprehensive Evaluation

• Relevant behavior noted during the observation
  and relationship of Bx to academic functioning
• Data from required observation

44
IDEIA Comprehensive Evaluation

• The child does not achieve adequately for the
  childs age or to meet state-approved grade-level
  standards
• GAP Analysis from Tier 1
• AND

45
IDEIA Comprehensive Evaluation

• The child does not make sufficient progress to
  meet age or to meet state-approved standards when
  using a process based on the child response to
  scientific, research-based intervention
• RtI Data from Tiers 2 and 3
• OR

46
IDEIA Comprehensive Evaluation

• The child exhibits a pattern of strengths and
  weaknesses in performance, achievement or both ,
  relative to age, state-approved grade level
  standards or intellectual development that is
  determined by the group to be relevant to the
  identification of a SLD, using appropriate
  assessments
• Differential Academic Performance Levels
• NOTE Requirement for a severe discrepancy
  between ability and achievement was removed.

47
IDEIA Comprehensive Evaluation

• The findings are not primarily the result of
• Sensory or Motor Disability
• Mental Retardation
• Assess Adaptive Behavior First
• Emotional Disturbance
• Data from observation
• Observation and performance data
• Cultural Factors
• AYP Data for Race (NCLB)
• Comparative AYP for Culture (Local Norms)
• Environmental or Economic Disadvantage
• AYP Data for Low SES
• Limited English Proficiency
• AYP Data for LEP