Title: DataBased Decision Making
1- Data-Based Decision Making
Institute for Small and Rural Districts July/Augus
t 2009
Michael McAuley Rebecca Sarlo Ann Selland RTI
Specialists Florida Department of Education and
the University of South Florida
2Some people want it to happen. Some people wish
it to happen. Others make no excuses and make it
happen.
3Using data to driveinstruction and intervention
4Why Data?
- For initial review
- To focus priorities
- To inform further study
- To inform instruction
- For documenting results
- Adapted from Getting Excited About Data by Edie
Holcomb
5Data-based Decision Making
- Is the students rate of progress (trendline)
less than the expected rate of progress
(aimline)? - Will the students rate of progress (trendline)
allow the student to meet identified standard
within the specified goal timeline or within a
reasonable amount of time? - Is the plan able to be maintained in the general
education setting? - If no, can student maintain rate of progress when
interventions are modified or faded?
6Major Purposes for Assessment
- Screening
- Progress Monitoring
- Development of Stronger Instructional Programs
72A P3 Success
- Alignment
- Achievement
- People
- Passion
- Proof
- Adapted from Getting Excited About Data by Edie
Holcomb
8Using Data for Alignment and Achievement
- between mission and school profile
- between concerns and priorities
- between priorities, strategies, and evidence
- Adapted from Getting Excited About Data by Edie
Holcomb
9- P3
- Engaging the People
- Arousing the Passion
- Include the Unmeasurables
- Finding the Proof
- Adapted from Getting Excited About Data by Edie
Holcomb
10Where is the proof?
- What evidence do you have that you are fulfilling
the commitments embedded in your mission
statement? - Do your goals have a baseline from which to
measure progress? - Is there more than one source of evidence for a
decision or more than one indicator of need for a
goal? - What are the assumptions we make about students
and their learning? How do you verify them?
- Adapted from Getting Excited About Data by Edie
Holcomb
11Data EngagementStart Somewhere
- Initial exploration of data will generate
reflective questions - The Snowball Effect
- Adapted from Getting Excited About Data by Edie
Holcomb
12Two Types of Data
- Generalized Outcome Measure
- Snap shot of general health, drilling down where
necessary - Mastery Measurement
- What was targeted by standard?
13Data Vocabulary
- Mean, Median, Mode
- Percentile Rank
- Aim line
- Goal line
- Trend line
- Learning Gains
- Catch up growth
14Data Display
- User-Friendly
- Longitudinal
- No-Numbers Approach
- Perceptual Data
- Adapted from Getting Excited About Data by Edie
Holcomb
15Designing a Data Day
- Define the Purpose
- District-level and School-level
- Establish Priorities
- Drilling Down Priority Data
- Carousel Data Analysis
- Adapted from Getting Excited About Data by Edie
Holcomb
16Carousel Ride
- What do these data seem to tell us?
- What do they not tell us? What else would we
need to know? - What good news is here to celebrate?
- What needs for school improvement might arise
from these data?
- Adapted from Getting Excited About Data by Edie
Holcomb
17Activity 1
- With table group, discuss the following
question - What are the barriers to the use of data by
teachers and school administrators? - (Choose one person to share out.)
18Sustaining the Struggle
- Analyzing Implementation
- Integrating Programs and Practice
- Celebrating!
- Leading with Relentless Resilience
- Adapted from Getting Excited About Data by Edie
Holcomb
19Limitations ofData-based Decision Making
- Lack of proper training
- Lack of time
- Fear of evaluation
- Fear of exposure
- Adapted from Getting Excited About Data by Edie
Holcomb
20Examples of data
- Outcome data (FCAT)
- Screening data (FAIR BS/PMT)
- Diagnostic data (FAIR, Benchmarks, ICEL)
- Formative assessments (mini-assessments FCIM)
- Progress monitoring
- Climate Survey
- Resource inventory
- Belief Survey/Implementation Rating
- School Profiles Attendance, discipline,
population, etc.
21Progress Monitoring
- Progress Monitoring Data is used to
- Determine effectiveness of
- Tier I, II, and III instruction/
- interventions
- Prompt Changes to the Instructional/Intervention
Plans - Distinguish between Intense and Severe problems
- Intensity- indicated by the difference between
what is expected and what is occurring - Severity- indicated by lack of or poor
- response to intervention
22Intervention is it working?
Making instructional decisions based on the
review and analysis of student data. Progress
monitoring always includes graphing.
Classroom Intervention I
Classroom Intervention 2
From material of the PS/RTI Project A
collaboration between FLDOE and USF
23(No Transcript)
24Positive Response to Intervention
Expected Trajectory
Performance
Time
Observed Trajectory
From material of the PS/RTI Project A
collaboration between FLDOE and USF
25Questionable Response to Intervention
Expected Trajectory
Performance
Observed Trajectory
Time
From material of the PS/RTI Project A
collaboration between FLDOE and USF
26Poor Response to Intervention
Expected Trajectory
Performance
Observed Trajectory
Time
From material of the PS/RTI Project A
collaboration between FLDOE and USF
27RtI Multi-Tier Model Tier I
- Provide effective academic and behavior core
instruction - Effective core instruction 80 or more of
students have met benchmarks with core
instruction - Provide differentiated instruction for at risk
students - Monitor and record academic and behavior growth
for all students - Typically 3 to 4 times per school year
- Review universal screening data after each
benchmark assessment to identify students in need
of more intensive intervention (i.e., Tier II
intervention).
28Universal Screening
- Given to everyone
- Critical Skills
- Brief
- Repeatable
- Cheap and easy to administer and score
- Tells us who needs more assessment
29Reviewing Core Data
30Problem Identification
From material of the PS/RTI Project A
collaboration between FLDOE and USF
31Problem Identification
From material of the PS/RTI Project A
collaboration between FLDOE and USF
32Problem Identification
From material of the PS/RTI Project A
collaboration between FLDOE and USF
33Southern Regional Education Board
- Experience
- Opportunity -- School Improvement
- Expectation Plan
- Achievement
- Until you close the first 3, you cant work on
the 4th
34Data Indicating Ineffective Core Instruction
Ineffective versus Effective Core Instruction
From material of the PS/RTI Project A
collaboration between FLDOE and USF
35Screening indicates math problem grades 3 to 5
Third Grade Math
Addition and Subtraction
About 81 Meeting minimum proficiency
Adapted from material by David Tilley Heartland
Area Educational Agency, Iowa
36Screening indicates math problem grades 3 to 5
Fourth Grade Math
About 32 Meeting Minimum Proficiency
Adapted from material by David Tilley Heartland
Area Educational Agency, Iowa
37Screening indicates math problem grades 3-5
Fifth Grade Math
About 42 Meeting Minimum Proficiency
Adapted from material by David Tilley Heartland
Area Educational Agency, Iowa
38- Universal Screening Data is used to
- Assess and monitor academic and behavioral
progress of all students - Identify ineffective core instruction
- Direct resources to Tier 1 intervention
- planning and implementation
- Identify students in need of intervention support
beyond core instruction
39- An opportunity to engage a team of persons with
expertise from across the system - Data analyst
- Curriculum Specialists
- Teachers
- Administrators
- Parents
40Use Data to Guide Questions
From material of the PS/RTI Project A
collaboration between FLDOE and USF
41What can we say?
42RtI Multi-Tier Model Tier II
- Provide intensive, research-based interventions
focused on remediation of identified problem - Interventions can be standard protocol or
determined through a problem-solving process - Interventions are typically highly structured and
are provided in a small group setting - Interventions are provided in addition to and
aligned with core instruction
43RtI Multi-Tier Model Tier II
- Monitor effectiveness of Tier II interventions
frequently (e.g., bi-weekly, monthly) - Effective Tier II Intervention 70 of students
receiving a specific Tier II intervention are
making adequate progress toward benchmark/goal - Change intervention plan if effectiveness
criteria is not met
44Tier II contd
- Students who meet benchmarks as a result of Tier
II interventions are reintegrated into Tier I
core instruction with support/scaffolding
provided through differentiation - Students who do not respond adequately to
effective Tier II intervention may be identified
for individual problem-solving and Tier III
intervention
45Determining Effectiveness of Tier II Interventions
46Data Indicating Ineffective Instruction for
Specific AYP Groups
47RtI Multi-Tier Model Tier III
- Determine intervention plan through an individual
Problem-Solving process - Interventions are intense in terms of time,
focus, and group size - Ensure interventions are research-based and
implemented with fidelity - Provide interventions in addition to and aligned
with Tier I and Tier II instruction - Measure intervention effectiveness very
frequently (e.g., weekly) and review data often
48aimline
trendline
- Check validity of data
- Emphasize rate and accuracy
- Increase power of reinforcement
- Make sure the reinforcer has value
- Have student set his or her own goals
- Challenge the student to monitor his or her own
performance
49aimline
trendline
- Recycling through PS process and address
- Does the student have needed prerequisite skills?
- Should alternative hypotheses be examined?
50aimline
trendline
- Continue Current interventions, extend monitoring
phase and consider - Increasing time of intervention
- Increasing intensity of intervention
- Have student self-monitor or self-evaluate
performance
51trendline
aimline
- Celebrate
- Keep current interventions until goal is met
- Once goal is met, determine if interventions can
be faded
52Problem-Solving andTreatment Fidelity
- Response to Intervention is based on the
actuality of interventions (Tier 1,2,3) delivered
as intended - We will not be successful with RtI if the
intervention was not implemented or was
implemented poorly - Therefore, intervention fidelity must be ensured
and documented if we are to use RtI
53Intervention Integrity Decisions
Evidence based intervention linked to verified
hypothesis planned
Evidence based intervention implemented
Student Outcomes (SO) Assessed
Treatment Integrity (TI) Assessed
Continue Intervention
SO TI
Implement strategies to promote treatment
integrity
Data-based Decisions
-SO -TI
Modify/change Intervention
-SO TI
From Lisa Hagermoser Sanetti, 2008 NASP Convention
54Implementation Integrity Data Sources
- Self Report
- Intervention Documentation Worksheet
- Permanent Product Reviews
- Lesson plans
- Graphic organizers
- Observations
- Global
- Intermediate
- Specific
55Intervention Documentation Worksheet
University of South Florida PS/RTI Project
56Monitoring Treatment Fidelity
- Strategies that improve fidelity
- Follow-up by a consultant/support staff
- Presentation of student data illustrating
response to intervention - Review of treatment implementation
- Frequency range from daily to weekly initially
- (Noell, Witt, Slider, Connell, Gatti, Williams,
Keonig, Resetar, Duhon, in press)
57Adequate Yearly Progress (AYP)
The following slides are from the Florida
Department of Education
58Subgroups Classifications
- AYP calculations are based upon nine subgroups.
- 1. Total School
- 2. White (W)
- 3. Black (B)
- 4. Hispanic (H)
- 5. Asian (A)
- 6. Native American (I)
- 7. Economically Disadvantaged Students
- 8. English Language Learners (ELL)
- 9. Students With Disabilities (SWD)
5939 Components of AYP
- Participation Rate in Reading for the 9 subgroups
- Participation Rate in Math for the 9 subgroups
- Reading Proficiency of 9 subgroups
- Math Proficiency of 9 subgroups
- Change in School Writing Proficiency
- Change in School Graduation Rate
- School Grade
60(No Transcript)
61(No Transcript)
62Whats the Size of School and Subgroup Enrollment
for AYP?
- A school must have at least 11 eligible students
in order to determine AYP for a school - A subgroup must have at least 30 representing
15 of the schools population or 100 eligible
students in order to determine AYP for a subgroup
63(No Transcript)
64(No Transcript)
65Determine if Participation Criteria was Met
- Two Ways to Compute
- First Method Is the CY Percent Tested 95 or
greater for the school and ALL subgroups? - If Yes Participation Target is Met
- If Not Check the Second Method
- Second Method Is Average Percent Tested (CY and
PY) 95 or greater for the school and/or ALL
subgroups - If Yes Participation Target is Met
- If Not Participation Target is Not Met
66(No Transcript)
67Determine if Reading/Math Proficiency Targets
were Met
- Three Ways to Compute
- First Method Did the school and ALL subgroups
meet the reading/math proficiency target? - If Yes Proficiency Targets are met
- If Not Check Second Method
- Second Method Did the school and/or ALL
subgroups that did not meet reading/math
proficiency targets meet Safe Harbor
requirements? - If Yes Reading/Math proficiency targets are Met
- If Not Check Third Method
- Third Method Did the school and/or ALL
subgroups that did not meet Safe Harbor
requirements meet Growth Model on track to be
proficient requirements? - If Yes Reading/Math proficiency targets are Met
- If Not Reading/Math proficiency targets are Not
Met
68What are Requirements for Safe Harbor?
- Safe Harbor requires that for each subgroup being
evaluated - The percent of non-proficient students decrease
by at least 10 from the preceding year - The group has met the writing criterion (the
increase in writing proficiency is at least 1 or
the school has a writing proficiency rate of 90
or better) and - The group has met the graduation rate criterion
(the increase in graduation rate is at least 1
or the school has a graduation rate of 85 or
better).
69Safe Harbor 10 Reduction inNon-Proficient
Students
Example School Prior Year 20 Proficient in
Reading Current Year 29
Proficient in Reading
Step 1 Compute Non-Proficient Prior
Year 100 20 80
Step 2 Compute 10 Target Reduction Ten
Percent of the Prior Years of Non Proficient
Students 80 10 8
Step 3 Compute Safe Harbor Proficient
Target Prior Year Proficient added to the 10
Target Reduction 20 8 28
Step 4 Determine if the 10 Reduction is
Met Does the Current Year Proficient exceed
the Safe Harbor Proficient Target? 29 gt 28
? YES, Subgroup Meets AYP through Safe Harbor
70What are Requirements for the Growth Model?
- The Growth Model requires that for each subgroup
being evaluated - The percent of students on track to be
proficient in three years or less is at least
72 in reading and 74 in math and - The group has met the writing criterion (the
increase in writing proficiency is at least 1 or
the school has a writing proficiency rate of 90
or better) and - The group has met the graduation rate criterion
(the increase in graduation rate is at least 1
or the school has a graduation rate of 85 or
better).
713rd Grade
5th Grade
4th Grade
6th Grade
Year 1 Score 1325 DSS ? Growth Target Met
Student on track to be Proficient
Year 2 Score 1500 DSS ? Growth Target
Met Student on track to be Proficient
Year 3 Score 1622 DSS ? Growth Target Met
Student Proficient in 3 Years
Gr. 5 Proficiency (1510 DSS)
Gr. 6 Proficiency (1622 DSS)
Gr. 3 Proficiency (1198 DSS)
Gr. 4 Proficiency (1456 DSS)
How to Calculate a Students 3-Year Growth
Trajectory Identify Gap Subtract Baseline Score
from Score Needed for Proficiency in Year 3 (1622
1001 621) Identify Amount of Growth Needed
Each Year Divide Gap by Three (621 / 3
207) Identify Growth Targets Year 1 Baseline
1/3 of Gap 1001 207 1208 Year 2
Baseline 2/3 of Gap 1001 414 1415 Year
3 Baseline 3/3 of Gap 1001 621 1622
72School Grades
73 School Grades What is Measured
P O I N T S C O M P O N E N T S
Points 400 for Performance 400 for Learning
Gains 800 Possible Points (100 for each of 8
components)
A 525 B 495 - 524 C 435 - 494 D 395 -
434 F lt 395
- Additional requirements applied AFTER a schools
points are calculated - Adequate Progress for Lowest 25required to earn
grade based on calculated points. If a school
does not meet this requirement, the schools
grade is lowered one letter grade. - Participation Requirement (Percent
Tested)required to earn grade based on
calculated points. Schools must test at least 95
of their students to earn an A, at least 90 to
earn any other grade.
74Floridas School Grading System50 based on
Performance
75Performance Components Points
Schools receive one point for each percent of
students meeting the performance criteria.
76Floridas School Grading System50 based on
Learning Gains
77Meeting the Learning Gains Criterion DSS Gain
for Non-Retained Level 1 and Level 2 Students
78 Recap School Grades
P O I N T S C O M P O N E N T S
Points 400 for Performance 400 for Learning
Gains 800 Possible Points (100 for each of 8
components)
A 525 B 495 - 524 C 435 - 494 D 395 -
434 F lt 395
- Additional requirements applied AFTER a schools
points are calculated - Adequate Progress for Lowest 25required to earn
grade based on calculated points. If a school
does not meet this requirement, the schools
grade is lowered one letter grade. - Participation Requirement (Percent
Tested)required to earn grade based on
calculated points. Schools must test at least 95
of their students to earn an A, at least 90 to
earn any other grade.
79Why Learning Gains?
- Reinforces concept of ALL students achieving a
years worth of growth for a years worth of
instruction - Measures growth of the individual student and
gives teachers feedback on the learning that has
occurred while under their tutelage - Represents the learning momentum a student
takes with them to the next grade - Constitutes 50 of Floridas accountability
system.the school grade - Is one of the indicators which determines pay
for performance for teachers
80http//fcat.fldoe.org/fcatpub2.asp FCAT
Achievement Level Definitions/Tables (PDF)
81Learning Gains
- Did this student make learning gains?
No! This is a Retained student and must increase
at least one achievement level to demonstrate a
learning gain.
82Learning Gains
- Did this student make learning gains?
Yes! This student made 146 DSS points gain
within level 2 and the targeted gain is 96 (95
plus 1).
83Lowest 25
What is adequate progress for the schools
lowest 25? At least 50 of the lowest
performing students must make learning gains in
Reading AND in Math Note SBE Rule modified in
2008 to include demonstration of progress toward
this benchmark
84Lowest 25
How do students in the lowest 25 demonstrate a
learning gain? Exactly the same as all other
students in the school Reasons A, B, or
C (Note If Nlt30 in lowest 25, the lowest 30
students lt Lvl 4 will be used. If still not 30
students, Learning Gains for all students will
be substituted.
85Lowest 25 Adequate Progress
- Additional opportunities for meeting adequate
progress criteria approved by SBE April 2008 - (Cannot be used for schools with sufficient
points to be A) - Lowest quartile at least 40 and increase at
least 1 percentage point over prior year - Lowest quartile less than 40 and increase at
least 5 percentage points over prior year
86REMEMBER
- Learning Gains are the
- No Excuse Data!
- It is the red flag for examining the learning
that is going on in our classrooms.
87Senate Bill 1908
- Beginning in the 2009-10 school year, 50 of the
schools grade will be based on the existing
FCAT-related factors and the remaining 50 will
be based on factors that include - A schools graduation rate
- As valid data become available, the performance
and participation of students in AP, IB, Dual
Enrollment, AICE, and industry certification - The postsecondary readiness of the students as
measured by the SAT, ACT, or CPT - The high school graduation rate of at-risk
students - The performance of a schools students on
statewide standardized end-of-course assessments,
when available and - Growth or decline in the data components from
year to year.
87
88New Component 1 Graduation Rate
- Graduation Rate Method of Calculation
- In 2009-10 and 2010-11, the graduation rate will
be calculated using the NGA 4-year cohort method
(GEDs counted as non-graduates) - Beginning in 2011-12 in order to ensure
consistency with federal reporting criteria the
graduation rate will be calculated using the new
federal uniform graduation rate criteria. - Under current regulations, 2011-12 is the first
year states are required to use this method for
federal reporting and AYP determinations. - At this time, federal regulations indicate that
GEDs, Special Diplomas, and transfers to Adult
Education would be counted as non-graduates under
this method. - These details may be subject to change with the
reauthorization of NCLB.
88
89New Component 2A Participation in Accelerated
Coursework
Proposed Calculation
- For a school to receive credit for participation
in an accelerated course that ends in an exam
(e.g., AP, IB, AICE), the student must take the
exam. - For dual enrollment, a student must earn a grade
in the course for a school to receive credit for
participation. - For industry certification, a student must be
enrolled in a CTE course and have taken an
industry certification exam on the SBE approved
Industry Certification Funding List for the
year.
89
90New Component 2B Performance in Accelerated
Coursework
Proposed Calculation
90
91New Component 2B Performance in Accelerated
Coursework
- Weighting Proposal for Performance
- Measure will be based on credits earned.
- Depending on their score on AP, IB, and/or AICE,
students will receive weight in the formula based
on the number of postsecondary courses for which
the student earns credit as determined by the
Articulation Coordinating Committees
Credit-by-Exam Equivalencies List.
(http//www.fldoe.org/articulation/pdf/ACC-CBE.pdf
) - Successful completion (a C or higher) of a Dual
Enrollment course leads to students earning
credit in one course. - Successful passage of an Industry Certification
exam.
91
92New Component 3 Postsecondary Readiness
Proposed Calculation
- Separate Measures for Reading and Math only
(recommendation to exclude Writing Readiness). - If student takes multiple tests (ACT, SAT, or
CPT), the students highest score by subtest is
used. - The scores used to define ready are set in
State Board of Education Rule 6A-10.0315, F.A.C.
92
93New Component 4 Graduation Rate for At-Risk
Students
- Use same method of calculation used for overall
graduation rate calculation. - Subset of overall cohort include only those
students that earned a Level 2 or lower on both
FCAT Reading and Math in 8th Grade. - If a school does not have at least 10 students in
that subgroup, the schools overall graduation
rate will be substituted for this measure. - This is consistent with what is done currently in
school grades in regard to the learning gains of
the lowest performing students (bottom quartile).
93
94New Component 5 Growth or Decline in
components
- Proposal
- Schools earn an escalating number of points based
on the magnitude of their improvement. - There is no deduction of points proposed at this
time. - Additional points would be awarded based on the
number of points the school improved (growth from
prior year) up to a 20 additional points. - EXAMPLE
- A schools acceleration performance improves from
25 to 32 the school earns an additional 7
points for a total of 39 points.
94
95Additional Requirement
- Law stipulates that in order for a high school to
be designated as having a grade of A, the
school must demonstrate that at-risk students are
making adequate progress. - In order for a school that earns enough points
for an A to be awarded an A, the schools
at-risk graduation rate must meet a certain
threshold to ensure adequate progress. - This requirement is akin to the current learning
gains requirement for the Low 25.
95
96High School Grading MatrixNEW 50 (with points
possible)
97Additional Issues2. Combination Schools
- In simulations, there were a total of 51 schools
labeled as combination schools that serve high
school grade levels in 2008. - 12 are K-12 schools.
- 32 are 6-12 schools.
- 7 are 9-12 schools that have been labeled as
combination. - Options on what to do with combination schools
range from - Separate school grades for the high school and
non-high school portion of the schools. - One grade per school with the high school
measures accounting for a proportional amount of
the grade based on the combination schools grade
levels served. - RECOMMENDATION One grade per combination school
using the proportional method.
97
98Combination Schools Proportional Method
98
99Additional Issues3. At-Risk Requirement for
A schools
- Law stipulates that in order for a high school to
be designated as having a grade of A, the
school must demonstrate that at-risk students are
making adequate progress. - This requirement is akin to the current learning
gains requirement for the Low 25. - RECOMMENDATION In order for a school that earns
enough points for an A to be awarded an A,
the schools at-risk graduation rate must be X
or they must demonstrate at least a 5 percentage
point increase from the prior year. Otherwise,
the schools grade is lowered to a B.
99
100Additional Issues3. At-Risk Requirement for
A schools (cont.)
- Proposed Threshold
- Using the NGA Rate, the average at-risk
graduation rate component among the graded
schools is 73. - Recommended Threshold
- 75 or
- 1 percentage point improvement over the prior
year if percentage is within 10 points of the
target - 5 percentage point improvement over the prior
year if percentage is beyond 10 points of the
target
100
101- EVALNRPT_at_fldoe.org or
- Juan Copa
- Department of Education
- 325 W. Gaines St., Room 844
- Tallahassee, FL 32399
- AYP Technical Assistance Paper Available at
http//schoolgrades.fldoe.org/pdf/0809/2009AYPTAP.
pdf
102Consider the Context
- Big data analysis question WHY???
- Why do the results look the way they do?
- Why did my students achieve the way they did?
- What are the external variables that could
possibly have affected the students
performance? - What did I, the teacher, bring to the table?
- NOTE
- Never make a decision based on one test score!
Look at longitudinal trends where possible.
103On-line Resources
- Studentprogress.org
- Rti4success.org
- Provides a consumer report of product/program for
technical features - Rtinetwork.org
104Data Disaggregation
- Never forgetbehind the data lies the face of a
child - FCAT Reading SSS 542 Science grade C
- 0 office referrals parents part of SAC
- FCAT Math SSS 467 Days Absent 2
- Days Tardy 1 Car rider
- Third school in four years
104
104
105- Thank you and Happy Data Mining!
- Questions?
- Comments?
106Line Graph with trend line
107Bar Graph
108Pie Chart