Initial Student Outcomes

1
Initial Student Outcomes
Evaluation of Massachusetts Reading First
An Analysis of Cohort 1 Assessment Data Spring
2004 vs. Spring 2005
Presented at the Massachusetts Reading First
Annual Conference June 1, 2006
Jennifer R. Gordon Research Manager
2
Questions well address today

• Have student assessment results in cohort 1
  schools improved?
• Are cohort 1 schools beginning to close the gap
  for key demographic subgroups?
• Do students benefit from longer exposure to
  Reading First?
• What are the initial results for cohort 2
  schools?

So far the evaluation does not systematically
address the relationship between effective
implementation and student outcomes. Moving
forward we are working on a plan to do so.
3
Notes about interpreting differences

• The analyses presented are comparing different
  groups of students and in all cases the
  demographics of those groups differ somewhat.
• There is a substantial research base
  demonstrating the demographic characteristics of
  students and the schools they attend have an
  impact on learning outcomes.
• Our analysis uses a mixed model regression
  procedure that controls for demographic
  differences in the schools and students being
  measured.
• A value of p 0.05 was used as the cut off for
  statistical significance.

4

• Have student assessment results in cohort 1
  schools improved?
• Fluency

5
DIBELS ORF Percentage at benchmark (low risk)
All differences are statistically significant
6
DIBELS ORF Percentage seriously behind (at risk)
All differences are statistically significant
7
Fluency in Cohort 1 Schools

• DIBELS ORF results show statistically significant
  improvement in fluency at all grade-levels.
• Increase in percentage of students in the low
  risk category
• Decrease in the percentage of students in the at
  risk category.
• Yet, many students continue to require support in
  developing this critical skill.
• 1st grade 40 below benchmark with 16 at
  considerable risk
• 2nd grade 50 below benchmark with 30 at
  considerable risk
• 3rd grade 57 below benchmark with 24 at
  considerable risk
• All third grade subgroups show improvement in
  fluency
• SPED statistically significant decrease in at
  risk
• Low Income statistically significant increase
  in low risk

8
Key Demographic Subgroups at Third GradeDIBELS
ORF Percentage at benchmark (low risk)
Statistically significant difference for low
income students
9
Key Racial/Ethnic Subgroups at Third GradeDIBELS
ORF Percentage at benchmark (low risk)
10

• Have student assessment results in cohort 1
  schools improved?
• Overall Reading Ability

11
GRADE Percentage at benchmark (stanine 5-9)
Difference for first grade is statistically
significant
12
GRADE Percentage seriously behind (stanine 1-3)
13
Overall Reading Ability in Cohort 1 Schools

• GRADE results show some improvement at all
  levels. Only the increase in first graders at
  benchmark is statistically significant.
• These are the first students to have attended
  both K and G1 in a Reading First school.
• Supports the case that early scientifically based
  instruction is key
• At all grade levels
• One-third are below benchmark
• One-fifth need intensive intervention
• All third grade subgroups show some improvement.
  Only statistically significant improvement is
  mean score increase for low income students.

14
Change in Percent at Benchmark and Seriously
Behind

• 32 schools had an overall improvement with
  increases in benchmark and decreases in seriously
  behind (upper left)
• 6 schools increased benchmark but also increased
  seriously behind (upper right)
• 5 schools decreased seriously behind but also
  decreased benchmark (bottom left)
• 10 schools had an overall decline with decreases
  in benchmark and increases in seriously behind
  (bottom right)

15
2005 GRADE Top Performers (based on cross grade
composite)

• 8 schools had at least 80 of students performing
  at benchmark on the spring 2005 GRADE assessment
• West Elementary (Plymouth) 89
• Neighborhood House Charter 88
• South Elementary (Plymouth) 87
• Walnut Square (Haverhill) 86
• Park Ave (Webster) 83
• Sanders Street (Athol-Royalston) 83
• Fall Brook (Leominster) 81
• Moseley (Westfield) 80

16
GRADE Most Improved (cohort 1 only)
17
Key Demographic Subgroups at Third GradeGRADE
Percentage at benchmark (stanine 5-9)
18
Key Racial/Ethnic Subgroups at Third GradeGRADE
Percentage at benchmark (stanine 5-9)
19
Comparing DIBELS ORF and GRADE results
20
Are there good overall readers without strong
fluency?

• Compared to DIBELS, GRADE results show larger
  percentages of students meeting benchmark.
• Could be interpreted to mean that some students
  are good overall readers but lacking fluency
  skills.
• More likely explanation is that benchmarks for
  the two assessments reflect different levels of
  expectation.
• It appears that low risk on DIBELS is associated
  with GRADE performance at or above the 6th
  stanine.
• Our GRADE analyses are based on a 5th stanine
  benchmark
• If we compare DIBELS low risk to a 6th stanine
  benchmark on GRADE we get more logical results

21

• Have student assessment results in cohort 1
  schools improved?
• MCAS Results

22
Statewide MCAS Performance 2003-2005
Due to large sample size the differences between
2003 and 2005 are statistically significant, but
not substantial.
23
RF Cohort 1 MCAS Performance 2003-2005
Overall, differences between 2003 and 2005 are
not statistically significant
24
So far MCAS results are resistant to change

• As with the state as a whole, Reading First
  schools show little change in performance on the
  3rd grade MCAS.
• Data do not support the hypothesis that
  improvement is hidden within the needs
  improvement category no positive movement from
  low to high needs improvement.
• There is some evidence that Reading First has had
  a positive impact on MCAS results.
• Mean score is the same (28.5) for both 2003 and
  2005. However demographics for the student
  tested in 2005 suggest they are more academically
  challenged. If this werent the case we would
  project an increase in mean score.
• However, the magnitude of the score increases is
  not sufficient to shift the distribution of
  students among the performance levels.

25
2005 MCAS Top Performers

• 6 schools had MCAS proficiency rates equal or
  better than the statewide rate of 63.
• Franklin Ave (Westfield) 90
• South Elementary (Plymouth) 74
• Sullivan (North Adams) 68
• Moseley (Westfield) 67
• West Elementary (Plymouth) 64
• Koziol (Ware) 63

26
MCAS Most Improved Cohort 1 Schools (2003 -
2005)
27
MCAS Most Improved Cohort 2 Schools (2004 -
2005)
28

• Do students benefit from longer exposure to
  Reading First?

29
Why look at the impact of longer exposure?

• Surrogate for a well-matched comparison group.
• Students who spent the previous full year are the
  treatment group
• Students with less exposure are the comparison
  group
• Comparison group is more demographically
  disadvantaged. Analysis controls for these
  differences.
• Analysis limited to 2nd and 3rd graders
• Since K testing is optional we cannot reliably
  code first graders
• Why not use a traditional comparison group?
• Well-matched schools are not available
• Strong dissemination efforts of DOE
• In ability to establish DIBELS and GRADE testing
  in schools that arent funded by RF or JSER.

30
GRADE Percentage at benchmark (stanines
5-9)Fall 2004 Treatment vs. Comparison
Differences are statistically significant
31
GRADE Percentage seriously behind (stanines
1-3)Fall 2004 Treatment vs. Comparison
Differences are statistically significant
32
Findings for Treatment vs. Comparison

• Second and third graders who did not spend the
  full previous year in a Reading First classroom
  started the 2004-2005 school year somewhat behind
  their classmates.
• Persistent disparity of about 15 percentage
  points all favoring the treatment group.
• Even after controlling for demographics the
  differences are statistically significant.

33

• Initial Results for
• Cohort 2 Schools

34
DIBELS ORF Cohort 2 Spring 2005Percentages at
Benchmark and Seriously Behind
35
GRADE Cohort 2 Spring 2005Percentages at
Benchmark and Seriously Behind
36
Cohort 2 Initial Outcomes Spring 2005

• DIBELS ORF
• 1st grade More than half failed to reach
  benchmark with more than one-quarter at
  considerable risk.
• 2nd grade More than 60 failed to reach
  benchmark with nearly 40 at considerable risk.
• 3rd grade More than two-thirds failed to reach
  benchmark with about one-quarter at considerable
  risk.
• GRADE
• All grade-levels had about half of the general
  population scoring in the average/strength
  categories.
• These results lag behind cohort 1, where about
  60 of all students met benchmark at the end of
  the first year of classroom implementation.

37

• Introduction to the Case Study

38
Student Assessment Results from the Arlington
School

• Three cohort 1 schools were selected for focused
  case study based on promising student assessment
  results
• Arlington School, Lawrence
• Stefanik School, Chicopee
• Franklin Avenue School, Westfield
• The next part of our presentation will share
  findings from the Arlington school case study.
• The following slides provide an overview of the
  schools GRADE data.

39
Arlington School (Lawrence) GRADE Percentage
at benchmark (stanine 5-9)
No significance testing due to small sample size
40
Arlington School (Lawrence) GRADE Percentage
seriously behind (stanine 1-3)
No significance testing due to small sample size
41
Arlington School Subgroup DataGRADE Percentage
at benchmark (stanine 5-9)
Excludes subgroups with fewer than 10 students
42
For additional information, please
contact Jennifer Gordon, Research
Manager 508-856-1349 jgordon_at_donahue.umassp.edu UM
ass Donahue Institute 333 South Street, Suite
400 Shrewsbury, MA 01545 www.donahue.umassp.edu