Evaluation Results

1
Missouri Reading Initiative

• Evaluation Results
• 2002-2003

2
MRIs Evaluation Activities

• Surveys
• Teacher Beliefs and Practices (pre/post)
• Annual Participant Questionnaire
• Data Collection
• Test Scores
• Standardized Tests
• Classroom Assessments (DRA)
• MAP
• Demographics
• Special Education Information
• MAP Analyses
• For schools beginning in 2002

3
MAP ANALYSES

• Map analyses compare schools that have finished
  the MRI program with randomly chosen samples of
  non-MRI elementary schools
• Results indicate MRI schools generally outperform
  non-MRI schools

4
Notes for MAP Analyses

• Note With all the following MAP Analyses charts
  the numbers are not as important as the
  comparative performance between MRI and non-MRI
  schools. This is because
• There is a lot of variation in the data from year
  to year and school to school
• Statewide 2003 MAP scores went down from 2002
• The calculation of the baseline is slightly
  different
• For the 2002 schools 1999 was the baseline
• For the 2003 schools an average of 1999/2000 was
  the baseline

5
Notes for Chart 1

• This chart compares MRI and non-MRI schools
  performance on MAP Communication Arts Index
• Baseline year is 1999 Outcome year is 2002
• Each sample has 15 schools number of schools
  that finished MRI Spring 2002
• Total random sample 150 (large enough number to
  satisfy statistical significance, high confidence
  levels)

6
(No Transcript)
7
Notes for Chart 2

• This chart compares MRI and non-MRI schools
  performance on MAP Reading Index
• Baseline year is 1999 Outcome year is 2002
• Each sample has 15 schools number of schools
  that finished MRI Spring 2002
• Total random sample 150 (large enough number to
  satisfy statistical significance, high confidence
  levels)

8
(No Transcript)
9
Notes for Chart 3

• This chart compares MRI and non-MRI schools
  performance on MAP Communication Arts Index 2003
• Baseline year is an average of 1999/2000
  (smoothes out variations) Outcome year is 2003
• Each sample has 20 schools number of schools
  that finished MRI Spring 2003
• Total random sample 200 (large enough number to
  satisfy statistical significance, high confidence
  levels)

10
(No Transcript)
11
Notes for Chart 4

• This chart compares MRI and non-MRI schools
  performance on MAP Reading Index 2003
• Baseline year is an average of 1999/2000
  (smoothes out variations) Outcome year is 2003
• Each sample has 20 schools number of schools
  that finished MRI Spring 2003
• Total sample 200 (large enough number to satisfy
  statistical significance, high confidence levels)

12
(No Transcript)
13
Notes for Chart 5

• Left Side For 20 MRI schools, there was an
  average 19.5 decrease (the desirable direction)
  in the number of students scoring in the bottom
  two categories (Step 1 and Progressing) of the
  2003 3rd Grade Communication Arts test as
  compared to a 2.8 increase (not the desirable
  direction) for 200 randomly selected Missouri
  elementary schools.
• Right Side For 20 MRI schools, there was an
  average 32.5 increase (the desirable direction)
  in the number of students scoring in the top two
  categories (Proficient and Advanced) of the 2003
  3rd Grade Communication Arts test as compared to
  a 6.6 increase for 200 randomly selected
  Missouri elementary schools.

14
(No Transcript)
15
Notes for Chart 6

• Left Side 75 of MRI schools had decreases (the
  desirable direction) in the number of students
  who scored in the two lowest categories (Step 1
  and Progressing) on the 2003 MAP 3rd Grade
  Communication Arts test as compared to 48 of the
  schools in the random sample.
• Right Side 75 of MRI schools had increases (the
  desirable direction) in the number of students
  who scored in the two lowest categories
  (Proficient and Advanced) on the 2003 MAP 3rd
  Grade Communication Arts test as compared to
  44.5 of the schools in the random sample.

16
(No Transcript)
17
2003 MAP Average Yearly Progress

• Statewide 49.7 Met AYP
• MRI 81 Met AYP (N60/74)
• Of the 14 that did not meet AYP
• Six met AYP for All Students
• 3 - Did not meet AYP for FRL
• 1 - Did not meet AYP for IEP
• 2 - lt95 of eligible group (30)

18
Participant Survey

• Participants rate the usefulness of component
  utilization,practice change, "buy in", attitudes
  toward the program and trainer, etc.
• Results drive program change e.g.,
  Orientation

19
Notes for Participant Survey Slide

• This slide introduces the survey and its uses.
  The example is the Table in the next slide which
  demonstrates how the survey is often used. In
  this case
• 2002 survey respondents identified the problem of
  being overwhelmed
• Program responded by redesigning orientation and
  other details
• Program satisfaction for 1st year schools
  improved from 02 to 03 surveys

20
Notes for School Rating Slide

• The year (e.g., 01-02 02-03) indicates the year
  of the survey
• The number (e.g., 1st, 2nd, 3rd) indicates the
  MRI year for those schools
• This slide provides evidence for two statements
• There is a higher degree of satisfaction
  expressed by participants the longer they are in
  the program. For example, 68 (23/34) of schools
  who began in 01-02 rated the program better than
  4.0 as in their 2nd year as opposed compared to
  47 (16/34) in their 1st year. At the same time
  only 9 (3/34) of those schools rated the program
  below 3.5 in their 2nd year as opposed to 29
  (10/34) of the same schools in their 1st year.
• There has been an increase in satisfaction each
  year by schools at the same point of the program.
  For example 72 (13/18) of 1st year schools in
  02-03 rated the program better than 4.0 as
  compared to 47 (16/34) of 1st year schools in
  01/02. (see previous slide notes)

21
School Ratings 01-03 Reflecting on the
effectiveness of the MRI program as a whole, how
would you rate it?
22
Special Education

• We track the effects of MRI on Special education
  in two ways
• Beginning with schools that started MRI in the
  Fall of 2002, all students with IEPs are
  identified and the type of IEP is described
  (Reading, Math, Speech, etc.)
• Annual reports are made from every school about
  their IEP evaluation process (see following slide)

23
Notes for Special Education Table

• This Table is for schools that were in their 3rd
  year 2001-2002
• Many schools do not have this data, or it is not
  easily accessed so many schools did not report.
• 9 of 18 in 2002
• 5 0f 23 in 2003
• The data we do have all points to a decrease in
  referrals, evaluations, and assignment of IEP
  over the time schools participate in MRI.
• We are collecting this data from the onset of
  schools that began in 2002-2003 so that we will
  have a complete report in 2005.

24
Special Education Results
25

• DRA GRADE LEVEL (GL) KEY
• (ltGL)
  (gtGL)
• 1st Grade Fall lt3 3-4 gt4
• 1st Grade Spring lt16 16-18 gt18
• 2nd Grade Fall lt16 16-18 gt18
• 2nd Grade Spring lt24 24-28
  gt28
• 3rd Grade Fall lt24 24-28 gt28
• 3rd Grade Spring lt34 34-38 gt38

26
FALL SPRING FALL
SPRING FALL SPRING