Making MAP More Meaningful

1
Making MAP More Meaningful

• David Dreher, Project Coordinator
• Dr. Kathryn Sprigg, Assistant Director
• Office of Accountability, Highline Public Schools
• Dr. Sandra L. Hunt , Literacy Coach
• Beverly Park Elementary, Highline Public Schools

2
Overview

• The needs of the data users
• The objectives of the data producers
• The products
• The process
• The implementation
• The results
• The future

3
What is MAP

• Measures of Academic Progress
• Developed by the Northwest Evaluation Association
  
• Norm-referenced assessment
• Computerized and adaptive
• Performance is reported as a RIT score
• The RIT Scale
• Uses individual item difficulty values to
  estimate student achievement
• A RIT score has the same meaning regardless of
  grade level
• Equal interval scale
• Highline Public Schools
• Three testing windows per year (Fall, Winter,
  Spring)
• Test students in the areas of math and reading
• Test students in grades 3-10

4
The Needs of the Data User

• Building staff were saying things like . . .
• How can we use MAP data to help us make
  decisions?
• How do MAP and WASL performance compare?
• I want to know what a students history is with
  MAP.
• What is a RIT score?
• Giving me a RIT score is like telling me the
  temperature in Celsius!

5
The Objectives for Us

• Include more historical data in reports.
• Make the data more accessible.
• Put MAP scores in context with WASL scores.
• Provide indication of a students likelihood of
  meeting standard.

6
Some General Challenges

• Fear of Numbers
• The products generated had to be fairly simple to
  explain and understand.
• Availability of Time
• Because it had to be there yesterday it has to be
  fairly simple for us to produce.

7
The Products

• Fall Predictions
• Our best guess about each students performance
  on the upcoming WASL.
• Used for
• Identifying level of risk for not meeting
  standard
• School- and District- level WASL forecasts
• Benchmark, Strategic, Intensive (BSI) Updates
• Status update produced after each testing
  window.
• Coarse filter based only on MAP.
• Cut Score Document
• A quick reference table that could be used to
  help put a MAP score in context.

8
Making The Predictions

• Snooped and found the best indicators of WASL
  success
• Applied linear regression models to generate WASL
  scores for each student
• Examined the predicted WASL scores

9
Snooping (Reading)R-Values
WASL 2007 Reading Scale vs. Grade Grade Grade Grade Grade Grade Grade
WASL 2007 Reading Scale vs. 3 4 5 6 7 8 10
WASL Reading 2006   0.730 0.769 0.764 0.745 0.755  
WASL Reading 2005       0.727      
WASL Reading 2004             0.723
MAP-R Spring 2007 0.792 0.778 0.799 0.774 0.744 0.726 0.755
MAP-R Winter 2007 0.782 0.782 0.796 0.767 0.780 0.711 0.786
MAP-R Fall 2006 0.769 0.750 0.793 0.768 0.768 0.755 0.806
High MAP-Read (F06, W07, S07) 0.804 0.779 0.816 0.784 0.791 0.746 0.804
High MAP-Read High MAP-Math 0.814            
High MAP-Read WASL06 Read scale   0.803 0.835 0.824 0.816  0.780  
High MAP-Read WASL 04_Rscale             0.787
10
Snooping (Math)R-values
WASL 2007 Math vs. Grade Grade Grade Grade Grade Grade Grade
WASL 2007 Math vs. 3 4 5 6 7 8 10
WASL Math 2006   0.801 0.840 0.875 0.873 0.890  
WASL Math 2005       0.603      
WASL Math 2004             0.856
MAP-Math Spring 2007 0.817 0.862 0.862 0.865 0.888 0.896 0.828
MAP-Math Winter 2007 0.809 0.851 0.860 0.863 0.883 0.906 0.856
MAP-Math Fall 2006 0.794 0.817 0.848 0.833 0.879 0.907 0.865
High MAP-Math (F06, W07, S07) 0.832 0.877 0.879 0.878 0.902 0.915 0.879
High MAP-Math High MAP-Read 0.846            
High MAP-Math WASL06 Math scale   0.892 0.896 0.910 0.921 0.930  
Highest MAP-M WASL04 Math scale             0.908
11
What we learned by snooping. . .

• Correlations were generally good.
• Reading R-value range 0.711 - 0.835
• Math R-value range 0.603 - 0.921
• Correlations in math were stronger than in
  reading.
• Highest MAP consistently correlated better than
  any single MAP score.
• Correlations were generally strongest when
  Highest MAP and WASL 2006 factors were combined.

12
Regression Models

• For students with both MAP and 2006 WASL scores
  (95)
• WASL 2007 b0 b1Highest MAP b2WASL 2006
• For students that only had MAP score(s) (3)
• WASL 2007 b0 b1Highest MAP
• For students that only had WASL 2006 score
  (2)
• WASL 2007 b0 b1WASL 2006
• Where
• Highest MAP The students highest score on MAP
  from the Fall 2006, Winter 2007, or Spring
  2007 windows.
• Typically Spring 2007.
• WASL 2006 The students raw score from the
  2006 WASL Spring testing.

13
Prediction Models

• For students with both MAP and 2007 WASL scores
• WASL 2008 b0 b1Projected MAP b2WASL 2007
• For students with only MAP score(s)
• WASL 2008 b0 b1Projected MAP
• For students with only WASL 2007 score
• WASL 2008 b0 b1 WASL 2007
• Where
• Projected MAP Projected Spring 2008 MAP score
  based on the students highest score on MAP
  from the Winter 2007, Spring 2007 or Fall
  2008 windows.
• WASL 2007 The students raw score from the
  2007 WASL Spring testing.

14
Projecting MAP to Spring

• For the models with Projected MAP as one of the
  factors individual student performance on MAP in
  the Spring of 2008 was projected.
• The amount of expected growth added to a
  students Highest MAP score came from NWEAs
  Growth Study

15
Example of Projection and Prediction7th Grade
Student in Reading
MAP Scores MAP Scores MAP Scores
Winter 2006 Spring 2007 Fall 2007 Highest MAP
210 212 216 216
Expected MAP Growth
3
WASL 2007 Projected MAP Spring 2008
392 219
Predicted Predicted
Prediction Model 2008 WASL Score 2008 WASL Range
MAP and WASL 399 392 - 406
MAP Only 402 395 - 409
WASL Only 397 390 - 404
16
WASL Prediction Range

• Constructed using the SEM values reported in the
  2001 WASL Technical Reports.
• Predicted Range Predicted WASL Score /- SEM

Grade Level SEM Reading SEM Math
3, 4, 5, 6 7 12
7,8 7 16
10 10 12
17
Examining the Predictions

• What are the predictions saying about how we
  might do in 2008?
• Forward look
• How would we have done if we had predicted 2007
  WASL scores in the fall of 2006?
• Backward look

18
What are the predictions saying about how we
might do in 2008?
19
What are the predictions saying about how we
might do in 2008?
20
What are the predictions saying about how we
might do in 2008?
21
What are the predictions saying about how we
might do in 2008?
22
What are the predictions saying about how we
might do in 2008?
23
What are the predictions saying about how we
might do in 2008?
24
Looking BackwardsHow would we have done
predicting the 2007 WASL?

• Successful prediction
• Accurately predicting whether a student would or
  would not meet standard on the WASL
• Unsuccessful prediction
• Predicted to meet standard and did not
• false positive (the kind we dont want)
• Predicted not to meet standard but did
• false negative (the kind we are okay with)

25
Looking Backwards - Math
Grade Successful Prediction () Unsuccessful Prediction Unsuccessful Prediction
Grade Successful Prediction () False Positives () False Negatives ()
3 80.8 7.2 12.0
4 87.3 6.3 6.4
5 86.8 5.2 8.0
6 86.8 4.8 8.4
7 89.8 4.5 5.7
8 89.0 5.3 5.7
10 86.9 5.5 7.6
District 86.8 5.5 7.7
26
Looking Backwards - Math
  Actual Met Predicted to Meet Actual Met Predicted to Meet
Grade Number of Students Percent of Total
3 -55 -4.8
4 -1 -0.1
5 -33 -2.8
6 -45 -3.6
7 -13 -1.2
8 -5 -0.4
10 -19 -2.1
District -171 -2.2
27
Looking Backwards - Reading
Grade Successful Prediction () Unsuccessful Prediction Unsuccessful Prediction
Grade Successful Prediction () False Positives () False Negatives ()
3 80.5 9.1 10.4
4 83.1 9.0 7.9
5 81.8 10.3 8.0
6 83.1 7.8 9.0
7 82.5 7.3 10.2
8 81.0 8.3 10.7
10 87.7 8.2 4.2
District 82.7 8.6 8.8
28
Looking Backwards - Reading
  Actual Met - Predicted to Meet Actual Met - Predicted to Meet
Grade Number of Students Percent of Total
3 -15 -1.3
4 14 1.2
5 27 2.3
6 -15 -1.2
7 -32 -2.9
8 -27 -2.5
10 35 4.0
District -13 -0.2
29
The Implementation

• Fall Predictions
• Rolled out at Fall Math Summit
• Cut Scores
• Released in November 2007
• BSI Status Updates
• Delivered in February 2008
• Use of the information was determined within each
  building by principals, coaches, and teachers.

30
The Results

• How good were the predictions?
• We wont know how good they are until after we
  get our WASL results.
• Come see our Fall WERA presentation!
• Did the products work for the end user.
• Feedback has been fairly limited
• Most feedback has been positive
• Some feedback says more work is still needed.

31
The Future

• Check the predictions after WASL results are
  released
• Continue to refine the products to make them
  work for the end user
• job security

32
Work in ProgressCut Scores Document

• Predictions Roll Out
• Cut Score Table
• Augmented with BSI graph
• NWEAs recently released Cut Scores document

33
Email to Principals

• If the prediction range is
• Entirely below 400 (ex. 380-396) student has
  less than a 20 chance on the WASL this spring
  unless we accelerate their learning.
• Straddles 400 (ex. 396-410) student has
  basically a coin-flip chance on the WASL, even if
  their prediction is above 400.
• Entirely above 400 (ex. 408-424) student has
  more than an 80 chance on the WASL in the
  spring, IF they continue to progress.

34
(No Transcript)
35
(No Transcript)
36
Contact Information

• David Dreher, Project Coordinator
• Dr. Kathryn Sprigg, Assistant Director
• Office of Accountability, Highline Public Schools
• www.hsd401.org
• 206-433-2334