Student Mobility and AYP

1
Student Mobility and AYP

• Valid Score Study
• June, 2006
• Wendell Callahan Claudette Inge, SDCOE
• Ken Taylor, KCSS
• Ted Price, OCDE
• Vicki Barber, EDCOE

2
The Problem

• County Offices of Education as LEAs experience
  student mobility of a magnitude much greater than
  school districts because of the nature of the
  COE-operated schools
• Higher student mobility makes accurate and
  statistically defensible determination of
  Adequate Yearly Progress (AYP) problematic

3
Percent Valid Method

• Data Source Corrected 2005 California
  Department of Education Accountability Progress
  Report (APR) Data File
• Percent Valid Scores-
• Calculation
• Number of Valid Scores
• ( ) X 100
• Enrollment on First Day of Testing
• students enrolled in the LEA or school
  continuously from CBEDS day to
• the first day of testing

4
Percent Valid Study Sample

• 10,290 AYP records were processed
• School records without AYP data were not included
  in the analysis (n156)
• AYP records included
• 1 state aggregate
• 58 county offices of education
• 14 COE direct-funded charter schools
• 977 school districts
• 273 school district direct funded charter schools

5
Percent Valid Results
6
Percent Valid Results
7
Percent Valid Results
8
Percent Valid Simulation

• Apply 70 Valid Score threshold to COE and School
  District AYP data
• Less than 70 Valid Scores poses a potential risk
  to the statistical integrity of the valid score
  sample for AYP determination

9
Simulation Results
10
Valid Score Study Purpose Rationale

• Investigate the effect of low Percent Valid
  Scores on the generalizability of the COE Valid
  Score Sample
• If the valid scores do not represent a
  generalizable sample of student performance, then
  the validity of an AYP determination made on the
  basis of these scores is questionable
• Based on a possibly faulty AYP determination,
  County Offices of Education will be identified
  for Program Improvement (PI) and be subject to a
  range of potential interventions, including
  possible reorganization

11
Low Percent Valid Scores in San Diego Resulted in
A Biased Valid Score Sample
Source SDCOE 2004-05 enrollment data
12
Valid Score Study Method

• An explanation of the study, data collection
  instructions and an Excel spreadsheet were
  e-mailed to 34 County Offices of Education
• Data from COE Special Education programs were not
  included in the analysis
• 11 COEs with fewer than 50 valid scores were not
  included in this phase of the study

13
County Offices of Education with Enrollment Less
than 50

• A sample size of 50 or less makes accurate
  statistical decision-making difficult because of
  the presence of error

Note An enlargement of this table is included
in the handout
14
Dealing with Less than 100 Valid Scores The
Confidence Interval

• Currently, the CDE applies a confidence interval
  to the Percent Proficient when there are less
  than 100 Valid Scores
• The confidence interval is based on the Standard
  Error of the Proportion
• The Standard Error of the Proportion tells us how
  much error is associated with an observed
  proportion
• i.e., the margin of error associated with the
  percent proficient calculated from a small number
  of test scores

15
County Offices of Education with Less than 100
Valid Scores
Note An enlargement of this table is included
in the handout
16
Valid Score Study Method

• After calculating the Standard Error of the
  Proportion, the valid score sample was deemed
  unrepresentative of the COEs annual enrollment
  if
• The Standard Error exceeded .05 (e.g., more than
  5 percent margin of error) for at least one
  school or program type
• In this case, the Standard Error of the
  Proportion (i.e., margin of error) tells us how
  much confidence we should place in the sample for
  further analysis and decision-making.

17
Valid Score Study Preliminary Results

• 100 of COEs had unrepresentative valid score
  samples
• 22 (out of 22) had unrepresentative valid score
  samples based on school or program type
• 18 (out of 20) had unrepresentative valid score
  samples based on grade span
• 2 COEs did not report complete grade span data

18
Valid Score StudyPreliminary Results Summary
Table
Note An enlargement of this table is included
in the handout
19
Tentative Conclusions

• High student mobility resulting in low Percent
  Valid Scores appears to influence the composition
  of the valid score sample
• The resulting valid score sample is typically not
  proportionally representative of the students
  enrolled in the school, program and/or grade span
  of the COE on an annual basis
• The disproportionality of the valid score sample
  potentially introduces significant amounts of
  error into the AYP determination

20
Discussion

• Propose expansion of current use of confidence
  interval method or County/District averages to
  LEAs and schools based on 70 Valid Score
  threshold
• Propose pilot alternative AYP method based on
  more appropriate and accurate measurement
  criteria
• i.e., ASAM Plus
• Include longitudinal (historical), value-added
  and transition measures

21
Appendix 1 Preliminary Sample Error Estimates
for County-Operated Programs