Special Data Opportunities in Florida

1
Special Data Opportunities in Florida

• David N. Figlio
• University of Florida and
• National Bureau of Economic Research

2
Whats so special about Florida?

• Florida has developed a remarkable ability to
  analyze data across a wide variety of settings
• Facilitated through legislative authority but
  reflective of exceptional nurturing of
  interagency relationships that facilitate
  data-sharing
• Easier also to collect large datasets directly
  from school districts in areas not present in
  state data
• Due in part to the relatively small number of
  highly-organized school districts that are
  acculturated to data-sharing and facilitating
  policy-motivated research

3
Two main special data opportunities in Florida

• Florida Education and Training Placement
  Information Program (FETPIP) and K-20 Education
  Data Warehouse
• Established in 1988 to document compliance with
  vocational education performance requirements
• Expanded to become a system of collecting and
  sharing individual-level data across all
  education sectors (K-12, vocational education,
  community colleges, state universities)
• Follows education, employment and earnings
  outcomes longitudinally for students integrated
  into a relational database

4
Two main special data opportunities in Florida

• Independent extensive surveys of school policies
  and practices
• Collaborative effort between David Figlio (U.
  Florida), Dan Goldhaber (U. Washington), Jane
  Hannaway (Urban Institute) and Cecilia Rouse
  (Princeton U.)
• Attempted census of all public schools in Florida
  
• First conducted in 1999-2000, then followed up in
  2001-02 and 2003-04
• Surveys of teachers in 2000-01, 2002-03, and
  2004-05 in 288 representative elementary schools
• Better than 70 percent response rates in all
  waves over 80 percent longitudinal response rates

5
Examples of FETPIP data

• Data from public two- and four-year institutions,
  as well as in-state private colleges and
  universities
• In-state, data are collected on courses taken,
  programs of study, and attainment
• Florida is working with the National Student
  Clearinghouse to include enrollment and
  credential records for a large fraction of
  out-of-state college students

6
Examples of FETPIP data

• Using social security numbers, FETPIP marches
  student records with other outcome data for all
  students exiting Florida public institutions as
  well as some private exits
• Examples include information on further
  education, job placement, compensation, military
  service, incarceration
• Result a remarkable tool for policy generation,
  evaluation and research

7
School Surveys

• All regular public schools in 2001-02 and
  2003-04
• 70 response rate in each year
• 2,095 schools responded in 2002 81 of these
  responded in 2004.

8
School policies/practices can be grouped into
domains

• Policies to improve low-performing students
• Lengthening instructional time
• Reduced class size for subject
• Minimum time required for tested subject
  instruction
• Minimum time required for non-tested subject
  instruction
• Scheduling systems
• Additional school resources

• Policies to improve low-performing teachers
• Teacher resources
• Teacher incentives
• Teacher autonomy
• District control
• Principal control
• School climate

9
Feeling the Florida Heat? How Low-Performing
Schools Respond to Voucher and Accountability
Pressure

• Cecilia Elena Rouse
• Princeton University NBER
• Jane Hannaway
• The Urban Institute
• Dan Goldhaber
• University of Washington
• David N. Figlio
• University of Florida NBER

10
There is little evidence on how schools respond
to accountability pressure

• Improved teacher effectiveness and greater focus
  on basics
• Teaching to the test
• Cheating
• Reclassification of students
• Strategic suspension of students.

11
Our question Do schools change their policies
and practices in response to school
accountability and voucher pressure?
12
Our approach

• Study effects of school accountability on student
  test score performance in Florida with a change
  in the A Plan for Education
• Analyze effects of accountability on schools
  using longitudinal data on school policies
  collected from surveys of school principals in
  1999-00, 2001-02, and 2003-04
• Attempt to determine if the policy changes
  explain the test score effects.

13
We find

• Among elementary schools, student achievement
  significantly increases among F-graded schools
• F-graded schools appear to respond with policy
  changes
• These policy changes appear to explain
  non-trivial portions of the student gains,
  particularly in math.

14
Since 2002, Grade Points

• percent students meeting levels 3 in reading,
  writing, and math
• percent students making learning gains in
  reading and math
• the percentage of the bottom 25 that have
  improved scale points in reading.

15
Table 1 The Distribution of Elementary School
Grades, by Year
16
Table 2 Transition Matrix in Predicted Grades
Based on 2002 Grade Change, Elementary
Schools(row percentages)
17
Appendix Table 1 (part 1) Mean School
Characteristics in 2002
18
Appendix Table 1 (part 2) Mean School
Characteristics in 2004
19
Table 3 (part 1) Regression-Discontinuity
Estimates of the Effect of Receiving an F Grade
in Summer 2002 on Student Performance
20
Table 3 (part 2) Regression-Discontinuity
Estimates of the Effect of Receiving an F Grade
in Summer 2002 on Student Performance
21
(No Transcript)
22
(No Transcript)
23
Table 4 (part 1) Regression-Discontinuity
Estimates of the Effect of Receiving an F Grade
in Summer 2002 on Student Performance
24
Table 4 (part 2) Regression-Discontinuity
Estimates of the Effect of Receiving an F Grade
in Summer 2002 on Subsequent Student Performance
Alternative specifications
25
Table 7 (part 1) Seemingly-Unrelated Regression
and OLS Results of the Effect of Receiving an F
Grade in Summer 2002 on School Policy in 2003-04
26
Table 7 (part 2) Seemingly-Unrelated Regression
and OLS Results of the Effect of Receiving an F
Grade in Summer 2002 on School Policy in 2003-04
27
Table 8 (part 1) OLS Results of the Impact of
Receiving an F Grade in Summer 2002 on School
Selected Individual Policies in 2003-04
28
Table 8 (part 2) OLS Results of the Impact of
Receiving an F Grade in Summer 2002 on School
Selected Individual Policies in 2003-04
29
Table 9 (part 1) The Effect of Including School
Policy/Practice Variables on Regression-Discontinu
ity Estimates of the Effect of Receiving an F
Grade in Summer 2002 on Subsequent Student
Performance Fifth-Grade Cohort of 2002-03
30
Table 9 (part 2) The Effect of Including School
Policy/Practice Variables on Regression-Discontinu
ity Estimates of the Effect of Receiving an F
Grade in Summer 2002 on Subsequent Student
Performance Fifth-Grade Cohort of 2002-03
31
In sum

• We estimate effect sizes in reading test scores
  among F-graded schools between 6-10s and effect
  sizes in math between 7-14s.
• We also find that F-graded schools appear to
  focus on low-performing students, lengthen the
  amount of time devoted to instruction, adopt
  different ways of organizing the instructional
  environment of students and teachers, increase
  resources available to teachers, and decrease
  principal control.
• These policies may explain at least 10 of the
  gains in reading and at least 25 of the gains in
  math.

32
Caveats.

• F-graded schools receive additional state
  assistance (e.g., assessment and course
  materials, increased professional development for
  teachers)
• While our results suggest that schools respond to
  accountability in potentially educationally
  meaningful ways, we do not observe student
  performance along all dimensions.