School Improvement and Educational Accountability

1
School Improvement and Educational Accountability

• M. David Miller
• University of Florida

2

• A major shortcoming of current
• accountability systems for purposes of making
  valid inferences about school quality is due to
  severe limitations on the data that are generally
  included in systems (Linn, 2007)

3

• The growing consensus is that in the era of
  standards based reform, a major missing piece of
  the puzzle is data. Every state must quickly
  develop a robust system of student data and
  information that allows us to not only report on
  assessment data for state and NCLB accountability
  purposes, but to use those data for diagnostic
  and instructional decision making (Wilhoit,
  2006)

4
Accountability Systems and Multiple Uses

• NCLB / AYP
• Individual Uses
• Diagnostic/ Instructional
• Graduation
• Retention/Promotion
• Course Credit

5
School Use of Assessments

• Designs for School (e.g., matrix sampling)
• fewer items per student
• broader coverage
• noncomparable student level scores
• reduced testing time

6
Diagnostic Testing

• Student v Group level
• Level of scores - Subscores v Benchmark
• Reliability
• Validity

7
Florida Content Focus Clusters
Content Area Content Focus Cluster
Reading Words and phrases in context Main idea, plot, and purpose Comparisons and cause/effect Reference and research
Mathematics Number sense, concepts and operations Measurement Geometry and spatial sense Algebraic thinking Data analysis and probability
Science Physical and chemical sciences Earth and Space sciences Life and environmental sciences Scientific thinking
8
Spearman-Brown projected subscore reliabilities
Subscores ?.85 ?.90 ?.95
3 .65 .75 .86
4 .59 .69 .83
5 .53 .64 .79
9
Reading benchmarks Main idea, plot, and purpose

• Chronological order
• Details and Facts
• Authors Purpose
• Character Development
• Plot Development

10
Mathematics benchmarks Algebraic Thinking

• Functions
• Graphic Patterns
• Linear Equations
• Solving Equations
• Equations
• Inequalities
• Solving Inequalities
• Solving Equations
• Translating Inequalities

11
Science benchmarks Scientific Thinking

• Experimental Design
• Drawing Conclusions
• Observation
• Communicating Results
• Understanding Models
• Predictable Events
• Using Machines
• Problem Solving
• Scientific Applications

12
Spearman-Brown projected benchmark reliabilities
Benchmarks ?.85 ?.90 ?.95
15 .27 .36 .56
25 .18 .26 .43
40 .12 .18 .32
13
Spearman-Brown projected factors for test length
with benchmark reliabilities.70
Benchmarks ?.85 ?.90 ?.95
15 6.31 3.81 1.83
25 10.63 6.64 3.09
40 17.11 10.63 4.96
14
Diagnostic Validity and Test Construction

• Adding items that fit overall test blueprint and
  information function
• Dropping items that do not fit overall test

15
Additional Indicators to Interpret Accountability
Data (Conditions of Education)

• Participation
• Student Effort and Progress
• student attitudes and aspirations
• student effort absenteeism
• Contexts of Education
• course taking and standards
• learning opportunities
• special programs
• teachers
• school characteristics and climate

16
Successful and Unsuccessful Schools

• In current accountability systems, student intake
  and instructional effectiveness are confounded to
  some unknown degree, calling into question any
  inferences about school effectiveness from these
  data (Raudenbush, 2004)

17
Summary

• More information needed to understand causes of
  success or lack of success
• Value-added adjustments?
• Diagnostic information reasonable in this context
  given issues of reliability and validity ?