Accountability Validity: Frameworks and Examples

1
Accountability ValidityFrameworks and Examples

• Brian Gong
• Center for Assessment
• Presentation at the CCSSO ASR/CAS SCASS Meeting
• April 29, 2004 Minneapolis, MN

2
Implementing a Validity Plan

• Start today
• Prioritize
• Know what is important
• Plan for three or four years ahead
• Gather baseline information
• Learn from others
• Weave it into your daily operations and policy

3
Overview

• Accountability System Validity
• Why?
• What?
• A general framework
• A specific framework
• How?
• Some examples (emphasis on study design, not
  results)

4
Accountability Validity Why?

• Would you like to be able to respond to questions
  such as
• How do you know your system is fair?
• How do you know it is working?
• Is the system making people do bad things?
• What improvements would you like to make? Why?

5
A High-level Validity Framework

• Accountability determinations accuracy and
  consistency
• Accountability actions intended consequences
• Accountability system unintended consequences

6
High-level Framework cont.

• The first component addresses the accuracy and
  consistency of the accountability determinations
  and includes such issues as misclassification
  rates, effects of additional indicators on AYP
  determinations, and the effects of various types
  of error on both the accuracy and consistency of
  these determinations. The questions subsumed by
  this category would include those of most
  interest to someone working within a typical
  accountability unit, researchers, policymakers.
• The second piece of the validity framework is
  what we have termed the whats next component.
  Questions addressed under this part of the
  framework would focus on both the effectiveness
  of the interventions being used to improve
  schools and student performance, as well as on
  the strategies used by the state and/or district
  to disseminate and support these interventions
  (including professional development). This
  component of the framework should be most
  relevant to curriculum, professional development,
  and school improvement specialists as well as
  leaders at all levels of the educational system,
  researchers, and policymakers.
• The third component focuses on questions most
  related to the positive and negative effects of
  the accountability system on other aspects of the
  educational enterprise. Study questions within
  this component would include examining changes in
  how Title I funds are distributed within
  districts, changes in how and where special
  education and English Language Learners (ELL) are
  identified and instructed in schools and
  districts, and examining whether and how school
  choices affects the educational opportunities of
  students. These questions would be most relevant
  to Title I and Title III personnel, Special
  Education directors, and policymakers.

7
A Specific Framework

• An Accountability System Technical Manual
• Similar in purpose, scope, rigor as an assessment
  system technical manual
• Should not duplicate (but remember that
  establishing assessment validity and reliability
  does not necessarily establish accountability
  validity/reliability, and vice versa!)
• Developed in a similar way
• Primary audience technical users and observers
• Single main source points to main studies, more
  extensive documentation, has technical detail
• Historical, incremental, added to yearly

8
A few notes

• I assume a NCLB AYP context, although framework
  could be applied to any state accountability
  system
• I focus on accountability validity, and assume
  appropriate work has been done to establish
  validity and reliability of assessments and other
  indicators as measures to be included in
  accountability system (see Hill, 1999 and 2000,
  for analyses of how highly reliable assessments
  could be used to produce lower reliability
  accountability systems, and vice versa)

9
Table of Contents

• Design of Accountability System
• Development of Accountability System
• Implementation of Accountability System
• Description of Results Identification
• Description of Results Consequences
• Evidence of Reliability of Accountability System
• Evidence of Validity of Accountability System
• Appendices

10
Design of Accountability System

• Design
• Legal requirements and charges of the
  accountability system (e.g., as found in statute,
  court decisions)
• Elaborated purposes and intents of the
  accountability system (e.g., All schools should
  be held accountable for student performance.
  Determination of school ratings should reflect
  student performance, and should not be based
  primarily on size, geographic location, or
  student demographics except as these are related
  to student performance.)
• Note The description of purposes and intents is
  important for setting the design, and critical
  for conducting validity studies.
• Sample 1 and 2 within NCLB framework
• All public schools and districts shall be held
  accountable for student performance. Performance
  shall be measured by valid and reliable
  indicators, including performance on the state
  assessment,
• Other contextual requirements of the
  accountability system (e.g., budget constraints,
  operational organization)
• Design of accountability system Data
• Descriptions and definitions of indicators used
• Performance standards
• Inclusion
• Who is assessed (student-level)
• Rules for non-inclusion in assessment
• Who is included in accountability
• Rules for non-inclusion and inclusion in
  accountability
• Notes about changes in population over time
• When data are collected
• How data are aggregated
• Quality assurance in data processing

11
Table of Contents

• Design of Accountability System
• Development of Accountability System
• Description of Results Identification
• Description of Results Consequences
• Evidence of Reliability of Accountability System
• Evidence of Validity of Accountability System
• Appendices

12
Development of Accountability System

• History of development Legal basis
• Development of design and rules (including
  rationale and other options considered)
• Description of changes in accountability system
  over time
• Description of changes since last edition of
  accountability technical manual

13
Table of Contents

• Design of Accountability System
• Development of Accountability System
• Description of Results Identification
• Description of Results Consequences
• Evidence of Reliability of Accountability System
• Evidence of Validity of Accountability System
• Appendices

14
Description of Results Identification

• Summary of accountability results since inception
  of program
• Description of most recent results since last
  edition of accountability technical manual
• How many schools met and didnt meet AYP total,
  for one year, two, three or more
• By grade level configuration
• Reasons for meeting (CI, exemptions, appeals)
• Reasons for not meeting
• Distance metric of how far from meeting AYP

15
Table of Contents

• Design of Accountability System
• Development of Accountability System
• Implementation of Accountability System
• Description of Results Identification
• Description of Results Consequences
• Evidence of Reliability of Accountability System
• Evidence of Validity of Accountability System
• Appendices

16
Description of Results Consequences

• Summary of intervention programs (rewards,
  assistance, sanctions) and results since
  inception of accountability system (and relevant
  history of prior accountability efforts)
• Description of most recent results since last
  edition of accountability technical manual
• Public school choice
• Eligible
• Programs (sample letters, etc.)
• How many did/did not transfer
• Description of general reasons
• Supplemental services
• Additional funding
• Corrective actions Distinguished educators,
  actions taken by districts
• Description of other supports from state,
  districts, and others (e.g., state-mandated
  school needs analysis student accountability
  tutoring and mentoring reading curriculum
  professional development)

17
Table of Contents

• Design of Accountability System
• Development of Accountability System
• Description of Results Identification
• Description of Results Consequences
• Evidence of Reliability of Accountability System
• Evidence of Validity of Accountability System
• Appendices

18
Evidence of Reliability of Accountability System

• Estimates of error rates in school/district
  identification (misclassification rates, or
  decision consistency)
• Estimates of Type I error (false identification
  as does not meet)
• Triangulate with other data on good school
• Discriminate analysis
• Estimates of Type II error (false identification
  as meets)
• Power analysis
• Triangulate with other data on schools just
  missing AMO in one or more subgroups
• Contrast school/district means for subgroups
  using above and below minimum-n
• Simulations to estimate effects of multiple
  conjunctive tests on actual alpha level
• Analyses of relative contributions of sources of
  error in addition to sampling to accountability
  decision consistency, especially for given
  distributions of student and school performance
• Description of reliability in terms of important
  school/district characteristics (e.g., size,
  geographic location, student mobility)

19
Minimizing error by design

• Type I error false positive e.g., The
  school did not meet AYP when it truly did meet
  AYP, or is a good school
• Type II error false negative e.g., The
  school DID meet AYP when it truly did NOT meet
  AYP or is not a good school
• Reliability decision consistency over time
  variation in accountability labels over time is
  not due to error (e.g., sampling error)

20
Table of Contents

• Design of Accountability System
• Development of Accountability System
• Description of Results Identification
• Description of Results Consequences
• Evidence of Reliability of Accountability System
• Evidence of Validity of Accountability System
• Appendices

21
Evidence of Validity of Accountability System

• The system was designed properly to meet the
  purposes
• The system was implemented correctly and fairly
• The system identifies schools as intended
• The system does not identify schools as not
  intended
• The system helps the priority students learn as
  intended
• The system does not disadvantage other students
  as not intended
• The system helps identified schools improve as
  intended
• The system does not disadvantage schools as not
  intended
• Check for other unintended consequences

22
Example 1 Design

• Design and Rationale
• Status, improvement, growth (effectiveness)
• Rules, targets

23
Example 2 Implementation

• Systematic quality control
• Systematic quality assurance

24
Example 3 Correct identification

• All schools should be held accountable for
  student performance. Determination of school
  ratings should reflect student performance, and
  should not be based primarily on size, geographic
  location, or student demographics except as these
  are related to student performance.

25
Met/Did not meet AYP - 1
26
Met/Did not meet by Size(!)
27
Example 4 Helps students learn

• Inclusion
• Services
• Effectiveness
• Full academic year
• SWD

28
Example 5 Helps schools improve

• Teachers
• Distribution of teachers by teacher
  qualification, degree institution, school
  performance, student demographics
• Curriculum
• School/district leadership
• Resources
• Multiple regression analysis of multiple
  variables on student achievement

29
Average Teacher Test Score by School Performance
Labels bars (state average 500)Average 2000
SPS by School Performance Labels line
30
Example 6 Unintended consequences
31
Implementing a Validity Plan

• Start today
• Prioritize
• Know what is important
• Plan for three or four years ahead
• Gather baseline information
• Learn from others
• Weave it into your daily operations and policy