Exploring Data Use

1
Exploring Data Use School Performance in an
Urban School District

• Kyo Yamashiro, Joan L. Herman, Kilchan Choi

UCLA Graduate School of Education Information
StudiesNational Center for Research on
Evaluation,Standards, and Student Testing
(CRESST) CRESST ConferenceUCLASeptember 8, 2005
2
Context Background

• Large urban school district in the Pacific
  Northwest
• Value-added Assessment System implemented in
  District
• Need for more info on schools use of data (VA
  and other)

3
Data Use Evidence-based Practice

• Data use at the heart of test-based reforms
  (NCLB) continuous improvement efforts
• Little evidence of effects of data use on
  performance
• Some evidence shows limited access and capacity
  of schools to use data

4
Study Components

• CRESST conducts multi-year, multi-faceted study
  of data use
• Transformation Plan Review - content analysis of
  school improvement plans
• Interviews, surveys, and observations from site
  visits of case study schools
• Analysis of district achievement and survey data
• Observations of school presentations about
  progress

5
Sampling

• Latent variable, multilevel analyses used to
  estimate gains (student-level, longitudinal ITBS
  data in reading math)
• Gains based on growth from 3rd to 5th grade for 2
  cohorts in each school
• 3rd graders in 1998
• 3rd graders in 2001
• Within each cohort, 3 performance subgroups
  (average, low, high)

6
Sampling (contd)

• 13 Schools met the following criteria
• Greater than district average of low-SES
  students
• Starting point below district average
• Beat the Odds Sample (7)
• Higher than average gains
• Relatively more consistent across
• 2 cohorts (98 01)
• reading and math
• performance subgroups (hi, avg, lo)

7
Sample

• Extremely diverse set of 13 small, elementary
  schools
• African American student populations between 11 -
  81
• Asian American student populations between 2 -
  59
• White student populations between 5-59
• Enrollment range 134 to 533

8
Transformation Plan Review

• TP Review Rubric (Rating of 1 to 3)
• Types of evidence or indicators used
• Breadth depth VA data technical sophistication
• Identification of goals/objectives or needs
  analysis
• Identification of solution strategies
• Specificity based on theory/ research/data
• Analysis of progress
• Inclusion of stakeholders

9
Case Study Site Visits

• 2-day visits to 4 case study sites
• Interviews/focus groups
• Principal
• Building Leadership Team (BLT)
• Teachers (primary, upper)
• Teacher Survey

10
Additional Achievement Analyses

• Latent Variable Multiple Cohort (LMC) Design
  (with SEMs)
• Estimating gains on ITBS based on data across 5
  cohorts (1998 to 2002)
• Gains for performance subgroups
• Average (students starting at school mean initial
  status)
• High (students starting at 15 points above
  schools average)
• Low (students starting at 15 points below
  schools average)
• Patterns of growth differ from 2-cohort analysis

11
Results Achievement

• Differences between Pre- and Post-Transformation
  Plan Reform
• High/Avg 4 schools - consistent growth across
  rdg math subgroups
• Low 6 schools - left some subgroups behind in
  math and/or rdg
• Very Low 3 schools - no growth or negative gains

12
Results Data Use

• Data Use Is Improving but Still Varied
• Over 3 years, schools increased use of assessment
  results and other evidence
• Schools increased mention of VA data
• Data Review Process is Inclusive When Capacity
  Exists
• Principal often conduit (filter, interpret)
• However, many schools developed collaborative
  processes for data review
• Transf Planning Process May become More
  Centralized (Less Inclusive) in Later Years

13
Results Data Use (contd)

• Accessible and Excessive Data
• Teachers use data for schoolwide reform and (to
  lesser degree) instructional planning
• Teachers are overwhelmed with amount of data
• More Capacity Needed
• Whether schools integrate data into instructional
  decisions tended to be person- or climate-driven
• Principals need help, too
• More Diagnostic, Instructionally Sensitive Data
  Needed
• State testing data not seen as useful, valid,
  timely, or interpretable
• lack of continuity in tests (from grade to grade)
• lack of diagnostic info (item analyses)
• lack of individual growth info (pre-post)
• District assessments seen as more helpful to
  instruction

14
Results Data Use Achievement

• Pre-Post Gains Data Use Practices

15
Results Data Use Achievement (contd)

• Ratings overlap for 7 of 13 schools
• For the most discrepant case (Polk)
• showing high gains but low data use
• school in chaos, with new leadership
• For remaining 5 moderate discrepancies, no case
  study data

16
Conclusions

• Less use of data for instructional planning
  probably a function of
• type of data provided
• leadership climate
• capacity
• Principals and teacher leaders need more help in
  interpreting and using data
• Data use and gains appear to have a moderate link
  for struggling schools more case study info
  needed
• Need for more research on how to use value-added
  (gains) in an accountability setting