The PASSCSIAC Data Study

1
The PASS/CSIAC Data Study on the Achievement Gap
WestEd The Partnership for the Assessment of
Standards-based Science Kathy Comfort, PI/PD and
Tamara Kushner, SRA Life Lab Science Program Jane
Lee Delgado, Executive Director Funded by the
National Science Foundation, ESR-0082024 (PASS
was formerly CSIAC--the California Systemic
Initiatives Assessment Collaborative)
2
Purpose
To conduct a secondary data analysis of PASS
Science Assessment results in order to address
several research questions related to the
achievement gap. To examine whether differences
in teacher and student characteristics,
attitudes, behaviors and opportunities-to-learn
are related to and can predict improved science
learning and a reduction in the achievement gap.
3
Overview of PASS

• Funded by NSF to assist systemic initiatives
  working to reform science education
• with the administration, scoring and reporting of
  valid and reliable
• standards-based science assessment in order to -
• Measure student achievement in science
• Measure growth against local and national
  standards
• Inform instruction and guide professional
  development
• Provide baseline and comparison data
• Show program improvement
• Analyze achievement and effectiveness of
  programs for different
• groups of students--ethnicity and gender within
  ethnicity
• PASS was developed by interdisciplinary team of
  SI leaders, measurement specialists, scientists,
  science educators and teachers.

4
The PASS Science Assessment

• Available at the elementary (5th), middle school
  (8th) and secondary (10th) levels
• Test design and specifications grounded in
  research
• Valid and reliable (good technical data)
• Aligned to the National Science Education
  Standards and to the Benchmarks for Science
  Literacy
• Contains multiple measures
• Results support educational decisions
• Meets the requirements of good science
  assessment, including NCLB

5
PASS Assessment Components

• Teacher Survey
• Professional Development
• Instructional Strategies/Practices
• Curricular Areas
• Standards
• Assessment

• Performance Tasks
• Inquiry and investigations using equipment.
• Solve problems and communicate understandings.
• Make observations, generate and manipulate data
  analyze data and draw conclusions based on their
  results.
• Scoring rubrics

• Constructed Response Investigations
• Inquiry and investigation, no equipment.
• Analyze a problem and conduct a secondary data
  analysis.
• Construct questions or models.
• Recommend solutions.
• Scoring rubrics.

• Open-ended Questions
• Communicate scientific understandings.
• Inquiry
• Design experiments.
• Use science to express positions on societal
  issues.
• Scoring rubrics.

6
PASS/CSIAC Data Study Research Questions

• Is there a gap in student achievement among
  groups?
• If a gap exists, is it becoming smaller or
  larger?
• Do different types of assessment components
  (multiple-choice questions, open-ended questions,
  performance tasks, constructed response
  investigations) widen or narrow the achievement
  gap among groups of students?
• What do the correlations between PASS/CSIAC
  student data, teacher surveys and
  opportunity-to-learn results for students from
  different groups tell us?

7
Methods

• I. Comprehensive Review of the Literature
• - Causes of the gap
• - Strategies and programs designed to reduce the
  gap
• Secondary Data Analysis
• Findings and Lessons Learned

8
I. Comprehensive Review of the Literature
Causes of the Gap Poverty and Inadequate
Resources (Johnson and Viadero, 2002
OSullivan, et. al., 1998
Kohr et al, 1991 Reyes Stanic,
1988) Academic Literacy Primary Language
(Rollnick, M., 1998 Lee, O., Fradd, S. Hl,
1996 Lee, O., Fradd, S.H.,
and Sutman, R. X., 1995) Resegregation and the
Urban Context (Wang Kovach, 1996 Yancy
Saporito, 1994 Allington Johnston, 1989
Oakes, 1985) Parenting and World View (Jenks
Phillips, 1998 Haycock, 1998 Johnson Viadero,
2000 Lee Fradd,
1998) Effects of School Practices (Phillips, et
al, 1998 Lynch, 2000 Ogbu Fordham, 1986
Oakes, 1990 Brewer, Rees Argys,
1995) Teacher Expectations and Quality (Jenks
Phillips, 1998 Good, 1987 Bamburg, 1994 Kahle,
2000 Haycock, 1998 Darling-Hammond, 2000
Ferguson, 1991 Goldhaber Brewer,
1996) Tracking and Access to Coursework
9

• Strategies and Programs for Closing the Gap
• Teacher Effects
• ( Haycock, 1998 Sanders Rivers, 1996
  Wright, Horn Sanders, 1997 Goldhaber Brewer,
  1996 Darling-Hammond, 2000 Clark, 1996
  Williams, 1996)
• School Effects
• (Johnson and Viadero, 2002 Oakes, 1990a
  Brewer, et al, 1995 CGCS, 1998 Hoffer, 1992
  Loveless, 1998)
• Parent Effects
• (Scarella Rumberger, 2000 Wang Kovich,
  1996 Viadero Johnson, 2000 Kirst Kelly,
  1995)
• Programs
• - School Reform
• - Programs Supporting Individual Students and
  Families

10
II. Secondary Data Analysis

• Between 1999 and 2002, approximately 200,000
  students in grades 5, 8, and 11 in NSF systemic
  initiative programs were administered all
  components of PASS. (PASS is a self-selected
  sample not stratified or random.)
• We initially focused our analysis on one year of
  PASS data from one large USP that used PASS over
  multiple years.
• In order to increase the reliability of our
  results, we replicated the analysis using four
  years (1999-2002) of data from two large USPs in
  two regions (Eastern and Western) of the US.
• Our total sample consisted of 31,000 students

11

• FYI
• This exploratory mapping of data is an NSF funded
  Small Grant for Exploratory Research.
• Not reporting on data that did not have
  significant relationship to student performance.
• Reporting our findings, not the answers!
• Each variable were reporting on is a study in
  its own!

12

• Variables Investigated
• Size of the gap.
• Academic Literacy and Primary Language
• Teacher Expectations, Quality and preparation
• Classroom Practice
• Resources and Technology
• Student Educational Motivation and Worldview
• Tracking and Access to coursework
• The Literature Review Preliminary Findings

13
III. Findings and Lessons Learned

• The size of the gap.
• Review of the Literature

• Size of the gap varies based on type of test and
  sampling strategy.
• On many standardized achievement tests with
  stratified random sampling the white-black score
  gap is about 1SD and the white-Hispanic gap is
  around .7 SD (NAEP, 1999 NELS, 1988).
• On self-selected tests (SAT I, ACT, GRE, etc.),
  the white-black score gap estimates range from
  .68 and .83 SDs to 1.08 and 1.11 SD. The gap
  between whites and Hispanics is around .6 SDs.
• Studies using simple composite SES measures of
  environmental factors have been able to to
  explain about 30 of the gap (Phillips et al.,
  1999).
• Studies using more detailed measures of
  environmental factors account for as much as 65
  of the gap (Phillips et al., 1999 Jencks
  Phillips, 1998).

14

• Preliminary Findings
• Our data analysis involved twelve data sets and
  over 31,000 students.
• Gap by item type for West district (Grades 5
  8)
• Estimates of white-African American gap range
  from
• 1.34 SD units on EMC to .35 SD units on PT.
• Estimates of white-Asian American gap ranged
  from .41
• on EMC to -.22 on PT.
• Gap by item type for East district (grades 5,
  8, and 11)
• Estimates of white-African American gap range
  from
• .92 SD units on EMC to .25 on PT.
• The trend appears to be a closing of the gap with
  use of Performance Tasks, and a widening of the
  gap with use of Multiple Choice.
• Possible explanations for smaller gap with
  non-multiple choice items demanding verbal
  reasoning and reading comprehension required by
  EMC and student ability to communicate
  content/concept understanding while documenting
  critical thinking and hands-on activities with
  Performance Tasks.

15
2. Academic Literacy and Primary Language Issues

• Review of the Literature
• Cognitive academic language proficiency is
  critical for demonstrating content achievement,
  especially as students advance in the educational
  system (Rollnick, M., 1998).
• Teachers interact differently with English
  Learner students in ways that limit their content
  and language development (Verplaetse, L. S.,
  1998).
• English Learner instructional environments
  exhibit differences in opportunities for science
  learning and in cultural patterns of science
  knowledge, vocabulary, and science strategy use
  (Lee, O., Fradd, S. H., 1996 Lee, O., Fradd,
  S. H., and Sutman, R.X., 1995).
• Research in cognitive psychology, learning
  theory, memory, and comprehension supports (1)
  the connection between language and conceptual
  development and (2) instructional practices that
  integrate a variety of modalities for cognitive
  processing (OMalley Chamot, 1990 Anderson,
  1990 Sadoski, Paivios Goetz, 1991, and others.

16

• Preliminary Findings
• Western District
• Hispanic/Latino and Asian American students who
  come from homes where English is the primary
  language used at home do better on PASS,
  particularly on the multiple-choice items.
• At 5th grade, difference is not significant for
  Open-Ended or Performance Task components.
• At 8th grade, all components are related to by
  the issue of English as the primary language used
  at home for Latinos, but not for Asians.

17
3. Teacher Expectations, Quality and Preparation

• Review of the Literature
• Teachers behave in different ways based on their
  estimation of student potential students
  considered low achievers have a very different
  classroom experience in terms of teacher-student
  interaction (Good, 1987).
• High quality teaching professional high
  performance expectations for all students verbal
  and mathematical competence subject-matter
  certification in science deep content knowledge
  effective pedagogical skill (Jencks Phillips,
  1998 Ferguson, 1991 Stevens, 1995 Sanders
  Rivers, 1998).
• Schools with high concentrations of poor and
  minority students are more likely to be taught by
  under-prepared teachers (Kahle, 2000
  Darling-Hammond, 2000).
• In one large Western urban district, 60 of
  3,000 newly hired teachers had emergency
  licenses.
• In same district, more than half of high school
  mathematics teachers had neither a major or minor
  in mathematics.

18

• Preliminary Findings
• Information on academic content preparation and
  credentialing were not collected 1999 2002 for
  PASS.
• PASS teacher surveys asked teachers about their
  experience with teacher professional development
  programs and the influence of standards. PD
  options included participation in systemic
  initiatives and other programs.
• In the West district, there were few significant
  correlations for teacher professional
  development. Particularly in grades 5 and 8,
  options were related negatively (possible
  interpretation some professional development
  programs are only offered in low performing
  schools.)
• In the East district, a special survey was
  created for grade 11 in 2002 referring to
  specific district PD programs. Here the effect
  size of the teacher professional development
  relationship was larger.

19
4. Classroom Practice

• Review of the Literature
• Low-income, minority and urban classrooms
  exhibit different instructional practices from
  those in high-performing schools (Oakes, 2001).
• Almost all goals are less emphasized,
  expectations are lower, and instruction is less
  engaging (Oakes, 1990).
• Students get a lower-level curriculum dominated
  by exercises, workbooks andbasic skills kits
  (Oakes, 1990).
• Teachers with less training in science teaching
  tend to emphasize memorization rather than
  laboratory techniques or discussions on
  application of content concepts (Darling-Hammond,
  2000).
• The difference between an effective and less
  effective teacher can be a full level of
  achievement in a single school year (Hanushek,
  1992).

20

• Preliminary Findings
• Students and teachers were asked to report how
  often they participated in or used various
  instructional strategies in the classroom.
• In grades 5, 8, and 11, experimentation with
  partners or groups was positively associated with
  the test performance of African American and
  Asian American students.
• For grades 5 and 8, spending time on a long-term
  project is consistently negatively correlated for
  all groups. (Possible explanation these students
  are spending less time on a breadth of science
  content.)
• For grade 8, as reported by teachers, the use of
  worksheets is negatively correlated with
  achievement. It is particularly negative for
  Performance Tasks. (Possible explanation
  students not thinking critically when spending
  time on worksheets).

21
5. Resources and Technology

• Review of the Literature
• Schools in impoverished areas often receive less
  funding and support even when funding is
  similar, suburban schools can access more
  parental support, community resources, and other
  education supports (Viadero, 2000).
• Disadvantaged schools more likely to have crowded
  and deteriorating school facilities, fewer
  materials, lower quality or no lab facilities,
  and fewer certified teachers in science (NSF,
  1996).
• School conditions impact subgroups differently
  within the same school scarce resources are more
  likely to be allocated to predominantly white
  classrooms (Kohr, et al., 1991 Reyes Stanic,
  1998).
• Students in urban schools appear to have less
  access to computers and computer staff, science
  labs, and related resources (Oakes, 1990).
• Achievement effects have been related to both
  funding for instruction and capital expenditures
  (Wenglinsky, 1998) upkeep (Harter, 1999) and
  classroom resources (Elliott, 1998).

22

• Preliminary Findings
• Teachers were asked if they had the resources
  they need. Fairly positive correlation.
  Interesting note both ends were positiveif
  resources supplied by school, or if teachers
  provided their own resources. (Possible
  explanation teachers who buy their own materials
  may be more motivated teachers.)
• Teachers were also asked how much time they
  spent using technology. In general, there was a
  negative correlation with student performance at
  grade 8 for all groups and item types.
• Grade 8 student reports about the use of
  computers also showed a negative correlation with
  performance, although smaller. (Possible
  explanation less discussion, less attention to
  critical thinking, processing, and/or ineffective
  use of technology resources.)

23
6. Student Educational Motivation and Worldview

• Review of the Literature
• Traditional view interprets the effect of peer
  pressure as a means of devaluing academic
  success as a cause of the achievement gap (Jencks
  and Phillips, 1998).
• Newer evidence suggests that negative peer
  pressure may instead be a symptom, resulting from
  real-life observation that education has failed
  to improve the conditions of life for members of
  the students home community (Jencks Phillips,
  1998 Ogbu Fordham, 1986 Ogbu, 1991).
• Peer pressure as a symptom rather than a cause
  may especially apply to involuntary minorities,
  African American or long-term English learners
  who have developed a protective oppositional
  culture over years or generations (Ogbu, 1991).
• Children from neighborhoods with few
  college-educated adults often have fewer
  successful role models, less encouragement to set
  high goals, and may have pressure to work or
  marry early (Hayward et al., 1997).

24

• Preliminary Findings
• Students were asked if knowing science would be
  helpful in the future (grades 5 8).
• Data showed a relatively strong relationship for
  African American and Asian Americans students in
  grade 5 for grade 8, relationship was less
  consistent or widespread, still mostly positive.
• Asked if they liked science (11th grade, East
  district), there was a relatively strong
  correlation with performance for African
  Americans.
• Time spent doing homework showed positive
  correlation with achievement. For 5th grade, the
  relationship was strongest in the East for 8th
  grade, it was equally strong in East and West.
  (Possible explanation could be student
  motivation, could also be home support for
  homework.)
• Time spent reading science textbooks in the
  classroom showed a positive relationship in the
  West for 8th grade, less in the East. (Question
  of time and support for reading as in homework
  effect appeared about the same for all item
  types.)

25
7. Tracking and Access to Coursework

• Review of the Literature
• One view tracking segregates students by class
  and race, as well as ability, and negatively
  affects performance of lower-track students
  (Oakes, 1990 Stockard Mayberry, 1992)
  tracking contributes to the achievement gap
  between low and high ability students (Brewer, et
  al., 1995).
• Other view tracking not convincingly proven to
  be associated with achievement differentials
  (Hoffer, 1992 Kulik Kulik, 1991).
• Science and math experiences differ by ethnic
  group as early as elementary school, become
  striking by high school--differential access by
  low income and minority students to rigorous
  coursework, less exposure to teaching goals and
  strategies most likely to generate interest and
  promote learning (Oakes, 1990).
• Prevalent ability grouping often places African
  American and Hispanic students into course-taking
  patterns, especially in science and math, that
  preclude entrance into four-year colleges and
  universities (Hayward, et al., (1997).

26

• Preliminary Findings
• Teachers were asked the question about how much
  time students spent on science content and
  processes of science.
• Grade 5 shows a positive correlation with
  achievement for all groups. This continues for
  African American students and Hispanic/Latinos in
  grade 8, but no longer for Asian Americans.
• At grades 5 and ,8 exposure to physical science
  content, and at grade 11 (East district), taking
  physics, is positively related to performance on
  PASS.
• Time spent on Earth Science at grade 5 (West)
  shows a positive relationship to achievement for
  Hispanic/Latino and Asian American.
• Time spent on Earth Science at grade 8 and 11in
  general shows a negative correlation for all
  groups. (Possible explanationby middle school,
  time spent on Earth Science may be reducing time
  spent on physical and life science and maybe an
  indicator of a tracked curriculum.

27
Possible Next Steps

• Potential next steps will focus on linking
  research and practice in science reform efforts
  related to reducing the gap, based on the
  findings described here, and in our final report.