Pittsburgh Public Schools

1
October 7, 2008
Pittsburgh Public Schools Magnet Program Audit
Presented By Alan J. Simon, Ph.D. Marilyn
Zlotnik Claire Aulicino
2
Overview of Presentation

• Context for magnet audit
• Research focus of magnet audit
• Magnet audit methodology
• Magnet audit preliminary findings and
  recommendations

1
3
Context for Magnet Audit

• Excellence for All reform agenda established in
  2005
• Review all programs for academic rigor
• Increase options for all students at the
  secondary level
• PPS experienced a 12 decline in enrollment over
  past five years, with alarming number of families
  leaving district, especially at high school level
• Supreme Court decision in June 2007 limited
  ability of districts to use students race in
  assignment for voluntary integration plans

2
4
Research Focus of Magnet Audit

• Analyze implementation and impact of magnet
  choices to understand alignment of offerings with
  social and economic demands
• Research components
• Market analysis of social and economic demand for
  choices
• Secondary research on best practices for magnet
  programs
• Longitudinal analysis of magnet programs over
  time
• Performance evaluation to assess program
  implementation
• Impact evaluation to examine effect of magnet
  programs on student outcomes
• Trend identification to identify patterns in
  enrollment and demand

3
5
Magnet Audit Methodology

• Mixed method research design to address questions
  in RFP
• Comprehensive review of literature
• Field visits to 29 magnet programs
• Interviews with 12 education and community
  experts and 18 Central Office staff
• Staff survey (548 staff in schools with magnet
  programs)
• Student surveys (2,930 students in grades 8, 10,
  and 12 in magnet and non-magnet schools)
• Telephone survey (1,047 parents with children in
  PPS magnet, PPS non-magnet, and non-PPS schools)
• Analysis of administrative and student data from
  PPS Stars Database
• Rubric developed and completed for each magnet
  program
• Collaboration with PPS Central Office staff on
  all aspects of audit

4
6
A comprehensive review of magnet program
literature identified a number of best practices
that were relevant to the magnet audit.

• Reputation, facilities, and safety are key to
  attracting racially and ethnically diverse mix of
  families
• Cooperative learning strategies are essential to
  impacting outcomes of all students, including low
  income and students of color
• Flexibility and autonomy in staffing, as well as
  specialized staff and professional development
  activities, are key
• Multi-pronged marketing strategy and linkages
  with business and community groups are successful
  strategies to engage community
• External and internal evaluation strategies,
  including peer reviews, are important to maintain
  accountability and quality in magnet programming

5
7
Magnet Audit Preliminary Findings and
Recommendations
6
8
There is no consensus within PPS around the
mission or vision for the magnet program.

• Audit Findings
• The PPS magnet program lacks leadership and
  vision at the district level.
• There are antiquated and incorrect perceptions
  about the goals of magnet schools and how to run
  them according to best practices.
• Recommendation
• Initiate a strategic planning process with key
  stakeholder groups to discuss and reach consensus
  about the mission and vision for the magnet
  school initiative.

7
9
There is a lack of district-level oversight for
the magnet program.

• Audit Findings
• In the absence of a well-staffed Magnet Office,
  the responsibility for designing and implementing
  magnet programs has been relegated to school
  principals.
• Principals have varying levels of experience or
  commitment to effectively implement programs as
  designed or intended.
• Recommendation
• Establish a Magnet Office directed by a
  senior-level educator with responsibility for
  program development and operations system-wide.

8
10
There are pockets of excellence in the PPS
magnet program that are exemplars for the
district.

• Audit Findings
• There are some magnet programs that exhibited
  best practices on elements contained in the
  scoring rubric.
• These programs should be recognized and serve as
  models for other programs.
• Recommendation
• Provide opportunities for instructional leaders
  from successful magnet schools/programs to share
  experience and expertise with the larger PPS
  community.

9
11
PPS magnet offerings are not adequately
addressing economic demand or realities.

• Audit Finding
• There are gaps in magnet programming in the areas
  of health care, biotechnology, financial
  services, engineering, technology, computer
  science, and critical languages.
• Recommendation
• Re-align magnet program options to meet needs of
  local, regional, and national economy.

10
12
PPS magnet offerings are not fully meeting the
needs or interests of the PPS community.

• Audit Findings
• Enrollment in magnet programs decreased by 12
  over past 5 years, which mirrors the decrease in
  the total PPS student population (Figure 1).
• Magnet program applications decreased from 4,284
  in 2003/04 to 2,555 in 2007/08from12.8 to 8.7
  of total PPS student population
• Quality of academic instruction, perceived
  reputation, safety, and the magnet theme are most
  important factors for parents (including PPS and
  non-PPS) and students in choosing a school.
• Parents identified technology, computer science,
  world languages, science, and arts as most
  important magnet themes.
• Students selected arts, college preparation,
  business, technology, math and science, and
  health as desirable themes.
• Recommendation
• Re-align magnet program options to meet interests
  of PPS community.

11
13
Enrollment in magnet programs has decreased by
12 over past five years same as district.
13
14
Applications to magnet programs have decreased by
40 over past five years.
12
15
Most magnet students remain in magnet programs as
they transition to the next school level.

• Audit Findings
• 86 of students in magnet programs in elementary
  school continue with a magnet program in middle
  school.
• 57 of students continue in magnet programs from
  middle to high school.
• At least half of students in vertical magnet
  tracks (K-12 or 6-12 themes) continued with
  magnet theme as they progressed in school level
  from 2006-07 to 2007-08.
• International Studies theme 63.6 from
  elementary to middle school 45.9 from middle to
  high school
• Traditional Academies 52.9 from elementary to
  middle 62.9 from middle to high school
• CAPA 71.3 of Rogers CAPA students went on to
  CAPA HS

14
16
PPS does not currently use effective outreach or
marketing strategies to inform the community
about the magnet program.

• Audit Findings
• Magnet Registration Guide is hard to navigate.
• The system of magnet applications and admissions
  is unnecessarily complex.
• The current admissions process unfairly
  advantages families of higher socio-economic
  status because those families have greater access
  to information (such as through social networks).
• Recommendation
• Revamp and expand the outreach materials and
  marketing strategies used to inform PPS families
  of the magnet choice options.

15
17
PPS use of race in magnet student selection
process is out of compliance with the law.

• Audit Findings
• The magnet application and assignment process is
  perceived by PPS community as inequitable.
• PPS magnet lottery uses a racially-balanced
  equation (50/50 rule) for student selection.
• Student assignment plan is not race neutral,
  and therefore not in compliance with Supreme
  Court ruling and not eligible for federal
  funding.
• Recommendation
• Revise PPS student assignment process so that it
  fully promotes equal access among all sectors of
  PPS community and is to fully aligned with
  Supreme Court decision regarding race neutral
  plans, thereby allowing the district to be
  eligible for Federal and other external funding.

16
18
Some PPS magnet programs have been able to
effectively implement rigorous magnet curriculum
and innovative teaching strategies within the
managed curriculum, yet most have not.

• Audit Findings
• Among magnet program leadership and staff, there
  is perceived incompatibility of PPS managed
  curriculum and magnet instructional programming.
• There is a general lack of rigorous,
  standards-based curricula for magnet themes.
• Yet some magnet programs in PPS are effectively
  integrating magnet programming with managed
  curriculum and are providing innovative and
  engaging teaching and learning strategies.
• Recommendation
• Provide support to the magnet programs for
  planning curriculum for the purpose of
  integrating magnet instructional programs and
  practices with district instructional initiatives
  and priorities.

17
19
PPS does not provide additional funding to
support magnet programs.

• Audit Findings
• Most PPS magnet programs are supported primarily
  through site-based budgets.
• Yet some entrepreneurial principals have
  leveraged additional resources for magnet
  programs.
• After moving to a race-neutral admissions policy,
  other school districts have secured
  district-level funding through Voluntary Public
  School Choice and Magnet Schools Assistance
  Program grants.
• Recommendation
• Apply for additional Voluntary Public School
  Choice and Magnet Schools Assistance Program
  grants and assist the principals of magnet
  programs in leveraging internal and external
  resources to support magnet program
  implementation.

18
20
Most staff in magnet programs are highly
qualified, but there is a dearth of staff with
specialized qualifications.

• Audit Findings
• There is a high percentage of qualified teachers
  in all PPS magnet programs.
• Programs have difficulty attracting and retaining
  teachers with specialized qualifications (e.g.,
  foreign languages, Montessori).
• There are limited resources and opportunities
  available for magnet-related professional
  development.
• Recommendation
• Support the principals of magnet programs in
  recruiting, retaining, and developing qualified
  staff with targeted skills and a commitment to
  the magnet program vision.
• Provide magnet-related professional development.

19
21
Monitoring or evaluation of magnet programs has
not been conducted by the district.

• Audit Findings
• The district does not conduct ongoing monitoring
  or evaluation of magnet programs.
• The districts data system does not include data
  necessary for complete picture of implementation,
  enrollment, or applications.
• Recommendations
• Using the rubric created for the audit as a
  guide, engage the magnet programs in an annual
  process of self-evaluation to identify areas of
  needed improvement.
• Enhance the districts data system so that
  accurate and reliable information can be captured
  and used for program monitoring.

20
22
PPS magnet programs receive high satisfaction
ratings from students and staff.

• Audit Findings
• Magnet students expressed high satisfaction with
  magnet programs, including quality of
  instruction, magnet themes, and how teachers
  teach magnet classes.
• Magnet staff identified value added areas
• Educational experiences
• Specialized classes
• College/career preparation
• Skills and values

21
23
Results suggest a positive relationship between
participation in magnet programs and student
outcomes in both reading and math, for all
students, as well as African-American and
low-income students.

• Audit Findings
• Magnet students scored higher on PSSA than
  non-magnet students when pre-existing differences
  were controlled for (Figure 3)
• Magnet students had higher average daily
  attendance, grade promotion rates, and graduation
  rates
• Among African-American and low income students,
  magnet students demonstrated higher achievement
  on PSSA, grade promotion, and attendance than
  non-magnet students
• Smaller gaps between African-American and White
  students were observed among magnet than
  non-magnet students (Figures 5 and 6).
• Recommendation
• Use results of the audit to identify strengths
  and areas for improvement within magnet programs
  to sustain positive impact of magnet programs on
  student outcomes.

22
24
Magnet students scored higher on PSSA than
non-magnet students when pre-existing differences
were controlled for.
Significant Differences -- Magnet G2
Non-Magnet
Significant Differences -- Magnet G2
Non-Magnet -- Magnet G1 Non-Magnet
Significant Differences -- Magnet Non-Magnet
Mean scores were adjusted to take into account
prior differences in academic achievement. For
grade 5 students, their 3rd-grade PSSA scale
scores were used as a covariate. For grade 8
students, their 5th-grade PSSA scale scores were
used as a covariate. For grade 11 students, their
8th-grade PSSA scale scores were used as a
covariate. Magnet G1 includes magnet students
who were not in a magnet program at the previous
school level (i.e., grade 5 for eighth-grade
students and grade 8 for eleventh-grade
students). Magnet G2 includes magnet students who
were in a magnet program at the previous school
level (i.e., grade 5 for eighth-grade students
and grade 8 for eleventh-grade students)
23
25
Among African-American students, students in
magnet programs demonstrated higher achievement
on PSSA than non-magnet students, after
accounting for prior differences.
An asterisk denotes a statistically significant
difference at the .05 level based on an analysis
of covariance. Mean scores were adjusted to
take into account prior differences in academic
achievement. For grade 5 students, their
3rd-grade PSSA scale scores were used as a
covariate. For grade 8 students, their 5th-grade
PSSA scale scores were used as a covariate. For
grade 11 students, their 8th-grade PSSA scale
scores were used as a covariate.
24
26
Smaller gaps between African-American and White
students were observed among magnet than
non-magnet students.
An asterisk denotes a statistically significant
difference at the .05 level between White and
African-American students.
25
27
27