Evaluation Study of Ford PAS

1
Evaluation Study of Ford PAS

• What do we want to know?

2
Evaluator

• Ford PAS contracted with
• MPR Associates, Inc. (Berkeley, CA)
• Education research and evaluation organization
• Gary Hoachlander, Chairman of the Board for MPR
  and President of affiliated organization
• ConnectEd (CA Center for College and Career)
• Background in Career Technical Education
• Project housed in two programs Evaluation and
  Preparation for College Career

3

4

5
(No Transcript)
6
Purpose of the Study

• Learn as much as possible about variations in
  implementation of Ford PAS curriculum and program
  across the country
• Collect evidence on the effects of the program on
  teacher practice, school culture
• Conduct rigorous study on the effects of the
  program on student academic achievement,
  attitudes, and 21st century skills

7
Research Questions

• Context
• How do organizations make decisions to use the
  Ford PAS program? What factors influence their
  decision?
• How is the model of implementation determined?
  How do outcomes vary for different models?
• How do agency (school, district, or other
  organization) factors influence implementation?
• How do teacher factors (demographics, knowledge,
  motivation, pedagogy) affect program outcomes?

8
Research Questions

• Implementation
• What range of models do communities use in
  implementing the Ford PAS program?
• Under what conditions does the program seem to be
  most effective?
• How do users integrate the program with existing
  curricular programs?
• What modifications/adaptations/innovations do
  users make to accommodate their particular
  context and needs?

9
Research Questions

• Implementation
• How are CLEs arranged and implemented, and can
  effects of these activities be related to
  particular student outcomes?
• How are classroom practices affected by use of
  the program?
• What barriers do organizations encounter as they
  implement the program?
• How prepared are teachers to implement the Ford
  PAS curriculum?

10
Research Questions

• Program Quality
• What do users and participants think about the
  Ford PAS program?
• How does the program align with national and/or
  state standards? Is it implemented in a way that
  is academically rigorous?
• How user-friendly is the program?
• To what degree does the enacted program reflect
  fidelity to the design and intent of the program?

11
Research Questions

• Support
• How effective are current methods of training and
  dissemination?
• How are BEACs (and other partnerships)
  instituted, and how do they function? What impact
  do they have on implementation of the program?
• What responsibilities do program coordinators
  have, and which strategies seem to affect program
  implementation?
• In what ways do agencies (schools, district,
  organization) support implementation of the
  program?

12
Research Questions

• Outcomes
• What impact does the program have on teachers in
  terms of knowledge, skills, and pedagogical
  practices?
• What impact does the program have on students in
  terms of a) cognitive performance and
  academic/business knowledge and skills b)
  affective student factors such as attitudes,
  behavior, future career and vocational
  aspirations c) future educational pursuits and
  success in those settings?

13
Research Questions

• Outcomes (cont.)
• In what ways has the program helped students
  connect classroom learning with real-world
  experiences? To what extent are students able to
  apply their academic/business knowledge and
  skills to real-world settings?
• Do program outcomes vary across different types
  of students and models of implementation?

14
Research Questions

• Outcomes (cont.)
• Does the program enhance teacher and student
  outcomes as compared to students/teachers who
  have not been exposed to the program?

15
Study Design - 3 Phases

• 1. Design Phase (Feb 2008 - Aug 2008)
• Organization of information about sites
• Implementation Survey
• gtSite Coordinators
• gtTeachers

16
Study Design

• 2. Pilot and Efficacy Study Phase
• (summer 2008-spring 2009)
• Development or identification of data collection
  instruments
• Identification of sites to participate in pilot
  and small efficacy study
• Piloting of instruments
• Implementation of small efficacy study in sample
  of sites

17
Study Design

• 3. Larger Scale Efficacy Study (July 2009 - June
  2012)
• Rigorous experimental or quasi-experimental
  design (likely a matched comparison study)
• Samples of sites that may be studied as separate
  cohorts
• Longitudinal analyses (possibility of including
  long-term outcomes such as postsecondary
  enrollment or progress, career placement

18
Study Design

• 4. Development of internal and local
• evaluation capacity
• Development of common instruments
• Guidelines for conducting evaluation
• Data analysis tools
• Reporting formats

19
Next Steps

• Complete analysis of implementation
• Development of instruments
• Identification of sites to participate in pilot
  and small efficacy study
• Visits to pilot sites

20
Collaborative Research

• Importance of participation/collaboration
• Have you completed your survey?
• Common goals
• Shared interest in improving education for all
  students

21
Its all about the data

• DATA
• DATA
• DATA
• DATA
• DATA
• DATA
• DATA
• DATA
• DATA
• DATA DATA DATA DATA DATA DATA!

22
To Contact Us

• MPR Associates, Inc. [phone]
• Bfarr_at_att.netBeverly Farr
• Rosio Pedroso
• Laurel Sipes