Getting Ready to Measure Outcomes of Ford PAS

1
Getting Ready to Measure Outcomes of Ford PAS

• Presenter
• Melanie Hwalek
• SPEC Associates

2
Phase 1Evaluation Advisory Group

• Warren Dubitsky, Advanced Technology Academy
• Melanie Hwalek, SPEC Associates
• Ilene Kantrov, Education Development Center, Inc.
• Dan Maggio, University of Detroit Mercy
• Janet Padilla, Ford Motor Company Fund
• Paul Poledink, Ford Motor Company Fund
• Angela Polk, Ford Motor Company Fund

3
Phase 2 Evaluation Advisory Group

• Cindy Anderson, Advanced Technology Academy
• Jim Hinga, University of Michigan Dearborn
• Melanie Hwalek, SPEC Associates
• Dan Maggio, University of Detroit Mercy
• Cathy Padesky, Shrine High School
• John Padesky, Shrine High School
• Paul Poledink, Ford Motor Company Fund
• Velma Smith, Southfield Regional Academy
• Kate Soderstrom, West Bloomfield High School
• Pamela A Rhoades-Todd, University of Detroit
  Mercy

4
Phase 1Activities

• Develop Logic Model
• Apply Logic Model to Advanced Technology Academy
  as a best practice application of Ford PAS
  (case study)

5
Purpose of Phase 2

• Get ready to obtain, analyze and present
  defensible data about Ford PAS implementation and
  outcomes in Detroit area schools
• Design a system that schools can use on an
  ongoing basis to continually measure and report
  on implementation and outcomes

6
Outcomes Solidly Supported by ATA Case Study

• Ford PAS helps students
• Prepare for and get admitted to college
• Learn about careers related to business and
  engineering
• Research, think, and problem solve in real world
  settings
• Manage time
• Work as team members
• Communicate

7
Outcomes Somewhat Supported by ATA Case Study

• Ford PAS helps students
• Increase self confidence
• Improve leadership skills
• Increase motivation to succeed
• Have a more global perspective
• Become lifelong learners

8
Implementation challenges identified through case
study

• Teachers do not necessarily know curriculum
  content
• Motivational teachers make challenging curriculum
  feel doable to students
• Technology is a major cost to Ford PAS
  implementation
• Some students and teachers are challenged in
  using Microsoft Office applications
• Completing all modules within suggested timeframe
  is not always realistic
• Parents may not know how to help students achieve
  in Ford PAS or how to pursue higher education

9
Questions to Be Answered in Phase 2

• What components of the Ford PAS logic model are
  important to measure?
• What methods and measures are most appropriate?
• What methods and tools already exist that are
  feasible to use?
• What processes must be undertaken in order to
  conduct evaluation within participating schools?
• What kinds of reports can be produced for CQI to
  measure outcomes and to add evidence to knowledge
  base about Ford PAS and similar programs?

10
Phase 2 Steps

• Convene an Evaluation Advisory Group (EAG)
• Validate/revise Ford PAS logic model
• Select components of logic model to measure
• Identify relevant methods and tools
• Draft evaluation plan and procedures
• Vet evaluation plan and procedures with
  participating schools
• Document the process

11

• What components of Ford PAS logic model are
  important to measure?

• Start with Modules 1, 2, and 3
• Maximize number of students
• Pilot measurement procedures on a small scale
• Get pre-post data within one semesters time

12
(No Transcript)
13
Logic Model Ford Partnership for Advanced
Studies (Ford PAS) Part 2 Outcomes and
Influencing Factors (Rev 5-24-07)
Outcomes that are in red, bold, and italicized
are those selected by EAG to be measured for
Modules 1, 2, and 3. Other outcomes in italics
are expected to be measured in the future.
FACTORS LIKELY TO INFLUENCE STUDENT
OUTCOMES Fidelity of implementation to the model
(amount of curriculum, inclusion of all
components) Student socioeconomic status School
effort related to ongoing professional
development and support for Ford PAS Extent of
integration of Ford PAS throughout the
school Size of school Teacher use of
inquiry-based pedagogy
14

• What methods and measures are most appropriate?

Observations (of presentations and
teamwork) Ratings of student work (of writing
assignments and PowerPoint presentations) Student
self-perceptions (retrospective pre-post test
survey and guided group discussion)
15

• What methods and tools already exist that are
  feasible to use?

Ford PAS is Getting Ready to Measure
Outcomes Test and Measures Search Results as of
6-6-07 Group Discussion Guide Re Outcomes of
Ford PAS Retrospective Pre-Post Test (RPT) of
Ford PAS Outcomes
16

• What processes must be undertaken in order to
  conduct evaluation within participating schools?

Agreement to use same module activities for
observations and ratings Module 1 Activity 1
and Activity 5 Module 2 Activity 2 and
Activity 6 Module 3 Activity 2 and Activity
6 Creation of observational and rating
tools Training raters to criterion Abide by
school system research policies and human rights
protections
17

• What kinds of reports can be produced for CQI to
  measure outcomes and to add evidence to knowledge
  base about Ford PAS and similar programs?

18
Summary

• Process was user-driven
• Start small and build big
• Minimize confusion of reading skills with
  outcomes achieved (i.e., limit use of written
  tests)
• Embed assessments
• Use of trained raters
• Next step test tools and procedures

19
Reflections
20
Contact Information

• SPEC Associates
• 615 Griswold, Suite 1505
• Detroit, Michigan 48226-3992
• [phone]
• www.specassociates.org
• Melanie Hwalek (ext. 202)
• mhwalek_at_specassociates.org