Measuring Program Success: The Importance of Program Evaluation

1
Measuring Program Success The Importance of
Program Evaluation

• OCAN 4th Annual Conference
• September 9, 2003
• Ramona S. Thomas, KnowledgeWorks Foundation
• David B. Swedlow, Ohio College Access Network

2
Objectives

• Evaluation Overview
• Working Definition
• Types of Evaluation
• Evaluation in Practice in College Access
• Why do evaluation?
• Evaluation Framework
• Principles and Suggestions for Evaluation
• OCAN Evaluation Overview
• Additional Resources

3
Evaluation Overview - Definition

• Program evaluation is not
• A method of quality assurance for published or
  public documents
• An account of expenditures or use of resources
• A status report on the development or
  implementation of a particular program

4
Evaluation Overview - Definition

• Program evaluation is the process through which
  research techniques are used to assess design,
  implementation, success, and impact
• The primary objective is to assess and determine
  whether goals, objectives, and desired outcomes
  were reached and, to the extent possible, explain
  why and how

5
Evaluation Overview - Types

• Based on length (or when conducted)
• Formative evaluations
• assess programs (or program activities) during
  their early stages or while still in progress
• focus on process
• Summative evaluations
• assess the effectiveness of a program at the end
  of the program activities (or over an extended
  time frame)
• focus on outcomes

6
Evaluation in PracticeWhy do evaluation?

• Program Evaluation
• Provides an opportunity to learn more about what
  you do (processes and activities) and the effects
  of your work (outcomes)
• Can help improve program goals, objectives, and
  service delivery
• Meets private and public accountability standards
  and requirements

7
Evaluation in Practice - Framework

• Common approach to program evaluation
• Identify problem
• Develop program
• Design/conduct evaluation
• Generate evaluation report
• Disseminate evaluation report to funders (or
  select group)

8
Evaluation in Practice - Framework
9
Evaluation in Practice - Framework
10
Evaluation in PracticePrinciples and Suggestions

• Align with program goals and objectives
• Clarify the purpose, goals, objectives, and
  desired results of your program
• Identify short-term, intermediate, and long-term
  outcomes
• Logic models are useful planning tools
• Objectives should encompass activities that
  reflect the experiences of those you serve
• Ensure that results focus on outcomes of your
  activities

11
Evaluation in PracticePrinciples and Suggestions

• Have clearly defined questions, outcomes,
  expectations, and uses
• Be clear about what you really want to know about
  your program and its outcomes
• Try to answer what works, how it works, and why
  it works
• Identify clear and concise evaluation objectives
  (they should match your programs services and/or
  activities)
• Carefully prioritize multiple purposes

12
Evaluation in PracticePrinciples and Suggestions

• Follow multiple approaches
• Collect and analyze data from multiple sources
  and perspectives
• Use various data collection methods (e.g.,
  surveys, interviews, focus groups, documents,
  other available data)
• Establish a baseline and collect baseline data
• Measure progress toward long-term goals
• Pay attention to rigor, flexibility, and
  cost-effectiveness

13
Evaluation in PracticePrinciples and Suggestions

• Include time for reflection and sharing
• Are you surprised by the findings or are they
  what you expected? In what ways?
• What new information about your program and/or
  its outcomes has surfaced?
• How can you use evaluation to improve your
  program and service delivery?
• Are there others who could benefit from what you
  have learned? What is the best way to share it
  with them?

14
OCAN Evaluation Overview

• Goals of the OCAN Evaluation
• Examine the effectiveness and influence of OCAN
  as a statewide, college access organization
• Assess sample of OCAN member programs
• Examine the partnership among KnowledgeWorks
  Foundation, OCAN, the Ohio Board of Regents
  (OBR), and others

15
OCAN Evaluation Overview

• Component on OCAN Member Programs
• Contextual and demographic information
• Activities, assistance received from OCAN,
  perceptions about the quality and value of OCAN
• Program administration and effectiveness of
  services provided to participants
• Member programs evaluation/assessment activities
• Relationship with OCAN

16
Resources

• Evaluation Toolkit. Project STAR for Corporation
  for National Service (http//www.projectstar.org/s
  tar/Library/ toolkit.html)
• Evaluation A systemic approach. P. Rossi H.
  Freeman. (1993). Sage Publications Newbury Park,
  CA.
• Evaluation Handbook. W.K. Kellogg Foundation
  (http//www.wkkf.org/Pubs/ Tools/Evaluation/Pub770
  .pdf)

17
Objectives

• Evaluation Overview
• Definition
• Types of Evaluation
• Evaluation in Practice in College Access
• Why do evaluation?
• Evaluation Framework
• Principles and Suggestions for Evaluation
• OCAN Evaluation Overview
• Additional Resources

18
Data for Evaluation Objectives

• How do we prepare for evaluation?
• What goals do we wish to achieve?
• What are some outcomes for college access?
• What are some outcomes for college success?
• How can the web-enabled student tracking
  database support evaluation?

19
How do we prepare for evaluation?

• What are our program goals objectives?
• Are student needs tied to the objectives?
• Are activities designed to meet objectives?
• Do outcomes relate to the activities?
• Will the achievement of the outcomes advance the
  goals of the program?

20
What are some goals of our programs?

• To increase the college-going rate of students
• To increase the number of community college
  students that transition to 4-yr colleges
• To increase the number of low-income students
  that can afford the rising costs of tuition
  fees
• To increase retention and graduation rates of
  underserved, low-income and first-generation
  college students
• To increase the number of adults in the
  population with bachelors degrees

21
What are some outcomes for access?

• Attending a college fair or visit
• Meeting with advisor on college applications
• Submitting a completed FAFSA on time
• Performing an electronic scholarship search
• Taking the SAT or ACT
• Shadowing with a mentor at a worksite
• Applying for a last-dollar scholarship
• Completing requirements for high school
  graduation and college admission

22
What are some outcomes for success?

• Admitted to postsecondary education
• Received financial aid package from college
• Awarded last-dollar scholarship
• Enrolled in postsecondary institution
• Made it! first term, successfully
• Retained! completed first year successfully
• Submitted transcript for scholarship renewal
• Selected major
• Graduated! - 2-yr or 4-yr degree

23
What can we collect on the student tracking
system?

• Demographic data
• Parent/guardian/contact information
• Financial data college cost data
• Secondary school academic progress
• Participation in college access activities
• College entrance exam scores (SAT/ACT)
• Scholarships and grants awarded
• Postsecondary education progress (enrollment,
  retention, grade point averages, majors,
  transfers, graduation)

24
What do we need to do to prepare to implement the
student tracking system?

• Gain commitment from board and staff
• Communicate with school districts
• Secure confidentiality signoffs
• Collect social security numbers
• Create individual accounts for advisors, staff
• Identify participation activities and events
• Work with OCAN to establish baselines