Using Data to Measure Outcomes

1
Using Data to Measure Outcomes

• Student Tracking and Using Data Effectively
• Geoff Zimmerman
• 2004 NTPN Minneapolis, MN

2
Impact?
Basic Question

• What impact are you having on the students and
  schools that you are serving?

3
Goals for Today

• Big Picture Impact/Outcomes
• Nuts Bolts Data Collection
• Tech Prep in Ohio Measuring Success
• Demonstration of Database Application
• The Basics Designing Your Own

4
(No Transcript)
5
Are you Data-Driven?

• In business and industry, data are king.
  Information about customers, inventory, sales,
  rates of return and employee turnover is crucial.
  Good data determines success or failure.

Educational Accountability
6
Why Collect Data?

• Measure progress
• Make program decisions
• Perkins driven shows accountability
• Provides hard data to state executive and
  legislators when making funding decisions

7
No Child Left Behind
Four Reform Principles
Accountability Guaranteeing Results Flexibility
Local Control for Local Challenges Research-Based
Reforms Proven Methods with Proven
Results Parental Options Choices for Parents,
Hope for Kids
8
Perkins III

• (1) Definition
• Section 202(a)(3) of Perkins III states that a
  "tech-prep program" means a program of study
  that
• combines at a minimum 2 years of secondary
  education (as determined under State law) with a
  minimum of 2 years of postsecondary education in
  a nonduplicative, sequential course of study
• integrates academic, and vocational and
  technical, instruction, and utilize work-based
  and worksite learning where appropriate and
  available
• provides technical preparation in a career field
  such as engineering technology, applied science,
  a mechanical, industrial, or practical art or
  trade, agriculture, health occupations, business,
  or applied economics
• builds student competence in mathematics,
  science, reading, writing, communications,
  economics, and workplace skills through applied,
  contextual academics, and integrated instruction,
  in a coherent sequence of courses
• leads to an associate or a baccalaureate degree
  or a postsecondary certificate in a specific
  career field and
• leads to placement in appropriate employment or
  to further education.
• An allowable tech prep program must meet the
  terms of this definition.

9

• Are your programs making a difference?
• Beginning Questions
• How many students are enrolled in a Tech Prep
  program?
• What is the college transition rate for Tech Prep
  Students? How many are persisting in college?
• What are the demographic characteristics of Tech
  Prep students?
• To what extent are Tech Prep students prepared
  for college?

10
Measuring Outcomes
Describe your program what does it do and what
does it accomplish?

• Why do you exist?
• What conditions need to change for this target
  group and why?

Source United Way of Greater St. Louis Guide for
Measuring and Reporting Outcomes
11
Measuring Outcomes
(cont.)

• What is your program designed to do?
• Who is the target of the change? Who benefits
  from your services?
• What information do you need to compile about
  participants to ensure delivery?
• What measurable changes will occur?

Source United Way of Greater St. Louis Guide for
Measuring and Reporting Outcomes
12
Describing what you do

• Inputs resources used to implement the program.
• Activities actions that define what is done,
  when, where, how, how often, and for whom.
• Outputs the direct products of the program
  activities. Measure of the amount and volume of
  work performed to implement the program.

Source United Way of Greater St. Louis Guide for
Measuring and Reporting Outcomes
13
Describing your intended results
Outcomes the results the program aspires to
accomplish they specify immediate, intermediate,
and long-term changes among students served by
the program Indicators measurable data about
benefits for and changes in the target group that
is collected at the individual level to show an
outcome has been achieved Impact what the
program aspires to accomplish over a long period
of time
Source United Way of Greater St. Louis Guide for
Measuring and Reporting Outcomes
14
Outputs vs. Outcomes

• Output what you do
• Outcome what changes because of what you do
• -refining selections as a result of college fair
  or visitas opposed to x number of students
  attending the college fair
• -completing and submitting college applications
  following advising sessionas opposed to advising
  x number of students
• -submitting a FAFSA on timeas opposed to giving
  the FAFSA form to x number of students
• -persisting in college as opposed to enrolling
  in college

15
Logic Model
Inputs
Activities
Outputs
Outcomes
Impact
16
Logic Model (alternative)
Long-Term Impact
Outcomes
Outcomes
Outcomes
Activities
Activities
Activities
17
Uses of Outcome Information
A key element in effective use of outcome
information is to continually ask why differences
occur and then assess whether actions can be
taken to improve results. (p.15)
Hatry, H., Morley, E., Rossman, S., and Wholey,
J. (2003). How Federal Programs Use Outcome
Information Opportunities for Federal Managers.
IBM Endowment for the Business of Government.
Online http//www.businessofgovernment.org/pdfs/H
atryReport.pdf
18
In other words

• In other words outcome information
• Tells us what is happening
• So we can ask why it is happening
• And fix it if it needs to be fixed
• Celebrate it and tell everyone if it works

19
Uses of Outcome Information

• How are you using it?
• Triggering corrective action
• Identifying and encouraging best practices
• Motivating
• Planning and Budgeting
• Reporting to Key leaders

Hatry, et.al, How Federal Programs Use Outcome
Information Opportunities for Federal Managers
20
Quality is not just a chart, or a standard, or a
specificationits a state of mind, a
commitment, a responsibility, a spirit. Its a
way of doing, being, living. Don Galer 
21
Data Collection
22
Insanity the belief that one can get different
results by doing the same thing. -Albert Einstein
23
How to Collect?
Primary Data Sources

• Student enrollment form
• Student list report of file from schools
• Surveys students, parents, teachers, etc.
• Focus groups, interviews
• Program site review process

Secondary Data Sources
U.S. Census Bureau, State Departments of
Education, National Center for Education
Statistics
24
What to Collect

• Determine what data to collect
• Demographics
• Participation in activities
• College aspirations
• Academic preparation
• Persistence in college
• Remediation in college
• Feelings about the program
• Perceptions of education and work

25
Survey Results
Source Greater Cincinnati Tech Prep Consortium,
May 2004
26
Survey Results
Source Greater Cincinnati Tech Prep Consortium,
May 2004
27
Survey Results
Source Greater Cincinnati Tech Prep Consortium,
May 2004
28
Survey Results
Source Greater Cincinnati Tech Prep Consortium,
May 2004
29
Survey Results
Source Greater Cincinnati Tech Prep Consortium,
May 2004
30
Focus Group Results
Source Greater Cincinnati Tech Prep Consortium,
September 2003
31
Educational Attainment, Bethel, Ohio
QT-P20. Educational Attainment by Sex  2000Data
Set Census 2000 Summary File 3 (SF 3) - Sample
Data
http//factfinder.census.gov
32
(No Transcript)
33
Tech Prep in Ohio
34

• Ohio College Tech Prep combines college
  preparatory academics and advanced
  career-technical education into a seamless
  program from high school to college to
  high-skill, high-wage careers.

35
Innovative Local Delivery Models

• CTP programs on college campuses
• CTP programs in industry setting
• Industry mentorships
• Dual Enrollment and Transcripted college credit
  options
• College faculty team-teaching with secondary
  instructors

36
College TP goals and Benchmarks

• Ohio Tech Prep Goals (1998)
• Expand CTP enrollment to meet Ohios critical
  workforce and economic needs
• Provide a CTP system that reflects the
  requirements for success in college and in
  high-wage, high-skill technical careers.
• Maximize student options through the integration
  of high school, associate degree and
  baccalaureate degree pathways to achieve a
  seamless, non-duplicative system.
• Engage the active support and involvement of all
  College Tech Prep stakeholders.

• Performance Benchmarks
• Expand enrollment 15 of all 11th and 12th
  graders
• Enter college remediation free 10 higher rate
  than traditional college population
• Transition to college 66
• Under-represented groups 25

?
?
?
?
37
3. Building the Secondary Pipeline
Annual Enrollment
38
Student Tracking and Analysis though HEI
Ohio Board of Regents Higher Education
Information (HEI)Student Tracking System
39
How is this done?

• Ohio has a comprehensive postsecondary data
  collection system in place - Higher Education
  Information System (HEI)
• Available to all public two four year
  institutions.
• Limited access to private colleges

40
How HEI Works

• Tracks students by SSN
• Follows student mobility in higher education.
• Tracks by campus type
• State Community College Community Colleges
• Technical Colleges
• University Branch or Main Campuses
• Reports on remediation levels and persistence

41
HEI Enrollment Data Submissions
Tech Prep Consortium Tracking (TC) File Revised
May 1, 2002
File Description The Tech Prep Consortium
Tracking (TC) File contains one record for each
student enrolled in a Tech Prep
consortium. Submission Schedule The TC file can
be submitted anytime throughout the year.
Data Fields
42
HEI Enrollment Data Submissions
Tech Prep Consortium Tracking (TC) File
Data Fields (continued)
43
Link to HEI 16 digit code

• Consortium identifier
• Student identifier (ss)
• Program Code
• Career Center Code (IRN)
• Delivery Site Code (IRN)
• High School Code (IRN)
• Graduation Year

(CNCI999990153Y11T0110514820065930065932004N)
44
Internal Data Tracking System

• Student Enrollment Forms (paper filing)
• Microsoft Excel, Lotus
• Microsoft Access
• College databases for transition enrollment,
  remediation

45
What Data are Gathered

• Consulted and sought input from consortia
  directors on four major topics.
• Tech Prep Completion
• Remediation
• Transition and Persistence
• Underrepresented Populations

46
Improved Remediation Rates
Tech Prep vs. Non-Tech Prep Remediation
Rates (18-19 year olds)
21337 students total
1604 students total
Population are 18-19 year old freshman enrolled
at community / technical colleges or university
branch campuses.
47
Remediation
(cont.)
Two Year Campuses 2003-04
Four - Year Campuses 2003-04
48
Persistence
First-time, Full-time, degree-seeking
Undergraduate Students age 19 and under in Autumn
Term. Number and percent persisting to Autumn of
next year, by type (tech prep vs. non tech prep).
Persisting includes enrollment at an Ohio public
institution, or private institution if full-time
and receiving a Student Choice Grant, or
proprietary institution if receiving a Workforce
Development Grant. Data as of 8/27/2003.
First Time Full Time Degree Seeking
Undergraduates
First Time Full Time Degree Seeking
Undergraduates
49
New Data to be Collected . .

• Post-program Placement

Annualized Full-Time Salaries (in thousands)
Associate Degree Graduates Fourth Year Following
Graduation by Discipline
50
(No Transcript)
51
Demo

• Consortium office database for tracking students,
  programs, articulation agreements, and contacts

52

• Contact informationGeoff ZimmermanGreater
  Cincinnati Tech Prep ConsortiumUniversity of
  Cincinnati513-936-1682geoffrey.zimmerman_at_uc.edu