6th Annual Summit on Vocational Rehabilitation Program Evaluation

1
Estimating Return on Investment for State
Vocational Rehabilitation Programs
Dr. David Dean, University of Richmond Dr.
Kirsten Rowe, Va. Dept. for Aging and
Rehabilitative Services

• 6th Annual Summit on Vocational Rehabilitation
  Program Evaluation Quality Assurance
• Providence, RI September 16, 2013

Dr. Dean died on August 11, 2013, after
confronting a very difficult year of illness with
characteristic humor, courage, and strength.
2
Project Overview

• NIDRR-funded 3-year grant to University of
  Richmond
• VR agency partners include
• Virginia General (DARS)
• Virginia Blind (DBVI)
• Maryland Combined (DORS)
• Oklahoma Combined (DRS)
• Purpose is to develop state of the science ROI
  estimates using readily available data

3
Key Features of Our Approach (1 of 2)

• Link readily-available longitudinal data from
  multiple systems to examine impacts
• Focus on individuals, not cases
• Use applicant cohorts, not closure cohorts
• Include everyone who applies for VR
• Start examining program impact with the first VR
  application

4
Key Features of Our Approach (2 of 2)

• Crack the black box of VR services
• Control for selection bias
• Develop individual-specific Rate of Return
  (ROR) estimates

5
Use Longitudinal Data on Employment, VR Services,
and DI/SSI Receipt

• Earnings Employment data from state
  Unemployment Insurance program records
• 3 years prior through 5 or 10 years post VR
  application date using quarterly data for all VR
  applicants in SFY 2000/2007
• VR Service Provision
• Longitudinal VR service provision (up to 10
  years) to account for multiple cases over time
  account for both purchased services and in-house
  costs
• DI/SSI data from Social Security Administration
• 3 years prior through 5 or 10 years post using
  monthly receipt dollar amounts

6
Study Individuals, Use VR Applicant Cohorts,
Evaluate Impact of Initial VR Case

• VR is no longer a one and done program Many
  individuals have multiple VR cases
• Closure cohorts enter the VR program over a
  number of years, spanning VR program and economic
  climate changes
• We separate multiple VR episodes into
• A base case the first application occurring
  in a given SFY
• All prior VR applications
• All subsequent VR episodes occurring within ten
  years of the base case
• We evaluate all applicants whose initial base
  case was in a given fiscal year (SFY 2000 or
  2007)

7
Account for Variation in VR Consumers and Types
of Services Provided

• VR consumers by type of impairment
• We estimate separate impacts by types of
  impairment (mental illness, cognitive
  impairments, physical impairments, learning
  disabilities)
• VR services
• We allow for different labor market effects of
  seven categories of VR services (DTERMPS
  diagnosis and evaluation, training, education,
  restoration, maintenance, placement and supported
  employment)
• We can calculate ROR by disability type or VR
  service category as well as agency-wide

8
DTERMPS Across All Agencies 25,765 Base Cases
Total D T E R M P S
Receiving 68 38 15 9 24 29 10 14
Avg. cost (if any) 3,297 491 2,161 3,657 1,786 1,179 1,404 3,307
D Diagnostic Evaluation T Training E
Education R Restorative
M Maintenance P Placement S Supported
Employment
9
DTERMPS by Agency
Agency D T E R M P S
1 Receiving 31 12 5 19 28 5 18
1 Avg. Cost 512 1,646 2,540 1,067 900 1,735 3,768
2 Receiving 34 24 12 37 53 14 --
2 Avg. Cost 435 4,147 7,480 2,321 3,887 4,237 --
3 Receiving 57 11 11 18 24 18 15
3 Avg. Cost 543 2,481 2,597 2,190 1,097 1,130 1,886
4 Rcvng 23 22 12 36 33 9 7
4 Avg. Cost 282 2,221 5,382 2,065 1,338 1,565 5,544
10
Use a State-of-the-Science Labor Economics Model
to Identify Employment Impacts

• We formalize and estimate a model of labor market
  outcomes (likelihood of employment and earnings
  increases) resulting from the choice of VR
  service mix
• Features of our model control for selection
  bias (unobservable differences between those who
  receive services and those who do not)
• "Instrumental variables" are variables correlated
  with service choice but not with unobservable
  influences on labor market outcomes
• Pre-program labor market outcomes aid in
  controlling for differences between those who do
  and do not receive VR-paid services
• Statistical model controls for interrelationships
  between service choices and labor market outcomes
  and aids in the interpretation of results

11
Measuring Rate of Return versus Return on
Investment

• ROR ROI both use net earnings impacts and cost
  of service provision to calculate a measure of VR
  service efficacy
• ROI requires the arbitrary selection of an
  interest rate, the choice of which becomes more
  important the longer the earnings time horizon
• ROR for VR can be readily compared to rates of
  return such as the 10 annual ROR for long-term
  U.S. stock market performance

12
Some Preliminary Estimates of VRs Impact

• Our estimates differ dramatically across
  impairment groups
• For people with mental illness
• Median annual rate of return is 17.5
• 88.5 have positive rates of return
• 10 exceed a 50.7 annual ROR
• For people with cognitive impairments
• Median annual rate of return is 34.5.
• 78.7 have positive rates of return
• 20 exceed a 101.9 annual ROR

Note These estimates are for SFY 2000 cohort
from Virginia
13
Some Preliminary Estimates of VRs Impact

• Our estimates also differ by type of service
  provided
• For people with mental illness
• Most effective Training (includes supported
  employment)
• 7,200 average present value for 10 years of
  earnings
• Education
• 1,700 average present value for 10 years of
  earnings
• For people with cognitive impairments
• Most effective Education
• 36,000 average present value for 10 years of
  earnings
• Training (includes supported employment)
• 10,000 average present value for 10 years of
  earnings

Note These estimates are for SFY 2000 cohort
from Virginia
14
Next Steps

• Develop population- and agency-specific ROR
  estimates using SFY 2007 cohorts
• Develop three-state estimate for individuals who
  are blind or vision impaired
• Work with all participating agencies to
  disseminate and use results

15
Contact Information

• Kirsten L. Rowe, Ph.D.
• VR-ROI Project Coordinator
• Va. Dept. for Aging and Rehabilitative Services
• 8004 Franklin Farms Dr.
• Richmond, VA 23229
• 804-640-0435
• Kirsten.Rowe_at_dars.virginia.gov

16
Acknowledgments

• This project is funded by Field Initiated Project
  grant H133G100169 from the National Institute on
  Disability and Rehabilitation Research to the
  University of Richmond.
• Dr. Rowe wishes to acknowledge the invaluable
  contributions of her friend and colleague Dr.
  Dean to both this presentation and the study of
  return on investment for vocational
  rehabilitation.