The National Evaluation of the PATH Program

1
The National Evaluation of the PATH Program

• Pamela J. Fischer, Ph.D.
• SAMHSAs Center for Mental Health Services
• Homeless Programs Branch

Presented at the 2010 PATH Grantee Meeting,
Washington, DC
2
Todays Presentation

• Findings from the 2005 National Evaluation
• of the PATH ProgramPamela Fischer, Ph.D.
• Design and upcoming activities of the current
  evaluationPaul Brounstein, Ph.D.
• Questions and discussion

3
Congressional Mandate for the National
Evaluation of the PATH Program

• Section 528 of the Public Health Service (PHS)
  Act requires the SAMHSA Administrator to evaluate
  the expenditures of PATH grants at least once
  every 3 years to ensure they are consistent with
  legislative requirements and to recommend changes
  to the program design or operations
• Evaluation design is determined by the Federal
  program staff and conducted through a
  competitively awarded contract

3
4
Findings from Previous Evaluations

• Outreach is emphasized by PATH providers and is
  carried out conscientiously and with quality
• PATH program uses funds appropriately and
  effectively
• Flexibility is widely praised
• High levels of job satisfaction
• PATH not necessarily recognized as a program as
  PATH funds represent only one of many small, but
  important funding streams received by providers

5
National Evaluation of the 2005 PATH Program
5
Conducted during 2006-9 by the MANILA Consulting
Group, Inc. Walter Leginski, Ph.D., and
Caroline Fernandez, Project Directors
6
Evaluation Questions

• Are services appropriate?
• Are services well-administered?
• Are outcome and process goals achieved?
• increases in the number of persons contacted,
  enrolled, and receiving mental health services
• clients placed in permanent housing

7
Focused Analyses

• PATH and rural homelessness
• Effect of local context on PATH performance
• Housing
• Poverty
• Other Federal funding

8
Methods

• 5 VPGs informed the evaluation approach
• emphasis on literal homelessness
• active State management
• specific guidance on reporting and definitions
• focus on exemplary practices
• active support of transition to mainstream
  services
• 2005 selected for analysis due to completeness of
  reports
• Data sources included administrative data and
  contextual information
• Design and interpretation guided by Technical
  Panel

9
Results

• Consistent with the findings of prior
  evaluations, PATH demonstrated that it continues
  to meet the three objectives of the evaluation

10
Appropriateness of Services

• Enrolled PATH clients match the eligibility
  profile and intent of the PATH legislation
  exceedingly well
• PATH providers offer a wide range of services
  from the PATH menu
• High proportions of enrolled clients participate
  in these services
• Engagement of clients into services and their
  transition to mainstream services are emphasized

11
Quality of Service Administration

• Program structure and approach
• Responsiveness of States and providers to
  information requested in applications
• Program monitoring
• States efforts toward adoption of Voluntary
  Performance Goals (VPGs)
• Variety of administrative approaches adopted by
  providers featuring collaborations that would
  expedite clients access to mainstream services
  and housing

12
Achievement of Identified Outcome and Process
Goals

• GPRA goal achievement was similar to past years
• Housing collaborations developed to assure access
  to affordable housing for PATH clients

13
PATH Providers in Rural Locations

• Received smaller awards
• Challenges to collaboration with providers in
  resource-poor environments
• Rural providers perform equally and in some cases
  better than providers in metropolitan and mixed
  (urban/rural) areas

14
Correlating PATH Performance with Poverty Rates,
Affordable Housing, and Other Targeted
Homelessness Funding Levels

• PATH provider performance was strong, even in
  locations where high poverty and lack of
  affordable housing were evident
• Additional sources of Federal homelessness
  funding in the same locale as PATH providers did
  not detract from PATH performance and supported
  the concept that multiple providers function as a
  system of homeless services, enhancing PATH
  performance

15
Summary

• PATH is a critical funding stream
• PATH leverages resources and systemic approaches
  directed to the most vulnerable subpopulations of
  homeless persons
• As populations and policies change in State and
  national environments, the triennial evaluations
  are ideal opportunities to assess whether
  adjustments are needed to sustain PATHs
  important contributions to the welfare of
  homeless persons with serious mental illnesses

16
For additional information contact

• Pamela J. Fischer, Ph.D.Homeless Programs
  BranchCenter for Mental Health
  ServicesSubstance Abuse and Mental Health
  Services Administration(240) 276-1901Pamela.Fis
  cher_at_samhsa.hhs.gov

17
The National Evaluation of the PATH Program

• Paul J. Brounstein, Ph.D.
• MANILA Consulting Group
• Project Director

Presented at the 2010 PATH Grantee Meeting,
Washington, DC
18
PATH

• Formula Grant to 50 states, D.C., and 5
  Territories
• To enable States and providers to better allocate
  resources and improve service availability,
  accessibility and fill gaps in existing service
  system to decrease homelessness for individuals
  and families affected by serious mental illness

19
Key Services

• Community-based outreach
• Screening and diagnostic services
• Habilitation and rehabilitation services
• Community mental health services
• Alcohol or drug treatment services (for people
  with mental illnesses and co-occurring substance
  abuse disorders)
• Case management services
• Supervisory services in residential settings
• Limited housing services and services to help
  clients use housing resources

20
Most Common Services

• 88 of grantees offer outreach services
• 82 of grantees offer case management services

21
Evaluation

• Congress requires triennial evaluation answering
  3 questions
• Are services using PATH monies appropriate
  (Stewart B. McKinney Homeless Assistance
  Amendments Act of 1990)?
• Are services well-administered?
• Have process and outcome goals been achieved?

22
Current Evaluation Plan goes further and assesses
the role of

• Grant Structure
• Organizational Structure
• Level of Collaboration
• Mix of Services
• Populations Served/Characteristics
• Role of Context on Outcomes
• Urbanicity, Poverty,
• Labor and Housing Markets

23
Evaluation Process

• Developed initial evaluation specifications in
  response to solicitation for proposals
• Assembled expert panel
• Multiple telephone conferences to review plan
• Plan finalized and approved by SAMHSA

24
Data Requirements

• Answers to the multiple questions and purposes of
  the evaluation require data from several
  different sources
• Archival/secondary data (e.g., Annual reports,
  State Applications, IUPs, Census data)
• Primary data collection (SPC and provider
  surveys, site visit assessments and focus groups
  with PATH clients)
• Data selected to support analysis of Program Model

25
PATH Logic Model
26
Other Measures

• Identification of PATH clients
• Sustained engagement of clients
• Linkage of PATH and other services
• Accessibility to and use of housing and MH care
  opportunities
• Acceptability of services
• Gaps (training, outreach, assessment, services,
  linkages, measurement and reporting
• Trends in homelessness (populations, mix of
  services)
• Identification of innovation and use of best
  practices

27
Data Sources

• Archival/Secondary Data
• 2009 State Applications and Annual Reports
  (context, grant/organizational structure,
  population and service characteristics, GPRA)
• Bureau of Census (American Community Survey), HUD
  (HMIS homelessness count), National Survey of
  Homeless Assistance Providers (context, community
  and population characteristics)

28
Primary Data Collection

• 14 Site Visits
• 4 site visits between 12/6/10 and 1/30/10
• Pre-test in-person interview and online survey
  data collection instruments and client focus
  group questions for comprehensiveness and
  usefulness
• Interviews w/SPC, at least 2 provider staff and
  1-2 clients
• 10 assessment visits after OMB approval
• Detailed assessment of grant structure process
  perceived outcomes and service gaps.
• Recommendations

29
Data Requirements

• Site Assessments
• SPC and key state staff
• 1-8 provider Directors and key staff offering
  direct service to clients
• Depends on size and distribution of provider
  network
• Focus group with 8-12 clients re acceptability,
  quality, gaps.

30
Online Surveys

• Developed with recommendations from SAMHSA
  staffers, Expert Panel and pre-test participants
• Will be approved by OMB
• Focused on assessing grant implementation
  process, throughput, outcomes and gaps
• 56 SPCs
• 483 Local providers (1/provider agencystaff who
  provide direct face-to-face services)

31
Online Surveys cont.

• Survey will require about 30 minutes to complete
• Unique password assigned to respondent will allow
  completion over multiple sessions
• Your cooperation in completing these surveys is
  ESSENTIAL!!! Please help.

32
Analyses

• Describe PATH implementations
• Look for commonalities in structure, populations
  served and service mix
• Create a typology of program implementations
• Relate PATH program models to outcomes to see if
  there is differential effectiveness
• Identify best practices
• Inform SAMHSA about service, training gaps and
  inform decisions on technical assistance still
  needed

33
Evaluation Project Status

• Developed reliable coding and data extraction
  procedures for State applications and IUPs
• Currently coding both, expect completion by
  mid-February
• Preparing Annual Report Data to merge with coded
  information extracted from Annual Reports and
  State Applications
• Pilot testing of protocols and instruments begun
  on December 6. Will complete around mid January
• OMB draft with revised instruments to SAMHSA by
  the end of January
• Field site assessments and surveys should begin
  by July 2011.

34
Questions and Comments

• Lisa Kleppel or Paul Brounstein
• MANILA Consulting Group
• 6707 Old Dominion Drive
• McLean, VA 22101
• 571-633-9797