MEASURE UP A Program Evaluation Training

1
MEASURE UPA Program Evaluation Training

• Stephen Day and Jim Yates
• Technical Assistance Collaborative
• Funded by
• US Department of Housing and Urban Development,
  Massachusetts State Office

2
Agenda

• 900-905 Introduction/ Administrative
  Matters
• 905-1000 Importance of Program Evaluations
  to CoC
• 1000-1100 Program Evaluation Process
• 1100-1105 Break
• 1105-Noon Introduction to HUDs Logic Model
• Noon-100 Lunch (on your own)
• 100-200 Logic Model Practical Exercise
• 200-210 Break
• 210-330 Report Back/Discussion
• 330-400 Wrap-Up

3
Importance of Program Evaluations to Continuums
of Care
4
Why the Focus on Program Evaluation?

• MA State CPD Office has recognized the increasing
  importance that program evaluation will play on
  McKinney-Vento programs in the future.
• Government Performance and Results Act of 1993.
• Clinton Administrations Reinventing Government
  Effort
• Office of Management and Budget requires each
  federal agency including HUD to prepare an annual
  performance plan.

5
Why the Focus on Program Evaluation?

• To comply with OMB requirements, HUD develops
  Strategic Goals and corresponding objectives.
• Within the SuperNOFA, HUD has rated each
  applicant on how well it ties the proposed
  outcomes to HUDs Strategic Goals and objectives.
• HUD introduced the Logic Model two years ago.
• HUD may require either Continuums of Care or
  individual McKinney-Vento programs to complete
  the Logic Model in the future.

6
HUDs Strategic Goals

• 1.) Increase Homeownership Opportunities.
• 2.) Promote Decent Affordable Housing.
• 3.) Strengthen Communities.
• 4.) Ensure Equal Opportunity in Housing.
• 5.) Embrace High Standards of Ethics,
• Management and Accountability.
• 6.) Promote Participation of Grassroots, Faith-
• Based and Other Community-Based
• Organizations.

7
HUDs Policy Priorities

• HUD establishes policy priorities in order to
  focus attention on issues of
• importance.
• Provide Increased Homeownership and Rental
  Opportunities for Low- and Moderate-Income
  Persons, Persons with Disabilities, the Elderly,
  Minorities, and Families with Limited English
  Proficiency
• Improving the Quality of Life in our Nations
  Communities
• Encouraging Accessible Design Features
• Providing Full and Equal Access to Grass-Roots,
  Faith-Based and Other Community-Based
  Organizations in HUD Program Implementation
• Participation of Minority-Serving Institutions in
  HUD Program
• Ending Chronic Homelessness within Ten Years and
• Removal of Barriers to Affordable Housing.
• Within the 2004 SuperNOFA, HUD encouraged
  applicants to propose
• specific activities that will assist them in
  implementing these priorities.

8
Importance of Program Evaluation to a Continuum
of Care

• Continuum of Care Funding Application
• Tool for Planning
• Internal Quality Improvement
• Assist Continuum of Care with the Continued
  Implementation of HMIS
• Public Relations

9
Continuum of Care Funding Application

• HUD will continue to put a greater importance on
  how well programs are performing within the CoC.
• The development of a program evaluation process
  would better prepare CoC to respond to future
  questions in the Exhibit 1 narrative.
• In the 2004 Exhibit 1, there were three
  questions that related to how a CoC is monitoring
  and evaluating their projects and how well
  renewal projects are performing.

10
Continuum of Care Funding Application

• 2004 CoC Application
• a.) In the CoC Priorities Section, HUD asked how
  a renewal was determined to be performing
  satisfactorily and addressing the needs for which
  it was designed.
• b.) In the Supplemental Resources Section, HUD
  asked CoCs to check whether the CoC
  systematically analyzes its projects APRs to
  assess and improve access to mainstream programs.
• c.) In the Project Performance Section, HUD asked
  CoCs to provide data from the renewal projects
  most recent APRs.
• Permanent Housing
• Transitional Housing
• Access to Mainstream Programs and Employment

11
Tool for Planning

• More exact and quantifiable data for gaps
  analysis.
• Performance-based justifications for changes in
  CoC priorities, renewal recommendations, and new
  project priorities.
• Performance-based data for engaging local
  officials, policy makers and funders in CoC
  planning and development.

12
Internal Quality Improvement

• Use a program evaluation process to identify
  systematic and individual program strengths and
  weaknesses.
• Once weaknesses are identified, the CoC can work
  with the provider to identify an intervention in
  order to improve performance.
• Over time, this follow-up assistance and
  monitoring will improve the quality of programs
  in the CoC.

13
HMIS Implementation

• HUD intends to use HMIS data to measure
  performance of both McKinney-Vento programs and
  Continuums in the future.
• It is likely that HMIS data will inform the
  evaluation process that HUD establishes in the
  future.
• Getting a head start on a program evaluation
  process will prepare a CoC for this move.

14
Public Relations

• Use outcome and performance data to highlight
  successes.
• Demonstrate progress that CoC/program(s) have
  made in moving homeless in individuals and
  families to permanent housing and greater
  self-sufficiency.
• Allows providers to take the high ground on
  homelessness issues.

15
Program Evaluation 101

• (Chapter Two)

16
Basic Questions

• Did we serve the people we planned to serve?
• Did we give them the housing/services we planned
  to provide?
• Having delivered the specified housing/services
  to the desired target population, did we actually
  achieve the results we wanted for those people?

17
Evaluation Accountability

• Are we assisting homeless people to move into
  permanent housing?
• Do housing and services programs make a
  difference in peoples lives?
• Are we using our dollars wisely?
• Are there improvements needed in our housing and
  services programs?
• Can we provide positive information to the
  community about the successes of our customers
  and programs?

18
Bottom Line

• Results have to be directly related to positive
  change for the people housed and served
  permanent housing community stability
  self-sufficiency.
• The only thing that really matters is whether
  the desired results have been achieved for
  defined priority customers.

19
Definition of Terms

• (Chapter Three)

20
The Program

• Program the thing you do to change peoples
  lives.
• Known as the intervention in research terms.
• For today, the program is considered to be the
  specific service or activity that is intended to
  produce the desired results.
• Not to be confused with other programs, such as
  McKinney/Vento Shelter Care Medicaid, etc.

21
Program Performance Measurement

• Bottom line is outcomes/results
• Also have to measure some basic items
• Meeting implementation schedules/milestones
• Complying with regulations and contract terms
• Collecting and reporting data about customers and
  program activities
• May also want to measure or document effects on
  the environment (e.g., new discharge planning
  protocols, interagency agreements related to
  employment, etc.)

22
Program Inputs

• Things that make the program happen
• Staff
• Money
• Shared resources
• Facilities
• Rental subsidies

23
Program Outputs

• Outputs are the things a program produced with
  its inputs
• Units of permanent housing developed or acquired
• Number of new chronically homeless individuals
  engaged in services
• Number of outreach encounters delivered

24
Program Outcomes

• Outcomes are the results or impact of the outputs
  produced by the inputs (services or activities)
• HUD Definition Outcomes or results are benefits
  to individuals, families, organizations and
  communities derived from participation in a
  program or service. Outcomes are always
  measurable.

25
Outcomes (2)

• Outcomes are ends, not means.
• Outcomes measure changes changes over time
  changes in status changes in direction.
• An outcome measure must state
• Who is intended to be impacted?
• What will the intended impact be? and
• When will the intended impact occur?

26
Proxy Outcomes

• Substitute measures when outcomes cant be
  directly measured (not enough time not enough
  information, etc.)
• Example being housed used as a proxy measure for
  housing stability or tenure.
• Example Staying in a job for 6 months as opposed
  to long term self-sufficiency

27
Outcome Targets or Benchmarks

• Defines the expected level of performance for
  each selected outcome measure.
• Example
• Retention of permanent housing for at least one
  year is an outcome measure.
• 70 of participants retain permanent housing for
  at least one year during the program term is a
  benchmark or target.

28
Quantitative Data

• Remember outcomes must be measurable.
• Measuring means counting counting means
  quantitative data
• How much?
• How many?
• How soon?
• What you count has to be timely, accurate,
  consistent, and comparable these are the
  criteria for objective, quantitative data.

29
Qualitative Data

• Cant be counted absolutely, but can be
  tabulated, summarized, compared
• Can be good in combination with quantitative data
• Example You can count how many people were
  housed, but you cant count with the same degree
  of accuracy how those people felt about being
  housed
• However, gathering information on how they felt
  about being housed can add understanding and
  depth to the quantitative data

30
Analysis and Interpretation

• Not enough to count and tabulate you have to
  understand what the data is telling you what
  can you learn about how the inputs and outputs
  worked (or didnt work) to produce outcomes
• Analysis is the process of judging the overall
  impact of a program, and comparing its results to
  other similar programs serving similar customers.

31
Conclusion

• What you measure is what you get
• Make sure that what you measure is consistent
  with your agencys mission and values
• Make sure that what you measure is really
  important to your priority customers
• Make sure that what you measure is directly
  related to what you are expected to accomplish
  with your resources

32
Program Performance Evaluation Process

• (Chapter Four)

33
Mission, Vision and Values

• Mission statement who the organization serves
  and how these people are expected to benefit from
  the services
• Vision statement future vision of how things
  will have changed for customers, the organization
  and the community as a result of the
  organizations actions
• Values statement what the organization believes
  about itself and the people it serves

34
Mission, Vision and Values (2)

• Mission, vision and values proscribe the context
  or framework within which program actions are
  designed, and in which consumer-based outcome
  measures are selected.
• For example, if your organizations mission and
  vision is to end chronic homelessness, then it is
  natural to measure the number of chronically
  homeless people moving into permanent housing
  every year

35
The Performance Evaluation Cycle
36
Data Sources HMIS
37
Use of Outcome Measurement for Quality Management

• Use the findings and conclusions from program
  performance evaluation to identify problems and
  decide of appropriate steps to improve the
  program in the future.
• Remember outcome evaluation produces answers,
  but it also produces questions Quality
  management is the process of asking why did this
  or that result occur, and how can we make it
  better in the future?

38
Useful Tips for Developing Effective Outcome
Measures

• Capture the impact of the program in Three or
  less outcomes.
• Minimize soft outcomes (qualitative or
  subjective).
• Use pre-existing measurement tools (if possible)

39
Common Mistakes in Developing Outcome Measures

• Services are stated as an outcome.
• Ex. - Participants will receive job readiness
  training.
• Absence of a clear relationship between the
  service and the outcome.
• Outcome cannot occur within established
  timeframe.
• Performance is under or over estimated.
• i.e. 100 retention rate

40
Program Evaluation Conclusions

• Use three or fewer outcome measures keep it
  simple
• Look at change in status as the major element to
  be tracked in HUD programs
• Audience is the Public and Congress, not just HUD

41
Introduction to HUDs Logic Model

• HUDs Logic Model is a tool that a CoC may use to
  implement a program evaluation process.
• TAC anticipates that HUD will require a CoC
  and/or individual grant applicants to develop a
  Logic Model for future SuperNOFA competitions.
• Step by Step overview of the Logic Model.

42
Simplified Version of the Logic Model
Transitional Housing Program for Homeless Men
Inputs
Activities
Output Goals
Outcome Goals
Program budget
of participants housed in the facility
and of participants obtain permanent housing
Case Manager
Provide transitional housing for homeless men
Property Mgmt. Staff
of Participants who participate in life skills
or employment training
and of participants obtain employment
On-site Case Management and Life/Employment
Skills Training
6 unit building
43
HUDs Logic Model

• Program Name
• - Fill in the HUD funding program which you are
  applying.
• Ex. For a CoC filling out the form, it would be
  McKinney-Vento Continuum of Care.
• Component Name
• - If the there is a component to the HUD program
  fill it in.
• Ex. If the specific project was an SHP funded
  program then it would be the Supportive Housing
  Program.

44
HUDs Logic Model

• Column 1, HUDs Strategic Goals/ Policy
  Priorities
• Fill out the numbers representing the specific
  HUD Strategic Goal and Policy Priority that are
  furthered by the project.
• HUDs Strategic Goals and Policy Priorities are
  listed at the bottom of the Logic Model form.

45
HUDs Logic Model

• Column 2, Problem, Need or Situation
• Provide a general statement of need that supports
  the start-up or continuation of the specific
  project.
• Example
• Annually, 100 families are homeless in
• the City of Aurora. Over the past 5 years,
• family homelessness has increased 28.

46
HUDs Logic Model

• Column 3, Service or Activity
• Specify the primary activities or services of the
  project that are critical to its success.
• Dont include all activities - just the primary
  ones.
• Example
• The program provides 15 beds of
• transitional housing. The program also
• provides an array of supportive services to
• include case management and substance
• abuse counseling.

47
HUDs Logic Model

• Column 4, Benchmarks/Output Goals
• Identify specific benchmarks that will be used to
  measure the projects success.
• Short, intermediate and long-term in duration.
• Need to be quantifiable including a timeframe for
  achievement.
• Example
• 20 participant will receive job placement
• assistance during the program year (12
• months).

48
HUDs Logic Model

• Column 5, Benchmarks/Output Results
• Identify the results for the Output Goals
  identified in Column 4.
• Output results are numerical measures.
• Short, intermediate and long-term in duration.
• Leave blank during initial application.
• Finished at the end of the grant period when the
  activities are complete and data is available.
• Example
• 17 participants (85 of goal) receive job
  placement
• assistance services during the program year.

49
HUDs Logic Model

• Column 6, Outcomes/ Achievement Outcome Goals
• Specify the outcome goals to measure the impact
  of the projects services and activities.
• Measure the impact the project on the community
  and the participant.
• Each output identified in Column 4 should have an
  associated outcome goal identified.
• Short, intermediate and long-term in duration.
• If the outcome is a proxy outcome, provide an
  explanation of why the proxy outcome was chosen.
• Example
• 10 participants or 50 of participants who
  receive
• job placement assistance will increase their
  income
• during the program year.

50
HUDs Logic Model

• Column 7, Outcomes/End Results
• Identify the achievements for each outcome
  identified in Column 6.
• Short, intermediate and long-term in duration.
• Leave blank during initial application.
• Finished at the end of the grant period when the
  activities are complete and data is available.
• Example
• 7 participants who receive job placement
• assistance (82 of goal) increase their income
• during the program year.

51
HUDs Logic Model

• Column 8, Measurement Reporting Tools
• Provide the specific measurement tools that are
  used to collect the projects data.
• HUD requests the following information in this
  column
• Tools used to track output/outcome data Ex
  HMIS, APRs
• Place where data is maintained Ex HMIS
  database, program database
• Location of the data Ex County office, program
  office
• How often the data is required to be collected,
  who will collect it and how often is it reported
  to HUD. - Ex Data collected monthly, annually to
  HUD through the APR
• Methods for retrieving the data. Ex Data
  retrieve electronically through HMIS reports.

52
HUDs Logic Model

• Column 9, Evaluation Process
• Describe the methodology the project will use to
  assess the success in meeting the output goals
  and outcome goals established.
• Tool to make necessary changes in the project in
  order to achieve the established goals.
• Example Agencys management team reviews data
  and outcomes from the program on an quarterly
  basis to monitor performance and make any needed
  changes to the program design or services in
  order to achieve the established goals.

53
Logic Model Practical Exercise

• Notional McKinney-Vento Projects.
• Mix of new and existing projects.
• Handout Program description and HUD
  McKinney-Vento Program Overviews.
• Prepare a Logic Model for submission in HUDs
  SuperNOFA funding round.

54
Report Back/Discussion
55
Wrap-Up
56
Concluding Thoughts

• Measure results for people, not programs
• Measure change in status
• Chose a very small number measures directly
  linked to your policy goals
• Use existing data
• Use what you measure quality improvement and
  public information
• What you measure is what you get