DHHS and Outcome Measurement

1
DHHS and Outcome Measurement

• Wes Albinger, Contract Services Coordinator
• Milwaukee County
• Department of Health and Human Services
• Contract Administration
• March 16th, 2006
• Presentation will be available at
• http//www.county.milwaukee.gov/display/router.asp
  ?docid15483
• Starting Monday, March 20, 2006

2
Intended Outcomes of OM Initiative

• Initial Agencies understand DHHS definition and
  application of OM concepts. Agencies understand
  the direction DHHS is headed with Outcome
  Measurement and their role in that process.
• Intermediate DHHS articulates long term outcomes
  for each program area. Agency program activities
  and initial outcomes are aligned with DHHS long
  term Outcomes identified for each program.
• Long Term Indicators and data collection
  protocols are established for each program area.
  OM is part of program and contract requirements.

3
DHHS and Outcomes WHY?

• To clearly communicate expectations (goals
  and objectives) to provider
  agencies.

• To establish an objective standard which is
  consistently applied to measure and report
  program impact.
• To provide DHHS and Agencies with a meaningful
  way to report success to stakeholders (e.g.
  client, provider, funding agencies, etc.).

4
DHHS and Outcomes WHY?

• To promote client choice.
• To increase benefits for clients from contract
  dollars.
• To improve program performance via feedback and
  program improvement.

5
Purpose of Todays presentation

• This is a multi-year project that will begin
  simply with establishment of outcomes and a
  requirement that agencies report to us how they
  think they are achieving those outcomes. At this
  point, DHHS will not be holding programs to a
  specific set of indicators. In subsequent years,
  contracts may actually include not only the
  outcomes, but also the indicators and performance
  goals.
• None of this is etched in stone yet -- we have
  called you together at this juncture specifically
  to gather your input.

6
FAQ

• Q Is DHHS telling us how to run our program?
• A No. Outcome Measurement focuses more on the
  impact of the program than the program itself.
  While the outcome is affected by the program, we
  are more concerned with the end of the program
  than the means to that end. Basically, we are
  interested in defining where we want programs to
  go, but not how to get there.
• In some instances, external factors prescribe
  program outcomes or processes. An example would
  be the new Federal Birth to Three Outcomes which
  will be required for all BT3 programs.
  Additionally, the growth in popularity of
  evidence based program design is putting
  increasing pressure on programs to adopt proven
  approaches to service delivery. These will
  likely play an increasing role in programs.

7
FAQ

• Q What role will Agencies play in this process?
• A DHHS will articulate long term outcomes for
  each program, distilled from estimation of client
  need, agency practice and capacity, current
  program requirements, federal, state and other
  mandates, internal staff, and research. DHHS
  will solicit information from agencies relating
  to outcomes, indicators, and data collection
  methods to help inform this process. Keep in
  mind that on a program by program basis, the
  amount of discretionary slack in setting
  outcomes varies greatly.
• Throughout the process, DHHS will release
  information for review and comment.

8
FAQ

• Q What are the implications of these changes for
  my Agency in the short term?
• A For 2007, we will not be prescribing
  indicators and performance goals in association
  with the long-term outcomes. Long term outcomes
  will be identified for several pilot programs for
  the FY 2007 RFP. For these programs, you will be
  asked to describe in detail how your program
  design and outcomes relate to the long term
  outcome identified for your program, using
  successful past performance or a research based
  justification. Additionally, you will be asked
  to identify what indicators you are or propose to
  use to measure the achievement of the outcome, as
  well as your data collection and or measurement
  tool(s) (concepts explained shortly).

9
FAQ, implications for short term, contd

• For those programs without DHHS identified long
  term outcomes in the FY 2007 RFP, you will be
  asked to describe your current or proposed
  program design and outcomes, using successful
  past performance or an evidence based
  justification. Additionally, you will be asked
  to identify what indicators you are or propose to
  use to measure the achievement of the outcome, as
  well as your data collection and or measurement
  tool(s).

10
FAQ, implications for short term, contd

• In other words, tell DHHS
• What do you intend to do? (program design)
• Why do you intend to do it? How does what you
  are proposing contribute toward the long term
  outcome? (rationale/justification)
• In what way(s) do you expect clients to benefit?
  (short, intermediate, and long-term outcomes)
• How will you know if you have accomplished this?
  (indicators)

11
FAQ

• Q What are the implications of these changes for
  my Agency in the long term?
• A Indicators and data collection protocols for
  each program area will be developed through an
  inclusive process. OM will be part of program
  and contract requirements.

12
FAQ

• Q How will this initiative create work for my
  agency?
• A This initiative is more focused on reporting
  differently, versus reporting additional
  information. The required reporting will be part
  of your semi annual evaluation reports.
• DHHS will seek to minimize the administrative
  burden. One way this will be accomplished is by
  soliciting Agency input to determine current
  capacity and potential technical assistance
  needs.

13
Example-Agency input

• It is worth noting that DHHS has done an
  extensive review of existing outcome data
  currently collected to help formulate our
  thinking and strategy. Many of the examples used
  in this presentation are actual outcome data
  already collected we are looking at the content
  currently being provided as a starting point.

14
Agency Input

• You may recall recently completing an Outcome
  Measurement Survey. These were sent to all
  agencies with purchase of service contracts.
  Some notable results, which have already been
  used to inform our progress on this initiative,
  are as follows

15
Survey responses

• Agencies have widely interpreted definitions of
  Outcome as such, there are widely varied
  applications of Outcome Measurement.

16
Survey responses

• Most indicated an interest in an outcome
  measurement presentation.
• Most indicated an interest in a formatted
  spreadsheet for measuring outcomes.
• Vast majority are currently reporting outcomes.
• Approximately half cited fundor requirements as
  the basis for deciding to report outcomes. It
  appears that many agencies are not finding an
  internal benefit to tracking outcomes, but are
  doing it because they have to.

17
Survey responses

• Most already track outcomes for ALL programs.
• Most common theme that emerged as concern was
  administrative/cost burden, with the second being
  data reliability and validity.
• Vast majority have developed a logic model or
  other visual representation of main programs.
• Most common methodology employed for measuring
  outcomes was United Ways model.

18
Technical Assistance

• DHHS will provide technical assistance throughout
  this process.
• Let DHHS know what your technical assistance
  needs are.

19
On to Outcomes

• This is intended to give you a sense of the
  process DHHS is using internally to determine
  outcomes, and may be helpful to you as you
  identify how your agencys program will work
  toward a specific long term outcome.
• DHHS will be using a methodology which is already
  familiar to many of you, and enjoys a broad base
  of support. It has become most familiar to
  practitioners through the United Way, but it is
  used more widely than that.

20
Some thoughts on Terminology

• Throughout this presentation, I will be using
  specific terminology. The concept behind the
  terminology is more important however, because
  there is a wide use of terminology and
  interpretation of terminology among agencies, it
  is important that
• We adopt a common language for discussing Outcome
  Measurement concepts, and that
• We use it consistently.

21
What are Outcomes and Why Measure Them?

• OUTCOMES-
• The intended benefit(s) for participants during
  or after their involvement with a program. You
  can also think of Outcomes as the OBJECTIVE of
  the program.

22
Assumptions and Organizational Challenges to
implementing OM

• Support for OM is based on the assumption that
  the information produced creates value that
  exceeds the time and effort to collect it.
• The challenge, then, is to make sure that the
  information that OM produces is useful, and that
  collecting it is administratively as simple as
  possible.
• DHHS wants OM information to serve useful
  purposes for all affected stakeholders.

23
Assumptions and Organizational Challenges to
implementing OM

• Concern about OM also comes from the belief that
  the outcomes measured are not an accurate or
  complete reflection of the program.
• Our challenge is to identify outcomes which are
  appropriate. Your input will help

24
An Outcomes Makeover

• The Before
• Served 350 people.
• Hosted 10 resume writing seminars.
• Provided 150 hours of individual career
  counseling.
• 25 staff attended Outcome Measurement training.
• Notice emphasis on delivery of service, or the
  program itself.

25
An Outcomes Makeover

• Gee, you sure did a lot, but what happened???
• Did your clients get jobs?
• Did your clients learn anything?
• Did your clients change their lifestyle?
• In general, how were your clients affected or
  changed?

26
An Outcomes Makeover

• The After
• 70 (160 out of 230) of clients over 50 years of
  age obtained employment .
• Of the total, 65 were able to maintain
  employment for at least 1 year.
• Notice emphasis on the benefit or change to the
  person served. No (or little) emphasis on the
  actual program. In fact, the program can only be
  inferred from the outcome statement. These
  statements reflect the OBJECTIVE of the program.

27
What Exactly is An Outcome?

• Benefits for participants during or after their
  involvement with a program.

Change in behavior
Change in knowledge or skills
Change in condition
28
Examples of Outcomes

• Clients decreased emergency room visits.
• Participants significantly reduced alcohol/drug
  use.
• Immigrants were able to secure legal residency
  and citizenship.
• Staff demonstrate proficiency with Outcome
  Measurement concepts.

29
Outcome or not?

• Clients increase community activities.
• Clients participate in skill training programs.
• Residents feel safer.
• In 2004, the Advocacy program assisted 75
  individuals.
• Clients reduce emergency room use.

30
Task 1 Choose the Outcomes You Want to Measure

• Hints
• Be thoughtful!!
• Define Your Target Population.
• Engage staff and clients in the discussion.
• Choose meaningful/compelling outcomes.
• Write outcomes so John Doe will be able to
  understand.
• Develop a Logic Model.

31
Logic Model A Picture of how a Program Works
Inputs
Activities
Outputs
Outcomes
Staff Outpatient Clinic Referring
agencies Individuals with mental illness Outside
environment
75 psychiatric sessions 150 drug prescriptions 65
subsidized housing applications 175 crisis
calls 80 resume writing/job placement sessions
Psychiatric Services Pharmacy Services Nursing
Services Housing Assistance 24 hour crisis
hotline Vocational Services
Reduce level of incarceration of adults with
mental illness Reduce level of hospitalization
of adults with mental illness
32
Levels of Outcomes

• In most cases, a program attempts to affect not
  one, but a series of changes for individuals,
  each building on the previous one. These can be
  thought of as the stepping stones toward the
  ultimate, or long term goal of the program. We
  can define these as
• Initial Outcomes
• Intermediate Outcomes
• Long term Outcomes

33
Relationships among levels of outcomes

• Levels of outcomes can be thought of in terms of
  a series of interrelated milestones. The
  achievement of one sets the condition which
  increases the chances of achievement of the next.
  
• In other words, IF a happens, THEN b is more
  likely to follow.

34
Relationships among levels of outcomes

• Example
• IF job seekers learn appropriate job interview
  skills (initial outcome), THEN they will use
  appropriate job interview skills in the interview
  (intermediate outcome). IF they use appropriate
  interview skills, THEN they will obtain jobs
  (long-term outcome).

35
Relationships among levels of outcomes

• Similarly, it can work backwards if you know
  the desired long term outcome for your program,
  you can ask yourself, what will make this more
  likely to occur?
• Example Intended benefit (Long Term Outcome)
  for Employment Program Clients get jobs. What
  will make this more likely to occur? Answer (one
  of many possible answers) Clients learn and
  demonstrate appropriate job seeking skills.
• There should be a theoretical/logical
  (research or practice supported) link supporting
  causality between the various outcomes.

36
Initial Outcomes

• Initial Outcomes-The most immediate changes and
  most heavily influenced by the program, and are
  typically expressed in terms of new knowledge,
  skills, or attitudes learned through the program.

37
Initial Outcomes

• Examples of Initial Outcomes
• Program Employment
• Initial Outcome Clients learn appropriate job
  seeking skills.
• Program Adult Day Services
• Initial Outcome Clients increase awareness of
  community resources.
• Program Outcome Measurement Initiative
• Initial Outcome Staff demonstrate proficiency
  with Outcome Measurement concepts.

38
Intermediate Outcomes

• Intermediate Outcomes-A link between the
  programs initial outcomes and long term
  outcomes, typically expressed in terms of changes
  in behavior connected to the new knowledge,
  attitude, or skill (initial outcome) learned
  through the program.

39
Intermediate Outcomes

• Examples of Intermediate Outcomes
• Program Employment
• Intermediate Outcome Clients utilize appropriate
  job seeking skills.
• Program Adult Day Services
• Intermediate Outcome Clients increase
  utilization of public and private services in
  their community.
• Program Outcome Measurement Initiative
• Intermediate Outcome Staff apply Outcome
  Measurement concepts to programs.

40
Long Term Outcomes

• Long Term Outcomes-The ultimate objective of the
  program, typically expressed in terms of changes
  in condition or status. When defining Long Term
  Outcomes, it is important to select only outcomes
  which the program can be reasonably expected to
  influence.

41
Long Term Outcomes

• Examples of Long Term Outcomes
• Program Employment
• Long Term Outcome Clients are hired.
• Program Adult Day Services
• Long Term Outcome Clients generalize acquired
  skills to other settings.

42
Important note, Long Term Outcomes

• Just because outside influences may affect
  achievement of an outcome, that does not mean the
  outcome is inappropriate. We can adjust for this
  by recognizing that outcomes will not be achieved
  100 of the time.
• As an example, consider the effect of the economy
  (outside influence) on an employment program.
  The state of the economy cannot rule out the
  outcome clients obtain jobs rather, the
  existence of an outside influence means that we
  should expect to see less than 100 of the
  outcome achieved.

43
Important note, Long Term Outcomes

• Long Term Outcomes are of greatest interest to
  DHHS and will be the primary focus of our efforts
  to define outcomes for each program.

44
Example

• Suppose we identify the long term outcome,
  Clients get hired. Well, we dont necessarily
  care if the means to that end are the initial
  outcome clients learn job seeking skills or
  clients are connected to warm job leads
  generated by job developers. Either initial
  outcome can lead to the long term outcome.
  Depending on the particular agencys expertise,
  service delivery method, or client mix, they may
  prefer one means to the other.

45
Example

• Another example would be the long term outcome,
  clients are appropriate weight. One program
  may choose to work toward this end by targeting
  the initial outcome, clients are able to
  identify healthy food and the intermediate
  outcome, clients eat healthy food, while
  another program may identify, clients learn the
  benefits of exercise as the initial outcome and
  clients exercise at least 2x per week as the
  intermediate outcome.

46
Implications of Different Means to a Common End

• Therefore, you can consider the Long Term
  Outcomes identified for your program to represent
  the direction we want you to go in. You will be
  responsible for deciding the best way to get
  there.
• In some instances, external factors prescribe
  program outcomes or processes. An example would
  be the new Federal Birth to Three Outcomes which
  will be required for all BT3 programs.
  Additionally, the growth in popularity of
  evidence based program design is putting
  increasing pressure on programs to adopt proven
  approaches to service delivery. These will
  likely play an increasing role in programs.

47
Task 2 Specify Indicators For Your Outcomes

• Hints
• Outcomes and Indicators are not the same thing.
• Indicators-Measurable approximations of the
  outcomes you are attempting to achieve. They
  determine the extent to which the outcome has
  been met.
• Indicators look at the outcome and ask, How do
  you know?

48
Task 2 Specify Indicators For Your Outcomes

• Think about the outcome of the program. Ask
  yourself
• How do I know this Outcome when I see it?
• What does it look like when this Outcome happens?
• What are the characteristics of someone who has
  achieved this Outcome?

49
Task 2 Specify Indicators For Your Outcomes

• Example
• Program- Recreation.
• Outcome- Clients socialize with peers and
  non-disabled individuals.
• Indicator- Number of integrated community
  activities participated in during the previous
  month.

50
Examples of Program Indicators
51
Indicators, continued

• Notice how Indicators decrease the ambiguity of
  determining whether the outcome has been met.

52
Task 3 Prepare to Collect Data on Your
Indicators

• Hints
• Identify data sources for your indicators
• i.e., Records, General Public, Program
  Participants, Parents
• Design Data Collection Methods
• i.e., Review of Program Records, Questionnaire,
  Interview

53
Things to Think About when developing a data
collection method

• Collect info that you want to know the answers
  to (Dont just do it to satisfy funder
  requirements!!)
• Dont reinvent the wheel (Look for instruments
  that have already been developed!!)

54
Task 4 Analyze Findings

• Hints
• Make it Easy to Tabulate Your Findings. If
  possible, computerize!!!
• Calculate the and of clients achieving the
  outcome.
• Look at how gender, race/ethnicity, income, etc.
  influenced achievement of outcomes.

55
Task 5 Use Your Findings

• If the outcome measurement system is not used
  for helping the organization improve its
  services, the time and effort will have been
  wasted.
• - Outcome Management in NP Orgs , Urban
  Institute, 2001.

56
Todays intended training outcomes, revisited

• Initial Agencies understand DHHS definition and
  application of OM concepts. Agencies understand
  the direction DHHS is headed with Outcome
  Measurement and their role in that process.
• Has this been accomplished?

57
The role for Agencies

• DHHS will identify long term outcomes for some
  programs by the release of the 2007 RFP.
• As mentioned previously, DHHS will define the
  long term outcomes for each program. Agencies
  will need to think about how they will work
  toward that outcome. In reviewing proposals DHHS
  will look at how logically connected your initial
  and intermediate outcomes are to the Long Term
  Outcome defined for the program, as well as how
  you propose to use indicators and data collection
  methods.

58
The role for Agencies

• Agencies can make suggestions for outcomes,
  indicators, and data collection based on their
  practices let DHHS learn from you by showing us
  what you are currently doing. You may submit
  this information at any time, but will be asked
  for it as part of your FY 2007 submission.

59
The role for Agencies

• DHHS will use your proposed strategies for
  outcomes, indicators, and data collection methods
  as a jumping off point as we consider
  establishing these for each program area.

60
Resources, References

• United Way of America Outcome Measurement
  Resource Network www.unitedway.org/outcomes
• Measuring Program Outcomes A Practical Approach,
  United Way of America, 1996. (Available through
  website for 5)
• Kellogg Foundation Evaluation Handbook Logic
  Model Guide www.wkkf.org
• Urban Institute (Publications by Harry Hatry)
• www.urban.org
• Fiscal Policy Studies Institute (Mark Friedman)
• www.resultsaccountability.com