Program Evaluation 201: Performance Measurement for Fun

1
Program Evaluation 201 Performance Measurement
for Fun Profit

• Presented by Barbara Dunn
• for
• The Wythe-Bland Community Foundation
• June 10, 2009

2
Purpose Objectives

• This presentation will focus on performance
  measurement to build nonprofit capacity.
• At the conclusion, participants will be able to
• Clearly document needs or problems
• Define measurable goals objectives
• Relate strategies activities to process
  outcome measures
• Draft a logic model evaluation plan

3
Proving vs. Improving(Innovation
Network/Evaluation Handbook, 2005)

• The old thinking Evaluation to prove something
  to someone else
• The new thinking Performance measurement to
  improve something for yourself your community
• The goal A meaningful process valuable
  findings (fun profit?)

4
Program Planning Evaluation Model (Va. Dept.
of Education, 2002)
5
Program Planning Process

• 1. Identify community or organizational needs
• 2. Develop goals to meet those needs
• Achieving goals generates an impact on
  clients/the organization
• 3. Establish objectives to work toward goals
• Meeting objectives produces outcomes for
  clients/the organization
• 4. Design structured activities that address
  established objectives
• Implementing activities creates organizational
  processes
• 5. Evaluate the results report your findings
• Evaluating the results provides information
  needed to document success improve performance

6
Making the Connections Evaluation Framework

• This framework helps make the connections among
• Needs ? Goals ? Objectives ? Activities
• Activities are designed to achieve established
  objectives, which support goals that address
  identified needs
• Logic models are popular tools that can be used
  to represent these relationships both graphically
  in narrative form

7
What does a logic model look like?

• Graphic display of boxes and arrows vertical or
  horizontal
• Relationships, linkages
• Any shape possible
• Circular, dynamic
• Cultural adaptations storyboards
• Level of detail
• Simple or complex
• Multiple models
• At least 3 purposes
• Risk Model
• Change Model
• Implementation Model

8
Making the Connections, Logically(Innovation
Network, Logic Model Workbook, 2005)
9
(No Transcript)
10
Making the Connections Evaluation Framework

• (Innovation Network, Evaluation Plan Workbook,
  2005)
• Relationships between the logic model
  evaluation plans

11
Definition of Terms

• Program A structured service area or
  administrative initiative that in some instances
  may be more appropriately called a project.
• These activities are organized to meet certain
  objectives and produce certain outcomes.
• Examples
• Home visiting (service area)
• Asthma case management (service project)
• Capacity-building (administrative initiative)
• Strategic planning (administrative project)

12
Definition of Terms

• Outcome A measurable result or benefit that you
  want to achieve, usually to change some condition
  in a particular population.
• AKA outcome objective, outcome goal, results
  indicator
• These outcomes will assist in meeting objectives
  that support your goals.
• Examples (Hint start with an active verb)
• Reduce school absenteeism (in children with
  asthma)
• Develop a strategic plan

13
Definition of Terms

• Indicator or Measure A descriptor that tells
  what is being measured (indicator) how youre
  measuring it (measure) to determine whether the
  outcome has been achieved.
• AKA outcome indicator, performance measure
• Examples
• Number of days absent (due to asthma-related
  illness)a number
• Percent reduction in days absenta percent
• Completed strategic plana milestone (yes/no)

14
Definition of Terms

• Target or Benchmark Specifies how much change to
  how many participants in what time period.
• AKA performance standard, performance target or
  goal
• To establish a target/benchmark, your guideline
  should be your previous experience (baseline),
  experience of others, a standard in the field, or
  your best estimate.
• Examples
• 20 (days absent baseline previous year was 30
  days)
• 33 (reduction in absenteeism from 30 to 20
  days)
• True/yes (strategic plan completed)

15
Definition of Terms

• Baseline This is your starting point or previous
  experience with this indicator. If you have no
  previous experience, you can use the experience
  of others or what research suggests.
• The target is often set in relation to this
  baseline, in which case the results will be
  compared to the target to the baseline.
• Examples
• Grades at entry in the program
• Pre-test scores on life skills inventory
• Last years participation rate or attendance

16
Definition of Terms

• Conditions or Limitations The criteria for
  inclusion in the group that further describes or
  defines the population or project being
  evaluated. It may also include the typical size
  of the group or expected dates of completion.
• Examples
• Includes all clients provided services during the
  program year (estimated 2,000)
• Includes only children with asthma enrolled for
  at least 6 months
• Includes all women over 18 years with at least
  one well-woman visit

17
Summary Examples

• Program Asthma case management
• Outcome Reduce school absenteeism
• Indicator Percent reduction in days absent
• Target/benchmark 33 (to 20 days)
• Baseline 30 days last year
• Conditions Children grades 1-3 enrolled 6 months
• Program Capacity-building
• Outcome Develop a strategic plan
• Indicator Completed strategic plan
• Target/benchmark True/Yes
• Baseline Consultant to be hired by 9/30/09
• Conditions Plan to be completed by 3/31/10

18
Define Goals Objectives

• Tasks
• State long-term goal of the program (3years)
• State short-term objectives (12-36 months
  knowledge, attitudes, behavior)
• Select important changeable personal external
  determinants (re at-risk group behavior
  environmental conditions)

19
Chain of Outcomes(Innovation Network)
20
Define Goals Objectives

• ABCDE Method Who, what, to what degree when
• Audiencethe population/target audience for whom
  the desired outcome is intended
• Behaviorwhat is to happen a clear statement of
  the behavior change/result expected (outcome
  objective)
• Conditionby when the conditions under which
  measurements will be made--may be a timeframe or
  upon implementation of a specific intervention
• Degreeby how much the level of results expected
  (ie, target or benchmark), which often involves
  measuring change in comparison to a baseline
• Evidenceas measured by defines the method of
  measuring the change expected (specific
  instrument or criterion) (indicator measure)

21
Define Goals Objectives

• Objectives should be SMART Specific, Measurable,
  Achievable, Realistic, Time Sensitive
• Example 1 In 12 months, there will be a 50
  reduction in unintended pregnancies among 17-18
  year old girls who live in Richmond City (
  participate in the project). Realistic?
  Achievable?
• Example 2 In 12 months, 75 of teen program
  participants will demonstrate increased knowledge
  about healthy sexual behavior, contraceptive
  practices prevention of sexually transmitted
  diseases.

22
Logic Models Evaluation Plans

• See Worksheets Examples!

23
Additional Resources

• Healthy People 2010 (www.healthypeople.gov)
• Provides targets for various conditions
• HRSA Health Disparities Collaborative
  (www.healthdisparities.net)
• Includes national core measures for
• Prevention Cancer Depression
• Asthma Diabetes Hypertension, etc.
• Va. Free Clinic Organizational Quality Assessment
  Tool (V1 2006)
• Program Services, Part IV. Program Evaluation,
  p.19

24
Thank You Have FUN!

• Barbara H. Dunn, PhD, RN
• Consultant, Health Human Services
• Email bhdunn114_at_comcast.net
• Phone [phone]
• Fax [phone]