Evaluation: A Sneak Peek

1
Evaluation A Sneak Peek

• William McPeck
• Director, Employee Health and Safety
• Maine State Government
• March 19, 2008

2
Agenda

• Why just a sneak peek?
• Introduction to evaluation
• Applying evaluation to worksite wellness
• Introduction to evaluation resources

3
What is Evaluation?

• Program evaluation is the use of scientific
  methods to collect data, some with emphasis on
  the process of monitoring, and others with
  emphasis on the use of data to inform program
  managers and other key policy makers about how
  well a program is meeting its goals and
  objectives.
• CDC

4
Why do evaluations?

• The CDC emphasizes evaluation as a way to improve
  and account for public health actions using
  methods that are useful, feasible, proper and
  accurate.
• Identification/minimization of unintended
  consequences
• Achieving goals and objectives
• Strategies used are working smoothly
• Cost effectiveness/ROI
• Answering leaders/funders important questions
  and expectations
• Answering staffs important questions and
  expectations
• Way to keep program up to date and on target
• CDC

5
When Do You Conduct Evaluations?

• The CDC recommends that specific, systematic
  evaluations be carried out throughout the life
  span of a program, from program inception and
  planning to implementation, during delivery and
  re-design.
• CDC, MMWR, 1999

6
Things to Consider

• What types of data will you need?
• From what sources will the data come?
• How will you collect the data?
• In what format(s) will the data be collected?
• How will you store the data until you are ready
  to use it?
• What are your IT capabilities?
• How will you report out the data and your
  evaluation results?

7
Four Standards in Evaluation

• Useful
• Feasible
• Proper
• Accurate
• CDC

8
Evaluation Uses

• In addition to program self-evaluation,
  evaluation may be needed in other situations,
  such as
• Prioritizing activities and guiding resource
  allocation
• Informing funders about program effectiveness
• Informing community members and stakeholders
  about a projects value
• Providing information that can be useful in the
  design or improvement of similar projects
• CDC

9
The Scope of Evaluations

• Different programs require and situations require
  different evaluation strategies
• Program Development Stage
• Focus on the extent and severity of the issues to
  be addressed and designing effective
  interventions to address them
• On-going Program Stage
• Examination of program methods to improve
  effectiveness or decrease operational costs

10
The Scope of Evaluations (Cont)

• Program Expansion or Curtailment Stage
• Cost Benefit studies
• Cost Effectiveness studies
• Cost Benefit/Effectiveness study of alternative
  strategies
• CDC

11
Evaluation Plan

• Numerous designs and methods exist
• Consult with a researcher or program evaluator
  early on in the program planning stage
• Three main types of evaluation
• Process
• Impact/Output
• Outcome

12
Process Evaluation

• Used to assess the implementation of a program or
  specific intervention
• Looks at program structure, delivery and to
  gather estimates of the magnitude of the expected
  impact
• Verifies what the program is, whether the program
  is being delivered to the intended audience as
  intended and in the intended dosage, frequency,
  etc.
• Assesses program consistency and quality
• Assesses the relationships between program
  components and other programs within the
  organization
• Understanding how a program works

13
Sample of Process Evaluation Type Questions

• Is the program delivery process being implemented
  with consistency?
• Are staff implementing program services with a
  high level of skill and attention to protocol,
  etc. and in the manner in which they have been
  trained?
• How does implementation of the program affect
  other policies, procedures and protocols within
  the organization?
• Are program materials interesting, easy to
  understand, easy to use and engaging?

14
Impact/Output Evaluation

• Used to determine if the program is meeting its
  goals and objectives
• Used to assess the effect of the program
• Measure outputs
• Things that can be counted
• Contacts, rates, frequency, etc.

15
Sample of Impact/Output Evaluation Type Questions

• Does the program increase awareness?
• Does the program increase knowledge?
• What were the
• Rates of contact
• Number of sessions delivered
• Number of materials handed out?
• Rates of engagement

16
Outcomes Evaluation

• Assess whether the program is doing the right
  activities to reach the anticipated or needed
  outcomes
• Determine whether the desired/expected outcomes
  occurred
• Three levels of outcomes evaluation
• Preliminary
• Intermediate
• Long Term
• Outcomes evaluation can focus on
• Individuals
• Systems
• Organizations

17
Sample of Outcomes Evaluation Type Questions

• Does the program result in greater _______ over
  time?
• Does the program result in increased (decreased)
  rates of _________ over time?
• Does modifying ________ improve employee ________?

18
The Dearth of Evaluation

• Why isnt wellness program evaluation more
  commonly used?
• Perceived as being costly
• Perceived as requiring an evaluation expert
• Design
• Implementation
• Evaluations are time consuming
• Tangential to program operations
• CDC

19
Measuring Changes Through Worksite Wellness

• Understanding the impact of what you are doing
  is the key to figuring out what still needs to be
  done.
• Evaluating Interventions
• Are employees aware of the available
  interventions?
• Did they participate?
• Were they satisfied?
• What new areas of interest do they have?

20
Measuring Changes Through Worksite Wellness - 2

• Impact on Health and Productivity
• Have employees adopted healthier behaviors?
• Have health risk factors been reduced across the
  employee population?
• Do employees/managers report higher levels of
  productivity and/or reduced levels of
  presenteeism?

21
Measuring Changes Through Worksite Wellness - 3

• Impact on the Organization
• Economic
• Decreased healthcare, disability and workers
  compensation utilization
• Greater productivity
• Non-economic
• Recruitment and retention
• Morale
• Company image
• Adapted from Andrew Spaulding, Medical Care
  Development, March 2008

22
Who Can I Call?

• Maine universities and colleges
• Public health programs
• Business administration programs
• Centers and institutes
• Muskie Institute USM
• Center on Aging UMO
• Maine Center for Public health

23
Resources

• Web sites
• CDC Evaluation Working Group
• http//www.cdc.gov/eval/index.htm
• Community Tool Box University of Kansas
• http//ctb.ku.edu

24
Resources

• Books
• Measuring Employee Productivity A Guide to
  Self-Assessment Tools - The Gold Book, Institute
  of Health and Productivity Management
• Chapman, Larry. Program Evaluation A Key to
  Wellness Program Survival, 2nd Edition (Health
  Management Guides, Volume 6), Summex Health
  Management, 2006
• Chenoweth, David. Evaluating Worksite Health
  Promotion. Human Kinetics, 2002.

25
Contact Information

• William McPeck
• Certified Worksite Wellness Program Consultant
• Director, Employee Health and Safety
• Maine State Government
• Augusta
• 207-287-6783
• william.c.mcpeck_at_maine.gov