Measuring Success

1
Measuring Success

• The Center on Philanthropy
• Third Sector Conference
• February 28, 2009

RON ANCRUM
2
Key Drivers

• To communicate impact
• To make better decisions programmatically
• Manage priorities
• Meet donor expectations

3
Defining Evaluation

• A systematic assessment research procedures
  using quantitative and/or qualitative data
• Operation studying the process
• Outcome of program practice, lessons learned
• Standard of comparison against expectations or
  set of benchmarks
• Contribution to the improvement of program focus
  and policy

4
Impact or ROI

• Often times what the donor seeks is different
  from what the nonprofit strives for
• Real impact can take years to develop and be
  measurable
• Marketing investment is about the perception for
  public relations purpose
• Does it all comes down to numbers?
• First step in the process is to define what
  success looks like

5
Program Theory of Change

• Useful to know not only what the program is
  expected to achieve, but also how it expects to
  achieve it (Weiss)
• Asks the question why or how was it successful or
  unsuccessful.
• What are the set of beliefs, assumptions, chain
  of activities leading to anticipated outcomes?

6
What to measure?

• What are the indicators for measuring success for
  the following?
• After school academic program as compared to
  youth recreational program
• Environmental advocacy initiative as compared to
  public policy health research center
• Job training program compared to community arts
  program for seniors
• Conference event as compared to membership
  association

7
Process

• Develop a logic model or other evaluation
  strategy
• Must commit the resources both funds and people
  up front
• Ensure the right metrics are identified
• Develop or adapt tools for collecting
  data/information
• Conducting proper and unbiased analysis
• Clarify how findings will be shared and used

8
Logic Model A Planning Tool

• Mission
• Input
• Systems Integration
• Activities
• Outputs
• Outcomes
• Systems Outcomes

9
Issues with the Design

• Validity does it really measure what you
  intended can the data be trusted
• Relevant is the data/information critical to
  understanding success or just interesting
• Source of data factual and complete
• Complexity
• Interpretation of findings/conclusions

10
Mission-Related Measurements

• Changed behavior
• Knowledge, awareness, action by constituents
• Program impact a decrease in problem/issue or
  increase in positive outcomes
• Participation, community change, quantifiable
  benchmarks
• Operations impact
• Efficiencies, effectiveness, capacity

11
Case Study

• A small nonprofit organization is planning a new
  program to teach financial literacy to single
  working mothers between the ages of 18-24. The
  goal is to help them understand household
  budgeting, banking products, credit cards. The
  purpose is to help this target group of consumers
  make better financial decisions and achieve
  economic self-sufficiency. The research shows
  that this group of women over extend their
  credit, are not home owners, and have little or
  no savings that negatively affects their
  livelihood and that of their children. The
  program teaches how to save and why, how to
  create a household budget, and building positive
  relationships with financial institutions. A
  local bank is interested in funding this project
  and want to know how the nonprofit will measure
  success and produce a return on the investment.
  How would you measure success?

12
Tools used for measuring

• Focus groups and interviews
• Data from independent sources income, economic,
  crime statistics, etc.
• Intake exit forms
• Observations site visits
• Review of documents
• Surveys
• Research on issue

13
Challenges

• Cost
• Expertise
• Time
• Commitment
• Asking the right questions

14
Reading Materials

• Weiss, Carol. Evaluation, Prentice Hall, second
  edition
• Mark, Melvin Henry, Gary Julnes, George.
  Evaluation An Integrated Framework for
  Understanding, Guiding, and Improving Public and
  Nonprofit Policies and Programs, Jossey-Bass.