Philadelphia

1
How to Measure and Evaluate the Impact of Your
Community Involvement Strategy
U.S. EPA 2003 Community Involvement Conference
and Training
Philadelphia July 2003
This document is confidential and is intended
solely for the use and information of the
audience to whom it is presented.
2
Presentation goals

• Clearly articulate measurement and evaluations
  critical role in implementing community
  involvement strategy.
• Provide practical guidance for applying
  measurement and evaluation to community
  involvement.
• Foster learning through exercises in designing
  strategy maps and developing appropriate
  measures.
• Handout Presentation Assessment Form

3
Three-Part Organization

• Why Measurement and Evaluation are Essential to
  CI Strategy
• Case study U.S. Air Force Real Property
  Agency/Division Central - Kelly
• Brainstorming Measures and Applying Principles

4

• Part One
• Why are Measurement and Evaluation Essential to
  Community Involvement Strategy?

5
The Performance Imperative

• PMA expectations
• Many agencies and programs lack rigorous data or
  evaluations to show that they work. Such evidence
  should be a prerequisite to continued funding.
• Presidents FY04 budget The burden is on the
  program to demonstrate performance achievement
  should determine which programs survive, and
  which do not.
• GPRA requirements
• Measurable goals, objectives
• Results-oriented performance planning and annual
  reporting
• Handout OMB EPA scorecard

6
The Performance Imperative (contd)

• Program effectiveness fulfillment of mission
  (technical mission) meaningful CI rather than
  CI for the sake of CI
• EPA Effective Public Involvement
• Plan and budget
• Identify whom to involve
• Consider providing assistance
• Provide information
• Conduct involvement
• Review, use input and provide feedback
• Evaluate public involvement
• Public accountability

7
Demonstration of Value

• Challenges to the value (or relevance) of CI
• Low program expectations (compared with technical
  objectives)
• Community involvement is a popular scapegoat for
  bad news, an easy target for budget cuts
• Real (hidden) value of CI is its strategic
  reinforcement of technical objectives (caution
  double-edged sword)

8
Guidance for Applying Varying Evaluation Designs
(contd)

• Recognize the critical role of perceptions,
  communications, and rewards in the performance
  measurement cycle

What matters gets measured
What gets measured gets done
What gets rewarded matters
What gets done gets rewarded
Adapted from The Performance Imperative (Risher
and Fay, 1995)
9
Overview Standard Approaches, Tools and Tactics

• General measurement models and their relevance to
  community involvement
• Different levels of performance measurement and
  their purpose
• A toolbox of measurement tools and processes
• Guidance for applying varying evaluation designs

10
General measurement models or frameworks

• Sink Tuttle (process-oriented input/output
  model logic model)
• Balanced Scorecard (family of measures)
• GPRA (to assist compliance with PL 103-62)
• NPR (validated through public/private sector
  review)

11
Sink Tuttle (1989)
Transformation Processes
Suppliers
Output
Input
Customers

• Seven performance criteria
• Effectiveness
• Efficiency
• Quality
• Productivity
• Quality of work life
• Innovation
• Budget-ability

12
Balanced Scorecard (Kaplan and Norton, 1990)
Customer perspective (how do customers see us?)
Results perspective (Are we achieving our
critical success factors?)
Internal business perspective (What must we excel
at?)
Innovation and learning perspective (Are we
learning, creating value?)
Financial perspective (how do we look to
stakeholders?)
13
Government Performance and Results Act (1993)
Processes (activities)
Mission (legislative mandate/ customers/
stakeholders)
Strategies
Strategic goals
Process indicators (productivity, efficiency,
quality)
Outcome indicators (how do we make a difference/
satisfy customers / stakeholders?)
Output indicators (quality)
Efficiency
Effectiveness
14
National Performance Review (1997)
Customer and stakeholder input Management
priorities and decisions Congressional priorities
and decisions
Performance reporting to customers and
stakeholders
Customer-driven strategic planning?Multi-year
goal-setting and resource planning?Annual
performance planning?Resource allocation
Establishing and updating performance measures
and goals
Measuring performance (data collection and
reporting)
Establishing accountability for performance
Analyzing and reviewing performance data
Evaluating and utilizing performance information
15
Different Levels of Performance Measurement

• Program/organization level visibility
• cost, schedule, quality, risks, resources,
  improvement
• Operational unit level plan/control
• cost, schedule, quality, risks, trade-offs
• Workforce level improvement
• quality, productivity, products, processes
• Customer level tracking
• cost, schedule, quality, risks

16
Toolbox of Measures, Tools and Processes

• Categories of Human Measures
• Observations
• Perceptions
• Attitudes
• Opinions
• Behaviors
• Social factors
• Personal factors

• Measurement Tools and Processes
• Surveys
• Questionnaires
• Focus groups
• Interviews
• Content analysis
• Performance testing
• Multi-source feedback
• Personality assessments
• Process observations

• Performance Measures Characteristics
• Leading
• Lagging
• Quantitative
• Qualitative

• Categories of Organizational Measures
• Innovation/ learning
• Productivity
• Efficiency (outputs divided by inputs)
• Quality
• Quality of work life
• Stakeholder satisfaction
• Effectiveness
• Financial/ accountability

17
Program Evaluation Design

• Experimental
• random assignment of program or intervention
• pre-test and post-test control and treatment
  groups
• most rigorous cause and effect
• least feasible very costly
• Quasi-experimental
• non-random comparison group
• interrupted time series with intervention as
  discontinuity
• Descriptive
• numerical descriptions of target groups only
  profiling

18
Guidance for Applying Measurement and Evaluation
Designs

• Avoid the three most common mistakes leadership
  makes in implementing measurement and evaluation
  
• Focusing only on short-term results (lagging
  indicators not leading indicators)
• Being unwilling to break down functional barriers
  to communication and coordination
• Viewing performance improvement as a program or
  department, not as everybodys job

Adapted from The Performance Imperative (Risher
and Fay, 1995)
19

• Part Three
• Brainstorming Measures andApplying Principles

20
How to Define Appropriate Measures

• Group discussion of measures, measurement
  approach
• Tips on most appropriate measures and methods

21
Actual Measures Lessons Learned

• Actual measures, findings (lesson start small)
• Prerequisites
• Implications

22
Applying Principles of Strategic Impact

• Principles
• Alignment of individual actions to performance
  objectives
• Building on existing approaches, mechanisms to
  enhance culture
• Creation of a strategic, targeted and measured
  communication approach
• Demonstrating a tangible impact on operational
  efficiency
• Conclusions

23
Questions
24
References

• D.S. Sink and T.C. Tuttle (1989). Planning and
  Measurement in Your Organization of the Future.
  Industrial Engineering and Management Press,
  Norcross, VA.
• Robert S. Kaplan and David P. Norton (2001). The
  Strategy-Focused Organization. Harvard Business
  School Press, Boston, Massachusetts
• Risher and Fay (1995). The Performance
  Imperative.Publisher information not available
• The National Performance Review Federal
  Benchmarking Consortium Study Report (1997).
  Serving the American Public Best Practices in
  Customer-Driven Strategic Planning. Government
  Printing Office, Washington, D.C.

25
Presentation/Session Assessment
26
Presenter Contact Information

• George Angerbauer
• Booz Allen Hamilton
• 1309 R Continental Drive
• Abingdon, MD 21009
• 410 612 8203
• Angerbauer_george_at_bah.com
• Timothy Sueltenfuss
• Booz Allen Hamilton
• 700 North St. Marys Street, Suite 700
• San Antonio, TX 78205
• 210 925 3021
• sueltenfuss_timothy_at_bah.com