Measuring Success in Community Involvement

1
Measuring Success in Community Involvement

• Information and Tools You Can Use
• EPA 2003 Community Involvement Conference and
  Training
• Philadelphia, PA
• July 24, 2003

2
EPAs 7 Basic Steps for Public Involvement

• Plan and budget for involvement
• Identify interested and affected parties
• Consider technical/financial assistance
• Provide information and outreach to public
• Conduct involvement activities
• Use input and provide feedback to public
• Evaluate involvement activities

3
Implementation Plan for Evaluation Activities

• Baseline assessment and follow-up
• Public involvement feedback tools
• Public Involvement Policy tracking table
• Regular sharing of lessons learned
• Project specific/program evaluations for public
  involvement efforts

4
Baseline Assessment of Public Involvement at EPA
Purposes

• To establish a baseline of current practices
  against which future public involvement efforts
  can be measured.
• To get practitioner perspective on the
• -- quality of public involvement efforts
• -- impact of public involvement efforts
• -- obstacles to effective public involvement

5
Baseline Assessment of Public Involvement at
EPA Approach

• Done March 2002 to February 2003
• Survey asked 52 questions
• Surveyed 39 projects from across Agency
• 19 regulatory projects
• 12 site specific projects
• 8 program development projects
• Work done by Steve Garon of SRA

6
Results Project Team Experience
7
Results Correlation of Team Training and Outcome
The numbers shown are based on a rating scale
from 1 being the least impact to 6 being the most
impact.
8
Results Planning and Budgeting

• 36 of projects had PI plan
• 38 of projects had budget
• Public Help Getting Outcome
• Plan? Input Impact
• Yes 4.95 5.19
• No 4.93
  3.50

9
Results How Do We Identify Affected Parties?

• 95 used print or broadcast notices
• 74 asked for names from others
• 38 reviewed dockets and publications
• 15 used approaches based on project factors

10
Results Providing Technical and Financial
Assistance

• Reg SS PD
• Agency/Contractor 21 67 37
• University 5
  33 13
• Travel Funding 5
  42 25
• Information Provided 5 58 13
• Other Support 16
  50 25

11
Results Providing Information and Outreach
12
Results Conducting Involvement Activities
13
Results Obstacles to Involvement

• Internal EPA factors
• Lack of resources
• Scheduling and timing issues
• Unreasonable public expectations
• Multiple and competing stakeholder interests

14
Results Use Input and Provide Feedback to Public

• 78 of respondents said that public input
  influenced the final decision
• In 79 of the projects, feedback was given to the
  public
• 41 used public meetings
• 36 used responsiveness summaries
• 26 used web sites

15
Results Evaluation Involvement Efforts

• A formal assessment of some PI activity was done
  at 41 of the projects
• An overall evaluation was done on 28 of the
  projects
• 11 of regulatory projects
• 50 of site specific projects
• 38 of program development projects

16
Results Impact of Involvement Efforts

• Increase Affect
• Involvement Outcome
• Regulatory 4.76 5.06
• Site Specific 4.60 4.50
• Program Development 5.14 4.64

17
Conclusions

• Good PI does make a difference
• PI success does not happen by accident,
  thoughtful planning is essential
• Use adaptive responses to project situation,
  dont just follow checklist of routine PI
  activities
• Need more management commitment/support
• Improve staff capacity for PI
• Have realistic expectations

18
Measuring Success in Community Involvement

• Information and Tools You Can Use Part Two
• EPA 2003 Community Involvement Conference and
  Training
• Philadelphia, PA
• July 24, 2003

19
Implementation Plan for Evaluation Activities

• Baseline assessment and follow-up
• Public involvement feedback tools
• Public Involvement Policy tracking table
• Regular sharing of lessons learned
• Project-specific/program evaluation for public
  involvement efforts

20
PI Feedback Tools Whats Included?

• User guide
• Ready-to-use questionnaires (pending
  OMB approval)
• Data input analysis spreadsheet program

21
PI Feedback Tools Who Developed Them?

• EPA Public Involvement Policy (PIP) Evaluation
  Task Group
• Cross-office/regional group
• Consulted with other staff involved in PI
  activities
• Currently testing refining feedback tools

22
PI Feedback Tools Purpose

• Enable EPA staff and managers to
• Understand minimum expectations of performance
• More easily assess effectiveness
• Enhance efforts
• Enhance trust between EPA and public
• Improve quality of Agency decisions

23
PI Feedback Tools Purpose

• Enable EPAs PIP Evaluation Task Group to
• Better understand what is working in regards to
  specific types of PI activities
• Better understand if improvements are taking
  place
• More easily share lessons learned

24
PI Feedback Tools Types of Questionnaires
25
PI Feedback Tools Listening Session
Questionnaire
26
PI Feedback Tools Listening Session
Questionnaire
27
Data Input AnalysisGetting Started
28
Data Input AnalysisEntering the Data
29
Data Input AnalysisReviewing Results
30
Data Input AnalysisReporting Results
31
Data Input AnalysisAssessing Overall
Performance for Multiple Events
32
Data Input AnalysisComparing Performance from
Event to Event
33
Data Input AnalysisReporting Overall
Performance for Multiple Events
34
Beyond Data Input AnalysisTaking Action

• Talk with managers/staff about effectiveness of
  activity and where changes may be necessary
• Communicate overall results to management and
  participants and any changes you intend to make
  based upon results

35
PI Feedback ToolsConclusion

• PI feedback tools will be available on EPA Public
  Involvement Intranet in 04
• Must get questionnaire approval from OMB
• Still need volunteers to test feedback tools
• EPA PI staff, managers, and public are keys to
  success!

36
PI Feedback ToolsFor More Information

• Contact PI Policy Evaluation Task Group
• Pat Bonner
• 202-566-2204
• bonner.patricia_at_epa.gov
• Bruce Engelbert
• 703-603-8711
• engelbert.bruce_at_epa.gov
• Shelley Fudge
• 202-566-0674
• fudge.shelley_at_epa.gov
• Eric Marsh (through mid-August 03)
• 202-566-2198
• marsh.eric_at_epa.gov