Environmental Protection Agency

1
Environmental Protection Agency

• Statistical Sampling Plans
• Collaborative Efforts between Statisticians
• and Subject Matter Experts
• by Marla Smith
• ICES-III
• June 2007

2
Overview

• Our program and its objective
• Surveys in our program
• Multi-disciplinary teams
• Survey development
• Lessons learned and examples

3
Milestones -- Environmental Protection

• 22 June 1969
• Cuyahoga River Fire (Cleveland, OH)
• Time Magazine Some river! Chocolate-brown,
  oily, bubbling with subsurface gases, it oozes
  rather than flows.

• 1970 President Nixon creates EPA
• 1972 Clean Water Act amendments shifts
• -- Reliance on violations of water and quality
  standards as the primary enforcement tool, to
• -- Establishment of specific technology-based
  effluent guidelines that are enforceable as
  permit conditions

4
What are Effluent Guidelines (EG)?

• Restrictions on the types and amounts of
  pollutants that can be discharged into navigable
  waters of the U.S. by various industries
• Based upon best available process and/or
  wastewater technologies and performance
• Separately implemented through permits

5
Why Do We Conduct Surveys?

• After collecting data from industry, EPA analyzes
  the data to identify or refine
• Technology Options and the related pollutant
  reductions and incremental compliance costs
• Economic Impacts
• Industry subsectors
• Environmental Benefits

Sampling, Industry Survey, etc.
Formulation of Options and Decision-making
Analysis of Data (including public comments)
Industry Identified
6
EG Surveys

• One-time surveys
• Duty to respond is mandatory under the Clean
  Water Act

7
EG Teams

• Each team has its own personality
• Statistician has to adapt accordingly
• Typical team includes
• Team Leader
• One or Two Engineers
• One Economist
• One Environment
• Assessor
• One Statistician
• Everyone supports multiple projects
• Some more part-time than others
• Each team member has contract resources
• Additional level of coordination
• Management and lawyers provide oversight

8
Survey Development EG Team

• Each project team expends considerable effort to
• Learn about each industry
• Determine the data needs for each type of
  analysis
• Refine study objectives
• Find a sample design that will satisfy all
  requirements
• Develop the survey instrument

9
Survey DevelopmentStatistician

• Leads survey design work
• Translates team objectives into specifications
  for statistics contractor
• Reviews statistics contractor products
• Often reviews multiple versions before sharing
  one with the team
• Discusses sample design concerns with management
• Documents and defends sample design as part of
  government approval process
• Reviews final draft of questionnaire

10
Examples from 3 Surveys

• Airport Deicing EG
• Airlines and airports perform deicing and
  anti-icing of aircraft and airfield pavement
• Can result in environmental impacts
• Survey Design includes two industries
• Airports Single phase
• Detailed questionnaire sent to stratified sample
• Airlines Two-phase
• Screener sent to census of relatively active
  airlines selected airports
• Detailed questionnaire sent to subset of deicers

11
Overview Examples (cont)

• Drinking Water EG
• Residuals are created when drinking water is
  treated. Sometimes the residuals are discharged
  back into the river, stream, etc.
• Survey Design Stratified sample
• Biosolids not an EG
• Biosolids result from the treatment of domestic
  sewage in a treatment facility (different than
  drinking water treatment)
• Survey Design Stratified sample
• Team aspects similar to EG

12
Lessons Learned General

• Listen
• Learn as much as possible
• Need to consider what isnt said
• Participate in non-survey meetings
• Ask questions
• Ask the same question at different meetings
• Keep asking until theres an answer
• Revisit and confirm previous decisions
• May get different answers at different times
• Leave the room last at a meeting
• Listen to what is (still) being discussed
• Allow time for questions about statistics and
  data
• Keep it simple
• Make sure everyone understands
• Bottom line is more important than how we got
  there

13
Lessons learned Target Pop.

• Concept can be difficult for non-statisticians
• Target population versus available data
• Not everyone needs or should get a questionnaire
• Sub-populations need to be defined
• Narrows down the target population
• Provides insight to study objectives
• Reduce eligible population
• Each pick means a loss of a pick elsewhere
• Verify that legal authority encompasses all
  members of the target population
• Decide if oversampling is appropriate
• Write down the definition
• Team needs to review and agree

14
Target Pop. Examples

• Airport Deicing
• Not everyone should get a questionnaire
• Military airports were excluded from survey
• Questionnaire not the right vehicle for
  collecting data
• Affordability questions are not relevant
• Still included in EG development
• Biosolids
• Target population definition
• Two alternatives facilities existing today or in
  1988
• Team knew a lot about the 1988 facilities
• Existing data indicated differences in population
  over time
• Management wanted estimates of current conditions
• Subpopulation review
• 0.2 percent of the facilities generate 32 percent
  of the total nationwide flows
• Team decided to oversample

15
Target Pop. Examples

• Drinking Water
• Target population definition
• Difficult to describe (series of criteria)
• Important during sample draw review
• Subpopulation considerations
• Excluded very small systems
• Approximately 98 percent of all 160,000 drinking
  water systems are very small
• Collectively, they have little environmental
  impact
• Exclusion greatly simplified sample frame issues
• Oversampled certain systems
• Team concluded they were most likely to have
  biggest impact in terms of engineering,
  economics, and environment assessment
• Ratios based upon engineering judgment

16
Lessons Learned Design

• Recognize that using a sample is a leap of faith
  for non-statisticians
• Sometimes need to involve management
• Estimate resources for different sample sizes
• Keep stratification under control
• Too many variables will complicate the analyses
  and unnecessarily increase sample sizes
• Accommodate favorite picks early on
• Team will reject sample draws without them
• Provides additional insight into study objectives
• Often can conclude these facilities are different
  enough that they can only represent themselves
• Prepare to develop multiple designs as the team
  goes through its thought process
• Keep management involved

17
Design Examples

• Airport Deicing
• Listen to whats not said
• Proposed design Stratified by four hub sizes
• Hub sizes readily available in sample frame
• But, team wasnt planning to stratify by size in
  any analyses
• Final design Stratified by two size categories
• This approach will simplify our data analyses
• Drinking Water
• Keep management involved
• Proposed two-phase design
• Management questioned sample size
• Final design
• One stage with reduced stratification
• Less detailed, shorter, questionnaire
• Keep stratification under control
• Removed one variable to reduce sample size

18
Lessons Learned Draw

• Include time in the schedule for a second draw
  and design tweak
• Team generally concludes that the first draw
  will never work
• Reasons are often valid
• Discussion may reveal additional study objectives
  not previously recognized by the team
• Design tweaks are generally very close to
  original design
• Statistical adjustments to responses are possible
• Team needs to compare analysis complications
  versus redrawing sample
• Also recognize that no draw will be perfect

19
Draw Examples

• Drinking Water No draw will be perfect
• Team wanted to reject certain systems based upon
  non-frame data
• Team retained the draw after further review
• Concerns about bias
• Judgment that statistically valid adjustments can
  be made later
• Strong management support for statistics
  arguments
• Biosolids Second draw was necessary
• Contact information not available for small
  facilities
• Tend to have part-time staffing
• Difficult to find person with the key
• Team redefined target population to exclude
  smalls
• Team able to move forward with the second draw

20
Conclusion

• Listening and asking questions is a critical part
  of developing survey designs
• Flexibility is important in redesigning and
  reevaluating to find a workable design