Survey of Benchmarking Methodologies

1

• Survey of Benchmarking Methodologies
• Maria Luisa Corton
• (in collaboration with Sanford Berg and the PURC
  Team)
• Public Utility Research Center
• University of Florida
• www.purc.ufl.edu

2
Survey Funded by World Bank

• Executive Summary Available
• Portions of Larger Report have been incorporated
  into Methodology sections of
• http//www.ib-net.org/index.php
• IBNET The International Benchmarking Network for
  Water and Sanitation Utilities
• Additional PURC Studies are forthcoming for use
  in ADERASA programs

3
Importance of Making Comparisons

• Performance comparisons are necessary but not
  sufficient for sound policy
• Benchmarking represents an important tool for
• Documenting past performance,
• Establishing baselines for gauging improvements,
• Making comparisons across service providers and
  over time, and
• Designing incentives.

4
Objectives of the Survey

• Bridge the gap between technical researchers and
  practitioners currently conducting studies for
  government agencies and water utilities.
• Encourage the application of more sophisticated
  quantitative tools to promote policies that
  improve company (and sector) performance.
• Provide rigorous tools that allow stakeholders to
  quantify utility progress towards meeting policy
  objectives,
• Help water specialists identify high performing
  utilities (whose production processes might be
  adopted by others)

5
Elements/Steps of Benchmarking

• Identify the problem What is going to be
  measured?
• Select (Preliminary) Methodology
• Collect Data
• Determine Estimation procedure
• Analyze results/sensitivity tests
• Communicate with targeted audiences
• Develop policy implications

6
Benchmarking Methodologies

• Performance indicators (partial)
• Total methods
• Parametric stochastic and non stochastic
• Non-parametric DEA
• Engineering approach
• Process Benchmarking
• Customer Survey Benchmarking

7
Chart Inputs, Processes, Outcomes, and
Performance Benchmarking
8
1. Performance Indicators

• Performance Indicators (Partial methods)
• Comparisons against best performers. Partial
  measures provide the simplest way to perform
  comparisons trends direct attention to potential
  problem areas, with data generally available from
  company annual reports.
• water delivered per worker, quality of service
  (continuity, water quality, complaints),
  unaccounted for water, coverage, and key
  financial data (operating expenses relative to
  total revenues, collections).

9
2.Total Methods

• Comparisons against a frontier. Based on the
  analysis of production patterns and/or cost
  structures.
• Production functions requires data on inputs
  and one output.
• Cost functions based on outputs and input
  prices.
• Distance Functions allow for multiple outputs in
  a production structure setting

10
2. Total Methods Parametric

• Parametric Identify relationship between firm
  performance, market conditions and
  characteristics of the production processes
  through functional forms
• Non Stochastic difference between firms and
  frontier is pure inefficiency (comprised by
  factors such as the effort of the producer and
  defective or damaged output)
• Stochastic difference between firms and frontier
  recognizes presence of noise (favorable as well
  as unfavorable external events such as luck,
  climate, machine performance and errors of
  observation and measurement of the variables
  comprising the estimated function)

11
2. Total Methods Non-Parametric

• Non Parametric No assumptions made about
  functional form
• Data Envelopment Analysis
• Method benchmarks firms only against the best
  performing producers
• Allows for multi-input, multi-output calculation
  of scores by means of linear programming
  techniques
• Can test for returns to scale.

12
3.Engineering/Model Company

• Requires the development of an optimized economic
  and engineering model
• Idealized benchmark specific to each
  utilityincorporating the topology, customer
  demand patterns, and density of the service
  territory.
• Artificial firm has optimized its network
  design and minimized its operating costs
• Production relationships can be obscured through
  a set of assumed coefficients used in the
  optimization process.
• Chile and Argentina have used this approach to
  establish infrastructure performance targets.

13
4.Process Benchmarking

• Focuses on individual production processes
• Detailed examination of facilities and their
  operations
• Identifies stages of the production process
  needing attention pumping up, intake, transport,
  clarification and filtration, purification and
  treatment.
• Studies of distribution processes (network
  design, pipeline construction and maintenance),
  sales processes (meter reading, data processing,
  billing, collections, and customer relations),
  and general processes (planning, staff
  recruitment and retention, and public relations).
  
• Provides a mechanism for identifying potential
  benchmarking partners, undertaking benchmarking
  visits, and implementing best practices

14
5.Customer Survey Benchmarking

• Customer Complaints one indicator
• SERVQUAL identifies five dimensions of service
  quality as perceived by customers
• External characteristics (tidy workplace,
  employee appearances),
• Reliability (meeting deadlines, consistency in
  interactions),
• Responsiveness (providing service promptly),
• Consideration (personnel who are courteous,
  friendly, and helpful),
• Empathy (giving individual care and attention).

15
Total methods Model Specification

• One output vs. multiple outputs
• Data availability prices or quantities
• Role of time
• Functional form
• Recognizing heterogeneity
• Recognizing noise

16
Total methods Estimation procedures

• Single Year observations
• OLS estimator
• COLS estimator
• Observations over Time
• Panel data
• Fixed effects dummy or within estimators
• Random effects GLS estimator
• Pooled observations
• Frontiers

17
Frontier Analysis
Output (Q)
Each X represents data for a water utility
Frontier From DEA
x
x
x
x
x
One observation far from the frontier!
x
x
x
x
x
Input
18
Efficiency Analysis
Input X2
Firm A
Firm B
P1
D
Firm C
P2
O
Input X1
19
Relative Inefficiency
Frontier
Frontier with time data
20
Examples of Model Specifications
Translog
Cobb-Douglas
Distance
21
Cost Example Ofwat Water Service Model (Y
Opex)

• Operating expenditures (less exceptionals,
  rates, third party services, abstraction
  charges, pumping costs)
• Ln Y 3.57 0.471 Ln X1 0.468 Ln X2
• - 1.575 Ln X3
• X1 water delivered in Ml/day
• X2 length of main in km
• X3 proportion of water delivered to measured
  non-households (Chaplin, United
  Utilities)

22
Consistency/Sensitivity Tests

• Accuracy and robustness of inefficiency
  estimates are important due to financial or
  social impacts.
• Cost Function vs. Production Function
• Functional form (linear, nonlinear)
• Outputs and inputs (e.g., network length vs.
  fixed assets)
• Alternative methodologies (e.g., DEA vs. SFA).
• Need to check whether estimated inefficiency
  scores or rankings are sensitive to the
  benchmarking method.

23
Develop Policy Implications

• Explore in greater detail the potential
  determinants of inefficiencies across firms and
  over time.
• Firms should not be ranked as poor performers if
  they operate under conditions that differ from
  those of the other firms.
• Identify the impact of factors like region,
  population density, regulatory environment,
  ownership structure, and network vintage.
• Seek public comments.
• Design incentives to improve performance.

24
Creating Appropriate Yardsticks

• Regulators want to induce outcomes comparable to
  those achieved under competition.
• Reward outstanding performance
• Penalize weak performance
• Benchmarking provides Yardsticks
• A note of caution
• Results can be misinterpreted and misused. The
  stakes are high, since affected parties have an
  interest in the relative and absolute performance
  comparisons prepared by analysts.

25
Summing Up

• Rankings can serve as catalysts for better
  stewardship of water and other resources.
• If regulators cannot identify historical trends,
  determine todays baseline performance, and
  quantify relative performance across utilities,
  then as an Indian regulator said, they may as
  well be writing pretty poetry.
• Report will be posted at www.purc.ufl.edu