European Conference on Quality in Survey Statistics

1
European Conference on Quality in Survey
Statistics
2
Composite indicators Data Envelopment Analysis

• Q2006Cardiff, April 25 Nicky Rogge(Catholic
  University of Leuven)

3
Structure

• Composite indicators useful but subject to
  controversy
• Normalization and weighting issue
• Background why DEA?
• Incorporating (expert) opinion(sub-indicator
  share restrictions)

4
Examples

• Human Development Index (HDI)
• Technology Achievement Index (TAI)
• Environmental Sustainability Index (ESI)
• Misery Index (MI)

5
Composite Indicators

• Indicators are observed (measured in different
  units)
• We want an overall score in order to compare
  countries
• but weights for aggregation are not known with
  certainty

• Useful tool for policy evaluation and
  communication
• Useful for comparing and ranking

However, still subject to controversy
6
The normalization issue
Common practice normalizing the sub-indicators
before aggregating.

• No consensus on suitable normalization method
• Results depend on normalization method

2 problems
A meaningful index is defined as an index whose
underlying preference ordering is independent of
admissible transformations of the variables
Ebert and Welsch (2004)
In practice, however, most indices and the
resulting ranks depend on the normalization
method
Potential point of criticism
Removing the necessity to normalize the original
data
7
The weighting issue
Ideally, individual indicators should be weighted
and combined in a manner reflecting the
underlying structure of the evaluated phenomenon.
However, insufficient knowledge on the underlying
structure
Often
Equal Weighting (EW)
Which weights to use? 2 issues

• Any predetermined common set of weights will
  favor some countries while harming others.
• The choice of weights may affect the composite
  indicators and their ranks, undermining their
  credibility.

Determine endogenously the optimal set of weights
8
Disagreement among experts proposed weights for
Technology Achievement Index
9
Classical DEA-problem

• Inputs and Outputs are observed (measured in
  different units)
• We want an overall expression of productivity in
  order to compare firms
• Productivity (weighted sum of outputs) /
  (weighted sum of inputs)
• but weights for aggregation are not known with
  certainty
• (production function, right prices, unknown)

Remarkable similarity between the problem of
measuringefficiency and the one of constructing
composite Indicators
10
Towards DEA composite indicatorStep 1 

• Weighted sum of indicators country total
• Since the eventual purpose of a composite
  indicator is to compare (with other countries),
  we will express this country total RELATIVE TO
  A SIMILARLY WEIGHTED SUM of BENCHMARK
  SUB-INDICATORS

11
Towards DEA composite indicatorStep 2 

• Which benchmark should we choose?
• Look WITHIN SAMPLE of COUNTRIES for the one that
  yields the HIGHEST POSSIBLE TOTAL (given the
  weights)
• Best Practice notion

12
The DEA composite indicatorStep 3
Benefit-of-the-doubt weighting

• Which weights should we choose?
• CHOOSE the weights such that the evaluated
  country has a MAXIMAL COMPOSITE INDICATOR VALUE
• Benefit-of-the-doubt-notion

13
Benefit-of-the-doubt-notion

• Weights? CHOOSE them (?0) such that the evaluated
  country has a maximal composite indicator value
• Benchmark ? Look within sample for country that
  maximizes CI, given these weights w c,i

14
Advantages

• Qua model better description of reality than
  (e.g.) equal weighting
• Embedded concern for (MS) diversity no other
  weighting scheme yields higher composite
  indicator value (political acceptance)
• Principle is easy to communicate
• Since we are not sure about the right weights, we
  look for  benefit of the doubt  weights (such
  that your overall relative performance index is
  as high as possible)

15

• We graphically represent sub indicators as pie
  shares with a bigger size reflecting a higher
  importance.

Finland 100.00
Poland 30.03
16
sub-indicator share restrictions

• Absolute sub-indicator share restrictions

• Relative sub-indicator share restrictions

• Proportional sub-indicator share restrictions

17

• For the TAI, for example, the lower and upper
  bound could be specified by respectively the
  lowest and highest weight assigned over all
  experts to that sub-indicator.

Netherlands 90.15
Norway 73.25
18

• Ordinal sub-indicator share restrictions

• The restriction imposes an importance ranking of
  the sub-indicators.

• Restrictions pertaining to category shares

Relative
Proportional

• Focusing on the importance of key dimensions of
  the evaluated phenomenon may allow a more swiftly
  expert consensus.

19
European Conference on Quality in Survey
Statistics