Composite indicators as models

1
Composite indicators as models

• Stefano Tarantola

2
Outline

• Models and composite indicators
• Craftmanship of model building
• The multi-dimensional nature of composite
  indicators
• Robustness and sensitivity analyses
• An example using the e-business index

3
Composite indicators as models
Model
Output
Input
Aggregation
Selection of indicators
Ind. 1
Ind. 1
Composite indicator
Analyst Expert group Policy makers
Ind. 2
Ind. 2
Ind. k
Ind. k
On rankings
Selection of weights
Equal weights, Based on statistical models (PCA,
regression, unobserved components) Participatory
approaches (budget allocation, AHP)
4

• Models are mathematical constructs
• Models are inspired by systems (natural, social)
  that one wishes to understand
• the modeller tries to encode the real system into
  a formal system

5
and the critique of models (Rosen)
Building a model (encoding) is the result of a
craftsmanship Interpreting the results of the
model (decoding) is also a craftsmanship
6

• The modeller has a perceived reality, another
  modeller has another.
• The encoding process depends on the different
  perceived realities
• One particular encoding process was never agreed
  among modellers

7

• The formalisation of the system generates an
  image, the theoretical framework.
• This is valid only within a given information
  space

8

• As a result the model of the system will reflect
  not only the characteristics of the real system
  but also the choices made by the scientists on
  how they observe the reality
• When building a model to describe a real world
  phenomenon, formal coherence is necessary yet not
  sufficient

9

• the observer and the observation are not
  separated the way human kind approaches the
  problem is part of the problem itself (Gough et
  al., 1998)
• No matter how subjective and imprecise the
  theoretical framework is, it implies the
  recognition of the multi-dimensional nature of
  the phenomenon that we try to measure (eg
  concepts like welfare, sustainability, etc.)

10

• all models are wrong, some are useful
• Box (1979).
• The quality of a composite indicator is in its
  fitness for purpose.

11

• Lets take the example of sustainability
• Defining sustainability within a
  multi-dimensional framework entails merging
  multi-disciplinary points of view
• All equally legitimate opinions of what
  sustainability is and how should be measured

12

• Conflicts may appear at multi-scale level change
  in scale may produce contradictory implications
• Conflicts about sustainability at national level
  may disappear at local level where other aspects
  prevail.

13
Need for robustness analysis
Leamer, 1990 (economist) I propose a form
of organised sensitivity analysis in which a
neighborhood of alternative assumptions is
selected and the corresponding interval of
inferences is identified.
14
Let us set the frame
Conclusions are judged to be sturdy only if the
neighborhood of assumptions is wide enough to be
credible and the corresponding interval of
inferences is narrow enough to be useful.
Space of the inferences
Edward E. Leamer, 1990 Sensitivity Analysis
would help, in Modelling Economic Series, Edited
by CWJ Granger, Clarendon Press, Oxford. Chair in
Man. Bus. Econ., UCLA
15
Robustness assessment (scheme)
16
Uncertainty analysis Mapping assumptions onto
inferences Sensitivity analysis The reverse
process
Simplified diagram - fixed model
17
Sensitivity analysis

A
B
Space of the assumptions
Output uncertainty
18
Sources for SA book (2000), ? primer (2004)
? ?free software
19
Settings for the sensitivity analysis
To validate or invalidate assessments
GSA used to show that the uncertainty in the
decision on whether to burn or dispose solid
waste depends on the choice of the system of
indicators, and not on the quality of the
available data. Money should not be spent to
improve quality in data, but to reach a
consensus on the proper system of indicators.
Tarantola et al., in Saltelli et al. Eds, (2000)
Sensitivity Analysis John Wiley
V(Y)VE(YXi)EV(YXi)

20
Settings for the sensitivity analysis
Problem simplification and dialogue optimisation
We look for those uncertain factors that have
negligible influence on the output. These can be
fixed to the most plausible value within their
range of variation. The dimensionality of the
input space is then reduced.
Useless discussing about the use of different
architectures to build the composite indicator,
when these do not influence the result.
21
Settings for the sensitivity analysis
Output uncertainty reduction
Joint use of UA and GSA (iterative procedure).
Perform UA and get a confidence interval for the
output
If it is unacceptably large, acquire better
knowledge on the most important factors. Perform
UA again to check ...
It the output quality exceeds the requirements,
the specifications on the input quality can be
relaxed, starting from the less important
factors.
Crosetto and Tarantola (2001) Int J Geogr Inf
Science
22

• An example the e-business index 2004
• The composite indicator aims to monitor country
  progress in the implementation of the e-Europe
  2005 Action Plan...

23
(No Transcript)
24
Propagation of uncertainty
Model
Output
E-business readiness index
xi indicators
UA independently on each country
25
The data set
Enterprise surveys in 2002 (22 missing values)
26
Weighting procedure
The e-Business Support Network (e-BSN) has
participated to the procedure of weights
assignment. Budget allocation exercise carried
out on 12 experts.
27
Budget Allocation
allocate 100 points
An example
Expert ...
A2 firms having www
A3 of firms with 2 security facilities
A4 employees using PC
A1 firms using Internet
A5 firms with broadband conne.
15 15 10 30 30
Expert 1
25 25 20 15 15
Expert 2
Expert 3
... ... ... ... ...
Normally the average weight across the experts is
used
28
Weights obtained from representatives of the
e-BSN.
29
Weights obtained from representatives of the
e-BSN.
30
Uncertainty Analysis (distributions)
31
Uncertainty analysis (box plots)
32
Sensitivity Analysis
Small fraction of uncertainty comes from
imputation procedure.
Large fraction comes from the choice of weights.
For all countries
91 97 ? weights
33
Sensitivity Analysis
Total indices for a few imputed variables are
below 0.01
Spending resources collecting data for such
variables would not improve accuracy of the
composite indicator.