Module Two MDG Indicators

1
Module Two- MDG Indicators
United Nations Statistical Institute for Asia
the Pacific
United Nations Statistical Institute for Asia
the Pacific
Millennium Development Goals Initiative in Asia
and the Pacific
Millennium Development Goals Initiative in Asia
and the Pacific
(UNDP Project RAS/04/060)
(UNDP Project RAS/04/060)
Sub
-
regional Course on Statistics for MDG Indicators
Sub
-
regional Course on Statistics for MDG Indicators
Developing data and institutions for MDG
monitoring
Developing data and institutions for MDG
monitoring

• Statistical Framework

Prepared byMargarita F Guerrero (UNSIAP)
2
Aim of Module Two

• To increase knowledge on
• Concepts, measurement issues, data sources, data
  gaps, methods and rationale for generating MDG
  indicators
• To improve skills in applying methods for
  constructing MDG indicators and related measures
• At national and subnational levels
• By sex and other relevant analysis subgroups
• With the required consistency, timeliness and
  frequency and international comparability from
  existing data collection systems and databases

3
Topics Covered Statistical Framework

• Construction of Indicators
• Errors in Indicators
• Data Sources
• Criteria for Selection of MDG Indicators
• Statistical Challenges

4
Goals, Targets and Indicators

• Goals are the objectives that policy makers wish
  to achieve often expressed in qualitative terms
• e.g., eradicate extreme hunger achieve universal
  primary education
• Targets are the quantified and time bound levels
  of the indicators that a country aims to achieve
• e.g., Halve, between 1990 and 2015, the
  proportion of people who suffer from hunger
• Indicators are the variables used to measure
  progress toward the targets

5
(No Transcript)
6
Classification of Indicators
7
MDG Progress Reports

• Global (UN Secretary General)?
• Regional (UNESCAP SPC)?
• National (Countries)?
• Sub-national

8
A. Construction of Indicators

• Formulation
• Method of calculation
• Interpretation

9
Formulation

• Means
• Ratios
• Proportions
• Percentages
• Rates
• Quantiles

MDG Indicators are in one of these forms.
10
1. Mean or Average

• Simple Mean
• Example HDI is simple average of three dimension
  indexes
• Weighted Mean
• Example Education dimension index of the HDI is
  a weighted average of the literacy index
  (weight2) and the gross enrolment ratio index
  (weight1)

11
2. Ratios- Definition

• Division of two numbers which are both measured
  in the same units
• Compares like quantities
• Result has no units (unitless)
• Example MDG i9 Ratio of girls to boys in
  primary, secondary and tertiary education

12
Ratios- Interpretation (1)

• If this ratio is consistent across the whole
  country and if enrolment reflects actual
  attendance, then there are slightly more boys
  than girls in school
• But, also look at relative number (ratio) of boys
  and girls of primary school age
• Example Ratio of 0.98 vs ratio of 1.05

13
Ratios- Interpretation (2)

• When using ratios, a difference between values
  over two time periods can be due to--
• Change in the numerator
• Change in denominator
• Change in both numerator and denominator
• Example If ratio of girls to boys enrolment
  increases from 0.98 to 0.99. Is this due to an
  increase in enrolment of girls?

14
Example Increase in girls enrolment?
MDG i9 0.98
MDG i9 0.99
More girls, same number of boys
Same number of girls, fewer boys
More girls, fewer boys
15
3. Proportions

• When the ratio takes the form of a part divided
  by the whole, it is called a proportion
• Example
• Rural population as a proportion of total
  population
• Urban population 1 889 622
• Rural population 14 782 083
• Total population 16 671 745
• 14,782,083/16,671,705 0.89

16
4. Percentages

• To express a proportion as a percentage, multiply
  it by 100
• So, (1889622/16671705) X 100 11 of the
  population live in urban areas

17
5. Rates

• When the numerator and denominator of a quotient
  do not have the same units, but are related in
  some other way, the result is a rate.
• For rare events we usually multiply this
  quotient by 100 or 1000 and express rates as per
  100 or per 1000.

Example Infant mortality rate Number of live
births 866,929 Number of infant deaths
105,765 IMR 1,000105,765/866,929 122 deaths
per 1000 births
18
6. Quantiles

• Quantiles divide a set of numbers into groups,
  each group consisting of roughly equal number of
  values
• Three quantiles divide a set of numbers into four
  groups
• There are four quintiles.
• There are 99 percentiles.

19
Quantiles Example

• Find the tertiles of the following numbers
• 9,6,2,14,8,15,7,3,14,11,12,5,10,1,17,12,13,8
• First, put the 18 values in order from smallest
  to largest, and then divide them into groups of
  size 6.
• 1,2,3,5,6,7,8,8,9,10,11,12,12,13,14,14,15,17
• Here, t1 7.5 and t2 12

20
Quintiles Example

• Find the quintiles of the numbers
• 9,6,2,14,8,15,7,3,14,11,12,5,10,1,17,12,13,8,7,18
  
• First, put the 20 values in order from smallest
  to largest, and then divide them into groups of
  size 4.
• 1,2,3,5,6,7,7,8,8,9,10,11,12,12,13,14,14,15,1
  7,18

21

• Estimate household consumption.
• Divide by number of people in household ? per
  capita consumption.
• Rank households by value of per capita
  consumption.
• Find the first quintile, Q1, of per capita
  consumption values.
• Aggregate consumption of all households in
  poorest quintile i.e., with per capita
  consumption less than Q1.
• i3 ratio of consumption of poorest quintile to
  total consumption

22
Example
23
B. Errors in Indicators

• MDG indicators are subject to error
• Computation error
• Bias error
• Sampling error

24
Where Does Error Come From?

• MDG indicators are derived from data
• Data represent the population from which they
  were collected
• Any shortfall in the data collection and handling
  system cause error in the MDG indicators (MDGi)

25
1. Computation Error

• Errors made in the calculation of the MDGi, or
  its components
• Purely due to avoidable mistakes
• Less likely when calculation is automated

26
2. Bias Error

• Estimating the wrong thing
• Wrong Population
• Wrong Time Period
• Almost always a big issue when administrative
  data are used in deriving the MDGi in developing
  countries-- undercoverage
• Measurement errors in surveys/census
  non-sampling errors

27
Wrong Population

• In many cases, bias arises because we obtain data
  from a population that is not the one we really
  should be using, called the target population

Example vital registration Target population
all deaths Population used urban areas
28
Wrong Time Period

• Base year is 1990 but no data for base year so
  use proxy value like value for year nearest 1990

29
Common Sources of Bias

• Errors in sampling frame
• Purposive selection selection
• Non-response
• Response errors
• Recording errors
• Deliberate false answers
• Memory recall

30
3. Sampling Error (SE)

• Difference between a sample and the population
  from which it was derived
• Always present when sample survey data are used
  to derive the MDGi
• Not an issue with administrative data (unless
  these are only collected from a sample)
• Not an issue with a census

31
Estimates of Sampling Error

• Design-based estimates
• Estimator
• Sample design
• Examples of measures of sampling error
• Standard error of a sample proportion calculated
  from a simple random sample
• Coefficient of variation of a sample proportion
  calculated from a two-stage stratified sample

32
Reporting SE Confidence Interval

• Means of reporting sampling error based on
  estimated standard error of the MDGi
• 95 percent Confidence Interval (CI)
• 95 in 100 of all possible 95 CIs (each one
  calculated from a different sample) will contain
  the true value of the MDGi

33
Confidence Interval Example

• Sample of 4200, 4700, 4500, 7000 has
• a mean of 5100
• a standard error of 524
• a 95 CI of
• 5100 ? (3.18)524 or
• 5100 ? 1666 or
• 3434 to 6766
• We are 95 confident that the population mean
  lies between 3434 and 6766

34
ExampleMaternal mortality ratio(with 95
confidence intervals)
Is the observed decrease in MMR from 1995 to 2000
statistically significant?
Is the observed decrease in MMR from 1990 to 1995
statistically significant?
35
Root Mean Square Error
The total error, sampling and bias combined, is
measured by the root mean square error, (RMSE)
36
Error and Sample Size

• Larger sample size have lower sampling errors
• But
• Tend to have higher bias errors

big sample
small sample
census
37
How can we minimise error?

• Calculation error may be avoided by careful
  arithmetic or appropriate use of software.
• Sampling error is unavoidable whenever sample
  survey data are used
• Use a larger sample size
• Use a better sample design (e.g. stratified)
• Bias error is often present, not always obvious
• Be more careful in survey administration (e.g.
  minimize non-response)
• Increase coverage of administrative data
• Use statistical models to average over time
  periods/countries etc. (e.g. Brass methods for
  infant mortality, MDGi14)

38
How should we treat error?

• Quantify all errors, if we can
• Usually, only possible for sampling error
• Treat small differences in MDGis with scepticism
  
• May be explainable by error
• Document errors through use of metadata
• Acknowledge errors but consider timing and
  audience- be sure this does not cause confusion
  or lead to lack of trust

39
C. Data Sources for MDGi
40
Typical Sources of Data

• Administrative records
• Education statistics
• Health statistics
• Civil registration systems (CRS)
• Birth rates
• Death rates
• Censuses
• Sample surveys
• International sources
• FAO, WHO/UNAIDS, UNESCO, WB, ITU

41
Comparison of Data Sources
42
Which Source to Use for MDGi?

• Consider --
• National ownership
• Periodicity and timeliness
• Quality
• Comparability across space and time

43
D. MDG Indicators

• Bases for Selection
• Why these 48 indicators?

44
MDG Indicators Criteria of Selection

• Provide relevant and robust measures of progress
  towards the targets of the Millennium Development
  Goals.
• Be clear and straightforward to interpret and
  provide a basis for international comparison.
• Be broadly consistent with other global lists and
  avoid imposing an unnecessary burden on country
  teams, governments and other partners.
• Be based to the greatest extent possible on
  international standards, recommendations and best
  practices.
• Be constructed from well-established data
  sources, be quantifiable and be consistent to
  enable measurement over time.

45
Understanding the MDG Indicators

• Simple operational definition
• Goal and target it addresses
• Rationale for use of the indicator
• Method of computation.
• Sources of data
• References, including relevant international Web
  sites
• Periodicity of measurement
• Gender and disaggregation issues
• Limitations of the indicator
• National and international agencies involved in
  the collection, compilation or dissemination of
  the data.

46
E. Statistical Challenges

• Read Topic 2.1.4 (Module Two Training Notes)
• Review your Country Report on Status of MDG
  Statistics (the one you were asked to prepare and
  bring with you)
• Revise this Country Report, as follows
• Reflect the issues discussed in Topic 2.1.4
• Complete Excel Worksheet on Summary of
  Statistical Challenges