Sampling issues in PIES and PETS

1
Sampling issues inPIES and PETS
2
Impact Evaluation Studies

• Need to compare two populations, differentiated
  by
• Time
• before and afterthe crisis,the boom,the
  earthquake,the program,
• Space
• North/South,rural/urban,mountain/forest,
• Policy
• program vs. non-programtreatment vs. control

3
The basic question

• Let y1 and y2 be the values of an indicator in
  populations 1 and 2
• Prevalence of malnutrition
• Average income
• Literacy
• etc
• We are interested in the difference y2 - y1
• y1 and y2 are both unknownWe estimate them from
  samples
• The estimations y1 and y2 are affected by
  sampling error
• What is the error of the difference?

4
The basic answer

• Let e1 and e2 be the standard errors of y1 and
  y2
• If the samples are independent, the standard
  error of y2 - y1 is

this is the case when the populations are
different
5
Panel Surveys can measure change better
y1
y2

2005 2005 2007 2007
It seems that y2 lt y1 but both measures are
affected by sampling errors (e1 et e2) The error
of the difference y2 - y1 is v (e²1 e²2) if
the two samples are independent only v(e²1
e²2 2?y1,y2) if the sample is the same
6
Advantages and disadvantagesof panels

• Analyticaladvantages
• Can measure changes better
• Permit understanding better why things changed
• Permits correlating past and present behavior

• Analyticaldisadvantages
• Become progressively less representative of the
  population

• Practicaldisadvantages
• Sample attrition
• Much harder to manage
• Vulnerable to manipulation
• Design them prospectively rather than in
  afterthought

• Practicaladvantages
• No sampling design needed for the second and
  subsequent surveys

7
For independent samples
State of the world
Decision rule
Null hypothesis H0 trueH0 y2 y1 0
Alternative hypothesis HA trueH0 y2 y1 ?
H0 accepted ify2 y1 D
OK
Type II error
P (correctly accepting H0) 1 - a
P (Type II error) ß
Researchfindingsy1 y2
H0 rejected ify2 y1 gt D
Type I error
OK
P (Type I error) a
P (correctly rejecting H0) 1 ß
With Simple Random Samples,e²1 s²1/n1 and e²2
s²2/n2 But real-life samples are almost never
SRS,design effects cannot be ignored !
The decision rule based on y2 y1 intends to
detect changes in one specific direction (a
1-sided test) To detect changes in any of the two
directions, it should be based on y2 y1
instead (a 2-sided test) But impact evaluation is
almost always one-sided !
8
Public Expenditure Tracking Studies

• PETS need to observe different kinds of units
• Large administrative areas (e.g., regions)
• Medium administrative areas (e.g. districts)
• Service delivery facilities (schools, hospital,
  clinics)
• Service providers (teachers, doctors, )
• Clients (students, patients, )
• Households
• Analyses need to account for their hierarchical
  relationships
• How should the samples be selected?

9
?
10
Two approaches

• Bottom-Up
• Take a sample of facilities first,
• This implicitly defines the sample of districts
• Which in turn defines the sample of regions

• Top-Down(A multi-stage sample)
• Select regions first
• Then districts in the selected regions
• Then Hospitals in the selected districts

• Disadvantages
• High design effects
• Poor understanding of the upper levels
• Vulnerable to unscientific choices

• Disadvantage
• Higher (slightly higher) costs

11
In each facility

• To select providers (teachers, doctors, nurses,
  etc.)
• The samples need to be random (perhaps
  stratified)
• The selection has to be entrusted to
  fieldworkers
• but it should be repeatable (for supervision)
• Can be done with Kish tables, or with random
  stickers
• To select clients in health facilities
• The selection also needs to be random and has to
  be entrusted to fieldworkers.
• It cannot be repeatable, but the procedures
  should try to avoid biases due to the
• Day of the week
• Time of the day