Estimating the Effects of Treatment on Outcomes with Confidence

1
Estimating the Effects of Treatment on Outcomes
with Confidence

• Sebastian Galiani
• Washington University in St. Louis

2
Parameters of Interest

• Two parameters of interest widely used in the
  literature
• Average Treatment Effect
• Average Treatment Effect on the Treated
• Under randomization and full compliance, they
  coincide.

3
Randomization

• In the absence of difficulties such as
  noncompliance or loss to follow up, assumptions
  play a minor role in randomized experiments, and
  no role at all in randomized tests of the
  hypothesis of no treatment effect.
• In contrast, inference in a nonrandomized
  experiment requires assumptions that are not at
  all innocuous.

4
Quasi-Experimental Designs

• If randomization is not feasible, we need to rely
  on quasi-experimental methods.
• In our case, the most promising strategy would be
  a Generalized Difference in Differences strategy.
  

5
Parameters of Interest

• We might want to response the following
  questions
• What is the effect of the intervention on a given
  outcome on a given population?
• What is the effect of the intervention on a given
  outcome on those that self-select as users of the
  facilities?
• The power for identifying the first parameters
  might be lower than the power for the second
  parameter identified by IV Methods.

6
Distance to Facilities and Sampling

• Think about stratifying the sample by distance to
  facilities, and over-sample households residing
  near facilities.

7
Testing absence of Treatment Effects. Type I Error

• Once we have chosen a Type I error rate a, the
  null hypothesis (pT- pC 0) is rejected whenever
  the statistics of contrast t gt ta/2 where
  ta/2 is the (critical) value of t that defines
  the a/2 percentile of the distribution of t.

No rejection
Reject Null
Reject Null
ta/2
-ta/2
0
8
Type II Error

• Now consider an alternative hypothesis pT- pC
  d
• Under this alternative hypothesis, the
  t-statistic will have a different distribution.
• If the alternative hypothesis is true, we want to
  reject the null hypothesis as often as possible.
• To fail to do so would be a Type II error.
• We want to restrict the probability of this type
  of error to b.
• Then b will be the type II error rate of the
  test.
• And 1-b will be the power of the statistical
  test.
• The power of a statistical hypothesis test
  measures the test's ability to reject the null
  hypothesis when it is actually false.

9
Type II Error
No rejection
Reject Null
Reject Null
Distribution of t under the null
Distribution of t under the alternative
Power 1-b
b
ta/2
0
-ta/2

• There is a trade-off between Type I and Type II
  errors

10
Type I and II Errors

• Both errors can be simultaneously reduced if the
  dispersion of the statistics is reduced.

No rejection
Reject Null
Reject Null
Distribution of t under the null
Distribution of t under the alternative
Power 1-b
b
ta/2
0
-ta/2
11
Simple Clinical Trial

• In this design m members are allocated to each
  condition treatment and control.
• The observed value to the i-th member in the l-th
  condition is a function of the grand mean and the
  effect of the l-th condition any difference
  between the observed and the predicted value is
  allocated to the residual error.
• The intervention effect is Cl. Its estimate is

12
Simple Clinical Trial

• Under the null hypothesis, H0 Cl 0, Let
  estimate the variance of .
• Assuming that the residual error is distributed
  Gaussian, the intervention effect is evaluated
  using a t-statistic with the appropriate df.
• The researcher determines the desired Type I and
  II error rates (say 5 and 20, respectively).
• The researcher expects a negative intervention
  effect but would be concerned about a positive
  effect as a result, she chooses a two-tailed
  test.

13
Simple Clinical Trial

• Given the random assignment of members to
  treatment and control conditions, it is
  reasonable to assume that the two study
  conditions are independent. Then
• The estimated variance of a single condition mean
  is
• If we assign the same number of members in each
  condition, the variances in the two conditions
  are assumed to be equal.

14
Simple Clinical Trial

• Then, the t-statistic is estimated as
• The parameters appearing in this formula are
  relatively easy to estimate using data from
  previous reports, from analyses of existing data
  or from preliminary studies.

15
Simple Clinical Trial
No rejection
Reject Null
Reject Null
H0
HA
Distribution of t under the null
Distribution of t under the alternative
Power 1-b
b
-ta/2-tb
ta/2
0
-ta/2

• True type I and II errors rates will be a and b
  respectively if

16
General Expression

• Or
• This expression is general to any design.
• We need
• Desired type I and II error rates.
• The expected magnitude of the treatment effect.
• The expression for the variance of the estimated
  treatment effect, which is a function of the
  sample size.
• We can express any of these variables in terms of
  the others.

17
Simple Clinical Trial

• Sample size

18
Sample Size in GRT

• Assume that there is only one individual per
  household.
• Probability that an individual has diarrhea
• Individual i, in group k, assigned to condition
  l.
• Within each condition the variance of any given
  observation is
• Where stands for the variance within groups
  and for the variance between groups

19
Sample Size in GRT

• Consider first the group mean.
• If that mean were based on m independent
  observations, the variance of that mean would be
  estimated as
• However, because the members within an
  identifiable group almost always show positive
  intra-class correlation, those observations are
  not independent.
• In fact, only the variance attributable to the
  individual effect will vanish as m increases. The
  variance attributable to the group effect will
  remain unaffected.

20
Sample Size

• Then, the variance of the group mean is
• Where, m stands for the number of households per
  group and ICC for the intra-group correlation.
• The variance of the condition mean is
• Where g is the number of groups in each condition

21
Sample Size

• When ICCgt0, the variance of the condition mean is
  always larger in a GRT than in a study based on
  random assignment of the same number of
  individuals to the study conditions.
• Statistic of interest
• Variance of the statistic

22
Sample Size

• Given a moderate number of groups per condition,
  the t-statistic to asses the difference between
  condition means is
• It is distributed t-student with gTgC-2 degrees
  of freedom

23
Sample Size

• Sample size
• Number of groups per condition
• Number of household per group (it requires a
  couple of iterations)

24
Sample Size

• When each household has more than one
  observation, we need to perform the following
  correction
• Where a is the number of observations per
  household and r is the intra-household
  correlation. See extreme cases r1 or 0.

25
Pretest - Posttest Repeat Observations on Groups

• Data are collected in each condition before and
  after the intervention has been delivered in the
  intervention condition.
• There are repeated observations on the same groups

26
Pretest - Posttest Repeat Observations on Groups

• The model
• The observed value for the i-th member nested
  within the k-th group and l-th condition and
  observed at the j-th time is expressed as a
  function of the grand mean, the effect of the
  l-th condition, the effect of the j-th time, the
  joint effect of condition and time, the realized
  value of the k-th group, the joint effect of
  group and time.
• Differences between this predicted value and the
  observed value are allocated to the residual error

27
Pretest- Posttest Repeat Observations on Groups

• Treatment effect Dif-in-dif
• There are two sources of variation among the
  groups
• Variation due to group effect
• Variation due to the interaction group x time
• The first difference eliminates the first source
  of variation.
• The group mean is
• This model can be easily transformed in the basic
  GRT

28
Pretest - Posttest Repeat Observations on Groups

• The variance of the group mean is
• Following the same steps as before
• The variance of the intervention effect can be
  written as
• Sample size can be solved as before

29
Pretest - Posttest Repeat Observations on Members

• Data are collected in each condition before and
  after the intervention has been delivered in the
  intervention condition.
• There are repeated observations on the same
  members

30
Pretest- Posttest Repeat Observations on Members

• The model
• The observed value for the i-th member nested
  within the k-th group and l-th condition and
  observed at the j-th time is expressed as a
  function of the grand mean, the effect of the
  l-th condition, the effect of the j-th time, the
  joint effect of condition and time, the realized
  value of the k-th group, the realized value of
  the i-th member, the joint effect of group and
  time and the joint effect of member and time.
• Differences between this predicted value and the
  observed value are allocated to the residual error

31
Pretest- Posttest Repeat Observations on Members

• Treatment effect Dif-in-dif
• There are three sources of variation among the
  members
• Variation due to member effect
• Variation due to the interaction member x time
• Error term
• The first difference eliminates the first source
  of variation.

32
Pretest- Posttest Repeat Observations on Members

• Taking differences by members
• This model can be easily transformed in the basic
  GRT
• The variance of the intervention effect can be
  written as
• Sample size can be solved as before