Social policy evaluation: concepts , methods and limits'

1
Social policy evaluation concepts , methods
and limits.

• Titre

2

Social policy evaluation concepts , methods
and limits.
1. Introduction Socio-political aspect of
evaluation concerns all of us beneficiary,
taxpayer, voter and decision maker Economic
aspects efficiency, costs and benefits analysis
QuestionWhat can be obtained at what cost and
at what probability Quantitative methods
Micro simulation ex ante and counterfactual ex
post reform valuation. Micro economic methods
ex post program evaluation Qualitative
methods (interviews, experts)

• Titre

3
Socio-political dimension beneficiary,
taxpayer, voter and decision makers

• Citizen Voter
• Taxpayer Beneficiary
• Parliament
• Government Administration and NGOs

4

Evaluation of social and fiscal policies
simulation and micro econometric methods

• Social policies an interactive process
• - Socio-economic needs
• labor market policies (unemployment -
  work incentive policies)
• social sector financing (retirement
  founding, social VAT, redistribution
  policies).
• - Reform projects (programs, expected results )
• - Ex ante expected effects evaluation.
• - Controlled experiments (pilot programs
  regions, populations)
• - Implementation (legal and administrative
  procedures, financing rules)
• - Ex post evaluation proportion of positive
  responses, result-expectation analysis, take up
  evaluation
• - Feedback (continuation, correction,
  abandonment)
• - Risks of political sanction (election)
  evaluation

5
Social policy evaluation concepts , methods and
limits.
Economic challenges of policy evaluation Costs
of implementation and costs of non implementation
!
6

Social policy evaluation concepts , methods and
limits.

• Evaluation criteria of programs and reforms
• Adequacy well defined and well targeted
• Efficiency economically rational equitably
  founded
• Efficacy fulfills the objectives

7

Social policy evaluation concepts , methods and
limits.

• Main difficulties
• The lack of the appropriate data
• statistics (surveys and administrtive files),
  experimental data (voluntary or natural
  experiments)
• The absence of the control group . (program
  participants versus non participants)
• The complexity and interdependence of different
  segments of socio-economic and tax-benefit
  systems sédimentation process and internal
  policy contradictions)
• Unobserved hetérogénéité of individual
  situations and individual behaviours

8
Social policy evaluation concepts , methods and
limits.Microsimulation methods

• Origin and interest Welfare state
  reforms in 1980ies (tax benefit systems,
  pension systems, redistribution rules
  modifications). It was highly political issue
  generating the need for independent control and
  monitoring (equity, redistribution problems)
• Opportunity the development of micro
  econometric methods and microeconomic data bases
  
• Micro - because based on the individual
  observations which allows the measure the program
  and reform impacts by modelling the individual
  behavior.
• Simulation - because many variants of changes in
  socio-economic systems rules can be
  simultaneously or sequentially introduced
  generating numerous predicted outputs in terms of
  new behaviors or new system states

9
Social and fiscal policies simulation and micro
econometric evaluation methods Microsimulation

• Micro-simulation model structure
• - Exogenoeus rule unit (tax benefit système
  income, consumption taxes, tax, contributions,
  benefits, with founding flows)
• - Individual data base (surveys,
  administrative data, with updating models, with
  transition probabilities, matching procedures)
• - Behavioural models individual reactions as a
  consequence of the (new) rules application
  (labour supply , consumption, tax evasion,
  informal market participation).

10
Social policy evaluation concepts , methods and
limits.Microsimulation

• The rules can be
• deterministic (tax burden)
• or stochastique (demographic events like
  marriages, births, divorces)
• The effects are measured in
• global level terms (tax expenditure for
  example)
• distributional terms (distribution of income
  effects).
• Frequently used indicators
• inequality (redistribution) measures
• equity
• work incentive measures (marginal tax rates)

11
Social policy evaluation concepts , methods and
limits.MicrosimulationData base

Database the essential element for micro
simulation. As exhaustive as possible on both
individual characteristics and socio-economic
environment description. The need for permanent
updating (time and coverage). This is somewhat
utopist postulate. No survey or administrative
file can provide all needed information. The
lacking information of interest can be completed
by indirect methods matching or imputation
(Family budgets survey with tax files for example
or imputing demographic events probabilities.
12
Evaluation des politiques sociales et fiscales
Modèles de microsimulation un outil daide à la
décision et dévaluation ex anteMicrosimulation
la base de données

13
Evaluation des politiques sociales et fiscales
Modèles de microsimulation un outil daide à la
décision et dévaluation ex anteMicrosimulation
Exemple imputationModèle INES (INSEE)

14
Social policy evaluation concepts , methods and
limits.Microsimulationtypology
Static models (no time dimension) Typical
household sets Artificially built different
socio-economic family structures (couple one
child, median income) to which the tax benefit
system rules are applied. Frustrating because
non representative they can give a good idea of
tax-benefit system interactions (effective
marginal tax rate (EMTR) for example), and more
generally threshold effects) Simple static
models (without behavioural adjustment) The
typical household set is replaced by an
individual data base. The policy impacts are
observed on representative sample of the total
population. The policy changes effects are
computed for every individual comparing the
situation after and before the policy change.
Typically the lost and gains in term of
disposable incomes can be computed for every
individual or for group of individuals (different
types of households, income groups). The results
is often given as a change in relative
individuals position in terms of well being or
income distribution.

15
Social policy evaluation concepts , methods and
limits.MicrosimulationSocial policy evaluation
concepts , methods and limits.Microsimulationtyp
ology

Static models (no time dimension) Static
Models with behavioral responses The individual
response behavioral model is added to the simple
static model (labour force supply model,
consumption behavior model). Then the obtained
individual behavioural parameters (elasticities)
are used to correct the results for the effect of
individual adaptation to the new situation.
Typically for the change in VAT taxation reform,
a Consumer Demand System is estimated and all
elasticities are derived.
16
Social policy evaluation concepts , methods and
limits.Microsimulationtypical household
analysis exemple(Tax reform effect on the
marriage versus partnership union after PPE ( low
wage workers allowance)

couple without children,, S. Guérin INSEE, Et.
Sociales
17
Social policy evaluation concepts , methods and
limits.Microsimulationtypology
Dynamic micro simulation models Similar
structure than those of static ones (typical
households, models with behavioral response,
models without behavioral response. The time
dimension is introduced with appropriate changes
especially as far as population evolution. The
data base individuals are moving in the time
(get married, have children , divorce, get a job,
become unemployed by associaton of the estimated
probabilities of all these events).
Simultanoeusly to these events the changing
socio-fiscal rules are applied over the life
cycle and ll outcomes are added.

18
Social policy evaluation concepts , methods and
limits.Microsimulationtypology
Dynamic typical household analysis typical
households situation over the life cycle Typical
households are multiplied in time dimensions
making their structure vary in the with respect
to their hypothetical life cycle events.
(Madinier, Sahut dIzarn, 1992). Dynamic models
with behavioral response, without feedback
first econometric simulation of data base
demographic evolution using the transition
probabilities of state changing (birth, marriage,
divorce, retirement, unemployment) second
simulation of life cycle income evolution
third the income effects different variants
of evolution of the tax benefit system, and
the labor market are estimated. Dynamic
models with behavioral response and with
"feed-back Inter temporal choice individual
model are added with possible response on the
change in socio-economic environment.

19
Social policy evaluation concepts , methods and
limits.MicrosimulationINES model
(INSEE)simulation examples
Scénario 1,2,3 Social security founding reforms
lowering work taxation by substituting health
contribution (paid as proportion of wages) by a
general, flat rate tax on all incomes.

20
Social policy evaluation concepts , methods and
limits.MicrosimulationMicrosimulation Model
(INSEE)simulation examples

21
Social policy evaluation concepts , methods and
limits.MicrosimulationINES model
(INSEE)simulation examples

Scénario 1 Suppression des cotisations
sociales maladie financée par CSG 4,5 points
et augmentation de limpôts par la non
déductibilité de la CSG
22
Social policy evaluation concepts , methods and
limits.MicrosimulationINES model
(INSEE)simulation examples
Tableau II-1-2 SCENARIO 1, variation en
rapportée au revenu déclaré initial du
prélèvement total selon la Cs du chef du ménage
et le type de famille

23
Evaluation des politiques sociales et fiscales
Modèles de microsimulation un outil daide à la
décision et dévaluation ex anteMicrosimulation
Modèle INES (INSEE)Exemples de simulations

24
Social policy evaluation concepts , methods and
limits.MicrosimulationINES model
(INSEE)simulation examples

25
Social policy evaluation concepts , methods and
limits.MicrosimulationINES model
(INSEE)simulation examples
Scenario 1- - by income level Advantageous for
low income increase in taxes, but lower
contributions, finally lower global tax
burden For 6-th-7th decile global tax burden
is unchanged . For _8th and higher deciles
reform is disadvantageous. Direct tax increase
is higher then the decrease in social
contribution. - by family type Large families
from lower social classe are beneficiaries of the
reform. Retired and selfemployed are loosing
independently on their family situation Middle
class professions improve their situation
proportionally to the family Globaly no change
in inequality, but another shape of redistribution

26
Social policy evaluation concepts , methods and
limits.MicrosimulationINES model
(INSEE)simulation examples

27
Social policy evaluation concepts , methods and
limits.MicrosimulationINES model
(INSEE)simulation examples

28
Social policy evaluation concepts , methods and
limits.MicrosimulationINES model
(INSEE)simulation examples

Conclusions are incomplete the lack of
essential response what impact on the
unemployment ? In order to answer that question
behavioral the model of labor supply is
necessary. The behavioral response model is
needed. In the case of employment the situation
is difficult many problems to obtain coherent
labor supply and labor demand
elasticities. Instead the example of
consumption.

29
Social policy evaluation concepts , methods and
limits.MicrosimulationINES model
(INSEE)simulation examplesBehavioural component
example value added tax reform

Reform projects  Social VAT, lowering VAT
as an incentive for legal employment in some
sectors hotel- restauration, construction. A
behavioural madel is needed to compute income
and price elasticities Rise in VATrise in
prices implying income and substitution effects.

30
Social policy evaluation concepts , methods and
limits.MicrosimulationINES model
(INSEE)simulation examplesBehavioural responses
VAT reform

Théoretical model Linear Expenditure System
(LES) With very well known direct and indirect
utility functions
31
Social polcy evaluation concepts , methods and
limits.MicrosimulationINES model
(INSEE)simulation examplesBehavioural responses
VAT reform

• Model estimated on grouped data from matched
  fiscal and consumption surveys

The demand system .
32
Social polcy evaluation concepts , methods and
limits.MicrosimulationINES model
(INSEE)simulation examplesBehavioural responses
VAT reform



.

33
Social polcy evaluation concepts , methods and
limits.MicrosimulationINES model
(INSEE)simulation examplesBehavioural responses
VAT reform


.

34
Social policy evaluation concepts , methods and
limits.MicrosimulationINES model
(INSEE)simulation examplesBehavioural responses
VAT reform


.

• Simulated reform unification of several VAT
  levels 1, 5.5 et 20.6 into one15
• Hypothesis
• the change of VAT is entirely integrated into
  retail prices
• The total expenditure remain unchanged only
  substitution effects, no change in saving
  behaviour
• tax revenue reamains constant.

35
Social policy evaluation concepts , methods and
limits.MicrosimulationINES model
(INSEE)simulation examplesBehavioural responses
VAT reform


.

36
Social policy evaluation concepts , methods and
limits.MicrosimulationINES model
(INSEE)simulation examplesBehavioural responses
VAT reform


.

37
Social policy evaluation concepts , methods and
limits.MicrosimulationINES model
(INSEE)simulation examplesBehavioural responses
VAT reform


.

Conclusion the behavioural effect introduction
did not change considerably the results despite
of rather large scale of the reform.
38
Social policy evaluation concepts , methods and
limits.MicrosimulationDESTINIE model
(INSEE)dynamic modelsimulation examples

Destinie microsimulation model was built to
analyse different scenarios in retirment
systems. Individual life trajectories are
simulated 50 years ahead Annual and individual
base- every year the probabilities of change in
individual demographic marriage, divorce,
situaitionsand economic (employment,
unemployment, wage evolution , retirment) are
estimated. (Remember in static models the
population does not change!) The fertility rate
is supposed to influence the retirment schemes.
Data base- Family History survey and Census
sample (250 000)
.
39
Social policy evaluation concepts , methods and
limits.MicrosimulationDESTINIE model
(INSEE)dynamic modelsimulation examples



.

40
Social policy evaluation concepts , methods and
limits.MicrosimulationDESTINIE model
(INSEE)dynamic modelsimulation examples



.
Essential estimated parameters Demographic
transition probabilities Labour market transition
probabilities (employment survey) Labour
participation probabilities evolution (especially
women) Wage evolution model (with individual
fixed effects, exogenous chocs and and
productivity evolution Wae dynamics in public
secotor Several specific surveys and
administrative files have to be used to obtain
these parameters.

41
Social policy evaluation concepts , methods and
limits.MicrosimulationDESTINIE model
(INSEE)dynamic modelsimulation examples, public
administration pension reform



.
2003 reform General objectif later retirment
for cicvil servants Longer working period shift
from 37,5 to 40 years for full pension. Bonus
malus system for longer or shorter working
period Minimum guarantee rules changed.

42
Social policy evaluation concepts , methods and
limits.MicrosimulationDESTINIE model
(INSEE)dynamic modelsimulation examples, public
administration pension reform



.
Effects on the number of retired Effects on
the total civil pension expenditure

43
Social policy evaluation concepts , methods and
limits.MicrosimulationDESTINIE model
(INSEE)dynamic modelsimulation examples, public
administration pension reform



.

44
Social policy evaluation concepts , methods and
limits.MicrosimulationDESTINIE model
(INSEE)dynamic modelsimulation examples, public
administration pension reform



.

45
Social policy evaluation concepts , methods and
limits.MicrosimulationDESTINIE model
(INSEE)dynamic modelsimulation examples, public
administration pension reform



.

46
Social policy evaluation concepts , methods and
limits.MicrosimulationEx ante evaluation final
remarks



.

47
Ex post policy and programs evaluation Methods
and measures for program participation and outputs




Question how to measure the program or reform
impact? The  output  of a program is often
defined in terms of additionnality
Difference between the outcomes under the new
program and the outcomes which would have
occurred without the program. Ex tax credit
for unemployed who would accept a low paid
job To what extent the decreasing unemployment
the effect of the program, or the economic
growth, demographic evolution. Problem how
distinguish between the two effects. The main
difficulty estimate the outcomes which would
have occurred without the program. This is
called the counterfactual. All ex post
evaluation methods try to estimate this figure
directly or indirectly, as an necessary element
to obtain the additionnality.

48
Ex post policy and programs evaluation Methods
and measures for program participation and outputs




More formally the programs output can be written
(Rubins model) Denoting Y the output
variable T the program participation ( 0 for
participation 1 for not participation) Y1 Y0 -
potential results - participation and not
participation respectively. They are never
observed simultaneously for the same individual.
He cannot be in two states at the same time. Y
T Y1 (1-T)Y0 In the case of participation
Y1, the Y0 represents the counterfactual Impac
t or additionnality I Y1 - Y0

49
Ex post policy and programs evaluation Methods
and measures for program participation and outputs




The problem of the counterfactual How to divide
the eligible population into treatment ( or
intervention group which participates in the
program), and control (or comparison group
which do not participate. This is essentially the
control group which is difficult to identify it
should be identical to treatment group except
from program participation. In reality, many
individual (often unobservable) factors influence
the decision to participate. Ex The monetary
incentive is not the only condition to
participate (others qualification, opportunity
costs (distance, domestic production, informal
markets availability) Thus, choosing a control
group for the counterfactual estimation is
exposed to a high risk of selectivity bias
Different methods to minimize that risk

50
Ex post policy and programs evaluation Methods
and measures for program participation and outputs

Different methods to minimise that
risk Randomised trials if not possible Quasi
experimental methods Before-after
design Difference in differences One-to one
matched comparison group design Matched area
comparison design Statistical micro econometric
modelin

51
Ex post policy and programs evaluation Methods
and measures for program participation and outputs

Randomised trials  Golden standard  Eligible
population is divided at random into two groups
a program (treatment) group and a control group .
(Like in medical research quasi single blind
experiment but not double or  triple  blind)
) Both group are balanced as far as all
charcteristcs which can influence the outcome
are concerned, except for program
participation. Advantages The only observed
differences are random differences and the
program impact Difficulties Moral the program
is refused to the control group Administrative
high administrative costs managing the selection
between participants and non participants (two
systems must be run) Conclusion very good
design but difficult to implement. Very rich
administrative data can facilitate the design (no
specific survey costs).

52
Ex post policy and programs evaluation Methods
and measures for program participation and
outputsQuasi experimental designs

Before-After design Used to evaluate programs
that are to be introduced nationally The
comparison group is drawn from the eligible
population before the program is implemented, The
program group is drawn from the eligible
population post- program implementation. The main
disadvantage of the before-after design is that
change, or additionality, due to the program can
not be separated from change that might naturally
occur between any two points in time. Different
factors can affect outcomes in different periods
(change in labor market conditions) Different
other programs interference possibility. Conclusio
n the most often used, but the identification of
particular programs effects can be
difficult. Can be improved, if a long time series
is available.

53
Ex post policy and programs evaluation Methods
and measures for program participation and
outputsQuasi experimental designs

Ex post micro-simulation A variant of the
previous method (before and after) is the use of
the micro simulation model (Constant population
structure analysis) A set of output indicators
is built (inequalities, poverty, unemployment,
effective marginal and average tax rate). The
introduced program are applied to the same
population sample generating the hypothetical
 after  outputs . The treatment and control
(comparison group) are the same and are compared
before and after program application. Problems
there is no control for population structure
evolution impact and the influence of economic
situation change
54
Ex post policy and programs evaluation Methods
and measures for program participation and
outputsQuasi experimental designs




Matched area comparison design First
selecting pilot areas to run a new program Second
these areas are matched to a set of control
areas (not necessarily on the one to one
basis The eligible population is followed up in
both areas and the outcomes compared Problem
controlling for observable or not observable
differences in areas Advantage no administrative
problems when running different programs.

55
Ex post policy and programs evaluation Methods
and measures for program participation and
outputsQuasi experimental designs

One to one matching methods (individuals,
groups, areas) Post program implementation
design Individuals or more often groups are
selected first among participants, Then the
 similar  ones are selected among non
participants. Similar means matching the
closest possible observable characteristics of
interest except from program participation. This
is one-to one matching i.e. to every
participating group (individual) another if
possible identical is associated from the non
participating population. Main problems the
weaknesses of matching methods especially when
many unobservable characteristics influencing
program participation or few common
characteristics between observations. .

56
Ex post policy and programs evaluation Methods
and measures for program participation and
outputsQuasi experimental designs


Différence in différences Two groups are compared
before and after program implementation Two
groups are selected from eligible population
participants (intervention group) non
participants in the program (control or
comparison group) Both groups are observed over
the time and outcomes of the variable of
interest (for example unemployment) are
calculated as differences  before  and
 after . For  non participants  natural
change is observed, For participants for
intervention group both  natural and program
impact change are observed. The second
difference between change in the treatment and
control group evolution gives the estimate of
program impact effect. main hypothesis natural
evolution is identical for both groups.

57
Ex post policy and programs evaluation Methods
and measures for program participation and outputs


Statistical modeling to estimate the
counterfactual It is a variant of one-to one
matching schemes associating an appropriate
control group (non-participant in the program) to
the treatment group (participants in the
program). If matching is difficult (usually it
is) it is interesting to get the non participant
group much larger then participants group
allowing a more precise reliable results when
estimating the counterfactual from a a larger
data set taking into account a large number of
potential candidate for match. Several matches
can be realized among non participants to
correspond to one participant observation.

58
Ex post policy and programs evaluation Methods
and measures for program participatStatistical
matching models and estimatorsi

• Matching on observable characteristics
• The estimation principle
• Use all information available on non all non
  participants to build for every participant a
  counterfactuel.
• Matching estimator on observable characteristics
  (Rubin 1977).
• The potential outcome for all non participants
  is estimated as a prediction based on the same
  characteristics and real outcomes of the variable
  of interest (unemployment) for program
  participants.
• The set of identical characteristics between
  participants and non participants can be
  difficult or even impossible to identify Then it
  can be replaced the closest possible individuals
  in sense of a defined distance mesure
  (Mahalanobis for exemple).
• The program result is as usual the average
  difference of outcomes between participant et
  non participants

59
Ex post policy and programs evaluation Methods
and measures for program participatStatistical
matching models and estimatorsi




Propensity score matching estimation The
estimation principle Probability of treatment
(participation) is estimated conditionally on the
observed individual characteristics. Then the
participants and non participants are matched on
the basis of the propensity score
proximity. Propensity score with kernel
weighting The basic idea is that every non
treated individual is participating in building
of the counterfactual of an treated individual
with the weight varying with respect to the
propensity score distance between both
individuals

60
Ex post policy and programs evaluation Methods
and measures for program participatStatistical
matching models and estimatorsi




The instrumental variable estimator The
estimation principle First find a variable
correlated with participation (treatment) but not
correlated with variables (observed and non
observed) related to outcomes Comparing outcomes
with this variable gives the information how
outcomes relate to participation and allows
additionnality estimation. Problem difficult to
find such a variable Propensity score is often
used. Heckman selection estimator Allows the
correlation of the instrument with outcome
equation errors by explicit estimation of both
(instrument and errors). Specification of that
relationships depends however on strong
hypothesis.

61
Ex post policy and programs evaluation Final
remarks

• What are the recommended evaluation schemes
• Both ex ante and ex post policy evaluations bring
  an information about programs reforms
  perspectives in terms of expected results.
• The reliability of this information depend on the
  use of the appropriate models but essentially on
  the quality of existing or created data sets
• Ex ante methods treat the general population
  effects, but need the development in the sense of
  macro-economic general equilibrium
• Ex post methods are adapted to treat rather small
  populations and suffer also from the lack of the
  general population or macro-economic feedback