Dia 1

1
Lecture 3 19.11.2004 Public programme
evaluation Introduction and logic behind of
quantitative methods Takis Venetoklis
2

• Lecture contents
• Link to previous and later lectures
• Data collection methods
• Data types
• Evaluation designs

3

• Data collection methods
• Literature search
• Review of administrative files/records
• Surveying via questionnaires/interviews
• Case studies
• Expert opinions
• Ready-made adminstrative data

4

• Literature research
• Always relevant with any type of evaluation
• Necessary to get a better idea of what others
  have done
• Methods they used
• Overall picture of process of program evaluated
• Legal aspects of program

5

• Review of administrative files/records
• Revealing and helpful data gathering method if
  existing data not very relevant to program
  evaluation objectives
• Example
• Process evaluation of business subsidies in
  Finland (Venetoklis 1999)
• 479 applications files of firms applyimg for
  subsidies read coded and analysed.

6

• Surveying via questionnaires/interviews
• Popular method
• Could produce biased (not acurate-wrong)
  responses
• Example of questions which can produce biased
  responses
• Program Subsidies to firms
• Respondents Owners/GM of firms
• Impact indicators measured via these questions
• a. Job creation/sustianed,
• b. Investment creation,
• c. Sales growth

7

• Actual questions asked by public officials
• Would you have made the investment had you not
  received the aid?
• What has been the real impact of the subsidy
  received, in terms of turnover growth in your
  firm?
• How many new jobs have been created because of
  this investment?
• How many jobs have been saved?
• Do you think that the turnover of your firm has
  grown due to the subsidy received/project
  invested (choose one)
• more than otherwise ?
• the same ?
• less than other wise ?
• Why could above questions produce biased
  answers?
• Potential dependency between respondent and
  interviewer
• Complexity of estimating true impacts problems

8
Another example of evaluation applying rather
biased data gathering methods

• Thematic evaluation of Structural impacts on SMEs
• Commissioned by (financed) European Commission
  (!)
• Evaluator Ernst and Young 1998-1999
• Aim
• Provide a thorough and systematic analysis of the
  contribution and impact of Structural Funds
  support to SMEs
• Draw up recommendations for future investment by
  Structural Funds in support of SMEs in the
  assisted regions based on the experience of past
  and current interventions.
• Gathering data method Interviews (telephone
  surveys)
• Who was targeted SMEs from 14 EU-countries FIN
  inclusive that got business subsidies (via the
  structural funds) between 1996-1998.
• No of obs 805 firms getting business subsidies,
  267 firms not getting subsidies, plus 90 case
  studies were done on projects financed with
  Structural Funds

9

• Questions asked dealt with
• Importance of Structural Funds assistance to the
  firms development
• Impact of Structural funds to the Firms
  performance and growth prospects
• Responses gathered for firms development
• Project would not have proceeded at all w/o
  assistance
• Project would have gone ahead w/o assistance, but
  delayed and/or modified
• Structural Funds aid made no difference to the
  SMEs plans and the firms would have
  proceeded with the project anyway.
• Responses gathered for firms performance and
  growth prospects
• Results indicated a positive impact

10
Overall results ..overall it is clear that
Structural Fund interventions have had a
significant impact on the SME sector and have
made an important contribution to wider regional
aim during the 1994-1999 period, around 2
million net jobs were created or saved as a
result of Structural Fund assistance to SMEs in
the absence of Structural Fund assistance, 70 of
SMEs said they would not have gone ahead with
their project or that it would have been
delayed/modified But the paper warns
also Relying on beneficiary feedback to assess
the extent of additionality demonstrated by
Structural Funds interventions in favour of SMEs
is clearly not ideal from a methodological point
of view. firms that claimed that the assistance
was fully additional could clearly be influenced
by an intention to apply for further aid.
...drawbacks of a survey-based approach to
assessing additionality are well knownbut
equally, alternative (econometric) methods are
not always possible to apply and would have not
been so in this study
11

• Case studies
• Assess program results through in-depth, rather
  than broad, coverage of specific cases or
  projects.
• It is vertical rather than horizontal gathering
  effort of data relevant to the program evaluated
• Specific cases are examined hoping to make
  inferences for the wholepopulation covered by
  (exposed to) the program.
• Selection is crucial (which cases to examine?)
• Difficulties in making generalisations because of
  small no of obs
• Time consuming and can be expensive per case.
• But allow in-depth analysis, impossible with
  other methods
• Can be complementary to other horizontal
  gathering data methods

12

• Expert opinions
• Utilise perceptions and knowledge of experts on
  parts of program evaluation
• It is just another type of survey but with less
  bias involved in answers because of less
  dependency problems.
• However estimation of impacts even by experts is
  still problematic (biased)unless estimated at a
  very broad and vague level.
• Results gathered are of course subjective.
• Delphi method is sometimes used to pass
  information around experts and re
  evaluate/estimate impacts topic examined after
  all share each others knowledge
• Best suited as a supplementary data gathering
  method
• Never should be used as the sole data source for
  program evaluation

13

• Ready-made (gathered) administrative (observed)
  data
• If existing data is
• relevant to program evaluated,
• already gathered by administrative/implementing
  authorities
• available (i.e. access not restricted due to
  confidentiality legislature), etc)
• then administrative data is probably the best way
  of gathering data
• Benefits
• No tedious phases as with previous methods
  (interviews, high costs, time consuming problems,
  limited access to sources
• Called also secondary (objective) data has an
  indirect characteristic thus less biased
• Survey data is called primary (subjective) data
  directly from the source thus more biased
• Can give the general picture, thus inferences for
  the general population, hence the program
  evaluated, can be made

14

• Problems with administrative data
• Poor data contents
• Missing data, wrongly inputted data, irrelevant
  data
• Inconsistent data
• Changes in data due to program implementation
  changes
• (i.e. exogenous changes due to new accounting
  systems in firms makes
• the maintenance of certain accounts say, for
  time series analysis very tedious)
• Difficulties in actually assessing data
• Legislature obstacles not to conflict with
  private confidentiality laws, not to breach
  competition laws, etc.

15
Qualitative data analysed quantitatively Can
use text classification software such as NUDIST.
Relevant URLs http//www.indiana.edu/statmath
/other/nudist/index.html http//www.qsrinternati
onal.com/index.htm
16

• Data types
• In quantitative evaluations one gathers and
  utilises four types of data
• Cross sectional data
• Consists of a sample of individuals, households ,
  firms cities states, countries or a variety of
  other units taken at a given point in time.
• Time series data
• Consists of observations on a variable or several
  variables of interest over time.
• Pooled cross sectional data
• Data sets that have both cross-sectional and time
  series features
• Panel ( Longitudinal) data
• Consist of a time series for EACH CROSS-SECTIONAL
  member of the data set.

17
Cross-sectional data set
18
Time series data
19
Panel (longitudinal) data (A)
20
Panel (longitudinal) data (B)
21

• Evaluation designs/data structures
• Here we concentrate on panel data. We follow the
  same units through time. We gather for variables
  of interest in different regular time intervals
• One group before after design, cross sectional
• Two groups before after design, cross
  sectional Difference in Difference DiD
• One group before after design, with trends
• Two groups before after design, with trends
• Difference in Difference DiD
• The methods applied depend on the data structure
  (design) at hand

22

• Example
• P (P)rofit (policy impact)
• S (S)ubsidies (policy intervention)
• T (T)REATed group
• C NON-TREATed group
• Data structure 1
• One group before after design, cross sectional
• PT t-1 STt PT t1
• Program applied horizontally (all units exposed
  to TREATment)
• Assumes temporal homogeneity
• Can not account for trends (historical behaviour
  of units of interest)
• Rather weak design

23

• Data structure 2
• Two groups before after design, cross sectional
  Difference in Difference DiD
  PT t-1 STt PT t1
• PC t-1 PC t1
• Program applied vertically only some (selected)
  units are exposed to TREAment
• some other units are not exposed)
• Counterfactual is created from units not exposed
  to Treatment
• Does not assume temporal homogeneity exogenous
  FIXED factors
• cancel out with DiD
• Can not account for trends (historical behaviour
  of units of interest)

24

• Data structure 3
• One group before after design, with trends
• PT t-3, PT t-2, PT t-1 STt PT t1, PT t2, PT
  t3
• Historical data very important (can reveal
  trends)
• Program applied horizontally
• Assumes temporal homogeneity
• Weak design but better than Design 1

25

• Data structure 4
• Two groups before after design, with trends
  Difference in Difference DiD
• PT t-3, PT t-2, PT t-1 STt PT t1, PT t2, PT
  t3
• PC t-3, PC t-2, PC t-1 PC t1, PC t2, PC t3
• Historical data very important (can reveal
  trends)
• Program applied vertically
• Counterfactual is created from units not exposed
  to Treatment
• Does not assume temporal homogeneity exogenous
  FIXED factors
• cancel out with DiD
• Best design of the four results in best
  estimations of the true program
• effect
• Designs can be enhanced with matching
  techniques
• and regression models