ETIC SESSION Oct 2002- MERIT L.Bach

1
Pecs Session / Week 2 - July 2007
Evaluation of STI policy performance and
impacts Laurent Bach bach_at_cournot.u-strasbg.fr
BETA, university Strasbourg
2
Pecs Session / Week 2 - July 2007
Evaluation of STI policy performance and impacts

• A few definitions
• The key question of additionality
• (Socio-economic) Effects and impacts
• Some examples
• New challenges for an "ideal" evaluation scheme ?

3
1. A few definitions
EVALUATION An evaluation is an assessment, as
systematic and objective as possible, of an
ongoing or completed project, programme or
policy, its design, implementation and results.
The aim is to determine the relevance and
fulfilment of objectives, development efficiency,
effectiveness, impact and sustainability. An
evaluation should provide information that is
credible and useful, enabling the incorporation
of lessons learned into the decision making
process of both recipients and donors. (OECD
1987) Analysis and assessment of goals,
instruments and impacts (Meyer -Krahmer /
Georghiou 1992)
4
EVALUATION OF ST POLICIES
Mechanisms / institutions (collaborative scheme,
funding mechanism, IPR regulation...)
Projects / programmes
Policy
( Evaluation of local / wider ST systems - Nat.
or Reg. SI )
5
Evaluation at different steps of the process of
public actions
6
(No Transcript)
7
THE DIFFERENT LEVELS OF EVALUATION
E1 - E4 Relevance (content) and "quality" of
conception (decision process) E5 - E7
Implementation and results E8 - E9 Efficiency
and legitimization
8
THE DIFFERENT LEVELS OF EVALUATION
Relevance (content) and "quality" of conception
(decision process) E1 Relevance of
objectives E2 Coherence of objectives E3
Relevance and coherence of the "institutional
arrangements" E4 Coherence between objectives
and institutional arrangements
9
THE DIFFERENT LEVELS OF EVALUATION
Implementation and results E5 Programme
management (cost-timing-quality) monitoring E6
Effects / outputs / impacts effectiveness E7
Match between effects and objectives efficacy
(1)
10
THE DIFFERENT LEVELS OF EVALUATION
Efficiency and legitimization E8 Match between
objectives - institutional arrangements and
effects-outputs-impacts do the same / do better
another way assessing adequate funding,
management, contractual behaviour in order for
objectives to be achieved in a cost-effective
manner efficiency
11
THE DIFFERENT LEVELS OF EVALUATION
Efficiency and legitimization E9 Ex-post
relevance of objectives, given the results of the
other evaluations assessing whether initial
objectives are still valid in the light of
evolving RTD, societal and environmental
conditions efficacy (2)
12
(No Transcript)
13
2. The key question of additionality
ADDITIONALITY and related concepts
What difference does State intervention (i.e.
policy) make ? ltgt What will happen (has
happened) over and above what will happen (would
have happened) anyway ? Does this difference
justify State intervention ? comparing
situation with policy vs situtation without
policy efficacy (1) with policy vs
objectives efficiency with policy vs with an
alternative policy aiming at similar objectives
opportunity cost with policy vs other use of
public funds
14
Different concept of additionality Without
policy alternative scenario null hypothesis
---gt before/after comparison counterfactual
scenario ---gt fictitious case control
group level of definition (policy, gvt,
actors,) Time scale
15
Policy Rationales, Evaluation Rationales and
Additionality
Neo-classical / standard approach (input-output
approach)
Market failures
Input additionality whether the public action
adds to, or substitutes for the agents inputs
(usually financial) crowding out effects,
displacement etc Output additionality whether
the same/different results would have been
obtained without policy action
Counterfactual -gt substitution/crowding out -
additive effects
Normative approach
State out of system
(Evaluator is independant)
16
Policy Rationales, Evaluation Rationales and
Additionality
Innovation system approach
System failures
Evolutionary approach
Selection failures
Knowledge-based approach
Knowledge/learning failures
Behavioural additionality (incl. Network
additionality ?) the differences to the agents
behaviour following the policy action the
persistence of differences to the agents
behaviour beyond actions carried out under
policy the differences in network structures and
actors' position in structures Cognitive capacity
additionality whether the policy action changes
the different dimensions of the cognitive
capacity of the agent. organizational add.,
structural add. .
Counterfactual -gt complementary, multiplicative,
catalytic effects etc
State part of the system
Normative approaches ?
(Evaluator is in the system)
17

• Additionality in FP5 assessment (Polt-Streicher,
  2005)
• What would you have done if the project had not
  received FP5 fundings ?
• (n939 participants)
• 57 would not have undertaken the project in the
  absence of EU funding
• 36 of the remainder would have gone ahead
• On a smaller scale 29
• Less partners 23
• Less ambitious objectives 15
• Lower expectations of profit 14
• More national partners 12
• A longer time-scale 11
• Less funds 7
• 10 would have replace EU funds with their own
  funds
• 20 would have replace funds with other external
  funds
• 13 would have done at least one thing "better"
  without EU funds

18
3. (Socioeconomic) effects and impacts
Economic evaluation vs economic effects, Direct
vs indirect effects, short term vs long term
effects, occasional vs steady effects, etc

• Competitiveness and exploitation/market-related
  effects (products, prototypes, patents,
  processes, interfirm technology transfer)
• Individual and organisational learning effects
  (such as finding of new partners, networking,
  building of research infrastructure) and
  behavioural changes (such as increased
  collaboration skills)
• Influencing norms and standards
• Structural effects (firm size, regional
  distribution, trusts, cartels)
• Externalities generic knowledge, RD flows,
  system and network externalities, impact on the
  environment, health, education
• Cohesion, diversity (in case of international
  programmes, especially European programmes)
• Contribution to skills and research manpower
• (Meyer-Krahmer / Georghiou 1992)

19
One reference model outputs - (outcome) - impact
From ToolBox Report 2002
20

• Outputs vs
• changes in routines
• behavioural changes
• structural organisational changes
• internally to actors - between actors

21
Key points PROPAGATION of effects - short term
-gt long term - participants -gt non participants -
micro -gt meso -gt macro SEPARABILITY among factors
generating effects - project fallacy (timing and
scope, cumulativeness) - complementary assets for
innovation (incl. strategy) - between partners if
collaborative projects
22
Dominant models 1
NO MODEL !! List of indicators "à la Prévert"
23
Dominant models 2 linear model / input-output
based
methods, models, data base, reports, lab
notebooks,
gt Econometric models (firm, sector
levels) output f (various inputs possibly incl.
those with public origin, structural variables,
dummy for participation to programme)
24
Dominant models 3 microeconomic market based
approaches sales, price, consumers, suppliers,
competitors, market interactions (market
structures, externalities) gt private /
public rate of return
25

• Other approaches, more scattered
• Network based approaches
• Social networks increase of number of links,
  centrality, connectivity gt growing emphasis
• Techno-economic networks CSI
• Evolutionnist approaches
• Knowledge-based approaches
• BETA approach (to some extent)

26
4. Some examples

• EU FP - see Annex below for institutional
  settings and some references
• EUREKA http//www.eureka.be CSE (Continuous and
  Systematic Evaluation) new evaluation study
  2006
• USA ATP programme http//www.atp.nist.gov
• implementation of 1993 GPRA (Government
  Performance and Results Act) and 2001 PART
  (Program Assessment Rating Tool) gt
  effectiveness, efficacy (type 1 goal attainment)
• Various national experiences in
• Conf. OECD Paris 1997 Conference
• Tokyo 2002 Conference
• Kuhlmann/Shapiro 2003 book,
• ASIF 2003
• Toolkit 2002
• Microeconometric apporaches (see Klette and al.)
• evaluation of basic research (2 conf in Vienna
  
• http//www.fteval.at/conference06/)
• 2005 Conf. of European Research Evaluation
  Network in Manchester
• http//www.mbs.ac.uk/Research/engineering-policy/d
  ti-evaluation-conference-2005.htm

27
(No Transcript)
28

• Case study of RD efficiency in an ATP Joint
  Venture - A. Link 1997
• Case Printed Wiring Board Research Joint
  Venture
• 91-96
• Cost 12.866 million ATP
• 13.693 million industry
• 5.2 million US DoE
• --------------
• 31.759 million
• ATT, Texas Instruments, Hamilton Standard,
  Allied Signal, Sandia, Hughes Electronics, IBM
• COUNTER-FACTUAL HYPOTHESIS - project based
• projects that would have been started in the
  absence of the ATP award (generally with at least
  one
• year delay)
• 91 7 / 29
• 96 31/62
• why ? cost of research / related risk
• gt Evaluation only on those projects
• comparing situation with ATP (actual) with what
  would have happened without (counter-factual)
  input and output additionality
• Effects

29
(No Transcript)
30
BETA approach RD programmes (RD content -
project characteristics limited in goals, time)
especially collaborative ones Ex. FP
programmes - RD collaborative projects (public
co-funding) European Space Agence programme
(public procurement) Material Ireland
(private/public funded technical centers) French
national programme for SMEs (refundable
grants) Sampling (participants /
projects) Micro evaluation and sum up of results
at sample level Data collection based on direct
interviews
31

• Effects for participants only
• Direct effects fit with objectives
• Indirect effects beyond objectives - various
  learning effects affecting other activities than
  those related to evaluated activities
• scientific and technological
• networking, relational
• organisational changes (internal)
• methods, modes of management
• reputation
• critical mass
• Identification of effects / learning processes
  (intra inter partners)
• Quantification
• products/services sales, cost reductions, new
  research projects, revenues from licenses etc
• "value" of critical mass
• "coefficient of influence" very seldom 100,
  minimal estimates

32
EUREKA 2006 the "Iceberg" Model Survey Case
studies - partly based on BETA approach
Sales of innovative product Reduced process
cost Licence income -------------------- Firm
strategy, organisation and method learning Use of
technology in other parts of the business New
contracts/netwoks prestige Employment,
competences and training ------------------- Spill
overs to non participants Use and social benefits

33
4. New challenges for an "ideal" evaluation
scheme ?
Trends / challenge for evaluators
Trends in ST policy making

• Complexity/Separability
• Multiple stakeholders
• Coherence of objectives
• Weight of different
• dimension of the evaluation
• Handling/using the results
• Evaluation of policy mix
• Mix of  evaluation cultures 
• Evolving schemes
• Evaluation of shifts
• Legitimacy, project fallacy

Multi-levels multi-agents decision Variety of
goals Variety of Inst. Arrang. Coupling with
other policies International collaboration /
integration (EU) Flexibility / adaptive -
learning policy makers Privatization /
contractualization Development of
competition-based programmes
34
Trends / challenge for evaluators
Trends in research activity
International dimension Benchmarking Bias toward
short term market-type output Limits of peer
reviews Separability/Complexity Network
evaluation Evaluation of knowledge/competences /c
apabilities / capacity Limits of bibliometrics
Globalization Solving problem orientation Interd
isciplinarity Cooperation S-I linkages Increasin
g knowledge content IPR regulations Devpt of
ICT
35
Towards a  good  evaluation system ?
Complexity no single method
multi-interdisciplinarity /quantitative and
qualitative WITH alignment between (implicit or
explicit) theoretical basis about which effects
are to be evaluated Guaranteing the scientific
value of evaluation methods (Robustness,
Repetability, Appropriability, Transparence,
Independance of the evaluators, Confidentiality,
Sampling) Balance between systematic /
standardized /simple approaches and exploratory
studies gt systematic Evaluation vs
monitoring, time fit between RD activity and
evaluation, Availability of data, Evaluation
fatigue, Cost of evaluation
36
Towards a  good  evaluation system ?
Ensuring that evaluation takes place on a
programmed and properly resourced
basis Providing  easy to understand  results
while avoiding meaningless list of indicators /
scoreboards Providing a mechanism for feedback
of the results into policy making (learning
policy maker) Interactions between academics,
practitioneers, policy makers and research actors
for better understanding of scope, relevance and
needs gt evaluation as a  social process 
37

• Evaluation as a  social process  (from L.
  Georghiou )
• Motives / interest relates to the actors
• Those  being  evaluated
• justification/legitimation
• learning at operational level
• gain new supports for public sources
• Those who are the audience of the evaluation
• accountability
• resources allocation
• learning at policy level (pro-active evaluation)
• Those performing the evaluation
• academic interest
• consultant business
• Make purpose and context-dependancy clear before
  choice of approach

38

• Evaluation as a  social process  (from L.
  Georghiou )
• Co-evolution of policy rationales/tools and
  evaluation rationale/tools
• 1970s modification of peer review to extend
  criteria
• 1980s interest began in collaborative RD
  programmes
• 1990s rise of
• performance indicators
• emphasis on knowledge transfer indicators
• commercialisation of research
• 2000s interest in
• evaluation of system capabilities eg national or
  regional systems
• aggregate or interactive effects of policies
  (policy mix)
• effect of soft policy tools such as foresight
• strategic and persistent effects on firms of
  public support (behavioural additionality)

39
ANNEX Institutional set-up for the evaluation
the case of the EU Framework Programme
Further information on 5th EU FP evaluation
system http//www.cordis.lu/fp5/monitoring/home
.html
E1-E4 EU Staff Scientific/ Industrial/
 External  Advisory Committees EURAB -
European Research Advisory Board (design and
implementation) Selection (ex ante) of proposals
Panel of independant experts /call E5-E9
Monitoring and Evaluation System ( from 1995
on) FP 3 validity, efficiency, effects
pro-active dimension FP 4 focus and
appropriateness, achievements and
effectiveness, management and efficiency
40
2 levels of evaluation Specific
programmes Framework programme 2 types of
evaluation Monitoring Five-year
assessment Creation of an Evaluation Unit
various evaluations from various High Level Groups
41

• Monitoring
• --gt day to day EC
• --gt annual (continuous annual reports) panel
  of independant experts (June 2005 on 2003
  activities)
• methods
• experts panels, indicators, qualitative
  evidence, interviews,etc
• target audience
• programme managers and committees, CREST, FP
  management
• cost-effective implementation E5
• progress in relation with objectives E6/E7
• objectives, resources still appropriate E8/E9
• http//europa.eu.int/comm/research/reports/index_
  en.html

42
5-year assessment before new FP proposal
panel of independant experts methods experts
panels, indicators, qualitative evidence,
interviews, surveys, report from monitoring,
target audience programme managers and
committees, CREST, FP management, European
Parliament, Council, Economic and Social
Committe relevance E9 management and
effectiveness E6-E7 efficiency E8
recommandations E1-E4
43

• Example of 5-year assessment
• dec. 2004  Ormala Report  on 1999-2003
  http//europa.eu.int/comm/research/reports/2004/
  fya_en.html
• need for (among others)
• focus on science / radical innovation
• -precise and simple definition of European Added
  Value
• -higher involvment of industry, high tech SME
• -simplification of administrative procedures
• -more support on HR/mobility

44
European Research Area and FP 6
Networks of Excellence progressive and lasting
integration of research capacities existing
advance knowledge by pooling a critical mass of
competence and skills. virtual centre of
excellence / long term objectives. evaluation
added value the scope of evaluation will extend
beyond activity directly funded by the Commission
multiple pathways by which research affects
the economy (focus on production of people
trained) additionality will lie both in the
immediate EAV generated and in the persistence of
the integration beyond the funding period long
term perspective, incompatible with the precise
specification of outputs and effects
45
European Research Area and FP 6
Integrated Projects aim to reinforce European
competitiveness or to contribute to the solution
of important societal problems through the
mobilisation of a critical mass of research and
technological development resources and skills
existing clearly defined objectives in terms of
scientific and technological knowledge evaluation
socio-economic dimension critical scale /
meso-level evaluation
46
European Research Area and FP 6
Integration of research policies extend beyond
the evaluation of the Framework Programme.
mutual understanding of what has been
achieved the balance of benefits what
constitutes quality and excellence in each
country. take into account widely varying
institutional settings for the same work
Benchmarking See for instance
http//trendchart.cordis.lu/index.cfm Misuse of
data without contextualisation Mobility and
Cooperation with Third Countries Towards
assessment of capability and potential.
47
Examples of strategic evaluations
Evaluation of the effectiveness of the new
instruments (Mid-term FP 6) Morimon High Level
Group Report July 2004 E8 http//www.cordis.lu/fp6
/instruments_review/
-clarification goals/roles of different
instruments (old and new ones) -need of
continuity of tools along different FPs -the
question of the size of NoE and
IP -administrative burden gt two-step
evaluation -special attention to SME and  new
comers 
EURAB report on FP 6 assessment (focus on
instruments and FP7) April 2005 E8-E9
(http//europa.eu.int/comm/research/eurab/index_en
.html)
-clarification, diversity and coherence of tools
(mission oriented strategic research JETI /
bottom-up ERC / old tools / FP6 tools) -5-10 of
budget for new initiaives - flexibility /
openess -new specific instruments for SME
(industrial PhD, SBIR-like, info centres) -stop
budget cuts / evaluation for stricter selection
(focus vs disseminate) -better multi-level
coordination (reg., nat., levels)
48
European Research Area and FP 6

• European Research Advisory Board (EURAB)
  recommandations for evaluation
• Excellence as a criterion
• Ensuring a quality Evaluators database
• Distinguish dissemnitation plans for
  basic/applied project proposals

Suggested Quantitative Indicators MID-TERM
Evaluation. Number of IPs and NoEs with
scores above threshold Researchers/countries
/disciplines involved Companies/SMEs involved
Joint publications (abstracts/proceedings)
Ph.D. Students in training in the consortium
Shared facilities resources Staff
exchanges per total no. of researchers
Patent submissions Percentage
of Proposals above threshold that were
funded Use of shared facilities by other
participants Budget used in project
administration Budget retained in the
project account Private investment so far
49
European Research Area and FP 6
European Research Advisory Board (EURAB)
recommandations for evaluation
Suggested Qualitative Indicators MID-TERM
Evaluation. Real participation and role
of key players of high reputation Are joint
research plans being made for a future period
? Joint standards and methodology (progress,
intent etc) Extent to which network has been
structured o Regular meetings o Consortium
agreements Submissions to journals with
top 20 impact factor ratings in relevant field
Base for new legislation Image/
Advertising/ participation in technical workshops
Views of participants regarding Management
etc Activities aimed at developing public
interest in science Gender disaggregated
data
50
European Research Area and FP 6
European Research Advisory Board (EURAB)
recommandations for evaluation
Suggested Quantitative Indicators FINAL
Evaluation Number of Companies/SMEs still
actively involved Joint publications
proceedings Newly trained Ph.D. Students
Shared facilities Staff exchanges per
total no. of researchers Patents and Patent
submissions Percentage of Use of facilities
by other participants Budget used in
project administration Private
investment so far Increasing market share/
new/ income New Business Success in
leveraging additional funding
51
European Research Area and FP 6
European Research Advisory Board (EURAB)
recommandations for evaluation
Suggested Qualitative Indicators FINAL
Evaluation Involvement of key players
of high reputation Joint research
development plans Joint plans for
continuation of collaboration activities to
develop same Degree of clustering with
other networks Joint standards and
methodology Publications in journals with
top 20 impact factor ratings in relevant field
Bases for new legislation Image
development/ Advertising/ participation in
technical workshops Teaching, Training and
Framework Development Activities aimed at
developing public interest in science
Gender disaggregated data
52
Ex-ante E1 E2-E4 (broad principles) Impact
assessment and ex-ante evaluation (April 2005)
Staff working paper See in "basic documents" at
http//www.kowi.de/en/fp/fp7/fp7.htm Context -
policy options - institutional arrangements
options - additionality / proportionality
Aggregate economic impact (econometric model for
growth - jobs - competitiveness) Aggregate
social impact Environmental impact
Contribution to other EU goals / policy RD
intensity - Research employment (Barcelona)
Sustainable development - etc
53
Towards FP 7 ...main features (see F.
Meyer-Krahmer)
Double the EU research budget ( 72.7 billion
over 2006-13) Knowledge-Education-Innovation
triangle - Lisbon/Barcelona strategy Foster the
fundamental research (European Research Council),
notably by competition between teams (programme
"Ideas") Human potential / reinforce
attractivity of European research ("People")
Reinforce European research infrastructure
("Capacities") "Cooperation" Develop poles
of excellences Externalize part of the
management Simplify institutional
arrangements Reinforce coordination of national
policies
54
Towards FP 7 ... Requirements for evaluation

• More needs for evaluation
• Evaluating critical mass, Euopean Added Value,
  infrastructure, coordination, externalization
• Evaluating fundamental research...
• Draw lessons from FP5/6 evaluation system -gt
  re-organization

55
From N. Reeve Planning Programming Evaluation
Directorate General Research , European
Commission, Nov. 2005
FP6 FP7
Research objectives Robust hierarchy of outcome objectives with appropriate indicators
Data collection (contract process, reporting) Systematic and simplified collection of data
Annual Monitoring by independent experts - Monitoring of implementation by senior Commission management Indicators to track progress
Mid-term evaluation (Science panels)
Five Year Assessment by high-level independent experts Ex post assessment of an FP, 2 years after its completion by high-level independent experts
Impact surveys at FP level Strengthened programme of coordinated strategic-level evaluations
Evaluation studies at operational level Evaluation studies at operational level (portfolio, programme)
National impact studies Coordinated national impact studies
Ad-hoc research-related activities FP research on evaluation tools and approaches