Jan Dvork, Jir Soucek: The R

1
Jan Dvorák, Jirí SoucekThe RD Efficiency
Evaluation in the Czech Republic
InfoScience Praha, s.r.o.

• CRIS2008 Maribor, Slovenia June 2008

2
Agenda

• The Czech RD IS (JD)?
• The Efficiency Evaluation in the CR (JD)?
• Efficiency Evaluation in general (JS)?

3
The RD Information System of the Czech
Republic

• A CRIS covering the RD subsidized from the State
  Budget of the Czech Republic
• Run by the Czech RD Council
• Contains data on
• Funding providers, Programmes, Calls
• RD Projects (since 1994, 30,000)?
• Institutional Research Plans (since 1999, 880)?
• RD Results publications, patents, technologies
  (since 1998, 530,000 records, 420,000 unique
  results)?
• Data from funding providers

4
http//aplikace.isvav.cvut.cz
5
The RD Efficiency Evaluation

• The setup
• The RD IS contains all results of all research
  that has been supported from public sources in
  the CR
• The State is the predominant supporter of basic
  research in the CR
• The investor's point of view
• Results per unit funding
• Runs in 2005, 2006, 2007

6
Results ? Points

• Results are assigned points
• Example from the 2007 Evaluation
• Article in an impacted journal ? 5 15 x IF
  / median( IF ) over discipline using data from
  ISI JCR
• Article in a non-impacted, reviewed periodical ?
  1 (Czech Slovak) 2 (other languages)?
• Monograph ? 12.5 (Czech Slovak) 25
  (other languages)?
• Proceedings contribution ? 0.1 (Czech
  Slovak) 0.2 (other languages)

7
Results ? Points

• Example from the 2007 Evaluation (cont'd)
• Patent ? 500 (EPO, USTPO, JPO) 50 (other
  issuer, incl. the Czech Industrial Property
  Office)?
• Verified technology, breed ? 75
• Prototype, software ? 25
• Special handling of Social Sciences and
  Humanities
• The point-assignment rules are agreed upon by the
  RD Council members

8
Results ? Points

• The point assignment has been evolving
• E.g. monograph

9
Point distribution
10
Point distribution

• MN relationships need a distribution factor
• result research activity ? result points
  divided uniformly
• research activity institution ? result points
  divided in the proportion of State Budget funding

11
Index

• Points( X ) / Cost( X ) ? Index( X )?
• Institution
• Programme
• Funding provider
• Whole RD
• Index( X ) / Index( Whole RD ) ? Relative Index(
  X )?

12
Relevant data selection

• Sliding window Evaluation in year Y Research
  activities ending in Y-5, Y-1
• Infrastructure projects excluded

13
Sample
14
Sample
15
Points by result type
16
Changes in 2008

• Modified point-assignment rules
• Esp. for articles in impacted journals
• Institutions evaluated by results
• Institution gets points for a result in the
  proportion of the authors affiliated with it to
  the total number of authors
• Part of the Reform of the system of research
  funding by the State

17
General principles of the Evaluation

• Systematic part (the logic of RD)?
• Results evaluated by the point-system
• Dividing points to individual projects
• Summing up points to Institutions
• Calculation of indexes and relative indexes
• Specific parameterization for points-systems
• Depends on the type of the result
• Using impact factor of the journal
• Using properties / attributes of the result
• Needed data on objects in RD
• Projects, institutions, results
• Links projects results
• Links institutions results

18
Principles or the impersonal (objective)
evaluation

• Is determined by rules (not by opinions)?
• Impartiality - does not depend on opinions
• Non-refutability depends on hard facts
• Is possibly imperfect, but the inaccuracy is
  random and suppressed by the aggregation (25
  results ? 5 times better precision)?
• Systematic inaccuracies can be traced and
  corrected
• Basic characteristics
• Non-refutability
• Impartiality

19
The importance of the impersonal (objective)
evaluation

• Serves not only for RD Council, but
• For all workers in RD field
• They can independently judge the contribution of
  others (projects, teams, institutions, )?
• Expected reception of a published paper is
  approximately equal to the impact factor of the
  journal (it is not necessary to wait many years
  for citations to accumulate)?
• Impersonal evaluation gain more weight than the
  peer review, since it is non-refutable

20
Needs of ERA (European Research Area)

• In the RD information infrastructure
  evaluation
• Accessibility of the information on projects,
  funds, calls, results (also on the national
  level)?
• Accessibility on the data supporting the
  evaluation
• The way
• Integration of data from national CRISes (just in
  progress)?
• Building evaluation data system
• Efficiency of the allocation of resources
• In economics market forces
• In research area - there are, in principle no
  market forces
• The role of market forces should be substituted
  by the impersonal (objective) evaluation.

21
Our vision

• The creation of the consortium focused to
  evaluation of RD results (in the 7 FP the
  infrastructure of the research)
• WE ARE LOOKING FOR PARTNERS AND COOPERATION!
• If you are interested in RD evaluation please
  contact us!

22
Thank you for your attention ?

• Jan Dvorák
• jan.dvorak_at_infoscience.cz
• Jirí Soucek
• jiri.soucek_at_infoscience.cz
• Martin Soucek
• martin.soucek_at_infoscience.cz
• Richard Papík
• richard.papik_at_ff.cuni.cz