BIBLIOMETRICS USE AND LIMITATIONS

1
BIBLIOMETRICS USE AND LIMITATIONS

• Wolfgang Glänzel
• KU Leuven, Belgium
• ISPR, HAS, Hungary

2

• STRUCTURE OF THE PRESENTATION
• Introduction
• Scope and structure of bibliometrics
• Data sources of bibliometric research
• Perspective shift in use of bibliometrics
• Bibliometrics as a tool in research evaluation
• Bibliometric research and services at the
  Steunpunt OO Statistieken

3
1. INTRODUCTION
4
1. Introduction
What is bibliometrics? The terms bibliometrics
and scientometrics were introduced by Pritchard
and Nalimov Mulchenko in 1969, and are nowadays
used almost synonymously. Pritchard bibliometri
cs is the application of mathematical and
statistical methods to books and other media of
communication. Nalimov Mulchenko Scientometri
cs is the application of those quantitative
methods which are dealing with the analysis of
science viewed as an information process.
5
1. Introduction
Gorkova (1988) Informetrics stands for a more
general subfield of information science dealing
with mathematical-statistical analysis of
communication processes in science. A quite
broad definition of the scope of bibliometrics
given by Glänzel Schoepflin (1994) includes
all quantitative aspects and models of science
communication, storage, dissemination and
retrieval of scientific information.
6
1. Introduction

• These notions also integrate many presently
  existing orientations such as applications to
  science policy, library science, technometrics,
  Web analyses and information retrieval. Prominent
  methodological issues are related to
• subject delineation
• subject evolution and structure
• publication activity/scientific productivity
• scientific collaboration, co-authorship networks
• citation impact and citation networks
• bibliographic coupling, co-citation analysis
• computational linguistics text mining, co-word
  analysis
• visualisation techniques
• data matching
• science-technology linkage

7
1. Introduction

• Common misbeliefs on bibliometrics
• Main task of bibliometrics should be the
  expeditious issuing of prompt and
  comprehensible indicators for science policy
  and research management.
• Bibliometrics is only an auxiliary tool, research
  on methodology is unnecessary. Instead,
  bibliometricians should elaborate guidelines
  explaining the use of their indicators.
• Bibliometrics might be reduced to simple counting
  activities in order to replace/supplement
  qualitative assessment by quantitative indicators
  and to set publication output off against funding.

8
1. Introduction

• Facts about bibliometrics
• Bibliometrics is a powerful, multifaceted
  endeavour encompassing sub-areas such as
• Structural scientometrics came up with results
  like the re-mapping of the epistemological
  structure of science.
• Dynamic scientometrics constructed sophisticated
  models of scientific growth, obsolescence,
  citation processes, etc.
• Evaluative scientometrics developed arrays of
  indicators to be used to characterise research
  performance at different levels of aggregation
  and come up with benchmarking solutions.

9
2. SCOPE AND STRUCTURE OF BIBLIOMETRICS
10
2. Structure of Bibliometrics
Three components of present-day bibliometrics
according to its three main target-groups
Bibliometrics for bibliometricians The domain
of bibliometric basic research. Bibliometrics
for scientific disciplines A large but also the
most diverse interest-group. Due to the
scientists primary scientific orientation, their
interests are strongly related to their
speciality. Bibliometrics for science policy and
research management The most important field of
application. Here the assessment and comparative
analysis of research performance are in the
foreground.
11
2. Structure of Bibliometrics
Links of bibliometrics with related research
fields and application services
Science policy
Scientific information
Research management
Librarianship
Services for
Research in
Economics
Sociology of science
History of science
Library and Information Science
Life sciences
Informetrics
Mathematics/Physics
Webometrics
12
3. DATA SOURCES OF BIBLIOMETRIC RESEARCH
13
3. Sources of Bibliometrics
Data sources of bibliometrics are bibliographies
and bibliographic databases. Large scale
analyses can only be based on bibliographic
databases. Despite objections to journal
coverage and data processing policy of the ISI,
the databases of the Institute for Scientific
Information (Thomson - ISI), above all, the
Science Citation Index (Expanded) have become the
most generally accepted data source. Recently
several countries have developed their own
national citation indexes which are also used (as
supplementary) data sources.
14
3. Sources of Bibliometrics

• Bibliographic databases are generally designed
  for the retrieval of information. However, their
  versatility is often limited in bibliometric use.
  
• Originally, bibliometrics was developed for
  studying communication in basic research.
  Present-day tasks
• How to extend bibliometrics towards application
  to engineering, social sciences and humanities?
• How to measure and evaluate aspects of technology
  linkage?
• Since non-periodicals are important conveyors of
  information in engineering, social sciences and
  humanities, journal based data-sources are
  accepted by scientists only with certain
  reservations.

15
4. PERSPECTIVE SHIFT IN USE OF BIBLIOMETRICS
16
4. Perspective Shift

• Bibliometrics deals with the quantitative
  analysis of the (social) network of scientific
  communication.
• From the historical perspective, information
  scientists and bibliometricians proceeded from
  the same approach. First citation analyses were
  designed to aid the decision which periodicals
  should best be purchased by small libraries. Many
  tools and indicators developed by the ISI were
  designed for use in scientific information,
  information retrieval and libraries.
• The application to science policy has brought a
  new perspective, and resulted in
  re-interpretation of bibliometric conceptions.

17
4. Perspective Shift
Example The notion of citations in information
science, bibliometrics and science
policy Citations became a widely used measure of
the impact of scientific publications, but the
notion of what citations actually express sheds
light on different aspects. These different
views might results in conflicts caused by
re-interpretation and perspective shift.
18
4. Perspective Shift
Some examples According to Merton, citations are
intimately connected with the reward system of
science. Cozzens Citation is only secondarily a
reward system. Primarily, it is rhetorical-part
of persuasively arguing for the knowledge claims
of the citing document. Cronin Citations are
"frozen footprints in the landscape of scholarly
achievement which bear witness to the passage
of ideas. Glänzel and Schoepflin Citations
give a formalised account of the information use
and can be taken as a strong indicator of
reception at this level.
19
4. Perspective Shift
The process of re-interpreting the notion of
citation and its consequences
interpretation
citation
Information use
Bibliometrics/Information science
uncitedness unused information frequent cite
good reception self-cite part of scient.
communication
repercussion (possible distortion of citation
behaviour)
re-interpretation
uncitedness low quality frequent cite high
quality self-cite distortion of impact
Reward system (quality measure)
Research evaluation/Science policy
20
5. BIBLIOMETRICS AS A TOOL IN RESEARCH EVALUATION
21
5. Research Evaluation

• Present-day use of bibliometrics
• The evaluation of scientific research performance
  is the most important application of present-days
  bibliometrics.
• Growing, often controversial, policy interest to
  use bibliometric techniques as output measures of
  research productivity and efficiency.
• Thus bibliometrics evolved to a standard tool in
  science policy and research management.
• Bibliometric components are used, among others,
  in national science reports, bibliometric
  indicators appear in the daily press,
  benchmarking results are used for research
  management and for allocation of research
  funding.

22
5. Research Evaluation

• What can bibliometrics not be responsible for?
• Bibliometrics can be used to develop and provide
  tools to be applied to research evaluation, but
  is not designed to evaluate research results.
• Bibliometrics does not aim at replacing
  qualitative methods by quantitative approaches.
• Consequently, bibliometrics is not designed to
  correct or even substitute peer reviews or
  evaluation by experts but qualitative and
  quantitative methods in science studies should
  complement each other.

23
5. Research Evaluation
Distorted behaviour based on policy use and
misuse of bibliometric data One issue concerns
the changes in the publication, citation and
collaboration behaviour of scientists (both
positive and negative) that the consistent policy
use of bibliometric indicators might potentially
induce. Uninformed use and misuse are not
always beyond the responsibility of
bibliometricians. Unfortunately, bibliometricians
do not always resist the temptation to follow
popular, even populist, trends in order to meet
the expectations of the customers. Any kind of
uninformed use or misuse of bibliometric results
involves the danger of bringing bibliometric
research itself into disrepute.
24
5. Research Evaluation

• The problem of inappropriate use ranges from
  uninformed use, over selecting and collecting
  most advantageous indicators to the obvious and
  deliberate misuse of data.
• Uninformed use
• incorrect presentation, interpretation of
  bibliometric indicators or their use in an
  inappropriate context caused by insufficient
  knowledge of methodology, background and data
  sources
• Misuse
• intentionally incorrect presentation,
  interpretation of bibliometric indicators or
  their deliberate use in inappropriate context
• tendentious application of biases or tendentious
  choice of (incompatible) indicators

25
5. Research Evaluation

• But even correct use might have undesired
  consequences
• Author self-citations are shown in an
  unfavourable light. Authors might thus feel urged
  avoiding self-citations a clear intervention
  into the mechanism of scientific communication.
• Indirect repercussions can be observed when
  bibliometric tools are used in decision-making
  and the scientific community recognises the
  feedback in terms of their funding.
• Butler (2004) showed on the example of Australia
  what might happen when funding is linked to
  publication counts. She found that this has
  stimulated an increased publication activity in
  the lower-impact journals.

26
5. Research Evaluation
27
5. Research Evaluation

• Possible positive effects
• Scientists might recognise that scientific
  collaboration and publishing in high-impact or
  even top journals pays off. Also their
  publication activity might be stimulated.
• Possible negative effects
• Exaggerated collaboration, even trends towards
  hyper-authorship, inflating publication output by
  splitting up publications to sequences, inflating
  citation impact by self-citations and forming
  citation cliques, etc.
• Trend towards replacing quality and recognition
  by visibility at any price or towards preferring
  journals as publication channels in social
  sciences and humanities might be among these
  effects.
• A certain champions league mentality is
  spreading among scientists (Shanghai ranking,
  H-index, IF Filter).

28
6. BIBLIOMETRIC RESEARCH AND SERVICES AT THE
STEUNPUNT OO STATISTIEKEN
29
6. The Steunpunt OO Statistieken
The Steunpunt OO Statistieken (Research Centre
for RD Statistics SOOS) was created in January
2002 after open calls for proposals. SOOS is a
unique and dedicated research center to support
its science and innovation policy in Flanders.
This center has to develop an appropriate system
of quantitative indicators to quantify RD
efforts at Flemish universities, research
institutes and industry that can be used by
policy makers to support the development of
appropriate regional science and innovation
policy for Flanders.
30
6. The Steunpunt OO Statistieken
ACTIVITY OVERVIEW The activities of SOOS can be
classified into four categories 1. Developing an
appropriate IT-infrastructure to support ST
indicator development 2. Providing the Flemish
government with appropriate ST indicators on
which to base its science and technology policy
resource allocation decisions 3. Executing
specific studies on science and technology
domains or science policy relevant topics for the
Flemish government 4. Doing original research in
the field of ST studies.
31
6. The Steunpunt OO Statistieken
LONG- AND MEDIUM-TERM SERVICES FOR THE FLEMISH
GOVERNMENT
I. The BOF-key One of the major funding
mechanisms for basic science in Flanders is the
BOF-mechanism. The Bijzonder Onderzoeksfonds is a
research fund of the Flemish government that is
distributed to the Flemish universities on the
basis of a specific allocation model. This
allocation key comprises both input and output
measures on the research performance of the
universities involved. Part of the key is based
on publication and citation data as derived from
WoS by SOOS.
32
6. The Steunpunt OO Statistieken
BOF-funding for Flanders amounts to 100 million
EUR annually. This amount is distributed in basic
science funding amongst the six Flemish
universities (LUC/UHasselt, KUB, KUL, VUB, UA and
UGent). II. The IOF-key The IOF-key is a
distribution key to determine the annual
allocation of 12 million EUR (to be augmented in
the year 2006) amongst the six Flemish
universities in order to support applied
research. The IOF-key is partially based on
patent statistics derived from the USPTO, EPO and
PCT databases. SOOS has been instrumental both in
the development of and the data delivery for
those distribution instruments.
33
6. The Steunpunt OO Statistieken
III. The Flemish Indicatorbook The Flemish
Indicatorbook on Science, Technology and
Innovation appears biannually, and does not only
support government questions on RD-proficiency
in Flanders, but also aims to supply adequate and
up-to-date information on the European
RD-potential of the Flanders region. Two
editions of the Indicatorbook have been published
by SOOS so far 2003 and 2005.
34
6. The Steunpunt OO Statistieken
IV. Domain Studies (here bibliometrics only) Aim
of these studies is to conduct and to provide
analyses of Flemish research and technology
activity in the international context in highly
important areas. Expected outcomes are relevant
information about performance and competitiveness
in Flanders. Three domain studies have been
prepared and published by SooS Nanoscience and
technology, Biotechnology and molecular biology
(both in 2003) and Stem-cell research (2004).
The studies are based on both publication and
patent data extracted from the WoS, the USPTO and
EPO databases.
35
6. The Steunpunt OO Statistieken
BIBLIOMETRIC RESEARCH PROJECTS AT SOOS
Methodological and applied research is closely
linked to the service tasks, for instance, in the
context of methodological validation, of
improvement of the efficiency of indicators or of
the subject classification. Basic research at
the Steunpunt is structured in a systematic way
to support policy-relevant research and services
and to contribute to the advancement of the field
of bibliometrics.   Similarly to the services,
research is done at the national/regional,
institutional and research-group level.
36
5. The Steunpunt OO Statistieken

• Development of a hierarchic subject
  classification scheme for evaluation purposes
• The role of author self-citations in scientific
  communication
• The myth of delayed recognition
• Analysis of possible extension of bibliometric
  methods to social science and humanities
• Studying the network of science communication
• Indicator research (improving journal and
  subject impact measures and science indicators)

37
6. The Steunpunt OO Statistieken

• Bibliometrics beyond the WoS (e.g., proceedings
  literature)
• Bibliometric approach to social sciences and
  humanities
• Combining text-mining and bibliometrics
• Mapping academic mobility and scientific
  collaboration in Europe
• Research group dynamics and their impact on
  performance and visibility
• Bisociation analysis of scientific communication