Benchmarking Core Methods

1
Benchmarking Core Methods

• Nicos Komninos
• URENIO Research Unit
• MLP Workshop
• Brussels, 20 June 2006

2
Benchmarking method

• The scope of the benchmarking methodology is to
  measure the performance of an entity (region,
  organisation, cluster) using numeric indicators
  and to compare the performance of the entity
  towards other entities.
• Two benchmarking methods
• One-to-one benchmarking comparing an entity
  with another one showing best practice, thus
  illustrating the deviation of the entity in focus
  from the organisation showing best practice.
• One-to-many benchmarking comparing an entity
  with the statistics of many other entities,
  better or worse, thus positioning the entity in
  focus into the range between the best and the
  worst performance.
• Regional benchmarking follows the second method.
  We usually compare a geographical entity (region,
  city, locality) with a sum of other regions.

3
Benchmarking method

• The steps for the implementation of the regional
  innovation benchmarking methodology include
• Selection of indicators, which should be able to
  bring on the surface the performance of a region
  in the field of innovation.
• Creation of the benchmarking database, which
  concerns the gathering of information on regional
  performance and the calculation of selected
  indicators for different regions.
• Production of the benchmarking data, which
  highlights the main statistics and graphs for the
  selected indicators (min, max, mean, mode,
  quartiles) and the position of the region in
  focus within the range of these statistics.
• Analysis and interpretation of statistics, which
  tries to find out the causes of the observed
  performance and the practices that are
  responsible for this performance.
• Suggestions for improvement, based on worldwide
  best practice, the benchmarking process concludes
  with the suggestion of measures which should be
  taken to improve the innovation performance of a
  region.

4
Benchmarking method

• Critical issues for the successful implementation
  of benchmarking
• Indicators are of major importance for the
  measurement of the innovation performance and the
  drawing of comparison tables and diagrams. In
  order to obtain reliable results, indicators have
  to be fully defined, in terms of concept,
  variables involved, calculation process, year of
  measurement, etc., and calculated with a uniform
  process.
• Data should be based on official sources which
  guaranty the validity and uniformity of the
  information collected.
• The selection of the comparison group depends on
  the scope of the benchmarking exercise. A region
  may be compared towards all entries of the
  database or towards a group of regions
  characterised by specific criteria set (i.e.,
  geographical area, GDP, population, innovative
  products, etc.). Best practice is usually linked
  to the top performance identified among the
  regions of the database.
• Interpretation of results defining the causes of
  a specific regional innovation performance
  depends greatly on the expertise of consultants
  involved in the benchmarking exercise.

5
1st level analysis

• First level benchmarking analysis is about the
  (1) calculation of benchmarks and (2) the
  positioning of the region in focus within these
  benchmarks.
• Benchmarks are calculated from a sample of
  regional profiles that were collected and stored
  into the database.
• Usual benchmarks are
• (1) the minimum value of the index into the
  sample, (2) the maximum value of the index into
  the sample, (3) the mean value of the index, (4)
  the quartile values, and (5) the standard
  deviation that measures the dispersion of values
  from the mean.

6
1st level analysis Benchmarks
Data collection and entry into the database
should be followed by a validation process. All
data should have the same format and extreme data
have to be excluded. The proposed way is to
normalize the sample by applying the standard
deviation rules, and exclude extreme values
outside /- 3 STDEV limits. In case of data from
field survey, if it is required the person who
provided the data will have to be contacted
again, for any clarification or modifications.
After these controls, the responsible for
validation person may characterise the data as
valid and proceed to the calculation of
benchmarks.
7
1st level analysis Positioning

Positioning of a region with respect to the
benchmarks calculated from the sample is based
on The real index value, which shows the
performance of the region in a specific field of
activity, i.e., patents, RD expenditure,
tertiary education, etc. The percent rank value,
which ranks a value in a data set as a percentage
of entities included in the data set. This
function can be used to evaluate the relative
standing of the entity having a value within a
data set of entities. The improvement index
value, which shows the distance of the current
index from the maximum value.
8
1st level analysis Positioning
9
2nd level analysis
10
2nd level analysisFocused analyses

• Various types of focused analysis may be useful
• Subject analysis comparing a theme across
  regions using summary index from a family of
  indicators (i.e. RD spending)
• Cluster analysis comparing a region with a
  group of similar regions (i.e. catching-up LFRs)
• Competition analysis comparing a region with a
  group of competing regions (i.e. capitals of EU
  on the attraction of RD FDI)
• Trend analysis comparing the evolution of
  indicators over a period of time

11
2nd level analysisPerformances - Practices

• Identifying practices behind the performances
• Definition of input output factors linking
  performances and underlying practices
• Modelling the relationships between the factors
  (variables)
• Estimation of parameters for the variables from
  available data
• Understanding the role of different practices in
  the overall performance (i.e. the significance of
  university RD in the innovation performance of a
  region)
• An example Griliches on Patents - RD

12
2nd level analysisPerformances - Practices

• RD practices and patent performance
• Model developed by Griliches (1979, 1984)
• Input RD in industries and universities
• Output Patented innovations
• The model is
• lnPATs ß1lnIRDs ß2lnURDs ß3lnCs POPs
  es
• lnPATs is the natural logarithm of the number of
  patents granted to private manufacturing firms
  in the state S
• ß1lnIRDs is the natural logarithm of RD
  expenditures by manufacturing firms in the state
  S
• ß2lnURDs is the natural logarithm of RD
  expenditures by universities in the state S
• ß3lnCs is the geographical coincidence index
• POPs is the total resident population in the
  state S
• es is a stochastic error
• Cs Sic UNIVic Tpic / SicUNIV2ic1/2
  SicTP2ic1/2
• UNIVic is RD expenditure within universities by
  industry i and metropolitan area c. Tpic is the
  number of researchers in the manufacturing sector
  by industry i and metropolitan area c.

13
2nd level analysisIntegrating intelligence

• Benchmarking Market watch Foresight
• The purpose of combining intelligence from
  benchmarking, market watch, foresight and other
  sources is to gather information from multiple
  sources, integrate multi-dimensional information,
  and widen the horizon of survey and watch.
• Complementary to benchmarking, market watch is
  the collection and dissemination of information
  about production, products, and prices. Market
  watch is better organized on an industry or
  cluster basis.
• Foresight can be defined as a systematic,
  participatory process, involving gathering
  intelligence and building visions for the
  medium-to-long-term future, and aimed at
  informing present-day decisions and mobilizing
  joint actions. Foresight involves thinking about
  emerging opportunities, challenges, trends and
  discontinuities

14
2nd level analysisIntegrating intelligence

• Meta-Foresight is an example of systemic
  intelligence, using information from regional,
  sectoral, and company sources and perspectives.
  Information from different sources (company,
  region, market, RD, etc.) is combined to give a
  holistic view to a subject (strategy, innovation,
  quality, demand, etc.) for the shake of better
  understanding and anticipating the future.
• Intelligence integration, as core concept of
  Meta-foresight, has two complementary sides. On
  one hand it denotes complementarity in the supply
  side, referring to combination of information and
  knowledge from organizations active in the above
  five fields of intelligence (foresight operators,
  benchmarking agencies, market and RD watch
  systems). On the other hand, it denotes
  participation of the users in the assessment and
  flow of information it is a feed back from
  users, thus integration of information between
  providers and users

15
2nd level analysisIntegrating intelligence
16
Benchmarking e-tools

• Most benchmarking applications use information
  technology tools to facilitate the process.
  Web-based applications may support the users
  during all steps of the benchmarking process
  from gathering information and data entry to the
  database, to the data analysis and generation of
  benchmarking reports.

17
e-ToolsAdded value

• Automation of data management and benchmarking
  reports creation All available data are stored
  into a database that is constantly increasing.
  The use of database is enabling the selection of
  the comparison sample in real time. Benchmarking
  may use alternative comparison groups. Real time
  benchmarking reporting may be produced in various
  output formats. There is also improved
  connectivity and ability to export / import data
  available in third party software and databases.
• Simplification of use There is no need of
  special knowledge from the user perspective as
  the application guides the user during all steps
  of benchmarking process. The intelligence is
  built into the application not into the user. The
  service can be offered remotely, on-line. The
  user using a web browser fills a questionnaire,
  choose the comparison sample, and obtains the
  benchmarking results easy, simple and quickly at
  any moment of time.
• Dissemination and awareness raising The Internet
  has become the mainstream dissemination channel
  for the benchmarking techniques. Huge amounts of
  data relating to all benchmarking areas are
  available through the web (methodologies,
  techniques, best practices, questionnaires,
  process models, sample reports, etc). By using
  common searching techniques the users can, easily
  and quickly, find critical information about
  benchmarking and how should use it in order to
  improve the performance.

18
e-ToolsURENIO application

• A questionnaire is filled in by the
  companies/organizations in the manufacturing,
  tourism, and ICT sectors
• The collected data are entered in a relational
  database through the web application of URENIO
• The Benchmarking Qualified Consultants are
  elaborating the indices, the statistical values
  and the spider graphs that on are presented on
  the report.
• The report is presented to the benchmarked
  company.
• After a set of meetings with the management of
  the company / organization the consultants are
  providing different options for the improvement
  of the performances and underlying practices of
  the company.

19
e-ToolsBest Practice Club
Best Practice Club Since 1993 it has been
facilitating learning and shared experience
opportunities to allow our members to identify
and adopt best business practice. BPC
D-Spaces (1) Dissemination of information Club
events Networking events (2) Membership
management Find members to benchmark an
organization (one-to-one b.) Community
events (3) Publications and reporting database
20
e-ToolsManufacturing Practices
Best Manufacturing Practices (BMP) Program was
created in 1985 to help businesses, academia and
government to identify, research, and promote
exceptional manufacturing practices, methods, and
procedures. BMP has three core competencies
represented by tools and resources that enable
organizations to identify and apply best
practices Best Practices Surveys Systems
Engineering - Web Technologies BMP digital
space opens to the core competences and
disseminate best practices
21
e-ToolsIndustry metrics
Industrymetrics.com Surveys Polls - The most
efficient and effective method of collecting and
seeing what has already been collected for
business process metrics. Self Assessor
Diagnostic - The best way to measure yourself,
based on National Quality Award Models such as
Baldrige, to determine where your organization
fits amongst others in similar and dissimilar
industries. And, results of your responses are
immediately displayed on your screen along side
the running industry standard.
22
Thank you very much for your attention