How to run an R

1
How to run an RD survey and setting up and
strengthening RD statistical systems
South Asian Regional Workshop on Science,
Technology and Innovation Statistics Kathmandu,
Nepal6-9 December 2010
2
Ch 7 FM - RD Survey Methodology

• Statistics on RD require regular, systematic and
  harmonised special surveys
• Other sources provide info, but
• concepts of RD used often different from FM
  concepts
• concepts may change over time
• very difficult to obtain all data for the same
  period
• difficult to avoid double counting when tracking
  flows from financial statements and other sources
  
• Estimates are a necessary supplement to surveys
• Especially in higher education sector

3
Scope of RD surveys

• RD surveys should identify and measure all
  financial and personnel resources devoted to all
  RD activities in all RD units
• RD surveys are mainly addressed to
  RD-performing units
• Chapter 7 of the FM only addresses
  performer-based surveys
• Statistical methodologies and other procedures
  have to be established to capture all RD,
  especially for units in the business enterprise
  sector with little RD

4
Identifying target population and survey
respondents - general

• Exhaustive survey not possible in most countries
• Constraints include
• number of respondents may have to be restricted
  to keep costs down
• RD survey may have to be taken in conjunction
  with another survey
• surveys of some groups may require the
  participation of other agencies with different
  data needs and hence different questions for
  respondents
• One size does not fit all every country has
  different constraints advice is therefore of
  general nature

5
Identifying target population and survey
respondents - business enterprise sector

• The enterprise is recommended as the main
  statistical unit in the business enterprise
  sector
• Some enterprises perform RD on a regular basis
  from year to year, and may have one or several
  RD units
• Other enterprises perform RD only occasionally
• It is recommended that all enterprises performing
  RD, either continuously or occasionally, should
  be included in RD surveys.

6
Survey population business enterprise sector
first possible approach

• A census-based survey of large enterprises and a
  sample of smaller ones in order to identify RD
  performers and request the information from them
• RD performed in the past in the enterprise is
  not considered
• this is the approach followed in innovation
  surveys
• very small enterprises and enterprises in certain
  less RD-intensive industries often excluded for
  cost reasons
• when the sample size is very small, estimates may
  be less reliable, owing to raising factors
• Method not strictly followed in any country

7
Survey population business enterprise sector
second possible approach

• Try to survey all enterprises known or assumed to
  perform RD, based on a register of
  RD-performing enterprises
• lists of enterprises receiving government grants
  and contracts for RD
• lists of enterprises reporting RD activities in
  previous RD surveys, in innovation surveys or
  other enterprise surveys
• directories of RD laboratories
• members of industrial research associations
• employers of very highly qualified personnel
• lists of enterprises claiming tax deductions for
  RD.

8
Survey population business enterprise sector
joint approach

• Recommendation
• To include in RD surveys of the business
  enterprise sector all firms known or supposed to
  perform RD.
• To identify RD performers not known or supposed
  to perform RD by a census/sample of all other
  firms
• In the industries on the next slide.
• In principle, enterprises in all size classes
  should be included, but if a cut-off point is
  necessary, it should be at ten employees.

9
Industries to be included
Industry ISIC Rev. 3/NACE Rev. 1
Mining 14
Manufacturing 15-37
Utilities, construction 40,41,45
Wholesale 50
Transport, storage and communication 60-64
Financial intermediation 65-67
Computer and related activities 72
RD services 73
Architectural, engineering and other technical activities 742
Plus any other industry relevant for the country
10
Identifying target population and survey
respondents - government sector

• Units to include in surveys are
• RD institutes.
• RD activities of general administrations of
  central or state government, statistical,
  meteorological, geological and other public
  services, museums, hospitals.
• RD activities at the municipality level.
• Recommendation the best way to survey is to send
  questionnaires to all units known or assumed to
  perform RD.

11
Identifying target population and survey
respondents - PNP sector

• The sources for identifying possible survey
  respondents are mainly the same as for the
  government sector.
• Register information may be less comprehensive
  and could be completed by information from
  researchers or research administrations.
• This sector may be more relevant for surveys on
  RD funding.

12
Identifying target population and survey
respondents higher education sector

• Recommendation The surveys and estimation
  procedures should cover all universities and
  corresponding institutions, especially those
  awarding degrees at the doctorate level. Other
  institutions in the sector known or assumed to
  perform RD should also be included.
• Identification generally easy.
• preferable to use smaller units, such as
  departments or institutes of the university, as
  statistical units.

13
Working with respondents

• Questionnaire simple and short, logical and with
  clear definitions and instructions
• Optional simpler survey for smaller units
• Test questionnaires on a sample of respondents

14
Who is the right respondent?

• RD Manager
• Better understanding of RD and FM norms
• But may not be able to supply exact figures

• Accountant or personnel manager
• May not refer exactly to RD as defined in FM
• But able to supply exact figures

• Cooperation of all three may be needed
• Useful to identify in advance the person
  responsible for providing information and for
  co-ordinating information from smaller sub-units

15
Encouraging co-operation

• Secure co-operation of respondent
• Make them appreciate the potential uses of the
  data
• Respect confidential data
• Minimise the response burden
• Share the results (option customised
  information)
• Provide technical assistance and contact details

16
Estimations

• RD measurement could be done in three stages
• Identification of all specialised RD units and
  measurement of their total activity.
• Estimates of the non-RD portions of their
  activity and subtraction of these estimates from
  the total.
• Estimates of the inputs used for RD in other
  units and addition of these estimates to the
  total.

17
Operational criteria

• Tools for translating theoretical FM concepts
  into practical questionnaire
• Explanatory notes
• Hypothetical examples
• Guidance to individual respondents
• Documentation on treatment of different cases

18
Estimation procedures

• Imputation methods for item non-response
• Use previous answer
• Hot decking (use info from same survey)
• Cold decking (use info from previous survey)
• Imputation methods for unit non-response
• Use past RD data (adjusted for sales or
  employment growth)
• Impute as a function of the relation to personnel
  or sales (test with non-response analysis)

19
Improving Statistical Systems Advice from the
UIS Technical Guide
20
Ch 7 UIS TG Strategies for setting up ST
statistics systems in developing countries

• Institutionalizing ST statistics
• Establishing registers
• Structural issues in the private sector and the
  private not-for-profit sector
• User-producer networks
• Science Technology Management Information
  Systems and other secondary sources
• Survey procedures and estimation

21
Institutionalization of ST statistics

• Political support
• Infrastructure and sustained staff
  training/capacity building
• Involvement of NSOs Official statistics status
  for RD surveys.
• Adequate legal framework

22
Establishing registers

• RD in developing countries tends to be very much
  the purview of public bodies
• Recommendations
• Establishing a database of public sector RD
  projects
• include human and financial resources align with
  national policies.
• design could reflect the RD statistical
  reporting/definitions.
• source for evaluation of such projects.
• Establishing STMIS
• provide overview of research system.
• framework for establishing complete registers as
  sample frames for RD surveys.

23
Establishing registers

• Other sources
• associations (trade, academic).
• learned societies.
• registers or databases of scientists and
  engineers.
• database of research grants.
• databases of scientific publications.
• patents and other IP documents.
• business registers.

24
Structural issues in the private sector and the
PNP sector

• Publicly-owned businesses play a major role in
  RD in some developing countries
• Recommendations
• should consider issuing data for publicly-owned
  businesses separately from the fully private
  enterprise sector.
• private enterprises could also be disaggregated
  by ownership, in particular the various degrees
  of foreign ownership.

25
Structural issues in the private sector and the
PNP sector

• Business enterprise RD is presumed to be
  generally weak in developing countries when
  compared to industrial countries.
• Recommendations
• take into account when conducting sample surveys,
  perhaps by over-sampling, especially amongst
  larger companies.
• big companies should not be missed out as it
  might imply significant error.
• invest time in interviewing key firms to
  understand their RD function and obtain a clear
  picture of their activity.
• Private-non-profit sector make a significant
  contribution to RD in developing countries, but
  the sector tends to be very volatile.

26
User-producer networks

• Recommendations
• user-producer networks and other forms of
  stakeholder consultation should be instituted.
• establishing national ST statistics groups.
• involve multiple actors.
• coordinating/networking among institutions/databas
  es.
• partnering with business associations.
• conducting face-to-face visits by statisticians
  and project leaders.
• exploit pre-existing personnel ties.
• get NSO involved to deal with privacy of
  information.
• training of interviewers/primary data producers.

27
Science and Technology Management Information
System and other secondary sources

• STMIS (e.g. database of scientists, research
  grants, etc) frequent source for the production
  of RD statistics.
• Recommendations
• need close integration between the statistical
  system and the STMIS.
• need adjustments to produce comparable
  statistics, taking into account issues of
  definitions and coverage.
• need a balanced approach using both STMIS and
  surveys.
• need different approach to Private sector
  organizations as they are frequently not covered
  by these systems.
• Combined RD and innovation surveys
• Recommendations
• the relative rarity of occurrence of RD in
  businesses needs to be taken into account.

28
Survey procedure and estimation

• Recommendations
• attention needs to be paid to questionnaire
  design.
• frequency of survey.
• prioritize area of work accompanied by
  step-by-step approach.
• use of survey questionnaires of other countries
  for inspiration need adaptations to local
  situation.
• get expertise from the NSO, in conducting survey,
  in sampling .
• different questionnaires might be designed for
  different sectors based on stakeholder
  consultations. One size does not fit all.
• procedures need to be developed for estimating
  missing data.

29
Thank you!

• http//www.uis.unesco.org
• m.schaaper_at_uis.unesco.org