The Use of Administrative Sources in the Evaluation

1
The Use of Administrative Sources in the
Evaluation Design of Data Collection Instruments

• Jacqui Jones, Sarah Williams, Marilyn Thomas
• Data Collection Methodology ,
• Methodology Group,
• Office for National Statistics

2
Overview

• ONSs statutory business surveys
• The framework for improving data collection
• Our evaluation approach
• Potential problems with data collection
• Administrative data
• Evaluation data
• NES evaluation review results
• E-Commerce Post-implementation evaluation review
  results
• Next steps

3
ONSs Statutory Business Surveys

• Number
• Types
• Modes of data collection
• Layout and structure
• How questionnaires have been developed
• Review programme

4
Framework for Improving Data Collection

Identification of data requirements
Stage 1
Expert Review
Evaluation Review


Based on results from stage 1, identify areas for
further development
Stage 2
Testing and Evaluation
Stage 3
Implementation
Stage 4
Post implementation evaluation
Stage 5
Identify and disseminate best practice
Stage 6
5
Our Evaluation Approach

• What type of evaluation?
• Case study evaluation approach
• Evaluation reviews - formative evaluation
• intended to help in the development of the
  programme
• Post-implementation reviews
• - summative evaluation
• assesses the effects and effectiveness of the
• programme

6
Our Evaluation Approach

• Triangulation
• to employ more than one method of
  investigation and hence more than one type of
  data
• (Bryman, 1998. p. 131)
• Triangulating data sources
• Comparing and cross-checking the consistency
  of information derived at different times, by
  different means, and from difference
  persons/sources

7
Potential Problems with Data Collection

• Non-sampling error
• Measurement error
• Errors that occur during the data collection
  phase (Eurostat, 2002)
• Processing error
• Errors introduced through coding, keying,
  editing, weighting tabulating (Eurostat, 2002)
• Also
• Respondent burden

8
Administrative Data

• Administrative data can be defined as
• data that are collected for one purpose and used
  for another purpose

9
Evaluation Data

• Quantitative data
• Analysis of a systematic sample of questionnaire
  images that have failed validation
• Survey results
• Qualitative data
• Focus groups DVB, RAP and FPC
• Written/verbal respondent comments

10
Evaluation Review Results

• New Earnings Survey
• Overview of questionnaire design

11
Evaluation Review Results

• New Earnings Survey
• Identified measurement error themes
• data item non response
• change of response
• problems with understanding questions/definitions

12
Evaluation Review Results
13
Evaluation Review Results

• New Earnings Survey
• Identified processing error themes
• change of response
• unclear numerical data
• crossing out of questions

14
Evaluation Review Results
15
Evaluation Review Results

• New Earnings Survey
• Identified respondent burden theme
• Unnecessary response

16
Evaluation Review Results
17
Post-Implementation Evaluation Review Results

• E-Commerce
• Overview of questionnaire design

18
Post-Implementation Evaluation Review Results

• E-Commerce
• Identified measurement error themes
• data item non response
• change of response
• problems with understanding questions/definitions
• problems following instructions/routing

19
Evaluation Review Results
20
Post-Implementation Evaluation Review Results

• E-Commerce
• Identified processing error themes
• marked both Yes and No boxes
• entered X in a numeric field
• all ? instead of X
• partial ?
• change of response
• unclear numerical data
• crossing out of questions

21
Evaluation Review Results
22
Post-Implementation Evaluation Review Results

• E-Commerce
• Identified respondent burden themes
• Unnecessary response due to instructions/routing

23
Evaluation Review Results
24
Next Steps

• continue with review programme using the
  framework for improving data collection
  instruments
• introduce experimental evaluation - split panel
  field test of Business Register Survey

25
Bibliography

• Eurostat (2002) Assessment of the Quality in
  Statistics
• Bryman, A. (1988). Quantity and Quality in Social
  Research. Unwin Hyman
• Patton, M. (1990). Qualitative Evaluation and
  Research Methods. Sage
• Forthcoming
• Jones, J. (2003). A Framework for evaluating and
  redesigning data collection instruments. Survey
  Methodology Bulletin, Office for National
  Statistics

26
Thank you

• Any questions?