Improved Questionnaire Design Yields Better Data: Experiences from the UK

1
Improved Questionnaire Design Yields Better Data
Experiences from the UKs Annual Survey of Hours
and Earnings

• Jacqui Jones, Pete Brodie, Sarah Williams Jane
  Carter
• UK Office for National Statistics

2
Overview

• Data Collection Methodology at ONS
• Methods standards
• Design standards
• Case study The Annual Survey of Hours and
  Earnings
• Slide 1

3
Background to DCM at ONS

• Mid 1990s 1st QTU set-up for Census
• Then expanded to include social surveys and
  business surveys
• Slide 2

4
Methods Standards

• Aim was to develop a framework of processes to be
  used for all DCM question(naire) development work
• Outcome was the Data Collection Methodology
  Improvement and Measurement Framework
• (DCM IMF)
• Slide 3

5
The DCM IMF

Identification of data requirements
Expert Review
Evaluation Review


Stage 1
Redesign Development
Stage 2
Pre-field Testing
Stage 3
Field Testing
Stage 4
Field Implementation
Stage 5
Post-implementation Evaluation
Stage 6
Identify and Disseminate Best Practice
Stage 7
Slide 4
6
The DCM IMF
Expert review
Evaluation review

• Stage 1
• Stage 2
• Stage 3
• Stage 4

Literature review
Instrument development
Expert review
Qualitative testing
Expert review
Re-development
Qualitative follow-up
Quantitative analysis
Slide 5
7
Design Standards

• Aim was to develop tested standards for the
  design of questions and questionnaires
• Outcome is draft standards
• with constraints colour and cross boxes
• Slide 6

8
Example of Design Standards
9
Example of Design Standards
10
Case Study ASHE

• ASHE collects data for earning statistics e.g.
  minimum wage
• ASHE questionnaires are sent out to employers to
  complete on behalf of selected employees
• 250,000 sample of employees
• Slide 9

11
ASHE Need for Change

• National Statistics Quality Review
• New data requirements e.g.
• Pay for other reasons
• Pension data
• Existing data problems e.g.
• Possible inclusion of pay elements in the wrong
  responses
• Slide 10

12
Overview of the Old Questionnaire

• Very cramped 2 sided questionnaire which
  collected 26 data items
• 4 sides of additional notes
• Respondent coding
• Slide 11

13
Old Questionnaire
Slide 12
14
(No Transcript)
15
Old Questionnaire
Slide 14
16
Overview of the Old Questionnaire

• Unclear data requirements
• Inconsistent terminology
• Double-barrelled questions
• Slide 15

17
Old Questionnaire

18
ASHE Development

• Followed DCM IMF processes
• Focus group (8 large businesses)
• Pre-field testing
• 30 cognitive interviews (5 waves of testing)
• Waves 1-3 concurrent interviewing
• Waves 4-5 retrospective interviewing
• Slide 17

19
ASHE Development

• 2004 Field tested 6 sided questionnaire
• Split sample design (98 old , 2 new)
• Quantitative analysis
• Non-response
• Respondent burden (actual perceived)
• Processing errors
• Slide 18

20
Results of ASHE Field Test

• Response (no statistical difference)
• Errors (no statistical difference)
• Respondent feedback (positive)
• Respondent burden (reduced per question)
• Some problems with question wording
• Costs (increased paper, printing, postage,
  processing)
• Time (increased processing time)
• Slide 19

21
6 sided to 4 sided compromises

• Minimum standard based on testing - keep
  question response category wording
  instructions to stay with questions
• Challenge was to promote positive respondent
  perceptions and clear navigational path
• No time for further testing redesign by DCM
  experts quick perception telephone interviews
• Slide 20

22
Old Questionnaire
Slide 21
23
ASHE 6 Sided Questionnaire
24
ASHE 4 Sided Questionnaire
25
2006 ASHE 4 Sided Questionnaire
Slide 24
26
2005 Qualitative Follow-Up

• Looked at differences between responses to the
  2004 old questionnaire and the 2005 4 sided
  questionnaire.
• Focussed on 5 key areas e.g.
• the affect on basic pay of a new other pay
  question
• 31 in-depth telephone interviews
• Slide 25

27
Old Basic Pay Question
Slide 26
28
New Basic Pay and Other Pay Questions
Slide 27
29
Other Pay Results

• Qualitative study found
• Better comprehension of new questions
• Improved reporting of other pay
• Respondents preferred new layout
• Quantitative analysis found
• Old questionnaire - other pay had been
  excluded by some
• Correction by imputation added 0.8 growth to
  basic pay at the all employee level

Slide 28
30
What did we learn?

• Must have
• clear agreed data requirements
• agreement on the maximum number of pages
• DCM experts involved in redesign testing
• method standards
• design standards
• improved methods for measuring data quality
• Slide 29

31

• Thank You
• Jacqui.Jones_at_ons.gov.uk