Greg Griffiths

1
A Coherent Approach to Questionnaire Design
Standards, Trends, and Implementation at the ABS
Greg Griffiths for Rob Burnside Emma Farrell
ICES 3 Montreal June 2007
2
Overview

• How ABS enterprise questionnaire design standards
  and guidelines are
• implemented organizationally
• created and maintained
• scope and content of the standards
• trends - changes over time
• role of form design methodologists
• new modes and questionnaire structures and
• the Total Respondent experience

3
Big Picture

• Mandatory use of agreed standards for in-scope
  instruments detailed in Corporate policy manual.
• Common conventions, look and feel. Bringing about
  use of best practice.
• Organization arrangements give responsibility to
  areas for different aspects of final forms
• Form development process is supported by
• Standards manual containing guidelines and
  examples available on the ABS intranet
• Organizational structures incl survey-specific
  meetings, an annual review of core questions and
  
• A forms image database with workflow/ sign-off
  process Holds images and linked discussion for
  interim, current and historical forms back to 1998

4
Responsibilities of areas involved

• Methodology Division Forms Consultancy Group
  (FCG)
• maintain and extend standards and guidelines
• understand mode effects and try to minimize
• RD into form design (all modes)
• Operational sections - manage survey process and
  produce output
• Economic Statistics Data Centre - includes a
  specialised area constructing forms to standards
  and coordinating testing
• Conceptual sections - establish user needs
• Economic Standards Section - consistency of core
  economic questions and classifications
• National Accounts Branch - to ensure inputs for
  Australian National Accounts

5
(No Transcript)
6
Scope and content

• Form standards cover
• format, structure and question wording
  principles, common components
• all modes - paper mail out, fax, CATI, Excel,
  TDE/IVR
• detailed specifications and examples for
  different types of questions and forms
• contact letters and emails
• Development and evaluation standards cover
• observational studies, usability testing,
  cognitive testing ...
• suitability of study design for answering
  development questions.

7
Creation and maintenance of standards

• Relies on maintaining knowledge of ABS and
  international theoretical and applied research
  and experience
• Maintenance and planned updating of standards
  based on
• Advice on development methods and approaches
• Emerging issues
• Involvement in field testing
• Organizing experiments to examine aspects of
  design form length effects, scanning design,
  effectiveness etc.
• Urgency of development need

8
Trends 1 - Evolving role of ABS form design
methodologists (FCG) ctd

• Pre 2000 Policing layout standards
• 2000 Methodology standards established. Need to
  examine broader measurement error recognized.
• 2001 More modes and more development methods
• 2002/03 Total Approach Management and a wider
  view - every interaction with the respondent is
  significant - appropriate, consistent and
  effective contact
• 2007 Multimodal Data Collection project funded
• 2007 Move of household based collections into
  same branch as FCG. New structure under
  consideration.

9
Trends - 2 more modes and question types

• More modes
• structure, presentation and wording changes to
  keep things the same conceptually
• implies more, different testing and development
  techniques
• increase testing and creation cost
• form design increasingly intertwined with IT
  capability and applications
• More question structures
• Complex
• Subjective (i.e. non-economic)
• More like household questionnaire questions

10
Trends 3 - More development techniques

• More development techniques
• right technique to explore the right thing
• more use of cognitive and exploratory methods
• Post Enumeration Studies, behavior coding, card
  sorting, interviewer debriefing
• Expert review
• Classic usability testing for eform and CATI
• Classic experimental design where appropriate

11
Trends 4 - Total Approach Management

• Every interaction with a data provider is
  important. The form is more than the form.
• pre approach and covering letters,
• brochures and explanatory material,
• Telephone, fax and e-mail contacts
• The response experience is more than filling in
  the form
• Response enhancing techniques - social
  influence theories
• mode of choice, incentives .

12
Summary
Strong senior management recognition of the
importance of form design which is reflected in
organizational structure. Formal form
development process with defined roles and
sign-off. Corporate systems ensuring
communication of standards. Visibility of
deliberations decisions.
13
Summary

• A specialized group of methodologists exists
  outside operational areas
• Responsible for standards and development
  methods
• Involved in day-to-day form development as well
  as RD into respondent related measurement
  error
• Centralization of methodology for all modes and
  methods ensuring consistency
• Maintains critical mass of expertise
• Maintains a strong commitment to research and
  the communication of results / new developments
  to operational areas