Mediating between Survey Users and Survey Respondents

1
Mediating between Survey Users and Survey
Respondents

• Roger Thomas
• Balkans Summer School 2006 Questionnaire Design 3

2
Aims of survey question design

• The question designer must mediate between survey
  users and survey Rs, taking account of -
• information needs of survey users
• survey concepts and definitions
• required levels of measurement
• how to minimise response error
• respondent psychology

3
Question design compromises 1

• In controlling the communication process the
  questionnaire designer aims to minimise -
• data collection costs
• interview time
• burden on respondents.

4
Question design compromises 2

• But she must also try to obtain from Rs
  information that is sufficiently -
• precise
• complete
• reliable
• sensitive to real differences
• These aims often clash with one another.

5
Steps towards a good question

• Analysing survey aims and concepts
• Reading relevant theory and research, consulting
  experts
• Reviewing other questionnaires (see e.g.
  http//qb.soc.surrey.ac.uk)
• Doing preliminary qualitative research
• Trying out drafts on colleagues
• Formal question-testing pilots

6
Respondent psychology

• Survey information needs must be turned into
  questions such that Rs -
• correctly interpret the question wording

7
Respondent psychology

• Survey information needs must be turned into
  questions so that Rs -
• correctly interpret the question wording
• have the knowledge needed to answer

8
Respondent psychology

• Survey information needs must be turned into
  questions such that Rs -
• correctly interpret the question wording
• have the knowledge needed to answer
• are willing in principle to answer

9
Respondent psychology

• Survey information needs must be turned into
  questions such that respondents -
• correctly interpret the question wording
• have the knowledge needed to answer
• are willing in principle to answer
• are willing to work hard to give accurate and
  precise answers

10
Failure of understanding

• If a question is not interpreted by all Rs in the
  same way, what might be the cause?
• Wording not consistently understood
• Reference of question not understood
• Purpose of question not understood
• Degree of accuracy needed not understood

11
Lack of relevant knowledge

• Why might a respondent lack the knowledge needed
  to give a good answer?
• Lack of relevant life experiences.
• Failed to memorise relevant details.
• Failure to recall relevant details.
• But ignorance often does not stop respondents
  from giving an answer.

12
Lack of willingness

• Why might a respondent be unwilling to give an
  accurate answer?
• Too much hard cognitive work involved
• To give a true answer would be embarrassing
• R feels that the researcher/interviewer was
  hoping for a different answer

13
Conceptual and operational definitions

• Factual questions are made of definitions.
• It is not enough to to provide precise conceptual
  definitions.
• These must be -
• turned into language that all Rs will understand
  consistently, and then -
• incorporated into clear survey questions.

14
Respondent suggestibility

• Rs are often not sure how to answer a survey
  question (e.g. opinion questions).
• But they tend not to say I dont know or I
  dont understand.
• Instead, they look for clues about what would be
  a safe, acceptable answer.
• They may seek clues in response options prompted
  as part of the question.

15
Response tasks

• Consider this question
• Please say how your health has been over the
  past 6 months.
• Exactly what cognitive tasks should R carry out
  to provide a satisfactory answer?
• If the researcher cannot say, this cannot be a
  good question.

16
Concepts and definitions Health

• Are we aiming to cover both mental and physical
  health?
• Should we include disabilities?
• How can we standardise/operationalise the concept
  of health?
• What level of measurement is needed?
• How sensitive must the measure be to health
  differences?

17
Formal measurement criteria

• Quantitative aim to measure attributes of
  persons, households etc.
• Measures should be -
• sufficiently precise for purpose
• reliable (random variance not too great)
• free of systematic bias
• able to detect real differences
• portable from one survey to another

18
Levels of measurement 1

• The question designer must decide at which level
  of measurement she is aiming
• ratio level
• Interval level
• ordinal level (including vague quantification)
• categorical level

19
Levels of measurement 2

• Levels of measurement differ in two ways that are
  very important for surveys.
• Some levels provide data that contain more
  statistical information than others.
• Some levels demand more effort from respondents
  than others.
• The more informative levels tend to be more
  demanding on respondents.

20
Ratio/interval-level measures 1

• The underlying operation is counting.
• Examples
• a persons height in cm (ratio)
• number of films seen over 4 weeks (ratio)
• standardised scores on a test (interval)
• a households annual income in Kn (ratio)
• The answer is a meaningful number.

21
Accurate answers hard work

• If Rs understand the question wording and are
  willing to answer, they may give an answer that
  seems OK.
• Most research users are happy to accept such an
  answer - but it is not enough.
• Rs may reach their answer-
• in different ways (non-comparability)
• in ways not intended by the researcher

22
How respondents avoid work

• If R is unwilling or unable to give an exact
  response, he may -
• give a vague answer
• make an estimate, using a good or a bad method
• give any answer that he thinks will satisfy the
  interviewer/researcher.
• It depends how hard he is willing to work.

23
Making the response task easier

• Survey Rs may be unable to give sufficiently
  precise numeric responses.
• Question designers have ways of making the
  response task easier.
• But these can -
• damage comparability
• lower the level of measurement
• allow response biases to creep in.

24
Ordinal measures

• Putting concepts, things or persons in order
  according to some criterion.
• e.g. Assigning ranks to each of a series of
  course presentations
• 1 most useful ltgt N least useful
• Note that perceived differences between ranks may
  not be equal and may differ for different Rs.

25
Vague quantification

• If she thinks Rs cannot or will not provide
  precise answers, a designer may use vague
  quantification devices, e.g. -
• value ranges (e.g. 0-5 ltgt 20 or more)
• verbal rating scales (e.g. very good/ fairly
  good/ not very good/ poor)
• words in the question text such as usually, or
  in a typical week

26
Categorical measurement

• Non-ordered classifications are common in surveys
  e.g.
• Yes/ No/ Dont know response options.
• classification of households by area of residence
• classification of persons by occupation
• classification (usually created by the
  researcher) of reasons for a decision

27
Classifications

• Some methods of classifying are standard in many
  surveys and censuses e.g.
• economic activity status
• ethnic group
• marital/family status
• To construct some classifications (e.g.
  occupation, family structure) responses to
  several questions may be needed.