Survey methodology

1
Survey methodology

• Cristina Giudici
• University La Sapienza
• Roma

2
References

• Groves R.M. et al. Survey methodology,
  Wiley-interscience, 2004
• Canada national statistical agency
  http//www.statcan.gc.ca/
• Eurostat methodological publications
• European Values Study (EVS)

3
INTRODUCTION
What is a survey ?
A survey is a systematic method of gathering
information from (a sample of) entities for the
purpose of constructing quantitative descriptors
of the attributes of the larger population of
wich the entities are members
A survey usually originates when an individual or
institution is confronted with an information
need and the existing data are insufficient
What is survey methodology ?
Survey methodology is the study of survey methods

4
How survey work to produce statistics
Characteristics of the population
Inference
Characteristics of a respondent
Statistical computing
Characteristics of the sample
Inference
Respondent answers to quenstions
5
Chapter IThe life cycle of a survey
6
A survey from a process perspective
Define research objectives
Choose mode of collection
Choose sampling frame
Construct and pretest a questionnaire
Design and select sample
Recruit and measure sample
Code and edit data
Make postsurvey adjustments
Perform analysis
7
The life cycle of a survey from a design
perspective
Target Population
Construct
Sampling Frame
What is the survey about?
Who is the survey about?
Measurement
Sample
Response
Respondent
Edited Response
Postsurvey Adjustments
Survey statistics
The Measurement dimension describes what data are
to be collected about the observational units in
the sample
The Representational dimention concerns what
population are described by the survey
8
The measurement dimension
Constructs are the elements of information that
are sought by the researcher How many incidents
of crimes with victims there were in the last
year The consumption of beer in the last
month The degree of knowledge of mathematics of
childrens
Construct
Measurements are ways to gather information about
constructs Questions posed to a respondent
(During the last 6 month, did you call the
police to report something that happened to you
that you thought was a crime?) NB the critical
task for maesurement is to design questions that
produce answers reflecting perfectly the
construct we are trying to measure.
Measurement
Response could be produced in a variety of
means But in general the nature of the response
is determined by the nature of the measurement
Response
Editing of data may examine the full distribution
of answers and look for atypical patterns of
responses Edited responses are the data from wich
inference is made about the values of the
construct for an individual respondent
Edited Response
9
The representational dimension
The target population is the set of unit to be
studied The adult population living in households
in 2009
Target Population
The frame population if the set of target
population members that has chance to be selected
into the survey sample In a simple case it is a
list of all units in the target population, but
sometimes it is a set of units imperfectly linked
to population members. i.e. a list of telephone
numbers when the target population is the adult
population
Sampling Frame
The sample is the group from wich measurement
will be sought. In many case it is a very small
fraction of the the sampling frame
Sample
Respondents are the elements successfully
measures. Non respondents is the complement
Respondent
Postsurvey adjustments consist on weighting up
the underrepresented groups in order to improve
the survey estimate Because of mismatches of the
sampling frame and the target population
(coverage problems) statistics based on the
respondents can differ from caracteristics of the
target population. Examination of non response
patterns may suggest an underrepresentation of
some groupes relative to the sampling frame
Postsurvey Adjustments
10
The life cycle of a survey from a quality
perspective
Target Population
Construct
Coverage error
Validity
Sampling Frame
Measurement
Sampling error
Measurement error
Sample
Nonresponse error
Response
Respondent
Processing error
Adjustments error
Edited Response
Postsurvey Adjustments
Survey statistics
11
Coverage of a target population by a frame
Undercoverage Elements in the target population
missing from the frame i.e.non telephone
household, using a telephone frame to cover the
full household population
Frame population
Ineligible units
Ineligible units Elements in the frame that are
no member of the target population i.e.business
telephone numbers, using a telephone frame to
cover the full household population
Covered population
Undercoverage
Target population
12
Evaluating survey questions

• Are the answers good measures of the intended
  construct?

Exemple of methods that can be used to evaluate
draft survey questions
The substantive expert review the wording,
the order and the structure of questions, the
response alternatives etc.

• Expert reviews

A small number of target population
participate in a systematic discussion about the
survey topic. The researcher learn about the
nomenclature of the concept, the common
perspective taken by the target population on key
issues etc

• Focus groups

Researcher test how questions are read and
answered. A behaviour coding is often used

• Questionnaire pretest

13
Evaluating survey questions
Exemple of behavior codes for interviewer and
respondent behaviors

• Interview Questioning Behavior (choose one)
• Reads questions exacty as worded
• Reads questions with minor changes
• Reads questions so that meaning is altering
• Respondent Behaviors (check as many as apply)
• Interrupt question reading
• Asks for clarification of question
• Gives adequate answer
• Gives answer qualifies about accuracy
• Gives answer inadequate for questions
• Answers dont know
• Refuses to answer

14
Chapter IIMethods of data collection
15
Traditional data collection methods

• Mailing paper questionnaires to respondents, who
  fill them out and mail them back
• Having interviewers call to respondents on the
  telephone and ask them the question in a
  telephone interview
• Sending the interviewers to the respondents
  home or office to administer the questions in
  face-to-face (FTF) interviews

16
Alternatives methods of data collection
OCR/ICR Optical/intelligent caracter recognition
FAX
Computerised Self Administered Questionnaires
Disk by Mail
E-mail
Web
Mail
Telephone
CATI computer assisted telephone interviewing
TDE Touchtone data entry
IVR Interactive voice response

• Face to face

CAPI computer assisted personal interviewing
Video CASI
SAQ Self administered questionnaire
Text CASI
Audio CASI
Walkman
17
Alternatives methods of data collection (a)
OCR/ICR Optical/intelligent caracter recognition
FAX
Disk by Mail
E-mail
Web
Mail
18
Alternatives methods of data collection (b)
CATI computer assisted telephone interviewing
TDE Touchtone data entry
IVR Interactive voice response
Telephone
19
Alternatives methods of data collection (c)

• Face to face

CAPI computer assisted personal interviewing
Video CASI
SAQ Self administered questionnaire
Text CASI
Audio CASI
Walkman
20
Chapter IIIDESIGNING A QUESTIONNAIRE
21
QUESTIONS AND ANSWERS IN SURVEYS
A questionaire is a standardised set of
questions administered to the respondents in a
survey
Respondents are required to interpret a
preestablished set of questions and to supply the
information these questions seek.
22
The cognitive processes in answering questions
A simple model of the survey response process
BUT
Respondents often take shortcuts to get through
the interview more quickly
Comprehension of the question
OR
Retrieval of information
they have motives that override their desire to
provide accurate information
Judgment and estimation

• Responses could be biased by
• acquiescence (the tendency to agree)
• social desirability (the tendency to present
  oneself in a favourable light by underreporting
  undesirable attributes and overreporting
  desirable one)

Reporting an answer
23
The cognitive processes in answering questions (2)
The satisficing model (Krosnik and Alwin,
1987)
Some respondents try to satisfy (to take a low
road answering more superficially) whereas
others try to optimise (to take an high road by
careful answering questions)
Satisficing respondents do not seek to understand
the question completely, just well enough to
provide a reasonable answer
24
Problems in answering survey questions

• Failure to encode the information sought

• Misinterpretation of the questions

• Forgetting and other memory problems

• Estimation strategies

• Problems in formatting answer

• More or less deliberate misreporting

• Failure to follow instruction

25
FORMATTING THE ANSWER

• Survey items can take a variety of formats the
  most common are
• Open-ended qustions that call for numerical
  answers
• Closed questions with ordered response scales
• Closed questions with categorial response options

26
1 - Open-ended qustions that call for numerical
answers

• Now, thinking about your physical health,
  which includes physical illness and injury, for
  how many days during the past 30 was your
  physical health not good?

Note that Open-ended items yield more exact
information than closed items
27
2 - Closed questions with ordered response scales

• Would you say that in general your health
  is
• Excellent
• Very good
• Good
• Fair
• Poor

The interviewer is instructed to please read
the answer categories, but not the number
attached to them!

• Note that
• with some type of rating respondents seem to shy
  away from the negative end of the scale
• When the scale points have numerical labels, the
  label can affect the answer (e.g. if respondents
  are asked to rate their success in life)

28
3 - Closed questions with categorial response
options

• Are you
• Married
• Divorced
• Widowed
• Separated
• Never married
• A member of an unmarried couple

• Note that
• The respondent may not wait to hear or read all
  the option they may select the firs reasonable
  answer they consider (primacy effect)
• The opposite coul happen the last option the
  interviewer read may be the first one that
  respondent think about (recency effect)

29
GUIDELINES FOR WRITING GOOD QUESTIONS (Sudman and
Bradburn)

• Non sensitive questions about behavior

The key problem with many questions about
behavios is that respondents may forget some or
all of the relevant information, or that their
answer may reflect inaccurate estimate
In order to reduce memory problems it is
essential to play attention to the wording of the
question and to provide memory help
Attitude questions
Attitude questions are a very commen class of
survey questions. The most frequent problems
deals with the wording of questions, the question
order and the format of response scales
30
Non sensitive questions about behavior Play
attention to the wording
With closed questions, include all reasonable
possibilities as explicit response options

• Are you
• Married
• Divorced
• Widowed
• Separated
• Never married

• Are you
• Married
• Single

31
Non sensitive questions about behavior Play
attention to the wording
Make the question as specific as possible (about
who it covers, what time period, which
behaviours)

• In a tipical week, how often do you read a
  newspaper?

Over the last month, that is .. how often do
you read a newspaper in a tipical week?
32
Non sensitive questions about behavior Play
attention to the wording
Use words that virtually all respondents will
understand
Have you ever had a heart attack?
Have you ever had a miocardial infarction?
33
Non sensitive questions about behavior Provide
memory help
Uses aided recall (or ask separate questions
about subcategories)
Please look cerefully at the following list
of volountary organisations which, if any, do
you belong to? A Religious organisations B Cultur
al organisations C Political groups D Other
To which volountary organisation do you
belong to?
34
Attitude questionsPlay attention to the wording
Clearly specify the attitude object of interest
Do you think the Government is spending
too litte, about the right amount, or too much on
education?
Do you think the Government is spending
too litte, about the right amount, or too much on
higher education?
35
Attitude questions play attention to the wording
Measure the strength of the attitute using a
response scale, a separate item or multiple
items that can be combined into a scale

Do you agree or disagree with the
following statement? Government is spending too
little on education 1 Agree strongly 2
Agree 3 Neither agree nor disagree 4 Disagree 5 D
isagree strongly
note that an attitude have generally a direction
(agree or disagree) and an intensity (strongly
disagree.strongly agree)
36
Attitude questions reduce impact of question
order

When asking general and specific questions about
a topic, ask the general question first
(otherwise, the answer to the general question
is likely to be affected by the number and
content of specific questions)

When asking questions about about multiple items,
start with the least popular (the unpopular
questions are likely to seem even less appealing
when they follow more popular questions)
37
When asking general and specific questions about
a topic, ask the general question first

• Please tell me whether or not you think it
  should be possible for a pregnant woman to obtain
  a legal abortion if the woman wants it for any
  reason? 1. Yes 2. No 3. Don't know

Please tell me whether or not you think it should
be possible for a pregnant woman to obtain a
legal abortion if there is a strong chance of a
serious defect in the baby? 1. Yes 2. No 3.
Don't know
USA GSS
38
Attitude questions play attention to the
response scale


Use closed questions for measuring
aptitudes(open answers are difficult to code)
Use five-to seven-point response scales and
label every scale point(verbal label ensure that
interpret the scale in the same way)
39

• Now is up to you!