Cornerstones of a Quality Survey

1
Cornerstones of a Quality Survey

• Sample surveying is the art and science of coming
  close!
• Sample surveys yield accurate results when
  researchers succeed in avoiding four kinds of
  error
• coverage error
• sampling error
• measurement error
• non-response error

2
Coverage Error

• Every member of the population that the
  researcher is trying to describe would have an
  equal chance of being selected for the sample.
  Coverage error is thus avoided
• sampling frame survey population
• ( list of respondents from (subset of the
  frame)
• which sample is drawn)

Avoid Coverage Error
Consider who might have been excluded from the
survey and how they might differ with respect to
characteristics important in the study
3
Sampling Error

• Enough people would be sampled randomly to
  achieve the needed level of precision. Thus,
  sampling error is avoided.
• Can never be totally avoided unless you do a
  census. Remember only a subset of the population
  is used.
• Control by increasing sample size

Avoid Sampling Error
Control error by selecting proper sample size
how much error can you tolerate? . 5 or do
you require more precision?.
4
Measurement Error

• Clear, unambiguous questions would be asked so
  that respondents are both capable of and
  motivated to answer correctly.
• Thus, measurement error is avoided.
• Respondents answers to given question is
• inaccurate
• imprecise
• cannot be compared usefully to other respondent
  answers

5
Measurement Error

• Method effect - mail, telephone survey,
  face-to-face survey may all yield different
  results.
• Questionnaire - words may confuse respondents,
  structure of question may be poor, answers may
  not be mutually exclusive.

6
Measurement Error

• The interviewer- lead respondents, bias answers,
  reinterpretation of questions to respondents.
• The respondent- May interpret questions
  differently according to their life experiences
  or perceptions. Cant get the correct answer

Avoid Measurement Error
Avoid biased or vague questions or other areas
where measurement error might creep into the
results of the study.
7
Non-response Error

• Everyone in the sample responds to the survey, or
  non-respondents are similar to respondents on
  characteristics of interest in the study.
• Thus, non-response error is avoided.
• Under-counts result from non-respondents may lead
  to inaccurate interpretation of survey results.

Avoid Non-response Error
Design the questionnaire and implement the
surveys to avoid non-response error. Response
under 60-70 is a red flag generally!
8
Improve Survey Quality

• Control for
• coverage error
• sampling error
• measurement error
• non-response error

9
Decide what information you need.

• What problems are you trying to solve?
• What new information do you need to solve it?
• What is the purpose of the survey?
• Identify the problem
• Clarify vagueness
• Eliminate bias
• Focus clearly on what you need to know not
  nice to know

10
Think in Terms of Results

• What do you want to get from the survey?
• Conceptualize this by preparing hypothetical
• tables of results. This
• forces us to separate need to know from nice
  to know

• Focus Groups or directed discussion groups.
• Help us to develop clarity in knowing which
  questions to ask in our survey. Stimulates
  thinking on a particular subject.

11
Choosing a Survey Method

• Mail Surveys
• Telephone Surveys
• Face-to-Face Interviews
• Drop-off-Survey

12
Choosing a Method

• Evaluate the method based on
• Specific Study topic and population
• Budget- how much can you spend on the project in
  time, effort and financial resources
• Staff - volunteers, paid, experienced
  non-experienced
• Time constraints - How soon do you need the
  results of the survey? What results must be
  produced?

13
Mailed Surveys

• Best suited for -
• When reliable addresses are available.
• Respondents likely to respond accurately and
  completely in writing.
• When immediate turnaround is not required.
• When money, qualified staff and professional help
  are scarce.

• Greatest weaknesses -
• Non-coverage error, incomplete lists.
• Some are less likely to respond than others,
  response error.
• Respondent ability to read and follow directions.
• Little control over who fills the survey out.
• Skipped items - item non-response.

14
Telephone Surveys

• Best suited for-
• Populations likely to have telephones.
• Questions are straightforward.
• Experienced help is available to conduct the
  interview.
• Quick turn around time is important

• Greatest weaknesses -
• Not all people have telephones.
• Directories are incomplete and inaccurate,
  multiple listings. ( random digit dialing and
  add-a-digit dialing)
• Sensitivity to measurement error. Leading
  question interviewers.

15
Face-to-Face Interviews

• Best suited for -
• Surveying a population for whom there is no list.
• Collecting responses from those who are unlikely
  to respond via mail or phone.
• Complex questionnaires
• well funded projects - cost of professional
  interviewers

• Greatest weaknesses -
• Expensive, and time consuming.
• Callbacks add greatly to cost.
• Requires well trained interviewers.
• Temptations to cut cost be reducing number of
  interviews -sampling errors.

16
Drop-off Surveys

• Best suited for -
• Small community surveys in which are in a small
  geographic area.
• Short and simple questionnaire.
• Projects with small staff but large sample size.

• Greatest weaknesses -
• Time to get questionnaires delivered and picked
  up.
• In ability to deliver questionnaire to the right
  individual.
• Delivery to a mail box which is illegal!

17
Selecting a Survey Sample

• A sample is a set of respondents selected from a
  larger population for the purpose of the survey.
• How much sampling error is acceptable? - Function
  of how large the sample needs to be.
• Three steps in sampling
• define the survey population
• obtain an adequate population list - list frame
• select the sample

18
How Large Should a Sample Be?Sample size depends
on

• How much sampling error can be tolerated.
• Smaller samples cheaper to collect info.
• Population size.
• Small populations gain little from sampling
  especially if need for precision is great.

• How varied the population is with respect to the
  characteristics of interest
• The smallest subgroups within the sample for
  which estimates are needed.

19
Sampling Rules of Thumb

• For large populations 100,000 - 1,000,000
• we need approximately the same sample size. Only
  when we compare groups of less than several
  thousand does sample size make a difference.
• The more diverse or variable populations require
  larger sample sizes. Take the conservative
  approach with respect to sample size

20
Sampling Rules of Thumb

• Sample size is influenced by not how large the
  overall population is but sample size is based on
  how big the subgroup being studied is. Sampling
  error depends on the size of the subgroup.
• Sampling error rates - Confidence Levels
• 3
• 5
• 10

See Sampling Table
21
Finding Good Lists

• Easy for small specific populations
• More difficult for general populations of state
  or national level.
• Get official sponsorship to improve list
  availability.
• Concept that every member of population has an
  equal and independent chance of selection

22
Uncomplicated Sample Designs

• Simple random sampling (SRS)
• a lottery (drawing out of a hat)
• random number table - how to use a table
• computer generated lists of random numbers
• Probability and Non-probability Sampling
• Non-probability- purposive sampling
• Results generalized from group likely not
  probable
• Focus groups good example

23
Sampling Considerations

• Face -to- Face Surveys
• small area cluster sampling Map based block map
  samples
• advantages in reducing travel time to collect
  sample
• larger area cluster sampling- two-stage approach
  400 blocks 40 selected then sample within this
  frame.

• Telephone Surveys
• avoid coverage errors
• Random digit dialing (RDD)
• 3 digit prefix plus random computer generated 4
  digit extensions
• Add-a-digit sampling
• random number selected e.g. 5 added to random
  selected numbers
• 883-0527. 5 .883-0532

24
Issues to Consider When Writing Survey Questions
Operationalizing Your Questions

• avoid emotional and biased questions
• how specific the question should be
• whether the questions will produce credible
  information
• whether respondents are able to answer the
  questions
• whether respondents will be willing to provide
  the information.

25
What Kind of Information are You Looking For?
Clarify What You Want to Know

• Four Types of Questions

• What people do or what they are
• Behavior or Attributes
• What people say they want or what they think is
  true.
• Attitudes or Beliefs

26
Open- Ended Questions

• Open-ended questions do not provide choices to
  select from. Respondents formulate the answer
• Drawbacks
• demanding for respondents
• difficult to analyze the data- time to prepare
  for computer entry or hand analysis
• produce many different responses
• rarely provide accurate consistent measurement
  when comparing whole group

27
Open-Ended Questions

• Advantages
• You have little prior knowledge about an area
• Want to give respondents opportunity to state
  strong opinions, vent frustrations
• Followed by a close ended question - respondents
  explain response
• When respondents are asked to estimate a routine
  behavior and they are likely to know an exact
  number
• When a precise piece of information is needed and
  cannot easily be listed in a closed ended
  question.

28
Close-ended with Ordered Response

• Each question choice represents a graduation of a
  single concept.
• 1. Under 25 years
• 2. 26-35 years
• 3. 36-45 years
• 4. 46-55 years
• 5. 56-65 years
• 6. over 65 years
• Less demanding for respondent, quite specific and
  easier to code and analyze.

29
Close-ended with Unordered Response Choices

• Responses do not fall on a continuum
• Unordered response choices generally more
  difficult to answer
• Often used to rank items -
• rank these items 1-6.
• these types of questions are difficult with
  telephone surveys. We loose track and cant
  remember.

30
Partially Close-ended Questions

• Compromise between open-ended and close-ended
  structures. Respondents have the opportunity to
  add their own response.
• Other ( Please Specify)__________________
• this format rarely yields additional responses
• Gives respondent choice to not be boxed in with
  their response. May generate new information

31
Which Question Structure Shall I Use?

• Question structure affects how people respond.
  Category Effect
• Mail survey -respondents more likely to choose
  first categories - telephone and face-to-face the
  last categories
• When list of choices gets too long
• Abstract ideas about which people havent given
  much thought. Use a series of leading questions-
  Cognitive Design
• Use a series of questions rather than one complex
  question - Scaling Technique
• Be aware of how category effect can increase
  measurement error.

32
There are NO Silver Bullets When Writing Survey
Questions

• Be Specific
• Use simple Words
• Dont be Vague
• Keep it Short
• Dont talk down to respondents
• Dont Be too specific

Conflicting Rules
33
Avoid these Question Traps

• Vaguely worded questions and responses
• Abbreviations or jargon that respondents may not
  understand
• Too much precision
• Bias from slanted introduction

• Bias from unequal comparison
• Bias from unbalanced response choices
• Bias from tone of question
• Objections to providing income information

34
Avoid these Question Traps - continued

• Objectionable statements
• Questions that are too difficult for respondents
• Double-barreled questions
• Answers that are not mutually exclusive

• Too much knowledge is assumed on the part of
  respondent
• Inaccurate statements
• In appropriate time references
• Responses that cant be compared with existing
  information
• Cryptic questions