Discussion Document for Survey Design and Implementation

1

• Discussion Document for Survey Design and
  Implementation
• Jennifer Lewis Priestley, Ph.D.

2
Discussion Topics

• Major Decisions in Survey Design
• Key Issues and Solutions
• Good, Bad and Ugly Examples
• Guidelines for Survey Formats
• Best Practices for Increasing Response Rates

3
The Major Decisions in Survey Design

• What should be asked?
• How will the results be actionable?
• How should each question be phrased?
• In what sequence should the questions be
  arranged?
• How should the questionnaire be pre-tested?
• Does the questionnaire need to be revised?

4
Key Issues and Solutions
Guidelines
Solution

• Avoid ambiguity

• Be clear and concise
• Response choices that are
• mutually exclusive
• exhaustive
• not double-barrelled

• Avoid complexity

• Use conversational, familiar language

• Avoid bias
• (i.e., leading or loaded
• questions, order bias)

• Be objective
• Be open to unpopular response
• Alternate order of questions

5
Key Issues and Solutions
Solution
Guidelines

• Avoid making assumptions

• Offer explicit alternatives

• Avoid questions that cant or wont be answered
• Omissions
• Telescoping
• Creation
• Unaided versus Aided

• Limit recall to reasonable time periods/events

6
Whats Wrong?
Q. How much red meat do you eat each day?

• On a typical day, at how many meals do you
  usually eat beef, pork or lamb?
• Think about the typical size of the serving of
  beef, pork, or lamb you eat. Is it about the
  size of a deck of cards, a small paperback book
  or a hardcover book?

7
Whats Wrong?
Q. When was the last time you used your wifes
hairspray?

• Recent studies have shown a high percentage of
  men use their wives hairspray to control their
  hair. Have you used your wifes hairspray within
  the past week?

8
Whats Wrong?
Q. If taxes must be raised to support education,
would you prefer a regressive or progressive tax?

• If taxes must be raised to support education,
  would you prefer an increase in a sales tax or
  property tax?

9
Whats Wrong?
Q. How do you spend your leisure time, watching
TV or reading?

• How do you spend your leisure time?

10
Whats Wrong?
Q. Would you support an amendment to limit the
Presidential term to one 6-year term?

• Which would you prefer for the Presidential term?
  
• keeping terms to 4-years, limited to 2 terms
• keeping terms to 4-years, change limit to 1 term
• changing terms to 6-years, limited to 1 term
• changing terms to 6-years, limited to 2 terms

11
Whats Wrong?

• Q. At what type of restaurant do you eat?
• Inexpensive
• Expensive
• Very expensive
• All of the above

• For work or school day lunches, would you
  describe the restaurants at which you typically
  eat as
• Inexpensive (less than 5)
• Moderately expensive (5-10)
• Expensive (more than 10)

12
Guidelines for Survey Format

• Introduce the questionnaire - tell the purpose of
  the studyfor mail surveys always provide a cover
  letter.
• Ask easy, non-threatening questions to establish
  rapport.
• Questionnaires should be presented in a logical
  flow - finish one topic before going to the next.
• Can be presented in a funnel (start general and
  get narrow) or inverted funnel (provide specifics
  and then ask overall) sequences.

13
Guidelines for Survey Format

• Determine response format for each set of
  questions - open-ended (respondent fills in
  response) or closed-ended (responses are provided
  for the respondent).
• Screening questions should be done first - dont
  work for completed questionnaires by ineligible
  people.
• When asking demographic questions or income
  questions, always include one category higher or
  lower than you think will be needed.

14
Guidelines for Survey Format

• Always include adequate instructions, and be sure
  they are noticeable and in the proper place.
• Make the questionnaire look easy to read and
  professional.
• Use blank space and large type font - dont try
  to crowd too many questions together to make the
  questionnaire shorter.
• Number the questions.

15
Guidelines for Survey Format

• Never split question wording or response
  categories across pages - avoid splitting groups
  of questions.
• Use proper response category formats - use plenty
  of spacing between categories and use boxes where
  possible. Using grids can save space and make
  filling out the survey less complicated for the
  respondent.

16
Guidelines for Survey Format

• Pre-code response categories and assign data
  layouts before administering the survey.
• Always think about how the data will be analyzed.
  
• Always use instructions for any possible
  confusing questions and use a different typeface
  to distinguish them.

17
Increasing Response Rate

• Pretest!
• Exposes problems with the survey as well as
  identifies possible respondent and interviewer
  error issues
• Two pretests should be conducted
• First pretest is done by personal interview,
  regardless of final mode, to determine
• Any confusing questions or instructions
• Any difficulty respondents have with generating
  answers
• Second pretest - done exactly as final survey
  will be done
• A small portion of the actual study sample

18
Increasing Response Rate

• Include a postage paid, pre-addressed return
  envelope.
• Include a personalized, brief cover letter
  explaining the purpose of the questionnaire and
  how the results will be used.
• Incentives (money) have proven to slightly lift
  response, but more is not better.
• Keep the survey to no more than two pages.
• Ensure anonymity.