Questionnaires

1
Questionnaires Data Collection Forms

• Dr. John T. Drea
• Professor of Marketing
• Western Illinois University

2
Choosing a data collection procedure

• Can be viewed by the degree of control available
• Sampling control control of selection of sample
  subjects.
• Information control obtaining the breadth and
  quality of information needed.
• Administrative control speed and cost.
• All of these have impacts on data quality.

3
Collecting Survey Data Control

• Sampling Control Telephone
• Good with US respondents, non-response bias is
  less of a problem than with mail surveys
• Can create random samples.
• Some under-representation of lower income
  individuals.
• Less control in many international markets.

4
Collecting Survey Data Control

• Sampling Control Mail
• More comprehensive coverage, but a greater
  problem with non-response bias.
• Randomization can be achieved through purchased
  mailing lists, usually with excellent quality
  (guaranteed 90-95 deliverable)

5
Collecting Survey Data Control

• Sampling Control Personal Interviews
• Can be done randomly for general population, but
  expensive.
• Can be done as a mall-intercept, but is the
  sample representative of the population?

6
Collecting Survey Data Control

• Information Control
• Telephones
• Limitations in complexity - no rank orders, no
  visual aids or other stimuli.
• Allows for complicated branching.
• Mail
• Problem of sequence bias
• No complicated branching, and can use more
  complicated questions
• Personal Interview
• Can use a variety of tools, no sequence problems

7
Collecting Survey Data Control

• Administrative Control
• Telephone
• Fast, data can be automatically tabulated and
  ready to edit, less expensive.
• Mail
• Slow, data can be coded quickly in some cases,
  can be very expensive depending on response
  rates.
• Personal Interview
• Speed depends on number of trained interviewers,
  can be very expensive.

8
Open Ended or Closed Ended Questions?

• Open ended questions
• Difficult to code and can introduce some bias
  problems in interpretation.
• Provides subjects with an opportunity for a
  variety of responses - good for pre-tests and
  creating multichotomous response categories.
• Ex Why did you choose to attend the last
  Milwaukee Brewers game?

9
Categories of Survey ErrorAdapted from Zikmund,
Essentials of Marketing Research (1999)
Non- response Error
Guessing
Attention Loss
Respondent Error
Sampling Error
Fatigue
Response Bias
Respondent Misunder- standing
Total Error
Non- Sampling Error
Data Processing Error
Auspices
Acquiescence
Sample Selection Error
Admin. Error
Extremity
Interviewer
Interviewer Error
Social Desirability
10
Sampling and Non-Sampling Errors

• Sampling Error the difference between the
  observed value of a measure and the long-run
  average of the observed values in repetitions of
  the measurement.
• The difference between the sample mean and the
  mean of the population (as estimated by multiple
  means of the population by the same instrument)
• A key to reducing sampling error is increasing
  sample size.

11
Sampling and Non-Sampling Errors

• Non-Sampling Errors types of errors not related
  to the sample (i.e., all other samples)
• Sampling errors decrease as sample sizes
  increase.
• Non-sampling errors do not decrease as sample
  sizes increase.
• Non-sampling errors have been found to be ten
  times the magnitude of sampling errors!

12
Types of Non-Sampling Errors

• Non-observation errors failure to obtain data
  from parts of the survey population (sometimes
  referred to as sample selection error)
• Are there segments of your populations from which
  you are failing to include in the sample? This is
  a non-coverage error. Are some units
  over-represented (listed more than once)? This
  is over-coverage error. They are both gaps in
  the sample frame.
• Are there characteristics members of the original
  sample that were not among the survey
  respondents? This is a non-response error.

13
Administrative Errors

• Data Processing Error
• errors from entering, coding, editing
• check each others work!
• Ideally, have one person read and the other
  enter.
• Interviewer Error/Cheating
• Can occur from marking the wrong response, or
  falsifying responses
• Check for falsification by calling a selected
  number of respondents back to confirm whether the
  original interview was conducted
• Sample Selection Error

14
Sample Selection Error(Non-coverage errors)

• How big of a problem is it?
• Compare data with available population data -
  does the sample compare with other known data?
• Be sure to scale questions similar to outside
  sources of known population data.
• Can we do anything to reduce it?
• Elimination of duplicates
• Carefully comparing the sample frame to the
  population.

15
Sample Selection Error (Non-response errors)

• Non-response error an inability to obtain
  information from sample elements.
• Just because someone doesnt respond, it doesnt
  mean we have a non-response error.
• If non-respondents are similar to respondents,
  there is no non-response error.

16
Response Rate Issues, Phone Surveys
Not at homes If there is no answer at a number,
or if you get an answering machine, place the
subject on a callback list to call during the
next calling cycle. Three attempted callbacks
is usually adequate - that reduces non-response
bias issues. Refusals best strategy is to
prevent them - what can you do to get the
respondent to complete the interview?
17
Response Rates
Response rates are much easier to calculate for
a mail survey, personal interview, or a survey
administered in a group setting.
If we mail 1000 surveys and receive 250 replies
of which 25 are not useable (illegible, not
completed by correct respondent, etc.), the
response rate is calculated as (250 - 25)/1000
22.5
18
Types of Non-Sampling Errors

• Observation Errors inaccurate information is
  obtained, created during processing, or
  communicated in the results.
• Harder to determine we need to compare measure
  results with true information to detect - in many
  cases, if we had the true information, we
  wouldnt need to do the research in the first
  place!

19
Response Bias

• Occurs when respondents consciously or
  subconsciously distort their responses Common
  types of response bias
• Acquiescence Bias some respondents tend to agree
  with all questions (ex Mizuno Gloves)
• Extremity Bias Some respondents avoid extremes,
  others gravitate towards them.
• Interviewer Bias Altering a question, presence
  of the interviewer tone of voice
• Auspices Bias Who is conducting the research?
• Social Desirability Desire of respondent to
  create a favorable impression (voting,
  readership, TV)

20
Controlling Response Bias

• Intentional bias
• Anonymity (name not associated with answers)
• Confidentiality (answers will remain private)
• Incentives (encourages participation, creates
  sense of obligation to tell the truth)
• Validation check (used in personal interview -
  asking respondent to demonstrate truthfulness,
  e.g., can I see your medication you take for
  condition X)
• Third-person technique (makes the question less
  personal, e.g., Do you think someone like
  yourself would consider a physicians assistant?)

21
Controlling Response Bias

• Respondent Misunderstanding respondent provides
  an answer w/o understanding the question (watch
  your terminology and instructions!)
• Guessing respondent gives an answer w/o know if
  it is accurate (introduces error - only ask
  questions the respondent can answer)
• Attention Loss decreasing motivation to complete
  the survey (encourage them in the instructions)
  or possible distractions (quiet place for
  interviews)
• Respondent Fatigue respondent tires of answer
  questions, accuracy drops (keep it short, offer
  incentive to complete)

22
Closed End Questions

• Multichotomous
• Provides three or more structured categories for
  responses.
• Dichotomous
• Provides two structured categories for responses.
• Use dichotomous only when its an either/or
  type of question - no middle position is possible

23
Examples, Dichotomous Questions

• Appropriate
• Has your household purchased an automobile in the
  past twelve months? ___ yes ___no ___ dont
  know
• What is your gender? ___female ___male
• Inappropriate
• Will your household purchase an automobile in the
  next twelve months? ___yes ___no ___dont know
• Are you satisfied with your current automobile?
  
• ___ yes ___no
• Do you consider yourself to be ____tall ___short

24
An alternative way to ask the same question

• Will your household purchase an automobile in the
  next twelve months?
• definitely will purchase
• probably will purchase
• probably will not purchase
• definitely will not purchase
• no opinion/dont know

25
Individual Question Content

• One question per issue (avoid double-barreled
  questions)
• No How important are academics and placement in
  choosing a college?
• Make your questions specific
• No Where do you buy your groceries?
• Yes At which local grocery store did you
  personally last purchase grocery items?

26
Individual Question Content

• Do subjects have the information to answer the
  question?
• Does the subject really know?
• Ex How much do you spend on food per week?
• Can they remember?
• Ex On what date did you last watch TV program X?
• Are subjects willing to answer?
• Place sensitive questions at the end of a survey,
  and/or place the subject at ease by suggesting a
  behavior is common.

27
Some other suggestions on question wording

• Is the question necessary?
• Avoid leading questions.
• Will respondents give the correct information?
• Use simple, ordinary wording.
• Avoid ambiguous words (occasionally, regularly)
• Avoid extreme words (always, all, never, ever)
• Avoid poorly chosen words
• Be careful with you and where