Developing a Questionnaire

1
Developing a Questionnaire

• Chapter 4

2
Types of Questions

• Open-ended
• high validity, low manipulative quality
• Closed-ended
• low validity, high manipulative quality

3
Open-ended

• An open-ended question is one in which you do not
  provide any standard answers to choose from.
• How old are you? ______ years.
• What do you like best about your job?

4
Closed-ended

• A closed-ended question is one in which you
  provide the response categories, and the
  respondent just chooses one
• What do you like best about your job?(a) The
  people(b) The diversity of skills you need to do
  it(c) The pay and/or benefits(d) Other
  ______________________________

5
Dichotomous Questions

• Dichotomous Question a question that has two
  possible responses
• Could be
• Yes/No
• True/False
• Agree/Disagree

6
Questions based on Level of Measurement

• Use a nominal question to measure a variable
• Assign a number next to each response that has no
  meaning simply a placeholder.
• Use an ordinal question to measure a variable
• Rank order preferences
• More than 5 10 items is difficult
• Does not measure intensity

7
Interval Level

• Attempt to measure on an interval level
• Likert response scale ask an opinion question on
  a 1-to-5, 1-to-7, etc. bipolar scale
• Bipolar has a neutral point and scale ends are
  at opposite positions of the opinion
• Semantic differential an object is assessed by
  the respondent on a set of bipolar adjective
  pairs
• Guttman scale respondent checks each item with
  which they agree constructed as cumulative, so
  if you agree to one, you probably agree to all of
  the ones above it in the list

8
Filter/Contingency Questions

• To determine if a respondent is qualified to
  answer questions, might need a filter or
  contingency question (also known as knowledge)
• Limit of jumps
• If only two levels, use graphic to jump
• If you can't fit the response to a filter on a
  single page, it's probably best to be send them
  to a page, rather than a question

9
How many steps in the response scale?

• Statistical reliability of the data increases
  sharply with the number of scale steps up to
  about 7 steps
• After 7, it increases slowly, leveling off around
  11
• After 20, it decreases sharply

10
Should there be a middle category?

• Does it make sense to offer it?
• Should not be used as the dont know or no
  opinion option.
•  
• The middle option is usually placed between the
  positive and negative responses.
• Sometimes its last in an interview.

11
Direct Magnitude Scaling

• Method of obtaining ratio-scaled data
• Idea is to give respondents an anchor point, and
  then ask them to answer questions relative to
  that
• Example
• Suppose you are interested in the severity of
  crimes.
• Begin by assigning a number to one crime and then
  have respondents assign numbers to the others
  based upon a ratio.

12
Filtering "Don't Know"

• Standard format
• No "don't know" option is presented to the
  respondent, but is recorded if the respondent
  volunteers it.
• Quasi filter
• A "don't know" option is included among the
  possible responses.
• Full filter
• First the respondent is asked if they have an
  opinion. If yes, the question is asked.

13
Question Placement

• It's a good idea to put difficult, embarrassing
  or threatening questions towards the end
• More likely to answer.
• If they get mad and quit, at least you've gotten
  most of your questions asked!
• Put related questions together to avoid giving
  the impression of lack of meticulousness
• Watch out for questions that influence the
  answers to other questions.

14
Wording of Questions

• Direction of Statements
• Response bias
• Socially desirable
• Always and never
• Avoid this
• Better to phrase as most, infrequently
• Language
• Reflect educational level and reading ability
• Need for various languages

15
Frequency and Quantity

• Consider both frequency and quantity
• Consider number of times
• Consider duration of times

16
Mutually Exclusive and Exhaustive

• Mutually exclusive not possible to select more
  than one category/value
• Exhaustive providing all possible
  categories/values

17
Forced Choice

• Choose between 2 choices
• Might not be relevant
• Other choices exist (or at least possible)
• Lesser of two evils

18
Recalling Behavior

• Can be difficult to remember
• Ask questions that can be answered
• Choose time frames that are reasonable
• Pilot test for time frame issues

19
Response Bias

• Exaggerating the truth
• Socially desirable answers
• Consider using trap questions
• Possibly fictional choice

20
Sensitive Items

• More comfortable answering in categories
• Minimize missing data
• Might loose statistical power

21
(No Transcript)
22
Evaluating Questions

• Pre-testing
• Cognitive interviewing
• Behavior coding
• Peer review
• Peer review has shown to be the best method but
  its the least used. 

23
Validity and Reliability Questions

• Evaluative strategies
• Analysis of data to evaluate the strength of
  predictable relationships among answers and with
  other characteristics of respondents.
• Comparisons of data from alternatively worded
  questions asked of comparable samples.
• Comparison of answers against records.
• Measuring the consistency of answers of the same
  respondents at two points in time.

24
Coding the Questionnaire

• Create a codebook reference guide for the data
  set
• Code assigning a value to a response category
• Often numeric code
• Pre-coding makes it easier
• Content analysis on open-ended items
• Yes/No often coded as present or not (0 or 1)

25
Missing Responses

• Why blank?
• Missed them
• Refusal to answer
• Didnt feel it applied
• Didnt know the answer
• To code or not
• Analyze the difference
• If know why, might consider

26
Piloting the Questionnaire

• Test it on yourself
• Possibly other experts
• Test on people similar to sample
• Dont reuse (some exceptions)
• Discuss the survey with individuals
• During completion or After

27
Finding Respondents

• Best Methods of Selection
• Even with a good survey, poorly chosen sample
  leads to poor results