THE BASICS OF CRAFTING GOOD QUESTIONS

1
THE BASICS OF CRAFTING GOOD QUESTIONS

• Damon Burton
• University of Idaho

2
QUESTION CONSTRUCTION CASE STUDY

• Smyth, Dillman Christian (2007) studied how
  question construction influenced responses on the
  number of hours a day students studied.
• 3 types of response patterns were tested
• A low range with 5 categories from .5 or less to
  2.5 hours a 6th category of more than 2.5 hours
• A high range consisting of 2.5 hours or less and
  5 categories between 2.5 and 4.5 hours
• An open-ended question with no categories.
• Responses for students studying more than 2.5
  hours varied by response categories, including
  30 for the low range, 71 for the high range and
  58 for the open-ended responses.

3
RATIONALE FOR FINDINGS

• For many students, estimating their average study
  time is a complicated task that requires
  averaging across weekdays and weekends and
  different times of the year.
• Respondents often look to the question and its
  accompanying response options for clues.
• Typically they assume that the range emphasized
  by the scale represents how many hours most
  students study.
• The middle option is often seen as representing
  the amount the average student studies.
• To save time, students may simply estimate
  whether they study more or less than an average
  student.

4

• What are the important issues to consider in
  crafting good survey questions?

5
ISSUES IN CRAFTING SURVEY QUESTIONS

• What survey mode(s) will be used to ask the
  questions?
• Is this question being repeated from another
  survey, and/or will answers be compared to
  previously-collected data?
• Will respondents be willing and motivated to
  answer accurately?
• What type of information is the question trying
  to obtain?

6
1. SURVEY MODE USED

• Different survey modes rely on different
  communication channels.
• Telephone interview respondents give and receive
  information through spoken words and their
  hearing system. Memory also becomes a more
  important factor.
• Web and mail questionnaires transmit information
  through the visual system.
• In mail and internet surveys, visual design
  elements become more critical.

7
2. COMPARING TO PREVIOUS SURVEYS

• If a question was used in another survey, the
  objective typically is to replicate the item with
  minimal change.
• Government surveys and longitudinal studies often
  repeat items.
• In self-administered studies, not only should the
  question be the same but also visual design and
  layout of the item.
• Can items be modified to remain contemporary in
  wording and focus?

8
3. ITEM MOTIVATES ACCURATE RESPONSE

• Respondents may ignore instructions, read
  questions carelessly or provide incomplete
  answers.
• Assume the worst in designing items.
• Poor item design may reduce motivation.
• Items may be (a) difficult to read and
  understand, (b) instructions are hard to find, or
  (c) the response task is too vague.

9
3. ITEM MOTIVATES ACCURATE RESPONSE

• Respondents are less likely to answer certain
  behavioral questions that they perceive as
  embarrassing or threatening such as sexual or
  criminal activity.
• Broad categories are more readily answered when
  dealing with financial information.
• For sensitive questions, wording and putting the
  question into context improve response rates.

10
4. TYPE OF INFORMATION REQUESTED

• Factual or demographic information is the easiest
  type of questions to answer (e.g., age).
• For attitude or opinion questions, elements of
  the item can influence the answering process,
• type of response requested,
• wording, and
• visual layout.

11
4. TYPE OF INFORMATION REQUESTED

• Respondents may be influenced by the context of
  the question (e.g., solving environmental
  problems).
• For behaviors or events, if the item requires
  remembering too many details, respondents rely on
  survey design for answers.
• Memory fades over time, so asking questions about
  easily-recalled or memorable behaviors enhances
  recall (e.g., days walked to work this week or
  recent trip to Yellowstone).

12

• What factors should you consider in choosing an
  effective item format?

13
A WHOLISTIC APPROACH TO QUESTION DESIGN

• Multiple aspects of the wording and design of the
  question must work together to convey meaning.
• Choose appropriate question format(s).
• Understand the anatomy of a survey question and
  use that information to design effective items.
• Question design benefits from informed, multiple
  and diverse perspectives.

14
A WHOLISTIC APPROACH TO QUESTION DESIGN

• Open-ended questions are general and give
  respondents great freedom in answering (e.g.,
  What are your greatest sources of enjoyment in
  basketball?).
• Closed-ended questions are used to force all
  respondents to make a choice from a limited range
  of options (e.g., multiple choice questions or
  rating agreement on a 5-point Likert scale.
• The choice of format needs to be based on the
  purpose of the question.

15
OPEN-ENDED QUESTION FORMATS

• Strengths of open-ended formats
• allows respondents to freely answer without
  limitations,
• numerical responses can be reported exactly
  without vague labels or ranges,
• Weaknesses of open-ended formats
• more respondents skip this item format,
• biases arise more often,
• more variations in responses making it more
  difficult to analyze, and
• short answers more likely.

16
CLOSED-ENDED QUESTION FORMATS

• Strengths of closed-ended formats
• can be analyzed statistically (i.e., ordinal or
  interval data),
• data results are produced quickly,
• numerical responses can be reported exactly
  without vague labels or ranges,
• Weaknesses of closed-ended formats
• ordering and grouping may have unintended effects
  on answers and make response harder,
• Responses are constrained by possible answers.
• Hard to identify new categories of information.

17
TYPES OF DATA

• Nominal data provides an unorder set of response
  choices (i.e., cant assume one response is
  higher or lower than another).
• Ordinal data has an order to the categories of
  response choices (i.e., good, average and poor)
  but cant assume the size of the scaled magnitude
  between categories is the same or represents
  equal distance.
• Interval data has both order and equal intervals
  between response choices (i.e., 5 categories from
  very dissatisfied to very satisfied).

18
ANATOMY OF A SURVEY QUESTION

• The most important part of a survey question is
  the stem or the words forming the question.
• Additional instructions help respondents
  comprehend the meaning of questions or concepts.
• Crafting survey questions involves both choosing
  words and visually displaying questions and
  responses.

19

• What are the most important guidelines for
  creating good questions?

20
GUIDELINES FOR CHOOSING WORDS FORMING QUESTIONS

• Make sure the question applies to all.
• Make sure the item is technically correct.
• Ask one question at a time.
• Use simple and familiar words.
• Select specific, concrete wording.
• Use as few words as possible.
• Use simple, complete sentences.
• Make sure yes means yes no no.
• Be sure the question specifies how to respond.

21
1. QUESTION APPLIES TO ALL

• Questions for mail surveys sometimes try to
  reduce the number of questions each person has to
  answer,
• Avoid skip instructions,
• Avoid the word if which implies a response is
  not needed.
• Avoid providing only 2 bad options, especially if
  respondents have to answer.
• Provide a range of categories so every person has
  to respond to every question.

22
2. ITEM TECHNICALLY CORRECT

• Use the right units of measurement so the
  question is easy to answer.
• For example, the question How many feet tall is
  your horse? is hard to answer because horses
  height is typically measured in hands (i.e., a
  hand 4 inches).
• Credibibility and trust come with using the
  appropriate terminology for the audience and
  being technically correct.

23
3. ONE QUESTION AT A TIME

• Multiple topics in the same item make answering
  inaccurate.
• For example, the question Do you subscribe to
  and read any magazines related to your job?
  could be answered based on (a) subscribing, (b)
  reading or (c) both.
• Responses are quicker and more accurate when the
  item has a single topic.
• Credibibility and trust come keeping questions
  simple and easy to respond to.

24
4. USE SIMPLE, FAMILIAR WORDS

• Keep readability at 5th grade or less and keep
  word length to 6-7 letters.
• Avoid formalized wording with too complex or
  technical wording.
• For example, use tired instead of exhausted,
  work instead of employment, and correct
  instead of rectify.
• Avoid abbreviations.
• Replace specialized terms and cliches.
• Credibility comes from simplicity.

25
5. USE SPECIFIC WORDING

• To get the response you want, you must be
  specific about the question you ask.
• For example, the question How many times did you
  get together as a family last week? is somewhat
  vague.
• What does get together as a family mean?
• How many meals did you sit down to eat as a
  family last week? is much more specific.
• Specificity in questions allows much more
  accurate responses.

26
6. USE FEW WORDS PER ITEM

• Keeping readability low usually reduces the
  number of words per item.
• Replace complex words or rewrite item more
  simply.
• Eliminate redundant words.
• However, keeping readability low and content
  specific is more important than question length.

27
7. USE COMPLETE SENTENCES

• Eliminate phrases or clauses.
• Write complete sentences that use a simple
  sentence structure.
• Avoid convoluted sentence structure with
  confusing wording.
• Keeping readability low requires simple sentences
  that are easily understood.
• Compound sentences work better than complex ones.

28
8. MAKE SURE YES MEANS YES

• The mental connection for the word not is
  difficult to comprehend.
• Specify meaning of words such as favor and
  oppose so for and against may be better
  terms.
• Ballet initiatives are often worded in a
  confusing ways so you are not sure whether you
  are for or against. Avoid this problem in
  your surveys.
• Credibility and trust come from making decisions
  easy to understand.

29
9. SPECIFY THE RESPONSE TASK

• Match the question stem and the response options.
• Make the task clear to the respondent after have
  read the question once.
• The question stem has to clearly state the
  response task.
• The response format and/or options provided must
  match the task as stated in the question stem.
• Mismatched question and response makes item
  unusable.

30

• What design elements should be considered in
  presentation of survey questions?

31
VISUAL PRESENTATION OF SURVEY QUESTIONS

• Visual layout influences how respondents organize
  information and focus attention on responses.
• 4 types of visual design elements communicate
  meaning and can be manipulated to draw attention
  to or take away from meaning.
• Words the primary source of meaning.
• Numbers convey meaning and sequence.
• Symbols figures that add special meaning.
• Graphics simple and complex shapes and visual
  images that convey meaning.

32
VISUAL PRESENTATION OF SURVEY QUESTIONS

• Visual design an be modified through font size
  and type, brightness and location of elements.
• Gestalt principles of pattern perception assign
  meaning and provide a sense of grouping and
  separation of elements.
• Proximity locating elements together denotes
  relatedness.
• Common Region boxes and highlighted areas
  designate connectedness.
• Connections linking elements thru lines.
• Continuity multiple elements layered
  continuously creates a perception of a group.

33

• What are the primary guidelines for using visual
  elements to design questions and the overall
  survey?

34
VISUAL PRESENTATION GUIDELINES

• Use darker and/or larger print for the question
  and lighter and/or smaller text for answer
  choices and spaces.
• Creates clear subgrouping and separation,
• Contrast helps delineate questions from answers.
• Use spacing to help create subgrouping within a
  question because proximity states that items
  located together are perceived as a group.
• Visually standardize all response spaces.
• Similarity states that items appear regular and
  similar are perceived as belonging together.

35
VISUAL PRESENTATION GUIDELINES

• 4. Use design properties to emphasize elements
  that are important to the respondent or
  de-emphasize ones that are unimportant.
• Manipulating size, contrast and location will
  emphasize or de-emphasize element importance.
• 5. Use design properties with consistency.
• Use each design element for only one purpose such
  as underlining to draw attention to important
  words or bolding for questions.
• 6. Visual elements in question must send
  consistent messages.
• Avoid conflicting messages communicated by
  incongruous visual elements.

36
VISUAL GUIDELINES

• 7. Integrate special instructions into the
  question where they will be used.
• When special instructions are outside the
  question stem, they may be ignored.
• 8. Separate optional or occasionally needed
  instructions from the question stem by font or
  symbol variation.
• Distinguish between words that are essential for
  everyone to read and those that may be needed by
  only some respondents.
• 9. Organize questions in ways that minimize the
  need to reread portions of instructions in order
  to comprehend the response task.
• Emphasize respondent efficiency so they are
  always clear about what to do.

37
VISUAL PRESENTATION GUIDELINES

• 10. Choose line spacing, font and text size to
  ensure legibility.
• Need appropriate font so avoid script and use
  serif or sans serif.
• Use 10-12 point font for most populations and
  larger fonts for older populations.
• Line length of 3-5 inches is recommended.

38
End
END