Survey Design and Measurement

1
Survey Design and Measurement

• Jeremy Kees, Ph.D.

2
Some practical issues.

• Qualtrics Research Platform
• Free you under VSBs site license
• Extremely user friendly, but also very robust
• www.qualtrics.com

3
Some practical issues.

• Amazon Mechanical Turk
• The most inexpensive way to collect consumer data
• Extremely user friendly, but also very robust
• www.mturk.com

4
Online Survey (created by you and housed on
Qualtrics server)

• Create HIT (Human Intelligence Task) on Mturk
• Description of your study and a (Qualtrics) link
  to it

Mturk workers (survey responders) work on your
HIT (i.e., they take your survey)
Data is recorded by Qualtrics. Participants who
complete the survey are given a code to input
into Mturk. Those that enter a valid code, get
paid.
Everyone is happy ? ? ?
5
The Mturk data I collected today.

• N200
• Cost 100
• Data collected in less than 1 hour
• Demographics
• Mean age 36
• 56 male
• 76 Caucasian
• 80 at least some college
• 41 are college grads
• Median income 35-50k
• Highly engaged!

6
Formulate Problem
Stages in the Research Process
Determine Research Design
Design Data Collection Method and Forms
Design Sample and Collect Data
Analyze and Interpret the Data
Prepare the Research Report
7
Surveys / Questionnaires

• The most common measurement instrument when
  quantitative data is sought
• Descriptive research
• Experiments
• Modeling
• Etc.

8
Developing Surveys

• Good, well-specified research objectives lead to
  good surveys
• Research design dictates what types of questions
  should be used
• Exploratory research unstructured script
• Confirmatory research structured survey

9
Desirable Characteristics

• Brief
• Objective
• Specific
• Relevant

10
Survey Methods

• Usually should determine administration method
  prior to developing items
• Can dictate what types of questions you should
  ask
• Internet panels have become the most efficient
  and versatile method to collect data
• Phone is still a viable option
• Mall intercepts can still be useful
• Mail/fax makes little sense anymore

11
Mall Intercepts vs. E-Panels
12
E-Panels vs. Phone
Jeff Miller and Alan Hogg Internet vs. Telephone
Data Collection Burke White Paper series 2 (4)
(www.burke.com). Also see Ashok Ranchhod and Fan
Zhou Comparing Respondents of E-Mail and Mail
Surveys, Marketing Intelligence Planning 19
(2001), 254.
13
Types of Questions

• Screening Variables
• Independent Variables
• Dependent Variables
• Classification Variables
• Segmentation
• Moderators
• Attention Filters

14
Primary Data Overview

• Types of Primary Data
• Demographic / Socioeconomic Characteristics
• Psychological / Lifestyle Characteristics
• Attitudes / Opinions
• Awareness / Knowledge
• Intentions
• Motivation
• Behavior
• What, how much, where, when, how, who
• Purchase behavior vs. use behavior
• E.g., --- basic hierarchy of effects models
• Example (CWL Study)

15
QUESTION WORDING - General Guidelines

• Use simple words and questions
• Avoid ambiguous words and questions
• Avoid leading questions---be objective
• Avoid implicit alternatives
• Avoid generalizations and estimates ---Be
  specific
• Avoid double-barreled questions

16

• What is your income?
• 10,000 or less.1
• 10,000 to 25,000..2
• 25,000 to 50,000..3
• 50,000 to 75,000...4
• 75,000 to 100,000.....5
• 100,000 or more..6

What is the problem and how would you revise the
question?
17

• Is the speed and efficiency of the drive-in
  teller services at your regular bank..(READ
  CATEGORIES)
• Very Satisfactory4
• Somewhat satisfactory3
• Somewhat unsatisfactory2
• Very unsatisfactory.1

18
Question Wording

• It is good practice to use scales whenever
  possible
• Likert or semantic differential
• Multi-item

19
Itemized Rating Scales

• The respondents are provided with a scale that
  has a number or brief description associated with
  each category.
• The categories are ordered in terms of scale
  position, and the respondents are required to
  select the specified category that best describes
  the object being rated.
• The commonly used itemized rating scales are the
  Likert and semantic differential

20
Types of Scales

• Nominal scales those that use only labels
• Ordinal scales those with which the researcher
  can rank-order the respondents or responses
• Interval scales those in which the distance
  between each descriptor is equal
• Ratio scales ones in which a true zero exists

21
Examples
21
22
Itemized Rating Scales

• Likert Scales
• requires the respondents to indicate a degree of
  agreement or disagreement with each of a series
  of statements about the stimulus objects

•  
• Strongly Disagree Neither Agree Strongly
• disagree agree nor agree
• disagree
•  
• 1. Wal-Mart sells high quality merchandise.
  1 2X 3 4 5
•  
• 2. Wal-Mart has poor in-store
  service. 1 2X 3 4 5
•  
• I like to shop at Wal-Mart . 1 2 3X 4 5
• 4. Wal-Mart has low prices . 1 2 3X 4 5

23
Itemized Rating Scales

• Semantic Differential Scales
• End points associated with bipolar labels that
  have semantic meaning
• SEARS IS
• Powerful ---------X----- Weak
• Unreliable -----------X--- Reliable
• Modern -------------X- Old-fashioned

24
Decisions for Itemized Scales

• Number of scale items
• More is better, but there is a diminishing return
  around 11 points (Nunnally 1978)
• 7-point scales are customary
• Enough to discriminate
• Allows for a scale midpoint
• Manageable
• Odd/even number of categories
• Forced vs. non-forced

25
Why Multi-Item Scales??
Construct Abstract Concept
Unobservable Latent Psychological
Single items are typically not sufficient to
assess unobservable constructs
26
Multi-Item Scales are More Reliable

• True Score Test Theory
• All measures have
• True Score
• Error (Random and Systematic)
• Good measures minimize the systematic error
  component of the score
• Types of Reliability
• Inter-Rater
• Test-Retest
• Internal Consistency (Cronbachs Alpha)

27
Developing Sound Measures
28
Question Sequencing

• After you have developed your measures, think
  about the order in which they should be asked

29
QUESTION SEQUENCING - General Guidelines

• Use (more) simple, interesting opening questions
• Use the funnel approach, asking broad questions
  first, and follow with more specific questions
• Carefully design branching questions
• Skip/display logic
• Ask for classification information last
• Place more difficult or sensitive questions near
  the end

30
QUESTION SEQUENCING - General Guidelines
Question ordering 1 1 EVALUATION OF FAT LEVEL
OF A PRODUCT 2 EVALUATION OF OVERALL PROD.
NUTRITIOUSNESS 3 EVALUATION OF OVERALL PRODUCT
ATTITUDE AND INTENTIONS TO PURCHASE
Question ordering 2 1 EVALUATION OF OVERALL
PRODUCT ATTITUDE AND INTENTIONS TO PURCHASE 2
EVALUATION OF LEVEL OF PROD. NUTRITIOUSNESS 3
EVALUATION OF FAT LEVEL OF PRODUCT
31
FOP Labeling Study

• We were interested in consumer evaluations of
• Facts Up Front
• All On-Package Labeling
• Front-of-Package Nutrition Info
• Why was question sequencing critical??

32
Tips for Maximizing Participation

• Offer an incentive ()
• Importance/relevance of the research project and
  its purpose
• Completing the questionnaire will take only a
  short time
• Answers are anonymous or confidential
• Reminder 2-3 days after the initial ask

33
Attention Filters

• Always include an attention filter to ensure that
  you are getting quality respondents
• Eliminate click throughs

34
Attention Filters (Case Study)

• Advertising Experiment
• Very stringent screening criteria
• Total that started the study 15,458
• Number that qualified 870
• Incidence Rate (IR) 5.6
• Number that qualified and passed the attention
  screener 451
• 48 failed the attention filter!!!
• NOT GOOD, criticalmix!

35
Easy Attention Filter
36
Difficult Attention Filter
37
And finally, remember the golden rule.

• Do unto your respondents as you would have them
  do unto you!!

38
Team Assignment 2

• Refine your research questions
• Need to be clear, concise, and testable
• Based on your research questions
• Design 2 potential studies that could address
  your research questions
• Explain the benefits and weaknesses of each
  approach
• Pick the best design and explain your decision
• (Note Dont worry about measurement or sampling
  too much---youll have your chance to do that
  later)

39
Team Assignment 3

• Dont start on this assignment until youve
  read Fowler (CH 6-7)
• Based on your research design
• Write a paragraph about what your measurement
  instrument is supposed to accomplish
• Make a list of what should be measured to
  accomplish the goals of the study
• Develop your measurement instrument

40
Team Assignment 3

• Deliverables include
• A very clean, polished version that you could use
  to actually collect data
• This means you will need to carefully think
  through all of the issues we covered tonight
  (e.g., set-up, ordering, length, multi-item
  scales, etc.)
• Intro paragraph and variable list (see previous
  slide)
• (Note Dont worry about defining your
  sample--youll have your chance to do that next
  week)