Comparing the Results of Probability and NonProbability Sample Surveys

1
Comparing the Results of Probability and
Non-Probability Sample Surveys
Sponsored by the Stanford Institute for the
Quantitative Study of Society
2
Alberto SimpserMatt LevenduskyLinChiat
ChangAmy GershkoffAditya Agrawal
3
House Effects

• Two definitions
• Any differences between the results produced by
  two firms measuring the same construct.
  
• Differences between the results of two firms
  measuring the same construct with the same
  methodology.
  
• Past findings
• Lots of house effects, according to the first
  definition.
  
• Few and small house effects, according to the
  second definition.
  

4
Conclusion

• The same survey conducted by two firms with the
  same methodology will usually yield comparable
  findings.
  
• But past studies have focused mostly on surveys
  conducted in the same mode with comparable
  sampling methods.
  
• What if mode and sampling method differ?

5
Data Collection Firms
6
Methods
Knowledge Networks declined to provide its
cumulative response rate
7
Usual Sample Balancing

• We asked for a nationally representative sample
  of adults living in the U.S. (18)
  
• Weighting
• SRBI
• Knowledge Networks
• Harris Interactive
• Quotas
• All the rest, at sample draw and/or completion

8
30-Minute Questionnaire

• Consumer Behaviors
• Movies
• Fast Food
• Airlines
• Rental Cars
• Beverages
• Automobiles

9
30-Minute Questionnaire

• Lifestyle Behaviors
• Exercise
• Cigarette smoking
• Vacation travel
• Frequent flier programs
• Passport
• Drivers license
• Voting in elections

10
30-Minute Questionnaire

• Self-Descriptions
• Health Status
• Insomnia
• Helpfulness
• Honesty
• Paranoia
• Enjoy thinking
• Have lots of opinions
• Efficacy
• Shyness
• Gregariousness
• Loneliness

11
30-Minute Questionnaire

• Self-Descriptions
• Self-Esteem
• Ability to think
• Ability to read
• Respect for authorities
• Religiosity
• Concern about privacy
• Attitudes toward politicians
• Opinions on political policy issues

12
30-Minute Questionnaire

• Demographics
• Age
• Race
• Gender
• Education
• Income
• Marital Status
• Work for Pay
• Telephone Lines
• Number of Bedrooms

13
30-Minute Questionnaire

• Demographics (and other things)
• Own/Rent Living Quarters
• Number of telephone lines
• Do Not Call list
• Own computers
• Comfort with computers
• Time spent on computers
• Caller ID
• Call Blocker
• Telephone Answering Machine

14
Today

• PRELIMINARY results.
• Subject to change as we refine the analyses,
  improve our precision and correct any errors that
  may be present.

15
Main Finding
16
Striking Similarity of DistributionsAcross
Nearly All Questions
Main Finding
17
Demographics(not used for quotas or weighting)
18
Never Married
19
Married
20
Separated
21
Divorced
22
Widowed
23
One Person in Household
24
Two People in Household
25
Three People in Household
26
Four People in Household
27
Five or more People in Household
28
Children Present in Household
29
Homeowners
30
Renters
31
No Bedroom in House
32
One Bedroom in House
33
Two Bedrooms in House
34
Three Bedrooms in House
35
Four Bedrooms in House
36
Five or more Bedrooms in House
37
No Vehicle
38
One Vehicle
39
Two Vehicles
40
Three Vehicles
41
Four or more Vehicles
42
Life-Style
43
Traveled in Past 12 Months
44
Continental vs. Delta
45
Delta vs. Northwest
46
United vs. US Airways
47
Holds Valid Drivers License
48
Holds Valid US Passport
49
How much free time do you have? None At All
50
How much free time do you have? A Little
51
How much free time do you have? A Moderate Amount
52
How much free time do you have? A Lot
53
How much free time do you have? A Great Deal
54
Average Percent of Free Time Spent at Home
55
Respondents Who Exercise
56
Attitudes/Beliefs
57
Would you say your health in general is
excellent, very good, good, fair or poor?
Excellent
58
Would you say your health in general is
excellent, very good, good, fair or poor? Very
Good
59
Would you say your health in general is
excellent, very good, good, fair or poor? Good
60
Would you say your health in general is
excellent, very good, good, fair or poor? Fair
61
Would you say your health in general is
excellent, very good, good, fair or poor? Poor
62
How often do you think people try to behelpful
to others? Always
63
How often do you think people try to behelpful
to others? Most of the Time
64
How often do you think people try to be helpful
to others? About Half of the Time
65
How often do you think people try to behelpful
to others? Rarely
66
How often do you think people try to behelpful
to others? Never
67
Do you like, dislike, or neither like nor dislike
expressing your opinions to others? Like a lot
68
Do you like, dislike, or neither like nor dislike
expressing your opinions to others?Like somewhat
69
Do you like, dislike, or neither like nor dislike
expressing your opinions to others?Lean toward
liking it
70
Do you like, dislike, or neither like nor dislike
expressing your opinions to others?Dont lean
either way
71
Do you like, dislike, or neither like nor dislike
expressing your opinions to others?Lean toward
disliking it
72
Do you like, dislike, or neither like nor dislike
expressing your opinions to others?Dislike
somewhat
73
Do you like, dislike, or neither like nor dislike
expressing your opinions to others? Dislike a lot
74
How much impact do you words and actionshave on
people around you? None
75
How much impact do you words and actionshave on
people around you? A little
76
How much impact do you words and actions have on
people around you? A moderate amount
77
How much impact do you words and actionshave on
people around you? A lot
78
How much impact do you words and actionshave on
people around you? A great deal
79
How important is religion in your life?Not at
all important
80
How important is religion in your life?Slightly
important
81
How important is religion in your
life?Moderately important
82
How important is religion in your life?Very
important
83
How important is religion in your life?Extremely
important
84
How concerned are you about threats to your
personal privacy? Not at all concerned
85
How concerned are you about threats to your
personal privacy? A little concerned
86
How concerned are you about threats to your
personal privacy? Moderately concerned
87
How concerned are you about threats to your
personal privacy? Very concerned
88
How concerned are you about threats to your
personal privacy? Extremely concerned
89
Some Sharper Differences
90
How comfortable are you using acomputer? Not at
all comfortable
91
How comfortable are you using acomputer?
Slightly comfortable
92
How comfortable are you using acomputer?
Moderately comfortable
93
How comfortable are you using acomputer? Very
comfortable
94
How comfortable are you using acomputer?
Extremely comfortable
95
SummariesAccuracy Against Benchmarks
We Focus Here on Modal Response Categories -
Average Errors are Much Smaller Than Those That
Follow
96
  Accuracy of Primary Demographics  
97
Accuracy of Secondary Demographics
98
Life-Style Variables
99
Magazine Subscriptions  
100
Summary of Survey Accuracy (Mean Absolute Error)
 
101
Conclusions

• 1) Remarkable comparability of results.
• 2) Most accuracy from Knowledge Networks.
• 3) SRBI Telephone and Harris Interactive tied
  for second place.
  
• 4) Others about the same.
• 5) GoZing in last place.