Survey Design and Implementation

1
Survey Design and Implementation

• Elizabeth C Wells
• Doctoral Candidate
• Ag Extension Education
• http//www.msu.edu/wellselz/

2
Why do people fill out surveys? What do you want
to ask/measure? Accuracy/Survey Error Random
Sampling Writing Questions Questionnaire
Design Survey Implementation Telephone, Interview
and Internet Surveys UCRIHS
3
Why People Fill Out Surveys

• Social Exchange Theory
• Theory of human behavior in which actions of
  individuals are motivated by the return these
  actions are expected to bring.
• Elements of reward, cost and trust

4
Social Exchange Theory

• Rewards
• show positive regard
• say thank you
• ask for advice
• support group values
• give tangible rewards
• make the questionnaire interesting

5
Social Exchange Theory

• Reduce Social Costs
• avoid subordinating language
• avoid inconvenience
• avoid embarrassment
• make it short and easy
• minimize requests for personal information
• keep requests consistent

6
Social Exchange Theory

• Establish Trust
• provide token of appreciation in advance
• sponsorship by legitimate authority
• make task appear important
• invoke other exchange

7
Constructs and Measurement

• What do you want to know?
• What are the variables?
• How will you measure it?
• behavior, attitudes, feelings, demographics

8
Survey Error

• Sampling Error Who are you sampling?
• Coverage Error Does your list include everyone?
• Measurement Error Does everyone answer a
  question the same way?
• Non-response Error Why did respondent not
  answer
• Instrument (whole questionnaire not
  returned)
• Item (question not answered)

9
Non Response Error

• Only an issue if non responders would answer
  differently and change results
• compare early to late respondents
• double dip

10
Random Sampling

• How much sampling error can you tolerate?
• How large is the population?
• How varied is the population?
• How confident do you wish to be about estimates
  made from the sample?

11
Random Sampling - Chart

• Pop Size 50/50 split 80/20 split
• 80 71
• 132 111
• 196 153
• 234 175
• 260 188
• 1000 278 198

Sample size for a 95 confidence level And a 5
sampling error
12
Random Sampling

• http//www.randomizer.org
• A www site that will generate random numbers
  which you can use for random selection of
  participants on a mailing list.

13
Writing Questions

• Everyone will interpret the same way
• Respondents are able to respond accurately
• Respondents are willing to answer

14
Writing Questions

• Use simple words
• Keep it short
• Be specific
• Do not talk down
• Avoid bias
• Avoid hypothetical questions
• Use complete sentences

15
Question Criteria

• Does the question require an answer?
• If you fixed dinner last night, did you eat
  meat as part of that meal?
• To what extent does the respondent have a ready
  made and accurate answer?
• Are tall people more likely to be elected
  President?
• Can respondent accurately recall and report past
  behavior?
• How much television did you watch last month?
  

16
Question Principles

• Use simple words and phrases
• tiredexhausted
• honestcandid
• workemployment
• most important..top priority
• free time.leisure
• doctor.physician
• your answers.your responses to this
• 
  questionnaire

17
Question Principles

• Avoid vague quantifiers
• How often did you attend religious services
  last year?
• ? Never
• ? Rarely
• ? Occasionally
• ? Regularly

18
Question Principles

• How often did you attend religious services
  during the past year?
• ? Not at all
• ? A few times
• ? About once a month
• ? Two or three times a month
• ? About once a week
• ? More than once a week

19
Question Principles

• Avoid bias by using both positive and negative
  sides in the question stem
• To what extent do you agree.
• vs
• To what extent do you agree or disagree with
  this statement.

20
Question Principles

• Avoid double-barreled questions
• 16. Should the new school have a swimming pool
  that includes lanes for swimming laps and is not
  enclosed for winter use?

21
Question Principles

• Avoid asking respondent to say yes in order to
  mean no.
• 5. Do you favor or oppose not allowing the state
  to approve a casino without approval from 60 of
  the voters?
• ? Favor
• ? Oppose

22
5. Do you favor or oppose requiring 60 approval
by voters in order for a casino to be built. ?
Favor requiring 60 approval ? Oppose requiring
60 approval
23
Question Principles

• Use equal numbers of positive and negative
  categories for scalar questions AND distinguish
  undecided from neutral by placing at the end of
  the scale.

24
To what extent do you agree or disagree with this
statement Listening to classical music is good
for a persons emotional health. ? Strongly
agree ? Somewhat agree ? Neither agree nor
disagree ? Somewhat disagree ? Strongly
disagree ? No opinion
25
Question Principles

• Soften impact of potentially objectionable
  questions
• Have you ever shoplifted something from a
  store?
• vs
• Have you ever taken something from a store
  without paying for it?

26
22. What was your total income from all sources
in 2003? ______________ Total income for
2003 22. Which category best describes your
total income from all sources in 2003? ?
10,000. or less ? 10,001. to 20,000. ?
20,001. to 35,000. ? 35,001. to 50,000. ?
50,001. or above
27
Question Principles

• Dont combine numbers and words
• Doing so may impact respondents perception
  of the scale.
• 1 strongly agree, 2 agree,
• 3 disagree, 4 strongly disagree

28
Question Principles

• Eliminate check-all-that-apply questions to
  reduce primacy and sufficiency effect.
• People tend to check the first ones more than
  the last ones and sometimes check a few and feel
  that is enough to satisfy the researcher.

29
Question Principles

• Extension Agent
• Other Cattlemen
• Fertilizer and other salesmen
• 59 selected
• Extension Agent

• Other Cattlemen
• Fertilizer and other salesmen
• Extension Agent
• 41 selected
• Extension Agent

30
Questionnaire Design

• Paper and layout choice
• booklet format preferred
• one column for legal size
• two columns for full size
• avoid upper left staple design
• avoid unusual size, shape or fold

31
Order of Questions

• A questionnaire is like a conversation. It
  evolves according to social norms. Constantly
  switching topics makes it appear as though the
  researcher is not listening to the respondents
  answers.
• Doing otherwise results in answers that are less
  thought out when you want them to focus!

32
Order of Questions

• What was your family income in 2003?
• Do you like to play golf?
• What is your opinion on Global Warming?
• Are you married?
• How often have you gone fishing this year?
• How adequate is your current health care?
• How old are you?
• What is your political party preference?
• What is your occupation?

33
Question Order

• Group alike questions together
• Begin asking questions in the upper left corner
• Ask one question at a time
• Use item-in-a-series format

34
The First Question

• Should clearly apply to everyone
• Question should be easy
• Question needs to be interesting
• Connectedness between the question and the
  purpose of the survey.

35
Unacceptable First Questions

• 1. Please think about the things that make MSU
  Extension a pleasant place to work and write down
  the five most important aspects and rate them one
  to five in importance to you.
• .

36
Unacceptable First Questions

• 1. Please describe in your own words what you
  consider good about working for MSU Extension.
• 1. What year were you born?
• _________year born

37
Good First Question

• 1. Thinking about MSU Extension, how would you
  rate it as a place to work?
• ? Excellent
• ? Good
• ? Fair
• ? Poor

38
Graphic Design

• Stimuli on page consists of two languages
• written words
• graphic symbols and arrangements

39
Graphic Design

• Desired navigational path
• Create visual navigational guides
• Develop additional guides for redirecting during
  skip patterns
• Be consistent through out your instrument

40
Questionnaire Design

• Instructions need to be exactly where needed not
  at the beginning of the whole questionnaire.
• Use italics or bolding or underline to emphasize
  instructions and directions

41
Questionnaire Design

• Pre-attentive processing is 210 degrees
• Focused reading is 2 degrees or about 8 to 9
  characters
• Use font, size, contrast, underline, spacing to
  lead the respondent through the questionnaire

42

• Start here
• Which of the following is your main work
  activity?
• ? Research
• ? Teaching
• ? Administration
• ? Other (Please Specify) _________________

43
START HERE ? Which of the following is your
main work activity? ? Research ?
Teaching ? Administration ?
Other (Please specify) ____________________
44
Questionnaire Format

• For paper questionnaires the standard font is 12
  pt Times New Roman also consider a larger font
  for senior audiences
• For on-line questionnaires the standard font is
  10 pt Ariel

45
Questionnaire Design

• Use a square for your answer and ask for an X
  in the square.
• ?
• 1

46
Questionnaire Design

• Place instructions exactly where needed within
  the question
• Differentiate with font, bolding, underlining etc
  

47
5. How many months have you worked at your
current job? _________Number of
months Please be as specific as possible in
answering the next question, including any area
of specialization. Example Extension Agent
Agriculture. If you had more than one job, answer
for the job for which you worked the most hours.
6. What kind of work do you do in your current
job? _________________________________ Kind of
work
48
5. How many months have you worked in your
current job? _____________________ Number of
months 6. What kind of work do you do in your
current job? Please be as specific as possible in
answering. Include any are of specialization, for
example Extension Agent Agriculture. If
you had more than one job, answer for the job
which you worked the most hours.
______________________________Kind of work
49
Questionnaire Design

• Vertical alignment eases response task
• Dont double or triple bank answers

50
1. Which one of the following best describes the
4-H project with the largest enrollment within
your club?

• market beef
• lambs
• hogs

• dogs
• cats
• rabbits

• goats
• poultry
• dairy

51
1. Which one of the following best describes the
4-H project with the largest enrollment within
your club?

• beef
• hogs
• lambs
• dogs
• cats
• rabbits
• goats
• poultry
• dairy

52
Questionnaire Design

• Major changes are needed for skip patterns
• Skip patterns are difficult to follow
• Skip patterns often lead to item non response

53

• Normally do you eat breakfast every day?
• ? Yes (Go to 13)
• ? No (Go to 18)
• 13. How many times a week do you drink orange
  juice with your breakfast?
• __________ Times a week

54
12. Normally do you eat breakfast every day? ?
No (Skip to 18) ? Yes 13. (If
Yes) How often do you drink orange juice with
your breakfast? ____________ Times a
week
55
Questionnaire Design

• Demographic data questions
• Place at end of questionnaire but not on
• back page
• Only ask what is necessary

56
Questionnaire Design

• Front cover needs to stand out
• Consider simple art work
• Include name of survey
• Include name and address for return

57
Questionnaire Design

• Back cover needs to be simple with white space
• Try not to ask specific questions
• May place open ended question here
• Give opportunity to add comments
• Conclude with a thank you

58
Implementation

• Respondent friendly questionnaire
• Multiple contacts
• Token pre-paid incentive
• SASA for return
• Personalized correspondence

59
Implementation

• Four contacts plus one special 5 contacts
• Brief pre notice letter
• Questionnaire mailing with SASE
• Thank you postcard
• Replacement questionnaire
• Final contact with special mailing

60
Implementation Contact 1

• Pre-notice letter
• Date at top
• Personalized letter
• What will happen
• What it is about
• Usefulness of survey
• Thank you
• Real signature
• P.S. Token incentive

Mail out a few days to one week before mailing
instrument
4-6 higher response rate
61
Implementation Contact 2

• Mail the questionnaire
• Include personalized cover letter
• Questionnaire
• Self addressed stamped envelope with
  real stamp
• Enclose incentive
• Use first class postage plain envelope

62
Implementation Cover Letter

• Inside address
• Request
• Why you were selected
• Usefulness of survey
• Confidentiality
• Token of appreciation
• Willingness to answer questions
• Thank you
• Real signature in contrasting ink

63
Implementation Token Incentive

• Cash in advance 1. or 2. 12 to 19 increase
  in response rate
• Less of an increase for larger amount later
• Increasing incentive not cost effective
• Non dollar incentive not as effective and may be
  effective because of packaging

64
Implementation Contact 3

• Postcard to all who were mailed a questionnaire
  thanking them
• Serves to jog memory
• Gives opportunity to call for replacement if
  needed

Mail one week after questionnaire Increase up to
13
65
Implementation Contact 4

• Replacement questionnaire to those who have not
  yet responded.
• Letter reminding them of the survey and asking
  again for their assistance.

Sent only to non-respondents Two to four weeks
after 3rd contact
66
Implementation Contact 5

• Special mailing to those who have not yet
  responded
• Use unique method of mailing certified,
  priority etc
• May also telephone and fill out over the phone

Final contact is a week or more after 4th contact
67
Tracking Responses

• Assign each questionnaire a number
• Assign each name and address a number
• Send respondents their individually numbered
  questionnaire
• Have assistant mark off numbers as they are
  returned
• Note surveys are confidential not anonymous
• Keep completed questionnaires in secure location

68
Mixed Mode Surveys

• Responses differ by mode
• Excellent Health 44 by personal interview
  
• 37 by telephone
• 30 by mail
• social desirability
• acquiescence
• primacy vs recency effect

69
Telephone Surveys

• Difficult to obtain phone numbers
• Caller id and answering machines may screen calls
• Need well trained interviewers
• Limited number of questions and answers

70
Internet Surveys

• Consider coverage error. Does your entire
  audience have and use the internet on a regular
  basis?
• E-mail addresses change often
• Difficult to include incentive
• Shorter less detailed questionnaire
• Survey Monkey

71
Research Ethics

• Anonymous No way to know who returned the
  survey
• Confidential Only researcher can connect the
  respondent with their answers. All answers remain
  confidential.

72
Research Ethics

• If you wish to publish results you MUST get
  approval from UCRIHS before you send out surveys.
  
• University Committee on Research Involving
  Human Subject

73
References

• Mail and Internet Surveys The Tailored
• Design Method by Don Dillman 2000