Survey Research

1
Survey Research
2
What is a Survey?

• Researchers are often interested in the opinions
  of a large group of people about a particular
  topic or issue.

3
Three Major Characteristics

• Information is collected from a group of people
  in order to describe some aspect of the
  population
  
• Information is collected by asking questions of
  the members of the selected group
  
• Information is collected from a sample rather
  than from every member of the population

4
Descriptive Survey Research

• End Sought
• Exploration
• If the purpose of the research is to become more
  familiar with phenomena, to gain new insights, or
  to formulate a more specific research problem or
  hypothesis.

5
Descriptive Survey Research

• End Sought
• Description
• If we want to portray accurately the incidence,
  distribution and characteristics of a group or
  situation.

6
Why Are Surveys Conducted?

• The purpose of surveys is to describe the
  characteristics of a population.
  
• Researchers find out how the members of a
  population distribute themselves on one or more
  variables.

7
Why Are Surveys Conducted?

• A selected sample is surveyed and the description
  of the population is inferred from what is found
  out about the sample.
  
• In descriptive surveys, researchers are not as
  concerned about why the observed distribution
  exists as with what the distribution is.

8
Social Exchange Theory

• A theory of human behavior used to explain the
  development continuation of human interaction.
  
• Asserts that actions of individuals are motivated
  by the return these actions are expected to
  bring, and in fact usually do bring, from others.

9
Social Exchange Theory

• Three elements are critical for predicting a
  particular action
  
• Rewards,
• Costs
• Trust

10
Rewards

• What one expects to gain from a particular
  activity

11
Costs

• What one gives up or spends to obtain the reward

12
Trust

• Is the expectation that in the long run the
  rewards of doing something will outweigh the
  costs
  
• The likelihood or responding to a request to
  complete a self-administered questionnaire, and
  doing so accurately, is greater when the
  respondent trusts that the expected rewards of
  responding will outweigh the anticipated costs.

13
Social Exchange Theory

• Implies 3 questions about the design of a
  questionnaire and the implementation process.
  
• How can we increase rewards for responding?
• How can perceived costs be reduced?
• How can trust be established so that the ultimate
  rewards will outweigh the costs of responding?

14
Show Positive Regard

• Thibaut Kelly (1959) noted that being regarded
  positively by another person has reward value to
  many people.
  
• Give respondents reasons why the survey is being
  done, provide toll-free numbers to call with
  questions, and personally address correspondences

15
Say Thank You

• Blau (1964) argues that a time-consuming service
  of great material benefit (like a survey) could
  be appropriately repaid in a social exchange
  context by mere verbal appreciation

16
Ask for Advice

• Both Blau (1964) and Homans (1961) have pointed
  out that the feeling of being asked for help,
  advice, or assistance provides a sense of reward
  to people.

17
Support Group Values

• Blau (1964) noted that supporting a persons
  values can instill a sense of reward in
  individuals.
  
• Depending on the survey population, sponsorship,
  and topic, one can often appeal to values shared
  widely by those who are surveyed.

18
Give Tangible Rewards

• Research has shown that token financial
  incentives will increase response rates
  (reciprocal obligation)
  
• Only a dollar or two, enclosed with the request
  to complete a questionnaire
  
• Outperforms promises to send a larger reward
  later.

19
Make the Questionnaire Interesting

• Cialdini (1984) argued that liking to do
  something is a powerful determinant of behavior.
  
  
• Heberlein Baumgartner (1978) showed that
  questionnaires on topics of high salience to
  recipients are more likely to be returned.

20
Make the Questionnaire Interesting

• Questionnaires can be made more interesting to
  respondents by
  
• improving layout and design,
• ordering questions so the more interesting ones
  are placed at the beginning, and
  
• making questions easy to understand and answer.

21
Give Social Validation

• Knowing that other people like themselves have
  completed a similar action can strongly influence
  peoples willingness to comply with a request.

22
Give Social Validation

• In other words, some people are socially
  validated by seeing themselves as similar to most
  others in a group.
  
• Therefore in later attempts to encourage
  response, telling people that many others have
  already responded encourages them to act in a
  similar way.

23
An Opportunity for You

• Inform respondents that opportunities to respond
  are scarce
  
• Telling people that there are relatively few
  opportunities to respond and that they may not
  have an opportunity to respond unless they do so
  quickly can influence people to act (Groves et
  al., 1992).

24
Deadline to Respond

• Deadlines to respond can make a difference
• If it is not too quick of a deadline that then
  feels like a cost (needing to put something else
  aside)

25
Avoid Subordinating Language

• For us to help solve the school problems in your
  community it is necessary for you to complete the
  questionnaire vs. Would you please do me a
  favor?

26
Avoid Subordinating Language

• Former implies that the respondent is dependent
  upon the letter writer.
  
• Blau (1964) argues that people prefer not to be
  subordinate to others. Not responding is an easy
  way of avoiding a sense of being subordinated.

27
Avoid Embarrassment

• Thibaut and Kelly (1959) pointed out that costs
  to an individual are higher when great physical
  or mental effort is required, and when
  embarrassment or anxiety accompany the action.

28
Ways of Reducing Social Costs

• Questionnaires often get discarded when the
  respondent peruses the questionnaire but cant
  figure out where to start, or what the first
  question means.
• Not being able to read well or not being able to
  express ones self in writing are similar social
  costs.

29
Avoid inconvenience

• Not including a return envelope lowers response
  rates.
  
• Including a return envelope with a real stamp(s)
  on it also improves response rate over a business
  reply envelope.

30
Make It Short and Easy

• Questionnaires that appear shorter and easy to
  fill out lessen the perceived costs of
  responding
  
• Such appearance can be reinforced by indicating
  in the cover letter that responding should only
  take a few minutes

31
Obtaining Personal Information

• Minimize requests to obtain personal information
• Some information people do not want to reveal to
  others
  
• E.g. annual income, past sexual behavior, method
  of disciplining children, or use of drugs

32
Obtaining Personal Information

• Minimize requests to obtain personal information
• Suggestions
• Use specific wording to try to soften the
  question.
  
• Add explanation as to why the question is
  important.
  
• Explain how the information will be kept
  confidential or even anonymous.

33
Similar Requests

• Keep requests similar to other requests to which
  a person has already responded

34
Similar Requests

• People who have committed themselves to a
  position are more likely to comply with requests
  to do something consistent with that position
  (Cialdini, 1984).
• e.g. someone who has completed a small survey for
  you is more likely to complete another survey
  later (pretest and then a posttest)

35
Tokens of Appreciation

• Provide a token of appreciation in advance
• Although 1-2 may have little direct reward, it
  seems to have a greater value in creating trust.

36
Up-front Rewards

• Rewards up-front shows the researcher trusts the
  potential respondent.
  
• Uncancelled stamps on return envelope create the
  same effect.
  
• Call it a small token of appreciation vs.
  payment for their time (subordinate)

37
Sponsorship by Legitimate Authority

• More likely to comply with a request if it comes
  from an authoritative source.
  
• Government surveys typically receive a higher
  response rate than marketing surveys.

38
Use of Authority

• Use of authority can increase response rates
• Most surveys are voluntary but many government
  surveys are not. In these cases, requiring a
  response does increase response rate.

39
Make the Task Appear Important

• Appeal to people on the basis that something
  important will ultimately happened as a result of
  the survey.
  
• Printing personalized cover letters on letterhead
  stationary and a well-designed questionnaire can
  create this perception.
  
• Form letters and sloppy questionnaires do not.

40
Invoke other Relationships

• Use other exchange relationships to help evoke a
  response (doing a favor)
  
• Alumni helping out.

41
Tailored Design

• Is the development of survey procedures that
  create respondent trust and perceptions of
  increased rewards and reduce costs for being a
  respondent
• Takes into account features of the survey
  situation and have as their goal the overall
  reduction of survey error.

42
Tailored Design

• Exchange concepts must be communicated visually
  (rather than verbally) through the use of visual
  design principles for the development of the
  questionnaire and implementation materials
• Successful tailored design seeks to reduce survey
  errors!

43
Types of Surveys

• There are two types of surveys
• Cross-sectional
• Longitudinal

44
Cross-sectional Survey

• Collects information from a sample that has been
  drawn from a predetermined population
  
• Information is collected at just one point in
  time, even though it could take days to gather
  all the data
  
• A census is when the entire population is surveyed

45
Types of Surveys

• Longitudinal survey
• Information is collected at different points in
  time in order to study changes over time
  
• Three types are employed
• Trend study
• Cohort Study
• Panel Study

46
Issues to Consider

• Population
• Sampling
• Questions
• Content
• Bias
• Administrative

47
Population Issues

• Can the population be enumerated?
• Is the population literate?
• Are there language issues?
• Will the population cooperate?
• What are the geographic restrictions?

48
Sampling Issues

• What data is available?
• Can respondents be found?
• Who is the respondent?
• Can all members of the population be sampled?
• Are response rates likely to be a problem?

49
Questions Issues

• What types of questions can you ask?
• How complex will the questions be?
• Will screening questions be needed?
• Can question sequence be controlled?
• Will lengthy questions be asked?
• Will long response scales be used?

50
Content Issues

• Can respondents be expected to know about the
  issue?
  
• Will respondents need to consult records?

51
Bias Issues

• Can social desirability be avoided?
• Can interviewer distortion and subversion be
  controlled?
  
• Can false respondents be avoided?

52
Administrative Issues

• Costs
• Facilities
• Time
• Personnel

53
Steps in Survey Research

• The following steps are used in conjunction with
  performing proper surveys in research
  
• Defining the problem
• Identifying the target population (unit of
  analysis)

54
Steps in Survey Research

• The following steps (cont.)
• Choosing the mode of data collection
• Direct administration
• Mail surveys
• Telephone surveys
• Personal interviews

55
Advantages and Disadvantages of Survey
Data-Collection Methods
56
Example of an Ideal vs. an Actual Telephone
Sample for a Specific Question
57
Steps in Survey Research (cont.)

• Selecting the sample
• Training interviewers

58
Steps in Survey Research (cont.)

• Preparing the instrument (questionnaire)
• Closed-ended questions
• Open-ended questions
• Pre-testing the questionnaire
• Format
• Cover letter

59
Constructing the Questionnaire

• Involves much more than the manipulation of
  words.
  
• Paper size and binding
• Determining which questions will encourage the
  recipient to start responding and keep going
  until the end.
  
• Layout decisions must be made that involve
  spacing, size, and brightness.

60
Alternative Questionnaire Formats

• Booklets higher response rates
• 8½" x 14" legal paper
• Creates an 8½" x 7" booklet
• Folded length-wise will fit in a standard
  business stationary envelope
  
• 11" x 17 paper
• Creates an 8½" x 11" booklet
• Can consider two columns per page

61
Alternative Questionnaire Formats

• IF not doing a booklet, print on only one side of
  the sheet with a staple in upper left corner.

62
Criteria for Ordering the Questions

• A questionnaire is like a conversation.
• One question should lead to the next like a
  person listening during a conversation.
  Constantly changing topics is confusing.

63
Criteria for Ordering Questions

• Begin by order questions from most salient to
  least.
  
• Take into account what is the stated purpose of
  the study.
  
• Objectionable questions are placed near the end
  of the questionnaire.

64
Criteria for Ordering Questions

• Try to order in a way that will be logical to the
  respondent
  
• Consider grouping similar questions in sections
• Consider how question may influence answers to
  subsequent questions
  
• e.g. dont ask summary questions before specific
  domain questions

65
Choose the 1st Question Carefully

• No question is more critical than the first one.
• Will more likely determine whether survey is
  returned or trashed.

66
Choose the 1st Question Carefully

• Suggestions
• First question should clearly apply to everyone
• Should be easy so that all respondents will need
  only a few seconds to read, comprehend, and
  respond to it.

67
Choose the 1st Question Carefully

• Suggestions
• Needs to be interesting
• Connectedness between the respondent and survey
  purpose (as understood by that person) is clearly
  evident

68
To Achieve a High Response Rate

• Respondent friendly questionnaire
• Four contacts by First Class mail, with an
  additional special contact
  
• Return envelopes with real First-Class stamps
• Personalization of correspondence
• Token prepaid financial incentive

69
First Contact

• A brief prenotice letter
• Sent to the respondent a few days prior to the
  questionnaire.

70
First Contact (cont.)

• It notes that a questionnaire for an important
  survey will arrive in a few more days and that
  the persons response would be greatly
  appreciated.
• Highlight what will happen, what it is about,
  usefulness of the survey, and a closing thank you
  statement.
  
• As a P.S., you could mention the token of
  appreciation.

71
Second Contact

• The initial questionnaire mailing. Packet to
  include
  
• Cover Letter
• Appropriately constructed, letterhead, signed
• Questionnaire
• Booklet, saddleback stapled, figure on front
  page, back page blank.

72
Second Contact Packet (cont.)

• Pre-addressed, stamped (real), return envelope
• Incentive
• Returned card (?), mailed flat, commemorative
  stamps, etc.

73
Third Contact

• A thank you postcard
• Sent a few days to a week after the
  questionnaire.
  
• The mailing expresses appreciation for
  responding, and indicates that if the completed
  questionnaire has not yet been mailed it is hoped
  that it will be returned soon.

74
Fourth Contacts

• A replacement questionnaire
• Sent to nonrespondents 2-4 weeks after the
  previous questionnaire.
  
• Indicates that the persons completed
  questionnaire has not yet been received and urges
  the recipient to respond.

75
Final Contact

• May be made by telephone a week or so after the
  fourth contact (if telephone numbers are
  available).
  
• This special contact has been shown to add an
  additional 13 response

76
More Contacts?

• Up to 20 follow-ups have been shown to be
  effective

77
Recommended Steps in Conducting a Mailed Survey

• Key Points
• Get as many responses as possible
• Control non-response error.

78
Internet or Web-based Surveys

• Becoming more popular
• Know you population sample
• Computer users?
• Dial-up or broadband?
• Design you own or use web-based survey companies
• e.g. Survey Monkey http//www.surveymonkey.com/

79
Mixed Mode or Bi-modal Survey

• Some recent work on combining methods.
• Start with web and use mailings for subsequent/
  final rounds.

80
Evaluating Threats to Internal Validity in Survey
Research

• There are four main threats to internal validity
  in survey research
  
• Mortality
• Location
• Instrumentation
• Instrument decay

81
Non-response Error

• Non-response is a serious problem in many
  surveys.
  
• Members of the sample cannot be reached
  (non-deliverables) or refuse to respond
  (non-respondents)

82
Non-response Error

• In almost all surveys, some members of the sample
  will not respond.
  
• Item non-response is due to unclear or
  questionable forms of wording.

83
Proposed Procedures for Handling Nonresponse
Issues

• When 85 not achieved
• Comparison of early to late respondents
• Waves?Min 30 or 50
• Using Days to Respond as a regression variable
• Compare respondents to nonrespondents
• Random sample ? Min 20 or use 1 or 2

84
Data Analysis in Survey Research

• After researchers receive the completed
  questionnaires, the task of summarizing the
  results remains.
  
• The total size of the sample and total percentage
  of returns should be reported.

85
Data Analysis (cont.)

• The percentage of the total sample responding for
  each item should be reported.
  
• The percentage of respondents who chose each
  alternative for each question should be given.

86
Summary of Descriptive Research

• Primary end sought is to describe or explore
• Describe present status, accurate description of
  the incidence and distribution of phenomena.

87
Summary of Descriptive Research

• Usually conducted in a natural setting
• Data collected by mail or Internet questionnaire
  interview, or telephone.

88
Summary of Survey Research

• Generally interested in making generalizations
  about populations, so sampling is involved.
  
• Necessitates probability (random) sampling.
• Census of all subjects can be done.

89
Summary of Survey Research

• Usually, survey research does not involve the
  testing of hypotheses about relationships between
  variables.
  
• Consequently, variables are not identified as
  independent or dependent since the variables are
  used to describe status only.
  
• Can use hypotheses to compare 2 or more groups.

90
Summary of Survey Research

• Specific objectives stated as subquestions to be
  answered.

91
Summary of Survey Research

• Advantages of Survey Research
• Possible to collect a wide scope of information
  from a large population.
  
• Deals with the real situation data are collected
  in the actual situation.
  
• May be necessary first step in developing
  hypotheses
  
• Or in identifying more specific problems for
  research that goes beyond description only.

92
Summary of Survey Research

• Disadvantages of Survey Research
• Usually more extensive than intensive.
• Ordinarily does not penetrate very deeply below
  the surface
  
• Sometimes demanding of time and money.

93
Summary of Survey Research

• Disadvantages (cont.)
• Frequently lacks external validity
• Frequently difficult to obtain valid data
• Involves slicing-time
• Is this time representative of other times?

94
Summary of Survey Research

• Require a great deal of research knowledge and
  sophistication.
  
• Looks easy but isnt
• Sampling
• Handling non-response error
• Design, construction, and testing of
  questionnaires and interview schedules get to be
  major undertakings.

95
Summary of Survey Research

• The interpretation possible is description only.
• Explanation or prediction of phenomena is not
  possible if research is designed to describe only.