So Now You Want to Do a Survey...

1
So Now You Want to Do a Survey...

• Words of Advice, Words of Caution
• Jay P. Paul, PhD
• UCSF Center for AIDS Prevention Studies

2
Key Questions To Consider When Asking Questions

• Is this item understandable to the participant?
• Is this item something the participant can
  answer?
• Is this item something the participant will be
  willing to answer honestly?
• How important is this item compared to others I
  have to avoid respondent burden?

3
Areas to Consider in Designing Your Survey

• What type of survey methods are out there and
  what can they do for me?
• What are the things to consider in maximizing the
  quality of data with respect to questions I ask?
• How do I consider using existing scales and
  putting it all together?

4
Modes of Survey Delivery
5
Survey Method Options

• IAQ
• SAQ
• Including Diary Methods, Mail Surveys
• CASI/CAPI (and A-CASI)
• CATI
• T-ACASI
• Internet

6
Interviewer-Administered Questionnaire (IAQ)

• Most flexible and responsive to respondent re
  comprehension of question, selection of response
  options.
• Able to handle skip patterns that could confuse
  the respondent in a self-administered
  questionnaire.
• Human element, rapport responsiveness also can
  introduce problems in lack of standardization of
  delivery, self-presentation.

7
Self-Administered Questionnaire (SAQ)

• Allows for greater privacy than face-to-face
  interviewing
• Tends to be associated with higher rates of
  reporting of sensitive behaviors
• Literacy requirements for respondent
• Cannot be adjusted to be responsive to needs of
  individual participant
• Cannot handle complex skip patterns
• Low-cost survey delivery option

8
Diary Methods

• Provides optimal potential for recording of
  accurate rates of target behaviors
• Recall
• Exploration of variations over time
• Allows for question design to examine
  co-occurrence of key behaviors (e.g., sexual
  behavior and drug use)
• Burden of consistent completion of information
  means that data may be missing in possibly
  non-random pattern, or completed at a later time
  than designed.
• NOTE Can be useful to design for Web-based
  implementation.

9
Mail Surveys

• Many of considerations of SAQ are applicable --
  with format even more crucial.
• Many respondents report liking the flexibility of
  completing surveys at their own pace.
• Need for clear and easy format to maintain
  respondent interest and ease of completion.
• Critical to have contact information and clear
  procedures to follow up on respondents to ensure
  adequate response rates.

10
Computer-Assisted Self-Interview /Personal
Interview (CASI/CAPI)

• Survey is delivered via computer (can be laptops
  for maximal flexibility), with respondent keying
  in responses.
• Usual mode is Audio-CASI or A-CASI, which
  involves respondent listening to audio recording
  of survey as they see items on-screen.
• For special PCs with touch screens, can avoid
  keyboard and use fingers to select choice by
  touching on-screen.

11
CASI/A-CASI Advantages

• Privacy
• Branching in complex questionnaires
• Automated consistency checks
• Automated range checks
• Automated adaptation of question wording based
  upon prior responses
• Audio-CASI allows for standardized verbal
  delivery of all questions and does not require
  respondent literacy

12
Computer-Assisted Telephone Interview (CATI)

• List samples of live residential phone lines
  are called to screen households and determine
  eligibility of members.
• Once consent is obtained, interviewer uses a
  computer to guide them through asking survey
  questions, providing prompts, entering data, etc.
  as in A-CASI.
• Interviewer also has some flexibility in dealing
  with respondent comprehension/literacy level.

13
CATI Advantages

• Opportunity to access representative sample
  (costly with eligibility requirements that are
  less common).
• Some advantages of live interviewer, but also
  anonymity of phone call.
• Has capacities of computer-assisted technology
  with respect to branching, adaptation of wording,
  consistency/range checks, etc.

14
Telephone Audio Computer-Assisted Self-Interview
(T-ACASI)

• First a telephone interviewer contacts a
  household, screens for an eligible respondent,
  and secures consent.
• Phone call then transferred to automated system
  where computer-controlled, pre-recorded questions
  are read aloud.
• Respondent provides answers by pressing keys on a
  touch-tone phone.
• At end, respondent can be returned to live person
  to close out interview.

15
Internet Surveys

• Access to large sample at relatively low cost in
  relatively short time frame.
• May provide access to difficult-to-reach
  populations.
• As with other computer-assisted methods, may
  result in higher rates of reporting of sensitive
  information, and provides consistency and
  validity checks.
• Rates of internet access are growing (if still
  disparities exist in different populations).
• Participation rates of target population hard to
  gauge
• Dropout rates
• Direct data entry into database facilitates
  cleaning and analysis.

16
Summary Survey Delivery Considerations

• Mode of survey delivery clearly influence
• Respondent comfort and likelihood of
  self-disclosure (due to level of privacy)
• Respondent ease of navigating through the survey
  if there are any skip patterns
• Time and effort expended per respondent
• The audience you can reach with your survey
• Quality of data, data entry and analysis
• While important, how you actually ask the
  questions is just as critical -- our next focus.

17
Survey Construction
18
Developing the Survey Question Organization

• Topics should be generally organized from least
  to most intimate
• Initial questions are crucial -- respondent must
  get hooked by at least some questions which are
  felt to be personally salient
• Initial questions must also require low effort
  (cognitive demands) to answer
• Thus, if possible, avoid demographic questions to
  start (especially income!)

19
Developing the Survey Question Organization

• A survey -- no matter how delivered -- is a type
  of conversation.
• Topics should follow an orderly sequence, with
  common questions grouped together.
• This flow allows transitions to more intimate
  matters without jarring or acute awareness.
• Transitional phrases can help in shifting set.
• All this also provides respondent with notion
  that there is a rationale to questions asked.

20
Question Ordering

• Be careful with filter questions if not worded
  carefully, respondents will learn that a no
  response means avoiding subsequent questions.
• Be careful about summary attitudinal questions
  more specific queries to begin will influence
  summary responses by forcing respondent to think
  about their overall beliefs more carefully.

21
Question Ordering

• Having similar response choices reduces cognitive
  burden on respondent grouping items with such
  options together is thus useful. It can also be
  useful to have the same number of options to
  reduce complexity (e.g., always using a
  four-point or five-point scale).
• However, arrange types of questions to provide
  some variety and reduce response set.
• Avoid asking unnecessary questions!!

22
Question Wording/Construction
23
Comprehension/Interpretation

• Not just question of literacy, but influenced by
  cognitive burden and attention required
• Wording choices
• Length and complexity of survey items
• Ordering of clauses
• Consistency of phrasing
• Avoid nots or items whose meaning will be lost
  by inattention to a single word
• Phrasing also influences response distribution
  along a Likert-type agree-disagree scale

24
Cultural Considerations

• Individuals can come up with very different
  understandings of survey items, based upon their
  personal experiences.
• The meanings that respondents bring to the survey
  experience can be strongly influenced by culture,
  acculturation and other background variables --
  not just education and economic status.
• These differences are highlighted in the vast
  literature that exists on different response
  patterns of different ethnic groups to the same
  standardized scales. (For a review of some
  issues, see Pasick, Stewart, Bird DOnofrio,
  2001.)

25
Key Considerations

• Examine all survey items with eye for their
  appropriateness for target population.
• Be aware of pre-existing measures so as to not
  have to reinvent the wheel.
• When using pre-existing scales, review in same
  way for clarity ease of response.
• Get feedback on item wording and construction
  from as many colleagues as possible.
• Pre-test survey, using methods first described by
  Cannell et al. (1989).

Cannell C, Oksenberg I, Kalton G, Bischoping K
Fowler F. (1989). New Techniques in
Pretesting Survey Questions (NCHSR HS 05616).
Survey Research Center, University of Michigan.
26
Cognitive Burden Recall Demand

• Recall accuracy is influenced by a variety of
  factors, including
• salience/vividness of behavior,
• time frame,
• complexity of response information necessary
  (ever, counts, close-ended categories, etc.),
• frequency and consistency of behavior,
• effort respondent is willing to exert,
• cueing techniques and prompts

27
Time Frame - Sexual/Drug Use Behaviors

• 3 month time frame generally considered the
  outside range of accurate reporting some suggest
  limiting to 1 month.
• Longer time frame may be possible if asking if a
  behavior ever happened.
• One important timeframe consideration is dealing
  with infrequent behaviors -- what do you want to
  capture with respect to either behavior events or
  possible behavior change (if longitudinal study)?

28
Sensitivity

• Item sensitivity not consistent across
  populations (e.g., sexual behavior among MSM vs.
  general population, drug use behavior among heavy
  drug abusers/IDUs, discussing sexuality among
  different ethnic groups).
• Rationale for sensitive questions helps response
  rates.
• Asking if ever occurred first, prior to asking
  about (for example) the last month, may be less
  charged for respondent.
• Importance of neutral wording of question and (if
  IAQ) careful training of interviewer

29
In Closing ...

• Selecting your methods of data collection must be
  carefully considered based upon target
  population, study questions and design, and
  costs.
• No matter what the method of data collection,
  your instrument is critical.
• Survey design is a combination of art and
  science, with the art strongly reliant upon
  interviewing skills.
• The science of survey design currently has a
  wealth of research literature to examine and
  guide you.