Cognitive Interviewing and Questionnaire Design

1
Cognitive Interviewing and Questionnaire Design

• Gordon Willis, Ph.D.National Cancer Institute
• National Institutes of Health
• willisg_at_mail.nih.gov

2
Preface Why worry about questionnairedesign
problems?

• On the one hand
• A self-report sample survey is a blunt
  instrument
• There is bound to be some slop
• To the extent questionnaire design IS rocket
  science, were content to just hit the moon,
  somewhere
• On the other hand
• Of the various types of errors afflicting surveys
  Sampling error, Non-response error, Processing
  error
• Response Error (e.g., bias or unreliability) is
  often the largest category -gt

3
Do small differences in wordingproduce
differences in data?

• VERSION 1 (No filter)
• On a typical day, how much time do you spend
  doing strenuous physical activities such as
  lifting, pushing, or pulling?
• __ None __ Less than 1 hour __ 1-4 hours __ 5
  hours
• VERSION 2 (Filtered)
• On a typical day, do you spend any time doing
  strenuous physical activities such as lifting,
  pushing, or pulling?
• IF YES Read Version 1

4
Evaluation study Embed target questions,
using three split-sample survey experiments

• 1) Field Pretest of NCHS National Health
  Interview Survey 78 adult household respondents
• 2) Womens Health Study 191 women, 18-41, in
  contractor offices
• 3) Random-Digit Dial (RDD) Survey 989
  respondents in Maryland Omnibus telephone survey
• (Willis Schechter, 1997)

5
Survey experiment results Screening for
physical activity

• On a typical day, how much time do you spend
  doing strenuous physical activities such as
  lifting, pushing, or pulling?
  0 lt1 1-4 5
• FIELD PRETEST (n78)
• No filter 32 32 35 0
• Filtered 72 18 10 0
• WOMENS HEALTH (n191)
• No filter 4 42 50 4
• Filtered 49 16 27 8
• RDD SURVEY(n989)
• No filter - 45 34 22
• Filtered - 62 29 9

6
Survey experiment results Screening for
physical activity

• On a typical day, how much time do you spend
  doing strenuous physical activities such as
  lifting, pushing, or pulling?
  0 lt1 1-4 5
• FIELD PRETEST (n78)
• No filter 32 32 35 0
• Filtered 72 18 10 0
• WOMENS HEALTH (n191)
• No filter 4 42 50 4
• Filtered 49 16 27 8
• RDD SURVEY(n989)
• No filter - 45 34 22
• Filtered - 62 29 9

7
The cognitive testing processin a nutshell

• Develop a questionnaire or material (advance
  letter, etc.) to be evaluated
• Recruit (paid) members of the targeted population
  (e.g., recipients of home loans, people without
  employment, cancer survivors)
• Conduct one-on-one interviews, in laboratory or
  other location
• Home
• Homeless shelter
• Health clinic
• Elderly center

8
The cognitive testing process in a nutshell

• THEN Make informed decisions, with cognitive
  testing as one source of input
• We also rely on
• a) Expert Review (Topic area and/or
  questionnaire design)
• b) Review of measurement objectives
• c) Consideration of past practice, need for
  comparability

9
The cognitive testing process, in a nutshell

• To conduct the cognitive interview, we use verbal
  probing techniques to elicit thinking about
  question
• Finding Note apparent problems related to
  question wording, ordering, format
• Fixing Suggest modifications that address
  problems
• Cognitive testing is best done as an iterative
  process (multiple small testing rounds)

10
Tourangeau (1984) cognitive model

• Encoding of question (understanding it)
• Have you ever received care from a podiatrist?
• Retrieval of information (knowing/remembering)
• Do you approve of the Budget Reconciliation Act
  of 1993?
• How many times have you ridden in a passenger
  airplane?
• Decision and judgment processes (truth, adequacy)
• How many sex partners have you had in the past 12
  months?
• Response (matching internal representation to
  given categories)
• Would you say your health is excellent, good,
  fair, or poor?

11
Focus Cognitive and other problems

• Focus is NOT just on cognitive problems
• (1) Problems with question assumptions
• Do you own or rent your house?
• (2) Problems for interviewer
• Question is tongue-twister, etc.
• (3) Format issues
• Where do I go next, for self-administered
  questionnaire (skips)
• (4) Issues that influence flow of the interview
• Subject thinks question is over and interrupts
  with answer
• As such, has been called Intensive
  interviewing (Cantril Fried, 1944(!) Royston,
  1989)

12
Think Aloud interviewing

• Mainly advocated by Ericsson and Simon (1980,
  1984) as a means to study cognitive processes
  used in problem solving (chess)
• The interviewer requests a verbal train as
  subject completes the task
• The interviewer interjects little except to say
  Keep talking or Tell me what youre thinking
• In the extreme case, avoid the use of references
  to me the interviewer(!)

13
Example of think-aloud

• INTERVIEWER How many times have you talked to a
  doctor in the last 12 months?
• SUBJECT I guess that depends on what you mean
  when you say talked. I talk to my neighbor,
  who is a doctorI go to my doctor about once a
  year, for a general check-up, so I would count
  that. Ive been toa specialist a couple of
  times in the past year - once to get a bad knee
  diagnosed, and I also saw an ENT about a coughing
  thing, which Im pretty sure was in the past
  year, although I wouldnt swear to it. I also
  talked to doctors when I brought my kids to the
  pediatrician - I assume that you dont want that,
  though Im not sure. Also, I saw a chiropractor,
  but I dont know if you consider that to be a
  doctor. So, Im not sure what number to give
  you

14
Limitations of pure Think-Aloud

• Many subjects are not good at this
• My experience male, teenage smokers
• Was not designed for a task that involves
  social interaction (survey)
• Is very different from the normal question
  asking-answering sequence
• Analysis of think-aloud protocol is not
  straightforward
• We can use a coding system (e.g., Bickart
  Felcher, 1996), but this is complex and
  burdensome

15
The alternative to Think-AloudVerbal Probing
Techniques

• Ask follow-up questions
• About the questions
• About the answer
• About the general context
• Not new Used by Cantril (1944), Belson (1980)
• Verbal probing is more active than think-aloud
• Requires decision by the interviewer about how to
  probe -gt This is what makes the activity
  interesting!

16
Basic varieties of (classic) verbal probes

• Comprehension probe What does the term
  dental sealant mean to you?
• Paraphrase Can you repeat the question in
  your own words?
• Confidence judgment How sure are you that your
  health insurance covers
• Recall probe How do you know that you went
  to the doctor 3 times?
• Specific probe Why do you think that breast
  cancer is the most serious health
  problem?
• General probe How did you arrive at that
  answer?

17
The combined approach

• Cognitive interviews are normally done as
  combination of Probing and Think-Aloud (DeMaio
  Landreth, 2004)
• But, the relative mix of these may vary across
  type of investigation

18
Verbal probing techniques Concurrent versus
retrospective approach

• Concurrent probing Probe immediately after the
  subject has answered each survey question
• Advantage Probing when the memory still exists
• Disadvantage Measurement process interrupts
  normal flow
• Retrospective probing Wait until after the
  interview, and then go back to probe
• Advantage Mirrors field procedures
• - Is particularly useful for
  self-administered Qs
• Disadvantage Subject may have forgotten key
  information

19

• Thesis
• C.I. depends on more than just the process of
  asking probe questions
• We need to start with a working knowledge of
  questionnaire design
• We cant just make up probes haphazardly
• We cant just leave it to the subject to tell us
  something is wrong!!!
• Active probing requires the targeting and
  recognition of problems
• Model Question Appraisal System -gt
• (Willis Lessler, 1999 Willis, 2009)

20
(No Transcript)
21
More evidence that we shoulddo Expert Review
22
Fashioning probes, in depth

23
Tested (classic) question Pain in the abdomen

• In the last year have you been bothered by pain
  in the abdomen?
• What (Anticipated) probes make sense here?
• What time period are you thinking about, exactly?
• What does bothered by pain mean to you?
• Where is your abdomen? ?

24
(No Transcript)
25
(No Transcript)
26
Revised questionPain in abdomen

• (Please look at this diagram. During the past 12
  months, have you had pain in this area)
• Remember to check out Ss interpretation of vague
  reference periods like year, month, week,
  or no reference period at all (e.g., How often
  do you X?).
• Argument Lose this example Its obvious
  that Abdomen is a problem
• Response Yes, but sometimes we need to
  demonstrate that which we think we know

27
TV, Smoking

• During the past 30 days, about how often have
  you seen anti-smoking commercials on TV?
• Probe that was used (Paraphrase) What was
  that question asking you?
• Subject About smoking commercials I dont
  think there are any
• So, he heard anti-smoking and thought smoking
  -- NO GOOD
• ? A straightforward answer can hide a Silent
  Misinterpretation (DeMaio and Rothgeb, 1996)

28
Fixing potential alternatives

• During the past 30 days, have you seen any TV
  commercials about the dangers of cigarette
  smoking?
• During the past 30 days, have you seen any TV
  commercials that warned people NOT to smoke?
• During the past 30 days, have you seen any
  messages on TV warning people NOT to smoke?
• -gt Which is best? Depends on objective may
  need to consult with sponsor/client
• -gt Cognitive interviewing is often a vagueness
  detector which brings out hidden ambiguity

29
Beyond Proactively searching

• Illustrative example of tested question
• Interviewer Inside your home, are there any
  walls that have peeling paint?
• Subject No not on the walls, anyway
• I (Probe) Is there any paint thats peeling?
• S Yeah, the window frame
• Problem wasnt anticipated beforehand so
  neither was the probe
• Cog Interviewing needs a capacity not only for
  problem Verification, but also for reaction to
  the unexpected Discovery

30
Easy default tell me more

• Example
• I How often is your household trash collected?
  S Twice a week.
• I Tell me more about your trash pickup.
• S The garbage gets picked up on Monday, and
  the recycling on Wednesday.
• NOT GOOD The objective concerns trash, not
  recycling
• Probing revealed that the tested version doesnt
  work
• -gt Good example of a Silent Misinterpretation

31
How would you probe?

• What types of smokeless tobacco products have you
  used in the past year?
• Proactive forms
• To you, whats a smokeless tobacco product?
• When did you last use (X)? How sure are you that
  this was within the past year?
• Reactive forms
• It depends on what we hear -gt

32
How would you probe?

• What types of smokeless tobacco products have you
  used in the past year?
• How about if given this response
• Well, I used to chew tobacco
• Does he mean in the past year, or longer ago?
• I would first follow up on the Reactive part,
  before Proactively fishing -
• -gt Follow problems before searching for them
• So How long ago did you last chew tobacco?
• You can then follow up with other (Anticipated)
  probes Besides chewing tobacco, what do you
  consider to be a smokeless tobacco product

33
An alternative to probing listening

• When you first thought you were pregnant, how did
  you feel? Were you -
• Happy,
• Unhappy,
• Both happy and unhappy, or
• Neither happy nor unhappy
• Say that we observe a favorite subject ploy She
  jumps in at how did you feel with I felt
  awful!
• So, the respondent thinks the question is over
  (or that it should be)

34
So, we change the question

• When you first thought you were pregnant, were
  you -
• Happy,
• Unhappy,
• Both happy and unhappy, or
• Neither happy nor unhappy
• Subject I dont get all those different
  happy-unhappy things
• -gt Again, we need only to listen no probing may
  be necessary

35
So, we change the question

• When you first thought you were pregnant, were
  you
• Happy,
• Unhappy,
• or did you have mixed feelings?
• Lessons
• 1) There may be multiple problems
• 2) Iterative testing is good
• 3) We may not need to probe but, we react
  differently from a field interviewer (who is
  motivated to make it work)

36
Avoiding Probing Pitfalls
37
Probing fouls

• I what about contamination of the ground
  water. Would you say its a very important
  problem, a somewhat important problem, a small
  problem, or not a problem?
• S I guess hmmm. Im not sure how to answer
  that, really.
• I Why dont you know how to answer it?
• Problems with probe?
• Kind of accusatory
• Im not sure that a person who cant answer can
  further report why they cant answer
• I would prefer Ok, tell me what youre thinking

38
Probing fouls continued

• (I Why dont you know how to answer it?)
• S Well, I dont know how to answer it, because
  I dont know really what your mean um
• I OK, whatever it means to you. Again, please
  indicate whether its a very important problem,
  a somewhat important problem, a small problem,
  or not a problem
• S Oh, its a very important problem
• Problems with probe? -gt

39
Probing fouls continued

• (I OK, whatever it means to you. Again, please
  indicate whether its a very important problem, a
  somewhat important problem, a small problem, or
  not a problem )
• (S Oh, its a very important problem)
• PROBLEMS WITH PROBING
• Whatever it means to you is a field interviewer
  activity designed only to get through the
  questionnaire
• Do we know that S is still thinking about the
  actual topic of the question (what is it thats
  an important problem?)
• Responses to probing are subject to social
  desirability effects (just as survey questions)
  

40
Rule of probing flaws

• Probes follow all the usual rules of
  questionnaire design
• They may be too long
• They may be vague
• They may produce huh? response
• They can be biasing
• They can be irrelevant to the exchange
• If so, they must be fixed, or abandoned!

41
Modifying probes to recover from communication
failure Wellens (1994)

• I Are you a citizen of the United States?
• S No.
• I In your own words, what does the term
  citizen mean to you?
• S I dont I dont understand
• I I just want to know what you think the
  definition of the word citizen is.
• S A citizen is a person who belongs to this
  country. That person has the right to vote or
  join the government.
• I Can you tell me more about why you are not a
  citizen?
• S A person must live here for five years, take a
  test, obtain history course, then the person can
  become citizen.

42
Adapting to Survey Administration Mode
43
Adapting cognitive interviewing to Face-to-face
administration

• This is the default the situation that is most
  similar to the in-person cognitive interview
• So, theres no not much to adapt to
• The main thing to keep in mind is that the
  cognitive interviewer is not the same as the
  field interviewer
• -gt So, pay attention to problems for the
  interviewer as well as for the subject

44
Adapting cognitive interviewing to Telephone
administration

• Important issue
• Can C.I. be done over the phone?
• Should C.I. be done over the phone?
• - We miss out on body language
• - But, phone may be the mode of fielded survey
  administration
• NCHS procedure
• Put subject in separate room and call him/her
• Conduct interview over phone, but with camera
• Go to Subject location, debrief in-person

45
Adapting cognitive interviewing to Telephone
administration

• Alternative -
• Conduct an initial interviewing round
  face-to-face, and a second round over the
  telephone (e.g., CDC Behavior Risk Factor
  Surveillance Survey)
• We are likely to find that the telephone produces
  impediments to question functioning

46
Adapting cognitive interviewing to
Self-administration

• Another interesting/unresolved area
• The cognitive demands of the S/A task are
  different from those of face-to-face and phone
• S/A relies on visual rather than auditory
  processing
• S/A sometimes requires navigational activities
  (find the starting point, follow skip patterns)
• So, for the C.I., it may not make sense to
  follow the usual procedure of having the
  interviewer read the questions to the subject

47
Adapting cognitive interviewing to
Self-administration

• So It makes sense in the C.I. to have the
  subject read the form
• But, do we then do the cog interview
  concurrently, or retrospectively?
• If concurrent, should we use (a) uninterrupted
  think-aloud, (b) interruption with probes?, or
  (c) both
• Issue Which procedure produces more distortion
  of the question answering process, and therefore
  produces reactivity effects?

48
Example Self-administered teen tobacco
questionnaire

• Do you think young people who smoke cigarettes
  have more friends?
• Definitely yes
• Probably yes
• Probably not
• Definitely not
• Retrospective probe What does young people
  mean? S 13 year old
• Observation S left the answer blank objects
  that It doesnt make any difference - but
  theres no way to say that
• Expert judgment Seems classically biased to me
  (presents one possibility, but not others)

49
Revised question

• Do you think that people your age who smoke
  cigarettes have more friends, fewer friends, or
  the same number of friends as those who dont
  smoke?
• People who smoke have MORE friends
• People who smoke have FEWER friends
• People who smoke and those who dont have the
  SAME number of friends
• ? Fixes may be appropriate for one mode but
  not another. The current example lengthens the
  question and seemed to be ok for
  self-administration, but may be too much for the
  phone.

50
Adapting C.I. to Web surveys

• The Web Usability Testing area is huge, and well
  entrenched (Mick Couper)
• It shares MANY commonalities with cognitive
  interviewing of S/A questionnaires
• But there is a major emphasis on use of the
  system navigation, problem-solving (e.g., how
  do I go back and change an answer, skip around)
• For this reason, it bears similarities to the
  classic Ericsson-Simon think-aloud task
• Cleo Redline (NSF), Jennifer Crafts (Westat)
  Think-aloud works for usability testing- we
  observe how the subject handles a complex,
  non-verbal activity

51
Adapting C.I. to Web surveys

• Challenge Integrating C.I. and Usability
  testing
• We need to consider both (a) the questionnaire
  and (b) Usability of the computer system
• Makes sense to (a) test questionnaire prior to
  programming, but to (b) include regular
  cognitive testing during Usability test (combine
  these so we dont forget about the questions!)
• Example of a complex Web questionnaire -gt

52

1. Where do users look?
2. Can they tell what the panel on the left is for?
3. Do they go directly to the question?
4. Can they tell what the colors signify?

53

1. Does it matter that there are no question
   numbers?
2. Are the questions understandable?

54
Examples of Cognitive Testing Reports

• Many cognitive testing reports are retrievable
  from the Web, through the Q-BANK database
• Miller, et al. (National Center for Health
  Statistics, Centers for Disease Control and
  Prevention)
• Q-Bank Home http//wwwn.cdc.gov/qbank/Home.aspx
  
• -gt Consult these reports for examples of how
  cognitive testing results are written up in U.S.
  Federal Agencies

55
Logistics of the Cognitive Interview Process
56
Practical considerations

• 1) How long should a Cog Interview be?
• One hour is reasonable
• Avoids fatigue (for Interviewer, and Subject)
• 2) How long a questionnaire can be tested in 1
  hr?
• Best measured in terms of administration time,
  not pages
• Depends on probing intensity
• DeMaio and Landreth 3 lab minutes for each
  questionnaire minute
• Me 2 lab minutes for 1 questionnaire minute

57
Practical considerations

• 3) How many interviews can you do in one day?
• A LOT but not for too many days!
• I have done up to 6
• Plan a maximum of three, on a regular basis
• 4) What is appropriate payment (remuneration)?
• Industry standard is US 25 - 50 for one-hour
• May need to pay more for special populations
• Interestingly, paying more brings in lower-income
  (NCHS finding), as theyre not just doing hobby
  research

58
Practical considerations

• 5) How many interviews should be conducted?
• The question sub-divides into (a) How many
  rounds and (b) How many per round?
• For pretesting purposes, I like doing no more
  than 12 before stopping to review and modify
• Note that different problems require different
  effective sample sizes to detect and to correct
• If we stop to change half the questions, and do
  another round, the other half get two rounds
  worth of unaltered testing
• A nice plan three iterative rounds of 10
  subjects
• Normally, we run out of time before running out
  of problems

59
Practical considerations

• 6) How much time is required for the various
  activities involved?
• For every 1-hour interview, plan another 3 hours
  for preparation, processing, and writing up
  results
• It is a good idea to listen to the interview
  again
• It takes awhile to make sense of and write up
  results
• You may have several debriefing meetings with
  project staff
• Concerning clock time, two activities tend to
  take time, and spread out interviewing rounds
• Recruitment (can take several weeks)
• Getting feedback from client/team before next
  testing round

60
Practical considerations

• 7) What staffing do you need to run a lab?
• Few people will have a true Cognitive Lab
• So, you can do everything yourself, and still get
  something out of it
• For a Lab
• Several staff should be trained as interviewers
• Good to have a staff member who is a Laboratory
  Manager (advertisements, recruitment, logistics)

61
Practical considerations

• 8) Do you need multiple interviewers?
• Some successful interviewing rounds have been
  done with one cognitive interviewer
• May be necessary As for multiple-language
  testing
• I prefer 2-3 interviewers
• Get differing approaches/perspectives on
  interviewing
• Have variation in questionnaire design expertise
• Can look for consistency across interviewers in
  results

62
Practical considerations

• 9) Is there value in very limited efforts?
• Good question What if I can only do 3 or 4
  interviews, myself --
• Is this better (or worse) than doing nothing?
• I find it hard to believe that this can be
  harmful
• The smaller the sample, the more conservative we
  need to be in accepting results as real
• But, if you administer a questionnaire to 4
  people and none of them have any idea what youre
  asking its time to revisit the approach

63
Practical considerations

• 10) What equipment do I need for my lab
• You dont need a lab
• You do need
• A private, quiet place to interview
• An audio recorder

64
Practical considerations

• 11) What about confidentiality?
• Cognitive Interviewing is research, so pay
  attention to issues of Human Subjects Protection
• Always good to use a consent form
• Dont keep forms with subject identifiers laying
  around
• Eventually, destroy tapes/videos
• If you plan to use interviews for demonstration
  purposes, have subjects sign a separate release
  for that purpose

65
Practical considerations

• 12) How can clients/team be integrated into the
  process?
• Dont isolate recipients of information include
  them!
• Seeing is believing Stakeholders can be
  influenced by observing a single interview
• Contractors normally invite clients to observe
  through video setup/one-way mirror, or watch tapes

66
Useful Internet Resources

• 1) US Census Bureau guide to self-administration
  http//www.census.gov/srd/Economic_Directorate_Gui
  delines_on_Questionnaire_Design.pdf
• 2) NCI resources
• a) The Question Appraisal System (Full version)
• http//appliedresearch.cancer.gov/areas/cognitive/
  qas99.pdf
• b) Cognitive Interviewing A How-To Guide
• http//appliedresearch.cancer.gov/areas/cognitive/
  qas99.pdf
• c) An Introduction to Modern Measurement Theory
• http//appliedresearch.cancer.gov/areas/cognitive/
  immt.pdf

67
In closing

• The uncreative mind can spot wrong answers,
• but it takes a very creative mind to spot wrong
  questions.
• - Anthony Jay