Cognitive Interviewing for Question Evaluation

1
Cognitive Interviewing for Question Evaluation
Kristen Miller, Ph.D. National Center for Health
Statistics ksmiller_at_cdc.gov
2
Why Question Evaluation?

• Ensure questions capture intended concept
• Identify incomparable survey data
• Cultural beliefs
• Response problems for vulnerable populations
• Lower socio-economic status
• Fewer resources
• Translation problems
• To fix problems

3
Why Question Evaluation?

• To identify and document what the question
  measures
• Not just what is wrong with the question
• Identify and describe subtle differences in
• Patterns of interpretation
• Patterns of calculation
• To support data users when conducting analysis of
  survey data

4
Question Evaluation should answer

• How do the respondents understand the survey
  question?
• Do respondents understand the survey question
  differently?
• Does the question mean the same in all the
  languages that it is asked?
• Does the question mean the same in all of the
  cultures that it is asked?

5
Question Evaluation should answer

• In processing a question, do all respondents
  recall information and form an answer the same
  way?
• What groups should be considered for
  comparability?
• Country? Language? Age? Education? Income?
  Gender? Health Status?

6
Presentation Outline

• Define tenets of good question design
• Describe cognitive interviewing
• What it is and how to do it
• Benefits for
• Question designer
• Data analysts
• Outline WG Cognitive Interviewing II

7
A good question is

• 1. relevant to the research agenda
• and
• 2. relevant to each potential respondents
  experience and knowledge.

8
What to Remember about Respondents

• Do not know or understand the research question
• Most likely, do not use scientific, abstract
  concepts
• Survey puts them in the position of operating as
  informants
• Reference aspects of their lives

9
When this relationship is broken, error is
introduced into the data.

• False Positives
• False Negatives
• An entirely different phenomena is measured than
  intended by the research agenda
• Example Terrorism

10
Cognitive Stages to Survey Question Response

1. Comprehension the respondent interprets the
   question
2. Retrieval the respondent searches memory for
   relevant information
3. Judgment the respondent assesses the
   completeness and relevance of memories, and makes
   an estimation
4. Response Maps judgment onto response category
   may need to edit response to fit the category

11
Question Response Process
Social Factors
Social Factors
Social Factors
Retrieval
Comprehension
Judgment
Response
Social Factors
Social Factors
Social Factors
Social Factors
12
Cognitive Interviews

• Designed to understand how respondents
  comprehend, retrieve, judge, respond to questions
• Through this examination, can identify
• potential response errors
• patterns of interpretation
• Provide insight into social-cultural factors that
  impact the response process

13
Cognitive Interviews are Semi-structured

• Core Question- interview is organized by the
  questions that are being tested
• Probe Questions- open-ended, spontaneous, not
  pre-scripted, based on the information that the
  respondent provides

14
Probing for story

• Why did the respondent answer the question the
  way that they did?
• Does this story match with the intent of the
  question?
• video

15
Cognitive Test

• Qualitative
• Small sample
• Sample selection purposive
• Examines thought processes of respondent
• How does the question work?
• Does the question work as intended?
• If not, how can it be fixed?

16
Data from Cognitive Testing

• Collected from semi-structured protocol
• Narrative format
• Validity tied to rich detail
• Findings are grounded
• Insight into question interpretation
• Insight into patterns of calculation

17
Cognitive Interview Findings

• Provide knowledge of question performance
• Illustrate what the question measures
• Varied patterns of interpretation
• Dimensions of response error

18
Overall, during the past 4 weeks, how much
difficulty did you have with thinking clearly and
solving daily problems?
Respondent 2
Respondent 1
Alzheimers disease
Busy
Long term, medical problem
Remembering detailed list
Respondent 3
Respondent 6
Specific experience- organizing tenants
Fiscal functioning
Respondent 5
Respondent 4
19
Analysis of Cognitive Interviews

• Yields an inventory of
• Interpretations
• Patterns of calculation
• Types of errors
• Provides an explanation of inventory
• Shows if and how patterns are interrelated
• E.g., does a particular country or group of
  respondents interpret a question differently than
  all others? If so, why?

20
Cognitive Interviewing Reports

• Methodology
• (e.g. N and demographics of respondents,
  recruitment and interviewing protocols)
• Analysis summary
• Question by question reviewdetails
• interpretations,
• calculation processes
• types of errors

21
Examples of Cognitive Reports

• Q-Bank Database of question evaluation reports
  searchable by question
• Q-Bank Website
• http//wwwn.cdc.gov/QBANK/Home.aspx

22
WG Cognitive Interviewing Round II
23
Overall Testing Plan

• First,
• Cognitive interviewing study (traditional)
• Small purposive samples
• Qualitative data and analysis
• To gain detailed insight processes
• i.e. interpretations, calculations, errors
• January-February, 2009

24
Overall Testing Plan

• Then,
• Field test
• Larger, non-purposive sample
• Quantitative data and analysis
• Questionnaire contains
• WG questions
• Issues identified in cognitive interviewing
• To learn prevalence or scope of problems
• March April, 2009

25
Cognitive Interviewing Project

• Novel approach using the best and current
  knowledge of pre-testing
• Utilization of new software for analyzing
  cognitive interviews
• Integrating lessons learned from WG Test Round 1
  and other pre-testing studies
• Based on group effort and collaboration

26
Cognitive Interviewing Study Timeline

• Interviewer Training January, 2009 Bangkok
• Participating countries each conduct 20
  interviews
• Preliminary Analysis Kristen and friends
• Analysis Meeting February, 2009 Bangkok
• Documentation and Report Kristen and friends

27
Country Invitation Requirements for
Participation

• One or Two cognitive interviewers
• NOT field interviewers
• Must understand question-response process
• Fluency in both English and language of
  respondents
• Attend Bangkok training
• Conduct 20 interviews
• Access internet for communication with group
• Document interviews in the provided template
• Prepare for attend Bangkok analysis meeting

28
Questions?