Questionnaire Design, Pretesting and Quality Control in Clinical Research

1
Questionnaire Design, Pretesting and Quality
Control in Clinical Research

• Celia P. Kaplan
• Division of General Internal Medicine
• August 24th 2009

2
Good Clinical Practice (GCP)

• A standard for the design, conduct,
  performance, monitoring, auditing, recording,
  analysis, and reporting of clinical trials that
  provides assurances that the data and reported
  results are credible and accurate, and that the
  rights, integrity, and confidentiality of trial
  subjects are protected
• WHO good clinical practice standards

3
Quality Control
4
Quality Control

• Minimizes the chance of obtaining faulty data and
  drawing erroneous conclusions

5
Quality Control

• Minimizes the chance of obtaining faulty data and
  drawing erroneous conclusions

• Fulfills administrative and regulatory obligations

6
Sources of Data Errors

• Data acquisition

7
Sources of Data Errors

• Data acquisition
• instrument problems

8
Sources of Data Errors

• Data acquisition
• problems instrument
• lost data

9
Sources of Data Errors

• Data acquisition
• instrument problems
• lost data
• falsified data (made up data, changed )

10
Sources of Data Errors

• Data acquisition
• instrument problems
• lost data
• falsified data (made up data, changed)
• Data recording

11
Sources of Data Errors

• Data acquisition
• instrument problems
• lost data
• falsified data (made up data or changed)
• Data recording
• Data entry

12
Sources of Data Errors

• Data acquisition
• instrument problems
• lost data
• falsified data (made up data or changed)
• Data recording
• Data entry
• Data management

13
Quality Control Stages

• Pre-implementation/Developmental Stage

14
Quality Control Stages

• Pre-implementation/Developmental Stage
• Design high-quality questionnaires/interviews and
  forms

15
Quality Control Stages

• Pre-implementation/Developmental Stage
• Design high-quality questionnaires/interviews and
  forms
• Pretest all aspects of the project

16
Quality Control Stages

• Pre-implementation/Developmental Stage
• Design high-quality questionnaires/interviews and
  forms
• Pretest all aspects of the project
• Develop an Operations Manual

17
Quality Control Stages

• Pre-implementation/Developmental Stage
• Design high-quality questionnaires and forms
• Pretest all aspects of the project
• Develop an Operations Manual
• Design a secure data-management system

18
Quality Control Stages

• Pre-implementation/Developmental Stage
• Design high-quality questionnaires and forms
• Pretest all aspects of the project
• Develop an Operations Manual
• Design a secure data-management system
• Conduct staff training and certification

19
Quality Control Stages

• Pre-implementation/Developmental Stage
• Design high-quality questionnaires and forms
• Pretest all aspects of the project
• Develop an Operations Manual
• Design a secure data-management system
• Conduct staff training and certification
• Implementation

20
Quality Control Stages

• Pre-implementation/Developmental Stage
• Design high-quality questionnaires and forms
• Pretest all aspects of the project
• Develop an Operations Manual
• Design a secure data-management system
• Conduct staff training and certification
• Implementation
• Make periodic study- and data-monitoring reports

21
Quality Control Stages

• Pre-implementation/Developmental Stage
• Design high-quality questionnaires and forms
• Pretest all aspects of the project
• Develop an Operations Manual
• Design a secure data-management system
• Conduct staff training and certification
• Implementation
• Make periodic study- and data-monitoring reports
• Conduct periodic project meetings

22
Quality Control Stages

• Pre-implementation/Developmental Stage
• Design high-quality questionnaires and forms
• Pretest all aspects of the project
• Develop an Operations Manual
• Design a secure data-management system
• Conduct staff training and certification
• Implementation
• Make periodic study- and data-monitoring reports
• Conduct periodic project meetings
• Complete interim staff training and performance
  reviews

23
Quality Control Stages

• Pre-implementation/Developmental Stage
• Design high-quality questionnaires/interviews and
  forms
• Pretest all aspects of the project
• Develop an Operations Manual
• Design a secure data-management system
• Conduct staff training and certification
• Implementation
• Make periodic study- and data-monitoring reports
• Conduct periodic project meetings
• Complete interim staff training and performance
  reviews

24
Questionnaires/interviews

• Much of the data in clinical and epidemiological
  research is gathered using questionnaires and
  interviews

25
Questionnaire Measures

• Desired qualities
• Reliability the extent to which the measures
  give consistent results
• A reliable measure is consistent, responses do
  not change
• Test-retest reliability
• Statistical techniques
• Validity the extent to which the measures
  reflect the truth

26
Survey measures Validity

• Face validity the researchers best judgment as
  to whether the item is measuring what it is
  supposed to measure
• Predictive validity the item accurately predicts
  an outcome (e.g., college entrance exams
  predicting college performance)
• Concurrent validity the item is validated by
  comparing it against a known valid measure (e.g.,
  a measure of job satisfaction might be correlated
  with work performance)

27
Survey measures Validity

• Content validity the items are a representative
  sample of the characteristics that comprise the
  area they intend to measure. Determination of
  content validity usually relies on experts
• Example Patient decision making style
• Construct validity the item demonstrates that it
  accurately measures a psychological construct
  like depression or satisfaction

28
Questionnaire MeasuresDesired qualities
Reliability the extent to which the measures
give consistent resultsValidity the extent to
which the measures reflect the truthQuestio
nnaire MeasuresDesired qualities Reliability
the extent to which the measures give consistent
resultsValidity the extent to which the
measures reflect the truth Questionnaire
Measures

• Desired qualities
• Accommodate participants age, literacy,
  cultural, linguistic, and SES characteristics
• Yield good variability in answers
• Result in low percent of missing data

29
Questionnaire Development Pitfalls

• Double-barreled questions Avoid and/or
  statements in questions
• e.g., Does your department have a special
  recruitment policy for ethnic minorities or
  women?
• Long questions and/or questionnaires
• Use of other as a response category

30
Use of Other as Response CategoryExample
10. Are you Latino/a or Hispanic? 1 Yes 0
No 11. What is your race/ethnicity? Please
check one answer only. 1 Black or African
American 2 Asian, Asian American or Pacific
Islander 3 White, European American or
Caucasian 4 American Indian or Alaska Native 5
Other Please specify ____________________
31
Use of Other as Response CategoryExample
1 Ameriasian 19 INDIAN (EAST) AMERICAN
2 American 20 Italian
3 Arab 21 MIDDLE EAST
4 Ashkenazi 22 Middle Eastern/ Indian Subcontinent
5 Asian and Caucasian 23 Mixed
6 Brazilian 24 NATIVE AMERICAN
7 Cuban 25 Oriental
8 East Indian 26 PERSIAN
9 Egyptian 27 POLISH AMERICAN
10 EGYPTIAN 28 South Asian
11 egyptian american 29 Southern Asian
12 From India 30 West Indian
13 half Latin, half black 31 White, Arab
14 Indain American 32
15 India- Indian 33
16 Indian 34
17 INDIAN 35
18 Indian (east) 36
32
Questionnaire Development Order

• First question should be a fact
• Beginning questions should stimulate interest
• Questions should be in logical order and
  groupings
• If ask attitude questions first and then behavior
  questions, may inflate reports of behavior
• Sensitive questions and open-ended questions
  should be at the end of the survey
• e.g., sexual behavior, religion, income

33
Questionnaire Development Language

• Use simple, clear, common, and natural language
• Avoid jargon, complex terms, clichés,
  colloquialisms, e.g., down in the dumps
• Check number of syllables in words
• Check reading level aim for lt 8th grade
• Define ambiguous terms, e.g., family or
  neighborhood
• Keep sentences short

34
Questionnaire Development Recall

• Solutions
• Narrow the reference period
• Average the response ask about a typical day or
  in general (e.g., hours of sleep)
• Use landmark events or milestones to anchor time
  frame Since you moved to the US, or use
  calendar with major holidays to establish major
  life events
• Use cues to generate associations to jog memory
  describe procedure, setting

35
Questionnaire Design
1
Select Interview Methodology
Interview methodology
36
Questionnaire Design
2
Review existing questionnaires and measures
Interview methodology
Review existing measures
37
Questionnaire Design
3
Conduct exploratory qualitative work
Interview methodology
Review existing measures
Conduct exploratory work
38
Questionnaire Design
4
Develop a draft survey
Draft survey
Interview methodology
Review existing measures
Conduct exploratory work
39
Questionnaire Design
5
Translation and back-translation
Draft survey
Interview methodology
Review existing measures
Conduct exploratory work
Translation/ back-translation
40
Questionnaire Design
6
Pre-testing
Draft survey
Interview methodology
Review existing measures
Conduct exploratory work
Translation/ back-translation
Pre-test the survey
41
Questionnaire Design
6
Pre-testing
Draft survey
Interview methodology
Review existing measures
Conduct exploratory work
Translation/ back-translation
Pre-test the survey
42
Questionnaire Design
7
Finalize the Questionnaire
Draft survey
Interview methodology
Review existing measures
Conduct exploratory work
Finalized instrument
Translation/ back-translation
Pre-test the survey
43
Questionnaire design

• Selection of interview methodology
• Face-to-face or in-person interview
• Telephone
• Self-administered, paper
• Self-administered, computer/web/e-mail

44
Questionnaire design

• Interview methodology Face-to-Face
• Advantages
• Enables the interviewer to establish rapport with
  respondent
• Permits more complex questions
• Permits the use of visual aids
• Longer interviews are possible
• Disadvantages
• Cost per interview
• Quality of answers too personal (sensitive
  questions)

45
Questionnaire design

• Interview methodology Telephone
• Advantages
• Faster contact of participants
• Better to elicit personal/sensitive information
• Results are available right away if using
  computer-assisted interviewing (CAI)
• Can do random selection of telephone numbers
• Disadvantages
• Certain sectors of the population are switching
  to cell phones only
• More expensive than mail surveys
• Difficulties in reaching participants during the
  day

46
Questionnaire design

• Interview methodology Self-administered and mail
  surveys (Paper)
• Advantages
• One of the least expensive methodologies
• Needs addresses of potential participants
• Allows participants to answer at their leisure
• Less intrusive
• Elimination of interviewer bias
• Disadvantages
• Response rate is low
• Difficulties to develop clear skip patterns
• It may be answered by a family member
• Very difficult to implement in low literacy
  populations

47
Questionnaire design
Example Self-administered Mail Surveys
48
Questionnaire design

• Interview methodology Self-administered,
  web-based surveys (Computer)
• Advantages
• Elimination of data entry and editing costs
• Get better answers to sensitive questions
• Elimination of interviewer bias
• Ensure that skip patterns are accurately followed
• Allows participants to answer at their leisure
• Disadvantages
• Interviewees must have access to a computer
• Must possess (or purchase) a list of email
  addresses
• Lower response rate
• People may quit in the middle of a questionnaire

49
Questionnaire design
Example Web-based survey
50
Questionnaire design

• Survey modalities Answer types
• Numeric open-ended
• How old are you? ____
• Text open ended
• How can we improve our services? ______________
• Multiple Choice--Categorical
• What is your marital status
• Single 1
• Married 2
• etc.

51
Questionnaire design

• Survey modalities Answer types
• Multiple choice-- Ordinal
• In general, would you say your health is?
• 1 Excellent
• 2 Very good
• 3 Good
• 4 Fair
• 5 Poor
• 77 DK
• Scales
• On a scale where 10 means you have a great
  amount of interest in a topic and 1 means you
  have none at all, how would you rate your
  interest in each of the following specialties?
• Cardiology ___
• Oncology __
• Dermatology__

52
Questionnaire design

• Survey modalities Answer types
• Multiple choice-- Ordinal
• In general, would you say your health is?
• 1 Excellent
• 2 Very good
• 3 Good
• 4 Fair
• 5 Poor
• 77 DK
• Scales
• On a scale where 10 means you have a great
  amount of interest in a topic and 1 means you
  have none at all, how would you rate your
  interest in each of the following specialties?
• Cardiology ___
• Oncology __
• Dermatology__

53
Questionnaire design

• Survey modalities Answer types
• Likert-Type scales
• Used to know respondents' feelings or attitudes
• The respondent indicates how closely their
  feelings match the question or statement on a
  rating scale.
• Response categories Strongly Disagree" (least
  agreement) to or "Strongly Agree." (most
  agreement)
• Example
• In general, to what degree is each of these
  factors a barrier for you in referring a patient
  to a clinical trial?
• Not a barrier A major barrier
• 1 ? 2 ? 3 ? 4 ? 5
  ?

54
Questionnaire design

• Review Existing Questionnaires Measures
• Reference databases
• Medline, Pubmed, Psychinfo, others
• Compendia of measures
• Books that compile various measures and review
  their characteristics
• Request questionnaires from other investigators

55
Questionnaire design

• Review Existing Questionnaires Measures

• Is there evidence the instrument works in your
  target population?
• Examine validity and reliability of measures when
  used in populations similar to your study
  population
• When gold standard does not exist, for construct
  validity, look at relationship of measure to
  other measures to see if in hypothesized direction

56
Questionnaire design

• Conduct Qualitative Exploratory Work
• Focus groups or semi-structured interviews
• Explores a topic before constructing formal
  questions
• Rationale
• Determine what is important
• Discover how respondents think about the topic
• Assess better language/words to use
• Identify response categories

57
Questionnaire design

• Create a Draft Survey Instrument
• Write initial draft
• Process of revision and refinement through cycles
  of translation/back-translation, reviews and
  pre-tests (Steps 5 and 6)

58
Questionnaire design

• Translation/Back-translation Procedures
• Both versions should have the same meaning
• Translation and back translation
• Review by a group of native speakers
• Challenging when more than one language involved
  in the study

59
Quality Control Stages

• Pre-implementation/Developmental Stage
• Design high-quality questionnaires and forms
• Pretest all aspects of the project
• Develop an Operations Manual
• Design a secure data-management system
• Conduct staff training and certification
• Implementation
• Make periodic study- and data-monitoring reports
• Conduct periodic project meetings
• Complete interim staff training and performance
  reviews

60
Quality Control

• Pre-testing the Questionnaire Overview
• Refines
• Instruments and questionnaires
• Evaluates
• Recruitment methods
• Interventions
• Data entry and management system
• Protocols
• Assists with training of personnel

61
Quality Control

• Pre-testing the Questionnaire
• Good to test
• Sensitive questions
• Complex or poorly defined topics
• Questions with terms respondents may not
  understand
• Translations
• Layout and instructions

62
Quality Control

• Pre-testing the Questionnaire
• Field Testing Techniques
• Expert reviews
• Cognitive interviews
• Full pretest

63
Quality Control

• Pre-testing the Questionnaire
• Expert Reviews
• Formal and systematic examination of a
  questionnaire by experts in the field
• Can cover question wording, layout, and skip
  patterns
• Fast method prior to formal testing
• May not be a complete review

64
Quality Control

• Pre-testing the Questionnaire
• Cognitive Interview Overview
• Diagnostic tool for pretesting survey questions
  derived from social and cognitive psychology
• Interview about an interview
• Explores the processes by which respondents reach
  answers
• Based on structured questionnaire and protocol

65
Quality Control

• Pre-testing the Questionnaire
• Cognitive Interview Techniques
• Concurrent proving
• Ask probes immediately after respondent has given
  answer to survey item
• Advantage information is fresh in respondents
  mind
• Disadvantage Disrupts interview flow and
  relationship between questions
• Retrospective probing
• Ask probes after entire interview or block of
  questions
• Advantage able to assess standard administration
  of items
• Disadvantage Participant may not remember
  thought process

66
Quality Control

• Pre-testing the Questionnaire
• Cognitive Interview Techniques
• Probing questions
• Think aloud interviews
• Interviewer asks respondent to think aloud as
  they answer question
• Paraphrasing
• Respondents rephrase the question in their own
  words

67
Quality Control

• Pre-testing the Questionnaire
• Cognitive Interview Key Features
• Comprehension of the question
• Do participants understand words and phrases as
  intended by the researchers (meaning)
• How did you arrive at your estimate of risk?
• What did you understand by the word risk?
• Retrieve of information
• Identify process respondents use to answer
  questions
• Examine strategies used to access memory
• Edit responses Decide what to report
• Are items unacceptable? Is answer embarrassing,
  socially undesirable?

68
Quality Control

• Pre-Test
• Pretest with a sample similar to the population
  to be studied
• Simultaneously pretest recruitment procedures and
  questionnaire implementation

69
Quality Control

• Pre-Test Behavior Coding
• Code interaction between interviewer and
  respondent
• Examples
• R answers before hearing whole question
• R looks puzzled
• R asks for clarification
• I repeats question

70
Quality Control

• Pre-Test Respondent Debriefing
• Structured follow-up questions at the end of the
  interview to assess
• Whether questions were clear
• Ease of completing the questionnaires
• If two versions are tested, which is preferable
• Whether an important question was missed

71
Quality Control

• Pre-Test Interviewer Debriefing
• Interviewers implement pretest with participants
• Interviewers detect problems with questions,
  response categories, and skip patterns
• Interviewers report issues to investigators

72
Quality Control

• Pre-Test Questionnaire Summary
• Testing takes time and resources
• Some procedures take place late in the
  developmental cycle, limiting the amount of
  change possible
• Danger of making changes that are not further
  tested
• Not testing leads to questionnaires that are
  potentially unreliable or will give you invalid
  results

73
Quality Control Stages

• Pre-implementation/Developmental Stage
• Design high-quality questionnaires and forms
• Pretest all aspects of the project
• Develop an Operations Manual
• Design a secure data-management system
• Conduct staff training and certification
• Implementation
• Make periodic study- and data-monitoring reports
• Conduct periodic project meetings
• Complete interim staff training and performance
  reviews

74
Quality Control

• Operations Manual
• Aim is to ensure high quality data
• Clearly written, detailed written instructions to
  obtain uniformity across sites, staff, and
  procedures
• Standardizes procedures for all aspects research
  project. Defines how to
• Recruit participants
• Consent participants
• Measure variables
• Transport clinical samples
• Enter data
• Monitor progress
• Conduct data analysis
• Requires updates, clarifications, and recording
  of decisions made

75
Quality Control

• Data Management System
• Desirable features
• Range and field type checks
• Ease of screen set up and use
• Double data entry (if possible)
• Security features
• protection of human subjects rights (privacy)
• For Web-based systems
• Virus protection to monitor and eliminate
  security threats
• Database server behind firewall
• All data backed up regularly

76
Quality Control

• Staff Training
• Training and certification of the research team
  members
• Rules to conduct interviews
• Rules for recruiting participants, including how
  to obtain consent
• Methods
• Train the trainer model
• Audio-visual techniques
• Certification/recertification to maintain skills

77
Quality Control Stages

• Pre-implementation/Developmental Stage
• Design high-quality questionnaires and forms
• Pretest all aspects of the project
• Develop an Operations Manual
• Design a secure data-management system
• Conduct staff training and certification
• Implementation
• Make periodic study- and data-monitoring reports
• Conduct periodic project meetings
• Complete interim staff training and performance
  reviews

78
Quality Control

• Data Monitoring Reports
• Standard procedures for
• Identifying missing, incomplete data
• Identifying incorrect skip patterns
• Identifying inconsistent and erroneous data
• Verifying correction

79
Quality Control

• Interim Staff Training and Performance Reviews
• Provide continuous supervision
• Hold frequent meetings of the study team
• Review all aspects related to implementation of
  the study, including
• Interviewers activities
• Listen to interviews
• Compare measures between interviewers
• Compare refusal rates between interviewers

80
Good Clinical Practice

• Good Instruments
• Complete and clear protocol manuals
• Extensive pretesting
• Ongoing monitoring procedures