Validation in Statistics Canada Health Surveys

1
Validation in Statistics Canada Health Surveys

• Presentation to RRFSS Workshop
• June 20, 2007
• Vincent Dale

2
Outline

• Statistics Canada quality assurance framework
• Ensuring data accuracy
• Past validation projects
• Future projects
• Future directions

3
Quality Assurance Framework
4
Statistics CanadaQuality Assurance Framework

• Trade-offs between aspects of quality
• These are actively managed through a variety of
  processes, including
• User and stakeholder feedback mechanisms
• Program review
• Data analysis and dissemination
• Standards and documentation (concepts, variables,
  classifications)

5
Ensuring Data Accuracy

• Questionnaire development
• Wherever possible, validated questionnaire
  modules are used
• Sometimes modified for use in population-based
  survey
• Sometimes not as valid as advertised
• Questionnaire testing
• STC policy requires testing of all new
  questionnaires
• Cognitive interviews and focus groups
• Coherence versus accuracy
• Sometimes better to keep measure stable even if
  imperfect

6
Ensuring Data Accuracy

• Sampling error
• error attributed to studying a fraction of a
  population rather than carrying out a census
• Non-sampling error
• coverage errors
• response errors
• non-response errors
• processing errors
• estimation errors
• analysis errors

7
Ensuring Data Accuracy

• Explosion of health survey data
• More data, more often for smaller levels of
  geography
• Increasing attention paid to validity
• Health measures
• Administrative data
• Complimentary surveys

8
What is validity?

• Face validity
• Internal validity
• construct validity
• External validity
• Criterion
• Sensitivity, specificity, predictive value

9
Past CCHS Validation Projects
10
Health Care Utilisation

• Data linkage of CCHS responses with BC
  administrative health records
• Supplemented with analysis of
• Respondent interpretation and formulation of
  responses
• Interviewer behaviour and training
• Patterns in response changes, edits and timing of
  response entry

11
Contacts with Health Professionals

• Results of linkage
• Compared to provincial health records
• Most CCHS respondents (58) reported fewer
  primary care physician contacts
• On average, CCHS respondents reported 1.7 fewer
  primary care physician contacts
• Older CCHS respondents and respondents with
  better self-perceived health tended to report
  fewer contacts
• Younger respondents and respondents with poorer
  self-perceived health tended to report more
  contacts

12
Contacts with Health Professionals

• Recommendations from study
• Revise wording of specific questions to minimize
  misinterpretation
• Facilitate consistent interviewer probing
  techniques
• Improved edits and CAPI/CATI application
  navigation for interviewers to facilitate changes
  to previously-answered questions

13
Evaluation of coverage of linked CCHS and
hospital inpatient records

• Probabilistic linkage used to identify CCHS 1.1
  respondents (excluding Québec) hospitalized over
  a 14-month period
• Health person-oriented information database
  (HPOI) is a virtual census of hospital admissions
  and used as the standard
• Survey weights applied to the 8230 CCHS records
  which were found in the HPOI database

14
Evaluation of coverage of linked CCHS and
hospital inpatient records
15
Evaluation of coverage of linked CCHS and
hospital inpatient records

• Under-reporting rates similar between women and
  men
• Lower among Manitoba residents (69.2)
• Higher among individuals aged 12-74 (86.1) than
  those aged 75 (70.3)
• Under-reporting is an essential prerequisite to
  further analyses based on the CCHS HPOI linked
  data
• Use of the linked file could lead to bias due
  depending on province/territory of residence and
  age

16
CCHS Measured Height Weight

• In 2005, height / weight were measured for a sub
  sample of CCHS Cycle 3.1 participants (n4567)
• Weight mean difference between measured and
  self-reported weight of 2.1 kg (2.5 kg for women)
• Height mean difference between measured and
  self-reported height of -0.7 cm (-1.0 cm for men)
• BMI mean difference between measured and
  self-reported BMI was 1.1

17
CCHS Measured Height Weight
18
CCHS Mode Effect Study

• Potential differences associated with two methods
  of collection used in CCHS
• CAPI computer assisted personal interview
• CATI computer assisted telephone interview
• Used a split-panel design with a unique sample
  frame
• secondary sampling units randomly assigned to
  CAPI or CATI.
• Fully integrated as part of CCHS cycle 2.1
• 11 sites selected to provide a good
  representation of each region in Canada

19
CCHS Mode Effect Study

• Important differences observed for obesity rates
• CAPI 17.9 CATI 13.2
• Physical activity index inactive persons
• CAPI 42.3 CATI 34.4
• Statistically significant differences for contact
  with medical doctors and unmet health care needs
• No significant differences observed in the vast
  majority of health indicators

20
CCHS Mode Effect Study

• Overall results show that cycles 1.1 and 2.1 are
  largely comparable despite an increase in CATI
  collection for Cycle 2.1 (2003)
• Results led to a decision to measure exact height
  and weight for a sub-sample of respondents in
  cycle 3.1 (2005)
• Led to improved standardization of interviewer
  procedures across the two collection modes

21
Future Validation Projects
22
Scale Reliability - Factor Analysis

• Construct validity / scale reliability
• Cronbachs Alpha calculated for scales used in
  CCHS questionnaire
• Results could be published in user guide
• What are standards?
• Some researchers feel that scores should be above
  0.8

23
CCHS Depression Module

• Currently, CIDI Short form for Major Depression
  (CIDI-SF) is used in CCHS
• Also used in NPHS and several regional and
  provincial surveys
• Some problems with its use in CCHS
• Has not been validated against International
  Classification of Disease (ICD)
• Evaluates 12-month prevalence, not necessarily
  current treatment need
• Does not evaluate some items related to clinical
  significance
• Patient Health Questionnaire (PHQ) identified as
  potential CIDI-SF replacement

24
CCHS Depression Module

• Primary goals of potential validation study
• Determine the validity of the CIDI-SF and PHQ in
  relation to a gold standard diagnostic interview
  (SCAN Schedules for Clinical Assessment in
  Neuropsychiatry)
• Identify optimal scoring procedures for the PHQ
  in Canadian population-based studies

25
CCHS Depression Module

• Samples of n200 subjects to be drawn in two
  sites (English and French)
• Supplemented with n100 subjects selected from
  psychiatric outpatient settings to increase the
  number of positive cases of major depression
• Each participant to be administered
• 1) Standard demographic module
• 2) PHQ-9
• 3) Module to distinguish between clinical
  depression and bereavement
• 4) SIDI-SF
• 5) Set of modules to assess consequences of
  construct in terms of quality of life

26
CCHS Depression Module

• Sensitivity and specificity of the CIDI-SF and
  PHQ to be measured using the SCAN as a gold
  standard
• Ordinal CIDI-SF ratings to be correlated with PHQ
  ordinal ratings using Spearman correlation
  coefficient
• Test of construct validity of PHQ to be performed
  using exploratory factor analysis
• Internal consistency of scales and subscales to
  be assessed using Cronbachs alpha
• Test-retest reliability of PHQ and CIDI-SF and
  inter-rater reliability of the SCAN will be
  evaluated for 50 respondents

27
CCHS Depression Module

• The estimated cost for the project exceeded
  200,000
• Due to our inability to secure external funding
  and the lack of available budget and personnel
  internally, there are no concrete plans to
  proceed with study

28
Directions Forward

• Focus on accuracy, interpretability and coherence
• Trade-offs between aspects of data quality
• Improved timeliness, accessibility and relevance
• How good is good enough?
• Partnerships
• Are there areas where CCHS, RRFSS and others can
  collaborate ?

29
Contact Information

• Vincent Dale
• Survey Manager, Canadian Community Health Survey
• 613-951-4265
• Sylvain Tremblay
• Content Manager, Canadian Community Health Survey
• 613-951-2528