Total Survey Error in Disability Assessments

1
Total Survey Error in Disability Assessments

• Measuring Physical and Cognitive Capacity in the
  National Health and Aging Trends Study (NHATS)
• Brad Edwards and Tamara Bruce, Westat
• Presented at the International Total Survey Error
  Workshop Quebec, Canada
• June 2011

2
Overview

• NHATS and capacity measures
• Self report and performance
• NHATS assessments
• Strategies for error reduction
• design, training, standardization, parsing out
  nonresponse
• Pretest results
• National experience
• Future research

3
NHATS

• Westat working with a team led by Johns Hopkins
  Bloomberg School of Public Health
• Funded through cooperative agreement from NIA
• CAPI panel study with 9,000 Medicare
  beneficiaries, annual interviews starting 2011
• New and improved measures of disability for
  producing data on trends and trajectories
  self-reports, mental and physical assessments,
  eventually biomarkers, links with admin data

4
Conceptual Framework

• Blend of Nagis model and WHO perspective
• Distinguish explicitly between
• Capacity to do something basic building blocks
• What people actually do in actual environment
  activities
• Accommodations may fill gap

5
Capacity Is Critical

• Measures of capacity over time are key elements
  in understanding individual patterns of
  progression to activity limitations. They allow
  us to track trends in function that are
  independent of environmental changes or
  accommodations, for understanding the disablement
  process, and as targets for interventions to
  prevent or slow disability.

6
Reducing Error in Capacity Measures

• Questionnaire design
• Importance of performance measures
• Challenges for quality management
• New to surveys
• Complex
• Expect high level of interviewer variance

7
Physical Capacity Upper Extremity

• Self report

• Performance

• Able to
• Put book on shelf/reach overhead
• Open jar/grasp small object

• Grip strength

8
Physical Capacity Lower Extremity

• Self Report

• Performance

• Able to
• Walk 6/3 blocks?
• Kneel/bend over?
• Lift and carry 20/10 pounds?
• Walk up 20/10 stairs?

• Walking speed
• Balance stands
• Side by side
• Semi-tandem
• Full-tandem
• One leg eyes open
• One leg eyes closed
• Chair stands

9
Cognitive Capacity Memory

• Self report

• Performance

• At present time?
• Memory problems interfere with activities?
• Memory compared to one year ago?

• 10 word recall
• Immediate
• Delayed

10
Other Cognitive Performance Measures

• Orientation
• Day of week, date, naming President and Vice
  President
• Overall cognitive screening/executive function
• Clock drawing
• Attention interference/executive function
• Stroop test (computerized)

11
Standardizing Administration

• Performance measures of capacity can be hard to
  standardize in large-scale surveys with lay
  interviewers
• NHATS uses multiple methods to improve
  consistency accuracy
• Activities Booklet design use within CAPI
  framework
• Standardized presentation of cognitive tasks
  using flash displays
• Survey design interviewer feedback using CARI
  coding
• Formal certification process using live
  respondents
• Follow-up web-based recertification

12
Training Approaches

• Broader scope of demands on NHATS interviewers
  for performance measures
• tests require using a variety of equipment
• navigation of unfamiliar environments
• potentially uncomfortable instructions to
  respondents
• Integration of video components
• recruitment
• in-person interviewer training modules such as
  walking course
• certification re-certification processes

13
Walking Course Timing 1
14
Walking Course Timing 2
15
Walking Course Timing 3
16
Training Video Demo
17
Item Nonresponse

• Performance measures used in depth set of
  exclusion criteria to minimize burden
• High completion rate for both cognitive
  performance tasks for those eligible
• Comprehensive list of reasons why test was not
  conducted allow detailed analyses
• Majority of unit nonresponse due to inability to
  complete easier task safety concerns
• Performance related to age, health rating, care
  setting, memory

18
Respondent Reactions

• Avoiding test and performance
• Training interviewers how to respond to requests
  (e.g., How did I do?)
• Physical activities 7th inning stretch
• Positive feedback at the end
• Interest in assessing respondent satisfaction and
  rapport in relation to nonresponse in later rounds

19
Comparison of Performance and Self Report

• High functioning performance group (attempting
  one-leg balance stands, meeting accuracy
  thresholds for Stroop) reported better
  self-reported health and memory and (for the
  Stroop) had higher performance on two memory tests

20
Comparison with Other Surveys

• NLTCS screening questions in NHATS form a bridge
  from previous trend data to detailed NHATS self
  reports and assessments
• Possibility of future bridge to 6 disability
  questions developed by NCHS and used on the
  American Community Survey

21
Conclusions

• Reducing error in capacity measures based on
  performance assessment
• sharpens understanding of individual trajectories
  and accommodations
• is expected to improve predictive ability for
  health outcomes
• NHATS protocol
• standardizes administration
• focuses on reasons for nonresponse

22
Conclusions (2)

• As a new survey, NHATS has been able to give
  consideration to the implications of doing these
  types of assessments from the outset, starting
  with interviewer recruitment and training.
  Experience from pretesting has led to refinements
  of these procedures (e.g., use of videos in
  recruitment certification procedures in
  training) and to the NHATS Activities Booklet.
  Training materials and data collection
  instruments will be available later this year at
  www.nhats.org.

23
Future Research

• Interviewer variance study
• Examination of
• interviewer and respondent conditioning effects
• drift trajectories
• item nonresponse as predictor of unit nonresponse
• Error comparison/tradeoffs between self-reports
  and performance
• Analysis of cognitive performance and data
  quality
• More development
• distance learning
• SPC charts, integration with paradata for
  management
• re-certification via WebEX, 2-way video

24
For more info, contact Brad Edwards
bradedwards_at_westat.com or Tamara Bruce
tamarabruce_at_westat.com