Identifying and Selecting Measures for Health Disparities Research

1
Identifying and Selecting Measures for Health
Disparities Research

• Anita L. Stewart, Ph.D.
• University of California, San Francisco
• Clinical Research with Diverse Communities
• EPI 222, Spring
• April 10, 2008

2
Selecting Measures for Your Own Study The Problem

• You are beginning a study
• You know the concepts (variables) of interest
• Question Which measure of ________ should I
  use?
• A popular measure
• One that a colleague used successfully
• Create your own

3
Inappropriate Measures can Result in

• Conceptual inadequacy
• Measuring wrong concept for your study
• Poor data quality (e.g. missing data)
• Poor variability
• Poor reliability and validity
• Inability to detect true associations
• e.g., no measured change in outcome when change
  occurred

4
Two Types of Considerations in Selecting Measures

• Contextual - factors unrelated to specific
  measurement tools
• Characteristics of target population
• Goals of research
• Practical constraints
• Psychometric - properties of measures within your
  context

5
Basic Steps in Selecting Appropriate Measures

• 1. Specify context
• 2. Define concept for your study
• 3. Locate potential measures for consideration
• 4. Review potential measures for
• a) conceptual match to your definition
• b) adequate psychometric properties in
  target group
• 5. Pretest potential measures in your target
  group
• 6. Choose best ones based on pretest results
  OR adapt if necessary to address problems

6
1. Specify Context

• Research question and how concept fits
  research (outcome, predictor, covariate)
• Nature of target population (health, age, SES,
  race/ethnicity, literacy)
• Practical constraints (time, personnel, budget,
  respondent burden)

7
Step 2 Define Concept For Your Study

• Define concept from your perspective, taking into
  account your
• study questions
• target population
• For outcomes, describe
• how intervention or independent variables might
  affect it
• specific types of changes you expect

8
Step 3. Locate Potential Measures

• Identify candidate measures for all concepts
• Redundancy OK for now
• Do NOT develop your own questions unless it is
  absolutely necessary

9
Locating Measures

• For major dependent or independent variables
• Multi-item measures with known psychometric
  properties
• For background variables and covariates
• Standardized survey measures

10
Locating Measures

• Compendia
• Organizations and research centers
• Government agencies
• National and state surveys
• Large research studies relevant to your topic
• Universities and individual researchers

11
Handout Locating Measures for Health Disparities
Research

• To link to websites, need to log on to CADC

http//medicine.ucsf.edu/cadc/cores/measurement/re
sourcescode.html
12
Locating Measures Compendia

• Specific measures of various concepts are
  compiled, reviewed, listed, or otherwise provided
• Books
• Many books review and critique various measures
• Web
• A few websites

13
Compendia by Web

• Health and Psychosocial Instruments (HaPI)
• Over 120,000 instruments
• http/gateway.ovid.com/
• Need ovid account UCSF has one if use computer
  at work (or vpn) no need to enter password

14
Compendia by Web

• National Cancer Institute website
• Health behavior constructs theory, measurement,
  and research
• Reviews concepts and measures of constructs such
  as perceived control, social support, and
  perceived vulnerability

http//dccps.cancer.gov/brp/constructs/index.html
15
Locating Measures Organizations and Research
Centers

• Some organizations and research centers
  specialize in measurement and provide public
  access websites
• RAND
• Michigan Diabetes Research and Training center

16
RAND Health Program

• Measures, scoring manuals, and citations
• Specialty
• Quality of care, patient satisfaction
• Health-related quality of life
• Generic and disease specific
• Medical Outcomes Study (MOS) measures
• www.rand.org/health/ (surveys and tools)

17
RAND Measures of Maternal, Child, and Adolescent
Health

• Pediatric asthma symptom scale
• Pediatric quality of life inventory

18
Michigan Diabetes Research and Training Center
Survey Instruments

• Diabetes Care Profile (DCP)
• Diabetes History (DMH)
• Diabetes Knowledge Test (DKT)
• Diabetes Attitude Scale (DAS-3)
• Diabetes Empowerment Scale (DES)
• Michigan Neuropathy Screening Instrument (MNSI)
  

www.med.umich.edu/mdrtc/survey/
19
Inter-University Consortium for Political and
Social Research

• Maintains archive of social science data
• Membership-based organization over 500 member
  colleges/universities
• UCSF is a member
• Can search website using keywords to locate
  studies, data, and questionnaires

www.icpsr.umich.edu/
20
Ottawa Health Decision Centre

• Patient and physician decision aids
• http//www.ohri.ca/centres/DecisionAids/default.as
  p
• Patient measures, e.g., decisional conflict,
  decisional regret, stage of decision making,
  decision self-efficacy
• http//decisionaid.ohri.ca/eval.html

21
Commonwealth Fund Surveys

• Health insurance
• Medicare
• Health care quality, patient centered care,
  underserved populations
• Child health development, care of the elderly
• State health policy, international health policy

• www.commonwealthfund.org/surveys/

22
Commonwealth Survey of Physicians

• 2006 International Health Policy Survey of
  Primary Care Doctors
• Use of information technology
• Access to care
• Availability of financial incentives
• Chronic care management

23
Commonwealth Fund Quality of Care Survey

• 2001 Health Care Quality Survey
• Racial and ethnic differences in patient
  experiences with health care
• focus on quality measures such as
  patientphysician communication and access to
  care 

24
Commonwealth Fund Surveys on Policy

• Modern Healthcare" Health Care Opinion Leaders
  Survey Assessing SCHIP
• State Childrens Health Insurance Program
• Health Care Opinion Leaders Survey Assessing
  Health Care Experts' Views on Medicaid and Its
  Future

25
Block Dietary Data Systems

• Now known as NutritionQuest
• Official source of Block Food Frequency
  Questionnaire
• Measures of nutrition, dietary intake, and
  physical activity.
• Questionnaire design and dietary analysis
  services to nutritional epidemiologists and
  public health researchers

www.nutritionquest.com/index.htm
26
Locating Measures Government Agencies

• Several federal and state government agencies
  provide measures for use in health and health
  disparities research
• Information about measures (e.g. clearinghouse)
• Actual measures

27
Agency for Healthcare Research and Quality (AHRQ)

• Medical Expenditure Panel Survey (MEPS)
• www.ahrq.gov/data/
• National Quality Measures Clearinghouse
• Consumer Assessment of Health Plans Survey
  (CAHPS)
• www.ahrq.gov/qual/

28
Department of Veterans Affairs

• National Survey of Veterans, 2001
• www.va.gov/vetdata/surveyresults/index.htm
• Survey of Veteran Enrollees Health and Reliance
  Upon VA, 2003
• www.va.gov/vetdata/healthcare/index.htm
• Both include measures of demographics and
  socioeconomic status, military background,
  health, health insurance, and VA and non-VA
  benefits usage.

29
National Cancer Institute

• Special initiatives on measures
• Health Information National Trends Survey (HINTS)
• Compiled cancer screening questions, identified
  best ones, conducted extensive pretesting

http//hints.cancer.gov/
30
National and State Surveys

• Population surveys
• Tend to have single-item measures rather than
  multi-item scales
• Good for standardized survey items
• Some exceptions
• e.g., National Health Interview Survey
  administered a 6-item scale of psychological
  distress (K6 scale) published by Kessler et al.
• Pratt LA, Advance Data, March 30, 2007 (No. 382)

31
Centers for Disease Control and Prevention

• Behavioral Risk Factor Surveillance System
• Surveys of health behaviors
• www.cdc.gov/brfss/questionnaires/index.htm

32
California Health Interview Survey (CHIS)

• A behavioral risk surveillance survey modeled
  after the CDC BRFSS
• Numerous languages
• www.chis.ucla.edu/
• Go to Methodology review questionnaires

33
CDC National School-Based Youth Risk Behavior
Survey (YRBS)

• Survey conducted every other year
• random national sample of youth in grades 9-12
• Most states conduct survey
• Measures of substance use, risky sexual
  behaviors, diet, physical activity, overweight
• www.cdc.gov/HealthyYouth/yrbs/index.htm

34
National Center for Health Statistics (NCHS)

• Surveys and data collection systems
• Can download
• Any survey or portion of survey
• See handouts for class 10
• Summary of all surveys
• Instructions for downloading

35
NCHS Sample of Population Surveys

• National Health Interview Survey
• National Health and Nutrition Examination Survey
• National Survey of Family Growth
• National Maternal and Infant Health Survey
• Longitudinal Studies of Aging (LSOA)

www.cdc.gov/nchs/express.htm
36
NCHS National Health Care Surveys Surveys of
Physicians

• Family of provider-based surveys
• Provide objective, reliable information about
• organizations and providers
• services rendered
• patients they serve

www.cdc.gov/nchs/nhcs.htm
37
Example National Ambulatory Medical Care Survey
(NAMCS)

• Samples MDs engaged in office-based patient care
• Samples visits of those MDs
• Measures of physician variables including
  practice characteristics

38
National Health Care Surveys Types of Measures

• Factors that influence use of health care
  resources
• Quality of health care, including safety
• Disparities in health care services provided to
  population subgroups in the U.S.

39
National Health Care Surveys Ambulatory and
Hospital Care

• Physician officesNational Ambulatory Medical
  Care Survey (NAMCS)
• Emergency and outpatient hospital departments
  National Hospital Ambulatory Hospital Medical
  Survey (NHAMCS)
• Ambulatory surgery facilities National Survey of
  Ambulatory Surgery (NSAS)
• Inpatient hospital departmentsNational Hospital
  Discharge Survey (NHDS)

40
Locating Measures Large Research Studies

• Many large-scale, multi-center and longitudinal
  studies have developed and used measures on
  health-related topics
• Increasingly, they are posting these on study
  websites

41
Study of Womens Health Across the Nation (SWAN)

• Physical, biological, psychological, and social
  changes of women during their middle years
• Questionnaire can be downloaded
• www.edc.gsph.pitt.edu/swan/public

42
Sacramento Area Latino Study on Aging (SALSA)
Study

• NIA funded longitudinal study of Latinos in the
  Sacramento region
• Started in 1996
• each person followed for up to 5 years
• http//sitemaker.umich.edu/salsa.study/home

43
Locating Measures Universities and Individual
Researchers

• Individual investigators who specialize in
  measurement and some universities have begun to
  post measures for public use
• Some of these include large research studies
  posted on university websites

44
University of Michigan Institute for Social
Science Research

• Two studies on antecedents and consequences of
  retirement (with NIA)
• Health and Retirement Study
• Asset and Health Dynamics Among the Oldest Old
  (AHEAD)
• Measures of health insurance, savings, and
  economic well-being including life course
  patterns of wealth accumulation and consumption

45
Dr. James Sallis, Ph.D., San Diego State
University

• Conducts research with children and parents,
  provides nearly 50 measures in English and
  Spanish
• 2003 Report Behavioral and Environmental
  Interventions to Promote Youth Physical Activity
  and Prevent Obesity  

www-rohan.sdsu.edu/faculty/sallis/measures.html
46
Duke University Center for Demographic Studies

• National Long Term Care Survey (with NIA)
• Prevalence and patterns of functional
  limitations, medical conditions, recent medical
  problems, use of health care, and housing and
  neighborhood characteristics
• www.nltcs.aas.duke.edu/index.htm

47
Tool Kit of Instruments to Measure End of Life
Care (TIME)

• Instruments to measure patient-focused, family
  centered care
• Annotated bibliographies of instruments that
  measure outcomes of care at the end of life
• www.chcr.brown.edu/pcoc/toolkit.htm

48
TIME Annotated Bibliographies Available

• Quality of life
• Pain and other symptoms
• Emotional and cognitive symptoms
• Advance care planning

• Spirituality
• Continuity of care
• Caregiver well-being
• Grief and bereavement
• and more

49
Roadmap K12 Data Resource Center

• Links to over 2 dozen national and state surveys
• Provides overview for each domains, time frame,
  population, scope, sample size, and contacts
• www.epibiostat.ucsf.edu/courses/RoadmapK12/PublicD
  ataSetResources/

50
Locating Measures Finding Authors of Measures

• Published research using measure you are
  interested in
• Unpublished measures often described in methods
• Authors may provide measures

51
Step 4 Review Potential Measures for

• Conceptual appropriateness relevance
• in your study
• in target group
• Psychometric adequacy in target group or groups
• Practicality
• Acceptability
• To respondents and interviewers

52
Conceptual Relevance

• Example you are interested in reports of
  perceived discrimination in the health care
  setting
• In reviewing measures of discrimination, most are
  about
• Discrimination over the lifecourse
• Discrimination in various life settings (work,
  school)
• Not relevant for your purpose

53
Concept Depicted as a Measurement Model

• Measurement model
• the dimensional structure of a measure
• how the items related to the construct
• Can be depicted as a list or visually

54
Measurement Models

• Physical Functioning (4 items)
• Psychological Distress (7 items)

55
Measurement Model (List format)

• Physical Functioning defined in terms of
• Walking
• Climbing stairs
• Bending
• Reaching

56
Measurement Model (Visual format)
Physical Functioning
Reaching
Climbing Stairs
Bending
Walking
57
Measurement Model (List format)

• Psychological distress
• Depression
• Sad
• Lost interest
• Cant get going
• Anxiety
• Restless
• Nervous

58
Measurement Model (Visual format)
Psychological Distress
Depression
Anxiety
Sad
Lost interest
Cant get going
Restless
Nervous
59
Psychometric Adequacy for Your Study

• In samples similar to your target group
• good variability
• low percent of missing data
• good reliability
• good validity
• As an outcome for planned intervention
• responsive, sensitive to change in similar
  population
• able to detect expected magnitude of change

60
Good Variability

• All (or nearly all) scale levels are represented
• Distribution approximates bell-shaped normal
• No floor or ceiling effects
• Scores bunched at either end

61
Reliability

• Extent to which an observed score is free of
  random error
• Population-specific reliability increases with
• sample size
• variability in scores (dispersion)
• a persons level on the scale

62
Reliability Coefficient

• Typically ranges from .00 - 1.00
• Higher scores indicate better reliability
• Types of reliability tests
• Internal-consistency
• Test-retest
• Inter-rater
• Intra-rater

63
Internal Consistency Reliability Cronbachs
Alpha

• Requires multiple items measuring same construct
• Extent to which items measure same construct
  (same latent variable)
• It is a function of
• Number of items
• Average correlation among items
• Variability in your sample

64
Minimum Standardsfor Internal Consistency
Reliability

• For group comparisons (e.g., regression,
  correlational analyses)
• .70 or above is minimum
• .80 is optimal
• above .90 is unnecessary
• For individual assessment (e.g., treatment
  decisions)
• .90 or above (.95) is preferred

JC Nunnally, Psychometric Theory 3rd ed,
McGraw-Hill, 1994
65
Validity

• Does a measure (or instrument) measure what it is
  supposed to measure?
• AndDoes a measure NOT measure what it is NOT
  supposed to measure?

66
Validation of Measures is an Iterative, Lengthy
Process

• Validity is not a property of the measure
• validity is a property of a measure for
  particular purpose and sample
• validation studies for one purpose and sample may
  not serve another purpose or sample
• Accumulation of evidence
• Different samples
• Longitudinal designs

67
Three Major Forms of Measurement Validity

• Content
• Criterion
• Construct

68
Construct Validity Basics

• A process of answering the following questions
• What is the hypothesis?
• What are the results?
• Do the results support (confirm) the hypothesis?

69
Construct Validity NOTE

• Sometimes the hypothesis is that the measure will
  NOT be correlated with certain other measures, or
  will be less correlated with some than with
  others
• THUS, observing a low or non-significant
  correlation can confirm construct validity

70
Limited Data on Psychometric Properties of Many
Measures

• Not easy to find this information
• Many studies do not report any psychometric
  properties
• Assume the properties from original study carry
  over

71
Review Measures for Practicality

• Method of administration appropriate for your
  study
• Costs of administration within study resources
• Scoring rules clearly documented
• Measure available at cost you can afford
• You are allowed to adapt it if necessary
• Translations available if needed

72
Practical - Scoring

• Know ahead of time how to score items
• Count of correct answers? Summated scale?
  Weighted?
• Are scoring instructions or computer scoring
  programs available?
• Can scoring programs be purchased from
  developers?
• Do you have a scoring codebook?

73
Review Measures for Availability of Translations
if Needed

• If you need the questionnaire in another
  language, are there translations available?
• Official (published and tested)
• Unofficial (by some other researcher)
• If not, you have to conduct translations
• Use state-of-the-art methods

74
Review Measures for Acceptability

• Acceptability to target population
• respondent burden (length, time needed, distress)
• culturally sensitive
• Acceptability to interviewers
• interviewer burden
• amount of training needed

75
Respondent Burden

• Perceived burden
• a function of item difficulty, distress due to
  content, perceived value of survey, expectations
  of length
• as important as time burden

76
5. Pretest Potential Measures in Your Target
Population

• Select best measures for all concepts in your
  conceptual framework
• existing instrument in its entirety
• subscales of relevant domains (e.g., only those
  that meet your needs)

77
Pretest

• Pretesting essential for priority measures (e.g.,
  outcomes)
• Pretest is to identify
• problems with method of administration
• unacceptable respondent burden
• problems with questions or response choices
• words and phrases that do not mean what you
  intended to target population

78
Summary

• Methods for selecting/reviewing measures
  described here are ideal
• Apply these methods to your most important
  measures
• e.g., outcomes, key independent variables
• Keep learning
• Good, appropriate measures remain the foundation
  of excellent research

79
Homework

• See handout for class 2 homework
• Complete rows 1-13 in matrix
• Use form posted on the website
• Email responses by Monday AM to
  anita.stewart_at_ucsf.edu