Accuracy: validity and precision

1
Accuracy validity and precision

• A major objective of epidemiologic studies
• to estimate measures of association as accurately
  as possible
• A major objective of methods used to design
  studies, collect data, analyze data, and
  interpret results
• to avoid error to the extent possible

2
Error

• Definition
• a discrepancy between a measured value and the
  true value
• Components of error
• Random non-systematic
• chance variation
• sampling error
• Systematic bias
• the extent to which the measurement procedure
  tends to underestimate or overestimate the true
  value on repeated application

3
Error

• All of the following should be considered as as
  alternate explanation for an observed association
• Random error (chance)
• systematic error (bias)
• Confounding
• Causal inference in epidemiology, depends on a
  clear understanding of what can go wrong
• Inferential statistics estimation of the
  probability of error due to random variation
  (quantification of the probability that the
  observed results could have occurred by chance
  alone)

4
Error

• Random error (chance)
• the difference between the estimate (result of
  the study) and the parameter actually being
  estimated
• inherent in all observations
• arises from measurement itself or from biologic
  phenomenon being measured
• called random since results on average just as
  likely to lie on one side of true value as on the
  other
• in the absence of bias, random variation is
  responsible for the uncertainty of the conclusion

5
Accuracy

• Accuracy - of measures of association or effect
• Definition
• the degree of absence of error
• Components of accuracy
• Precision
• the degree of absence of random error
• the degree to which the measure is reproducible
  on repeated sampling
• a precise measure is statistically stable
• Validity
• the degree of absence of systematic error
• the degree to which the measure reflects the true
  value
• a valid measure is unbiased

6
Accuracy
7
Evaluating validity and precision

• Precision
• evaluated by statistical measures of error
• standard error
• confidence interval

8
Evaluating validity and precision

• Validity
• Study methods scrutinized for potential sources
  of systematic error
• likely direction and magnitude of biases should
  be assessed
• external information can be used to quantify
  probable degrees of bias
• in quantifying effects of bias we can explore
  what is likely to have occurred in evaluating
  specific studies we are generally uncertain about
  the actual bias that has occurred

9
Improving precision

• Increase sample size
• Increase statistical efficiency
• greater precision for the same sample size
• determined by the apportionment of subjects into
  comparison groups
• optimal apportionment must be anticipated in
  design of study as it depends on
• the true magnitude of the exposure effect
• disease frequency and exposure prevalence
• number of variables used for stratification
• relative cost per unit of information in
  comparison groups

10
Improving validity

• Design studies to minimize selection bias,
  information bias and confounding
• Obtain information on validity of measurement
  instruments
• Obtain information on the distribution of study
  factors in target population
• Employ appropriate analytic methods to estimate
  effects
• Use analytic methods to control for systematic
  errors in data
• Use sensitivity analyses to quantify probable
  degrees of systematic error

11
Internal validity

• Synonym study validity
• The extent to which study results are free of
  bias
• Valid measures of association reflect the true
  value in the target (the population the study
  sample was intended to represent)

12
External validity

• Synonym generalizability
• The extent to which study results apply to
  populations other than the source population from
  which the study sample was drawn
• lack of generalizability is a reflection of
  reality, not a type of error
• non-generalizability reflects effect-measure
  modification
• the effect of interest varies by the distribution
  of population characteristics such as age, sex,
  genetic traits, behaviors, or environmental
  exposures that influence disease frequency

13
Generalizability versus representativeness

• Scientific generalization is more than
  statistical generalization from a sample to the
  source population from which the sample was drawn
• Valid statistical generalization from a sample to
  a source population requires that the sample
  represent the source with respect to study
  factors
• If scientific generalization required
  representative study groups, study findings would
  only apply to those who could have been included
  in the study through sampling
• Subject selection for epidemiologic studies that
  estimate measures of association should
  prioritize the need for valid comparison groups
  over attempts to make the study groups
  representative, in a random sampling sense, of
  source or target populations
• Valid comparisons may require restriction of
  subjects to a narrow range of characteristics

14
Generalizability versus representativeness

• Scientific generalization involves applying what
  is learned from particular observations to a
  broader domain using subject matter knowledge
  regarding aspects of hypothesized relationships
  that may or may not hold across distinct
  populations
• Study groups should be selected for
  characteristics that enable them to distinguish
  effectively between competing scientific
  hypotheses of interest

15
Bias

• Definition
• any systematic error that distorts the truth and
  results in an incorrect estimate of association
  (non-random error)
• always a potential regardless of how rigorously
  designed the investigation is
• Types of bias in estimating causal effects in
  epidemiologic studies
• Selection bias
• Information bias
• Confounding

16
Selection bias

• Occurs when the relations for the sampling
  validity of the causal contrast study design do
  not hold
• Results from procedures used to select subjects
  and from factors that influence study
  participation such that
• the relation between exposure and disease is
  different for those who participate in the study
  and all those who should be eligible for study,
  including non-participants

17
Selection bias

• Does not result from differential selection by
  exposure status alone or by outcome status alone
• Occurs if
• cohort study
• disease status influences selection or
  participation to a different degree in exposed
  and unexposed comparison groups
• case-control study
• exposure status influences selection or
  participation to a different degree among cases
  and controls
• cross-sectional study
• either variable influences selection or
  participation to a different degree in comparison
  groups

18
Information bias

• Information bias
• errors in obtaining information (asymmetric
  collection or measurement errors)
• a study design issue
• recall bias
• difference in the manner in which exposure
  information is reported must be considered in
  both design and interpretation
• interviewer (observer, investigator) bias
• difference in the soliciting, recording, or
  interpreting of information from participants

19
Information bias

• Bias in estimating measures of effect can result
  from either random or systematic error in the
  measurement of either exposure status or disease
  status
• Measurement error in categorical variables
  results in misclassification of observations

20
Information bias

• Misclassification
• Misclassification of exposure status
• Nondifferential the probability of exposure
  misclassification is the same for cases and
  noncases
• Differential the probability of exposure
  misclassification differs for cases and noncases
• Misclassification of disease status
• Nondifferential the probability of disease
  misclassification is the same for exposed and
  unexposed subjects
• Differential the probability of disease
  misclassification differs for exposed and
  unexposed subjects

21
Bias

• Control of bias
• primarily an issue in the design of a study
• crucial to the validity of the study results
• features that help to minimize bias
• choice of study population
• methods of data collection
• sources of information

22
Confounding

• A confusion of effects that occurs when the
  effect of an extraneous factor is mistaken for or
  mixed with the actual exposure effect
• Occurs when disease risk factors are unevenly
  distributed across exposure groups, making these
  groups incomparable with respect to factors that
  influence disease frequency

23
Confounding

• A confounder can
• create the appearance of an association when the
  true association is null
• create the appearance of a null association when
  there is a true association
• bias the measure of a true effect toward or away
  from the null value
• reverse the direction of a true association

24
Confounding

• A confounder, considered in isolation, must
  fulfill 3 conditions
• a risk factor for disease in the unexposed
• associated with exposure in the population from
  which cases arose
• not an intermediate step in the causal pathway
  between exposure and disease
• Valid identification of confounders is done in
  the multivariate context, considering all
  potential confounders, because confounding
  influences can be altered (heightened, lessened
  or canceled) by the presence of other confounders

25
Confounding

• Control of confounding
• in study design
• randomization - only achieved in intervention
  studies unique strength is the ability to
  control confounding
• restriction - establish inclusion/exclusion
  criteria to limit selection of individuals that
  fall in a specific category of the confounder
• matching - primarily used in case-control
  studies removes the effect by making groups
  equivalent in terms of a particular confounding
  variable

26
Confounding

• Control of confounding
• in analysis
• stratified analysis - determining an estimate of
  the association within homogeneous categories
  (e.g., sex, age groups)
• multivariate analysis - allows for efficient
  estimation of measures of association while
  controlling for a number of potential confounding
  variables simultaneously normally done with
  regression modeling

27
Hypothesis testing

• conducting a statistical test (e.g., chi-square,
  t-test) to quantify significance of difference
• requires making an explicit statement about the
  difference or relationship between the two groups
  regarding exposure
• Ho (no difference) Ha (difference exists)
• ? ? ? ? ?
• pe po pe ? p0
• RR 1 RR ? 1

28
Hypothesis testing

• Point estimate the exact size of the effect as
  determined by the study (e.g., the study result)
  can be a sample mean, median, measure of
  association (RR or OR)

29
Hypothesis testing

• Interval estimate (confidence interval) A
  measure of the precision (stability) of an
  observed effect
• the range within which the true magnitude of
  effect lies with a particular degree of certainty
• 95 C.I. means that value of the effect (mean,
  risk, rate) lies within 2 standard errors of the
  population mean 95 time out of 100
• width indicates the amount of variability the
  more narrow, the less variability
• If C.I. does not include value indicating no
  effect (e.g., mean difference 0 RR or OR 1),
  then results are statistically significant

30
Hypothesis testing

• p-value the probability that an effect at least
  as extreme as that observed could have occurred
  by chance alone, given there is truly no
  relationship between exposure and disease
• convention ? 0.05 (1 in 20) level
• if p lt 0.05 , statistically significant, not
  likely due to chance
• if p gt 0.05, not significant, chance cannot be
  excluded

31
Hypothesis testing

• p-value one-tailed vs two-tailed
• normally, test for possibility of a difference in
  both directions (two-tailed test) allows for
  uncertainty about the direction of an effect
• a priori evidence usually dictates which is
  appropriate

32
Hypothesis testing

• H0 Pet owners who are asthmatic are no more
  likely to have asthma attacks than non pet owners.

RR (risk ratio) 33/78 ? 30/75 1.06 ?2 0.08
p 0.77 95 C.I. 0.72 ltRR lt1.55
H0 not rejected pet ownership not associated
with asthma attacks
33
Hypothesis testing

• H0 Pet owners who are asthmatic are no more
  likely to have asthma attacks than non pet owners.

RR (risk ratio) 200/210 ? 300/450 1.43 ?2
63.64 p lt0.0001 95 C.I. 1.33 ltRR lt1.54
H0 rejected pet ownership associated with asthma
attacks
34
Point estimates and confidence intervals
? Risk
Protection ?
35
Hypothesis testing

• In determining statistical significance, the
  precision of the estimate should be based on both
  p-value and confidence interval
• p-value is susceptible to variability and sample
  size large sample can detect a statistically
  significant difference that may not be important
  and vice versa
• confidence interval--width depends on variability
  in data location of the no effect value (in or
  out of interval) and width both informative

36
Hypothesis testing

• Statistical significance vs clinical importance
• although the p-value and confidence interval may
  lead to the conclusion that chance is an unlikely
  explanation for the findings, they provide no
  information regarding the effects of uncontrolled
  bias or confounding
• all factors must be weighed in light of clinical
  importance

37
Type I and Type II error
Type I error false positive Type II error
false negative