Precision and Validity

1
Precision and Validity

• Kim Angelon-Gaetz, MSPH
• Office of Healthy Carolinians/
• Health Education
• Kim.angelon_at_ncmail.net

2
Precision vs. Validity

• Precision how much variability
• Validity how close to the true value

Precise but not valid
Valid but imprecise
3
2 Types of Validity

• Internal Validity
• What is the quality of the information that you
  gathered about the sample?
• External Validity
• Do your sample data pertain to the general
  population?
• Can you generalize your findings?

4
Speaking of Validity

• You can never know what the true association
  between exposure and disease is, so you cant
  measure validity.
• Plan your survey to minimize bias.
• Report any leftover problems that might have
  biased your survey.

5
The Balancing Act

• Reducing

• Increases

• Random error
• By increasing sample size
• By making your study more efficient
• Bias/ Systematic error
• By improving measurement, selection and
  participation

• Precision
• Validity

6
Measuring Statistical Significance
7
Statistical vs. Clinical Significance

• Statistical significance the finding is
  important based on a statistical test
• How likely is it that the observed effect
  happened by chance?
• Clinical significance the finding is important
  based on its clinical or practical implications

8
Statistical significance cont.

• Usually we test whether the p-value is below some
  threshold, called alpha (a)
• Ex If a 0.05
• P-value lt 0.05 statistical
  significance
• Alpha is
• A fixed value that we decide on BEFORE looking
  at our data!
• How much type I error we think is acceptable

9
Type 1 vs. Type 2 Error

• Type 1 Error Saying that a factor has an
  association with the outcome when it really
  doesnt
• Type 2 Error Saying that a factor doesnt have
  an association with the outcome when it really
  does

10
Hypothesis Testing

• Using data to test a hypothesis about what is
  associated with an outcome
• Null hypothesis no association between exposure
  and outcome
• 1.0 for ratios or 0 for absolute measures
• The exposure is Innocent until proven guilty
• This is what we are testing!
• Alternative hypothesis
• Opposite of the null hypothesis
• Any/ Positive/ Negative Association

11
P-value

• The probability of finding an association as
  strong or stronger than the observed association,
  if the null hypothesis is true.
• Compared to a to test our hypothesis
• Usually a0.05, meaning that a valid test should
  (incorrectly) reject a true null hypothesis, no
  more than 5 of the time.

12
Interpreting the P-value

• A low p-value (lt 0.05) means that the effect that
  you found is statistically significant
• BUT
• never interpret your findings based
• on the p-value alone!

13
Confidence Intervals (CI)

• Tell you something about both the precision of an
  estimate and strength of association.
• 1-a confidence level (usually 95)
• If you could repeat your study over and over
  again using a correct model, valid test, and no
  bias, the confidence intervals should contain the
  true value 95 of the time.

14
Precision and Confidence Intervals

• Narrow confidence intervals precise!
• Confidence limit ratio (CLR) upper limit/ lower
  limit
• 1 is narrow
• Ex Odds Ratio 1.7 Confidence interval (0.9,
  3.2)
• CLR 3.2/ 0.9 3.56
• Interpretation
• This confidence interval is pretty narrow so it
  is fairly precise!
• It includes the null (1.0) so we fail to reject
  the null hypothesis.
• The association between exposure and outcome is
  not statistically significant.

15
95 Confidence Interval
Point Estimate
0 0.9 1.0 1.7
3.2 5
Null
Upper Limit (UL)
Lower Limit (LL)
16
How are p-values and confidence intervals related?

• A rule of thumb
• When a confidence interval includes the null
  value (1 or 0), the p-value will be a
• When a confidence interval does not include the
  null, the p-value will be lt a

17
Example Hypothesis Test

• In a recent outbreak, peanut butter was found to
  be associated with salmonellosis.
• Odds ratio and 95 confidence interval
• 2.53 (1.26, 5.31), p-value 0.007
• Interpret The odds of getting ill after eating
  peanut butter were 2.53 (1.26, 5.31) times the
  odds of getting ill without eating peanut butter.
  
• CI did not include the null and was fairly
  precise (CLR5.31/1.26 4.2)
• Result is statistically significant

18
Example 2 Large Sample Size

• Remember that the p-value doesnt tell you how
  large the observed effect was.
• If you have a really large sample size, you might
  get a very small p-value even though the effect
  is not very large
• Risk Ratio1.80 (1.322.46) p-valuelt0.0001
• CLR 2.46/1.321.86
• The measurement is precise.
• The association is statistically significant but
  not very large.

19
Information from p-values and confidence intervals
20
15 Variance Rule

• SCHS guideline for comparing health indicators
  Pay attention to 15 difference
• Compare your county to
• Peer or neighboring counties
• State
• Itself! (In years before)
• Look at improvement and problems

21
What DONT Statistical Measures Tell Us?

• Clinical relevance
• Validity
• Biologic plausibility
• Costs
• Benefits
• Feasibility

22
Making Informed Public Health Goals
23
References

• Rothman, KJ. (2002) Epidemiology An
  Introduction. Oxford University Press.
• Rothman, KJ and S Greenland. (1998) Modern
  Epidemiology. 2nd Edition. Lippincott- Raven
  Publishers.