Section 1.4 ~ Should You Believe a Statistical Study?

1
Section 1.4 Should You Believe a Statistical
Study?

• Introduction to Probability and Statistics
• Ms. Young

2
Objective
Sec. 1.4

• In this section we will discuss how you can
  evaluate statistical studies to determine if the
  results are meaningful.

3
Eight Guidelines for Critically Evaluating a
Statistical Study
Sec. 1.4

1. Identify the goal of the study, the population
   considered, and the type of study.
2. Consider the source, particularly with regard to
   whether the researchers may be biased.
3. Examine the sampling method to decide whether
   its likely to produce a representative sample.
4. Look for problems in defining or measuring the
   variables of interest.
5. Watch out for confounding variables that can
   invalidate the conclusions of a study.
6. Consider the setting and wording in survey polls,
   looking for anything that might tend to produce
   inaccurate or dishonest responses.
7. Check that results are fairly represented in
   graphics and concluding statements, because
   researchers and media often create misleading
   graphics or jump to conclusions that the results
   do not support.
8. Stand back and consider the conclusions. Did the
   study achieve its goals? Do the conclusions make
   sense? Do the results have any practical
   significance?

4
Guideline 1 Identify the Goal, Population, and
Type of Study
Sec. 1.4

• In evaluating a statistical study, you must
  understand the goal and the approach of the study
• To do this, try to answer these basic questions
• What was the study designed to determine?
• What was the population under study? Was the
  population clearly and appropriately defined?
• Was the study an observational study, an
  experiment, or a meta-analysis?
• If observational, was it retrospective or
  prospective?
• If it was an experiment, was it single- or
  double-blind, and were the treatment and control
  groups properly randomized?
• Given the goal, was the type of study appropriate?

5
Example 1
Sec. 1.4

• A study sought to determine whether aspirin is
  effective in preventing heart attacks. It
  involved 22,000 male physicians considered to be
  at risk for heart attacks. The men were divided
  into a treatment group that took aspirin and a
  control group that did not. The results were so
  convincing in favor of the benefits of aspirin
  that the experiment was stopped for ethical
  reasons before it was completed and the subjects
  were informed of the results. Many news reports
  led with the headline that taking aspirin can
  help prevent heart attacks. Analyze the headline
  according to Guideline 1.
• The study was designed to determine whether
  aspirin is effective in preventing heart attacks.
• The headline did not specify the population, but
  it gives the impression that they are talking
  about adults in general. However, the study only
  consisted of men which means that the headline is
  misleading since they cannot assume that aspirin
  has the same effect on women.
• The study was an experiment which is appropriate,
  but with the information given, we are unsure if
  it was double-blind or single-blind. It did seem
  to be very convincing though.

6
Guideline 2 Consider the Source
Sec. 1.4

• It is important to consider the source of a study
  and evaluate potential for biases
• Examples
• 4 out of 5 dentists prefer our brand
• This statement appears to be statistically based,
  but we are given no details about how the survey
  was conducted and since advertisers would only
  want to say good things about their brand, if
  they themselves conducted the survey, it would
  have a lot of potential for bias
• A carefully conducted study concludes that a new
  drug helps cure cancer
• Might seem believable at first, but it was later
  found that the study was funded by the drug
  company, which means that they would have a lot
  to gain by making their drug appear better than
  it may actually be
• A study that has undergone peer review a
  process in which several experts in a field
  evaluate a research report before it is published
  has much more credibility because it implies
  that other experts agree that the study was
  carried out properly

7
Example 2
Sec. 1.4

• By 1963, research had so clearly shown the health
  dangers of smoking that the Surgeon General of
  the United States publicly announced that smoking
  is bad for health. Research done since that time
  built further support for this claim. However,
  while the vast majority of peer reviewed studies
  showed that smoking is unhealthy, a few studies
  found no dangers from smoking and perhaps even
  health benefits. These studies generally were
  carried out by the Tobacco Research Institute,
  funded by the tobacco companies. Analyze these
  studies according to Guideline 2.
• Since the peer reviewed studies are more credible
  than the studies conducted by the tobacco
  companies, there is reason to believe that the
  studies showing health benefits are biased and
  before believing the results presented, you
  should analyze the study further

8
Guideline 3 Examine the Sampling Method
Sec. 1.4

• A statistical study cannot be valid unless the
  sample is representative of the population, so
  you should examine the sampling method for signs
  of bias.
• Selection bias occurs whenever researchers
  select their sample in a way that tends to make
  it unrepresentative of the population
• Ex. A pre-election poll that surveyed only
  registered republicans would be considered
  selection bias since the survey didnt include
  all parties
• Participation bias occurs when people choose to
  be part of a study
• This mostly occurs in self-selected surveys since
  people who feel strongly about an issue are more
  likely to participate and their opinions may not
  represent the opinions of the population in
  general
• Ex. A survey was mailed to random households
  asking if they supported a womans right to
  abortion or not.
• Since this would require the survey to be mailed
  back, it is likely that only those who feel
  strong about this topic would participate, thus
  creating a biased study

9
Example 3
Sec. 1.4

• The television show Nightline conducted a poll in
  which viewers were asked whether the United
  Nations headquarters should be kept in the United
  States. Viewers could respond to the poll by
  paying 50 cents to call a 900 phone number with
  their opinions. The poll drew 186,000 responses,
  of which 67 favored moving the United Nations
  out of the United States. Around the same time, a
  poll using simple random sampling of 500 people
  found that 72 wanted the United Nations to stay
  in the United States. Which poll is more likely
  to be representative of the general opinions of
  Americans?
• The random sample is more likely to be
  representative of the general opinions of
  Americans
• The phone poll had severe bias
• There was a selection bias because they only drew
  from Nightline viewers
• There was participation bias because this was a
  self-selected study in which those who were
  willing to respond not only had to take the time
  to call in, but they also had to pay 0.50

10
Guideline 4 Look for Problems in Defining or
Measuring the Variables of Interest
Sec. 1.4

• Results of a study may be difficult to interpret
  if the variables under study are difficult to
  define or measure
• Ex. A study on how exercise affects resting
  heart rates
• Both variables of interest (amount of exercise
  and resting heart rate) are hard to define and
  measure how would you define and measure amount
  of exercise?

11
Example 4
Sec. 1.4

• A commonly quoted statistic is that law
  enforcement authorities succeed in stopping only
  about 10 to 20 of the illegal drugs entering
  the United States. Should you believe this
  statistic?
• In this study there is a comparison between the
  amount of illegal drugs intercepted and the
  amount that is not intercepted.
• The statistic presented is very hard to believe
  since there is no way for us to know the amount
  of illegal drugs that are entering the U.S that
  arent being stopped

12
Guideline 5 Watch Out for Confounding Variables
Sec. 1.4

• Recall that confounding variables are variables
  that were not intended to be part of the study,
  but may play a large role in interpreting the
  results properly
• Not always easy to discover these variables, but
  can be accomplished by simply by thinking hard
  about the factors that may have influenced a
  studys results

13
Example 5
Sec. 1.4

• Radon is a radioactive gas produced by natural
  processes (the decay of uranium) in the ground.
  The gas can leach into buildings through the
  foundation and can accumulate to relatively high
  concentrations if doors and windows are closed.
  Imagine a (hypothetical) study that seeks to
  determine whether radon gas causes lung cancer by
  comparing the lung cancer rate in Colorado, where
  radon gas is fairly common, with the lung cancer
  rate in Hong Kong, where radon gas is less
  common. Suppose the study finds that the lung
  cancer rates are nearly the same. Would it be
  reasonable to conclude that radon is not a
  significant cause of lung cancer?
• Since radon gas is not the only cause of lung
  cancer, smoking is also known to cause it, it
  would be hard to believe the statement made in
  the study without further investigation

14
Guideline 6 Consider the Setting and Wording in
Surveys
Sec. 1.4

• Watch for problems in the setting or wording that
  may produce dishonest responses
• Ex. Do you cheat on your income taxes?
• Unlikely to elicit honest answers from those who
  cheat

15
Example 6
Sec. 1.4

• At a time when the U.S. government was running
  annual budget surpluses, Republicans in Congress
  proposed a tax cut and the Republican National
  Committee commissioned a poll to find out whether
  Americans supported the proposal. Asked Do you
  favor a tax cut?, 67 of respondents answered
  yes. Should we conclude that Americans supported
  the proposal?
• A question like Do you favor a tax cut? is
  biased because it does not give other options for
  uses of the money such as social security or
  using it towards national debt

16
Guideline 7 Check That Results Are Fairly
Represented in Graphics or Concluding Statements
Sec. 1.4

• Even if a statistical study is done well, it may
  be misinterpreted in graphics or concluding
  statements
• News reporters may misinterpret a survey or jump
  to unwarranted conclusions to make a story seem
  more spectacular
• Ex. What if you were told that your
  son/daughter was in the bottom 3 of the class in
  a particular test? Its sounds bad, but what if
  the top result was 98 and the lowest result was
  76. Then the bottom 3 doesn't sound all that
  bad.

17
Guideline 8Stand Back and Consider the
Conclusions
Sec. 1.4

• Even if a study seems reasonable according to all
  the previous guidelines, ask yourself these
  questions
• Did the study achieve its goals?
• Do the conclusions make sense?
• Can you rule out alternative explanations for the
  results?
• If the conclusions make sense, do they have
  practical significance?

18
Example 7
Sec. 1.4

• Suppose a (hypothetical) study concludes that
  wearing a gold chain increases your chances of
  surviving a car accident by 10. The claim is
  based on a statistical analysis of data about
  survival rates and what people were wearing.
  Careful analysis of the research shows that it
  was conducted properly and carefully. Should you
  start wearing a gold chain whenever you drive a
  car?
• Despite the care that went into the study, the
  claim that a gold chain can save your life in a
  collision is difficult to believe. After all, how
  could a thin chain help in a highspeed collision?
  
• Its certainly possible that some unknown effect
  of gold chains makes the conclusion correct, but
  it seems far more likely that the results were
  either a fluke or due to an unidentified
  confounding variable
• For example, perhaps those who wear gold chains
  are wealthier and drive newer cars with more
  advanced safety features, lowering their fatality
  rate