Experiments: Background and terminology

1
Experiments Background and terminology

• A study is an experiment when we actually do
  something to people, animals or objects in order
  to observe the response.

2

• The purpose of an experiment is to reveal the
  change in one variable in response to changes in
  other variables
• - The distinction between explanatory and
  response variables is important.
• - The explanatory variables in an experiment are
  often called factors.
• - The specific values that a factor (explanatory
  variable) can take are called levels of that
  factor.

3

• In an experiment, we study the specific factor(s)
  we are interested in, while controlling the
  effects of lurking variables. As a result,
  experiments can give good evidence for causation.
• - If we study just one factor, then the levels
  of that factor are considered as treatments. This
  kind of experiment is called a single-factor
  experiment.
• - Many experiments study the joint effects of
  several factors. In such an experiment, each
  treatment is formed by combining levels of each
  of the factors. This kind of experiment is called
  a multi-factor experiment.

4
Example

• To study the effect of alcohol on the drop in
  body temperature, researchers give several
  different amounts of alcohol (5 ml, 10 ml and 15
  ml) to mice and then measure the change in each
  mouses body temperature in the 15 minutes after
  taking the alcohol.
• 1. What are the experimental units?
• 2. What is(are) the factor(s)?
• 3. What kind of experiment is this single-factor
  or multi-factor?
• 4. What are the treatments?
• 5. What is the response variable?

5
Example

• A study is designed to determine the effect of
  temperature and pressure on a chemical process.
  Three different temperature settings (50F,150F
  and 200F) are considered, and for each
  temperature setting, two pressure settings (1 bar
  and 2 bar) are considered. For each of the
  setups, the process is run and the completion
  time (in minutes) is recorded.
• 1. What are the experimental units?
• 2. What is(are) the factor(s)?
• 3. What kind of experiment is this single-factor
  or multi-factor?
• 4. What are the treatments?
• 5. What is the response variable?

6
Confounding

• Two factors (variables) are confounded if we
  cannot determine which one caused the differences
  in the response.
• Confounding can result when there exists a
  variable which influences the response, but isnt
  properly controlled.

7
Examples of confounding

• Suppose we compared the writing speed of three
  different brands of pens, each with a different
  tester.
• - Even if we find difference in the
  writing speed, we would not know if that is due
  to the different brands or due to different
  testers.
• - We say, brand and tester are confounded.
• Suppose we compared the effectiveness of two
  teaching methods, each with a different group of
  students.
• -Even if we find difference in the
  effectiveness, we would not know if that is due
  to the different teaching methods or different
  student groups.
• - In this case, teaching method and student
  group are confounded.

8
Randomized Experiment

• The key to a randomized experiment the
  treatment (explanatory variable) is randomly
  assigned to the experimental units or subjects.

Random Assignment
Compare
9
Example of Randomized Experiment

• Suppose that before we want to test the effect of
  aspirin on the physicians, we wish to do a study
  on the effect of aspirin on mice, comparing heart
  rates.
• We obtain a random sample of 100 mice.
• We randomly assign 50 mice to receive a placebo.
• We randomly assign 50 mice to receive aspirin.
• After 20 days of administering the placebo and
  aspirin, we measure the heart rates and obtain
  summary statistics for comparison.

10
Advantages of Randomized Experiment

• The single greatest advantage of a randomized
  experiment is that we can infer causation.
• Through randomization to groups, we have
  controlled all other factors and eliminated the
  possibility of a confounding variable.
• Unfortunately or perhaps fortunately, we cannot
  always use a randomized experiment
• Often impossible or unethical, particularly with
  humans.

11
Comparative experiments

• Laboratory experiments in science and engineering
  often have a simple design with only a single
  treatment, which is applied to all of the
  experimental units.
• We rely on the controlled environment of the
  laboratory to protect us from lurking variables.
• When experiments are conducted in the field or
  with living subjects, such simple designs often
  yield invalid data.

12
The Bias

• Uncontrolled experiments in medicine and the
  behavioral sciences can be dominated by such
  influences as the details of the experimental
  arrangement, the selection of subjects, and the
  placebo effect. The result is often bias.

13
Examples of Bias

• A researcher wants to study the effect of two
  drugs (drug A and drug B) on reduction of blood
  pressure. She forms two groups of subjects one
  group with 20 people, all aged 25 years and
  another group with 20 people aged 50 and above.
  The first group was given drug A while the second
  group got drug B. After the study it was found
  that the decrease in blood pressure is higher in
  group 2 (which got drug B). Can you conclude that
  drug B is more effective than drug A in reducing
  blood pressure? Why or why not?

14
Examples of Bias

• A company wants to know peoples views on the
  products they sell. To achieve this, the company
  conducts a survey where they interview the
  existing customers and records their opinions on
  the products. Based on the collected data, it is
  then found out that people are generally happy
  about using the companys products. Do you have
  any reasons to doubt this conclusion?

15
Randomization

• Q. How can we assign experimental units to
  treatments in a way that is fair to all of the
  treatments?
• A. The statisticians remedy is to rely on chance
  to make an assignment that does not depend on any
  characteristic of the experimental units and that
  does not rely on the judgment of the experimenter
  in any way.

16
Example of Randomization

• A food company assesses the nutritional quality
  of a new instant breakfast product by feeding
  it to newly weaned male white rats and measuring
  their weight gain over a 28-day period. A control
  group of rats receives a standard diet for
  comparison. The researchers use 30 rats for the
  experiment and so must divide them into two
  groups of 15. What should be done to do this in a
  completely unbiased fashion?
• The use of chance to divide experimental units
  into groups is called randomization.

17
Advantages

• Randomization produces groups of rats that should
  be similar in all respects before the treatments
  are applied.
• Comparative design ensures that the influences
  other than the diets operate equally on both
  groups.
• Therefore, differences in average weight gain
  must be due either to the diets or to the play of
  chance in the random assignment of the rats to
  the two diets.
• We hope to see a difference in the responses so
  large that it is unlikely to happen just because
  of chance variation.

18
(No Transcript)
19
Cautions about experimentation

• The logic of a randomized comparative experiment
  depends on our ability to treat all the
  experimental units identically in every way
  except for the actual treatments being compared.
• Many experiments have some weaknesses in detail.
• - The environment of an experiment can influence
  the outcomes in unexpected ways.
• The most serious potential weakness of
  experiments is lack of realism.

20

• -The subjects or treatments or setting of an
  experiment may not realistically duplicate the
  conditions we really want to study.
• - Lack of realism can limit our ability to apply
  the conclusions of an experiment to the settings
  of greatest interest.
• - Although experiments are the gold standard for
  evidence of cause and effect, really convincing
  evidence usually requires that a number of
  studies in different places with different detail
  produce similar results.

21

• Most experimenters want to generalize their
  conclusions to some setting wider than that of
  the actual experiment.
• Statistical analysis of an experiment cannot
  tell us how far the results will generalize to
  other settings.
• Nonetheless, the randomized comparative
  experiment, because of its ability to give
  convincing evidence for causation, is one of the
  most important ideas in statistics.

22
Inference Overview

• Recall that inference is using statistics from a
  sample to talk about a population.
• We need some background in how we talk about
  populations and how we talk about samples.

23
Example

• A market research farm interviews a random sample
  of 2500
• adults. Result 66 find shopping for clothes
  frustrating and time consuming.
• Note that here 66 actually means 66 of
  the 2500 people in the sample. But what is this
  percentage for all the 220 million American
  adults who make up the population?
• Because the sample is random, it is reasonable
  to think that the sample represents the
  population well. So the market researchers turn
  the fact that 66 of the sample find shopping
  frustrating into an estimate that 66 of all
  adults in the population feel this way.
• This is a basic move in statistics use a fact
  about a sample to estimate the truth about the
  whole population. We call this statistical
  inference we infer conclusions about the wider
  population based on the data from on some
  selected individuals.

24
Inference Overview

• Describing a Population
• It is common practice to use Greek letters when
  talking about a population.
• We call the mean of a population m.
• We call the standard deviation of a population s
  and the variance s2.
• When we are talking about percentages, we call
  the population proportion p. (or pi).
• It is important to know that for a given
  population there is only one true mean and one
  true standard deviation and variance or one true
  proportion.
• There is a special name for these values
  parameters.

25
Inference Overview

• Describing a Sample
• It is common practice to use Roman letters when
  talking about a sample.
• We call the mean of a sample .
• We call the standard deviation of a sample s and
  the variance s2.
• When we are talking about percentages, we call
  the sample proportion p.
• There are many different possible samples that
  could be taken from a given population. For each
  sample there may be a different mean, standard
  deviation, variance, or proportion.
• There is a special name for these values
  statistics.

26
Inference Overview

• We use sample statistics to make inference about
  population parameters

m
s
s
p
p
27
(No Transcript)
28
Examples

• Voter registration records show that 68 of all
  voters in Indianapolis are registered as
  Republicans. To test a random digit dialing
  device, you use the device to call 150 randomly
  chosen residential telephones in Indianapolis.
  Among the call recipients 73 are registered
  Republicans.
• A carload lot of ball bearings has a mean
  diameter of 2.503 centimeters This is within the
  specifications for acceptance of the lot by the
  purchaser. The inspector happens to inspect 100
  bearings from the lot with a mean diameter of
  2.515 cm. This is outside specified limit so the
  lot was rejected
• 3. A telemarketing firm in Los Angeles uses a
  device that dials residential telephone numbers
  in that city at random. Of the first 100 numbers
  dialed, 43 are unlisted. This is not surprising,
  because from a census it is known that 52 of all
  Los Angeles residential phones are unlisted.

29
Sampling Variability

• There are many different samples that you can
  take from the population.
• Statistics can be computed on each sample.
• Since different members of the population are in
  each sample, the value of a statistic varies from
  sample to sample.

30
Sampling Distribution

• The sampling distribution of a statistic is the
  distribution of values taken by the statistic in
  all possible samples of the same size from the
  same population.
• We can then examine the shape, center, and spread
  of the sampling distribution.

31

• To assess sampling variability, one can use the
  following procedure
• Take a large number of samples
• Compute the statistic for each of those
  samples
• Compute the standard deviation of those
  statistic values. This is called the standard
  error of the statistic.
• Look at the distribution of those values by
  drawing a histogram. This distribution is called
  the sampling distribution of the statistic.

32
Bias and Variability

• Bias concerns the center of the sampling
  distribution. A statistic used to a parameter is
  unbiased if the mean of the sampling distribution
  is equal to the true value of the parameter being
  estimated.
• To reduce bias, use random sampling. The values
  of a statistic computed from an SRS neither
  consistently overestimates nor consistently
  underestimates the value of the population
  parameter.
• Variability is described by the spread of the
  sampling distribution.
• To reduce the variability of a statistic from an
  SRS, use a larger sample. You can make the
  variability as small as you want by taking a
  large enough sample.

33
Bias and Variability