Experimental, QuasiExperimental, and Ex Post Facto CausalComparative Research

1
Experimental, Quasi-Experimental, and Ex Post
Facto (Causal-Comparative) Research
2
Characteristics of Experimental Research

• There is a control or comparison group
• Subjects are randomly assigned to groups
• The treatment is randomly assigned to groups.

3
Characteristics of Quasi-Experimental Research

• There is a control or comparison group
• Intact groups are used
• The treatment is randomly assigned to groups.

4
Characteristics of Ex Post Facto Research

• There is a control or comparison group
• Intact groups are used
• The treatment is not manipulated, it has already
  occurred.

5
(No Transcript)
6
Diagramming Research

• To illustrate research designs, a number of
  symbols are used
• X1 Treatment
• X2 Control Group
• O Observation (pretest or posttest)
• R Random Assignment

7
A Sample Research Design

• Single-Group Pretest-Treatment-Posttest Design
• R O X1 O

This means subjects are randomly assigned to a
group, which is then given a pretest, then there
is a treatment, then there is a posttest.
8
R O X1 O

• This is not really an experimental design because
  there is no control group
• It is often referred to as a preexperimental
  design
• Novice researchers often use this research design
• There are some major problems with this design
  did the treatment really make the difference or
  was something else happening.

9
R O X1 O

• What are the threats to the Internal Validity of
  this type of research (Did the treatment really
  cause a difference?)

10
Internal Validity Threats
R O X1 O

• History
• Another event occurs during the time of the
  experiment that might cause the difference
• An experiment to heighten racial awareness was
  conducted by a researcher during February. This
  is Black History month so the results might be
  affected by events that occur during Black
  History month and not the treatment.

11
Internal Validity Threats
R O X1 O

• Maturation
• People naturally change and evolve over time.
  This may cause the difference.
• A college develops a new housing plan to promote
  more open-mindness and acceptance of others. The
  students are tested when they enter college and
  when they graduate. The results show they are now
  more open-minded and tolerant of others. Did the
  housing plan work or do students just mature and
  grow as a result of the college experience.

12
Internal Validity Threats
R O X1 O

• Mortality
• Some people drop out during an experiment. This
  may affect the outcome.
• I am teaching a new experimental seminar on study
  skills. About half of the class stopped coming to
  the seminar before the semester was over. The
  students who remained improved their study
  skills. So my course was effective!
• Probably not. The half that stopped coming might
  not have gained anything that is why they
  stopped attending.

13
Internal Validity Threats
R O X1 O

• Testing
• Whenever you give a pretest, the students may
  remember the test questions, and get them correct
  on the posttest.
• I gave a test to my study skills group on Monday,
  presented some unique concepts on Tuesday, then
  gave them the posttest on Wednesday. The grades
  were significantly higher on the posttest.
• It is possible the grades were higher because the
  students still remembered the questions from the
  pretest.

14
Internal Validity Threats
R O X1 O

• Instrumentation
• To overcome the testing threat to internal
  validity, a researcher develops a different form
  of the test instrument, but it is not really
  equivalent.
• I gave a test to my study skills group on Monday,
  presented some unique concepts on Tuesday, then
  gave them an alternative form of the pretest on
  Wednesday. The grades were significantly higher
  on the posttest.
• It is possible the grades were higher because the
  second test was easier than the first.

15
Internal Validity Threats
O X1 O

• Regression
• When subjects are selected because of extreme
  scores on some type of instrument, there is
  tendency for their scores to move more toward the
  average on subsequent tests.
• An experimenter selected students for a reading
  program based on their low test scores. At the
  end of the treatment, the test scores had
  improved.
• Extreme scores naturally move toward the mean on
  subsequent tests.

16
How to Handle Internal Validity Threats

• Have a control group and use randomization.This
  design is the Two-Group Pretest-Treatment-Posttest
  Design.

The Control Group would experience the same
history and maturation. Mortality should be the
same because of random assignment. Random
assignment eliminates the selection threat.
However testing and instrumentation could still
be a threat.
R O X1 O R O X2 O
17
Other Research Designs

• Two-Group Treatment-Posttest-Only Design

There is no pretest so this eliminates the
testing and instrumentation threat to internal
validly but you dont know about their knowledge
or attitude coming into the study.
R X1 O R X2 O
18
Other Research Designs

• Solomon 4-Group Design

Note A blank indicates the control group, same
as X2
R O X1 O R X1 O R O O R
O
19
Quasi-Experimental Designs

• Posttest Only Nonequivalent Group Design

The absence of R indicates there is no random
assignment. Sometimes you will see a dotted line
between the two groups. This indicates the two
groups may not be equivalent.
X1 O X2 O
20
Quasi-Experimental Designs

• Pretest-Posttest Nonequivalent Group Design

O X1 O O X2 O
21
Time Series Designs

• O O O X1 O O O

In the next course, AEE 579 Research Design, many
more research designs are examined.
22
External Validity

• Can the research be generalized to other
  settings?
• Population Validity
• Personological Variables
• Ecological Validity

23
Population Validity

• Is the sample population similar to the
  population the researchers wishes to generalize to

24
Personological Variables

• Different people have different personalities,
  learning styles, etc., so the results may not be
  generalizable to people who are substantially
  different on these personological variables.

25
Ecological Validity

• The setting or situation in which the experiment
  occurred may be different than other settings.

26
Social Interaction Validity Threats

• Diffusion or Imitation of Treatment
• This occurs when a comparison group learns about
  the program either directly or indirectly from
  program group participants.
• This group may try to imitate or emulate what the
  treatment group is getting.

27
Social Interaction Validity Threats

• Compensatory Rivalry
• The comparison group knows what the program group
  is getting and develops a competitive attitude
  with them.

28
Social Interaction Validity Threats

• Resentful Demoralization
• This is almost the opposite of compensatory
  rivalry. Here, students in the comparison group
  know what the program group is getting. But here,
  instead of developing a rivalry, they get
  discouraged or angry and they give up.

29
Social Interaction Validity Threats

• Compensatory Equalization of Treatment
• The researcher is under pressure to enrich the
  experiences of the control group. This pressure
  may come from parents, school administrators, etc.

30
Ex Post Facto (Causal-Comparative) Research

• Explores possible causes and effects
• The independent variable is not manipulated, it
  has already been applied
• Focuses first on the effect, then attempts to
  determine what caused the observed effect.

31
Statistical Analysis

• If we are comparing the scores of two groups a
  t-test is normally used. The value of t means
  nothing by itself (unlike the value of R). We
  have to determine if t is statistically
  significant

Tea for two
32
Statistical Analysis

• If we are comparing the scores of three (or more)
  groups Analysis of Variance (ANVOA) is used.
  This test gives us a f value which means nothing
  by itself. We have to determine if it is
  statistically significant.

33
Statistical Analysis

• If we want to statistically equate two or more
  groups (because one group had a high pretest
  score) we use Analysis of Covariance (ANCOVA).
  This test gives us a f value which means nothing
  by itself. We have to determine if it is
  statistically significant.