Quasiexperiment lack control group or pretest observation

1
Quasi-experiment lack control group or pretest
observation

• Ilham El-saleh Alma Rangel

2
Definition

• Quasi-experimental designs are commonly employed
  in the evaluation of educational programs when
  random assignment is not possible or practical.
  However, these designs are subjected to many
  limitations and interpretations that we are going
  to see in different examples.

3
Why do researchers using such designs? The best
design for a given study

• Practical necessities imposed by funding, ethics,
  or administrators.
• Logistical constraints that occur when an
  intervention has already been fielded before the
  evaluation of the intervention is designed.

4
Principles address requirements in such design

• Identification and study of plausible threats to
  internal validity.
• Primacy of control by design.
• Coherent pattern matching.

5
What does Control or equivalent group design
mean?

• The presence of control group in a design will
  give the researcher the chance to compare
  between the result of two different groups. Each
  of two groups received two different kinds of
  treatment. For example, one group of students
  might receive math instruction using a whole new
  program while the other receives a different math
  program (nothing). After certain time, a test can
  be administered to see which program was more
  effective.

6
Threats to Validity

• There are several factors that can jeopardize the
  validity of an experimental design
• They can be broken into internal and external
  validity
• Internal Validity. (interpretability) Are our
  claims concerning the effects of the treatment
  valid in this particular experiment?
• External Validity. (generalizability) Can our
  results from this experiment be generalized to
  other populations and settings?

7
Internal Validity

• Eight different classes of extraneous variables,
  which, if not controlled in the experimental
  design, might produce effects confounded with the
  effect of the experimental stimulus.
• History - the specific events occurring between
  the first and second measurement in addition to
  the experimental value, creating rival hypotheses
  (O1 X O2)
• Maturation - processes within the respondents
  operating as a function of the passage of time
  per se (not specific to the particular events),
  including growing older, hungrier, more tired,
  etc.
• Testing - the effects of taking a first test upon
  the scores of a second testing.
• Instrumentation - changes in the calibration of a
  measuring instrument or changes in the observers
  or scorers used, may produce changes in the
  obtained measurements.

8
Internal Validity

• Statistical Regression - operating where groups
  have been selected on the basis of their extreme
  scores, i.e., tendency toward the mean.
• Selection - biases resulting in differential
  selection of respondents for the comparison
  groups (X O1, O2)
• Experimental Mortality - differential loss of
  respondents from the comparison groups.
• Selection-Maturation Interaction, etc. - any of
  the extraneous variables can have a combined
  effect that can be mistaken for the effect of the
  experimental variable.

9
Designs without control groups( non-treatment
control group)

• One group posttest only
• X O1
• Improving onegroup posttest with multiple
  posttest
• X O1A O1B. O 1N

treatment
posttest
treatment
10
More..

• One group pretest posttest
• O1 X O2

pretest
treatment
posttest
11
Example One Group Pre-test Post-test Design

• O1 X O2
• X learn SPSS
• O1 time to solve problems in another
• language (e.g. SAS)
• O2 time to solve problems in SPSS
• Does not control for history, maturation, etc.

12
Pretest-Posttest Design
13
More

• Improving the one group pretest posttest using
  double pretest
• O1
  O2 X O3

treatment
posttest
pretest
pretest
14
More

• One group pretest posttest using a non
  equivalent dependent variable. (same group at
  times 1 2)
• O1A , O1B X O2A
  , O2B

pretest
pretest
treatment
posttest
posttest
15
More

• The removed- treatment design
• O1 X O2
  O3 X O4
• (The outcome rises and falls with the presence of
  the treatment)
• The repeated- treatment design
• O1 X O2 X
  O3 X O4

pretest
treatment
posttest
pretest
posttest
No treatment
treatment
remove
pretest
posttest
Re treatment
posttest
pretest
16
Designs that use control group but no pretest

• Add a control group that receives no treatment,
  with the control group selected to be as similar
  as possible to the treatment group.
• In this design there is not a pretest measure on
  the outcome variable.

17
Notation Symbols

• O Observation
• X Treatment
• R Random Assignment to Group
• -- or NR No Random Assignment

18
Quasi-Experimental Designs that use Control
Groups but No Pretest

• Posttest-Only Design With Nonequivalent Groups.
• NR X O1
• --------------------------------------
• NR O2
• Posttest-Only Design using an Independent Pretest
  Sample.
• NR O1 X O2
• --------------------------------------
• NR O1 O2
• Posttest-Only Design using Proxy Pretests.
• NR OA1 X OB2
• ----------------------------------------
• NR OA1 OB2

Dashed line between groups means that they were
not randomly formed
19
Proxy-Pretest Design

• In this design one gathers the pretest
  information after the experimental treatment has
  started. In other words, one finds find a proxy
  variable that would estimate pretest performance.
  
• For example, suppose I ask the following
  question Does completion of EDCI 627 have an
  effect on a students knowledge of statistics?
  Ideally I would measure the students statistical
  knowledge at the beginning of the semester, but
  suppose that the question did not occur to me
  until the middle of the semester. I might decide
  to use as a proxy-pretest students performance
  in their EDCI 627 (statistics) class. My control
  group might consist of a group of students taking
  some other class (not 627).

20
Cont. Proxy-Pretest Design

• For each student I would obtain a continuous
  measurement of the students performance in EDCI
  627 and, at the end of the semester, a continuous
  measurement of the students knowledge of
  statistics.
• The proxy pretest design is not one you should
  ever select by choice. But, if you find yourself
  in a situation where you have to evaluate a
  program that has already begun, it may be the
  best you can do and would almost certainly be
  better than relying only on a posttest-only
  design.

21
The Posttest-Only Design Using Matching or
Stratifying

• Matching. The units in the Treatment and Control
  groups are exactly equal (rather than just
  similar) on a matching variable.
• Stratifying. The units are placed into
  homogeneous sets that contain more units than the
  experiment has conditions.
• Methods for Matching. Exact matching requires
  units to have exactly the same score within a
  match. However, some units will not have an exact
  match if samples are small, if the distribution
  of participants between groups on the matching
  variable is uneven, or if variables are measured
  using very fine gradations.

22
Problems with matching..

• There is always the possibility of selection
  bias.
• Matching in quasi-experimentation works least
  effectively when it is done on an unstable or
  unreliable variable and when the nonequivalent
  groups from which the matched sets are drawn are
  increasingly dissimilar when matched
• Two methods that can help to counteract these
  problems are
• - Select groups that are similar as possible
  before matching, as much as the context and
  research question allow.
• - Use matching variables that are stable and
  reliable.

23
Other Quasi-Experimental Designs..

• The Posttest-Only Design Using Internal Controls.
• - Internal control group are plausibly drawn
  from a population similar to that from which the
  treatment units are taken. Internal controls do
  not guarantee similarity.
• The Posttest-Only Design Using Multiple Controls
  Groups.
• - The use of multiple nonequivalent control
  group.
• The Posttest-Only Design Using Predicted
  Interaction.
• - The objective is predict statistical
  interaction. It can be useful for predict a
  complex data pattern, outcomes measures can be
  reliably measured, etc..

24
Improving Designs without Control groups by
Constructing Contrasts other than with
independent control groups.

• When it is not possible to gather prospective
  data on the kinds of independent control groups,
  it is sometimes possible to construct contrast
  that try to mimic the function of an independent
  control group. Three such contrasts are
• - Regression Extrapolation Contrasts. It
  compares actual and projected posttest scores.
• - Normed Comparison Contrasts. It compares
  treatment recipients to normed sampled.
• - Secondary source Contrasts. It compares
  treatment recipients to samples drawn form
  previously gathered data, such as
  population-based surveys.

25
The Case Control Design

• In this design, one group consists of cases that
  have the outcome of interest, and the other group
  consists of controls that do not have it.
• - Ex. If we compare the exposure
  distribution between the groups of subjects with
  and without disease.
• The outcome in this design is typically
  dichotomous.
• Cases and controls are then compared using
  retrospective data to see if cases experienced
  the hypothesized cause more often than controls.
• The case-control design is excellent for
  generating hypotheses about causal connections.
  Some case-control studies include birth control
  pills, smoke and cancer, etc.. Case-control
  studies are more feasible than experiments in
  cases in which an outcome is rare or takes years
  to develop they are often cheaper and
  logistically easier to conduct and they may
  decrease risk to participants who could be
  needlessly exposed to a harmful experimental
  treatment.

26
Sources

• http//www.socialresearchmethods.net/kb-old/quasio
  th.htm
• http//faculty.ncwc.edu/toconnor/308/308lect06.htm
  
• http//core.ecu.edu/psyc/wuenschk/docs2210/Researc
  h-8-QuasiExpDesign.doc
• Campbell, Shadish Cook, 2002. Experimental and
  Quasi-Experimental Designs for Generalized Causal
  Inference. Houghton Mifflin Company