Experimental Research

1
Experimental Research
Attempts to Establish Cause and Effect
Relationships
2
Cause and Affect Cannot exist unless two
variables are correlated

• i.e. evaluate if two variables are correlated
  before trying to manipulate one to change the
  other.

3
The Logic

• Cause and Effect can only be established if there
  is no other reasonable explanation for the
  changes in the dependent variable except the
  manipulation of the independent variable

4
Important Factors

• Good theoretical framework
• Appropriate experimental design
• Correct statistical model Design
• Proper selection Control of Independent
  variable
• Appropriate selection Measurement of dependent
  variable
• Use of appropriate subjects
• Correct interpretation of results

5
Internal Validity

• The extent to which the results of the study can
  be attributed to the treatments used in the study.

Requirement without which study is
uninterpretable.
6
External Validity

• Generalizability of the study results.

7
Strong internal validity may be at odds to strong
external validity

• In real-world settings everything is not
  controlled and may not operate in the same way as
  the laboratory.

8
The Dilemma!!!

• Is it more important to ascertain that the
  manipulation of the independent variable caused
  the observed changes in the dependent variable...
  Or, is it more important to generalize the
  results to other populations?

9
Threats to Internal ValidityCampbell Stanley
(1963)
10

• History Events occurring during the experiment
  that are not a part of the treatment.i.e.
  Evaluating effects of semester-long fitness
  program on 8th graders. 60 of students also
  participated in recreational soccer will affect
  results.

11

• MaturationProcesses within the subjects that
  operate as a result of time passing. Occurs most
  frequently when one group is tested several times
  over a long period of time. i.e. Fatigue,
  again, hunger. Fitness testing in fall and
  spring--students do better in spring.

12

• TestingThe effects of one test on subsequent
  administrations of the same test. i.e. Giving
  test knowledge about Performance enhancing drugs
  to students today and re-testing two days later.

13

• InstrumentationChanges in the instrument
  calibration, including lack of agreement within
  and between observers.Same observer may
  systematically vary rating across time or
  subjects (observer drift) or different observers
  may not rate the same performance the same way.
  (i.e. Skin Fold Study)

14

• Statistical RegressionThe fact that Groups
  selected on the basis of extreme scores are not
  as extreme on subsequent testing.May occur when
  groups are not randomly formed but selected on
  some extreme score on a measure. (i.e. observing
  active and inactive behaviors. In other settings
  may regress toward the mean.)

15

• Selection BiasIdentification of comparison
  groups in other than a random manner. Occurs when
  groups are not randomly assigned.Did the study
  use volunteers? Are subjects extremely
  non-representative of the population?

16

• Experimental MortalityLoss of subjects from
  comparison groups due to nonrandom reasons.Loss
  of subjects from the treatment groups.

17

• Selection-Maturation InteractionSpecific to
  non-equivalent group designs where the passage of
  time might affect one group but not the
  other.i.e. Difference between expert and novice
  tennis payers. Experts might have higher
  socio-economic status, access to country clubs,
  and better instruction.

18
Additional Threat

• ExpectancyResearchers anticipating certain
  subjects will perform better.i.e. Rating
  subjects labeled as skilled better than those
  labeled as unskilled regardless of the
  treatment.

19
Threats to External ValidityCampbell Stanley
(1963)
20

• 1) Reactive or Interactive Effects of
  TestingFact that the pretest may make the
  subject more aware of or sensitive to the
  upcoming treatment. Results in treatments not
  being as effective without the pretest.

21

• Interaction of Selection Biases and the
  Experimental TreatmentWhen a group is selected
  on some characteristic, the treatment may work
  only on groups possessing that characteristic.

22

• Reactive Effects of Experimental
  ArrangementsTreatments that are effective in
  very constrained situations (labs) may not be
  effective in less constrained (real-world)
  settings.

23

• Multiple-Treatment InterferenceWhen subjects
  receive more than one treatment, the effects of
  previous treatments may influence subsequent ones.

24
Controlling Threats
25
Internal Validity

• RandomizationControls For1) History (up
  until experiment)2) Maturation3) Statistical
  Regression4) selection Biases5) Selection
  Maturation Interaction

26

• Test ReliabilityInstrumental cannot be
  controlled or evaluated by any design only
  experimenter can control.Psychometrics
  Establishing appropriate instrumentation.

27

• Subject RetentionCannot be controlled by
  experimental design. Careful discussions about
  importance of study with subjects.

28

• Placebos, Blind, Double-Blind SetupsPlacebo
  is treatment effect real or due to psychological
  effects?Blind Study Subject does not know
  whether receiving treatment or control.Double-Bl
  ind Neither subject or Researcher knows who is
  receiving treatment or control.

29
External Validity

• Random selection is the key.Random selection of
  subjects, treatments, and experimental situations.