Experimental Design

1
Experimental Design

• Week 9 Lecture 1

2
Agenda

• Purpose of experimental design
• Key elements in experimental design
• Various types of experimental design
• Causal-comparative design
• Field and controlled laboratory experiment

3
Purpose of experimental design

• Examine the possible influences that one factor
  or condition may have on another factor or
  condition
• Build on positivist approach
• Research questions appropriate for an experiment
• Issues that have a narrow scope or small scale
• Can rarely address question that may require
  looking at conditions across an entire society or
  across decade
• Small number of participants (subjects) compared
  with survey research

4
Independent variables

• Independent variable
• Cause
• Research measure (manipulate) independent
  variable by creating a condition or situation
• Manipulation of independent variable create
  different treatments.
• Event manipulation
• Affecting the independent variable by altering
  the events that subjects experience
• Presence versus absence
• Instructional manipulation
• Varying the independent variable by giving
  different sets of instructions to the subjects

5
Dependent variables

• Effect (outcome)
• Physical conditions, behaviors, attitudes,
  feelings, or beliefs of subjects that change in
  response to a treatment.
• How to measure
• Questionnaire, interviews
• Observation
• Test
• Direct outcome

6
The importance of control

• Internal validity -- The extent to which we can
  accurately state that the independent variable
  produced the observed effect

7
Strategies to achieve control

• Keep some things constant
• What are variables that need to be held constant
  in most experiments?
• Include a control group
• Treatment group (experimental group)
• Between-subjects design
• Randomly assign people to groups
• Use matched pairs
• Matched-subject design
• Expose participants to both or all treatment
  conditions
• Within group design

8
Between and matched-subjects design
9
Steps in conducting an experiment

• State hypotheses
• Decide on an experimental design
• Decide the way to manipulate independent
  variables
• Develop a valid and reliable measure for
  dependent variable
• Pilot testing the treatment and dependent
  variable measures
• Recruit subjects (or locate cases)
• Assign subject to groups
• Introduce treatment to treatment groups
• Gather data for measure of the dependent
  variables
• Hypotheses testing

10
Mini-workshop

• Identify the IV and DV in the following
  hypotheses
• Adults find it easier to remember a list of
  meaningful words than to remember a list of
  nonsense syllables
• Nationality of salesperson will affect customers
  intention to buy a foreign made product
• Perceptions of the characteristics of the good
  or effective teacher are in part determined by
  the perceivers attitudes toward education.
• Determine how might you manipulate the IV and
  measure the DV

11
Design elements and notations

• Observations or Measures
• Symbolized by O in design notation
• Treatments or Programs
• A manipulated independent variable
• Symbolized by X in design notation
• Time
• Used to indicate the time you make the
  observation or take the measure
• Time moves from left to right. Elements that are
  listed on the left occur before elements that are
  listed on the right
• Assignment to group
• R random assignment
• N Nonequivalent groups
• C assignment by cutoff

12
Classic true experimental design

• pretest-posttest
• Treatment Versus control group
• Randomized
• Experimental design

http//trochim.human.cornell.edu/kb/desintro.htm
13
Pre-experimental design

• One-shot case study
• Result usually not valid
• Seldom used in serious research
• One-group pretest-posttest design
• Some effect after the treatment
• Can not rule out alternative explanations for the
  effect
• Static group comparison
• Any posttest outcome difference between the
  groups could be due to group differences prior to
  the experiment instead of to the treatment

14
Quasi-experimental design

• Non-equivalent control group design
• What does the term nonequivalent mean?
• Assignment to group was not random
• The groups may be different prior to the study
• Susceptible to internal validity threat of
  selection
• Selection bias groups were not comparable before
  the study

15
Time-series design

• Interrupted time-series design
• Why do we use time-series design
• Difficult to find an equivalent group of subjects
  to serve as a control group
• Single group pretest-posttest design is
  susceptible to lots of internal validity threat

16
An example

• Research question
• The impact of distance learning on university
  students learning outcome and cognitive skills
• Units of analysis college students
• Independent variables
• Distance learning (2 level)
• Dependent variables
• Learning outcome
• Cognitive skills

17
An example (cont)

• Hypotheses
• H1. Students using distance learning system will
  achieve better learning outcome than students
  with traditional classroom learning do
• H2. The cognitive skills of students using
  distance learning system will be lower than those
  of students with traditional classroom learning
• Experimental design?
• Subject recruitment?
• Manipulation of independent variable
• Measurement of dependent variables

18
Causal-comparative designs

• What can we do if we are not able to manipulate
  the independent variable
• Consider the following hypotheses
• Girls who follow science courses in Year 12 are
  more aggressive than girls following non-science
  courses.
• Working class children will learn nonsense
  syllables slower than middle-class children
• Causal-comparative designs provide the means by
  which a research can examine how specific IVs
  (personal trait, history of family violence, etc)
  affect the dependent variable (s) of interest

19
Causal-comparative designs (cont)

• Look at conditions that have already occurred and
  then collects data to investigate a possible
  relationship between these conditions and
  dependent variables of interest (ex post facto
  research)
• Difference between causal-comparative design and
  correlational design
• Less able to draw firm conclusion about cause and
  effect

20
Field and controlled laboratory experiment

• Field experiment
• Experiments conducted in real-life or field
  settings
• Researcher has less control over the experimental
  condition
• Greater external validity but lower internal
  validity
• Controlled laboratory experiment
• Conducted under controlled conditions of a
  laboratory
• Greater internal validity but lower external
  validity
• Practical consideration
• Planning and pilot testing
• Instruction to subjects
• Post experiment interview