Linking Data Collection to Causality

1
Linking Data Collection to Causality
2
Collecting for a Causality

• When we collect data, we have varying purposes.
  Sometimes we just want to describe a population.
• Other times we want to determine whether
  variables are causally related.
• When measuring phenomena, the timing and whom we
  contact should match our objectives Describe or
  Explain.
• How does timing and whom we contact (design)
  affect ability to make causal statements?

3
Collecting for a Causality

• X Y
• Independent Dependent
• Variable Variable

Y
X
z
Z
One thing causes another when there is a)
Associationwhen X and Y change in tandem b)
Time Orderfor X to cause Y, value of X must
occur prior to value of Y c)
Nonspuriousnessrelationship between X and Y is
not coincidental or
caused by changes in a third variable (z)
4
Collecting for a Causality

• Cross-Sectional Design
• Collecting data at one point in time, using same
  instruments for everyoneobserving or asking
  questions only during a single limited
  time-frame.
• Great for descriptive work.
• Effect on Causality 1. Can establish
  association,
• 2. Time-order is hard to establish
• Answers on variables such as sex and race can be
  assumed to have pre-dated answers on other
  variables
• Answers to many variables, however, do not
  clearly precede answers to others
• We often rely on respondents memories to
  establish time order and this can be erroneous
  (why?)

5
Collecting for a Causality

• Longitudinal Designs
• Collecting data, using same instrument for
  everyone, at more than one point in
  timeobserving or asking questions across time,
  typically at discrete points.
• Works for description over time.
• Effect on Causality Depends on design, trend or
  fixed-sample design?

6
Collecting for a Causality

• Longitudinal Designs
• Repeated Cross-Sectional Designs or Trend Studies
• New sample used to collect data at each new time
  point.
• Political Polls, General Social Survey.
• Descriptive Can see change over time.
• Effect on Causality
• 1. Can establish association at distinct times.
  Cannot establish association at the individual
  level over time points. May establish macro
  level association over time points.
• 2. Like cross-sectional design at each time
  point. Cannot establish time order at the
  individual level over time points. May establish
  macro level time order over time points.

7
Collecting for a Causality

• Longitudinal Designs
• Fixed-Sample Panel Design or Panel Study
• Same sample used to collect data at each new time
  point.
• Descriptive Can see change over time.
• Effect on Causality
• 1. Can establish association at distinct times,
  at the individual level over time points, and at
  the macro level over time points.
• 2. Can establish time order at the individual
  level over time points and at the macro level
  over time points. An independent variables
  value at a previous time can be linked to a
  dependent variables value at a subsequent time.
• Very time-consuming, expensive.

8
Collecting for a Causality

• Non-Spuriousness
• Cross-sectional and longitudinal designs cannot
  establish that associations are not spurious.
  Breadth of data collectionhaving collected
  enough of the right variablesallows one to take
  into account other extraneous variables.
• Can you establish nonspuriousness with your
  papers analyses?

9
Collecting for a Causality

• Experiments
• Treating groups differently, but collecting the
  same information from them.
• True experiments have
• At least two comparison groups (experimental and
  control)
• Random assignment of subjects to comparison
  groups.
• Variation (or manipulation) in an independent
  variable before assessment of outcome on the
  dependent variable

Independent Variable
Dependent Variable
Experimental Group
Vary a condition, X
Measure Y
Random Assignment
Compare scores
Sample
Control Group
Measure Y
Do nothing, X
10
Collecting for a Causality

• Experiments
• Devised to assess causality by controlling
  everything possible while allowing for a change
  in just one variable to see how it would affect
  variables of interest in subjects.
• Control is created by randomly placing persons in
  two or more groups and treating them the same
  except
• Time-order is established by manipulating an
  independent variable between groupschanging just
  one thing for one group but not the other.
• Association is determined by observing change in
  the dependent variable after allowing only the
  independent variable to vary.
• Non-spuriousness is determined by not allowing
  anything else to vary between groups. If nothing
  else is changing, there is no extraneous variable
  influencing those of interest.
• Random assignment (NOT RANDOM SAMPLING) of
  persons to comparison groups eliminates
  possibility of systematic variation between
  groups.

11
Collecting for a Causality

• Experiments
• Sometimes, pretests are used prior to
  manipulation of the independent variable.
• This does not establish causality as much as it
  provides a baseline allowing one to determine
  just how much the dependent variable changes
• and can demonstrate similarity of comparison
  groups prior to manipulation.

Pre-Measure Y
Independent Variable
Dependent Variable
Experimental Group
Vary a condition, X
Measure Y
Random Assignment
Compare scores
Sample
Control Group
Measure Y
Do nothing, X
12
Collecting for a Causality

• Experiments
• Sometimes, matching of subjects influences
  assignment. This is so that one can guarantee
  similarity along certain dimensions across
  comparison groups.
• If using matching alone, the design is quasi
  experimental, quasi meaning something that
  appears to be something it is not
• Matching can be used with random assignment

Independent Variable
Dependent Variable
Matching
Experimental Group
Vary a condition, X
Measure Y
Random Assignment
Compare scores
Sample
Control Group
Measure Y
Do nothing, X
13
Collecting for a Causality

• Experiments are good when one can control and
  manipulate.
• Experiments are much more common in the natural
  sciences
• Sociologists rarely use experiments,
  generalizeability for complex social phenomena is
  limited
• Ethical concerns lead us to observe rather than
  control and manipulate (we just cant control the
  way wed have to)
• Control is artificial, setting up
  nonrepresentative contexts
• Observation changes the observed, especially
  among humans
• Variables of interest are more complex than can
  be represented in a controlled setting
• Subjects forming the sample are typically
  recruited, leading to nonrepresentative samples

14
Collecting for a Causality

• Quasi-Experiments
• Quasi-experiments attempt to adapt good things
  about experiments to situations where controlled
  experiments are impossible.
• Helpful if it is impossible to randomly assign
  people to groups that determine their
  experienceslike when studying real-world
  situations or interventions
• They are common in evaluation researchdetermining
  whether an intervention is effective.
• Missing typically is Random Assignment to groups.
• Technically, groups should be determined prior to
  manipulation of the independent variable or
  intervention.

Independent Variable
Dependent Variable
Experimental Group
Vary a condition, X
Measure Y
Random Assignment
Compare scores
Sample
Control Group
Measure Y
Do nothing, X
15
Collecting for a Causality

• Quasi-Experiments
• Nonequivalent control group designs A ?T-O ?
  N-S ?
• Individual Matching
• Persons are assigned to different groups in
  pairs so that experimental and control groups
  will be similar.
• Aggregate Matching
• Another group of persons that resembles the
  experimental group is selected to act as the
  control group.

matching
Independent Variable
Dependent Variable
Experimental Group
Vary a condition, X
Measure Y
Random Assignment
Compare scores
Sample
Control Group
Measure Y
Do nothing, X
16
Collecting for a Causality

• Quasi-Experiments
• Before-and-After designs A ?T-O ? N-S ?
• 1. A group acts as its own control. A pretest
  measure (the control) is compared with a posttest
  measure.
• The control group becomes the experimental group
  and is then compared with itself.
• Helpful when a control group is almost impossible
  to create or find, such as when an entire
  organization changes procedures.

Independent Variable
Dependent Variable
continue
Experimental Group
Vary a condition, X
Measure Y
Random Assignment
Compare scores
Sample
Start here
Control Group
Measure Y
Do nothing, X
17
Collecting for a Causality

• Quasi-Experiments
• Before-and-After designs
• 2. Comparing multiple groups that experience the
  same independent variable manipulation improves
  confidence in conclusions about causality.
  Repeated measurement prior to and after change in
  the independent variable provides even more
  evidence for causality and permits analysis of
  how long effects last.

Independent Variable
Dependent Variable
continue
Experimental Group
Vary a condition, X
Measure Y
Random Assignment
Compare scores
Sample
Start here
Control Group
Measure Y
Do nothing, X
18
Collecting for a Causality

• Nonexperiments
• These lack some key element of experiments such
  as lacking random assignment to groups, lacking
  matching prior to manipulation of the independent
  variable or lacking comparison groups.
• Ex Post Facto Control Group Design A ?T-O ?
  N-S ?
• The groups cannot be determined in advance, so
  there is the possibility of extraneous factors
  determining group membership.
• This is often necessary when studying events that
  have occurred or practices that are already in
  place.

Experimental Group
Vary a condition, X
Measure Y
Compare scores on Y
Find another similar group.
Control Group
Measure Y
Do nothing, X
19
Collecting for a Causality

• Factorial Surveys A ?T-O ? N-S ?
• A research bright spot where researchers
  attempt to combine generalizability of a random
  sample with random assignment to groups.
• Randomly selected participants randomly get
  treatment or no treatment in the survey,
  typically vignettes, and then dependent variable
  is measured later.
• Often survey methods are tested this way, with
  randomly selected sample being randomly surveyed
  with different techniques such as with interview,
  paper/pencil, or web-based.
• The biggest issue is typically that only
  attitudes can be measured, not particular
  behaviors.

Independent Variable
Dependent Variable
Experimental Group
Vary a condition, X
Measure Y
Random Sample
Random Assignment
Control Group
Measure Y
Do nothing, X
20
Collecting for a Causality

• A Note
• Regardless of the research method you employ, you
  should be thinking in terms of
• Association
• Time-order
• Nonspuriousness

21
Collecting for a Causality

• Some other things to consider, threats to
  determining causality and validity. Make sure
  you study these.
• Selection bias
• Differential attrition
• Endogenous Change
• Testing
• Maturation
• Regression Effect
• External Events
• Contamination
• Treatment Misidentification
• Researcher demand
• Self-fulfilling prophesies
• Placebo effect
• Hawthorne effect

22
Collecting for a Causality

• In-class Group Assignment (worth 2 Q A)
• The Fantasy Island Preservation Society has
  offered you a lot of money to do research. They
  believe that watching Fantasy Island increases
  willingness to pursue dreams.
• Your job is to devise an experiment that is
  reasonably feasible that will determine whether
  watching Fantasy Island affects pursuit of
  dreams.
• Due at the end of class!