Validity, Sampling

1
Validity, Sampling Experimental Control

• Psych 231 Research Methods in Psychology

2
Announcements

• The required articles for the class experiment
  paper are already on-line at the Milner course
  reserves page

3
Class Experiment

• Collect the forms (consent forms and data summary
  sheets) - pass to the front
• Brief discussion
• So how did it go?
• What happened?
• Any thing unusual/unexpected?
• Any problems?

4
External Validity

• Are experiments real life behavioral
  situations, or does the process of control put
  too much limitation on the way things really
  work?
• Will the same basic conclusions be supported with
  different operational definitions, different
  participants, different research settings?

5
External Validity

• Variable representativeness
• relevant variables for the behavior studied along
  which the sample may vary
• Subject representativeness
• characteristics of sample and target population
  along these relevant variables
• Setting representativeness (ecological validity)
• how do the characteristics of the research
  setting compare with the real world

6
Internal Validity

• The precision of the results
• Did the change result from the changes in the DV
  or does it come from something else?

7
Threats to internal validity

• History an event happens during the experiment
• Maturation participants get older (and other
  changes)
• Selection nonrandom selection may lead to
  biases
• Mortality participants drop out or cant
  continue
• Testing being in the study actually influences
  how the participants respond
• Statistical regression regression towards the
  mean, if you select participants based on high
  (or low) scores (e.g., IQ, SAT, etc.) their
  scores later tend to move towards the mean.

8
Debugging your study

• Pilot studies
• A trial run through
• Dont plan to publish these results, just try out
  the methods
• Manipulation checks
• An attempt to directly measure whether the IV
  variable really affects the DV.
• Look for correlations with other measures of the
  desired effects.

9
Sampling

• Why do we do we use sampling methods?
• Typically dont have the resources to test
  everybody
• Population - everybody that the research results
  are targeted
• Sample - the subset of the population that
  actually participates in the research

10
Sampling

• Goals
• Maximize
• Representativeness - to what extent do the
  characteristics of those in the sample reflect
  those in the population
• Reduce
• Bias - a systematic difference between those in
  the sample and those in the population

11
Sampling Methods

• Probability sampling
• Simple random sampling
• Systematic sampling
• Stratified sampling
• Non-probability sampling
• Convenience sampling
• Quota sampling

12
Simple random sampling

• Every individual has a equal and independent
  chance of being selected from the population

13
Systematic sampling

• Selecting every nth person

14
Stratified sampling

• Step 1 Identify groups (strata)
• Step 2 randomly select from each group

15
Convenience sampling

• Use the participants who are easy to get

16
Quota sampling

• Step 1 identify the specific subgroups
• Step 2 take from each group until desired number
  of individuals

17
Experimental Control

• Our goal
• to test the possibility of a relationship between
  the variability in our IV and how that affects
  our DV.
• Control is used to minimize excessive
  variability.
• To reduce the potential of confoundings.
• if there are other variables that influence our
  DV, how do we know that the observed differences
  are due to our IV and not some other variable

18
Sources of variability (noise)

• Sources of Total (T) Variability
• T NonRandomexp NonRandomother Random

19
Sources of variability (noise)

• I. Nonrandom (NR) Variability systematic
  variation
• A. (NRexp)manipulated independent variables (IV)
• i. our hypothesis is that changes in the IV will
  result in changes in the DV
• B. (NRother)extraneous variables (EV) which
  covary with IV
• i. other variables that also vary along with the
  changes in the IV, which may in turn influence
  changes in the DV

20
Sources of variability (noise)

• II. Random (R) Variability
• A. imprecision in manipulation (IV) and/or
  measurement (DV)
• B. randomly varying extraneous variables (EV)

21
Sources of variability (noise)

• Sources of Total (T) Variability
• T NRexp NRother R
• Our goal is to reduce R and NRother so that we
  can detect NRexp.
• That is, so we can see the changes in the DV that
  are due to the changes in the independent
  variable(s).

22
Weight analogy

• Imagine the different sources of variabilility as
  weights

Treatment group
control group
23
Weight analogy

• If NRother and R are large relative to NRexp then
  detecting a difference may be difficult

24
Weight analogy

• But if we reduce the size of NRother and R
  relative to NRexp then detecting gets easier

25
Methods of Controlling Variability

• Comparison
• Production
• Constancy/Randomization

26
Methods of Controlling Variability

• Comparison
• An experiment always makes a comparison, so it
  must have at least two groups
• Sometimes there are baseline, or control groups
• This is typically the absence of the treatment
• Without control groups if is harder to see what
  is really happening in the experiment
• it is easier to be swayed by plausibility or
  inappropriate comparisons
• Sometimes there are just a range of values of the
  IV

27
Methods of Controlling Variability

• Production
• The experimenter selects the specific values of
  the Independent Variables
• (as opposed to allowing the levels to freely vary
  as in observational studies)
• Need to do this carefully
• Suppose that you dont find a difference in the
  DV across your different groups
• Is this because the IV and DV arent related?
• Or is it because your levels of IV werent
  different enough

28
Methods of Controlling Variability

• Constancy/Randomization
• If there is a variable that may be related to the
  DV that you cant (or dont want to) manipulate
• Then you should either hold it constant, or let
  it vary randomly across all of the experimental
  conditions

29
Potential Problems of Experimental Control

• Excessive random variability
• If control procedures are not applied, then R
  component of data will be excessively large, and
  may make NR undetectable
• Confounding
• If relevant EV covaries with IV, then NR
  component of data will be "significantly" large,
  and may lead to misattribution of effect to IV
• Dissimulation
• If EV which interacts with IV is held constant,
  then effect of IV is known only for that level of
  EV, and may lead to overgeneralization of IV
  effect

30
Next time

• Read Chpt 8