Hypothesis Testing, Validity, and Threats to Validity

1
Hypothesis Testing, Validity,and Threats to
Validity

• Graziano and Raulin
• Research Methods Chapter 8

2
Testing Hypotheses

• Scientific research tests specific hypotheses
• Generated from the initial research idea through
  a series of steps (Figure 8.1)
• A given research idea can easily generate dozens
  of research hypotheses
• Depending on how it is translated into a
  statement of the problem and how the variables in
  the statement of the problem are operationally
  defined

3
The Initial Idea

• The initial idea is the starting point
• Often vague or general, it requires refining
  before research hypotheses can be generated
• Refinement of the initial idea is based on
• (1) a search of relevant research literature
• (2) initial observations of the phenomenon
• Want to narrow and formalize the initial idea
  into a statement of the problem

4
Statement of the Problem

• In the form of a question that clearly indicates
  an expected relationship
• The nature of the question will dictate the
  required level of constraint of a study
• Causal questions will require experimental
  research
• Questions about relationships can be answered
  with lower constraint research
• Convert into research hypotheses by operationally
  defining the variables

5
Generating Research Hypotheses

• Ideas lead to
• observations
• library research
• Statement of problem
• Problem statements become research hypotheses
  when constructs are operationalized

6
Operational Definitions

• Initially covered in Chapter 3
• The procedures used to measure and/or manipulate
  a variable
• Most variables can be operationally defined in
  many different ways, thus creating many different
  research hypotheses from a single statement of
  the problem

7
Research Hypotheses

• States clearly the expected relationship between
  the variables
• The form is a declarative statement, but it is a
  tentative statement to be tested in research
• Implicitly or explicitly, the variables in the
  research hypothesis are stated in operational
  definition terms

8
The Role of Theory

• In all research planning, theory guides the
  process
• Theory is often the primary source of the
  research hypothesis
• Theory guides the selection of variables as well
  as their operational definitions
• Most research is based on multiple, overlapping
  and interacting theories

9
Testing Research Hypotheses

• Actually testing three hypotheses
• The null hypothesis
• The confounding variable hypothesis
• The causal hypothesis
• Accept causal hypothesis only if you
• reject null hypothesis (statistical analysis)
• rule out each potential confounding variable
  hypothesis (based on appropriate controls)

10
Types of Validity

• Statistical Validity
• Are the statistical tests accurate?
• Construct Validity
• Is our theory the best explanation for the
  results?
• External Validity
• Do the results apply to the broader population?
• Internal Validity
• Is the independent variable responsible for the
  observed changes in the dependent variable?

11
Avoiding Confounding

• Confounding and internal validity
• Many sources for confounding (covered next)
• With proper controls (see Chapter 9), confounding
  can be virtually eliminated
• Confounding and construct validity
• Make sure that you have considered alternative
  theoretical explanations for the anticipated
  phenomenon

12
Confounding Variables (1/3)

• Maturation
• Changes due to normal growth or predictable
  changes
• History
• Changes due to an event that occurs during the
  study, which might have affects the results
• Testing
• Changes due to the effects of previous testing

13
Confounding Variables (2/3)

• Instrumentation
• Any change in the calibration of the measuring
  instrument over the course of the study
• Regression to the Mean
• Tendency for participants selected because of
  extreme scores to be less extreme on a retest
• Selection
• Any factor that creates groups that are not equal
  at the start of the study

14
Confounding Variables (3/3)

• Attrition
• Loss of participants during a study are the
  participants who drop out different from those
  who continue?
• Diffusion of treatment
• Changes in participants behavior in one
  condition because of information they obtained
  about the procedures in other conditions
• Sequence Effects
• Effects on performance in one condition due to
  experience with previous conditions

15
Subject Effects

• Participants are not passive
• They try to understand the study to help them to
  know what they should do
• This behavior termed subject effects
• Participants respond to subtle cues about what is
  expected (termed demand characteristics)
• Placebo effect treatment effect that is due to
  expectations that the treatment will work

16
Experimenter Effects

• Any preconceived idea of the researcher about how
  the experiment should turn out
• Affect the outcome of studies if not properly
  controlled
• May be due to the experimenter providing demand
  characteristics to the participant
• Not the same as scientific fraud (which is
  deliberate)

17
Validity, Control, Constraint

• Three closely-tied concepts
• Validity
• Refers to the scientific accuracy of the study or
  procedure
• It is increased by using appropriate control
  procedures
• The more controls we employ in a study, the
  higher the level of constraint

18
Summary

• Start by building a research hypothesis
• Testing the research hypothesis is actually
  testing three hypotheses
• (1) null (2) confounding-variable (3) causal
• Several types of validity
• Many potential confounding variables
• Subject and experimenter effects can also affect
  the outcome of the study