Conclusion Validity

1
Conclusion Validity
Conclusion validity is the degree to which
conclusions we reach about relationships in our
data are reasonable
2
Conclusion Validity

• Conclusion validity is the degree to which the
  conclusion we reach is credible or believable
• A statistically significant result
• OR
• A nonsignificant result
• Statistically labeled - Often Misunderstood,
  Least Considered
• Relevance in Qualitative Research as well as
  Quantitative Research

3
Conclusion Validity
Quantitative Example An inventor has developed a
new, energy-efficient lawn mower engine. The
inventor claims that the engine will run
continuously for 5 hours (300 minutes) on a
single gallon of regular gasoline Qualitative
Example Accountability as practices in our
primary health care system creates an undesirable
atmosphere of anxiety among nurses
4
Validity- How they differ?
Internal Validity
Conclusion Validity
How Credible?
Program
Cause-Effect
Relation
What you do What you See
Accept / Reject
External Validity
Construct Validity
5
Threats to Conclusion Validity

• The primary threat to conclusion validity is the
  possibility of making an error in the inference
  process concerning the relationship between the
  program and the outcome(s) of the program
• Error Types
• Type I error Conclude there is a relationship
  when in fact there is not (concluding/seeing an
  effect that in reality is not there)
• Type II error Conclude there is no relationship
  when in fact there is a relationship (miss a
  true effect)

6
Fishing the error rate problem
Looking for a specific result by analyzing the
data repeatedly under slightly differing
conditions or assumptions Conducting multiple
analyses and treating each one as though it was
independent without error rate adjustment Likely
to see a relationship when there isn't one when
you keep reanalyzing your data and don't take
that fishing into account when drawing your
conclusions
7
Signal-to-noise ratio problem
"noise" consists of factors that make it hard to
see the relationship

• low reliability of measures
• poor reliability of treatment implementation
• random irrelevancies in the setting
• random heterogeneity of respondents

signal" amount of information collected and the
amount of risk taken for decision

• low statistical power

8
Problems that can lead to either conclusion error
Assumptions behind analysis when violated -
likely to draw erroneous conclusions about
relationships Quantitative Research
Example Assumption that data is normally
distributed is violated Quantitative Research
Example Assumption that the respondent is free
to say anything but under pressure from
supervisors respond in a particular to
9
Improving Conclusion Validity
Good Statistical Power Sample size Collect
more information -- use a larger sample size
Effect size Improve the impact of
the program relative to the noise
Alpha Level Increase your risk of
making a Type I error Power
Ability to see effect thats there Good
Reliability Consistency and
Repeatability for measures, reducing situational
distractions in the measurement context Good
Implementation Training program
operators and standardizing the protocols for
administering the program
10
The Four Components to a Statistical Conclusion

• Amount of information
• Impact of program
• Willingness to risk being wrong in
• finding an effect (rejecting the null hypothesis)
• Ability to see effect thats there

Sample size
Effect size
Alpha level
Power
11
Given Values for Any Three, Possible to Compute
the Fourth

• n f(effect size, a, power)
• effect size f(n, a, power)
• a f(n, effect size, power)
• power f(n, effect size, a)

12
Statistical Conclusions

• Statistical conclusions involve constructing two
  mutually
• exclusive hypotheses, termed the null (labeled
  H0) and
• alternative (labeled H1)
• H0 Program Effect 0
• H1 Program Effect ltgt 0

13
The Decision Matrix
14
The Decision Matrix
H0 (null hypothesis) true
Alternative H1 false
In reality...

• There is no real program effect
• There is no difference, gain
• Our theory is wrong

15
The Decision Matrix
H0 (null hypothesis) true
Alternative H1 false
In reality...

• There is no real program effect
• There is no difference, gain
• Our theory is wrong

Accept null
Reject alternative
We say...

• There is no real program effect
• There is no difference, gain
• Our theory is wrong

16
The Decision Matrix
H0 (null hypothesis) true
Alternative H1 false
In reality...

• There is no real program effect
• There is no difference, gain
• Our theory is wrong

Reject null
Accept alternative
We say...

• There is a real program effect
• There is a difference, gain
• Our theory is correct

17
The Decision Matrix
H0 (null hypothesis) true
Alternative H1 false
In reality...

• There is no real program effect
• There is no difference, gain
• Our theory is wrong

Reject null
?
Accept alternative
TYPE I ERROR
We say...
The odds of saying there is an effect or gain
when in fact there is none

• There is a real program effect
• There is a difference, gain
• Our theory is correct

of times out of 100 when there is no effect,
well say there is one
18
The Decision Matrix
H0 (null hypothesis) false
Alternative H1 true
In reality...

• There is a real program effect
• There is a difference, gain
• Our theory is correct

19
The Decision Matrix
H0 (null hypothesis) false
Alternative H1 true
In reality...

• There is a real program effect
• There is a difference, gain
• Our theory is correct

Accept null
Reject alternative
We say...

• There is no real program effect
• There is no difference, gain
• Our theory is wrong

20
The Decision Matrix
H0 (null hypothesis) false
Alternative H1 true
In reality...

• There is a real program effect
• There is a difference, gain
• Our theory is correct

Accept null
?
Reject alternative
TYPE II ERROR
We say...
The odds of saying there is no effect or gain
when in fact there is one

• There is no real program effect
• There is no difference, gain
• Our theory is wrong

of times out of 100 when there is an effect,
well say there is none
21
The Decision Matrix
H0 (null hypothesis) false
Alternative H1 true
In reality...

• There is a real program effect
• There is a difference, gain
• Our theory is correct

Reject null
Accept alternative
We say...

• There is a real program effect
• There is a difference, gain
• Our theory is correct

22
The Decision Matrix
H0 (null hypothesis) false
Alternative H1 true
In reality...

• There is a real program effect
• There is a difference, gain
• Our theory is correct

Reject null
1-?
Accept alternative
POWER
We say...
The odds of saying there is an effect or gain
when in fact there is one

• There is a real program effect
• There is a difference, gain
• Our theory is correct

of times out of 100 when there is an effect,
well say there is one
23
The Decision Matrix
In reality
H0 (null hypothesis) false
H0 (null hypothesis) true
Alternative H1 true
Alternative H1 false
In reality...
In reality...
What we conclude

• There is no real program effect
• There is no difference, gain
• Our theory is wrong

• There is a real program effect
• There is a difference, gain
• Our theory is correct

x
Accept null
1-?
?
Reject alternative
THE CONFIDENCE LEVEL
TYPE II ERROR
We say...
The odds of saying there is no effect or gain
when in fact there is none
The odds of saying there is no effect or gain
when in fact there is one

• There is no real program effect
• There is no difference, gain
• Our theory is wrong

of times out of 100 when there is no effect,
well say there is none
of times out of 100 when there is an effect,
well say there is none
Reject null
?
1-?
Accept alternative
TYPE I ERROR
POWER
We say...
The odds of saying there is an effect or gain
when in fact there is none
The odds of saying there is an effect or gain
when in fact there is one

• There is a real program effect
• There is a difference, gain
• Our theory is correct

of times out of 100 when there is no effect,
well say there is one
of times out of 100 when there is an effect,
well say there is one
24
The Decision Matrix
H0 (null hypothesis) true
H0 (null hypothesis) false
Alternative H1 true
Alternative H1 false
In reality...
In reality...

• There is no real program effect
• There is no difference, gain
• Our theory is wrong

• There is a real program effect
• There is a difference, gain
• Our theory is correct

Accept null
1-?
?
Reject alternative
THE CONFIDENCE LEVEL
TYPE II ERROR
We say...

• There is no real program effect
• There is no difference, gain
• Our theory is wrong

Reject null
?
1-?
Accept alternative
TYPE I ERROR
POWER
We say...

• There is a real program effect
• There is a difference, gain
• Our theory is correct

25
The Decision Matrix
H0 (null hypothesis) true
H0 (null hypothesis) false
Alternative H1 true
Alternative H1 false
In reality...
In reality...

• There is no real program effect
• There is no difference, gain
• Our theory is wrong

• There is a real program effect
• There is a difference, gain
• Our theory is correct

Accept null
1-?
?
Reject alternative
THE CONFIDENCE LEVEL
TYPE II ERROR
We say...
CORRECT

• There is no real program effect
• There is no difference, gain
• Our theory is wrong

Reject null
?
1-?
Accept alternative
TYPE I ERROR
POWER
We say...
CORRECT

• There is a real program effect
• There is a difference, gain
• Our theory is correct

26

• Questions ?