Validity

1
Validity

• Does my research method actually tell me what I
  want to know?

2
Validity

• The most important consideration when developing
  research methods is whether they actually provide
  data that addresses your research questions
  
• Do your results really tell you what you think
  they tell you?
  
• When you conclude, based on your research, that
  depictions of race on television reduce
  stereotyping among children, is it true?

3
What is validity?

• When your conclusions based on your research
  reflect whats true in the real world, they are
  valid.
  
• Researchers must produce findings that can be
  trusted and generalized to other people/texts,
  situations, and time periods.

4
How do we try to assure that the results of our
research are valid?

• We need to engage in careful explication of our
  theories.
  
• Explication is guided by scientific knowledge,
  experience and common sense
  
• We constantly update our theories and methods
  based on new information
  
• During the research process
• From one study to the next

5
Trochim identifies four components of validity

• Conclusion validity
• Internal validity
• Construct validity
• External validity

6
Conclusion validity

• Is your acceptance or rejection of the hypothesis
  correct?
  
• Type 1 and Type 2 errors
• Did you interpret your findings correctly?

7
Internal validity

• Whether or not the manipulation in the study (or
  the questions asked, etc.) actually was the
  reason for the outcomes found within the study.
  
• Were you able to either prevent or identify the
  effects of variables not included in your theory?

8
Construct validity

• Can you claim that the empirical indicators you
  used actually did closely reflect your
  constructs?
  
• Did you operationalize well the ideas of the
  cause and the effect?
  
• Did you pick a method and measures that really do
  reflect your theoretical thinking?

9
External validity

• Can you generalize the findings to other persons,
  places or times? That is, was the research done
  in a way so that your findings are not only true
  for the specific context in which you carried out
  your research?
• Dependent upon the quality of sample 

10
(No Transcript)
11
Sources of measurement variation

• There are three things that affect a
  respondent/individuals score on a measure
  
• Three components
• A true score component
• Non-systematic (random) error
• Systematic error (bias)

12
Hypothetical variance distributions
13
Variance goods and bads

• (Good) The true score variance reflects the
  variance that directly reflects the correct
  measurement of the concept under study. The
  researcher wants to maximize the portion of
  measurement variance that reflects the true score.

14
Variance goods and bads

• (Kind of bad) The nonsystematic error variance
  increases the distribution of scores around the
  true score, but does not change the estimates of
  population mean, percentages, etc.

15
Variance goods and bads

• (Very bad) Systematic error misleads the
  researcher to believe that the population mean,
  percentages, etc. are different from the true
  value, that relationships exist that really dont
  (and vice versa) and is harder to estimate than
  non-systematic error.

16
Evaluating the validity of your study

• Validity is hard to test--it is usually tested
  indirectly.
  
• Even so, researchers have developed a number of
  ways to assess validity.

17
Face validity

• Researchers apply the term face validity to the
  confidence gained from careful inspection of a
  concept to see if it is appropriate on its
  face.
• The relationship of measure to concept is
  obvious
  
• This is a particularly weak test of validity as
  everyone has different ideas about what is
  obvious (and people can be wrong)

18
Content validity

• Content validity establishes that the measure
  covers the full range of the concepts meaning.
  To determine that range of meaning, the
  researcher may solicit the opinions of experts
  and review literature that identifies the
  different aspects, or dimensions, of the concept.

19
Criterion Validity

• Criterion validity is established when the
  scores obtained on one measure can be accurately
  compared to those obtained with a more direct or
  already validated measure of the same phenomenon
  (the criterion).

20
Two forms of criterion validity

• Concurrent validity
• Concurrent validity exists when a measure yields
  scores that are closely related to scores on a
  criterion measured at the same time.
  
• Measure of sales ability given to sales force and
  compared to actual sales performance.
  
• Predictive validity
• Predictive validity is the ability of a measure
  to predict scores on a criterion measured in the
  future.
  
• Measure of sales ability given to new sales
  recruits compared to sales performance two years
  later.

21
Construct validity

• Construct validity refers to the degree to which
  inferences can legitimately be made from the
  operationalizations in your study to the
  theoretical constructs on which those
  operationalizations were based. (Trochim)
• Are you really measuring what you think you are
  measuring?

22
Two types of construct validity

• Convergent validity is achieved when one measure
  of a concept is associated with different types
  of measures of the same concept
  
• Discriminant validity In this approach, scores
  on the measure to be validated are compared to
  scores on measures of different but related
  concepts.
• Should find moderate correlations with measures
  of related but not identical concepts

23
(No Transcript)
24
Telcom example

• Transportation
• Relation to enjoyment (r.60 r.77)
• Relation to absorption (r.24 r
• Relation to dissassociative oblivion (r.23)

25
External validity

• Trochim names two approaches to external
  validity
  
• Sampling approach
• If you have a good, random sample, you can
  automatically generalize
  
• Samples are never perfect (may not have a real
  understanding of population, cannot sample across
  all time periods you want to generalize to)
  

26

• Proximal similarity model
• The more the sample resembles the population you
  want to generalize to, the more valid your claims
  to be able to take results as indicative of
  population behavior
• Gradients of similarity
• Three threatspeople, places, and times

27
Triangulating validity

• Attempt to demonstrate multiple types of validity
  for measures

28
Humor orientation scale

• 1strongly agree, 2agree, 3neutral, 4disagree,
  5strongly disagree
  
• 1. I regularly tell jokes and funny stories when
  I am in a group
  
• 2. People usually laugh when I tell a joke or
  story
  
• 3. I have no memory for jokes or funny stories
• 4. I can be funny without having to rehearse a
  joke
  
• 5. Being funny is a natural communication style
  with me
  
• 6. I cannot tell a joke well
• 7. People seldom ask me to tell stories
• 8. My friends would say that I am a funny person
• 9. People dont seem to pay close attention when
  I tell a joke
  
• 10. Even funny jokes seem flat when I tell them