Scale Development

1
Scale Development

• Chapter 4 Validity

2
Overview

• Overview of Validity
• Types of Validity
• Content
• Criterion-related
• Relation to accuracy
• Construct
• Relation to criterion-related validity
• Multitrait-Multimethod Matrix
• Convergent Discriminant

3
Validity Defined

• The degree to which the variable of interest is
  the underlying cause of item covariation
• How well the scale measures a specific variable
• Three types of validity
• Content, Criterion-related, and Construct

4
Content Validity

• Content Validity
• How well a set of items reflects a content domain
• Requires a well defined domain
• Theoretically comprises a random subset of items
  from the entire pool of relevant items

5
Criterion-Related Validity

• How well the item or scale is empirically related
  to a gold standard
• Predicting a process, not understanding a process
• Often called predictive validity
• Is NOT a causal relation (even when the order
  between predictor and criterion is clear)

6
Criterion-Related Validity (continued)

• Order of predictor and criterion is irrelevant
• Sometimes called concurrent validity and less
  often called postdictive validity
• Strength of the relation is a key aspect of
  criterion-related validity

7
Criterion-Related Validity versus Accuracy

• Correlation coefficient does not measure
  classification accuracy
• Categorizing predictor and criterion into
  discrete groups
• Accuracy proportion of correct classifications
• Division of categories is important and very
  difficult
• Leads to errors ?
• Take efforts to minimize one

8
Criterion-Related Validity versus Accuracy
(continued)

• Scores on the predictor are not an estimate of
  the criterion (even with a perfect correlation
  between the two)
• May need to transform the units of the predictor
  to be equivalent to those of the criterion
• Failing to transform leads to inaccurate
  conclusions

9
Construct Validity

• Theoretical relations between the variable of
  interest and other variables
• Not to be confused with criterion-related
  validity
• Difference is in the intent of the researcher not
  in the value obtained

10
Construct Validity Its a matter of intent

• Known-groups validation
• Differentiating members of one group from another
  group based on their scores on a scale
• Construct validity theory driven
• Criterion-related prediction/identification
  driven

11
Construct Validity Proof of its existence

• Whats needed for proof of construct validity?
• There is no cut-off for the strength of the
  correlation
• Covariation between variables should be greater
  than method variance
• Relation between two variables should not be an
  artifact of covariation due to similar
  measurement methods

12
Multitrait-Multimethod Matrix

• Measure more models manipulating multiple methods
  ?
• Goal is to create a fully crossed
  method-by-measure matrix
• Can include unrelated traits

13
Convergent Discriminant Validity

• Convergent Validity
• Evidence of similarity between measures of
  theoretically related constructs
• Discriminant Validity
• Absence of a relation between measures of
  unrelated constructs
• Sometimes called Divergent validity

14
Mitchell says

• Multitrait-multimethod matrix makes the mark!
• Its a two-facet G-Study
• Traits and method facets
• Allows for more precise statements about
  construct validity
• Differentiates covariation due to construct
  similarity from covariation due to measurement
  artifacts

15
Comments on face validity

• Face validity items that assess what they
  appear to measure on their face
• The assumption can be wrong
• Is not enough to support evidence of validity
• May not want the variable being measured to be
  evident
• Problems determining to whom the scales purpose
  should be evident

16
Summary

• Three broad types of validity
• Intent of the researcher is key to distinguishing
  between types of validity
• Multitrait-Multimethod Matrix provides precision
  for construct validity