Conceptualization, Operationalization, and Measurement: Measurement Quality

1
Conceptualization, Operationalization, and
MeasurementMeasurement Quality

• November 1, 2004
• Sociology 487

2
Lecture Topics

• Reliability
• What is reliability?
• How do we assess reliability?
• Validity
• What is validity?
• What are the 4 ways in which we assess validity?

3
Measurement Components
4
Reliability

• Measurement procedures result in consistent and
  stable scores / responses
• Pre-requisite for measurement validity

5
Ways to Assess Reliability

• Test-Retest Reliability
• Interitem Reliability
• Alternate-Forms Reliability
• Interobserver Reliability

6
Test-Retest Reliability

• Take the same measurement more than once
• If answers are the same AND phenomena has not
  changed, measurement is reliable
• Interobserver reliability
• Assessing extent to which observer ratings at 2
  time points are similar

7
Interitem Reliability

• Association of answers to a set of questions
  designed to measure same concept
• Cronbachs alpha is the statistic used to measure
  interitem reliability
• How strong is the association among the
  individual questions

8
Example of Interitem Reliability
9
Alternate-Forms Reliability

• Compare respondents answers to slightly
  different versions of survey question
• Split-half method
• Divide sample into two subsamples and administer
  slightly different version of question to each
  subsample
• Correspondence between subsamples used to assess
  reliability

10
Interobserver Reliability

• Correspondence between measures made by different
  observers

11
Validity

• Extent to which empirical measures reflect
  meaning of concept
• Four ways to assess validity
• Face validity
• Content validity
• Criterion Validity
• Construct Validity

12
Face Validity

• Determine if a measure is appropriate on its
  face
• Congruence with common-sense notions of concept
  

13
Content Validity

• Measure encompasses full range of meaning of
  concept
• Example Measure of mathematics ability would
  need to cover addition, subtraction,
  multiplication, division

14
Criterion Validity

• Results from one measure of a concept match
  results obtained from more direct measure of the
  same concept
• Concurrent validity
• Criterion measure and measure being validated
  administered at same time
• Predictive validity
• Measure being validated administered prior to
  criterion measure

15
Construct Validity

• Measure is related to other measures as specified
  by theory and literature
• Convergent validity
• One measure of a concept is associated with
  different measures of the same concept
• Discriminant validity
• Measure being validated is NOT strongly
  associated with measures of related but DIFFERENT
  concepts

16
Relationship Between Reliability and Validity
Adapted from Babbie, 2004
17
Improving Reliability and Measurement Validity

• Use previously tested measures
• Use multiple measures
• When creating new measures
• Use focus groups
• Conduct cognitive interviews
• Pre-test and test again

18
Key Points

• Measures can be reliable but not valid
• Measurement reliability is necessary to have
  measurement validity
• Measurement validity is the necessary starting
  point for social research
• Appropriate conceptualization and
  operationalization helps us obtain valid measures