Measurement Issues In Quantitative Research

1
Measurement Issues In Quantitative Research

• J. Stephen Guffey, Fall, 1999

2
Why Measure

• to describe the quality or quantity of an
  existing variable
• to make absolute decisions based on a criterion
  or standard of performance
• to form a basis for choosing between two courses
  of action
• to compare and discriminate between individuals
  or groups
• to draw conclusions about the predictive
  relationship between variables

3
Measurement

• The process of
• assigning numerals to objects to
• represent quantities of characteristics
• according to certain rules

4
Whats in a Number?

• Continuous Variables
• Discrete Variables
• Precision

5
Measures are Frequently Indirect

• Values represent quantities of characteristics
• accurately define
• usually measuring constructs

6
Rules of Measurement

• Nominal Scale - number to name
• Ordinal Scale - ranking
• Interval Scale - ranks with known differences
• Ratio - intervals with absolute zero, empirical
  rather than arbitrary meaning

7
Nominal

• sex (gender)
• nationality
• blood type
• clinical diagnosis

8
Ordinal

• Manual muscle test
• pain scale
• functional scale
• attitude scale

9
Interval

• Calendar years
• IQ
• temperature in F or C
• course grade

10
Ratio

• Distance
• age
• time
• weight
• strength
• blood pressure

11
Consider an example where we are
measuring the stride length of a group of
first graders. We measure with a
ruler. Subject Ratio Interval Ordinal
Nominal A 23 4 2 Long B 24 5 3
Long C 19 0 1 Short D 28 9 4 Long
12
Reliability

• Measurement Error

13
Types of Errors

• Systematic Errors
• Random Errors

14
Sources of Error

• The individual taking the measurement
• Inaccuracies in the measuring instrument
• Variability of the characteristic being measured.

15
Regression to the Mean

• Positive Error
• Negative Error
• Be careful when using measurement to classify or
  group by virtue of one measurement

16
Estimates of Reliability

• Variance is the Key Issue
• True score variance ______________________________
  _____
• true score variance error variance
• T / T E

17
Correlation v. Agreement

• A B
• T1 T2 T1 T2
• 1 5 1 1
• 2 6 2 2
• 3 7 3 3
• 4 8 4 4

18
Interpretation of Reliability Coefficients

• lt .50
• .50 - .75
• .75 - 1.00
• Turns out to be a judgment call

19
Test-Retest Reliability

• Pearson product-moment
• ICC

20
Testing Effects

• When the test itself (or something closely
  related) is responsible for observed changes in
  the variable

21
Rater Bias

• Blind the Rater
• Use only Objective Criteria

22
Test-Retest Intervals

• Stability of the variable being measured is the
  key
• Also avoid testing effects

23
Rater Reliability

• Intra-rater
• Inter-rater

24
Equivalence

• Alternate forms reliability

25
Internal Consistency

• Split-Half Reliability
• Test Item Reliability

26
Generalizability

• Population-Specific Reliability

27
Validity

• Did we measure what we thought we measured?

28
Issues of Validity

• Is the test capable of discriminating among
  individuals with or without certain traits?
• Can it evaluate change in the magnitude or
  quality of a variable from one time to another?
• Can we make useful and accurate predictions about
  a subject's potential ability or function based
  on the outcomes of the test?

29
Validity and Reliability

• They are related
• A valid test is usually reliable
• But an invalid test can also be reliable

30
Validity of Inferences

• Related
• Proportional
• Representative of the Response

31
Validity

• Not inherent to the instrument
• Evaluated in the context of the instruments
  intended purpose and specific population

32
Face Validity

• Appears to do what is says it does
• Is a reasonable way to do what is says it does
• Better do better than this

33
Content Validity

• Covers the content
• Reflects the relative importance of each part of
  the content

34
Criterion-Related Validity

• How does the target test (your instrument)
  measure up against the gold standard
• need a high correlation
• concurrent validity
• predictive validity

35
Construct Validity

• Ability to measure an abstraction or construct

36
Establishing Construct Validity

• Known Groups Method
• Multitrait-Multimethod Matrix
• Convergent validity
• Discriminant validity
• Factor Analysis
• Hypothesis Testing
• Criterion Validation

37
Screening

• Sensitivity and Specificity
• Predictive Value
• Prevalence

38
Other Types of Validity

• Cross-Validation
• Calibration