Method Variance

1
Method Variance

• Spector (1987)
• Williams et al. (1989)
• Bagozzi et al. (1990)
• Tepper Tepper (1993)

2
What is Method Variance?

• Variance attributable to the measurement method
• Method Variance biases results when correlating
  different constructs that used the same
  measurement method

3
How big a problem is it?

• Common bias sources due to measurement method can
  be correlated to produce incorrect results
• May artificially inflate or suppress correlations
  between two variables
• I/O adage all self-report measures
  intercorrelate at .30

4
Examples of Method Variance

• Acquiescence
• Tendency for respondent to agree with items
  regardless of content
• Most common in ambiguous test items and poorly
  designed tests
• Social Desirability
• Tendency to report socially desirable answers
• Is sometimes used as a personality construct in
  its own right

5
Spector(1987)

• Analyzed ten published studies with multi-method
  designs and strong correlations between methods
  for each construct
• Procedure (developed by Campbell and Fiske, 1959)
• Correlate between traits using same method
• Correlate between traits using different methods
• If there is method bias, first correlation will
  be significantly larger

6
Spector(1987)

• Did not find significant differences attributable
  to method variance
• Social desirability was also correlated and had
  minimal effect
• Acquiescence was examined by comparing responses
  to positively and negatively worded items. The
  effect was small to nonexistant

7
Williams et al. (1989)

• Limitations of the statistical analysis Spector
  used
• Inability to account for differential reliability
• Implicit assumptions
• assumption of uncorrelated and maximally
  dissimilar methods

8
Williams et al. (1989)

• Reanalysis of Spectors data using confirmatory
  factor analysis (CFA)
• Tested five models with each data sample
• M1 null model (no correlation)
• M2 trait model (no method variance)
• M3 method model (no trait variance)
• M4 trait and method model with no correlation
  between method factors
• M5 trait and method model with correlated method
  factors

9
Williams et al. (1989)

• Results
• Comparing M1 to M3, M3 was significantly better
  based on chi-square difference, and accounted for
  a large amount of the correlation based on the
  normed-fit index (NFI)
• Comparing M2 to M4, M4 was significantly better
  in 9 studies based on chi-square difference, and
  accounted for a large amount of the correlation
  based on the NFI

10
Williams et al. (1989)

• Results (cont)
• Comparing M4 to M5, M5 was significantly better
  in 7 studies based on chi-square difference, but
  did not account for much difference in fit

11
Williams et al. (1989)

• Conclusions
• Based on CFA, method factors play a large role in
  the constructs analyzed by Spector
• There is also evidence that the method factors
  are not independent, as required by the Campbell
  and Fiske analysis procedure

12
Bagozzi et al. (1990)

• Limitations of Williams et al. analysis
• Tests examined only overall effects of method
  factors
• No information for conclusions about individual
  tests
• Ignored other chi-square indicators for
  additional model information
• Did not include possibility of trait/model
  interaction

13
Bagozzi et al. (1990)

• Reanalysis of the Spector data
• Used traits-only model as a baseline against the
  traits and methods model
• 10 of 11 data sets fit the model for traits and
  methods
• Examined statistical significance of method
  factors on individual measures
• Five data sets had no significant method factor
  loads on any individual measures, 4 had
  significant effects on some measures, and 2 had
  significant effects on all measures

14
Bagozzi et al. (1990)

• Used additional criteria to evaluate the trait
  and method model
• 9 of the 11 data sets fit the model without large
  unexplained correlations
• 2 of the data sets were not suited for the trait
  and method model

15
Bagozzi et al. (1990)

• Examined the data sets to see if an alternative
  model, the direct-product model, would fit better
  than the CFA
• One data set fit the alternate model, that did
  not fit the CFA model
• None of the other data sets fit the DPA model

16
Bagozzi et al. (1990)

• Conclusions
• The CFA model, but not DPA, fits 9 of the data
  sets
• The DPA model, but not the CFA, fit one data set
• One data set did not fit either model
• Two data sets would have been miscategorized if
  only chi-squared goodness of fit tests were used

17
Tepper Tepper (1993)

• Method variance within measures Covariance among
  items on the same measurement due to item format
• Inflates interitem correlations
• Inflates reliability estimates

18
Tepper Tepper (1993)

• Common measurement guidelines encourage practices
  that increase method variance within measures
• Simplifying questionnaires by using homogeneous
  item formats and scales
• Examining test-retest reliability by examining
  consistency in responses across repeated items
  within the test

19
Tepper Tepper (1993)

• Scale development suggestions
• Heterogeneous item formats
• Placing dissimilar items together, rather than
  grouping items by content and format
• Random placement of dummy items that capture
  irrelevent content
• Skip back and forth in questionnaire to break up
  respondents concentration
• Spread the questionnaire administration out over
  several sittings