Structural Equation Modeling: Problems and Ambiguities with

1
Structural Equation Modeling Problems and
Ambiguities with Well Fitting Models

• Andrew Tomarken

2
Overview of Talk

• Brief introduction to structural equation
  modeling (SEM) with emphasis on core concept of
  model fit
• Review of several ambiguities and problems
  associated with well-fitting models that are
  typically ignored by users
• Conclusions
• It is important for users to bear in mind what
  precisely is being tested when assessing model
  fit
• Users need to look beyond omnibus measures of fit

3
What is SEM?

• A set of methods for estimating and testing
  models that are hypothesized to account for the
  variances and covariances (and possibly mean
  structures) among a set of variables
• Such models typically consist of sets of linear
  equations containing free, fixed, or otherwise
  constrained parameters
• Two types of linear relations can be specified
• between latent constructs (or factors) and their
  observable indicators (measurement model)
• between latent constructs (structural model)
• One way to think about it Combines simultaneous
  equation/econometric approaches and
  factor-analytic/ psychometric approaches

4
SEM as a General Statistical Approach

• Most statistical procedures conventionally used
  to test hypotheses can be considered special
  cases of SEM
• Parallels development of GLMs in 1970s and
  1980s as liberalization of classic linear models
  
• Recent development of multilevel and mixture
  modeling within SEM domain represents further
  extension of GLMs to latent continuous and
  categorical variables
• Thus SEM may arguably be most general
  data-analytic framework at present time (Tomarken
  Waller, 2005)

5
Some Advantages of SEM

• High level of explicitness Forces researchers to
  specify a model with a high level of detail
• Typically aligns the statistical null hypothesis
  with the research hypothesis
• In principle, allows for separate assessments of
  relations between observable indicators and
  latent variables (measurement model) and among
  latent variables
• Can test models that are difficult or impossible
  to test with other procedures (e.g., factor of
  curves, associative growth)
• Allows you to test the overall fit of even very
  complex models and thats the focus of todays
  talk

6
Path Analysis Model (Lynam et al., 1993)
7
The Figure Implies Linear Equations

• Figure

• Equations

• Imp a SES b TE c VIQ e1
• Del d SES e Imp e2

8
Confirmatory Factor Analysis Model
9
The Figure Implies Linear Equations

• Figure

• Equations

• Visperc Spatial e_v
• Cubes a Spatial e_c
• Lozenges b Spatial e_l
• Paragraph Verbal e_p
• Sentence c Verbal e_s
• Wordmean d Verbal e_w

10
Latent Variable Causal Model (Trull, 2001)
11
A SEM Analysis What Do We Want to Do?
12
Estimate Coefficients and Standard Errors
Estimate S.E. C.R. P Label
visperc lt--- spatial 1.000
cubes lt--- spatial .610 .143 4.250 a
lozenges lt--- spatial 1.198 .272 4.405 b
paragrap lt--- verbal 1.000
sentence lt--- verbal 1.334 .160 8.322 c
wordmean lt--- verbal 2.234 .263 8.482 d
13
Assess Overall Fit

MODEL NPAR CHI-SQUARE DF P
Correlated Factors 13 7.853 8 .448

RMSEA LO 90 HI 90 PCLOSE
Correlated Factors .000 .000 .137 .577


14
Model Comparisons

Model DF CHI-SQUARE P
Orthogonal Factors 9 19.860
Correlated Factors 8 7.853
Nested Comparison 1 12.008 .001
15
The Concept of Model Fit in SEM

• The question Does the structure implied by the
  model account for the observed variances and
  covariances among a set of variables?
• We compare the observed covariance matrix to the
  covariance matrix implied by the model
• A fitting function (F) assesses the discrepancy
  between S (sample cov. matrix) and (estimated
  population covariance matrix implied by the
  model)
• Example ML fitting function
• F or something very much like F appears in
  the formulae for all conventionally used
  statistical tests of fit and fit indices
• Estimates of free parameters are chosen that meet
  two potentially competing goals
• minimize the discrepancy between the implied and
  observed matrices
• respect the restrictions (constraints) on the
  covariance matrix implied by the model

16
Example of Model-Imposed Restrictions 3
Variable Mediational Model
17
Example of Model-Imposed Restrictions
Confirmatory Factor Model
10 knowns the 4 variances and 6 covariances
among v1-v4 8 free parameters to estimate 4
factor loadings (a-d) and the variances of the
four error terms (e1-e4) This model also implies
a set of constraints on the covariances among the
observable variables C(1,3)C(2,4) C(1,4)C(2,3)
C(1,2)C(3,4) This model will result in an
estimated or implied covariance matrix that
respects these constraints If the sample and
implied matrices agree, the model fits
18
This Should Sound Familiar

• Although the models and the specific criteria
  minimized may differ, the notion that statistical
  tests and fit indices evaluate model-imposed
  restrictions is completely consistent with
  general principles of statistical modeling,
  particularly in specific contexts (e.g., ML
  estimation)

19
How Do Users Typically Assess Overall Fit?

• Hypothesis-testing using inferential statistical
  tests
• Likelihood ratio chi-square test of exact fit
  Compares target model to a saturated
  (just-identified model)
• Nested chi-square tests for competing models
  (very important for model comparisons)
• Fit indices that indicate degree of fit
• Historically, more methodological papers on SEM
  have focused on measures of fit than any other
  topic

20
Below the Radar

• Both methodological literature and empirical
  applications heavily emphasize statistical tests
  and descriptive indices of fit
• This focus can blind users to an important point
  Even well-fitting models can have substantial
  problems and uncertainties that are often ignored
  by researchers
• Tomarken and Wallers (2003) review indicated a
  number of respects in which users ignore several
  potential problems with models that appear to fit
  well
• Ironically, these issues are not particularly
  subtle. Rather they are linked to core features
  of the concept of model fit in the SEM context

21
Potential Problems/Ambiguities with Well-Fitting
Models ---- and/or the Researchers Who Test Them

1. Lack of clarity concerning what exactly is being
   tested.
2. A poorly fitting structural (i.e., path)
   component that is masked by a well-fitting
   composite model
3. A large number of equivalent models that will
   always yield identical fit to the target model
4. Questionable lower-order components of fit
5. Omitted variables that influence constructs
   included in the model
6. The presence of a number of non-equivalent and
   non-nested alternative models that could fit
   better but are rarely ever tested
7. Low power or sensitivity to detect critical
   misspecifications
8. Specifications driven by hidden post-hoc
   modifications that lower the validity and
   replicability of the results

22
Issue 1 Do you Know What Exactly is Being
Tested?

• SEM models impose restrictions on variances and
  covariances among the observed variables (and
  sometimes on means too).
• Unfortunately
• Researchers are often unaware of the restrictions
  tested by even simple models
• Such restrictions sometimes do not reflect what
  the researcher would identify as core features of
  the model --- questions that motivated the study
  in the first place
• Many models impose so many restrictions that its
  typically impossible for even specialists to
  figure them all out or render them comprehensible
  in a more global way
• In short
• People often are unaware of what exactly is being
  assessed by statistical tests of fit or fit
  measures and what is being assessed is often
  not exactly what the researcher had in mind

23
(No Transcript)
24
(No Transcript)
25
(No Transcript)
26
(No Transcript)
27
This Does Not Mean Overall Model Fit is
Irrelevant!

• One might argue Lets just ignore fit indices
  and look at what were really interested in
• Flawed argument One would not want to test
  coefficients, estimate direct and indirect
  effects, estimate proportion of variance, etc.,
  etc in a model that does not fit well and appears
  to be mis-specified. Parameter estimates and
  standard errors will be inaccurate.
• Dont ignore fit but see it as a first step or
  necessary condition for looking at what you
  really are interested in. It is not an end in
  itself.

28
Why the Problem?

• Educational
• Perceptual/cognitive biases
• Feature-positive effect We attend more to
  presence (whats there) than to absence (whats
  not there)
• Model restrictions are usually characterized by
  absence (e.g., coefficients that are fixed at 0).
  
• Reliance on graphics and other user-friendly
  mechanisms for specifying models in software

29
(No Transcript)
30
Why the Problem?

• Educational
• Perceptual/cognitive biases
• Feature-positive effect We attend more to
  presence (whats there) than to absence (whats
  not there)
• Model restrictions are usually characterized by
  absence (e.g., coefficients that are fixed at 0).
  
• Reliance on graphics and other user-friendly
  mechanisms for specifying models in software
• Complexity of many models makes it impossible to
  catalogue all restrictions

31
(No Transcript)
32
Issue 2 A poorly fitting composite model that
masks an ill-fitting structural (path) model

• In many latent variable SEM models its important
  to distinguish between
• Measurement model Relations between manifest
  indicators and latent constructs
• Structural (path) model Relations among latent
  constructs
• Composite model The whole model that combines
  both the measurement and structural components
• Typically, in latent variable models the clear
  majority of the restrictions are imposed at the
  level of the measurement model -- and that often
  fits well
• Common result A well-fitting composite model
  that masks an ill-fitting structural component
• But the main motivation for the study typically
  is the structural component!

33
(No Transcript)
34
Chi-Square Tests of the C, M, and S Models

• Composite Global test of the composite
  model
• Measurement Global test of the measurement
  model
• Structural Nested Test assessing relative
  fit of the composite and mesurement models

35
Illustrating the Problem
Model df p RMSEA
Composite 35.46 25 .080 .0290
Measurement 24.66 24 .425 .0074
Structural 10.80 1 .001 .1402
36
Issue 3 Equivalent Models

• Two models are equivalent when their assessed fit
  across all possible samples is identical because
  they impose identical restrictions on the data
• Such models are ubiquitous in statistics
• In the context of SEM, two models are equivalent
  when their implied covariance matrices are
  identical because they impose the same
  restrictions on the variances and covariances
• If their implied covariance matrices are
  identical, then for any given sample, their
  discrepancy functions will be identical.
• If their discrepancy functions are identical, the
  values of all conventionally used fit indices
  will be identical.

37
The Problem

• The typical structural equation model has many
  equivalent models that impose the same
  restrictions on the data
• Typically, at least some are compelling
  theoretical alternatives to the target model of
  interest
• Such equivalent models are almost never
  acknowledged by researchers

38
3 Equivalent Causal Models

• These 3 models share the same restriction
  Cov(x,z)Var(y)-Cov(x,y)Cov(y,
  z) 0
• If variables are standardized, this restriction
  is rxz-rxyryz0
• All 3 models predict that the partial correlation
  between x and z, adjusting for y equals 0
• The overall fit of these 3 models will always be
  the same
• However, they represent three radically different
  claims about causal structure
• 
  

39
Three Equivalent Measurement Models
All 3 models impose the same restriction on the
implied covariance matrix
Cov(x1,x3)Cov(x2,x4)-Cov(x1,x4)Cov(x2,x3)
0
40
(No Transcript)
41
(No Transcript)
42
Recommendations

• Researchers need to acknowledge presence of
  equivalent models
• Use designs that limit number of plausible
  equivalents (e.g., one rarely noted advantage of
  longitudinal relative to cross-sectional
  designs).

43
Issue 4 Fixated on FitInattention to
Lower-order Components

• What are lower-order components ?
• Specific model parameters (e.g., path
  coefficents)
• Measures that can be derived from parameters
• Direct, indirect, and total effects
• Proportion of variance
• In most other statistical procedures that we use
  (e.g., multiple regression), the focus is on
  lower-order components
• There can be dissociations between measures of
  overall fit and lower-order components
• A model can fit perfectly, yet have problematic
  or disappointing lower-order components
• Lower-order components can indicate very strong
  effects, yet the overall fit can be terrible
• Problem Applied researchers often
  inappropriately de-emphasize lower-order
  components in favor of reliance on global fit
  indices

44
(No Transcript)
45

• Sample Covariance Matrix SB

• Sample Covariance Matrix SA

X Q Y Z
X 100
Q 20 100
Y 55 65 100
Z 65 75 80 100
X Q Y Z
X 100
Q 30 100
Y 6.5 6.5 100
Z 0.52 0.52 8.0 100
46
(No Transcript)
47
How Can a Model with Problematic Lower-order
Components Fit Well?

• Residuals are part of the model!
• Two types of residuals in SEM
• Residual matrix that is difference between
  observed and implied covariance matrices
• Residual variances and covariances (e.g.,
  variance of an endogenous variable not accounted
  for by its predictors) that are model parameters
• Residual variances
• Typically, are just-identified (impose no
  restrictions)
• Can easily fill in the difference to reproduce
  the observed variance of a variable even when
  predictors account for very small proportion of
  variance
• In essence, a weak theory can be bailed out by
  residual terms

48
Residual Covariances are Often Critical Too
49
Other Respects in Which Local Features of a Model
are Ignored

• Confidence intervals around parameter estimates
  rarely reported
• Potential problems with tests of parameters often
  ignored
• Reliance on Wald tests
• Incorrect chi-square distributions for tests at
  the boundary of the parameter space
• Often invariance across different
  parameterizations is mistakenly assumed
• Issue of assessment of fit at the level of
  individual subjects is typically ignored (e.g. no
  analysis of residuals or of individual
  contributions to fit)
• Irony In many cases, a more rigorous assessment
  of a model is afforded by a more traditional
  multiple regression approach!

50
Issue 5 Omitted Variables

• Sometimes measures of fit are sensitive to the
  problem of omitted variables (4A tested model, 4B
  true model)
• Sometimes they are not (4A tested, 4C true model)
  
• Thus, a well-fitting model could -- and typically
  does -- omit important variables

51
Omitted Variables Can Residual Covariance Terms
Bail us Out?

• By representing the omitted influences that do
  variables may share in common, residual
  covariance terms can improve model fit
• However, there are limits on the covariances that
  can be specified
• In addition, they typically do not correct for
  biased estimates due to omitted variables

52
(No Transcript)
53
(No Transcript)
54
(No Transcript)
55
Summary

• SEM is a powerful and comprehensive data-analytic
  technique
• There are a number of issues regarding
  model-imposed restrictions and assessment of fit
  that commonly operate under the radar of the
  applied user
• A well-fitting model can have substantial
  problems and ambiguities