Model Diagnostics

1
Model Diagnostics
2
Model-Data Agreement

• Model represents how data were generated
• Important to check model-data agreement
• Support model assumptions?
• Should model components be refined?
• Remove regressors
• Alter covariance structure
• Sensitivity of results to model and/or data?

3
Classical Linear Models

• Residual analysis
• Residual plots
• Histograms / QQplots
• Goodness of fit
• Likelihood ratio test
• AIC / BIC
• Influence analysis
• Cooks D
• DFFITS

4
Influence

• The ability of a single or multiple data points,
  through their presence or absence, to
• alter important aspects of the analysis
• yield qualitatively different inferences
• violate model assumptions
• Want to determine these points to hopefully
  develop a better model

5
Linear Mixed Model

• Y is nx1 vector of observations
• X is an nxp fixed effects design matrix
• Z is an nxg random effects design matrix

6
Mixed vs Fixed Model

• Both and depend on covariance parameter
  estimates
• Consider mixed in both conditional and
  unconditional sense
• conditional, particular values of
  random effects
• unconditional, average over
  random effects
• Observation might influence fixed effects given
  covariance parameters, covariance parameters, or
  both

7
Mixed vs Fixed Model

• Often used to analyze repeated measures
• Influence of single observation?
• Influence of set of observations from same EU
• Cannot determine effect of deleted observation
  without refitting the model unless you assume
  covariance unaffected
• Traditional OLS summaries can give
  counter-intuitive results in mixed

8
Residuals

• Marginal Residual
• Conditional Residual
• MIXED produces these with the OUTP and OUTPM
  option

9
Standardized Residuals

• Starting in SAS 9.1, will have options to also
  obtain studentized, Pearson, and scaled
  residuals.

10
Influence Diagnostics

• Basic procedure
• Fit model and obtain estimates
• Remove one or more data points and recompute
  estimates
• May not be possible (G not positive definite)
• Contrast quantities of interest to see which
  observations are influential

11
Global Influence Summary

• Likelihood distance (or restricted likelihood)
• Measures overall change in fit
• Likelihood based on all the data, not subset
• If global measure suggests influence, must
  determine the nature of influence

12
State of Influence

• Change in fixed parameter estimates
• SAS 9.1 provides modified versions of Cooks D
  and multivariate DFFITS
• Change in parameter estimate precision
• SAS 9.1 provides summaries of the trace and
  determinants of the variance matrices
• Change in predicted values
• SAS 9.1 provides the PRESS residual and DFFITS

13
State of Influence

• Outlier Detection
• SAS 9.1 and studentized residuals within 2
• Leverage
• SAS 9.1 provides two modified Hat Matrix
• Iterative analysis approach can be very time
  consuming
• Can use full parameter values as starting pts

14
Reference

• Schabenberger, O. (2004) Mixed Model Influence
  Diagnostics, SUGI 29 Statistics and Data
  Analysis, Paper 189-29.