Estimating the Predictive Quality after Model Selection

1
Estimating the Predictive Quality after Model
Selection

• Chuanpu Hu, Ph.D.
• Yingwen Dong
• CP/PK Statistics, sanofi aventis
• May 3, 2006

2
Outline

• Introduction
• Use of model for prediction
• Data Perturbation
• Simulations
• Investigate influence of model selection on
  predictive qualities
• Insights on candidate models
• Evaluate performance of Data Perturbation
• Conclusions

3
Use of Modeling 2 Scenarios

• Use of model is known
• E.g., to predict the dose corresponding to a
  desired effect level ?
• MTD / MED
• Can be thought of as a parameter in the model
• Of interest standard error of parameter estimate
• Use of model is unknown
• Purpose of modeling is for general scientific
  understanding
• Of interest overall prediction error

4
Model Selection

• Predictive qualities difficult to compute
• Common practice base conclusions on final model
• Ignoring model selection process, an important
  source of uncertainty
• Known Consequences
• Underestimating standard errors and prediction
  errors
• Overly optimisitic inferences

5
Data Perturbation

• A recently proposed methodology
• Shen et al, JASA (2004)
• Allows estimation of variance of complex
  statistics, and prediction error
• Idea vairance and prediction errors relate to
  sensitivity of model prediction to small data
  changes
• Evaluation usually needs Monte Carlo
  approximation
• Variance of data must be known in order to
  perterb data
• In practice, need to use an estimate
• Nonparametric estimates are available in
  regression setting

6
Simulations Dose-Response

• Explore impact of model selection on
• Estimation bias, standard error estimation
• Of X?, the dose that would produce effect ?
• Through re-parameterization
• Prediction error
• Different scenarios
• Desired effect levels (?, or X?)
• Dose range
• Sample size
• Study performance of Data Perturbation
• Gain insight on candidiate model influence

7
Candidate Models
8
Simulation Design

• Candidate Model Schemes
• Linear, Emax (12)
• Linear, Quadratic (14)
• Linear, Emax, Quadratic (124)
• Linear, Emax, Quadratic, Linear-log (1245)
• All 7 models (1234567)
• Model selection is conducted using AICc
• Estimating doses X? corresponding to the desired
  effect levels ? 0.1, 0.5
• Parallel groups with equal sample sizes per
  treatment
• Dose range 0, 1, 0, 2, 0, 5, 0, 10
• of doses 4, 15
• Sample size per dose group 5, 20
• Total 160 scenarios

9
Simulations

• True model Emax model
• E0 0, Emax 1, ED50 1
• 2000 simulation runs for each scenario
• Bias
• Comparing estimation bias in different
  scenarios

10
(No Transcript)
11
Bias Conclusions

• True model could have larger bias
• Good models reduce bias
• Good depends on situation
• Dose range
• Parameter of interest
• Candidate model set

12
Standard Error Estimation

• Parameter of interest dose X? corresponding to
  the desired effect levels ?
• Naïve
• Based on the final selected model only
• Data perturbation
• True
• Computed from sample standard deviation of
  parameter estimates, using the selected model,
  over 2000 simulations

13
(No Transcript)
14
Standard Error Estimation Conclusions

• Naïve estimate generally under-estimate, due to
  ignorance of uncertainty
• May over-estimate due to asymptotic approximation
• Data perturbation may over-estimate, due to MC
  variability
• Advantages vs. naïve at small sample sizes
• Adding false models could decrease error
• Adding true model could increase error

15
Prediction Error Estimation

• Each scenario, 100 simulation runs
• 4 estimates calculated
• Apparent based on residuals
• Naïve Based on the final selected model only
• Data perturbation
• True the true error averaged from 100
  simulations

16
(No Transcript)
17
Prediction Error Estimation Conclusions

• Naïve estimate generally under-estimate
• Data perturbation performs well
• Model selection does not seem to have notable
  influence, perhaps due to similar complexities in
  candidate models

18
Conclusion

• Candidate model sets should include good
  models, and avoid bad ones
• Good or bad depends on intended use
• True model, even if not overly complex, can be
  bad
• False model can be good
• Data perturbation provides an approach to
  understand predictive qualities
• Standard error
• Prediction error

19
(No Transcript)
20
Backup Slides Theory
21
Notation and assumption

• Y (Y1, ... , Yn)T with Yis independently
  distributed as N(µi, s2)
• s is assumed to be known
• Estimating unknown s is a completely different
  task, which can be done separately, depending on
  specific applications.
• Methods of estimating s are available in the
  literature.
• The new method can be extended straightforwardly
  by replacing the unknown s by an estimator.

22
2

• g(Y) arbitrary statistics of Y defined by a
  known procedure
• Its analytic expression can be easy to obtain.
  For example, in linear regression, g is a vector
  of the estimated regression parameters
• Its analytic expression may be difficult or
  impossible to obtain,
• In nonlinear regression, g is a vector of the
  estimated regression parameters.
• In linear regression, g is a vector of the
  estimated regression parameters based on a
  selected model.

23
3

• Taylors Expansion
• In the situation where g(Y) is continuously
  differentiable
• This in turn yields
• where and

24
4

• General Global Linear Approximation
• In the context of model selection, g is
  nondifferentiable. Consider an approximation of
  the form
• where each ai is determined by minimizing

25
5

• Optimal a0 is
• Furthermore
• where

26
Estimation 1

• Perturbed data
• Let be an independent random variable with
  distribution following
• Define
• where is called the perturbation size.
  In practice,
• suggesting using
• Conditional distribution of given is
  

27
Estimation 2

• Optimal
• Proposed estimator of

28
Estimation 3

• Estimation for variance of g(Y)