Bayesian Analysis of the Thermal Challenge Problem

1
Bayesian Analysis of the Thermal Challenge Problem

• F. Liu1, M.J. Bayarri2, J. O. Berger1, R.
  Paulo3,
• J. Sacks4
• 1Duke, 2Valencia, 3Bristol, 4National Institute
  of Statistical Sciences
• May 22, 2006
• Sandia Laboratories
• Albuquerque, NM

2
Formulation

• Reality Code Output Bias
• Field Reality Error
• Assumption in Challenge Error 0
• i 1,, nexpt where nexpt is the number of
  observations in a particular experiment for fixed
  inputs x,L,q

3
Strategy Six Step (Statistical) Framework

• 1. Specify code inputs and associated ranges
• 2. Determine evaluation criteria
• Steps 1 and 2 define the problem
• 3. Experimental strategy
• Includes design of data collection(s)
• 4. Approximation of functions
• Necessary to deal with limited data use of
  GASP Gaussian process priors on functions
• 5. Analysis
• Comparison of field and model by Bayesian
  analysis
• 6. Feedback
• Continuous improvement of the model

4
Evaluation CriteriaRegulatory Requirement

• Criteria Is bias b 0? Is b(..., 1000) 0?
• Regulatory Requirement

5
Pure Model Prediction of Regulatory Requirement
Left for medium level pf.08Right for high
level pf.06
6
Formulation (contd)

• Simplifying Assumption
• Additional Simplification

7
Material Characterization (MC) Data

• Consider 2 levels medium, high (contains added
  data)
• Use data (only for temperatures 500) to
  estimate ?, ?
• assume ?, ? are independent with ? N(µ?s?), ?
  N(µ?s?) and non-informative priors p (µ?) 1
  and p(s?) 1/ s?and same for ?

8
Ensemble Experiment (EE) Data

• Use data to estimate bias b predict at
  accreditation configuration
• EE data are for x 0 and for the following
  design (configuration) for L, q

9
Bayes Analysis

• Bayes Theorem
• Use MCMC methods to produce a sample from ppost
  from which everything needed is obtained

10
Bayes AnalysisPrior Distributions on Unknowns

• p(b) Gaussian Process (GP) on functions of
  z(x,L,q,t)

11
Explanation

• Think of b as a random function, a realization of
  the GP
• When z z' Corr (b(z),b(z')) is high so b(z)
  b(z')
• When z far from z' Corr (b(z),b(z')) is low so
  b(z) and b(z') are not connected
• Degree of (smoothness) affected by as (a 2,
  infinitely differentiable)
• Scale determined by ßs

12
Bayes Analysis EE Data

• For EE data run MCMC for (simplified) model (2)
  and as2
• Use the MCMC samples to obtain
• Run MCMC on model (1) with fixed
• and get samples

13
ppost(b) Lt L.0127, q1000 Rt L.0127,
q2000 Medium-level Data
14
Prediction of Accreditation Data (AC) at x0

• For predicting at new setting (the AC
  configuration) from the EE analysis there is a
  new ?new, ?new with posterior distribution MC
  Data found earlier
• To the MCMC samples above adjoin
  drawn from the posterior distribution of ?new,
  ?new
• Adjoin draws from the distribution of b(0,
  L.019, q3000,) bEE, (bEE, -- bias
  functions at the four configurations of EE)

15
Prediction of AC Data at x0Lt Medium-level
Rt High-levelBlueprediction dashed80 (10
below 10 above) boundsRed Accreditation
dataGreen Plug-in pure model prediction
16
Bayesian Analysis EE AC Data

• Use x 0 data only
• Follow same strategy as for EE analysis (get
  ßs, ? first)

17
Prediction at Regulatory Requirement

• For predicting at regulatory configuration) from
  the EEAC analysis use new ?new, ?new with
  posterior distribution MC Data found earlier
• To the MCMC samples from EEAC analysis above
  adjoin drawn from the posterior
  distribution of ?new, ?new
• Adjoin draws from the distribution of b(0,
  L.019, q3500,) bEEAC, (bEEAC, -- bias
  functions at the five configurations of EEAC)
• Produce sample prediction as

18
Prediction at Regulatory Requirement blue
bias-corrected prediction with 80 boundsred
pure model prediction
19
Regulatory Results Lt medium-level pf .02
Rt high-level pf .01
20
Bayesian Analysis 2 (1)Run MCMC assuming b 0
(2) run full MCMC but sample ?i, ?i from their
posteriors from (1)
21
Prediction at Regulatory Requirement pf .03
blue bias-corrected prediction with 80
boundsgreen pure model prediction
22
Conclusions

• Model is somewhat imperfect (bias appears)
• Bias-correction suggests (marginally) that device
  may pass regulatory requirement
• Variability in ?, ? is substantial
• Constancy of ?, ? is not supportable improve the
  model

23
The Specific Tasking Questions

• The statistical strategy (SAVE Simulator
  Assessment and Validation Engine) described takes
  into account all the issues raised and provides
  the essential uncertainty bounds

24
References

• Bayarri, Berger, Higdon, Kennedy, Kottas, Paulo,
  Sacks, Cafeo, Cavendish, Lin, Tu. (2002).A
  Framework for Validation of Computer Models. In
  D. Pace, S. Stevenson, eds., Proceedings of the
  Workshop on Foundations for VV in the 21st
  Century. Society for Computer Simulation. NISS
  Technical Report 128, www.niss.org/downloadabletec
  hreports.html
• Kennedy, OHagan (2001). Bayesian Calibration of
  Computer Models. JRSS B 425-464