The AutoSimOA Project

1
The AutoSimOA Project
AUTOMATING D.E.S OUTPUT ANALYSIS
HOW MANY REPLICATIONS TO RUN
A 3 year, EPSRC funded project in collaboration
with SIMUL8 Corporation. http//www.wbs.ac.uk/go/
autosimoa

• Katy Hoad, Stewart Robinson, Ruth Davies
• Warwick Business School
• WSC 07

2

• Objective
• To provide an easy to use method, that can be
  incorporated into existing simulation software,
  that enables practitioners to obtain results of a
  specified accuracy from their discrete event
  simulation model.
• (Only looking at analysis of a single scenario)

3
OUTLINE Introduction Methods in literature Our
Algorithm Test Methodology Results Discussion
Summary
4

• Underlying Assumptions
• Any warm-up problems already dealt with.
• Run length (m) decided upon.
• Modeller decided to use multiple replications
  to obtain better estimate of mean performance.

5
QUESTION IS

• How many replications are needed?
• Limiting factors computing time and expense.
• 4 main methods found in the literature for
  choosing the number of replications N to perform.

6

• Rule of Thumb (Law McComas 1990)
• Run at least 3 to 5 replications.
• Advantage Very simple.
• Disadvantage Does not use characteristics of
  model output.
• No measured precision level.

7
2. Simple Graphical Method (Robinson 2004)
Advantages Simple Uses output of
interest in decision. Disadvantages
Subjective No measured precision level.
8
3. Confidence Interval Method (Robinson 2004,
Law 2007, Banks et al. 2005).

• Advantages Uses statistical inference to
  determine N.
• Uses output of interest in decision.
• Provides specified precision.
• Disadvantage Many simulation users do not have
  the skills to apply approach.

9
4. Prediction Formula (Banks et al. 2005)

• Decide size of error e that can be can tolerated.
• Run 2 replications - estimate variance s2.
• Solve to predict N.
• Check desired precision achieved if not
  recalculate N with new estimate of variance.
• Advantages Uses statistical inference to
  determine N.
• Uses output of interest in decision.
• Provides specified precision.
• Disadvantage Can be very inaccurate especially
  for small number of replications.

10
AUTOMATE Confidence Interval Method Algorithm
interacts with simulation model sequentially.
11
ALGORITHM DEFINITIONS
12

• Stopping Criteria
• Simplest method
• Stop when dn 1st found to be desired
  precision, drequired . Recommend that number of
  replications, Nsol, to user.
• Problem Data series could prematurely converge,
  by chance, to incorrect estimate of the mean,
  with precision drequired , then diverge again.
• Look-ahead procedure When dn 1st found to be
  drequired, algorithm performs set number of
  extra replications, to check that precision
  remains drequired.

13
Look-ahead procedure kLimit look ahead
value. Actual number of replications checked
ahead is Relates look ahead period length
with current value of n.
14
Replication Algorithm
95 confidence limits
Precision 5
Cumulative mean,
f(kLimit)
Nsol f(kLimit)
Nsol
15
Precision 5
Precision gt 5
Precision 5
f(kLimit)
Nsol2 f(kLimit)
Nsol2
Nsol1
16
TESTING METHODOLOGY

• 24 artificial data sets
• Left skewed, symmetric, right skewed
• Varying values of relative st.dev (st.dev/mean).
• 100 sequences of 2000 data values.
• 8 real models selected.
• Different lengths of look ahead period tested
• kLimit values 0 (i.e. no look ahead), 5, 10,
  25.
• drequired value kept constant at 5.

17

• 5 performance measures
• Coverage of the true mean
• Bias
• Absolute Bias
• Average Nsol value
• Comparison of 4. with Theoretical Nsol value
• For real models true mean variance values -
  estimated from whole sets of output data
  (3000 to 11000 data points).

18
Results

• Nsol values for individual algorithm runs are
  very variable.
• Average Nsol values for 100 runs per model close
  to the theoretical values of Nsol.
• Normality assumption appears robust.
• Using a look ahead period improves performance
  of the algorithm.

19
Mean bias significantly different to zero Failed in coverage of true mean Mean est. Nsol significantly different to theoretical Nsol (gt3)
No look-ahead period Proportion of Artificial models 4/24 2/24 9/18
No look-ahead period Proportion of Real models 1/8 1/8 3/5
kLimit 5 Proportion of Artificial models 1/24 0 1/18
kLimit 5 Proportion of Real models 0 0 0
20
Impact of different look ahead periods on
performance of algorithm
decrease in absolute mean bias decrease in absolute mean bias decrease in absolute mean bias
kLimit 0 to kLimit 5 kLimit 5 to kLimit 10 kLimit 10 to kLimit 25
Artificial Models 8.76 0.07 0.26
Real Models 10.45 0.14 0.33
21
Number of times the Nsol value changes (out of 100 runs of the algorithm per model) because of the lengthening of the look ahead period. Number of times the Nsol value changes (out of 100 runs of the algorithm per model) because of the lengthening of the look ahead period. Number of times the Nsol value changes (out of 100 runs of the algorithm per model) because of the lengthening of the look ahead period. Number of times the Nsol value changes (out of 100 runs of the algorithm per model) because of the lengthening of the look ahead period.
Model ID kLimit 0 to kLimit 5 kLimit 5 to kLimit 10 kLimit 10 to kLimit 25
R1 0 0 0
R3 2 0 0
R5 24 0 1
R8 24 4 1
A5 30 1 3
A6 26 6 3
A15 1 0 0
A17 22 0 1
A21 25 2 1
A24 37 0 0
22
Examples of changes in Nsol improvement in
estimate of true mean
Model ID kLimit Nsol Theoretical Nsol (approx) Mean estimate significantly different to the true mean?
A9 0 4 112 Yes
  5 120 112 No
A24 0 3 755 Yes
  5 718 755 No
R7 0 3 10 Yes
  5 8 10 No
R4 0 3 6 Yes
5 7 6 No
R8 0 3 45 Yes
  5 46 45 No
23
DISCUSSION

• kLimit default value set to 5.
• Initial number of replications set to 3.
• Multiple response variables - Algorithm run with
  each response - use maximum estimated value for
  Nsol.
• Different scenarios - advisable to repeat
  algorithm every few scenarios to check that
  precision has not degraded significantly.
• Implementation into Simul8 simulation package.

24
SUMMARY

• Selection and automation of Confidence Interval
  Method for estimating the number of replications
  to be run in a simulation.
• Algorithm created with look ahead period
  -efficient and performs well on wide selection of
  artificial and real model output.
• Black box - fully automated and does not
  require user intervention.

25
ACKNOWLEDGMENTSThis work is part of the
Automating Simulation Output Analysis (AutoSimOA)
project (http//www.wbs.ac.uk/go/autosimoa) that
is funded by the UK Engineering and Physical
Sciences Research Council (EP/D033640/1). The
work is being carried out in collaboration with
SIMUL8 Corporation, who are also providing
sponsorship for the project.
Thank you for listening.

• Katy Hoad, Stewart Robinson, Ruth Davies
• Warwick Business School
• WSC 07