Automated Analysis of Simulation Output Data and the AutoSimOA Project

1
Automated Analysis of Simulation Output Data and
the AutoSimOA Project

• Stewart Robinson and Katy Hoad and Ruth Davies
• Warwick Business School
• Simulation Group Seminar, 5 May 2006

2
Outline

• The problem
• An automated output Analyser
• Warm-up analysis
• Replications analysis
• Run-length analysis batch means method
• Example (demonstration)
• Discussion
• The AutoSimOA Project

3
The Problem

• Prevalence of simulation software
  easy-to-develop models and use by non-experts.

Simulation software generally have very limited
facilities for directing/advising on simulation
experiments.
Main exception is directing scenario selection
through optimisers.
With a lack of the necessary skills and support,
it is highly likely that simulation users are
using their models poorly.
4
The Problem
Despite continued theoretical developments in
simulation output analysis, little is being put
into practical use.

• There are 3 factors that seem to inhibit the
  adoption of output analysis methods
• Limited testing of methods
• Requirement for detailed statistical knowledge
• Methods generally not implemented in simulation
  software (AutoMod/AutoStat is an exception)

A solution would be to provide an automated
output Analyser.
5
An Automated Output Analyser

• For this project the Analyser looked at
• Warm-up
• Run-length
• Number of replications
• Scenario analysis could be added.

6
An Automated Output Analyser
A prototype Analyser has been developed in
Microsoft Excel. At present it links to the
SIMUL8 software, but it could be used with any
software that can be controlled from Excel VBA.

7
Warm-up Analysis

• The Analyser uses 3 procedures from which the
  user can select the desired warm-up period
  MSER-5, Batch Means Bias Detection, Welchs
  Method.
• The 3 procedures were chosen on the basis of
• Accuracy
• Reliability
• Generality
• Ease of implementation (in Excel)
• Requires minimum user intervention
• Varied (e.g. not all graphical procedures)

8
Warm-up Analysis
Adaptation of Welchs Method Smoothness
Criterion ith jump Average
jump Increase window size until average jump is
reduced to 10 of its value in the raw data.
9
Warm-up Analysis
Adaptation of Welchs Method Convergence
Criterion (average difference rule) Suppose the
moving average plot becomes smooth at observation
Xj. Then Cj should have a low value Obtain a
value for Cj such that Cj/MltL M is the difference
between the max and min Xi for igtj Tests showed
that a value of L0.0025 gave convergence close
to that chosen by visual inspection.
10
Replications Analysis
Option to run normal streams or mixed normal and
antithetic streams.
Set significance level and confidence interval
half width ().
11
Run-Length Selection Batch Means Method

• Three procedures are used for selecting the batch
  size
• Fishmans algorithm (Fishman, 1978)
• Law and Carsons algorithm (Law and Carson, 1979)
• ABATCH algorithm (Fishman and Yarberry, 1997)

12
Example
The Analyser is applied to an M/M/1 queuing model
in SIMUL8 Arrival rate 1 Service rate
0.67 Queue limit 100 Output statistic
customers in the system Demonstration!
13
Example
Run Length Batch Means example
14
Example
Run Length Batch Means example results
95 Confidence Interval Confidence Interval Confidence Interval      
Algorithm Batch mean St. dev. Lower Upper Size of half width Relative width Batch size Batches Data points used
Fishman 97.54 1.44 97.10 97.98 0.442 0.453 16 43 688
Law and Carson 97.48 1.21 96.91 98.05 0.566 0.581 30 20 600
ABATCH 97.51 0.88 97.06 97.96 0.450 0.462 48 17 816
15
Discussion
It is possible to link an Automated Analyser in
Excel to a simulation software tool.
At present this is just a proof of concept.

• Key issues to address
• More thorough testing of output analysis methods
  for their accuracy and their generality.
• Adaptation of methods to sequential procedures
  and to minimise the need for user intervention.

16
The AutoSimOA Project
A 3 year, EPSRC funded project in collaboration
with SIMUL8 Corporation.

• Objectives
• To determine the most appropriate methods for
  automating simulation output analysis
• To determine the effectiveness of the analysis
  methods
• To revise the methods where necessary in order to
  improve their effectiveness and capacity for
  automation
• To propose a procedure for automated output
  analysis of warm-up, replications and run-length
• Only looking at analysis of a single scenario

17
The AutoSimOA Project
Programme of work
Milestone Timescale (months)
Literature review of warm-up, replications and run-length methods 3
Development of artificial data sets and collection of simulation models 2
Testing of warm-up methods 6
Testing of replications methods 2
Testing of run-length methods 6
Development, testing and revision of candidate methods 6
Develop and test automated procedure (including prototype software) 5
Dissemination 6
Total 36
18
The AutoSimOA Project

• CURRENT WORK
• Literature review of warm-up, replications and
  run-length methods
• Development of artificial data sets
  (Auto-Regressive Moving average M/M/n/p
  Queues)
• and collection of real simulation models
• Produce output data
• Analyse and categorise output
  Auto-correlation Normality M.A.D..

19
Example artificial models
1. Auto-Regressive (2) series
20
(No Transcript)
21
Example artificial models
2. E4 Erlang(4) / M / 1 Queue
mean 1.8
Traffic Intensity 0.8
22
Example real models
1. Coventry Train Station Queuing Times
23
Example real models
2. Argos Queuing Times
24
Example real models
3. Tesco petrol Station Queuing Times
25
Example real models
4. Café Library Queuing Times
26
Auto Correlation
Spread round mean
In/out of control
Terminating
Group B
Trend
Non-terminating
Normality
Transient
Seasonality
Steady state