V' Simulation Management and Output Analysis

1
V. Simulation Management and Output Analysis

• M. Peter Jurkat
• CS452/Mgt532 Simulation for Managerial Decisions
• The Robert O. Anderson Schools of Management
• University of New Mexico

2
Simulation Management

• Simulation output depends on
• Model being simulation
• Parameter values (see DOE next slide set)
• Initial conditions, including random number
  generator(s) and seed(s)
• Terminating conditions
• Number of replications

3
Stochastic Process Simulation

• Each stochastic variable (e.g., time between
  arrivals) is generated by a stream of random
  numbers from a LCG beginning with a particular
  seed value gt need multiple runs and statistical
  analysis for stochastic models (i.e., sampling)
• The same LCG with the same seed, x0, will always
  generate the same sequence of pseudo-random
  numbers gt repeated simulation with the same
  inputs and the same seed will result in identical
  outputs
• For independent replications need a different
  seed for each replication (F9 in Excel will do
  that)

4
Simulation Sampling

• Each simulation run may yield only one value (a
  single sample value) of each DV (e.g., average
  waiting time, proportion of time server is idle)
• Need replications to gain statistical stability
  and significance in output distribution
• Repeated runs are akin to a sample from a
  population of all such runs with a given set of
  exogenous variables stochastic inputs cause the
  variation need good data for distribution (at
  least mean and variance) of stochastic variables

5
Simulation Replications

• Two approaches
• Independent replications/samples different
  seeds/ random number generators for random
  variable experimental variation (EV/DOE) for
  others
• Good news standard statistics based on
  independent random samples apply
• Bad news differences may be due to different
  random number
• Generators may mask lack of differences in system
  avoid with sufficient sample size and
  statistical tests
• Correlated replications same seeds/random number
  generators for each random variable EV/DOE
• Good news differences must be due to model
  EV/DOE
• Bad news need to use special statistical
  procedures for analysis (e.g., stat procedures
  which require independent samples not applicable)
  may not actually need statistics

6
Collecting Output Measures(usual 7 and others)

• During simulation
• many values for a given DV per run (e.g., Yi
  in Q)
• Compare at same time between various runs
• At end of simulation
• one value per run (e.g., LQ time average of Yi
  in Q)
• Time of termination, TE
• Maximum, minimum of Yi and/or Y(t)
• Simple average of Yi over one run for discrete
  measures possibly not independent samples due
  to serial correlation can test
• Time-Averages of Y(t) for continuous measures
• Quantiles value for which given proportion of
  Yi and/or Y(t) values are lower or higher

7
Terminating Simulations

• Seek description of one system instance
  termination at TE due to
• Schedule then analyze other measures
• System failure time of termination (i.e., time
  of system failure) is often desired output
  measure
• Multiple replications yield distribution of
  expected time to failure and time to failure
  distribution
• Other output measures gathered during simulation
  may indicated reasons for failure

8
Non-Terminating

• Seek steady-state behavior
• Often difficult to achieve how recognized under
  stochastic input?
• Stability of measure or its moving
  average/derivative
• Initialization bias (chaos?) avoid by
• Use near steady state conditions for initial
  conditions often unknown!!!
• Analytic solution of simpler system
• Analyze by batching/lag analysis/deleting batches
  at beginning of run see Banks et al, Table 11.6
• Usually needs many, long, independent
  replications with different starting parameters
  can lead to chaos for non-linear systems

9
Serial Correlation

• Positive correlation (serious!) between values
  gathered during simulation arises from taking
  data at intervals short compared to systems
  natural frequency negative correlation (often
  benign) from intervals long compared to system
  natural frequency
• Positive serial correlation can lead to false
  assessment of steady state (Banks et al, Figure
  11.7)
• Can determine serial correlation by calculating
  the lag-1 correlation if small, may conclude
  low serial correlation since higher lag
  correlations usually smaller see TIMXCORR.XLS

10
Single System

• Single system often implies steady-state analysis
• For independent replications output estimates
  based on averages, variances, and confidence
  intervals
• Needs sample size determination for given desired
  precision
• If variance of results is unknown, may need pilot
  study to determine variance and needed sample
  size
• For correlated replications can use Bonferroni
  methods (weak)

11
Confidence Intervals

• Single system analysis based on confidence
  intervals for (1-a) confidence, interval for
  true mean is given by
• sample mean /- increment
• Increment ta,n-1sample s.d./sqrt(n)
• (see Section 11.5.3 and equation 11.38)
• Excel procedure Descriptive Statistics calculates
  increment
• see MonteCarlo-NewFab.xls

12
Competing Systems Analysis

• Most simulation studies can be considered
  analyzing competing systems - system with changed
  parameter value vs. base system
• One basic analytic measure is the confidence
  interval for the difference of the means of the
  two run outcomes (equations 12.1 12.6) if it
  contains zero it is concluded there is no
  difference
• For C such runs there will be need to be
  C!/2!(C-2)! confidence intervals for the
  difference of the two means see Bonferroni
  approximation to significance (Banks Section
  12.2.1)

13
Other Competing Systems Tests

• Intervals not only, nor best, analysis
• For independent replication comparison of
• Two systems t-test for independent samples see
  TwoPopTests.xls
• More than two systems ANOVA see ANOVA.XLS
• Example 12.1 Banks12t2.xls
• For correlated replications comparison of
• Two systems t-test for matched samples see
  TwoPopTests.xls
• More than two confidence intervals using
  Bonferroni methods Weak result since it does
  not specify probability of Type I error exactly
  and lower a increases chance of Type II error

14
Analysis of Results of Simulating More than Two
Models

• May want to consider improvement to current
  situation due to variation of several factors at
  several levels, e.g.,
• Number of servers 1, 2, or 3, and
• Amount of training none, one week, or one month
• All combinations yields 9 different models
• T-test of all differences inefficient since
  probability of Type I error for all levels
  increases as the sum of the Type I error for each
  level need large sample sizes to get error
  below acceptable level Bonferroni approach
• Use Analysis of Variance (ANOVA) or DOE

15
ANOVA Terminology and Notation

• SS sum of squared deviations from the mean -
  usually just called sum of squares
• k number of levels indexed by j 1k
• different values of a parameter
• yields k simulations whose results to be compared
• nj number of replications for level j indexed
  by i 1nj equivalent to k samples
• N total number of trials across all levels and
  replications N n1 n2 nk

16
Example One Factor at Three Levels with Three
Replications
17
Basic Measure of DifferencesSums of Square
Deviations

• Sum of Squares Total (SST) measures overall
  variation
• Sum of Squares Regression (SSR) for the level
  means measures the between levels effect - the
  signal - also often called the between (SSB) or
  treatment (SST) SS
• Sum of Squares Error (SSE) for all levels
  measures the within levels variation - the
  noise - also often called the within (SSW) or
  residual (SSR) SS
• See ANOVA.XLS for example SSR, SSE, and SST

18
Mean Squares (MS) Variances

• Sum of Squares divided by degrees of freedom (df
  data used in estimate minus number of
  population parameters estimated)
• dfR k - 1 dfT N - 1 dfE dfT - dfR
• MSR SSR/dfR MSE SSE/dfE
• Since mean squares are variances, ratios of mean
  squares (which are estimates) divided by true
  variances have an F distribution

19
ANOVA Hypothesis Test

• H0 sR sE vs. HA sR ¹ sE
• Test statistics under null hypothesis is just
  FdfR,dfE MSR/MSE
• Calculate P-value
• If greater than, say, 5 then reject H0 and
  conclude levels made a significant difference
• Else, conclude there is no significant difference
  between levels and the factor (e.g., number of
  servers) did not have a significant effect
• ANOVA procedures in Excel make all the above
  calculation see ANOVA.XLS

20
Three or More Factor Designs

• Use Excel to analyze one (e.g., number of
  servers) and two factor (e.g., number of servers
  and amount of server training) designs
• To analyze more than two factor multilevel
  designs
• Use other software such as Minitab, SPSS, SAS,
  etc. using ANOVA
• Use DOE techniques (allows t-tests for
  independent factors)

21
Design of Analysis Approach

• None of the prior analysis is considered complete
  complete analysis needs consistency between the
  design of the experiment and its analysis
• Recommended approach is Design of Analysis (DOE)
  not only consistent but also efficient
• Focuses on changes in a single DV resulting from
  simultaneous variation of multiple IVs
• Much studied efficient designs available
• Appropriate for determination of
• Important (few?) parameters
• Model formula relationship between IVs and DV
• Optimum seeking by response surface method