Experiences with the DEVStone Synthetic Benchmark

1
Experiences with the DEVStone Synthetic Benchmark

• J. Marcelo Gutierrez-Alcaraz
• Proxy speaker Rodrigo Castro
• April 2008

2
Agenda

• Introduction
• DEVS Model
• DEVS based Simulators, CD, ADEVS
• DEVStone
• DEVStone Models
• DEVStone Test case
• CD Initialization Problem
• DEVStone Comparison Tests
• Conclusions
• Questions

3
Introduction

• DEVS (Discrete Event systems Specifications)
  formalism was successfully applied in a wide
  variety of applications due to the ease for model
  definition, improved composition and reuse.
• Several tools have implemented DEVS theory,
  including CD, ADEVS, DEVS-C, DEVS/HLA,
  DEVSJAVA, DEVSim and others.
• Growing use, growing challenges
• Systems under study become larger and more
  complex
• Performance of the simulator becomes critical
• Evaluating simulators performance is a complex
  process
• A uniform means for obtaining meaningful metrics
  is needed
• We introduced the DEVStone benchmark
• A synthetic model generator that automatically
  creates models according to our performance
  analysis goals.
• Initially developed to analyze the performance of
  different simulation techniques in the CD
  toolkit.

4
Introduction

• A second step presented here, extends the
  execution of DEVStone in CD and ADEVS.
• to perform thorough cross-tool comparative
  performance analysis
• with a demanding set of experiments
• DEVStone is used to measure and improve DEVS
  simulation tools, by providing a unified
  mechanism for comparing different tools and
  environments.
• No previous means for cross-tool thorough
  comparison of DEVS simulators efficiency.

5
DEVS Atomic Model
6
DEVS-based Simulators

• ADEVS
• A Discrete EVent System simulator, developed by
  Jim Nutaro of the University of Arizona.
• Self-contained C library for constructing
  discrete event simulations based on the DEVS
  Formalism.
• Models are constructed based on templates in C
  and compiled and linked to the library to produce
  the executable.
• CD
• An implementation of the DEVS formalism developed
  at Carleton University (and other Universities).
• Modular simulator built as a DEVS model in C.
• To date is comprised by multiple versions for
  multiple platforms Embedded CD, Parallel CD,
  Stand-alone CD, etc.

7
DEVStone

• Synthetic benchmark based on the Dhrystone
  Benchmark
• Originally used to measure performance of CD
  simulators
• 4 representative model structures available
  (configurable).
• Configuration variables depth, width, transition
  functions load.
• Dhrystones running in external and internal
  transition functions.
• DEVStone metrics based on the performance of the
  simulator
• for a given model (or models) compared to
• the same model (or models) running on another
  simulator.
• DEVStone was ported to ADEVS in this work
• The newest (highly coupled) model was developed
  in this work

8
DEVStone Model Overview
Low Level Interconnect Model - LI
n
input
L2
output
L1
w
High Level Interconnect Model HI, HO
n
input
L2
output
w
L1
9
DEVStone LI Models
DEVStone deepest model
DEVStone LI model
10
DEVStone HI, HO Models
DEVStone HI model
DEVStone HO model
11
DEVStone HOmod Model
DEVStone HOmod model (w2)
12
DEVStone Test Case

• An Initialization Problem
• Its been drawn to our attention that
  initialization times in CD are greater than
  any other time spent during the execution of the
  CD model simulation.
• To test this hypothesis a DEVStone test setup was
  constructed and compared against ADEVS.
• Additional DEVStone performance tests were built
  for CD and ADEVS.
• With the DEVStone setup a debugging process is
  presented
• helps the user find and correct performance
  problems using standard tools

13
DEVStone - Initialization
Compilation 10 sec.. (not included)
Compilation gt 1 hr. (included)
14
DEVStone Profiler
GDB Profiler (intrusive) Detailed output broken
down into function calls
15
DEVStone Profiler - detailed
Due to secuentially-based search method in gcc
2.95.3. Improved in new versions of the compiler.
GDB Profiler Detailed Output
16
DEVStone LIw, LIl Test Case
LI variable width, LI variable depth test results
LIw Plot for ?int lt ?ext
LIl Plot for ?int lt ?ext
17
DEVStone HIw, HIl Test Case
HI variable width, HI variable depth test results
HIw Plot for ?int lt ?ext
HIl Plot for ?int lt ?ext
18
DEVStone HOw, HOl Test Case
HO variable width, HO variable depth test results
HOw Plot for ?int ?ext
HOl Plot for ?int ?ext
19
DEVStone HOw, HOl Test Case
HO variable width, HO variable depth test results
HOl Plot for ?int gt ?ext
20
DEVStone HOmodw, HOmodl Test Case
HOmod variable width, HOmod variable depth test
results
HOmodw Plot for ?int lt ?ext
HOmodl Plot for ?int ?ext
21
DEVStone Test Case
HO vs. HOmod test results
HO Plot for ?int ?ext
HOmod Plot for ?int ?ext
22
DEVStone Test Case
HO vs. HOmod test results for variable width
HO Plot for ?int ?ext
HOmod Plot for ?int ?ext
23
Conclusions

• DEVStone provides a common metric for different
  simulation tools.
• Enables efficiency analysis of successive
  versions of several simulators.
• New DEVStone model HOmod, represents message
  intensive systems with enormous overhead.
• DEVStone becomes an even more useful tool when it
  is combined with standard debugging tools.
• Based on DEVStone results we can classify
  simulators by their performance.
• The CD Toolkit shows better performance than
  ADEVS for those models dealing with intensive use
  of internal transition function.
• Next Steps
• Try different benchmarks based on floating point
  operations.
• Recompile Stand-alone CD with a recent compiler
  version and analyze the results.
• Port DEVStone to different DEVS-based simulators
• Port DEVStone to ECD and other CD versions

24
Questions?
Thanks for your attention.