RTI Benchmark Studies

1
RTI Benchmark Studies

• Brad Fitzgibbons
• Thom McLean
• Richard Fujimoto
• College of Computing/GTRI
• Georgia Tech

2
Outline

• Intro/Motivation
• Synchronization Models
• Federate
• RTI
• Benchmarking
• Test Setup
• Benchmark Results
• Backups
• FDK Information
• RT Execution Models

3
Intro/Motivation

• Task Real-time RTI services
• Observations
• Previous benchmark (e.g., DMSO) tests emphasize
  raw performance rather that a realistic workload
• Existing benchmarks do not provide adequate model
  of federate or federation performance in an
  actual execution
• HLA I/F Spec does not accommodate a R/T
  synchronization model
• Thoughts
• RTI developers and consumers may need a more
  reasonable way to assess RTI design (a buyers
  guide to RTI performance)

4
Federate Synchronization Models

• (How the federate works with simulation time)
• Time-Stepped
• Typically uses receive order delivery
• Simple to understand and implement
• Subject to many anomalies
• Normally non-deterministic (not repeatable)
• Hard Real-time
• Requires a real-time infrastructure (QoS, RTOS,
  etc.)
• Common in HWITL, distributed computing
• Requires scheduled delivery services (not just
  during tick())

5
Time Stepped Execution

• The simulation is executed as a series of
  discrete intervals, length dt.
• Regular updates
• Predominant model for Distributed Interactive
  Simulation (DIS)
• Simulation receives all incoming events on each
  iteration.
• Processing must be completed in (dt scalefactor)
  time
• Slack is needed to ensure simulation time does
  not lag wall-clock time (WCT)

Code Sample while (not done)
flush message queue (deliver all incoming
messages) process messages
for each object integrate (over
dt) post results (send messages)
now now dt
wait until WallClockTime (now scalefactor)

6
Time Stepped Execution
Messages are Delivered
Messages are Sent
Slack Time
Tick
Process Events
Send Updates
Each Step (dt)
Missed Deadline (Gasp)
dt
dt
dt
T0
T0dt
T02dt
T03dt
Slack Time
The Execution
7
Hard RT (Scheduled) Execution

• Common outside the distributed simulation
  community
• Focused on scheduling remote events with
  real-time constraints.

ProcessEvent
Schedule Event(s)
T Scheduled Time
New Event
Each Event
The Execution
8
RTI Synchronization Models

• (How does the federate work with the RTI)
• Synchronous RTI callbacks
• requires tick() from federate
• RTI-NG (multi-threaded)
• DRTI (single-threaded)
• Cannot ensure deadlines are met
• Asynchronous RTI callbacks
• interrupt driven
• DRTI-ECho (implicitly multi-threaded)
• Appropriate for remotely scheduled deadlines

This is not the same as synchronous event
delivery!
9
Real-Time Message Delivery Assurance

• The RTI will provide message delivery assurances.
  To implement delivery assurances, the RTI will
  obtain the quality of service required for the
  requested delivery performance. This requires no
  change the the I/F Specification!
• Fedfile modifications needed
• Federates use Receive Order message delivery.

10
Real-Time Service Assurances

• Federates make service requests
• RTI initiates the service processing and
  schedules additional processing as needed
• RTI continues to process the request
  asynchronously
• Upon completion, by the deadline, the RTI makes
  the results of the request available
• Federate interaction after the deadline is
  assured of service request completion

11
Fully Distributed State Assurances

• Federates make service requests directly to
  remote federates, local federates, or to the LRC.
• RTI service processing is completely
  asynchronous LRC operates autonomously
• RTI makes federate callbacks as required, in RT.
• Where required, I/F Spec is modified to provide
  greater flexibility in modeling distributed RT
  functions.

12
DMSO Benchmarks

• Average One-Way Delay (BmLatency)
• 1000 updates between 2 computers
• Average Throughput (BmThruput)
• 30 cycles of 1000 updates with increasing
  attribute and federation size (sender/receiver)
• Average Advancement Rate (BmTimeAdv)
• 30 cycles of 1000 TAGs with increasing federation
  size

13
Test Setup

• RTIs
• DMSO RTI-NG 1.3 v4 (synch)
• FDK DRTI (synch)
• FDK DRTI-Echo (asynch)
• Hardware
• 2-8 dual-Pentium II 300Mhz
• RedHat Linux 7.1
• TCP over 100Mb Ethernet

14
Bundled One-Way Latency
Q Why would you want to measure latency with
bundling on? A Because thats what I expect to
do during my execution!
15
Non-Bundled One-Way Latency
16
Average Sender Throughput
17
Average Receiver Throughput
18
Time Advancement
19
Caveats (Your results may vary)

• No way to change federate behavior
• No federation behaviors represented at all
• Execution models are too simplistic
• Therefore
• Results do not transfer to real federation
  executions
• Benefits of Real-time (for example) are not
  apparent
• Need to improve/advance RTI technology is hidden
• User requirements are not met (they cannot even
  be expressed!)

20
Enhancements to BenchmarkingBehavior-based
Performance

• Federate Behavior Affects Performance
• Tick() behaviors (regular/sporadic)
• Computational load (low/high)
• Federation behavior is a function of its
  individuals
• Modeling federate/federation behavior can lead to
  a better understanding of RTI and federate
  performance

21
Enhancements to Benchmarking Relative
Performance Metrics

• Fine-grained metrics lead to a more complete
  characterization
• Individual updates, grants, etc.
• Produces large, complex sample space
• Coarse-grained metrics produce snapshots
• Performance gauged over finite window of time
• Leads to general characterization (inappropriate
  for real-time)

22
Enhancements to BenchmarkingModeling Load
23
Conclusions/Next Steps

• More Robust Federate Models
• Better Federation Dynamics
• Agreement! (community buy-in)
• Benchmarking Working group or standards
  (interested?)
• RTIC Performance Special Session..

24
FIN

• For More info
• www.cc.gatech.edu/computing/pads

25
How we do RT now - Receive Order

• Messages are delivered in the order they arrive
  from the network.

FEDT1
MESSAGE3
MESSAGE1
FEDT3
FEDT2
NETWORK
MESSAGE2
Delivery Order 2, 3, 1, 4
FEDT3
MESSAGE4
Send Order 1, 2, 3, 4
26
Time-Stamp Order

• Messages are delivered in the order of their
  timestamps
• TIME must be managed (TM service group)

FEDT1
T/S 3
T/S 1
FEDT3
FEDT2
NETWORK
T/S 4
Delivery Order 1, 2, 3, 4
FEDT3
T/S 2
Send Order 1, 2, 3, 4
27
Receive Order Anomalies

• Temporal Anomalies - events are observed out of
  order
• Causal Anomalies - impossible behavior occurs

Errors can occur when 1) a soldier
mis-perceives, and shoots a dead soldier, or 2)
when a dead soldier shoots. We may not need to
preclude these errors, but must, at least, allow
them to be consistently recreated.
LP1
LP2
Scheduling a Fire, TSO Delivery
RO Delivery
Sim Time
28
Problems

• The usefulness of real-time distributed
  simulations is limited because the behavior of
  the distributed system is not easy to predict or
  repeat.
• New networking technology can help address these
  problems, but existing infrastructure software
  must be modified to make use of the network
  capabilities.
• Specific Problems
• No Delivery Assurances - we cannot specify a
  deadline for the delivery of a message
• No Service Assurances - we cannot know be
  guaranteed that a service will be completed by a
  certain time
• No Asynchrony - the infrastructure depends on the
  federate synchronization model

29
HRT Federation Execution

• Hard real-time execution implies that meeting a
  deadline is an absolute requirement for
  correctness in the simulation system.
• For a hard real-time federate, an HRT execution
  constraint could be applied to any one of the
  execution models above.
• A hard real-time federation - must also meet
  execution deadlines for federation-level,
  distributed processes.
• This new model for federation execution
  necessitates that RTI processes execute in
  real-time.

30
Prerequisites for HRT

• Bounded Message Latency - it is impossible to
  agree on (or enforce) a distributed deadline
  without a delivery deadline
• Common clock - Federates and RTIs must agree
  upon time (to within some bounded latency)

31
Addressing the Problems

• Dead Reckoning - dont rely on timely delivery
• Transfer-of-control (ownership) - separate
  closely coupled interaction
• Virtual Timestamps - achievable ordering
• RT-RTI - RTOS-like RTI

32
Changes to the HLA I/F Specification

• The I/F Spec is insufficient for HRT execution
• Additional RTI configuration data or
  vendor-supplied specifications are needed for
  designing and executing in a HRT environment.
• HLA does not
• include an execution model for real-time
  simulation,
• provide sufficient detail to construct a
  real-time synchronization model,
• provide enough information to map federation
  requirements into RTI performance requirements.

33
Ongoing Efforts

• We are studying the following modifications to
  the IFSpec.
• Extending RTI delivery specification to support
  QoS requests
• Specifying timeliness requirements for RTI and
  Federation Ambassador calls
• Providing real-time synchronization requirements
  for RTI Service processing
• Specifying real-time simulation execution models

34
WIP Asynchronous Communications

• As a proof-of-concept prototype, we implemented
  asynchronous communications using ECho, an event
  channel comm library developed here at GT. See
  http//www.cc.gatech.edu/systems/projects/ECho

Preliminary result It worked
35
WIP RT-Time Management

• We would like to specify a deadline (in real-time
  or Wall Clock Time) for the completion of a
  distributed (LBTS) computation. This requires 2
  things
• 1) All participants in the computation by a
  specific WCT.
• 2) the computation can be completed bounded time.
• This gives two basic problems to be addressed
• How can we guarantee that all LPs will rendezvous
  at a particular time to begin the computation?
• Can we formulate an LBTS reduction that will
  always complete in bounded time?
• Additionally, we would like to understand how we
  can minimize the amount of time dedicated to the
  LBTS computation itself, since that time is
  simply overhead to the simulation execution.

36
Offset-Epoch Method for Real-Time LBTS Computation

• Federates may use normal time advancing requests
• Lookahead must be greater than 2(lbts_bound)
  latency_bound.
• LBTS will always remain gt WCT, therefore,
• all TAGs will be issued at or before the
  corresponding WCT
• All reflects/receives will occur at or before WCT.

We developed a simple technique to leverage QoS
bounded latency in a time management scheme. An
added benefit of RT TM is that it allows NRT
federates to interoperate with real-time
applications. The real-time features do not
impact the federates that have no RT demands,
provided that they meet the performance
requirements for real-time execution.
37
RT Time Management Simulation

• LBTS will always remain gt WCT, therefore,
• All TAGs will be issued at or before the
  corresponding WCT
• All reflects/receives will occur at or before WCT.

Tick()
NER()
Tick()
NER()
Tick()
TAG()
TAG()
TAG()
Receive()
LBTS
X- LBTS
LBTS
LBTS
LBTS
LBTS
In an NRT Execution, LBTS can be started at any
time. In RT, LBTS is executed on schedule.