Optimistic Parallel Discrete Event Simulations of Physical Systems using Reverse Computation

1
Optimistic Parallel Discrete Event Simulations of
Physical Systems using Reverse Computation
Yarong Tang Kalyan S. Perumalla Richard M. Fujimoto College of Computing, Georgia Tech Homa Karimabadi Jonathan Driscoll Yuri Omelchenko SciberNet Inc.

• 06/01/2005

2
The Big Picture

• Global Multi-Scale Kinetic Simulations of the
  Earth's Magnetosphere
• Multi-physics multi-scale
• The goal to understand how solar wind interacts
  with the Earths magnetosphere (space weather)

• People
• PDES simulation lead by Richard Fujimoto and
  Kalyan Perumalla
• Compiler optimization lead by Pande Santosh
• Physics modeling lead by Homa Karimabadi _at_
  Scibernet Inc.

3
Outline

• Motivation
• Overview
• The PIC model (Particle-in-Cell)
• Implementation
• Performance
• Conclusion Future work

4
Why optimistic simulation?

• Synchronization in PDES ensures LPs to process
  events in time stamp order
• Conservative approach
• Avoids violation of local causality constraint
• Inefficient for applications with little
  lookahead
• (low predictability)
• Plasma simulation highly dynamic, nearly-zero
  lookahead in certain regions

5
Why reverse computation?

• Optimistic synchronization
• Allows causality errors to happen, but uses a
  rollback mechanism to recover
• State saving saves state prior to computation
  restores saved state
• Plasma simulation has a lots of events ? lots of
  memory!
• Event computation cost is small compared to state
  saving overhead ? significant overhead!
• Reverse computation rollback by performing the
  inverse of the event computation
• Reduces both memory and time needed for state
  saving
• Can be automated
• Excellent match for large, high-performance
  models

6
Outline

• Motivation
• Overview
• The PIC model (Particle-in-Cell)
• Implementation
• Performance
• Conclusion
• Future work

7
Particle-in-Cell (PIC) Model

• Charging a spacecraft due to a periodical beam
  injection from its surface
• Conceptually simple, yet sufficiently complex to
  capture characteristics of plasma dynamics

• What are we interested in the simulation?
• Charge on spacecraft surface
• Plasma dynamics

8
Challenges

• Complicated interactions between physical
  entities cells (modeled by LPs) and particles

Cell1 Cell2

• Cell1 sends a particle to cell2
• Cell1 updates state and may wake up all its
  particles

• Cell2 updates state and may wake up all its
  particles

9
Particle Movement Illustrated
Cell 1
Cell 2
Cell 3
Simulation Time
10
Challenges cont.d

• Memory requirement
• Particle state
• Velocity, acceleration, position, direction,
  type,
• exit_time, lastmove_time, etc.
• Cell state
• Center field, boundary fields, coordinates,
  particle queue
• Q How to minimize the number of auxiliary bits
  
• for efficient reverse code?

11
Challenges cont.d

• Floating point arithmetic throughout simulation
• Simple increment/decrement wont work
• Destructive operations are dominant
• Particles exit_time is computed by solving a
  pair of quadratic equations ? not reversible
• Wakeup event ? recompute all particle states
• Delete operations in particle queue

12
Our observations

• delete in queue is the inverse of insert
• and vise versa
• Particles never disappear one deletion in a
  cell corresponds to an insertion in another cell
• Partial particle state is carried over by msgs
• No need to re-compute these state variables
• Reversing physical states can resort to physics
  laws
• No need for brute-force when reversing code
• Application semantics is the magic for
  irreversible code

13
Implementation example
Shellarrival( ParticleArrivalEvent e )
if( this cell is active ) update
cell state insert particle in cell
else if ( e is a beam particle )
activate cell Shelldeparture(
ParticleDepartureEvent e ) if( particle
bounced from right neighbor ) bounce
particle back // no state change else
update cell state Shellinject(
ParticleInjectEvent e ) update cell
state insert beam particles
Shellundo_arrival( ParticleArrivalEvent e )
if ( cell was activated )
undo_activate cell else if ( cell
already active ) delete particle in
cell undo_update cell state
Shellundo_departure( ParticleDepartureEvent
e ) if ( particle was bounced from right
neighbor ) undo_bounce particle
else undo_update cell state
Shellundo_inject( ParticleInjectEvent e
) delete beam particles
undo_update cell state
14
Reverse Computation Illustrated
ForwardAddParticleToPQ() Q Q'q/c.w2a
Q.q/mv v'a.dtx x'v.dt½a.dt2
Reversex' x-v.dt-½a.dt2v' v-a.dtQ'
Q-q/c.w2a' Q'.q/mDelParticleFromPQ()
?

Cell 2
dt
Cell 3
Simulation Time
15
Experiment

• Simulation engine µsik (by Kalyan Perumalla)
• General-purpose parallel/distributed sim engine
• Supports multiple synchronization approaches
• Hardware platform
• SMP machines running Red Hat Linux with a
  customized 2.4.18-10smp kernel
• 8 PIII 550MHz Xeon processors sharing 4GB of
  memory

16
Simulation configuration

• 1 Karimabadi, H, et al. A New Asynchronous
  Methodology for Modeling of Physical Systems
  Breaking the Curse of Courant Condition. J.
  Computational Physics, 2005
• Normalized units
• Simulate 7000 cells with 70 initially active
  cells
• Spacecraft has a radius of 500 cell has a width
  of 0.24
• Solar wind plasma initially loaded with uniformly
  distributed electrons and protons (30 of each per
  cell)
• Injected positron beam has an energy of 10 keV
  with a period of 4ms

17
Phase space comparison
Time-stepped
Sequential DES
18
Validation by phase space comparison
Sequential DES
PDES
19
PDES vs. sequential DES
20
Events distribution
Total Number of Committed Events
21
Conclusions

• Being the First to apply reverse computation to
  parallel simulations of physical systems
• Propose application-level reverse computation
• Encouraging results for 1D model, better
  performance expected for models with a higher
  dimension

22
Ongoing and future work

• Performance comparison with check-pointing
  approaches
• More complex applications on larger-scale systems
• Apply reverse computation to other domains of
  physics applications involving partial
  differential equations

23

• Any questions?
• Thank you!