Metacomputing Within the Cactus Framework

1
Metacomputing Within the Cactus Framework
Gabrielle Allen, Thomas Radke, Ed
Seidel. Albert-Einstein-Institut MPI-Gravitationsp
hysik

• What and why is Cactus?
• What has Cactus got to do with Globus?

2
Cactus 4.0
3
Why Cactus?

• Numerical relativity has very high computational
  requirements. Not every group has the resources
  or desire to develop a 3D code. (Especially IO,
  elliptic solvers, AMR)
• Previous experiences show that even a few people
  using one code is problematic. Need a structure
  that is maintainable and collaborative
• Scientists like to program in Fortran
• Want the ability to make new computational
  advances instantly and transparently available,
  without users modifying code

4
What Is Cactus?

• Cactus was developed as a general, computational
  framework for solving PDEs (originally in
  numerical relativity and astrophysics)
• Modular for easy development, maintenance and
  collaborations. Users supply thorns which
  plug-into compact core flesh
• Configurable thorns register parameter,
  variable and scheduling information with runtime
  function registry (RFR). Object-orientated
  inspired features
• Scientist friendly thorns written in F77, F90,
  C or C
• Accessible parallelism driver layer (thorn) is
  hidden from physics thorns by a fixed flesh
  interface

5
What Is Cactus?

• Standard interfaces interpolation, reduction,
  IO, coordinates. Actual routines supplied by
  thorns
• Portable Cray T3E, Origin, NT/Win9, Linux, O2,
  Dec Alpha, Exemplar, SP2
• Free distributed under the GNU GPL. Uses as
  much free software as possible
• Up-to-date new computational developments
  and/or thorns immediately available to users
  (optimisations, AMR, Globus, IO)
• Collaborative thorn structure makes it possible
  for large number of people to use and development
  toolkits
• New version Cactus beta-4.0 released 30th August

6
Cactus 4.0 Credits

• Cactus flesh and design
• Gabrielle allen
• Tom goodale
• Joan massó
• Paul walker
• Computational toolkit
• Flesh authors
• Gerd lanferman
• Thomas radke
• John shalf
• Development toolkit
• Bernd bruegmann
• Manish parashar
• Many others
• Relativity and astrophysics
• Flesh authors
• Miguel alcubierre

• Toni Arbona
• Carles Bona
• Steve Brandt
• Bernd Bruegmann
• Thomas Dramlitsch
• Ed Evans
• Carsten Gundlach
• Gerd Lanferman
• Lars Nerger
• Mark Miller
• Hisaaki Shinkai
• Ryoji Takahashi
• Malcolm Tobias
• Vision and Motivation
• Bernard Schutz
• Ed Seidel "the Evangelist"
• Wai-Mo Suen

7
Full GR Neutron Star Collision With Cactus
8
Thorn Arrangements
9
Cactus 4.0
Boundary
CartGrid3D
WaveToyF77
WaveToyF90
PUGH
FLESH (Parameters, Variables, Scheduling)
GrACE
IOFlexIO
IOHDF5
10
Cactus Many Developers
DAGH/AMR (UTexas)
AEI
NCSA
FlexIO
ZIB
Wash. U
HDF5
NASA
SGI
Valencia
Petsc (Argonne)
Globus (Foster)
Panda I/O (UIUC CS)
11
What Has It Got to Do With Globus?

• Easy access to available resources
• Access to more resources
• Einstein equations require extreme memory, speed
• Largest supercomputers too small!
• Networks very fast!
• DFN gigabit testbed 622 mbits potsdam-berlin-garc
  hing, connect multiple supercomputers
• Gigabit networking to US possible
• Connect workstations to make supercomputer
• Acquire resources dynamically during simulation!
• Interactive visualization and steering from
  anywhere
• Metacomputing experiments in progress with
  CactusGlobus

12
TIKSLTele Immersion Collision of Black Holes

• German research project aimed to exploit the
  newly installed gigabit testbed SüdBerlin
• Project partners
• Albert-Einstein-Institut Potsdam
• Konrad-Zuse-Institut Berlin
• Rechenzentrum Garching
• Main project goals
• Distributed simulations of black hole collisions
  with Cactus
• Remote visualization and application steering
  with Amira

AEI
13
Running Cactus in a Distributed Environment

• Using the Globus services to
• Locate computing resources via MDS
• Authenticate the cactus users (GSS)
• Transfer necessary files to remote
  sites(executable, parameter files) via GASS
• Start the Cactus job via GRAM
• Do parallel communication and file I/Ousing
  Nexus MPI and MPI-IO extensions
• Access output data via GASS

14
Computational Needs for 3D Numerical Relativity

• Explicit finite difference codes
• 104 flops/zone/time step
• 100 3D arrays
• Require 10003 zones or more
• 1000 gbytes
• Double resolution 8x memory, 16x flops
• Tflop, tbyte machine required
• Parallel AMR, I/O essential
• Etc

t100
t0

• InitialData 4 coupled nonlinear elliptics
• Time step update
• explicit hyperbolic update
• also solve elliptics

15
(A Single) Such Large Scale Computation Requires
Incredible Mix of Varied Technologies and
Expertise!

• Many scientific/engineering components
• Formulation of ees, gauge conditions, equation
  of state, astrophysics, hydrodynamics, etc
• Many numerical algorithm components
• Finite differences? Finite elements? Structured
  meshes?
• Hyperbolic equations implicit vs implicit,
  shock treatments, dozens of methods (and
  presently nothing is fully satisfactory!)
• Elliptic equations multigrid, krylov subspace,
  spectral, preconditioners (elliptics currently
  require most of the time)
• Mesh refinement?
• Many different computational components
• Parallelism (HPF, MPI, PVM, ???)
• Architecture efficiency (MPP, DSM, vector, NOW,
  ???)
• I/O bottlenecks (generate gigabytes per
  simulation, checkpointing)
• Visualization of all that comes out!

16
Distributing Spacetime SC97 Intercontinental
Metacomputing at Aei/Argonne/Garching/NCSA
Immersadesk
512 Node T3E
17
Metacomputing the Einstein EquationsConnecting
T3es in Berlin, Garching, San Diego
18
The Dream not far away...
Physics Module 1
BH Initial Data
Cactus/Einstein solver
Budding Einstein in Berlin...
MPI, MG, AMR, DAGH, Viz, I/O, ...
Mass storage
Globus Resource Manager
Ultra 3000 Whatever-Wherever
Garching T3E
NCSA Origin 2000 array