The Cactus Code: A Problem Solving Environment for the Grid

1
The Cactus Code A Problem Solving Environment
for the Grid

• Gabrielle Allen, Gerd Lanfermann
• Max Planck Institute for Gravitational Physics

2
What is Cactus?

• Cactus is a freely available, modular,
• portable and manageable environment
• for collaboratively developing parallel, high-
• performance multidimensional simulations

3
Cactus
remote steering
Plug-In Thorns (modules)
extensible APIs
Fortran/C/C Java/Perl/Python
ANSI C
parameters
driver
scheduling
equations of state
Core Flesh
input/output
error handling
black holes
interpolation
make system
boundary conditions
grid variables
SOR solver
coordinates
multigrid
wave evolvers
4
Cactus Architecture
Cactus
Thorns
Computational Toolkit
Toolkit
Toolkit
Flesh
Make
Configure
CST
Operating Systems
SuperUX
Irix
Linux
Unicos
HP-UX
Solaris
OSF
NT
AIX
5
Thorn Architecture
Thorn
Configuration Files
Parameter Files and Testsuites
Source Code
????
????
Fortran Routines
C Routines
C Routines
Make Information
Documentation!
6
State of the Art
Numerical Relativity Simulations Albert Einstein
Institute Washington University Viz Werner Benger
7
Current Version Cactus 4.0

• Cactus 4.0 beta 1 released September 1999
• Flesh and many thorns distributed under GNU GPL
• Currently Cactus 4.0 beta 8
• Supported Architectures
• SGI Origin
• SGI 32/64
• Cray T3E
• Dec Alpha
• Intel Linux IA32/IA64
• Windows NT
• HP Exemplar
• IBM SP2
• Sun Solaris
• Hitachi SR8000-F
• NEC SX-5
• Mac Linux

8
Flesh API

• Abstract Flesh API for
• Driver functions (storage, communication)
• Interpolation
• Reduction
• IO, checkpointing
• Coordinates
• etc, etc
• In general, thorns overload or register their
  capabilities with the Flesh, agreeing to provide
  a function with the correct interface
• e.g. CCTK_SyncGroup (overloaded)
• e.g. CCTK_OutputVar(variable,IOASCII) (registe
  red)

9
Application View
10
Parallelism in Cactus

• Cactus is designed around a distributed memory
  model. Each thorn is passed a section of the
  global grid.
• The actual parallel driver (implemented in a
  thorn) can use whatever method it likes to
  decompose the grid across processors and exchange
  ghost zone information - each thorn is presented
  with a standard interface, independent of the
  driver.

• Standard driver distributed with Cactus (PUGH) is
  for a parallel unigrid and uses MPI for the
  communication layer
• PUGH can do custom processor decomposition and
  static load balancing

11
Configuration files

• Each thorn provides 3 configuration files,
  detailing its interface with the Flesh and with
  other thorns
• CCL Cactus Configuration Language
• interface.ccl
• implementation, this thorns variables and
  variables used from other thorns
• param.ccl
• this thorns parameters, parameters used and
  extended from other thorns
• schedule.ccl
• when and how this thorns routines should be
  executed, optionally with respect to routines
  from other thorns

12
Cactus Computational Toolkit

• CactusBase
• Boundary, IOUtil, IOBasic, CartGrid3D, IOASCII,
  Time
• CactusBench
• BenchADM
• CactusExample
• WaveToy1DF77, WaveToy2DF77
• CactusElliptic
• EllBase, EllPETSc, EllSOR, EllTest
• CactusPUGH
• Interp, PUGH, PUGHSlab

• CactusPUGHIO
• IOFlexIO, IOHDF5, IsoSurfacer
• CactusTest
• TestArrays, TestCoordinates, TestInclude1,
  TestInclude2, TestComplex, TestInterp
• CactusWave
• IDScalarWave, IDScalarWaveC, IDScalarWaveCXX,
  WaveBinarySource, WaveToyC, WaveToyCXX,
  WaveToyF77, WaveToyF90, WaveToyFreeF90
• external
• IEEEIO, RemoteIO, TCPXX

13
Cactus can make use of ...

• Autopilot
• FlexIO (IEEEIO/HDF5)
• Globus
• GrACE
• HDF5
• MPI
• Panda IO
• PAPI
• PETSc

14
AutoPilot

• Dynamic performance instrumentation, on-the-fly
  performance data reduction, resource management
  algorithms, real-time adaptive control mechanism
• Cactus provides a mechanism to register timers,
  and Autopilot is currently being integrated.

http//www-pablo.cs.uiuc.edu/Project/Autopilot/Aut
opilotOverview.htm http//www.cactuscode.org/Docum
entation/HOWTO/Performance-HOWTO http//www.cactus
code.org/Projects.html
15
FlexIO (IEEEIO)
IEEEIO readers for Amira AVS IDL LCA Vision NAG
Explorer

• FlexIO is a compact multi-platform API for
  storing multidimensional scientific data. It
  hides the differences between underlying file
  formats including HDF5 and IEEEIO.

• IEEEIO is a compact library for storing
  multidimensional scientific data in a binary
  format that can be transported between different
  computer systems.
• Cactus thorn CactusPUGHIO/IOFlexIO outputs
  multidimensional data using the IEEEIO library.

http//zeus.ncsa.uiuc.edu/jshalf/FlexIO/ http//z
eus.ncsa.uiuc.edu/jshalf/FlexIO/IEEEIO.html http
//www.cactuscode.org/Documentation/HOWTO/Visualiza
tion-HOWTO Documentation in thorns
CactusBase/IOUtil and CactusPUGHIO/IEEEIO
16
Globus Toolkit

• Globus Toolkit Enables application of Grid
  concepts to scientific and engineering computing
• Cactus (with the default MPI driver) compiles
  with Globus (1.0/1.1), using MPICH-G.
• Cactus can then be run using RSL scripts as usual
  with Globus

The Grid Dependable, consistent, pervasive
access to high-end resources Collaborative
engineering Browsing of remote datasets Use of
remote software Data-intensive computing Very
large-scale simulation Large-scale parameter
studies
http//www.globus.org/ http//www.cactuscode.org/D
ocumentation/HOWTO/Globus-HOWTO http//jean-luc.ae
i-potsdam.mpg.de/SC98/
17
HDF5

• Hierarchical data format for scientific data
  management (I/O libraries and tools).
• Future standard, overcomes limitations of HDF4.
  Simple but powerful model, includes hyperslabs,
  datatype conversion, parallel IO.
• Used for 2D/3D output in Computational Toolkit
  (CactusPUGHIO/IOHDF5)
• Much development in (remote) visualization and
  steering with Cactus uses HDF5
• Readers for Amira, OpenDX, (LCA Vision).

http//hdf.ncsa.uiuc.edu/HDF5/ http//www.CactusCo
de.org/Documentation/UsersGuide_html/node15.html h
ttp//www.cactuscode.org/Documentation/HOWTO/Visua
lization-HOWTO Documentation in thorns
CactusBase/IOUtil and CactusPUGHIO/IOHDF5
18
Panda IO

• Data management techniques for I/O intensive
  applications in high-performance scientific
  computing.
• Simpler, more abstract interfaces, efficient
  layout alternatives for multidimensional arrays,
  high performance array I/O operations.
• Thorn IOPanda

http//cdr.cs.uiuc.edu/panda/ http//www.cactuscod
e.org/Workshops/NCSA99/talk13/sld003.htm
19
PAPI

• Standard API for accessing the hardware
  performance counters on most microprocessors.
• Useful for tuning, optimisation, debugging,
  benchmarking, etc.

• Java GUI available for monitoring the metrics
• Cactus thorn CactusPerformance/PAPI

http//icl.cs.utk.edu/projects/papi/ http//www.ca
ctuscode.org/Documentation/HOWTO/Performance-HOWTO
http//www.cactuscode.org/Projects.html
20
Grid-Enabled Cactus

• Cactus and its ancestor codes have been using
  Grid infrastructure since 1993
• Support for Grid computing was part of the design
  requirements for Cactus 4.0 (experiences with
  Cactus 3)
• Cactus compiles out-of-the-box with Globus
  using globus device of MPICH-G(2)
• Design of Cactus means that applications are
  unaware of the underlying machine/s that the
  simulation is running on applications become
  trivially Grid-enabled
• Infrastructure thorns (I/O, driver layers) can be
  enhanced to make most effective use of the
  underlying Grid architecture

21
Grid Experiments

• SC93
• remote CM-5 simulation with live viz in CAVE
• SC95
• Heroic I-Way experiments leads to development of
  Globus. Cornell SP-2, Power Challenge, with live
  viz in San Diego CAVE
• SC97
• Garching 512 node T3E, launched, controlled,
  visualized in San Jose
• SC98
• HPC Challenge. SDSC, ZIB, and Garching T3E
  compute collision of 2 Neutron Stars, controlled
  from Orlando
• SC99
• Colliding Black Holes using Garching, ZIB T3Es,
  with remote collaborative interaction and viz at
  ANL and NCSA booths
• 2000
• Single simulation LANL, NCSA, NERSC, SDSC, ZIB,
  Garching,
• Dynamic distributed computing spawning new
  simulations

22
Cactus Globus
Cactus Application Thorns Distribution
information hidden from programmer Initial data,
Evolution, Analysis, etc

Grid Aware Application Thorns Drivers for
parallelism, IO, communication, data
mapping PUGH parallelism via MPI (MPICH-G2,
grid enabled message passing library)
Grid Enabled Communication Library MPICH-G2
implementation of MPI, can run MPI programs
across heterogeneous computing resources
Standard MPI
Single Proc
23
Grand Picture
Viz of data from previous simulations in SF café
Remote steering and monitoring from airport
Remote Viz in St Louis
Remote Viz and steering from Berlin
DataGrid/DPSS Downsampling
IsoSurfaces
http
HDF5
T3E Garching
Origin NCSA
Globus
Simulations launched from Cactus Portal
Grid enabled Cactus runs on distributed machines
24
Grid Related Projects

• ASC Astrophysics Simulation Collaboratory
• NSF Funded (WashU, Rutgers, Argonne, U. Chicago,
  NCSA)
• Collaboratory tools, Cactus Portal
• Currently setting up testbed (Globus, Cactus,
  Portal at NCSA, ZIB, AEI)
• E-Grid European Grid Forum
• Members from academic and government
  institutions, computer centers and industry
• Test application CactusGlobus
• Currently working towards distributed computing
  project for SC2000 (spawning Cactus jobs to new
  machines)
• GrADs Grid Application Development Software
• NSF Funded (Rice, NCSA, U. Illinois, UCSD, U.
  Chicago, U. Indiana...)
• Application driver for grid software

25
Grid Related Projects (2)

• Grid Forum
• Experiments
• Transparency appearances
• Distributed Runs
• AEI, Argonne, U. Chicago
• Working towards running on several computers,
  1000s of processors (different processors,
  memories, OSs, resource management, varied
  networks, bandwidths and latencies)
• TIKSL
• German DFN funded AEI, ZIB, Garching
• Remote online and offline visualization, remote
  steering/monitoring
• Cactus Team
• Dynamic distributed computing
• Testing of alternative communication protocols
  MPI, PVM, SHMEM, pthreads, OpenMP, Corba, RDMA,
  ...

26
Dynamic Distributed Computing

• Make use of
• Running with management tools such as Condor,
  Entropia, etc.
• Scripting thorns (management, launching new jobs,
  etc)
• Dynamic use of MDS for finding available
  resources
• Applications
• Portal for simulation launching and management
• Intelligent parameter surveys (Cactus control
  thorn)
• Spawning off independent jobs to new machines
  e.g. analysis tasks
• Dynamic staging seeking out and moving to
  faster/larger/cheaper machines as they become
  available (Cactus worm)
• Dynamic load balancing (e.g. inhomogeneous loads,
  multiple grids)

27
Remote Visualization
OpenDX
OpenDX
Amira
Contour plots (download)
LCA Vision
IsoSurfaces and Geodesics
Grid FunctionsStreaming HDF5
Amira
28
Remote Visualization

• Streaming data from Cactus simulation to viz
  client
• Clients OpenDX, Amira, LCA Vision, ...
• Protocols
• Proprietary
• Isosurfaces, geodesics
• HTTP
• Parameters, xgraph data, JPegs
• Streaming HDF5
• HDF5 provides downsampling and hyperslabbing
• all above data, and all possible HDF5 data (e.g.
  2D/3D)
• two different technologies
• Streaming Virtual File Driver (I/O rerouted over
  network stream)
• XML-wrapper (HDF5 calls wrapped and translated
  into XML)

29
Remote Visualization (2)

• Clients
• Proprietary
• Amira
• HTTP
• Any browser ( xgraph helper application)
• HDF5
• Any HDF5 aware application
• h5dump
• Amira
• OpenDX
• LCA Vision (soon)
• XML
• Any XML aware application
• Perl/Tk GUI
• Future browsers (need XSL-Stylesheets)

30
OpenDX

• Open source, (free), multiplatform, large active
  development community, easy to program
• Reads HDF5 (Cactus) data from file or remotely
  streamed from Cactus
• Simple GUI, select different hyperslabs from 3D
  data
• Also support for streamed ASCII data from Cactus

31
Remote Visualization - Issues

• Parallel streaming
• Cactus can do this, but readers not yet available
  on the client side
• Handling of port numbers
• clients currently have no method for finding the
  port number that Cactus is using for streaming
• development of external meta-data server needed
  (ASC/TIKSL)
• Generic protocols
• Data server
• Cactus should pass data to a separate server that
  will handle multiple clients without interfering
  with simulation
• TIKSL provides middleware (streaming HDF5) to
  implement this
• Output parameters for each client

32
Remote Steering
Any Viz Client
HTTP
Remote Viz data
XML
HDF5
Amira
Remote Viz data
33
Remote Steering

• Stream parameters from Cactus simulation to
  remote client, which changes parameters (GUI,
  command line, viz tool), and streams them back to
  Cactus where they change the state of the
  simulation.
• Cactus has a special STEERABLE tag for
  parameters, indicating it makes sense to change
  them during a simulation, and there is support
  for them to be changed.
• Example IO parameters, frequency, fields
• Current protocols
• XML (HDF5) to standalone GUI
• HDF5 to viz tools (Amira)
• HTTP to Web browser (HTML forms)

34
Thorn http

• Thorn which allows simulation to act as a web
  server
• Connect to simulation from any browser
• Monitor run parameters, basic visualization, ...
• Change steerable parameters
• See running example at www.CactusCode.org
• Wireless remote viz, monitoring
  and steering

35
Remote Steering - Issues

• Same kinds of problems as remote visualization
• generic protocols
• handling of port numbers
• broadcasting of active Cactus simulations
• Security
• Logins
• Who can change parameters?
• Lots of issues still to resolve ...

36
Remote Offline Visualization
Viz in Berlin
VisualizationClient
Downsampling, hyperslabs
Only what is needed
Remote Data Server
4TB at NCSA
37
Remote Offline Visualization

• Accessing remote data for local visualization
• Should allow downsampling, hyperslabbing, etc.
• Access via DPSS is working (TIKSL)
• Waiting for DataGrid support for HTTP and FTP to
  remove dependency on the DPSS file systems.

38
More Information ...

• Web site www.CactusCode.org
• Users Guide
• Development projects
• HOWTOs
• References for common questions, includes
  HOWTO-QuickStart
• Tutorials
• Help desk
• cactusmaint_at_cactuscode.org
• Bug tracking
• Mailing lists