CCTK The Cactus Computational Toolkit

1
CCTKThe Cactus Computational Toolkit
Werner Benger Max-PIanck-Institut für
Gravitationsphysik (Albert-Einstein-Institute at
Golm/Potsdam AEI) and Konrad-Zuse-Center for
Information Technology Berlin (ZIB) Werner.Benger
_at_aei-potsdam.mpg.de
2
Introduction

• Cactus Original Motivation
• Numerical Relativity
• Ongoing Research
• What is the Cactus Computational Toolkit?
• Cactus Design - Flesh and Thorns
• What does Cactus provide?
• The Cactus Framework
• How to use Cactus
• Interfacing Cactus
• Future Developments
• Metacomputing for Numerical Relativity
• Plans...

3
Cactus - Original Motivation Numerical Relativity
The Two-Body problem in General Relativity is
still unsolved.
Kepler stable ellipses
Einstein no stable orbits because of
gravitational radiation (GR required for extreme
gravity)
Astronomy with gravitational wave detectors
(taken into operation by this year 2000!)
4
Axial symmetric Collisions of Black Holes (NCSA
1993-96)
Pre-Cactus (H-Code, G-Code), Cactus 1.0, 2.x,
Cactus 3.xUS - NSF Grand Challenge
ProjectsRequirement for Collaborative Code
Two Black Holes
A distorted single black hole
5
Focus of 3D Numerical Evolutions during the last
couple years

• Black Holes (prime source for GW) from Misner
  BH collisions to grazing BH collisions with
  initial spin and momentum (Brandt-Bruegmann
  initial data)
• Gravitational Waves (Evolution of Brill Waves,
  collapse of pure GW, investigation of critical
  amplitude, i.e. when do black holes form)
• Neutron Stars (NASA Neutron Star Grand Challenge,
  GR hydrodynamics, neutron stars colliding to
  black holes,...)

6
Visualization of 3D data with ZIB's Amira
Efforts for Remote Visualization, Remote
Monitoring, Remote Steering German Gigabit
Project (TIKSL), KDI Portal
7
German Gigabit Project TIKSL

• Remote Visualization
• Remote Steering
• Online Monitoring
• Globus Services
• HDF5 Interface
• Data Grid Access (DPSS)

8
What is the Cactus Computational Toolkit?

• Portable Application Framework
• "Flesh" provides registration and scheduling
  facilities
• Memory management ("gridarrays","gridfunctions")
• "Thorns" are exchangeable code segments
• Runtime activation/deactivation of thorns
• I/O layers (parallel I/O)
• Parameter handling
• Exchangeable Parallelization layers (MPI,
  Globus-MPI, Shared Memory,...)

9
Data Types (Portability)

• Cactus data types to provide portability across
  platforms
• CCTK_REAL
• CCTK_REAL4, CCTK_REAL8, CCTK_REAL16
• CCTK_INT
• CCTK_INT2, CCTK_INT4, CCTK_INT8
• CCTK_CHAR
• CCTK_COMPLEX
• CCTK_COMPLEX8, CCTK_COMPLEX16, CCTK_COMPLEX32

10
Cactus Flesh

• Make System
• Organizes builds as configurations which hold
  everything needed to build with a particular set
  of options on a particular architecture.
• API
• Functions which must be there for thorns to
  operate.
• Scheduler
• Sophisticated scheduler which calls
  thorn-provided functions as and when needed.
• CCL
• Configuration language which tells the flesh all
  it needs to know about the thorns.

11
Data Structures (Memory Management)

• Grid Arrays
• A multidimensional and arbitrarily sized array
  distributed among processors
• Grid Functions
• A field distributed on the multidimensional
  computational grid (a Grid Array sized to the
  grid)
• Every point in a grid may hold a different value
  f(x,y,z)
• Grid Scalars
• Values common to all the grid points
• Parameters
• Values/Keywords that affect the behavior of the
  code (initialization, evolution, output, etc..)
• parameter checking, steerable parameters

12
Cactus Thorns

• Flesh (core) written in C
• Thorns (modules) grouped in packages written in
  F77, F90, C, C
• Thorn-Flesh interface fixed in 3 files written in
  CCL (Cactus Configuration Language)
• interface.ccl Grid Functions, Arrays, Scalars
  (integer, real, logical, complex)
• param.ccl Parameters and their allowed values
• schedule.ccl Entry point of routines, dynamic
  memory and communication allocations
• Object oriented features for thorns (public,
  private, protected variables, implementations,
  inheritance) for clearer interfaces
• Compilation
• PERL parses the CCL files and creates the
  flesh-thorn interface code at compile time
• Particularly important for the FORTRAN-C
  interface. FORTRAN arg. lists must be known at
  compile time, but depend on the thorn list

13
Interface

• The concept contract with the rest of the code
• Now it is only for the data structures variables
  and parameters
• adding routines and arguments
• Private
• The variables that you want the flesh to
  allocate/communicate but no other thorn to see.
• Public
• The variables that you want everybody to see
  (that means that everybody can modify them too!)
• Inheritance
• Protected
• Variables that you want only your friends to see!

14
Implementation

• Why
• Two or more thorns that provide the same
  functionality but different internal
  implementation
• Interchangeable pieces that allow easy comparison
  and evolution in the development process
• They are compiled together and only one is
  activated at runtime
• How
• If all the other thorns need to see the same
  contract, then thorns implementing a certain
  functionality must
• Have the same public and protected variables
• The same concept applies to parameters and
  scheduling
• Example
• Wildly different evolution approaches for the
  same equations, so all the analysis and initial
  data thorns remain the same.

15
Scheduling

• Thorns schedule when their routines should be
  executed
• Basic evolution skeleton idea
• standard scheduling points INITIAL, EVOL,
  ANALYSIS
• fine control run this routine BEFORE/AFTER that
  routine
• Extend/customize with scheduling groups
• Add my routine to this group of routines
• Run the group WHILE some condition is met
• Future redesign
• The scheduler is really a runtime selector of the
  computation flow.
• Much more power can be added to this concept

16
Parallelizing an Application Thorn

• CCTK_SyncGroup
• synchronize ghostzones for a group of grid
  variables
• just add Synchronization to Scheduler
  configuration file as well
• CCTK_Reduce
• call any registered reduction operator, e.g.
  maximum value over the grid
• CCTK_Interpolate
• call any registered interpolation operator
• CCTK_MyProc
• unique processor number within the computation
• CCTK_nProcs
• total number of processors
• CCTK_Barrier
• waits for all processors to reach this point

17
PUGH

• The standard parallel driver supplied with Cactus
  is supplied by thorn PUGH
• Driver thorn Sets up grid variables, handles
  processor decomposition, deals with processor
  communications
• 1,2,3D Grid Arrays and Grid Functions (beta6)
• Uses MPI
• Custom processor decomposition
• Otherwise decomposes in z, then y, then x
  directions

18
How to use Cactus

• Optional Develop thorns, according to some
  rules
• e.g. specify variables through interface.ccl
• Specify calling sequence of the thorn subroutines
  for given problem and algorithm (schedule.ccl)
• Specify which thorns are desired for simulation
  (e.g. Einstein equations special method 1 HRSC
  hydrowave finder AMR live visualization
  module remote steering tool)
• Specified code is then created, with only those
  modules, those variables, those I/O routines,
  this MPI layer, that AMR system,, needed
  (minimal binary)
• Subroutine calling lists generated automatically
• Automatically created for desired computer
  architecture
• Run it(local, remote, on the Grid using Globus
  environment)

19
How does Cactus help ?

• Collaborative working
• Different people are experts on different parts
  of the physical problem
• Each person can write a thorn which solves their
  part, e.g. the metric evolution, hydrodynamics,
  apparent horizon finding, ...
• Thorns are encapsulated, and different thorns
  with the same functionality are interchangeable
• Parallelism
• Cactus provides a parallel layer
• Layer is independent of underlying machine
  architecture
• Researchers don't need to think deeply about
  parallelism
• Choice of parallel library layers (Native MPI,
  MPICH, MPICH-G(2), LAM, WMPI, PACX, HPVM, MPIPro)

20
...

• IO
• Cactus provides optimized IO layers for the
  various machines
• Possible to output very large datasets in a short
  time
• Parallel I/O (Parallel HDF5, Panda, John Shalf's
  FlexIO - various interfaces to MPI-I/O)
• Checkpointing
• A lot of runs take more time than queing systems
  allow
• Cactus provides mechanisms to dump out the entire
  state of the simulation and then to read it in
  again either on the same machine or another
• Platform independence
• Cactus provides a platform independent
  environment
• Various "strange compilation" issues on different
  machines are already handled by the CCTK
  environment

21
CACTUS on NT
The NT Supercluster was used in three demos and
two of the HPC challenges at SC 98 , held in
Orlando, Florida, November 9-13,
1998. http//www.ncsa.uiuc.edu/General/CC/ntcluste
r/sc98/
22
Who Uses Cactus

• Numerical Relativity
• AEI
• WashU
• NCSA
• Penn State
• UIB
• Southampton
• PRL
• ...

• Computer Science
• Panda IO Project (UIUC)
• Globus (Argonne)
• Cluster evaluation
• Roadrunner (UNM)
• NT cluster (NCSA)
• ...
• Autopilot (UIUC)
• Gigabit Project (DFN)

23
Horizons (A Metacomputing Application)

• Singularity at center hidden by event horizon
• Surface through which nothing in the interior can
  escape
• Event horizon only really detectable if we have
  the whole spacetime
• Not possible while running the simulation
• Apparent horizon always within event horizon
• Many methods to detect this
• If a horizon is found
• Definitely have a black hole
• Can compute Gaussian Curvature to inspect
  oscillations and correlation to emitted
  gravitational waves

24
Distributed Computation of AH

• SpaceTime Evolution

Apparent Horizon Computation
AH(t9.0)
Globus Services
AEI Origin 2000, 16 Nodes
AH(t11.0)
Cornell NT Cluster, 16 Nodes
AH(t16.0)
ZIB T3E, 16 Nodes
RZG CRAY T3E, 512 Nodes (Garching/Munich)
All the required technique (TIKSL NetHDF5, Globus
MDS Queries, ...) is already there!
25
Coming up (Cactus 4.2)

• Cactus communication layer
• Parallel driver thorn (e.g. PUGH) currently
  provides both variable management and
  communication
• abstract send and receives etc
• Abstract communication from driver thorn
• easily implement different parallel paradigms
• shared memory, threads, Corba, OpenMP, PVM, ...
• Compact groups (different layout in memory for
  improved Cache performance)
• Unstructured Meshes/Finite Elements/Spectral
  Methods
• Unstructured Multigrid Solver
• Convergence/Multiple Coordinate Patches
• Capability browsing mechanism
• Command line interface connect directly to
  Cactus, scheduling
• GUIs, Documentation, GUIs, Documentation .

26
What physical systems are there now ?

• Initial Data
• Schwarzschild
• Misner
• Brandt-Bruegmann puncture data
• Brill waves
• Teukolsky waves
• Distorted Brill wave and black hole
• TOV
• Colliding neutron stars
• Orbiting neutron stars
• Boson star
• Dust

27
...

• Analysis
• Apparent horizon finders
• fast flow
• minimisation
• Wave extraction
• Riemann Invariants
• Newman-Penrose quantities
• Constraint evaluation
• Geodesic tracking

28
Overview

• Introduction
• Cactus - Original Motivation Numerical
  Relativity
• Axial symmetric Collisions of Black Holes (NCSA
  1993-96)
• Focus of 3D Numerical Evolutions during the last
  couple years
• Visualization of 3D data with ZIB's Amira
• German Gigabit Project TIKSL

29
Overview, II

• What is the Cactus Computational Toolkit?
• Data Types (Portability)
• Cactus Flesh
• Data Structures (Memory Management)
• Cactus Thorns
• Interface
• Implementation
• Scheduling
• Parallelizing an Application Thorn
• PUGH
• How to use Cactus
• How does Cactus help ?
• CACTUS on NT
• Who Uses Cactus
• Horizons (A Metacomputing Application)
• Distributed Computation of AH