The Cactus Code

1
The Cactus Code

• Gabrielle Allen
• Albert Einstein Institute
• Max Planck Institute for Gravitational Physics
• allen_at_cactuscode.org

2
Cactus Code Versions 1,2,3

• A code for Numerical Relativity
• collaborative, portable, parallel, ...
• Model of Flesh Thorns
• Flesh core code, provides parallelisation, IO.
• Thorns plug in modules, written in Fortran or C,
  which provide the application code.
  
• Successful for numerical relativity, but problems
  as number of thorns and number of users
  increased
  
• Redesign, incorporating lessons learnt from
  previous versions
  
• Cactus 4.0

3
Current Version Cactus 4.0

• Cactus 4.0 beta 1 released September 1999
• Flesh and many thorns distributed under GNU GPL
• Wednesday Cactus 4.0 beta 6
• Supported Architectures
• SGI Origin
• SGI 32/64
• Cray T3E (142GF on 1024 nodes)
• Dec Alpha
• Intel Linux
• Windows NT
• (HP Exemplar)
• (IBM SP2)
• (Sun Solaris)

4
Cactus 4 Design Goals

• Generalization
• meta-code that can be applied to any system of
  PDEs
  
• mainly 3D cartesian finite differencing codes,
  but not only...
  
• Abstraction
• Identify key concepts that can be abstracted
• Evolution skeleton. Reduction operators. I/O.
  Etc...
  
• Encapsulation
• Protect the developers of thorns from other
  thorns ...
  
• Extension
• Prepare for new concepts in future thorns
• Overloading, Inheritance, etc...
• In some way, make it a little Object Oriented

5
Design issues

• Modular and Object Oriented
• Keep the concept of thorns
• Encapsulation, Polymorphism, Inheritance, ...
• Fortran
• influences most design issues
• Portable Parallelism
• Support for FMR and AMR as well as Unigrid
• Powerful Make system
• Tools such as Testsuite checking technology

6
Realization

• Perl
• The final code is created from Thorn
  configuration files by perl scripts that are some
  sort of seed for a new language
  
• The Cactus Configuration Language (CCL)
• variables, (functions),
• parameters,
• scheduling
• Perl scripts also take care of testsuite
  checking
  
• Flesh written in ANSI C
• Thorns written in C, C, Fortran77, Fortran90

7
Cactus Flesh interface between Application
Thorns and Computational Infrastructure Thorns
8
The Flesh

• Abstract API
• evolve the same PDE with unigrid, AMR (MPI or
  shared memory, etc) without having to change any
  of the application code.
  
• Interfaces
• set of data structures that a thorn exports to
  the world (global), to its friends (protected)
  and to nobody (private) and how these are
  inherited.
• Implementations
• Different thorns may implement e.g. the evolution
  of the same PDE and we select the one we want at
  runtime.
  
• Scheduling
• call in a certain order the routines of every
  thorn and how to handle their interdependencies.
  
  
• Parameters
• many types of parameters and all of their
  essential consistency checked before running

9
Cactus Computational Toolkit

• Parallel Evolution Drivers
• PUGH
• MPI domain decomposition based unigrid driver
• Distributed using globus
• GrACE/PAGH
• Adaptive Mesh Refinement driver
• Elliptic Solvers
• PETSc
• BAM
• Interpolators
• I/O
• FlexIO, ASCII, HDF5, Panda, etc...
• Visualization, etc...

10
Data Structures

• Grid Arrays
• An multidimensional and arbitrarily sized array
  distributed among processors
  
• Grid Functions
• A field distributed on the multidimensional
  computational grid (a Grid Array sized to the
  grid)
  
• Every point in a grid may hold a different value
  f(x,y,z)
  
• Grid Scalars
• Values common to all the grid points
• Parameters
• Values/Keywords that affect the behavior of the
  code (initialization, evolution, output, etc..)
  
• parameter checking, steerable parameters

11
Data Types

• Cactus data types to provide portability across
  platforms
  
• CCTK_REAL
• CCTK_REAL4, CCTK_REAL8, CCTK_REAL16
• CCTK_INT
• CCTK_INT2, CCTK_INT4, CCTK_INT8
• CCTK_CHAR
• CCTK_COMPLEX
• CCTK_COMPLEX8, CCTK_COMPLEX16, CCTK_COMPLEX32

12
Scheduling

• Thorns schedule when their routines should be
  executed
  
• Basic evolution skeleton idea
• standard scheduling points INITIAL, EVOL,
  ANALYSIS
  
• fine control run this routine BEFORE/AFTER that
  routine
  
• Extend/customise with scheduling groups
• Add my routine to this group of routines
• Run the group WHILE some condition is met
• Future redesign
• The scheduler is really a runtime selector of the
  computation flow.
  
• We can add much more power to this concept

13
Interface

• The concept contract with the rest of the code
• Now it is only for the data structures
  variables and parameters
  
• adding routines and arguments
• Private
• The variables that you want the flesh to
  allocate/communicate but no other thorn to see.
  
• Public
• The variables that you want everybody to see
  (that means that everybody can modify them too!)
  
• Inheritance
• Protected
• Variables that you want only your friends to see!
  
  
• Watch out for the change of meaning from C
  names

14
Implementation

• Why
• Two or more thorns that provide the same
  functionality but different internal
  implementation
  
• Interchangeable pieces that allow easy comparison
  and evolution in the development process
  
• They are compiled together and only one is
  activated at runtime
  
• How
• If all the other thorns need to see the same
  contract, then thorns implementing a certain
  functionality must
  
• Have the same public variables
• and their protected ones!!
• The same concept applies to parameters and
  scheduling
  
• Example
• Wildly different evolution approaches for the
  same equations, so all the analysis and initial
  data thorns remain the same.
  

15
Parallelism in Cactus

• Cactus is designed around a distributed memory
  model. Each thorn is passed a section of the
  global grid.
  
• The actual parallel driver (implemented in a
  thorn) can use whatever method it likes to
  decompose the grid across processors and exchange
  ghost zone information - each thorn is presented
  with a standard interface, independent of the
  driver.

drivernghostzones 1
16
PUGH

• The standard parallel driver supplied with Cactus
  is supplied by thorn PUGH
  
• Driver thorn Sets up grid variables, handles
  processor decomposition, deals with processor
  communications
  
• 1,2,3D Grid Arrays and Grid Functions (beta6)
• Uses MPI
• Custom processor decomposition
• Otherwise decomposes in z, then y, then x
  directions
  

17
Parallizing an Application Thorn

• CCTK_SyncGroup
• synchronise ghostzones for a group of grid
  variables
  
• just added Synchronization to Scheduler
  configuration file as well
  
• CCTK_Reduce
• call any registered reduction operator, e.g.
  maximum value over the grid
  
• CCTK_Interpolate
• call any registered interpolation operator
• CCTK_MyProc
• unique processor number within the computation
• CCTK_nProcs
• total number of processors
• CCTK_Barrier
• waits for all processors to reach this point

18
Building an executable

• Compiling Cactus involves two stages
• creating a configuration
• compiling the source files to an executable
• Configuration
• Cactus can be compiled with different compilers,
  different compilation options, with different
  lists of thorns, with different external
  libraries (e.g. MPICH or LAM), and on different
  architectures.
• To facilitate this Cactus uses configurations,
  which store all the distinct information used to
  build a particular executable (Cactus/configs)
  
• Each configuration is given a unique name.

19
Configuration Options

• gmake MyConfig-config (or options
  file)
  
• Default options decided by autoconf
• Compiler and tool specification e.g.
• F77/weirdplace/pgf90
• Compilation and tool flags e.g.
• CFLAGSsave-temps
• DEBUGALL
• Library and include file specification
• Precision options e.g.
• REAL_PRECISION16

20
Configuring with External Packages

• Cactus currently knows about the external
  packages
  
• MPI (NATIVE, MPICH, LAM, WMPI,CUSTOM)
• HDF5
• GRACE
• and will search standard locations for them
• gmake MyConfig MPINATIVE
• gmake MyConfig MPIMPICH MPICH_DEVICEglobus
  GLOBUS_LIB_DIR/usr/local/globus/lib
  
• gmake MyConfig MPICUSTOM MPI_LIBSmpi
  MPI_LIB_DIRS/usr/lib MPI_INC_DIRS/usr/include

21
Compile Options

• gmake MyConfig
• Parallel build
• FJOBS
• TJOBS
• Compilation debugging
• SILENTno
• Compiler warnings
• WARNyes

22
Running Cactus

• ./exe/cactus_MyConfig MyParameterFile.par
• Additional command line options
• -h help
• -Ov details about all parameters
• -o details about one parameter
• -v version number, compile date
• -T list all thorns
• -t is thorn compiled
• -r redirect stdout
• -W reset warning level
• -E reset error level

23
Parameter Files

• Cactus runs from a users parameter file
• chooses the thorns to be used for the run (so
  that inactive thorns cant do any damage)
  
• sets parameters which are different from default
  values
  
• !desc Demonstrates my new application
• ActiveThorns PUGH WaveToyF77 Boundary
  CartGrid3D
  
• driverglobal_size 30 Change the grid size
• wavetoy amplitude 1.56 Initial wave
  amplitude

24
Coming up (Cactus 4.2)

• Cactus communication layer
• Parallel driver thorn (e.g. PUGH) currently
  provides both variable management and
  communication
  
• abstract send and receives etc
• Abstract communication from driver thorn
• easily implement different parallel paradigms
• shared memory, threads, Corba, OpenMP, PVM, ...
• Compact groups (different layout in memory for
  improved Cache performance)
  
• Unstructured Meshes/Finite Elements/Spectral
  Methods
  
• Unstructured Multigrid Solver
• Convergence/Multiple Coordinate Patches
• Capability browsing mechanism
• Command line interface connect directly to
  Cactus, scheduling
  
• GUIs, Documentation, GUIs, Documentation .

25
www.CactusCode.org

• Documentation
• IEEE Computer December 1999
• Users Guide
• Maintainers Guide
• Download
• CVS distribution (stable and development
  versions)
  
• Development
• Bugs and feature requests
• Mailing lists (e.g. cactususers_at_cactuscode.org,
  flesh_at_cactuscode.org)
  
• Showcase
• Presentations, publications, movies...
• News, and Links to related institutions, software