Cactus and Solving Einsteins Equations

1
Cactus and Solving Einsteins Equations

• Edward Wang
• 2002.04.09
• Yfwang_at_mail.ustc.edu.cn

2
Episode oneWhat is cactus?
http//www.cactus.org
3
Description

• Cactus is an open source problem solving
  environment designed for scientists and
  engineers. Its modular structure easily enables
  parallel computation across different
  architectures and collaborative code development
  between different groups. Cactus originated in
  the academic research community, where it was
  developed and used over many years by a large
  international collaboration of physicists and
  computational scientists.

4

• The name Cactus comes from the design of a
  central core (or "flesh") which connects to
  application modules (or "thorns") through an
  extensible interface. Thorns can implement custom
  developed scientific or engineering applications,
  such as computational fluid dynamics. Other
  thorns from a standard computational toolkit
  provide a range of computational capabilities,
  such as parallel I/O, data distribution, or
  checkpointing.

5

• Cactus runs on many architectures.
  Applications, developed on standard workstations
  or laptops, can be seamlessly run on clusters or
  supercomputers. Cactus provides easy access to
  many cutting edge software technologies being
  developed in the academic research community,
  including the Globus Metacomputing Toolkit, HDF5
  parallel file I/O, the PETSc scientific library,
  adaptive mesh refinement, web interfaces, and
  advanced visualization tools.

6
http//www.globus.org/default.asp The Globus
Project is developing fundamental technologies
needed to build computational grids.
http//hdf.ncsa.uiuc.edu/HDF5/ HDF5 is a library
and file format for storing scientific data.
http//www-fp.mcs.anl.gov/petsc/ PETSc is a suite
of data structures and routines for the scalable
(parallel) solution of scientific applications
modeled by partial differential equations.

• PETSc

7
Features

• Highly Portable
• Supported on most architectures
• Sophisticated make system
• Powerful Application Programming Interface
• User modules (thorns) plug-into compact core
  (flesh)
• Configurable interfaces, schedules and parameters
  

8

• Advanced Computational Toolkit
• Accessible MPI-based parallelism for finite
  difference grids
• Access to a variety of supercomputing
  architectures and clusters
• Several parallel I/O layers
• Fixed and Adaptive mesh refinement under
  development
• Elliptic solvers ?
• Parallel interpolators and reductions ?
• Metacomputing and distributed computing

9

• Collaborative Development
• Interactive monitoring, steering and
  visualization
• Enables sharing code base.
• TestSuite checking technology
• Visualization tools
• Exhaustive Numerical Relativity and Astrophysical
  Applications
• Black Hole coalescence
• Neutron star collisions
• Other cataclysms

10
Applications

• If your only goal is to implement a
  simulation of a 1D wave equation, you probably
  don't want to use Cactus. You should consider
  using Cactus if you need to perform any of the
  following tasks - now or in future
• perform parallel programming in an easy but
  powerful manner using the language of your
  choice F77, F90, C, C.
• run on a wide range of architectures and
  operating systems.
• develop your code on the most convenient machine
  available, for example your workstation or laptop
  ...

11

• and simply move your code to a supercomputer
  (like a Cray T3E or Origin) to go into production
  right away!
• engage in high performance cluster computing or
• get first parallel experience by turning your
  networked PC pool into a computing cluster.
• work with collaborators on the same code and
  avoid having your programs fragmented.
• make use of the latest software technology e.g.
  take advantage of research groups that spend
  their time thinking about the fastest way to
  bring simulation data to disk or how to visualize
  it while the code is running.

12
Supported Architectures

•  Intel IA32 Linux
• IBM SP
• SGI 32 bit
• Intel IA32 Windows NT
• Cray T3E
• SGI 64 bit
• Intel IA32 Windows 2000
• Intel IA64 Linux
• Compaq Alpha
• Sun Sparc
• Hitachi SR8000-F1
• MacOS X
• Fujitsu(Beta 11) 

13

• Cactus uses the cygwin package to provide a
  unix-like environment in windows os.

cvgwin
14
Associated Packages

• MPI
• LAM Version 6.5.4 and earlier
• MPICH Version 1.2.2.1 and earlier
• MPICH-G2
• HDF5
• Version 1.4 and 1.2

15
The Computational Toolkit

• Cactus is architectured to consist of modules
  (we call them thorns) which plug into a core code
  (we call it the flesh) which contains the APIs
  and infrastructure to glue the thorns together.
• The flesh on it's own doesn't actually do
  anything. The thorns contain all the real
  activity, and can usually be divided into
  application thorns (typically written by
  scientists, solving problems in physics,
  astrophysics, engineering, etc) and
  infrastructure thorns (typically written by
  computational scientists, providing IO,
  interpolations, drivers, elliptic solvers etc).
  We group thorns together into arrangements,
  depending on their functionality and
  applicability.

16

• Drivers
• Boundary Conditions
• Coordinate Systems
• Output Methods
• Interpolators and Reduction Operators
• Elliptic Solvers
• Utilities

17
(No Transcript)
18
Live Demos

• WaveToy DemoOur standard demonstration, see the
  description page for an explanation and how to
  run this yourself.
• Cactus WormOur prototype dynamic Grid computing
  example ... this is being developed to add new
  features and fault tolerance ... please be
  understanding if it is down!

19
E-Grid and Cactus

• The Cactus Team are closely involved with the
  activities of the European Grid Forum, in
  particular with the Working Group for Testbeds,
  which is headed by Ed Seidel. Cactus is being
  used by members of the E-Grid, along with the
  distributed computing toolkit Globus as an
  initial test application in a number of
  institutions.

20

• Through work with the EGrid Consortium, Cactus
  will benefitfrom the development of
• Access to a Europe wide Grid-TestBed, joining
  computing resources at over ten institutions.
• More stable and secure methods of streaming data
  between simulation and various visualisation
  clients
• Development of procedures and thorns for
  automatic migrating Cactus jobs
• Ability to read streamed data into Cactus, e.g.
  checkpoint files
• Ability to stream checkpoint files from Cactus
• Increasing expertise in performing distributed
  runs

21
E-Grid TestBed

• The E-Grid TestBed comprises a varied set of
  computational resources which was set up for
  demonstrations at SC2000 in Dallas, and provides
  a common and supported grid infrastructure with
  each site implementing
• Globus 1.1.4
• gsiftpd
• gsisshd (using port 2222)
• MPICH-G2 (MPICH 1.2.1 with Globus device)
• HDF5 1.3.30 or above
• GRIS reporting to TestBed GIIS
• Cactus 4.0 Beta 9

22
Episode TwoSolving Einsteins Equations
http//www.computer.org/computer /articles/einstei
n_1299_1.htm
23
Einsteins Equations

• Globally distributed scientific teams, linked
  to the most powerful supercomputers, are running
  visual simulations of Einsteins equations on the
  gravitational effects of colliding black holes.

24

• In 1916, Albert Einstein published his
  famous general theory of relativity, which
  contains the rules of gravity and provides the
  basis for modern theories of astrophysics and
  cosmology. It describes phenomena on all scales
  in the universe, from compact objects such as
  black holes, neutron stars, and supernovae to
  large-scale structure formation such as that
  involved in creating the distribution of clusters
  of galaxies. For many years, physicists,
  astrophysicists, and mathematicians have striven
  to develop techniques for unlocking the secrets
  contained in Einsteins theory of gravity more
  recently, computational-science research groups
  have added their expertise to the endeavor.

25

• Those who study these objects face a daunting
  challenge The equations are among the most
  complicated seen in mathematical physics.
  Together, they form a set of 10 coupled,
  nonlinear, hyperbolic-elliptic partial
  differential equations(??????????-???????) that
  contain many thousands of terms. Despite more
  than 80 years of intense analytical study, these
  equations have yielded only a handful of special
  solutions relevant for astrophysics and
  cosmology, giving only tantalizing snapshots of
  the dynamics that occur in our universe.

26

• Scientists have gradually realized that
  numerical studies of Einsteins equations will
  play an essential role in uncovering the full
  picture. Realizing that this work requires new
  tools, developers have created computer programs
  to investigate gravity, giving birth to the field
  of numerical relativity. Progress here has been
  initially slow, due to the complexity of the
  equations, the lack of computer resources, and
  the wide variety of numerical algorithms,
  computational techniques, and underlying physics
  needed to solve these equations.

27

• A realistic 3D simulation based on the full
  Einstein equations is an enormous task A single
  simulation of coalescing neutron stars or black
  holes would require more than a teraflop per
  second for reasonable performance, and a terabyte
  of memory. Even if the computers needed to
  perform such work existed, we must still increase
  our understanding of the underlying physics,
  mathematics, numerical algorithms, high-speed
  networking, and computational science. These
  cross-disciplinary requirements ensure that no
  single group of researchers has the expertise to
  solve the full problem.

28

• Further, many traditional relativists trained
  in more mathematical aspects of the problem lack
  expertise in computational science, as do many
  astrophysicists trained in the techniques of
  numerical relativity. We need an effective
  community framework for bringing together experts
  from disparate disciplines and geographically
  distributed locations that will enable
  mathematical relativists, astrophysicists, and
  computer scientists to work together on this
  immensely difficult problem.

29
Some examples

• Figure 1. Gravitational waves from a full 3D
  grazing merger of two black holes. The image
  shows the objects immediately after the
  coalescence, when the two holes (seen just
  inside) have merged to form a single, larger hole
  at the center. The distortions of the horizon
  (the Gaussian curvature of the surface,????)
  appear as a color map, while the resulting burst
  of gravitational waves (the even-parity
  polarization or real part of Y4) appears in red
  and yellow.

30

• Figure 2. Additional polarization of
  gravitational waves (imaginary part of Y4) from a
  3D merging collision of two black holes. The
  merged single black-hole horizon can just be seen
  through the cloud of radiation emitted in the
  process.

31

• Figure 3. Close-up of the horizonwith
  Gaussian curvature to show the distorted nature
  of the surfaceof a black hole formed by the
  collision of two black holes. The two individual
  horizon surfaces can just be seen inside the
  larger horizon formed during the collision
  process.

32

• Figure 4. Gravitational waves and horizon of
  a newly formed black hole, caused by massive
  collapse of a strong gravitational wave on
  itself. The dotted surface shows the horizon,
  where green colors indicate high curvature and
  yellow means zero curvature. The highest
  curvature indicates the largest gravitational
  radiation. The distortion (the non-sphericity)
  of the black hole radiates away over time, in
  accordance with mathematical theorems about black
  holes.

33

• Figure 5. Horizon of a gravitational wave
  that implodes to form a black hole, with leftover
  waves escaping, shown at a later time in the same
  evolution presented in Figure 4. We see that the
  horizon curvature is affected by, and correlated
  with, the evolving gravitational wave.

34
Episode ThreeWaveToy Demo
35
including

• Remote monitoring and steering of an application
  from any web browser
• Streaming of isosurfaces(???) from a simulation,
  which can then be viewed on a local machine
• Remote visualization of 2D slices from any grid
  function in a simulation as jpegs in a web
  browser

36

• The application we are using is the
  simulation of the 3D scalar field produced by two
  orbiting sources. The solution is found by finite
  differencing a hyperbolic partial differential
  equation(????????) for the scalar field. This is
  a very simple application, however it is
  representative of a large class of more complex
  systems, including Einstein's Equations,
  Maxwell's Equations, or the Navier-Stokes
  Equations. We use it for demonstrations since the
  simulation is not computationally intensive, is
  very robust, has simple parameter choices, and
  has reasonable graphics.

37
Before you start, make sure you have the
following items

• GetCactus the perl script for easily checking a
  Cactus application out from our CVS server.
• WaveDemo.th the ThornList for the demo, this is
  used to tell GetCactus which thorns to get.
• WaveDemo.par a parameter file for running the
  demonstration.
• IsoView the isosurface visualization client.
• A web browser.
• Note that you'll need a live network connection
  to checkout the code, but you can run the demo on
  a single machine, the remote tools will look more
  impressive though if you use two networked
  machines, preferably a long way apart.

38
Check out and compile

• Checkout the source code using something like
• perl GetCactus WaveDemo.th
• You should be able to use the default answers for
  all the questions
• Move into the Cactus directory (cd Cactus), and
  compile the application using either
• gmake WaveDemo-config ltconfiguration optionsgt
• gmake WaveDemo

39

• or if you have a configuration file (this is
  easiest, you don't need to remember the options
  you use)
• gmake WaveDemo-config optionsltconfiguration
  filenamegt
• gmake WaveDemo
• Hopefully that went OK, and you now have an
  executable, exe/cactus_WaveDemo. Check it really
  worked by running the testsuites, just type
• gmake WaveDemo-testsuite
• and use the default answers to each question.

40
Run the demo

• Move the downloaded demo parameter file into the
  Cactus directory. To start the simulation, run
  your new executable with the demo parameter file,
  if you have a single processor executable
• ./exe/cactus_WaveDemo WaveDemo.par
• If you compiled with MPI and have a
  multiprocessor version, you will need to use the
  appropriate mpi command for running.

41

• When the simulation starts, you will see output
  describing for example the activated thorns and
  the scheduling tree.
• If you have the simple visualization client
  xgraph installed, you can look at the 1D output

42
Connecting with a web browser

• To connect to the simulation, move to another
  machine if you have one, and start up a web
  browser. Connect to
• http//ltmachine namegt5555
• where ltmachine namegt5555 is the name of the
  machine where the simulation is running. Note
  that this information was part of the standard
  output when the simulation started for example
• Server started on http//gullveig.aei-potsdam.mpg.
  de5555/
• Now you should see a screen with information
  about the simulation.

43
(No Transcript)
44
Viewing IsoSurfaces

• Start up the IsoView client, using
• IsoView -h ltmachine namegt -dp 5557 -cp 5558
• Again, this information can be found in the
  standard output, for example
• Isosurfacer listening for control connections on
  gullveig.aei-potsdam.mpg.de port 5558/
• Isosurfacer listening for data connections on
  gullveig.aei-potsdam.mpg.de port 5557/

45
You should now see rotating blobs appearing in
the client, looking something like this
46
Steering the Simulation
47
If you are watching the isosurfaces you should
see the blobs move together.
48
Episode FourSome Other Sites

• NCSA Datalink Article about Cactus from the NCSA
  Datalink news letter. http//archive.ncsa.uiuc.edu
  /datalink/9911/Cactus1.html
• TASC Review Review of Cactus Framework by
  NASA/TASC http//sdcd.gsfc.nasa.gov/ESS/esmf_tasc/
  Files/Cactus_b.html
• IEEE Computing Cover story article about Cactus
  in IEEE Computing. http//www.computer.org/compute
  r/articles/einstein_1299_1.htm
• Albert Einstein Institut The birth place of the
  Cactus Code. http//www.aei-potsdam.mpg.de/

49

• Thank you!