Cactus in GrADS

1
Cactus in GrADS

• Dave Angulo, Ian Foster
• Matei Ripeanu, Michael Russell
• Distributed Systems Laboratory
• The University of Chicago
• With Gabrielle Allen, Thomas Dramlitsch, Ed
  Seidel, John Shalf, Thomas Radke

2
Presentation Outline

• Cactus Overview
• Architecture
• Applications
• Cactus and Grid computing
• Metacomputing, Worms,
• Proposed Cactus-GrADS project
• The Cactus-G worm
• Tequila thorn and architecture
• Issues

3
What is Cactus?

• Cactus is a freely available, modular,
  portable and manageable environment for
  collaboratively developing parallel,
  high-performance multidimensional simulations
• Originally developed for astrophysics, but
  nothing about it is astrophysics-specific

4
Cactus Applications
Example output from Numerical Relativity
Simulations
5
Cactus Architecture

• Codes are constructed by linking a small core
  (flesh) with selected modules (thorns)
• Custom linking/configuration tools
• Core provides basic management services
• A wide variety of thorns are supported
• Numerical methods
• Grids and domain decompositions
• Visualization and steering
• Etc.

6
Cactus Architecture
Cactus
Thorns
Computational Toolkit
Toolkit
Toolkit
Flesh
Make
Configure
CST
Operating Systems
SuperUX
Irix
Linux
Unicos
HP-UX
Solaris
OSF
NT
AIX
7
Cactus Applications

• A Cactus application is just another thorn,
  linked with other tool thorns
• Numerous Astrophysics applications
• E.g., Calculate Schwartzchild Event Horizons for
  colliding black holes
• Potential candidates for GrADS work
• Elliptical Solver, BenchADM
• Both use 3-D grid abstract topology

8
Cactus Model (cont.)
9
Running Cactus
10
Parallelism in Cactus

• Distributed memory model each thorn is passed a
  section of the global grid
• The parallel driver (implemented in a thorn) can
  use whatever method it likes to decompose the
  grid across processors and exchange ghost zone
  information - each thorn is presented with a
  standard interface, independent of the driver

• Standard driver distributed with Cactus (PUGH) is
  for a parallel unigrid and uses MPI for the
  communication layer
• PUGH can do custom processor decomposition and
  static load balancing
• AMR driver also provided

11
Cactus and Grid ComputingGeneral Observations

• Reasons to work with Cactus
• Rich structure, computationally intensive,
  numerous opportunities for Grid computing
• Talented and motivated developer/user community
• Issues
• At core, relatively simple structure
• Cactus system is relatively complex
• User community is relatively small

12
Cactus-G Possible Opportunities

• Metacomputing use heterogeneous systems as
  source of low-cost cycles
• Departmental pool or multi-site system
• Dynamic resource selection, e.g.
• Cheapest resources to achieve interactivity
• Fastest resource for best turnaround
• Best resolution to meet turnaround goal
• Spawn independent tasks e.g., analysis
• Migration to better resource for all above

13
Cactus-G Common Building Blocks

• Resource selection based on resource and
  application characterizations
• Implementation and management of distributed
  output
• (De)centralized logging, accounting for resource
  usage, parameter selection, etc.
• Fault discovery, recovery, tolerance
• Code/executable management and creation
• Next-generation Cactus that increases flexibility
  with respect to parameter selection

14
Proposed Cactus-G Challenge Problem Cactus-G Worm

• Migrate to faster/cheaper/bigger system
• When system identified by resource discovery
• When resource requirements change
• Why?
• Tests much of the machinery required for Cactus-G
  (source code mgmt, discovery, )
• Places substantial demands on GrADS
• Good potential to show real benefit
• Migration approach simplifies infrastructure
  demands (MPI-2 support not required)

15
Cactus-G WormBasic Architecture and Operation
Application Manager
Compute resource

Compute resource
Appln other thorns
Cactus flesh
Tequila Thorn
Storage resource

GrADS Resource Selector
Storage resource
Code repository

Grid Information Service
Code repository
16
Tequila Thorn Functions

• Initiates adaptation on application request or on
  notification of new resources
• Can include user input (e.g., HTTP thorn)
• Requests resources from external entity
• GIS or ResourceSelector
• Checkpoints application
• Contacts Application Manager to request restart
  on new resources
• AppManager has security, robustness advantages
  vs. direct restart

17
Cactus-G Worm Approach

1. Uniproc Tequila thorn that speaks to GIS, adapts
   periodically done Cactus group
2. Tequila thorn that speaks to UCSD Resource
   Selector current focus
3. Integrate accurate performance models
4. Support multiprocessor execution
5. Detailed evaluation
6. Add adaptation triggers e.g., contract
   violation, new regime, user input

18
Tequila Thorn ResourceSelector

• ResourceSelector must be set up as service
• Tequila thorn sends request for new bag of
  resources
• ResourceSelector responds with the new bag

19
Current Status

• Tequila thorn prototype developed that speaks to
  ResourceSelector
• Dummy ResourceSelector that returns a static bag
  of resources
• Demonstrated CactusTequila operating
• Performance model developed
• Expected by May multiprocessor support,
  ResourceSelector interface, real performance model

20
Open Issues

• Should we move more management logic into
  Application Manager?
• How does Contract Monitor fit into architecture?
• How does PPS fit into architecture?
• How does COP and Aplication Launcher fit into
  architecture (Cactus has its own launcher and
  compiles its own code)?
• How does Pablo fit into architecture (Which
  thorns are monitored, is flesh monitored)?

21
The End
22
Request and Response

• The Request to the ResourceSelector will be
  stored in the InformationService
• Only the pointer to the data in the IS will be
  passed to the ResourceSelector
• The Response from the ResourceSelector will also
  be stored in the IS
• Only the pointer to the data in the IS will be
  passed back.

23
Tequila communication overview
24
Cactus Architecture in GrADS
Cactus
Thorns
Computational Toolkit
Toolkit
Flesh
Make
Configure
CST
Operating Systems
SuperUX
Irix
Linux
Unicos
HP-UX
Solaris
OSF
NT
AIX
25
Communication Details step 1

• Event sent to Tequila thorn requesting restart

26
Communication Details step 2

• Tequila store AART in IS

27
Communication Details step 3

• Tequila sends request to ResourceSelector passing
  pointer to data in IS

28
Communication Details step 4

• ResourceSelector retrieves AART from IS

29
Communication Details step 5

• ResourceSelector stores bag of resources (in
  AART) in IS

30
Communication Details step 6

• ResourceSelector responds to Tequila passing
  pointer to data in IS

31
Communication Details step 7

• Tequila retrieves AART with new bag of resources
  from IS

32
Requirements

• Using the IS for communication adds overhead.
• Why do this?
• GrADS requirement 1 do some things (e.g.
  compile) at one time and have the results stored
  in a persistent storage area. Pick these stored
  results up later and complete other phases.

33
Sample Tequila Scenario

• User asks to run an ADM simulation 400x400x400
  for 1000 timesteps in 10s.
• Resource selector contacted to obtain virtual
  machines
• Best virtual machine selected based on
  performance model.
• AM starts Cactus on that virtual machine (and
  monitors execution Contracts?)
• User (or application manager) decides that
  computation advances too slow and decides to
  search for a better virtual machine
• AM finds a better machine, commands the Cactus
  run to Checkpoint, transfers files and restart
  Cactus