Technologies and Tools for High-Performance Distributed Computing

1
David Keyes, project lead
http//www.tops-scidac.org
2
Plan

• What TOPS aims to provide
• Some uses of TOPS solver software
• TOPS posters what to find Tues. aft.
• TOPS attendees who to meet Tues. aft.
• Preview of TOPS linear solver interface
• How to work with TOPS

• Get you to lunch on time!

3
Shameless advertisement

• 16th Domain Decomposition meeting
• USA hosts for first time since 1997
• January 12-16, 2005
• Courant Institute, NYU
• Co-organized by O. Widlund D. K.
• Invited speakers
• Participant-initiated minisymposia
• Contributed talks
• Pre-conference short course on software for
  prototyping domain decomposition methods

4
Shameless advertisement

• 16th Domain Decomposition meeting
• USA hosts for first time since 1997
• January 12-16, 2005
• Courant Institute, NYU
• Co-organized by O. Widlund D. K.
• Invited speakers
• Participant-initiated minisymposia
• Contributed talks
• Pre-conference short course on software for
  prototyping domain decomposition methods

March
5
Who we are
the PETSc and TAO people
the hypre and Sundials people
the SuperLU and PARPACK people
6
Plus some university collaborators
with a history of lab collaborations in high
performance computing
7
Beyond the on-line Templates guides
www.netlib.org/etemplates
www.netlib.org/templates
124 pp.
410 pp.
these are good starts, but not adequate for
SciDAC scales!
8
Themes affirmed in our May03 review

• Central concept software toolchain
• solutions ? sensitivity, stability, optimization
• modules are nested
• the greatest coding work is in the distributed
  data structures and should be leveraged
• users deal with mathematical objects throughout
  software hides the MPI and ugly details for
  performance
• Ordered goals usability, robustness, algorithmic
  efficiency, and performance efficiency
• Growth industry multirate problems are forcing
  applications codes into implicitness

9
Scope for TOPS

• Design and implementation of solvers
• Time integrators
• Nonlinear solvers
• Optimizers
• Linear solvers
• Eigensolvers
• Software integration
• Performance optimization

(w/ sens. anal.)
(w/ sens. anal.)
10
The power of optimal algorithms

• Advances in algorithmic efficiency rival advances
  in hardware architecture
• Consider Poissons equation on a cube of size
  Nn3
• If n64, this implies an overall reduction in
  flops of 16 million

Year Method Reference Storage Flops
1947 GE (banded) Von Neumann Goldstine n5 n7
1950 Optimal SOR Young n3 n4 log n
1971 CG Reid n3 n3.5 log n
1984 Full MG Brandt n3 n3
11
Algorithms and Moores Law

• This advance took place over a span of about 36
  years, or 24 doubling times for Moores Law
• 224?16 million ? the same as the factor from
  algorithms alone!

12
Where to go past O(N) ?

• Since O(N) is already optimal, there is nowhere
  further upward to go in efficiency, but one
  must extend optimality outward, to more general
  problems
• Hence, for instance, algebraic multigrid (AMG),
  obtaining O(N) in indefinite, anisotropic, or
  inhomogeneous problems

algebraically smooth error
13
TOPS software in SciDAC collabs
current
planned
APDEC Hypre
ASCTKD PETSc
AST PARPACK/SuperLU
CCTTSS TAO, SUNDIALS
CEMM PETSc/Hypre, SUNDIALS
CMRS PETSc/Hypre
CLGT
PERC PETSc
SSC
TSI custom
TSTT TAO
supporting NWChem, MPQC supporting CFRFS
14
TOPS software in SciDAC collabs
current
planned
APDEC Hypre
ASCTKD PETSc
AST PARPACK/SuperLU Veltisto/PETSc
CCTTSS TAO, SUNDIALS PETSc/Hypre
CEMM PETSc/Hypre, SUNDIALS
CMRS PETSc/Hypre
CLGT Hypre, PARPACK
PERC PETSc
SSC PETSc/Hypre
TSI custom PETSc/Hypre, SUNDIALS
TSTT TAO
supporting NWChem, MPQC supporting CFRFS
15
TOPS citations outside of SciDAC
25 jour., proc., theses (2002-04)
Widely distributed software

• Astronomy
• Biomechanics
• Chemistry
• Cognitive Sciences
• Combustion
• Electrical Engineering
• Geosciences
• Hydrodynamics
• Materials Science
• Mechanics
• Micromechanics/Nanotechnology
• Numerical Analysis
• Optics
• Porous Media
• Shape Optimization

• Dspice
• EMSolve
• FEMLAB
• FIDAP
• GlobalArrays
• HP Mathematical Library
• libMesh
• Magpar
• Mathematica
• NIKE
• Prometheus
• SCIRun
• SLEPc
• Snark
• Trilinos

16
Bell prize news

• TOPS PETSc software has been employed in two
  Bell Prizes (special category) in 1999 2003
• 2003 prize geological parameter estimation
  problem
• Forward PDE 17 million unknowns
• Inverse problem 70 billion unknowns (over time
  history)
• 2048 procs of HPAlphaServer for 24 hours

target
reconstruction
17
Poster iconography, 2004
Overview
Iterative Solvers
Integrators
Eigensolvers
Optimizers
Direct Solvers
Implementation Performance
Convergence Performance
18
Who to talk with at this meeting

• Linear solvers
  (Esmond Ng)
• Eigensolvers
  (Esmond Ng, Osni Marques)
• Nonlinear solvers (David
  Keyes)
• ODE/DAEs/sensitivity
  (David Keyes)
• Optimizers
  (Omar Ghattas, Jorge Moré)
• Software integration
  (David Keyes)
• Performance optimization (Victor
  Eijkhout, Rich Vuduc)

Ng, LBNL
19
Who to talk with at this meeting

• Linear solvers
  (Esmond Ng)
• Eigensolvers
  (Esmond Ng, Osni Marques)
• Nonlinear solvers (David
  Keyes)
• ODE/DAEs/sensitivity
  (David Keyes)
• Optimizers
  (Omar Ghattas, Jorge Moré)
• Software integration
  (David Keyes)
• Performance optimization (Victor
  Eijkhout, Rich Vuduc)

Marques, LBNL
20
Who to talk with at this meeting

• Linear solvers
  (Esmond Ng)
• Eigensolvers
  (Esmond Ng, Osni Marques)
• Nonlinear solvers (David
  Keyes)
• ODE/DAEs/sensitivity
  (David Keyes)
• Optimizers
  (Omar Ghattas, Jorge Moré)
• Software integration
  (David Keyes)
• Performance optimization (Victor
  Eijkhout, Rich Vuduc)

Keyes, Columbia
21
Who to talk with at this meeting

• Linear solvers
  (Esmond Ng)
• Eigensolvers
  (Esmond Ng, Osni Marques)
• Nonlinear solvers (David
  Keyes)
• ODE/DAEs/sensitivity
  (David Keyes)
• Optimizers
  (Omar Ghattas, Jorge Moré)
• Software integration
  (David Keyes)
• Performance optimization (Victor
  Eijkhout, Rich Vuduc)

Ghattas, CMU
22
Who to talk with at this meeting

• Linear solvers
  (Esmond Ng)
• Eigensolvers
  (Esmond Ng, Osni Marques)
• Nonlinear solvers (David
  Keyes)
• ODE/DAEs/sensitivity
  (David Keyes)
• Optimizers
  (Omar Ghattas, Jorge Moré)
• Software integration
  (David Keyes)
• Performance optimization (Victor
  Eijkhout, Rich Vuduc)

Moré, ANL
23
Who to talk with at this meeting

• Linear solvers
  (Esmond Ng)
• Eigensolvers
  (Esmond Ng, Osni Marques)
• Nonlinear solvers (David
  Keyes)
• ODE/DAEs/sensitivity
  (David Keyes)
• Optimizers
  (Omar Ghattas, Jorge Moré)
• Software integration
  (David Keyes)
• Performance optimization (Victor
  Eijkhout, Rich Vuduc)

Eijkhout, UT
24
Who to talk with at this meeting

• Linear solvers
  (Esmond Ng)
• Eigensolvers
  (Esmond Ng, Osni Marques)
• Nonlinear solvers (David
  Keyes)
• ODE/DAEs/sensitivity
  (David Keyes)
• Optimizers
  (Omar Ghattas, Jorge Moré)
• Software integration
  (David Keyes)
• Performance optimization (Victor
  Eijkhout, Rich Vuduc)

Vuduc, UC Berkeley
25
TOPS Solver Interface
26
Desiderata

• Simplicity
• as few required concepts as possible
• Generality
• richness of instances of each concept
• Language independence
• callable from anything you want, but easy for us
  to maintain
• High performance
• no forced provision or recopying of data beyond
  what best method needs
• Extensibility
• something we can live with as new algorithms come
  along and have to be integrated underneath

27
Some progenitors

• FEI - finite element interface/C
• developed at SNL
• ESI - equation solver interface/C
• multi-lab effort
• hypre conceptual interfaces/C
• developed at LLNL
• PETSc/C
• developed at ANL

28
TOPS solver interfaces today

• PETSc is an API that calls
• Hypre, SuperLU, PVODE (SUNDIALS), many other
  solvers outside of TOPS responsibility to
  maintain
• TAO, Veltisto call PETSc
• Other interoperability combinations
• e.g., PARPACK calls SuperLU, Hypre calls SuperLU
  and PETSc,
• For linear solvers, want common interface to
  PETSc, Hypre, SuperLU, etc.
• Purpose is not to avoid subroutine call-through
  overhead, but to offer full functionality of each

29
Language independence

• Implemented using LLNLs SIDL (Scientific
  Interface Definition Language), a winning card of
  CCTTSS
• Will not require additional packages
• Does not require learning a new programming
  language
• F77/F90, C/C, Python

30
Abstract interface concepts

• Solver is a labeled object that contains visible
  operator and vector(s), and invisible associated
  storage
• A given application code may have dozens of
  persistent solver objects, each with reusable
  factorizations, preconditioners, coarse grid
  hierarchies, etc.
• Vector represents field data
• Operators and vectors can be accessed with a
  viewer
• Operators and vectors have distributed, blocked
  layouts

31
View

• Allows user to access vector and matrix values in
  the language of the application, e.g, conceptual
  interfaces of Hypre
• Handles any communication of the data
  transparently

32
Accessing values in vectors/matrices
interface View // Begin getting or setting
values void startAccessible() //
Send values to correct (distributed) locations
void endAccessibleStartCommunication() //
Receive values to correct locations void
endAccessibleEndCommunication() // Set all
entries to zero void zero()
Sub-interface specific methods
33
Classical linear algebra view
interface View_Vector_Rn extends TOPS.View
ltdoublegt getValues(ltintgt indices) void
setValues(ltintgt indices, ltdoublegt values)
void addValues(ltintgt indices, ltdoublegt
values) ltdoublegt accessArrayRead()
ltdoublegt accessArrayWrite() void
restoreArray(ltdoublegt values)
Indices represent locations in Rn
34
Structured-grid view
interface View_Vector_S extends TOPS.View
ltdouble,3gt getValues(ltint,3gt indices) void
setValues(ltint,3gt indices, ltdouble,3gt
values) void addValues(ltint,3gt
indices, ltdouble,3gt values) ltdouble,3gt
accessArrayRead() ltdouble,3gt
accessArrayWrite() void
restoreArray(ltdouble,3gt values)
Indices represent locations in Rm x Rn x Rp
35
Views/Layouts

• Rn - classical linear algebra access
• S - single structured grid
• Fe - finite element interface
• Ss - semi-structured grids block-structured grid
  with additional arbitrary node-to-node
  connections

36
For further discussion

• Open source interface to PETSc and Hypre, to
  begin with
• Will not require installing Babel (?)
• Working code available soon
• Comments welcome!

bk//tops.bkbits.net/tops-solver-interface
37
Working with TOPS

• Hundreds of groups around the world use TOPS
  software without directly collaborating with TOPS
  co-PIs
• You can, too, but SciDAC does provide limited
  opportunity for direct collaboration

38
Benefits of working with TOPS

• Apps groups tend to under-employ complicated
  iterative libraries on their own
• underexploitation of available structure
• underexploitation of algorithmic options
• underexploitation of profiling tools
• TOPS thinks of its work as adding options, not
  making changes
• TOPS can often help a lot before adding solver
  options

39
May03 review recommendations

1. Down-select to a smaller set of applications and
   a subset of the algorithms to be implemented in
   the solver software tools.
2. On a high priority basis, allocate resources in
   the later years to algorithm and software
   consultation and deep collaboration with selected
   applications.

40
Requirements for future collaborations

• Version control systems are essential to having
  any lasting impact or insertion path for solver
  improvements
• Automated build system for user code required
• Portability to Linux desktop/laptop required for
  development purposes
• Access to users production machine required for
  performance tuning purposes

41
Expectations TOPS has of users

• Be willing to experiment with novel algorithmic
  choices optimality is rarely achieved (beyond
  model problems) without interplay between physics
  and algorithmics!
• Adopt flexible, extensible programming styles in
  which solver algorithms and data structures are
  not hardwired
• Be willing to let us play with the real code you
  care about, but be willing, if appropriate, to
  abstract out relevant compact tests

42
TOPS success metrics
TOPS users

• Understand range of algorithmic options and their
  tradeoffs (e.g., memory vs. time, inner iteration
  work vs. outer)
• Can try all reasonable options easily without
  recoding or extensive recompilation
• Know how their solvers are performing
• Spend more time in their physics than in their
  solvers
• Are intelligently driving solver research, and
  publishing joint papers with TOPS researchers
• Can simulate truly new physics, as solver limits
  are steadily pushed back (finer meshes, complex
  coupling, etc.)

43
For more information ...
http//www.tops-scidac.org