PETSc

1
PETSc

• Portable, Extensible Toolkit for Scientific
  computing

2
Overview

• Parallel Sequential
• Object-oriented
• Available for virtually all UNIX platforms, as
  well as Windows 95/NT
• Flexible many options for solver algorithms and
  parameters

3
Motivation

• Developing parallel, non-trivial PDE solvers that
  deliver high performance is still difficult and
  requires months (or even years) of concentrated
  effort.
• PETSc is a toolkit that can ease these
  difficulties and reduce the development time

4
Introduction

• A freely available and supported research code
• Available via http//www.mcs.anl.gov/petsc
• Free for everyone, including industrial users
• Hyperlinked documentation and manual pages for
  all routines. Many tutorial-style examples
• Support via email petsc-maint_at_mcs.anl.gov
• Usable from Fortran 77/90, C, and C
• Portable to any parallel system supporting MPI,
  including
• Tightly coupled systems
• Cray T3E, SGI Origin, IBM SP, HP 9000, Sun
  Enterprise
• Loosely coupled systems, e.g., networks of
  workstations
• Compaq, HP, IBM, SGI, Sun, PCs running Linux or
  Windows

5
History

• PETSc history
• Begun in September 1991
• Keep on development
• Now over 8,500 downloads since 1995 (versions
  2.0 and 2.1)

6
PETSc Concepts

• How to specify the mathematics of the problem
• Data objects
• - vectors, matrices
• How to solve the problem
• Solvers
• - linear, nonlinear, and time stepping (ODE)
  solvers
• Parallel computing complications
• Parallel data layout
• - structured and unstructured meshes

7
Structure of PETSc
8
PETSc Numerical Components
Nonlinear Solvers
Time Steppers(For ODE)
Newton-based Methods
Other
Euler
Backward Euler
Pseudo Time Stepping
Other
Line Search
Trust Region
Krylov Subspace Methods
GMRES
CG
CGS
Bi-CG-STAB
TFQMR
Richardson
Chebychev
Other
Preconditioners
Additive Schwartz
Block Jacobi
Jacobi
ILU
ICC
LU (Sequential only)
Others
Matrices
Compressed Sparse Row (AIJ)
Blocked Compressed Sparse Row (BAIJ)
Block Diagonal (BDIAG)
Dense
Other
Matrix-free
9
What is not in PETSc?

• Discretizations
• Unstructured mesh generation and refinement tools
• Load balancing tools
• Sophisticated visualization capabilities

10
BUT!

• PETSc has interface to external software that
  provides some
• of this functionality
• Linear solvers (AMG BlockSolve95 LUSOL SPAI
  SuperLU)
• Optimization software (TAO, Veltisto)
• Mesh and discretization tools (overture, SAMRAI)
• ODE solvers (PVODE)
• Others (Matlab, ParMETIS)

11
Flow of Control for PDE Solution
Main Routine
Timestepping Solvers (TS)
Nonlinear Solvers (SNES)
Linear Solvers (SLES)
PETSc
PC
KSP
Application Initialization
Function Evaluation
Jacobian Evaluation
Post- Processing
PETSc code
User code
12
A simple PETSc program

• include petsc.hint main( int argc, char
  argv ) int rank
• PetscInitialize(argc,argv,PETSC_NULL,PETS
  C_NULL)
• MPI_Comm_rank(PETSC_COMM_WORLD,rank )
  PetscSynchronizedPrintf(PETSC_COMM_WORLD,
• Hello World from
  d\n,rank)
• PetscFinalize() return 0

13
Sparse Matrix Computation in PETSc

• Variety Data Structures
• Variety Preconditioners
• Variety Iterative Solvers
• Variety interface to External Softwares

14
Data Structure for sparse matrix

• Compressed Sparse Row
• Blocked compressed sparse Row
• Block Diagonal

15
Example(BCRS)

• Block Compressed Sparse Row
• NR Number of blocks
  per side of the matrix
• NNZB
  Number of Non-Zero Blocks
• NB Number
  of elements per side of the Block
• N Number of elements per side of the
  matrix

16
Preconditioners

• Incomplete LU factorization
• Jacobi, Gauss-Seidel,SOR
• Schwarz within blocks use LU, ILU, SOR, etc.

17
Solvers

• Direct method (LU)
• Krylov method (CG, GMRES, BiCGstab, CGS, QMR, )
• Non-Krylov iterative method (Jacobi, G-S, SOR)
• Pre-conditioned Krylov method

18
Interface to external solver

• Both Iterative and Director solvers
• AMG
• BlockSolve95
• LUSOL
• SPAI
• SuperLU

19
Functions

• Define the linear system (Axb)
• MatCreate(), MatSetValue(), VecCreate()
• Create the Solver
• SLESCreate(), SLESSetOperators()
• Solve System of Equations
• SLESSolve()
• Clean up
• SLESDestroy()

20
Example

• Solve Ax b
• www.cs.ucsb.edu/jh/example.cc

21
ComparisonI Iterative methods provided for
different packages
22
ComparisonII Preconditioners provided by
different packages
23
ComparisonIII Performance

• Test Matrix

24
ComparisonIII Performance

• Feed these matrices to Aztec and PETSc
• Generate a zero vector as initial guess and a
  vector of 1 as RHS
• Use GMRES without preconditoners, max iteration
  500
• Run on 512-processor Cray T3E900

25
ComparisonIII Performance
26
ComparisonIII Performance
27
ComparisonIII Performance

• Aztect suffers from setup varization
• PETSc did not optimize partition algorithm( PE256
  -gt PE512, performance decrease). Communication
  takes more time than computation

28
Conclusion

• PETSc
• - Well designed and widely used
• - First MPI-based program to public
• - Good set of iterative methods and
  preconditioners
• - Good Support and excellent technical
  documents
• - Still under developement

29
Thanks

• Comments ?