ACTS A Reliable Software Infrastructure for Scientific Computing

1
- ACTS -A Reliable Software Infrastructure for
Scientific Computing
UC Berkeley - CS267

• Osni Marques
• Lawrence Berkeley National Laboratory (LBNL)
• oamarques_at_lbl.gov

2
Outline

• Keeping the pace with the software and hardware
• Hardware evolution
• Performance tuning
• Software selection
• What is missing?
• The DOE ACTS Collection Project
• Goals
• Current features
• Lessons learned

3
IBM BlueGene/L
A computation that took 1 full year to complete
in 1980 could be done in 10 hours in 1992, in
16 minutes in 1997, in 27 seconds in 2001 and
in 1.7 seconds today!
4
Challenges in the Development of Scientific Codes

• Research in computational sciences is
  fundamentally interdisciplinary
• The development of complex simulation codes on
  high-end computers is not a trivial task
• Productivity
• Time to the first solution (prototype)
• Time to solution (production)
• Other requirements
• Complexity
• Increasingly sophisticated models
• Model coupling
• Interdisciplinarity
• Performance
• Increasingly complex algorithms
• Increasingly complex architectures
• Increasingly demanding applications

• Libraries written in different languages
• Discussions about standardizing interfaces are
  often sidetracked into implementation issues
• Difficulties managing multiple libraries
  developed by third-parties
• Need to use more than one language in one
  application
• The code is long-lived and different pieces
  evolve at different rates
• Swapping competing implementations of the same
  idea and testing without modifying the code
• Need to compose an application with some other(s)
  that were not originally designed to be combined

5
Automatic Tuning

• For each kernel
• Identify and generate a space of algorithms
• Search for the fastest one, by running them
• What is a space of algorithms?
• Depending on kernel and input, may vary
• instruction mix and order
• memory access patterns
• data structures
• mathematical formulation
• When do we search?
• Once per kernel and architecture
• At compile time
• At run time
• All of the above

• PHiPAC www.icsi.berkeley.edu/bilmes/phipac
• ATLAS
• www.netlib.org/atlas
• XBLAS
• www.nersc.gov/xiaoye/XBLAS
• Sparsity www.cs.berkeley.edu/yelick/sparsity
• FFTs and Signal Processing
• FFTW www.fftw.org
• Won 1999 Wilkinson Prize for Numerical Software
• SPIRAL www.ece.cmu.edu/spiral
• Extensions to other transforms, DSPs
• UHFFT
• Extensions to higher dimension, parallelism

6
What About Software Selection?

• Use a direct solver (ALU) if
• Time and storage space acceptable
• Iterative methods dont converge
• Many bs for same A
• Criteria for choosing a direct solver
• Symmetric positive definite (SPD)
• Symmetric
• Symmetric-pattern
• Unsymmetric
• Row/column ordering schemes available
• MMD, AMD, ND, graph partitioning
• Hardware

Build a preconditioning matrix K such that Kxb
is much easier to solve than Axb and K is
somehow close to A (incomplete LU
decompositions, sparse approximate inverses,
polynomial preconditioners, preconditioning by
blocks or domains, element-by-element, etc). See
Templates for the Solution of Linear Systems
Building Blocks for Iterative Methods.
7
Components simple example
8
The DOE ACTS Collection
http//acts.nersc.gov

• Goals
• Collection of tools for developing parallel
  applications
• Extended support for experimental software
• Make ACTS tools available on DOE computers
• Provide technical support (acts-support_at_nersc.gov)
  
• Maintain ACTS information center
  (http//acts.nersc.gov)
• Coordinate efforts with other supercomputing
  centers
• Enable large scale scientific applications
• Educate and train

• High Performance Tools
• portable
• library calls
• robust algorithms
• help code optimization
• More code development in less time
• More simulation in less computer time

9
Current ACTS Tools and their Functionalities
10
Use of ACTS Tools
Advanced Computational Research in Fusion (SciDAC
Project, PI Mitch Pindzola). Point of contact
Dario Mitnik (Dept. of Physics, Rollins College).
Mitnik attended the workshop on the ACTS
Collection in September 2000. Since then he has
been actively using some of the ACTS tools, in
particular ScaLAPACK, for which he has provided
insightful feedback. Dario is currently working
on the development, testing and support of new
scientific simulation codes related to the study
of atomic dynamics using time-dependent close
coupling lattice and time-independent methods. He
reports that this work could not be carried out
in sequential machines and that ScaLAPACK is
fundamental for the parallelization of these
codes.
11
Use of ACTS Tools
12
Use of ACTS Tools
13
ScaLAPACK software structure
http//acts.nersc.gov/scalapack
Version 1.7 released in August 2001 recent NSF
funding for further development.
ScaLAPACK
PBLAS
Global
Parallel BLAS.
Local
LAPACK
BLACS
Linear systems, least squares, singular value
decomposition, eigenvalues.
Communication routines targeting linear algebra
operations.
platform specific
BLAS
MPI/PVM/...
Clarity,modularity, performance and portability.
Atlas can be used here for automatic tuning.
Communication layer (message passing).
14
PBLAS
(Parallel Basic Linear Algebra Subroutines)

• Similar to the BLAS in portability, functionality
  and naming
• Level 1 vector-vector operations
• Level 2 matrix-vector operations
• Level 3 matrix-matrix operations
• CALL DGEXXX( M, N, A( IA, JA ), LDA, ... )
• CALL PDGEXXX( M, N, A, IA, JA, DESCA, ... )
• Built atop the BLAS and BLACS
• Provide global view of
• the matrix operands

BLAS
PBLAS
array descriptor (see next slides)
15
BLACS
(Basic Linear Algebra Communication Subroutines)

• A design tool, they are a conceptual aid in
  design and coding.
• Associate widely recognized mnemonic names with
  communication operations. This improves
• program readability
• self-documenting quality of the code.
• Promote efficiency by identifying frequently
  occurring operations of linear algebra which can
  be optimized on various computers.

16
BLACS basics

• Processes are embedded in a two-dimensional grid.
• An operation which involves more than one sender
  and one receiver is called a scoped operation.

Example a 3x4 grid
17
ScaLAPACK data layouts

• 1D block and cyclic column distributions
• 1D block-cycle column and 2D block-cyclic
  distribution
• 2D block-cyclic used in ScaLAPACK for dense
  matrices

18
ScaLAPACK 2D Block-Cyclic Distribution
5x5 matrix partitioned in 2x2 blocks
2x2 process grid point of view
19
2D Block-Cyclic Distribution
http//acts.nersc.gov/scalapack/hands-on/datadist.
html
20
ScaLAPACK array descriptors
SUBROUTINE PSGESV( N, NRHS, A, IA, JA, DESCA,
IPIV, B, IB, JB, DESCB, INFO )

• Each global data object is assigned an array
  descriptor
• The array descriptor
• Contains information required to establish
  mapping between a global array entry and its
  corresponding process and memory location (uses
  concept of BLACS context).
• Is differentiated by the DTYPE_ (first entry) in
  the descriptor.
• Provides a flexible framework to easily specify
  additional data distributions or matrix types.
• User must distribute all global arrays prior to
  the invocation of a ScaLAPACK routine, for
  example
• Each process generates its own submatrix.
• One processor reads the matrix from a file and
  send pieces to other processors (may require
  message-passing for this).

21
Array Descriptor for Dense Matrices

22
ScaLAPACK Functionality
23
On line tutorial http//acts.nersc.gov/scalapack
/hands-on/main.html
24
Global Arrays (GA) Wrappers
http//www.emsl.pnl.gov/docs/global/ga.html

• Simpler than message-passing for many
  applications
• Complete environment for parallel code
  development
• Data locality control similar to distributed
  memory/message passing model
• Compatible with MPI
• Scalable

• Distributed Data data is explicitly associated
  with each processor, accessing data requires
  specifying the location of the data on the
  processor and the processor itself.
• Shared Memory data is an a globally accessible
  address space, any processor can access data by
  specifying its location using a global index.
• GA distributed dense arrays that can be accessed
  through a shared memory-like style.

25
TAU Tuning and Performance Analysis

• Multi-level performance instrumentation
• Multi-language automatic source instrumentation
• Flexible and configurable performance measurement
• Widely-ported parallel performance profiling
  system
• Computer system architectures and operating
  systems
• Different programming languages and compilers
• Support for multiple parallel programming
  paradigms
• Multi-threading, message passing, mixed-mode,
  hybrid
• Support for performance mapping
• Support for object-oriented and generic
  programming
• Integration in complex software systems and
  applications

26
Definitions Profiling

• Profiling
• Recording of summary information during execution
• inclusive, exclusive time, calls, hardware
  statistics,
• Reflects performance behavior of program entities
• functions, loops, basic blocks
• user-defined semantic entities
• Very good for low-cost performance assessment
• Helps to expose performance bottlenecks and
  hotspots
• Implemented through
• sampling periodic OS interrupts or hardware
  counter traps
• instrumentation direct insertion of measurement
  code

27
Definitions Tracing

• Tracing
• Recording of information about significant points
  (events) during program execution
• entering/exiting code region (function, loop,
  block, )
• thread/process interactions (e.g., send/receive
  message)
• Save information in event record
• timestamp
• CPU identifier, thread identifier
• Event type and event-specific information
• Event trace is a time-sequenced stream of event
  records
• Can be used to reconstruct dynamic program
  behavior
• Typically requires code instrumentation

28
TAU Example 1 (1/4)
http//acts.nersc.gov/tau/programs/psgesv
29
TAU Example 1 (2/4)


30
TAU Example 1 (3/4)
psgesvdriver.int.f90
PROGRAM PSGESVDRIVER ! ! Example Program
solving Axb via ScaLAPACK routine PSGESV ! !
.. Parameters .. ! a bunch of things omitted
for the sake of space ! .. Executable
Statements .. ! ! INITIALIZE THE PROCESS
GRID ! integer profiler(2) save
profiler call TAU_PROFILE_INIT()
call TAU_PROFILE_TIMER(profiler,'PSGESVDRIVER')
call TAU_PROFILE_START(profiler) CALL
SL_INIT( ICTXT, NPROW, NPCOL ) CALL
BLACS_GRIDINFO( ICTXT, NPROW, NPCOL, MYROW, MYCOL
) ! a bunch of things omitted for the sake
of space CALL PSGESV( N, NRHS, A, IA,
JA, DESCA, IPIV, B, IB, JB, DESCB,
INFO ) ! a bunch of things omitted for
the sake of space call
TAU_PROFILE_STOP(profiler) STOP END
31
TAU Example 2 (1/2)
http//acts.nersc.gov/tau/programs/pdgssvx
tau-multiplecounters-mpi-papi-pdt
32
TAU Example 2 (2/2)
PAPI provides access to hardware performance
counters (see http//icl.cs.utk.edu/papi for
details and contact acts-support_at_nersc.gov for
the corresponding TAU events). In this example we
are just measuring FLOPS.
33
Who Benefits from these tools?
http//acts.nersc.gov/AppMat
Enabling sciences and discoveries with high
performance and scalability...
... More Applications
34
http//acts.nersc.gov

• High Performance Tools
• portable
• library calls
• robust algorithms
• help code optimization
• Scientific Computing Centers
• Reduce users code development time that sums up
  in more production runs and faster and effective
  scientific research results
• Overall better system utilization
• Facilitate the accumulation and distribution of
  high performance computing expertise
• Provide better scientific parameters for
  procurement and characterization of specific user
  needs

Tool descriptions, installation details,
examples, etc
Agenda, accomplishments, conferences, releases,
etc
Goals and other relevant information
Points of contact
Search engine

• VECPAR 2006
• ACTS Workshop 2006

35
Journals Featuring ACTS Tools
September 2005 Issue
36
ACTS Numerical Tools Functionality
37
ACTS Numerical Tools Functionality
38
ACTS Numerical Tools Functionality
39
ACTS Numerical Tools Functionality
40
ACTS Numerical Tools Functionality
41
ACTS Numerical Tools Functionality
42
ACTS Numerical Tools Functionality
43
ACTS Numerical Tools Functionality
44
ACTS Numerical Tools Functionality
45
ACTS Tools Functionality
46
ACTS Tools Functionality