A dynamic parallel coupler

1

• A dynamic parallel coupler
• E-Mail palm_at_cerfacs.fr
• URL http//www.cerfacs.fr/palm

Andrea Piacentini
Rennes - November 2007
2
Overview

• The genesis of the PALM project
• Code coupling issues
• PALM main features
• How to set up a PALM coupled application
• Some PALM applications

3
The genesis of the PALM project
Origin of the PALM project in 1996 with the
operational ocean forecasting project
MERCATOR. A data assimilation suite can be
designed and implemented as a coupling. The
different tasks - running a forecast, apply the
observation operator, computing the misfit,
approximate the error statistics, invert
matrices, minimize a cost function and so on -
are thought of as independent pieces of code to
be assembled within a portable, flexible and
efficient framework.
4
The genesis of the PALM project

• The PALM Team at CERFACS was in charge of the
  development of the software backbone of the
  MERCATOR project.
• First a fully featured prototype was implemented
  using the available technology at the end of the
  90's. At the issue of this phase the first
  version of the coupler was released with the name
  PALM_RESEARCH.
• Based on the expertise developed on the
  prototype, the final MPI2 based version of PALM
  was designed and implemented. It is currently
  released with the name PALM MP.
• Thanks to the flexibility of the approach, the
  non-specific formalism used to describe the
  coupling interfaces and the portability of the
  software, PALM established itself as a natural
  choice for all sorts of dynamic parallel coupling
  and code assembling projects.

5
Overview

• The genesis of the PALM project
• Code coupling issues
• PALM main features
• How to set up a PALM coupled application
• Some PALM applications

6
Code coupling issues

• To build modular applications by assembling their
  elementary components (data assimilation)
• Split the problem in order to reach a better
  task scheduling, taking advantage of the
  intrinsic parallelism of the application

7
Code coupling issues

• What does it imply?
• Manage executions (serial or parallel)
• Drive the information exchanges between codes

Commitments -Easy set-up non intrusive
interfaces -Efficiency no loss in performances
-Portability standard technical
solutions -Flexibility test different
configurations with a few changes -Genericity
reuse components in other couplings
8
Code coupling issuesImplementation

• First solution code merging

prog1 prog2
Program prog1 Call sub_prog2(in, out) end
Program prog2 Subroutine sub_prog2(in,out) end

• Very efficient in computing time (data exchange
  by reference, )
• BUT
• Integration problems (the code independency is
  not respected) common blocks, logical units,
  compilation options, languages,
• No flexibility (problems in maintenance,
  evolution, )
• Potential memory wastes
• Task parallelism cannot be implemented

• N.B. This is merging and not coupling
  (parallelism not taken into account)

9
Code coupling issuesImplementation

• Second solution use a communication mechanism
  (PVM, MPI, CORBA, pipe UNIX, files, )

Prog1
Prog2
Program prog2 Call XXX_recv(prog1, data) end
Program prog1 Call XXX_send(prog2, data) end

• More or less efficient depending on the
  mechanism
• BUT
• Coupling is not generic
• Good experience in parallel computing required
• Lack of flexibility (parallelism, interpolation,
  )
• Not always portable depending on the chosen
  mechanism
• Too complex with more than two codes and many
  exchanges

10
Code coupling issuesImplementation
OASIS
Third solution choose a coupler
PALM
Prog1
Prog2

• The coupler starts the executables in a sequence
  or in parallel
• Data exchanges are invoked by general primitives
  PUT/GET
• The coupler uses a portable and efficient
  communication protocol (MPI)
• The coupler provides interpolation tools, data
  redistributions,
• Huge gain in integration ease, performances,
  maintenance and evolution of the coupled
  application.

11
Overview

• The genesis of the PALM project
• Code coupling issues
• PALM main features
• How to set up a PALM coupled application
• Some PALM applications

12
PALM main features
First function of the coupler Managing the
components execution
13
Spawning and execution
Classical couplers
PALM coupler
Static coupling
Dynamic coupling
condition
Code1
Loop
time
Code4
Code3
Code2
Both codes are started at the beginning of the
simulation and exit together at the end of the
application Example Ocean/Atmosphere coupling
in climate modelling
Parallel or serial executions
Loops and conditional executions
Complex algorithms can be easily end efficiently
implemented
Dynamic resources management (processors and
memory)
14
Spawning and execution
How to describe complex algorithms?

• A Graphic User Interface for
• Representing the components
• Handling the parallelism
• Describing loops and conditions
• Describing the communications (exchanges)

• With all the benefits of a G.U.I. as, for
  instance
• Coherency checks on the exchanged data
• Pictorial flow chart representation of the
  algorithm
• Performance analysis, debugging and run-time
  monitoring

15
Snapshot of the G.U.I.
16
PALM glossary

• Unit computational component
• - Code
• - Module of a code
• - Data loaders
• - Pre-defined units (F77/F90 subroutines, C or
  C functions, Java, Python, Matlab and so on
  through F90 or C/C interfaces)

32

• Parallel unit

17
PALM glossary

• Branch sequence of units and instructions gt
  algorithm

18
PALM main featuresTwo levels of parallelism
Task parallelism branches Parallel computing
simply drawing
Distributed components units
Components assembling blocks
19
PALM main features
Second function of the coupler Driving the data
exchanges
20
PALM main features
End point communication paradigm
? NO explicit indication of the receiving units
on the producer side, NOR of the producer on
the receivers side
In the sources of the units potential sending or
reception
Announce data production PROVIDER CALL
PALM_Put () Register a data request CLIENT
CALL PALM_Get ()
21
PALM glossary

• Object data chunk produced or received by a unit

N.B. Packed data for coherent data batches can
simplify the data flow representation
22
PALM main featuresCommunication mechanism
Actual communications are described by connecting
the plugs in the graphic interface
Handshaking between clients and providers
23
PALM main featuresCommunication mechanism
A communication takes place only if
24
PALM main featuresIndependent units

• Parallel data exchanges
• Automatic remapping of the objects if the
  distribution on the provider side is different
  from the distribution on the client side

25
PALM main featuresCommunication mechanism
Optional use of files for too large objects
26
Overview

• The genesis of the PALM project
• Code coupling issues
• PALM main features
• How to set up a PALM coupled application
• Some PALM applications

27
How to set up a PALM coupled application
Preliminary step How a code becomes a PALM
unit ?
28
How a code becomes a PALM unit

• Pre-requisites
• Code sources should be available
• (it is possible to couple black boxes, but
  efficiency is much lower)
• Codes have to be written in a compiled language
  (C, C, FORTRAN 77 ou 90) or interfaced with
  such languages (Python, Java)
• Codes must run under Linux/Unix

29
How a code becomes a PALM unit
For all kinds of units

• Replace PROGRAM (FORTRAN) by subroutine or main
  (C) by a function name

For a parallel unit

• Skip the calls to MPI_INIT and MPI_FINALIZE
• Replace MPI_COMM_WORLD by PL_COMM_EXEC
• Replace the calls to STOP, EXIT by PALM_Abort

30
How a code becomes a PALM unit
Create an identity card
!PALM_UNIT \ ! -name unit name\ !
-functions function to be
invoked ! -comment optional
comment ! !PALM_SPACE \ ! -name
space name \ ! -shape (.,.,
,.) \ ! -element_size PL_INTEGER,
REAL, DOUBLE_PRECISION, \ ! -comment
optional comment ! !PALM_OBJECT \ !
-name object name \ ! -space
name of its space \ ! -intent
IN, OUT, INOUT\ ! -time ON,
NO \ ! -tag ON , NO \ !
-comment optional comment
! !PALM_DISTRIBUTOR ! -name
distributor name \ ! -type
regular, custom \ ! -nbproc
number of processes in the distribution \ !
-function name of the distribution
function \ ! -comment optional
comment
31
How a code becomes a PALM unit
Add the PALM primitives
CALL PALM_Put (space1, obj1, time, tag ,
array, error) CALL PALM_Get (space2, obj2,
time, tag , array, error)
32
Pre-defined PALM algebra units
Definition
Pre-defined units interfacing common mathematical
libraries (BLAS, LAPACK, ) applied on the fly to
objects exchanged between units. (Interesting for
units conversion, grid to grid interpolation, )
Users can compose (and share) their specific
algebra units.
33
How to set up a PALM coupled application
Main step Once the units have been defined,
the coupled application (algorithm and
communications) is described in the graphic user
interface PrePALM
34
Description of the coupling algorithm
In the graphic interface the user define the
execution sequences (branches), with control
structures
Observations Branch
Model Branch
35
Description of the communication fluxes
The actual communications are described in the
graphic interface by linking the plugs
corresponding to PALM_Put and PALM_Get
KEY POINT Analysis of the algorithm data flow
and coherent definition of the interfaces between
sub-models
N.B. PALM handles time interpolations for
objects produced by units with a different time
step.
36
Compile and run a PALM_MP application
37
Debugging, analysis and monitoring tools

• A PALM output file per branch with different
  verbosity levels for different classes of messages

• A user defined debugging function, called on the
  PALM_Put and PALM_Get sides of an active
  communication to check the coherency of the
  objects

• A performance analyser accounting for the elapsed
  and the CPU time

• A graphic post-mortem replay or run-time
  monitoring of the application

38
Performance analyser in PrePALM
39
Post-mortem replay and run-time monitoring
40
To sum up
PALM is a software dealing with couplings or with
complex applications built upon sub-modules with
no loss of performances.
A friendly and simple G.U.I. to describe,
implement and supervise applications.

• Many pros
• Evolution and maintenance of a complex
  application and its components are easier
• A framework for the integration of existing
  independent codes in an application (sub-models
  assembling, multi physics coupling, collaborative
  development, )
• Take the best advantage from the intrinsic
  parallelism of an application

41
Overview

• The genesis of the PALM project
• Code coupling issues
• PALM main features
• How to set up a PALM coupled application
• Some PALM applications

42
PALM in the SEVE project
Sol Eau Végétation Energie integrated modelling
of land surface processes at different space and
time scales (water and carbon cycles)
Pool of actors INRA CSE (Avignon) , INRA
EPHYSE (Bordeaux) , LISAH (Montpellier) , CESBIO
(Toulouse) , CNRM/GMME (Toulouse), LTHE,
CEMAGREF, CERMICS
Ongoing actionsComplex landscapes description
and integrated interactions modelling
Segmentation of landscapes Hetherogeneity,
different scales, discontinuities and linear
structuresMulti-physics and multi-scale
coupling, dynamic time stepping3D transferts
modelling in different compartments at a
landscape scale Models enhancement data
assimilation from teledetectionData bases
43
PALM for NitroScape ??
44
PALM in the ANR corayl projectCoupling
combustion and radiatif
chambre de combustion très hautes températures.
phénomènes radiatifs prépondérants. calcul
massivement parallèle.
PhD Jorge Amaya
CERFACS EM2C (Ecole Centrale Paris) LGPSD
(Ecole des mines dAlbi)
DOMASIUM
AVBP
Radiation equations in complex geometry
Reactive Filtered Navier-Stokes equations.
Two levels of parallelism. Independent coupled
models
Energy sources. Thermal fluxes on the walls.
TemperatureSpecies
Coupling scheme n CFD time steps for each
radiative step
Parallel model
Parallel model
CFD - LES
Radiative model
45
Model nestingthe COMODO-PALM project
IMAG/CNRS - Laurence Viry
OPA
1/3 NATL3
OPA
North Atlantic
1/15 BABY15
Gascogne Bay
With PALM
Before PALM

• The components communicate by message passing
  (MPI)
• Very aggressive method, with deep specific
  changes in the source coedes rewriting at every
  model or algorithm update
• Implementation of the two levels of parallelism
  rather cumbersome

• Independent coupled components (MPI2)
• Flexibility, modularity, performances
• Run-time components spawning
• Loops around parts of the application (iterative
  algorithms of domain decomposition)
• Coupled units does not depend on the nesting
  algorithm
• Automatic parallel communications
• Code reuse in new developments

46
Temperature forecast in combustion chambers
Eric Mercier (SNECMA Moteur), Florent Duchaine
Itération 1
Itération 3
47
Shape optimisation MIPTO (INTELLECT D.M.)
Florent Duchaine CERFACS - TURBOMECA
48
RANS LES coupling - Florent Duchaine
Code reuse
49
PALM in MERCATOR
Operational oceanography
All the data assimilation suites have been
developed with PALMThe same methods are used for
the different resolutions. The upgrades of the
direct model do not impact on assimilationThe
PALM_MP coupled implementation is less memory
consuming than a standard implementation
50
PALM in the ADOMOCA project
Assimilation de DOnnées dans les MOdèles de
Chimie Atmosphérique
Pool of the main actors in the atmospheric
chemistry french research SA, LA, LSCE,
LPCE, LMD , IPSL, CERFACS, L3AB, CNRM, ACRI, CNES
Choice of a common tool for sharing components
models and operators methods algorithms and
diagnostics
3D-FGAT
The assimilation algorithm is edited in the
graphic interface PrePALM no changes in the
source codes. In this case, new units are
plugged in e.g. the tangent linear and adjoint
models
CERFACS
CNRM
51
Assimilation de DOnnées NeutronIqueS
PALM in the ADONIS project

• Aim best use of the measurements collected in
  the nuclear reactors for the tuning and
  improvement of the kernel modelling
• 3 applications
• Manara Best estimate of the physical fields by
  variational assimilation
• Kafeine Optimal tuning of the EDF neutronic
  numerical model
• PhD thesis  Compare different data assimilation
  algorithms without the constraints of industrial
  direct application.

52
More info
PALM is portable on most platforms (linux PCs,
unix workstations, mainframe, supercomputers). We
b page http//www.cerfacs.fr/palm Training
At CERFACS 2-3 times/year Or on demand E-mail
palm_at_cerfacs.fr morel_at_cerfacs.fr