ScalES

1
ScalES OASIS4 _at_AWI


K. Fieg, W. Hiller Alfred Wegener Institute
for Polar and Marine Research Bremerhaven, Germa
ny
2
ScalES

• Scalable ESM for high productivity climate
• simulations
• BMBF funded project, started in Jan. 2009
• gives the opportunity, to study scalability of
  ESMs
• (-gt COSMOS) on many- and multiprocessor
• systems (IBM Blizzard, DKRZ).
• focus on then technical part of code
  optimization
• (not physical!)

3
ScalES identified reasons for performance
reduction

• Parallel I/O sequential data transport between
  storage
• and the (highly?) parallel
  application
• Load Balancing physical processes and
  communication
• requests can lead to a
  serious imbalance
• in different
  partitions of the model domain
• Coupler a serial coupler (e.g. OASIS3) will
  become
• the bottleneck for highly
  parallel applications
• (up to 2000 CPUs) -gt Amdahls
  Law
• System Architecture communication in the model
  is not
• optimized for the use of the
  specific computers
• communication network

Result optimized version of COSMOS in Dec. 2011
4
ScalES
Topics Sustainability Parallel I/O Load
Balancing System architecture Coupler, AP3
Partners DKRZ MPI-M MPI-Chemie Uni
Karlsruhe AWI IBM (NEC IT)
5
OASIS3 vs. OASIS4 general features

• fully parallel coupler
• Beta plus status
• 3 user groups up to now
• automatic repartitioning of data
• couples 1d 2d 3d fields
• unlimited in number of
• - exchange fields
• - any high resolution
• - any coupling frequency

• Serial coupler
• Well tested
• 15 user groups
• not very efficient
• couples only 2d fields
• limited in number of
• - exchange fields
• - resolution
• - low coupling frequency

6
OASIS3 vs. OASIS4 general remarks

• Just a short reminder
• Amdahls Law serial Coupler time takes 5
• of the overall serial computing time budget of
  a
• coupled model setup ---gt
• limits the speedup to 20 even with infinite
• number of processors in the parallel case
• DKRZ IBM Power6 gt 6000 CPU cores!

7
OASIS4 - Example
8
BUT up to now, the serial coupler OASIS3 is
included in most COSMOS applications
OASIS4
9

OASIS4

Consequence (1) We see a strong need for all
groups, which need to have an (even) reasonable
performance on many- and multi processor systems
to switch from OASIS3 to
OASIS4 with their COSMOS applications!

If you are interested in using OASIS4, please
let us know!!!
10

Unstructured grids _at_ OASIS4
Furthermore, we see an increasing demand to
include unstructured grids into ESMs. Neither
OASIS3 nor OASIS4 is supporting the coupling from
or to unstructured grids Consequence (2) -gt
including unstructured grids into OASIS4 is one
of AWIs tasks in ScalES

11

Coupling unstructured grids problems
The coupling of unstructured grids was not tried
yet for large scale climate model components


unpredictable things can happen!
12

Coupling unstructured grids problems Ratio of
source processes target processes can vary
significantly, mapping of partioned grid geometry
is complicated

Processor distribution In regular atmospheric
model
This can lead to strong load imbalance
Need to implement appropriate, performant search
algorithms interpolation schemes into OASIS4

• 1 2 3 4 5
• 7 8 9 10
• 12 13 14 15
• 16 17 18 19 20

13

Coupling unstructured grids problems mass
conservation coupling algorithm has to
consider - non matching grids
at land / ocean interface -
different element shapes ( / )


We have to deal with mainly the same problems,
but in a more extreme way
model drift Coupled values will be calculated in
one grid, passed over to the second grid and
distributed there.
14

First step Coupling unstructured grid to
intermediate structured grid - 2 D lon / lat
structured intermediate grid with appropriate
resolution, properties cell centered
unstructured




Promised to be finished until end of the year






coupled via OASIS4
to other components
15

Further steps after interpolation to
intermediate 2d lon / lat grid

• direct coupling!!!!
• extending coupler interface to unstructured
  grids by
• amending unstructured grid nomenclatures and
  extending
• the existing xml communication scheme of OASIS4
• integration of additional interpolation routines
  like cell
• integrated interpolation scheme (e.g. CISL)
• testing different model setups optimizing code
  for HPC
• secure portability by platform independency

16



Thank you
17

As a summary, coupling unstructured grids means

• dealing with a similar class of problems as
  coupling
• structured grid
• the administration of the unstructured geometry
  requires
• an additional type of data structure
• special care in transformation, mass
  conservation
• and interpolation is needed to avoid model
  drift
• setting up a model run needs more manual
  interaction
• strong coordination work is needed to integrate
• unstructured grid concepts into existing coupler

18
interaction with other projects

• like STORM.
• Other projects are also highly interested to
  
• use OASIS4 as reliable coupler for their ESM
  
• -gt collaboration and communication is
  necessary
• 2. Continuous development of Community Model
  FEOM according to code and infrastructure (pre-
  and post processing)
• -gt regular updates of release version have
  to
• be expected

19
My preliminary ToDo List

• - metadata standards for unstructured grids have
  to
• be defined and discussed in the community
• a way to pass additional grid information to
  OASIS4
• has to be discussed with the OASIS developer
• implementing performant search algorithms (e.g.
  CISL)
• into the OASIS4 code
• secure load balancing also for very unbalanced
  element
• distribution in the grids
• secure mass conservation also for coupling to
  strange
• element shapes
• find transformation routines appropriate for
  unstructured
• grids