Patrick Marchesiello

1
Le modèle ROMS et son utilisation sur NYMPHEA
Centre IRD de Bretagne

• Patrick Marchesiello
• Brest, 13 Janvier 2005

2
(No Transcript)
3
ROMS History

• Descendant of SPEM SCRUM (relative of POM)
• (Song Haidvogel 1994 Barnier et al., 1998)
• UCLA more like developers code (Shchepetkin et
  al., 1998, 2003, 2004 Marchesiello et al., 2001,
  2003 )
• http//www.atmos.ucla.edu/cesr/ROMS_page.html
• Rutgers larger user community support
• http//marine.rutgers.edu/po/index.php?modelroms
• IRD Brest UCLA INRIA
• http//www.brest.ird.fr/Roms_tools
• - AGRIF Adaptive Grid Refinement In
  Fortran (Debreu 1999)
• - Pre-processing tools (Penven,
  Marchesiello)

4
Collaborators and Users

• FRANCE
• IRD Brest Penven, Marchesiello et al.
• LMC Grenoble Debreu et al.
• LPO Brest Le Gentil et al.
• USA
• UCLA McWilliams, Shchepetkin, et al.
• JPL Chao et al.
• Rutgers U. Arango et al.
• USERS
• France Brest, Paris, Toulouse, Noumea
• Europe Germany (U. Bremerhaven), Italy (JRC),
  Portugal (IPIMAR), Spain (AZTI)
• Africa Morocco (INRH), Senegal (LPA), South
  Africa (U. Captown)
• America California, Peru (IMARPE), Chili (U.
  Conception), Brazil

5
ROMS Main features

• Hydrostatic, Boussinesq Primitive Equations
• Free surface
• Generalized vertical s-coordinate
• Horizontal curvilinear coordinates
• High order, low dispersion numerics
• Embedded domains AGRIF
• Open boundary conditions
• Boundary layers parameterizations
• Parallelization OMP, MPI
• Domain partitionning
• Optimized for vector computers
• Fortran 95
• UNIX/Linux
• C preprocessor
• NetCDF library, used for all I/O

6
Numerics Motivation
Kantha and Clayson (2000) after Durran (1991)
7
Numerics Strategy

• High order accurate methods
• Sanderson (1998) optimal choice (lower cost for
  a given accuracy) for general ocean circulation
  models is 3RD OR 4TH ORDER accurate methods
• With special care to
• Numerical dispersion
• Pressure gradient
• Mode splitting
• Combination of methods

8
Numerics in ROMS(Shchepetkin McWilliams, 1998,
2003, 2004)

• Horizontal (C) and vertical staggered grids
• Time stepping
• Split-explicit barotropic and baroclinic modes
  with 2-way time filter
• Predictor-corrector Leapfrog-Adams-Molton 3rd
  order scheme with feed-back between momentum
  tracer equations
• Non-uniform density in barotropic mode
• Conservative constancy preserving advection for
  tracers.
• Advection
• 3rd order upstream biased (QUICK)
• Vertical terms
• parabolic spline reconstruction for horiz.
  pressure gradient and advection terms (equivalent
  8th order)
• Implicite Crank-Nicholson scheme for vertical
  mixing terms

9
Numerics Perfomances
ROMS 0.25 deg
POG - 0.25 deg
C. Blanc
C. Blanc
10
ROMS_AGRIF
The same model (executable) runs on grids with
different space/time resolutions
2 20 45 34 59 3 3 3 30 55 70 89 3 3 2 0 1 10 30
20 40 5 3 5 0

• Each domain has its own input/output files
• Grids locations specified in AGRIF_FixedGrids.in
  
• Works in OPENMP/MPI
• Forcings, initial conditions generated with an
  interactive matlab tool  nesting gui 

11
Nymphea
12
Implementation

• Compilation
• Software required Fortran95, Unix, C
  preprocessor, NetCDF library
• Compilation interface in ROMS which defines
  machine dependent options (Tru64 UNIX)
• Parallelisation
• OpenMP 1 knot of 4 processors
• MPI for process studies (S. Le Gentil) needs
  work for realistic applications
• Applications
• Realistic coastal regions of West Africa
  (Morocco and Senegal), Iroise sea,Bay of Brest
• Process studies at high resolution

13
ROMS_AGRIF for West AFRICA
Mercator
Sahara 5 km
W. Africa 25 km
Levitus
C. Blanc
C. Blanc
Clipper
C. Vert
24225232 points dt720s
14
PERFORMANCES COST

• CONFIGURATION
• 2 Embedded grids with refinement coef5
• Size (child grid) 24225232 points with
  dt720s
• Duration of simulation 10 model years
• Processors 1 knot of 4 processors Alpha EV68
  (1GHz)
• Parallelization with OpenMP
• Partitionning 48
• Cost c 6. 10-6 CPU seconds / grid point /
  time step
• (Total run time 15 days)

• Comparisons
• PC Xeon 2.8Ghz c1.10-5
• SGI/CRAY Origin2000 c8.10-5
• Earth Simulator (NEC SX) c5. 10-7

15
PERFORMANCES SCALABILITY

• Nymphea 95 for 1-4 proc.
• SGI/CRAY-Origin2000 95 with saturation above
  128 proc.
• Earth Simulator 95-60 for 1-512 proc.

OMP opt. part.
MPI (1 sub/proc)
OMP (1 sub/proc)
16
Partitioning

• Senegal ideal case on Nymphea
• (P. Estrade)
• Domain 15050040 with dt480s
• Partitioning 11 Cost 7.5 10-6
• Partitioning 164 Cost 6. 10-6
• (units CPU s/ grid point/ time step)
• 25 gain due to optimal cache use

NSUB_E
NSUB_X
New Caledonia region on PC (J. Lefêvre)
100 gain due to optimal cache use
Domain 15917120 with dt480s
17
CONCLUSION

• ROMS is well optimized (code and methods) and
  adapted to Nymphea which allows to perform large
  runs in a reasonable time without excessive
  queuing time
• The model is ready for faster, more numerous
  processors (provided AGRIF is fully tested with
  MPI)
• More storage would be welcome