GENIE Earth System Modelling workshop

1
(No Transcript)
2
Outline

• PRISM
• the project
• the Areas of Expertise
• OASIS
• historical background
• the community today
• some key notes
• The OASIS4 coupler
• model adaptation
• component model description
• coupled model configuration
• communication
• regridding/transformations
• grids supported
• future developments and conclusions

3
PRISM the project

• Promote the development of a standard
  software infrastructure for
  Earth system coupled modelling
• (http//prism.enes.org)

Help climate modellers spend more time on science
4
PRISM the project

• 2001-2004 the PRISM EU project
• a European project funded for 4.8 M by the EC
• 22 partners
• 2005-2008 the PRISM Support Initiative
• 7 partners CERFACS, CGAM, CNRS, ECMWF, MPI-MD,
  UK MetOffice, NEC-CCRLE
• 8 associate partners CRAY, IPSL, Météo-France,
  MPI-M, NEC-HPCE, SGI, SMHI, SUN
• investing their own manpower in PRISM (8 py/y for
  3 years)

5
PRISM the Areas of Expertise

• 2006 reorganisation in 5 PRISM Areas of
  Expertise (PAE)
• Promotion and, if needed, development of software
  tools for ESM
• Organisation of related network of experts
• Technology watch
• Promotion of community standards
• Coordination with other international efforts
• 1. Code coupling and I/O (S. Valcke, CERFACS)
• 2. Integration and modelling environments (M.
  Carter, UK Met Office)
• 3. Data processing, visualisation management
  (M. Lautenschlager,MD)
• 4. Meta-data (L. Steenman-Clark, CGAM)
• 5. Computing issues (R. Redler, NEC-CCRLE M.-A.
  Foujols, IPSL)

6
PRISM the Areas of Expertise

• 1. Code coupling and I/O
• OASIS3 and OASIS4 couplers, the PALM coupler
  
• 2. Integration and modelling environments (source
  version control, compilation, job configuration
  running)
• Standard Compiling and Running Environments
  (SCE/SRE)
• GUI for model configuration prepIFS, prepOASIS4
• GUI for job monitoring SMS
• FMS software control and compilation management
• 3. Data processing, visualisation and management
  includes data networking and federated archive
  architecture
• CERA database (Hamburg)
• 4. Meta-data definition of metadata standards
  with other ESM projects
• 5. Computing issues petacomputing, performance,
  portability, parallelism, algorithmic
  developments

7
OASIS historical background

• OASIS1, OASIS2, OASIS3
• low resolution, low number of 2D fields, low
  coupling frequency
• flexibility very important, efficiency not so
  much!

• OASIS4
• higher resolution parallel models, massively
  parallel platforms, 3D fields
• need to optimise and parallelise the coupler

8
OASIS the community today

• CERFACS (France)
  ARPEGE3 - ORCA2LIM ARPEGE4 -
  OPA9/NEMO
• METEO-FRANCE (France) ARPEGE4 -
  ORCA2 ARPEGE medias -OPAmed ARPEGE3 -
  OPA8.1-GELATO
• IPSL- LODYC, LMD, LSCE (France)
  LMDz -
  ORCA2LIM LMDz - ORCA4
• MERCATOR (France) (for interpolation only)
• MPI - MD (Germany) ECHAM5
  - MPI-OM ECHAM5 - C-HOPE PUMA
  - C-HOPE EMAD - E-HOPE ECHAM5
  - E-HOPE ECHAM4 - E-HOPE
• ECMWF IFS - CTM IFS
  Cy23r4 - E-HOPE

9
OASIS the community today

• IFM-GEOMAR (Germany) ECHAM5 - NEMO
• CGAM-Reading (UK) UM4.5 - OPA
  (diff. res.)
• SMHI (Sweden)
  RCA(region.) RCO(region.)
• NERSC (Norway) ARPEGE
  - MICOM
• U. of Bergen (Norway) MM5 - ROMS
  
• KNMI (Netherlands) ECHAM5 - TM5
• INGV (Italy) ECHAM5 MPI-OM
• ENEA (Italy)
  ECHAM5 - OPA8.2
• MITgcm
  - REGgcm
• IRI (USA) ECHAM5 - MOM3
• JAMSTEC (Japan) ECHAM4 - OPA 8.2
• IAP-CAS (China)
  AGCM - LSM
• BMRC (Australia)
  BAM4 - MOM4
• U. of Tasmania (Australia) Data Atm. - MOM4
• RPN-Environment Canada (Canada) MEC - GOM

10
OASIS Some key notes

• Developers CERFACS, NEC CCRLE, CNRS, SGI, (NEC
  HPCE)
• Public domain open source license (LGPL)
• Programming language Fortran 90 and C
• Public domain libraries vendor optimized
  versions may exist
• MPI1 and/or MPI2 NetCDF/parallel NetCDF libXML
• mpp_io SCRIP

11
The OASIS4 coupler

• A parallel model interface library (PSMILe) that
  performs
• weight-and-address calculation for the coupling
  field interpolations
• MPI-based coupling exchanges between components
• component I/O (GFDL mpp_io library)
• A parallel central Driver/Transformer
• launches models at the beginning of the run
  (MPI2)
• reads the user-defined configuration information
  and distributes it to the component PSMILes
• performs parallel transformations of the coupling
  fields during the run

12
OASIS4 model adaptation (1/3)

• Initialization
• call prism_init_comp (comp_id, comp_name, ierr)
• call prism_get_localcomm (comp_id, local_comm,
  ierr)
• Definition of grid (3D)
• call prism_def_grid (grd_id, grd_name, comp_id,
  grd_shape, type, ierr)
• call prism_set_corners(grd_id, nbr_crnr,
  crnr_shape, crnr_array, ierr)
• Placement of scalar points and mask on the grid
• call prism_set_points (pt_id, pt_name, grd_id,
  pt_shape, pt_lon, pt_lat, pt_vert
  ,ierr)
• call prism_set_mask (msk_id, grd_id, msk_shape,
  msk_array, ierr)
• Function overloading to keep the interface
  concise and flexible

13
OASIS4 model adaptation (2/3)
prism_def_grid
Example
prism-set-corners
prism_set_points
prism_set_points
prism_set_points
prism_set_mask
14
OASIS4 model adaptation (3/3)

• Coupling or I/O field declaration
• call prism_def_var
• (var_id, var_name, grd_id, pt_id, msk_id,
  var_nbrdims,
• var_shape, var_type, ierr)
• End of definition
• call prism_enddef (ierr)
• Coupling or I/O field sending and receiving
• in model time stepping loop
• depending on users specifications in SMIOC
• users defined source or target, component or
  file (end-point communication)
• coupling or I/O sending or receiving at
  appropriate times
• averaging/accumulation
• call prism_put (var_id, date, date_bounds,
  var_array, info, ierr)
• call prism_get (var_id, date, date_bounds,
  var_array, info, ierr)
• Termination
• call prism_terminate (ierr)

15
OASIS4 component model description

• Application and component description (XML
  files)
• For each application (code)
• one Application Description (AD)
• possible number of processes
• components included
• For each component in the application
• one Potential Model Input and Output Description
  (PMIOD)
• component general characteristics name,
  component simulated,
• grid information domain, resolution(s), grid
  type,
• potential I/O or coupling variables
• local name, standard name
• units, valid min and max
• numerical type
• associated grid and points
• intent input and/or output

16
OASIS4 coupled model configuration

• (Through a GUI), the user produces
• a Specific Coupling Configuration (SCC)
• experiment and run start date and end date
• applications, components for each application
• host(s), number of processes per host, ranks for
  each component
• For each component,
• a Specific Model Input and Output Configuration
  (SMIOC)
• grid information chosen resolution,
• I/O or coupling variables
• name, units, valid min max, numerical type, grid
• activated intent input and/or output
• source and/or target (component and/or file)
• coupling or I/O dates
• transformations/interpolations/combinations

17
(No Transcript)
18
OASIS4 communication (1/2)

• Model interface library PSMILe based on MPI1 or
  MPI2
• Parallel communication including repartitioning
• based on geographical description of the
  partitions
• parallel calculation of communication patterns in
  source PSMILe

19
OASIS4 communication (2/2)

• end-point communication (source doesnt know
  target and vice-versa)
• parallel 3D neighbourhood search, based on
  efficient multigrid algorithm, in each source
  process PSMILe
• extraction of useful part of source field only
• one-to-one, one-to-many

• parallel I/O (vector, bundles, vector bundles)
  GFDL mpp_io, parNetCDF

20
OASIS4 regridding/transformations

• source time transformations (prism_put)
• average, accumulation
• target time transformations (prism_get)
• time interpolation (for I/O only)
• statistics
• local transformations
• addition/multiplication by scalar
• interpolation/regridding (3D)
• nearest-neighbour 2D in the horizontal, none
  in the vertical
• nearest-neighbour 3D
• bilinear in the horizontal, none in the
  vertical
• bicubic (gradient, 16 nghbrs) in the horizontal,
  none in the vertical
• trilinear

21
Grids supported by OASIS4

• Regridding, repartitioning, I/O
• Regular in lon, lat, vert (Reglonlatvrt)
• lon(i), lat(j), height(k)
• Irregular in lon and lat, regular in the vert
  (irrlonlat_regvrt)
• lon(i,j), lat(i,j), height(k)
• Irregular in lon, lat, and vert (irrlonlatvrt)
  (not fully tested)
• lon(i,j,k), lat(i,j,k), height(i,j,k)
• Gaussian Reduced in lon and lat, regular in the
  vert (Gaussreduced_regvrt)
• lon(nbr_pt_hor), lat(nbr_pt_hor), height(k)
• Repartitioning and I/O only
• Non-geographical fields
• no geographical information attached
• local partitions described in the global index
  space (prism_def_partition)
• I/O only
• Unstructured grids (unstructlonlatvrt)
• lon(npt_tot), lat(npt_tot), height(npt_tot)

22
OASIS4 future developments

• Short term
• 2D conservative remapping
• Parallel global search for the interpolation
• Transformer efficiency
• Error messages, debugging info, documentation,
  user support
• Evaluation of OASIS4 for CICLE (French ANR
  funded project)
• LMDZ - OPA9/NEMO (IPSL)
• reg. atm reg. ocean global atm. global
  ocean (Météo-France)
• Work on the OASIS4 GUI (prepOASIS4) in
  collaboration with ECMWF
• Test UK Met Office FCM environment for
  compiling.
• Coupling workshop (PALM, BFG, FLUME, ESMF,
  GENIE?)

• Long term
• Support types of exchange dates other than fixed
  frequency
• 3D conservative remapping
• User-defined 3D and 2D remapping
• Field reduction, combination
• Full support of unstructured grid
• Support of adaptive grids

23
OASIS4 conclusions

• OASIS4 tested and run with toy examples on
• Intel Pentium 4 Workstation Cluster
• SGI O3000/2000 server with MIPS 4 processors and
  IRIX64
• SGI IA64 Linux server Altix 3000
• NEC SX
• AMD 2800 Cluster
• IBM Power 4
• OASIS4 now being used in a reduced number of real
  applications
• IN EU project GEMS, for atmospheric dynamic and
  chemistry coupling
• At SMHI, for ocean-atmosphere regional coupling
• At UK Met Office for global ocean-atmosphere
  coupling (currently prototyping)
• At GFDL, for MOM4 toy atmosphere coupling
• At IFM-GEOMAR (Kiel) in pseudo-models to
  interpolate high-resolution fields.
• OASIS4 full public release planned beginning 2007.

24
The end
25
Oasis3 model adaptation (1/3)
PRISM System Model Interface Library (PSMILe) API

• Initialization
• call prism_init_comp (comp_id, comp_name, ierr)
• Retrieval of component local communicator
• call prism_get_localcomm (local_comm, ierr)
• Global grid definition (master process only)
• call prism_write_grid (grd_name, nx, ny, lon,
  lat)
• call prism_write_corner (grd_name, nx, ny, nc,
  clon, clat)
• call prism_write_mask (grd_name, nx, ny, mask)
• call prism_write_areas (grd_name, nx, ny, area)

26
Oasis3 model adaptation (2/3)
PRISM System Model Interface Library (PSMILe) API

• Local partition definition (by each process in
  the global index space)
• call prism_def_partition (part_id, ig_paral,
  ierr)
• Coupling or I/O field declaration (name,
  partition id, status, shape, type)
• call prism_def_var
• (var_id, var_name, part_id, var_nbrdims,
  var_inout,
• var_shape, var_type, ierr)
• End of definition
• call prism_enddef (ierr)

27
Oasis3 model adaptation (3/3)
PRISM System Model Interface Library (PSMILe)
API

• Coupling or I/O field sending and receiving
• in model time stepping loop
• depending on users specifications in namcouple
• users defined source or target (end-point
  communication)
• coupling or I/O sending or receiving at
  appropriate times
• automatic averaging/accumulation
• automatic writing of coupling restart file at end
  of run
• call prism_put (var_id, time, var_array, ierr)
• call prism_get (var_id, time, var_array, ierr)
• Termination
• call prism_terminate (ierr)
• Auxiliary routines
• call prism_put_inquire (var_id, time, info)
• call prism_put_restart (var_id, time, info)

28
OASIS3 coupled model configuration

• In text file namcouple, read by OASIS3 main
  process, and distributed to component model
  PSMILes at beginning of run
• total run time
• component models
• number of coupling fields
• for each coupling field
• source and target names (end-point communication)
  (var_name)
• grid acronym (grid_name)
• coupling and/or I/O status
• coupling or I/O period
• transformations/interpolations

29
OASIS3 communication
PSMILe based on MPI1 or MPI2 message passing
30
OASIS3 interpolations/transformations

• on 2D scalar or vector fields
• Interfacing with RPN Fast Scalar INTerpolator
  package
• nearest-neighbour, bilinear, bicubic for regular
  Lat-Lon grids
• Interfacing with SCRIP1.4 library
• nearest-neighbour, 1st and 2nd order conservative
  remapping for all grids
• bilinear and bicubic interpolation for
  logically-rectangular grids
• Bilinear and bicubic interpolation for reduced
  atmospheric grids
• Other spatial transformations flux correction,
  merging, etc.
• General algebraic operations

31
OASIS3 conclusions

• OASIS3 prism_2-4 delivered last December
• OASIS3 tested and run on
• IBM Power 4
• SGI O3800 IRIX64
• Linux Opteron
• NEC SX5 SX6
• Fujitsu VPP5000
• Current work at RPN
• Implement RPN GOSSIP as communication layer in
  PSMIle
• Use new OASIS to couple GEM (in MEC) to IML ocean
  model and realize coupled simulation over Hudson
  Bay

32
OASIS4 component model description

• Application and component description (XML
  files)
• For each application (code)
• one Application Description (AD)
• possible number of processes, components
  included, etc.
• For each component in the application
• one Potential Model Input and Output Description
  (PMIOD)
• component general characteristics name,
  component simulated,
• grid information domain, resolution(s), grid
  type,
• potential I/O or coupling variables
• local name, standard name (NetCDF CF convention)
• units, valid min and max
• numerical type
• associated grid and points
• intent input and/or output

33
OASIS4 coupled model configuration

• Coupled model configuration (XML files)
• (Through a GUI,) the user produces
• a Specific Coupling Configuration (SCC)
• start date and end date
• start mode (MPI1, MPI2)
• applications, components for each application
• host(s), number of processes per host, ranks for
  each component
• For each component,
• a Specific Model Input and Output Configuration
  (SMIOC)
• grid information chosen resolution,
• I/O or coupling variables
• local and standard name, units, valid min max,
  numerical type, grid
• activated intent input and/or output
• source and/or target (component and/or file)
• coupling or I/O dates
• transformations/interpolations

34
Driver
ifs_ad.xml
user
ifs_ifs_pmiod.xml ifs_tmp output ifs_q
output ifs_pl input
scc.xml IFS... CTM..
ifs_tmp
ctm_tmp
ifs_q
ifs_q
ifs_pl
ctm_pl
ifs_ifs_smioc.xml ifs_tmp-gtctm_tmp ctm 1 hr
statistics ifs_q-gtctm_q ctm 1 hr
statistics ifs_pllt-ctm_pl 1hr, trilinear,
statistics
ctm_ctm_smioc.xml ctm_tmplt-ifs_tmp ifs 1hr,
trilinear, statistics ctm_q lt- ifs_q ifs 1hr,
trilinear, statistics ctm_pl-gtifs_pl 1hr,
statistics
35
The end
36
(No Transcript)
37
What is an XML file?

• Structured way of providing information
• Hierarchy of elements and attributes
• Structure of an XML file given by an XSD (schema)
  file

• SCC.xsd
• ltxselement name"host"gt
• ltxscomplexTypegt
• ltxssequencegt
• ltxselement name"nbr_procs"/gt
• lt/xssequencegt
• ltxsattribute name"local_name"
• type"xsstring" use"required"/gt
• lt/xscomplexTypegt
• lt/xselementgt
• ltxselement namenbr_procs
• type"xsinteger /gt

SCC.xml lthost local_name"host1"gt
ltnbr_procsgt2lt/nbr_procsgt lt/hostgt
38
Oasis4 communication (2/3)
Parallel calculation of communication patterns in
source PSMILe
For each pair of source and target processes
39
PALM le contexte

• Implémentation modulaire de chaînes
  dassimilation de données
• - recherche de méthodes
• flexibilité
• - exploitation opérationnelle
• performances

40
PALM le principe

• Les différentes techniques dassimilation de
  données peuvent se décomposer en opérations
  élémentaires
• Un enchaînement donné de ces opérations
  élémentaires (unités) génère une technique
  dassimilation particulière
• Loutil PALM assure (suivant la description faite
  par lutilisateur)
• lenchaînement et la synchronisation des unités
• les échanges efficaces dobjets entre les unités

41
PALM lutilisation

• 1. Décomposer lapplication en unités de
  traitement
• décrire les données produites (output) ou
  requises (input) de lunités objets
• implémenter dans lunité les requêtes dobjets
  input PALM_GET
• implémenter dans lunité les mises à disposition
  dobjets output PALM_PUT

• 2. Utiliser linterface graphique pré-palm pour
• décrire lenchaînement dynamique de ces unités
  (for, while, if , select) les branches
• décrire les échanges dobjets entre les
  différentes unités

• 3. Compiler, lancer lapplication et laisser PALM
  assurer
• lenchaînement et la synchronisation des unités
  décrits avec pre-Palm
• les échanges efficaces dobjets entre les unités

42
PALM linterface graphique pre_PALM
43
PALM lanalyse de performance de pre_PALM
44
PALM le développement
45
PALM les utilisateurs

• Projet MERCATOR
• assimilation de données en océanographie
  opérationnelle
• Projet Européen ASSET
• assimilation de données en chimie atmosphérique
  (ENVISAT)
• Projet ADOMOCA (PNCA)
• PALM choisi comme outil fédérateur pour
  lassimilation de données en chimie atmosphérique
  en France
• EDF
• assimilation de données dans les curs de
  réacteurs nucléaires
• SNECMA
• couplage 3D fluideradiatifsolide (simulation
  des températures de paroi des chambres de
  combustion de moteur davion).
• elsA (CERFACS)
• couplage fluide-structure
• Projet OPAVAR (CERFACS)
• assimilation variationnelle 3/4DVAR dans l'océan
• CESBIO, SCHAPI, TURBOMECA, IUSTI, IMFT,