Mars_Agrif2_V9.06

1
Mars_Agrif2_V9.06 MARS ENVIRONMENTs
manualValérie GARNIER, Sébastien THEETTEN
(easier to consult this documentation activating
slide (diaporama)in order to show links )
2
MARS environmentSommaire (click on each
chapter to go directly in)

• Why this MARS environment
• What is the MARS environment
• Setting up MARS environment
• MARS requirements
• List of useful commands
• How to implement a new configuration
• How to update a configuration
• MARS environment SUMMARY
• Choice of CPP Keys
• How to prepare parametrization before running
• How to run a job
• Run a job SUMMARY
• How to decompose domain for MPI purpose
• How to use a test case
• How to set a new test case
• How to modify MARS - coding rules

2
3
Why this MARS environment (1)

• Frequently observed problems
• Difficulty in following MARS evolutions led to
  different MARS codes !
• Numerical developments and configurations must be
  performed following a few usual rules, otherwise
  they get lost
• MARS is used through a working environment based
  on C-shell scripts
• Goals
• Management of MARS versions
• Easier tracking of configuration changes
• Easier keeping up to date of configurations
• Direct visualization of users modifications
• Sharing of configurations
• Saving of disk space

4
Why this MARS environment (2)

• Super-users
• management and distribution (V. Garnier)
• developments / evolutions (F. Dumas, V. Garnier,
  B. Thouvenin)
• teaching and users help (S. Theetten)
• MARS code
• The complete MARS code (sources, namelist,
  environment, documentation and netcdf libraries)
  is located in a directory defined by the
  environment variable HOMEMARS
• Users
• Run the code for personal configuration or use
  test cases
• Improve the code (numerical developments)
• Report encountered problems or needs
• Follow code improvements, keep configurations up
  to date

5
Reference MARS environment (3)
Retour au Sommaire

• Reference
• MARS environment is originally derived from
  utility named Drakkar Config Manager and
  developped by J. M. Molines and S. Theetten.
• Similar environments are used with MARS code,
  NEMO code (DRAKKAR projects and followings) and
  SYMPHONIE code (soon).

6
What is the MARS environment (1)

• 3 directories (user)

3 scripts (user)

• mkconfdir
• create the directories of a configuration

MARS_CONFIG compilation / code modification

• makefile
• manage the configuration
• (choice of rank, test case, previous
  configurations),
• compile the code by calling
• Makefile.linux
• (or Makefile.caparmor)

COMPILE_MARS compilation directory, where the
user must not go (otherwise users changes might
be lost)
RUN_MARS run simulation, input and ouput files
7
What is the MARS environment (2)
Retour au Sommaire

• The originial code (reference code) is placed in
  the
• Reference directory HOMEMARS/.. (super user)
• which contains
• MARS sources,
• Scripts of MARS environment,
• Makefile for the compilation,
• Input files as namelists and .dat (river.dat,
  outflow.dat)
• Documentation manuals, faq
• Module example (fortran)
• Libraries (netcdf, io_netcdf)
• Code rules

HOMEMARS/../
HOMEMARS/
Mars_Agrif2 Reference Sources code
DOC Manual, faq
EXAMPLES input files, Makefile..
LIB Netcdf, io_netcdf..
TOOLS bathy, connect,
7
8
MARS environment setting up (1)
1/ Inside your .cshrc, add following
environment variables and paths

• ------------------------------
•   MARS DEVELOPMENT ENVIRONMENT
• -----------------------------
• setenv HOMEMARS /export/home11/mars/CODE_MARS/CODE
  _MARS_V9/V9.06/Mars_Agrif2
• setenv UDIR /export/home/USER/MARS/MARS_CONFIG
• setenv CDIR /export/home/USER/MARS/COMPILE_MARS
  
• setenv RDIR /export/home/USER/MARS/RUN_MARS
• setenv HTTPSVNROOT https//forge.ifremer.fr/svn/ma
  rs3d
• setenv extranet_login yours
• setenv PATH PATHHOMEMARS/../TOOLS/MARSENV
• alias mkconfdir  "HOMEMARS/../TOOLS/MARSENV/mkcon
  fdir_caparmor
• alias mkconfdir_bisc  "HOMEMARS/../TOOLS/MARSENV/
  mkconfdir_caparmor_bisc

2/ type source .cshrc
8
9
MARS environment setting up (2) (bash use)

• If bash use, add following environmental
  variables and paths
• 1/ create a new file (env_MARS_V9.06 for
  instance)
• ------------------------------
•   MARS DEVELOPMENT ENVIRONMENT
• ------------------------------
• export HOMEMARS /export/home11/mars/CODE_MARS/CO
  DE_MARS_V9/V9.06/Mars_Agrif2
• export UDIR/export/home/USER/MARS/MARS_CONFIG
• export CDIR/export/home/USER/MARS/COMPILE_MARS
  
• export RDIR/export/home/USER/MARS/RUN_MARS
• alias mkconfdirHOMEMARS/../TOOLS/MARSENV/mkconf
  dir_linux
• alias getfileHOMEMARS/../TOOLS/MARSENV/getfile
  
• alias updateHOMEMARS/../TOOLS/MARSENV/update
• 2/ type source env_MARS_V9.06

9
10
MARS environment setting up (3)

• 3/ create the 3 directories necessary for MARS
  environment
• UDIR
• type mkdir /export/home/USER/MARS/MARS_CONFI
  G
• CDIR
• type mkdir /export/home/USER/MARS/COMPILE_MAR
  S
• RDIR
• type mkdir /export/home/USER/MARS/RUN_MARS
• CODE_MARS (if the reference MARS code is
  locally saved not for IFREMER users)
• type mkdir /export/home/USER/MARS/CODE_MARS

11
MARS environment setting up (4)

• Important remark
• meant for research institutes, private societies
  or labtop use
• IFREMER users have no need to duplicate the code
  under their account
• Step local setting up (not for
  caparmor users)
• Get the code from mars3d projet under forge with
  the following command svn username
  extranet_login checkout HTTPSVNROOT/tags/V9.06
• cd V9.06/EXAMPLES
• vi Makefile.linux modify library paths of
  netcdf, io_netcdf.
• (if make use, replace all gmake by make in
  makefile_env_linux or link gmake to make in your
  linux environment)
• Rq 1 changes inside Makefile.linux (and
  makefile_env_linux) are the only ones allowed in
  CODE_MARS directory
• Rq 2 no help from PHYSED if the source code has
  been modified

12
MARS environment setting up (5)

• Important remarks
• MARS requires fortran compiler, netcdf4 library
  (netcdf4 requires hdf5 and zlib librairies)
• MARS results are easily visualized with ferret
  (free) or matlab
• Library NCO is useful for files management

For users working on caparmor Return to
sommaire Informations on Fortran, netcdf4,
netcdf, click here
13
MARS environment local setting up (6)

• Fortran compiler setting up
• Download free Intel Fortran compiler from
  http//www.intel.com/cd/software/products/asmo-na/
  eng/index.htm (a -
  register at  Free Non Commercial Download  to
  receive a number authorization by mail b
  choose Intel Fortran Compiler Profressional
  Edition for linux)
• Installation from file /export/home/l_fc_p_10.X.XX
  X.tar.gz
• If library stcd is missing yum install
  libstdc.so.5
• Set compiler environment source
  /opt/intel/fce/10.X.XXX/bin/ifortvars.csh

14
MARS environment local setting up (7)Netcdf4
(1/3)

• We assume the netcdf4 library is installed under
  /export/home/logiciel/netcdf4
• zlib library
• Create installation directory
• cd export/home/logiciel/netcdf4 mkdir install
  cd install
• Download of zlib-1.2.5.tar from http//zlib.net/
  (get the sources and not the binaries)
• bunzip2 zlib-1.2.5.tar.bz2 tar xvf
  zlib-1.2.5.tar
• Installation

sequential or OMP purpose MPI purpose
Module purge module load intel-comp/11.1.073 (path for intel fortran and c) Module purge module load intel-comp/11.1.073 module load intel-mpi/4.0.0.028 (path mpi)
export NC4DIR/export/home/logiciels/netcdf4/lib/seq export NC4DIR/export/home/logiciels/netcdf4/lib/mpi
cd /export/home/logiciels/netcdf4/install/zlib-1.2.5 cd /export/home/logiciels/netcdf4/install/zlib-1.2.5
./configure --prefixNC4DIR ./configure --prefixNC4DIR
make check install make check install
14
15
MARS environment local setting up (8)Netcdf4
(2/3)

• We assume the netcdf4 library is installed under
  /export/home/logiciel/netcdf4
• hdf5 library
• Go to installation directory
• cd export/home/logiciel/netcdf4/install
• Download http//www.hdfgroup.org/ftp/HDF5/prev-rel
  eases/hdf5-1.8.6/src/hdf5-1.8.6.tar.gz (get the
  sources and not the binaries)
• gunzip file tar xvf file
• Installation

sequential or OMP purpose MPI purpose
Module purge module load gfortran (path for gfortran and c) Module purge module load gfortran module load gfortran-mpi (path mpi)
export NC4DIR/export/home/logiciels/netcdf4/lib/seq export NC4DIR/export/home/logiciels/netcdf4/lib/mpi
cd /export/home/logiciels/netcdf4/install/hdf5-1.8.6 cd /export/home/logiciels/netcdf4/install/hdf5-1.8.6
./configure --prefixNC4DIR --with-zlibNC4DIR --disable-shared CCgcc CPP'gcc -E' ./configure --prefixNC4DIR --with-zlibNC4DIR --disable-shared --enable-parallel CCmpicc CPP'mpicc -E'
make make
make install make install
15
16
MARS environment local setting up (9)Netcdf4
(3/3)

• We assume the netcdf4 library is installed under
  /export/home/logiciel/netcdf4
• netcdf4 library
• Go to the installation directory (cd
  export/home/logiciel/netcdf4/install)
• Download of netcdf-4.1.2.tar.gz from
  http//www.unidata.ucar.edu/downloads/netcdf/netcd
  f-4_1_2/index.jsp (get the sources and not the
  binaries)
• gunzip file tar xvf file
• Installation

sequential or OMP purpose MPI purpose
Module purge module load gfortran (path for gfortran and c) Module purge module load gfortran module load gfortran-mpi (path mpi)
export NC4DIR/export/home/logiciels/netcdf4/lib/seq export NC4DIR/export/home/logiciels/netcdf4/lib/mpi
cd /export/home/logiciels/netcdf4/install/netcdf-4.1.2 cd /export/home/logiciels/netcdf4/install/netcdf-4.1.2
export LDFLAGS"-lm"  (access to a mathematics library)
./configure --prefixNC4DIR --with-hdf5NC4DIR --with-zlibNC4DIR --enable-ncgen4 --enable-netcdf-4 --disable-shared --disable-cxx --disable-dap FCgfortran CCgcc CPP'gcc -E' ./configure --prefixNC4DIR --with-hdf5NC4DIR --with-zlibNC4DIR --enable-ncgen4 --enable-netcdf-4 --disable-shared --disable-cxx --disable-dap CPPFLAGS'-DNDEBUG -DpgiFortran' FCmpif90 CCmpicc CPP'mpicc -E'
make make
make check install make check install
16
17
MARS environment local setting up (10)

• IO_NETCDF library (to be compiled only if used by
  software other than MARS)
• Libionc4 is an interface between NetCDF4 and MARS
  (or other softwares). This library is developed
  by IFREMER/PHYSED and supplied with MARS under
  HOMEMARS/../LIB/libionc4. It is a module of MARS
  software.
• cd HOMEMARS/../LIB/libionc4/src
• ln -s HOMEMARS/INC/comionc4.F90
• ln -s HOMEMARS/PHYS/OUT/ionc4.F90
• Prepare directories
• cd ..
• ./install.csh (2 directories are created
  intel-seq and intel-mpi the will store the
  library and the include files)
• edit Makefile and modify

sequential or OMP purpose MPI purpose
FC ifort FC mpiifort
CPPFLAGS CPPFLAGS -Dkey_MPI_2D
NETCDF4 /home1/caparmor/rramel/netcdf4.1.2 NETCDF4 /home1/caparmor/rramel/netcdf4.1.2_par
OUTDIR ./intel-seq OUTDIR ./intel-mpi
FFLAGS required compilation options FFLAGS required compilation options
18
MARS environment local setting up (11)

• File connexion
• This fortran code developed by IFREMER/PHYSED to
  gather MPI output files is supplied with MARS
  under HOMEMARS/../TOOLS/CONNECT
• cd HOMEMARS/../TOOLS/connect
• Modify the library paths in compile_cap
• Compile_cap
• The connect.exe is created under the directory
  HOMEMARS/../TOOLS/connect. It will be
  automatically copied under RDIR/CONF/CONF-CASE/r
  ank_X

18
19
MARS environment local setting up (12)

• Domain cutting out executable (for MPI purpose)
• This fortran code has been developed by
  IFREMER/RIC to divide the regional model in X
  sub-domains. It is available in
  HOMEMARS/../TOOLS/MPI2D_DOMAIN
• cd HOMEMARS/../TOOLS/MPI2D_DOMAIN
• Modify the library paths in makefile
• make
• The ./run executable is created under the
  directory HOMEMARS/../TOOLS/MPI2D_DOMAIN.
• A user manual is available from
  http//wwz.ifremer.fr/dyneco/Moyens-Outils/Logicie
  ls/MARS/Documentation

19
20
MARS environment local setting up (13)
Retour au Sommaire

• FERRET
• Download files from http//ferret.pmel.noaa.gov/st
  atic/Downloads/
• Follow instructions of installation
• NCO tools yum install nco
• nedit editor yum install nedit

21
MARS environment commands (1/2)

• Useful commands
• mkconfdir creates all directories required for
  the implementation of a configuration (for more
  details of what mkconfdir does, click here)
• gmake install sets the reference code into the
  directory of compilation
• (for more details,
  click here)
• gmake compiles with users routines
• (for more details,
  click here)
• gmake clean cleans .o and .f
• gmake cleaninst empties the directory of
  compilation (useful when you want to remove a
  users routine)
• gmake update "OLDREVXXX" "NEWREVYYY" sets
  the reference code into the directory of
  compilation and updates users routines from
  revision XXX to revision YYY
• All these commands are launched from the
  directory MARS_CONFIG/conf/conf-case, location
  where the user prepares the code he will run.

21
22
Retour au Sommaire
MARS environment commands (2/2)

• Useful commands
• getfile WORK/routine.F90 allows the user to
  modify the reference code source (for more
  details, click here)
• rmfile routine.F90 remove a file from
  MARS_CONFIG/CONF/CONF-CASE.
• (for more details, click here)
• cmpfile routine.F90 Easy way for making either
  a diff between a file in the MARS_CONFIG/CONF/CON
  F-CASE directory and the corresponding file in
  the HOMEMARS directory. (for more details, click
  here)
• All these commands are launched from the
  directory MARS_CONFIG/conf/conf-case, location
  where the user prepares the code he will run.

22
23
MARS environmentmkconfdir command creates all
directories required for the implementation of a
configuration

• without rank (click here)
• with rank (click here)

Usage mkconfdir CONF CASE (number of
ranks)
23
24
MARS environmentmkconfdir command (without rank)
Return Liste commands
Return new config

• mkconfdir CONF CASE
• (CONF for geographic configuration CASE for one
  case of this configuration)

UDIR/CONF/CONF-CASE

• 1. Create in UDIR
• create empty directories
• copy the makefile and
• Makefile.caparmor files

INC
PHYS
TRAJ
BIOLO
SEDIM
SWAN
AGRIF
makefile
The makefile file manages the MARS environment
and calls to Makefile.caparmor for compilation
Makefile.caparmor
Makefile. where user chooses cpp keys and
compiling option

• 2. Create in CDIR
• Create empty directory

CDIR/WCONF-CASE

• 3. Create in RDIR
• Create directories

RDIR/CONF/CONF-CASE
RDIR/CONF/inputs

• default namelist files
• output.dat file
• script and batchs
• batch_connect and connect.exe files

- examples of files .dat
to give the formats of the files managing
rivers, outflows, variables, trajectories
example of head.CONF file
25
MARS environmentmkconfdir command (with rank)
Return Liste commands
Return new config

• mkconfdir CONF CASE 3 (if X3 ranks)
• (CONF for geographic configuration CASE for one
  case of this configuration)

UDIR/CONF/CONF-CASE

• 1. Create in UDIR
• create empty directories
• copy the makefile and
• Makefile.caparmor files

INC
PHYS
TRAJ
BIOLO
SEDIM
SWAN
AGRIF
makefile_rankX
The makefile file manages the MARS environment
and calls to Makefile.caparmor for compilation
Makefile.caparmor_rankX
Makefile. where user chooses cpp keys and
compiling option

• 2. Create in CDIR
• Create empty directory

CDIR/WCONF-CASE

• 3. Create in RDIR
• Create directories

RDIR/CONF/CONF-CASE

RDIR/CONF/inputs
RDIR/CONF/CONF-CASE /rank_0, rank_1 et
rank_2 each contains

• default namelist files

- examples of files .dat
to give the formats of the files managing
rivers, outflows, variables, trajectories

• default namelist files
• output.dat file
• script and batchs
• batch_connect and connect.exe files

example of head.CONF file
25
26
MARS environmentgmake install command
Return Liste commands
Return new config

• The setting up (users modification of the code,
  compiling)
• of the configuration is done from directory
  UDIR/CONF/CONF-CASE
• Usage cd UDIR/CONF/CONF-CASE
• gmake install

• This command copies the reference MARS code in
  CDIR/WCONF-CASE
• And creates a link to the directory WORK in
  UDIR/CONF/CONF-CASE

CODE_MARS Reference (HOMEMARS)
COMPILE_MARS/WCONF-CASE Reference code
MARS_CONFIG/CONF/CONF-CASE/WORK In WORK all
MARS routines as link to CDIR
27
MARS environmentExecution gmake
Return Liste commands
Return new config

• This command prepares the code, compiles it and
  copy the executable to RDIR

• Usage cd UDIR/CONF/CONF-CASE
• gmake

• Copy modified users routines in CDIR

executable
exe
smallf90
smallf90

• Compiles the code to get the executable (exe gt
  mars_exe in RDIR)

• Creates smallf90 directory

27
28
MARS environmentExecution modification of the
reference code source
Return Liste commands
Return new config

• Usage cd UDIR/CONF/CONF-CASE
• getfile
  WORK/routine.F90

• gets a routine and saves it at the correct
  location in MARS_CONFIG/CONF/CONF-CASE.
• Then, this routine can be modified by the user
  directly from the directory MARS_CONFIG/CONF/CONF-
  CASE.
• (links are automatically made)

29
MARS environmentremove a file modified by user
and back to reference
Return Liste commands

• Usage cd UDIR/CONF/CONF-CASE
• rmfile routine.F90

• Remove a file from MARS_CONFIG/CONF/CONF-CASE
• Restore the removed file from the REFERENCES
  (HOMEMARS,REF_CONFIG),

29
30
MARS environmentvisualize differences between
users routine and reference code
Return Liste commands

• Usage cd UDIR/CONF/CONF-CASE
• cmpfile
  routine.F90

• Easy way for making either a diff between a file
  in the MARS_CONFIG/CONF/CONF-CASE directory and
  the corresponding file in the HOMEMARS directory.
  
• Default option launches tkdiff wheter -t option
  produces a diff -bic listing.
• It can be used as template for personnal
  improvement.

30
31
MARS environmentnew configuration
Retour au Sommaire

• New configuration without rank
• New configuration with ranks
• New configuration using (copying) an other
  configuration (PREV_CONFIG)
• New configuration recovering an other
  configuration updated by an other user
  (REF_CONFIG) WITHOUT RANK
• New configuration recovering an other
  configuration updated by an other user
  (REF_CONFIG) WITH RANKS
• Usage of PREV_CONFIG and REF_CONFIG
• Before compilation choice of CPP Keys (list)
• After compilation and before running

32
MARS environmentnew configuration without rank
Return new configuration list

• The rank_0 does not supply the rank_1 with open
  boundary conditions
• ssh -X -l username caparmor (connection to
  caparmor )
• mkconfdir MENOR V9.06 (create directories for
  your configuration
• 
  (here CONF
  MENOR, CASE V9.06)
• cd (follow the screen directive) (go in the
  directory UDIR/CONFIG/CASE)
• gmake install (install the code for your
  configuration)
• vi Makefile.caparmor and choice of CPP keys,
  compilation option
• getfile WORK/parameters.F90 and change specific
  parameters to your configuration
• gmake (compilation)
• modification of the code by the user
• getfile WORK/routine.F90
• modification of routine.F90
• gmake

32
33
MARS environmentnew configuration with ranks
Return new configuration list

• The rank_0 supplies the rank_1 with open boundary
  conditions (SSH)
• ssh -X -l username caparmor (connection to
  caparmor )
• mkconfdir MENOR V9.06 4 (for ranks 0, 1 ,2 et 3)
  (create directories for your
• 
  configuration (here CONF
  MENOR, CASE V9.06, 4 ranks)
• cd (follow the screen directive) (go in the
  directory UDIR/CONFIG/CASE)
• gmake install (install the code for your
  configuration)
• vi Makefile.caparmor_rankX and choice of CPP
  keys, compilation option for each rank
• getfile WORK/parameters.F90_rank0
• cp INC/parameters.F90_rank0 INC/parameters.F90_ran
  k1,
• cp INC/parameters.F90_rank0 INC/parameters.F90_ran
  k2 ...
• modification of INC/parameters.F90_rank (no,
  nrac,imax,jmax,kmax)
• vi makefile and change rank0,1 or 2 in order to
  compile the code for this rank
• gmake clean gmake (compilation)

33
34
MARS environment why variables REF_CONFIG et
PREV_CONFIG (1/2)

• PREV_CONFIG (with gmake copyconfig)
• During the evolution of a configuration, the
  creation of a new configuration CONF-CASE and the
  use of gmake copyconfig
• allows to get exactly the same code as in
  configuration PREV_CONFIG (gmake copyconfig
  command copies users routines and
  Makefile.linux(or .caparmor) from directory
  PREV_CONFIG to UDIR/CONF/CONF-CASE)
• allows to keep a saving of configuration
  PREV_CONFIG
• This command is used when updating a
  configuration with a new version of the reference
  code (the user get directly the routines he may
  have to update)
• REF_CONFIG
• Useful when several users use the same
  configuration
• Only one user is in charge of updating the
  REF_CONFIG configuration
• At each evolution of the reference MARS code,
  other users have to update their owns routines
  only
• Routines of directory REF_CONFIG are directly
  copied from REF_CONFIG directory to the compiling
  directory the user does not see them in
  UDIR/CONF/CONF-CASE but he uses them

35
MARS environment Remarks on variables REF_CONFIG
et PREV_CONFIG (2/2)
Return new configuration list

• PREV_CONFIG and REF_CONFIG
• The variables PREV_CONFIG and REF_CONFIG allow to
  manage the code but not the input files they
  have an influence in directory UDIR/CONF/CONF-CAS
  E only
• The user must copy namelists paraspec.txt,
  paramain.txt, paracom.txt, parasubs.txt,
  parasedim.txt and parabiolo.txt ( output.dat
  file and mpi.txt) relative to the configuration
  PREV_CONFIG or REF_CONFIG
• PREV_CONFIG
• Makefile.linux files is copied from PREV_CONFIG
  directory gt update the path in INCDIR and CPP
  variables
• REF_CONFIG
• Makefile.linux is not changed gt the user has to
  change it in order to use the same cpp keys and
  the same compiling options as the ones used in
  REF_CONFIG directory

36
MARS environment Recovery of a previous
configurationuse of variable PREV_CONFIG in
makefile
Return new configuration list

• ssh -X -l username caparmor (connection to
  caparmor )
• mkconfdir MENOR V9.06 3 (3 if ranks) (create
  directories for your configuration
• 
  
  (here CONF MENOR, CASE V9.06, 3 ranks)
• cd (follow the screen directive) (go in the
  directory UDIR/CONFIG/CASE)
• gmake install (install the code for your
  configuration)
• vi makefile PREV_CONFIG /export/home11/mars/CON
  FIG/MENOR/MARS/MARS_CONFIG/MENOR/MENOR-V9.06
  (choose the name of the configuration you want to
  copy without any blank at the end of the line)
• gmake copyconfig (Rk all users routines of
  MENOR-V9.06 are copied to the configuration).
• vi Makefile.caparmor_rankX
• update the file (if new cpp key or routines have
  been introduced in a new version of the reference
  code ) (compare with Makefile.caparmor_ref)
• vi makefile and change rank0,1 or 2 in order to
  compile the code for this rank
• (gmake clean ) gmake (compilation)
• (To make sure you have the same code, compare
  smallf90 directory with PREV_CONFIG/smallf90
  directory)

Click for summary of what PREV_CONFIG is doing
36
37
MARS environment (PREV_CONFIG summary)
Return new configuration list
HOMEMARS (reference code)
mkconfdir CONF CASE (rank)
gmake install
gmake copyconfig
gmake
OLD_CONFIG (configuration to be used)

• UDIR/CONF/CONF-CASE
• makefile
• Makefile.caparmor_rankX
• tree of the reference code

CDIR/WCONF-CASE
reference code
PREV_CONFIG OLD_CONFIG (makefile)
OLD_CONFIG routines users routines
users routines of OLD_CONFIG Makefile.caparmor_r
ankX users routines

• RDIR/CONF/CONF-CASE

mars_exe
Verification diff WORK OLD_CONFIG/WORK diff
smallf90_rankX OLD_CONFIG/ smallf90_rankX
38
MARS environment Recovery of a previous
configuration (without rank)use of variable
REF_CONFIG in makefile
Return new configuration list

• ssh -X -l username caparmor (connection to
  caparmor )
• mkconfdir MENOR V9.06 (create directories for
  your configuration
• 
  
  (here CONF MENOR, CASE V9.06)
• cd (follow the screen directive) (go in the
  directory UDIR/CONFIG/CASE)
• vi makefile REF_CONFIG /export/home11/mars/CONF
  IG/MENOR/MARS/MARS_CONFIG/MENOR/MENOR-V9.06
  (choice the name of the configuration you wants
  to copy without blank at the end of the line)
• vi Makefile.caparmor
• Choose the same compiling options and cpp keys
  as those in REF_CONFIG/Makefile.caparmor
• If ranks
• cp REF_CONFIG/INC/parameters.F90_rank INC/.
• If test case
• cp REF_CONFIG/INC/parameters.F90_casXXX INC/.
• gmake install (install the code for your
  configuration)
• gmake (compilation)

Rk all users routines of MENOR-V9.06 are
compiled but they are not copied in your
directory UDIR/CONF/CONF-CASE. NEVER ERASE
DIRECTORY REF_CONFIG !!!!!
38
39
MARS environment Recovery of a previous
configuration (with rank)use of variable
REF_CONFIG in makefile
Return new configuration list

• ssh -X -l username caparmor (connection to
  caparmor )
• mkconfdir MENOR V9.06 2 (create directories for
  your configuration
• 
  
  (here CONF MENOR, CASE V9.06, 2 ranks)
• cd (follow the screen directive) (go in the
  directory UDIR/CONFIG/CASE)
• vi makefile REF_CONFIG /export/home1/vgarnier/M
  ARS_CONFIG/calv/calv-V9.06
• (choice the name of the configuration you wants
  to copy without blank at the end of the line)
• get routines parameters.F90 of each rank
• cp /export/home1/vgarnier/MARS_CONFIG/calv/calv-V9
  .06/INC/parameters.F90_rank INC/.
• vi Makefile.caparmor_rankX
• Choose the same compiling options and cpp keys
  as those in REF_CONFIG/Makefile.caparmor
• gmake install (install the code for your
  configuration)
• vi makefile (change RANK ) gmake clean gmake
  (compilation of each rank)

Rk all users routines of MENOR-V9.06 are
compiled but they are not copied in your
directory UDIR/CONF/CONF-CASE. NEVER ERASE THE
REF_CONFIG DIRECTORY !!!!!
39
40
MARS environment (REF_CONFIG summary)
Return new configuration list
HOMEMARS (reference code)
mkconfdir CONF CASE (rank)
gmake install
gmake
OLD_CONFIG (configuration to be used)

• UDIR/CONF/CONF-CASE
• makefile
• Makefile.caparmor_rankX
• tree of the reference code

CDIR/WCONF-CASE
reference code
REF_CONFIG OLD_CONFIG (makefile) users
routines
OLD_CONFIG routines users routines
cp OLD_CONFIG/Makefile_caparmor_rankX .
(modify INCDIR,CPP) cp
OLD_CONFIG/parameters.F90_rank INC/.

• RDIR/CONF/CONF-CASE

mars_exe
Verification diff WORK OLD_CONFIG/WORK diff
smallf90_rankX OLD_CONFIG/ smallf90_rankX
40
41
MARS environment Update of a configuration (1)

• ? change the number of reference code version
• in .cshrc.caparmor (or env_MARS_VX.XX)
• setenv HOMEMARS /export/home11/mars/CODE_MARS/CODE
  _MARS_V9/V9.06/Mars_Agrif2

? Verify you have in .cshrc.caparmor setenv
HTTPSVNROOT https//forge.ifremer.fr/svn/mars3d se
tenv extranet_login yours (Set your
extranet_login name)  
? Run .cshrc.caparmor typing source
.cshrc.caparmor  
41
42
MARS environment Update of a configuration (2)
? create a new configuration mkconfdir MENOR
V9.06 (CONFIGMENOR, new case or new version
V9.06) cd ... (follow the screen directive)
gmake install (install the code for your
configuration useless if you update with
subversion) vi makefile PREV_CONFIG
/export/home1/vgarnier/MARS_CONFIG/MENOR/MENOR-old
gmake copyconfig vi Makefile.caparmor_rankX
update if new cpp key or routines have been
introduced in the new version (compare to
Makefile.caparmor_ref)  

• ?update user's routines
• without SubVersioN use
  click here
• or
• with SubversioN use
  click here

WARNING No gmake at this step
43
MARS environment Update of a configuration (2)
without SubVersioN use

•  
• ?update user's routines without SubVersioN use
• a - notice which users routines has been
  modified in reference MARS code
• (the list of modified routines is
  published at each new version or compare the
  routines between the old and the new HOMEMARS)
•  
• b visualize the modifications introduced by
  the user
• vimdiff WORK/routine.F90
  HOMEMARS/../Mars_Agrif2_old/...../routine.F90
• c - visualize the modifications introduced
  into the new reference version in comparison to
  the old reference version
• vimdiff HOMEMARS/../Mars_Agrif2_XXX/.....
  /routine.F90 HOMEMARS/../Mars_Agrif2_old/...../ro
  utine.F90
•  

43
44
MARS environment Update of a configuration (3)
without SubVersioN use

•  
• d - 2 cases
• 1st case the number of modifications
  introduced by the user is small
• ---gt getfile WORK/routine.F90
• (get the routine of the new reference
  code because the code has not been compiled yet)
• ---gt introduce users modifications
•  
• 2nd case the number of modifications
  introduced by the user is larger that the number
  of modifications introduced into the new
  reference code
• ---gt introduce modifications of the new
  reference code into the users routine
•  
• ? choice CPP Keys (in Makefile_caparmor) then
  gmake
• ? cd RDIR/CONF/CASE (click here to continue)

45
MARS environment Update of a configuration (2)
with SubVersion

•   ? install and update user's routines
• gmake update "OLDREVXXX"
  "NEWREVYYY"
• (check the
  number into HOMEMARS/../DOC/list_version -FIRST
  COLUMN- )
•   vi HOMEMARS/../DOC/list_version to get XXX
  YYY relative to the old and new versions
• this command updates all the routines of the
  current tree from an old revision XXX to the
  current one YYY using SubVersioN
• (you need a client of SubVersioN installed on
  your machine. Available under caparmor for all
  the users)

  ? resolve conflicts for routines described by
C a open 3 windows b windows 1 visualize
modifications introduced by the user in old
version Setenv HOMEMARS /export/home11/mars/CODE_M
ARS/V7.71 (only in this window) Cmpfile
routine.F90 b windows 2 visualize
modifications introduced into the new reference
version by comparison to the old reference
version vimdiff HOMEMARS/...../routine.F90
/export/home11/mars/CODE_MARS/V7.71/Mars_Agrif2//
routine.F90 c windows 3 correct users
changes taking care of the modifications added in
the reference code. Delete warning comments
relative to the merge (ltltltlt.working
gtgtgtgt.merge.right.rXXX gtgtgtgtgtgt.merge.left.rXXX) Cm
pfile routine.F90   ? choice CPP Keys (in
Makefile_caparmor) then gmake ?cd
RDIR/CONF/CASE
46
MARS environment Update of a configuration (4)
Return new configuration list

•  ? update files paramain.txt under
  RDIR/CONF/CONF-CASE
• ? Run the new configuration and the old one in
  sequential with no optimization option de
  compilation (FLAG -r8 O0)
• The update is finished and secured
• If you get exactly the same results (all digits)
  when you compare files named listing, files
  that contain maximum velocities at different time
  steps,
• Otherwise, if you can explain differences.
• Consequently, you need to get exactly the same
  results before introducing the corrections or
  evolutions that modify results. These
  corrections/evolutions are listed in the file
  list_modif_V6 supplied at each new version.

47
MARS environment (summary)
mkconfdir CONF CASE (rank) gmake install gmake
getfile WORK/casvortex.F90 UDIR/CONF/CONF-CASE m
akefile Makefile.caparmor_rankX tree of the
reference code INC PHYS/ BIOLO TRAJ SEDIM
SWAN .. WORK/all the routines of the reference
code (link to CDIR) Version (contains used
HOMEMARS) (edit makefile and choose the RANK or
test case) (edit Makefile.caparmor_rankX and
choose CPP keys and compiling options) smallf90_r
ankX (link to CDIR) PHYS/KTEST/casvortex.F90 cas
vortex.F90 (link to PHYS/KTEST/casvortex.F90) edit
and modify casvortex.F90, then
compile CDIR/WCONF-CASE copy of the entire
reference code WORK contains all the routine
(links to PHYS/) smallf90_rankX contains all
the compiled routines after preprocessing
47
48
Retour au Sommaire
MARS environment (summary)

• RDIR/CONF/CONF-CASE
• namelists paramain.txt, paraspec.txt,
  paracom.txt, parasubs.txt,parasedim.txt,parabiolo.
  txt
• output.dat
• batch_seq, batch_omp, runmpi_connect (which uses
  batch_mpi and batch_connect(bydate) ),
  runmpt_connect
• mpi.txt (for MPI purpose)
• mars_exe
• Before running, the user has to
• define all the parameters in the different
  namelist (para)
• under inputs directory, define head.CONF
• under inputs directory, add the input files (like
  bathymetry, initial condition, open boundary
  conditions)
• under inputs directory, define all default .dat
  files read by the simulation (ex river.dat)

48
49
Before compilation choice of CPP keys (1/5)

• In Makefile.caparmor add CPP_keys in
• CPPFLAGS

FORMAT BATHY FILE TO READ -Dkey_bathy_final to
read definite bathymetry in netCDF format file
(no more changes)
TIDES - choice of harmonic components -Dkey_tide_s
chwid schwiderski model (1980), modele de pg
Islande-Portugal (5 km) (on progress) -Dkey_tide_
fes2004 FES 2004 -Dkey_tide_shom CST -Dkey_tide_
lagoon lagoons of Banuyls and
Marseilles -Dkey_tide_lagoon_marseille tide for
Marseilles -Dkey_tide_lagoon_banyuls tide for
Banyuls
MOMENTUM NON LINEAR TERMS default centered -Dkey_
dyn_adv_quick QUICK -Dkey_dyn_adv_quickest QUIC
KEST -Dkey_dyn_adv_linear no nonlinear terms
PRESSURE GRADIENT default pressure
jacobian -Dkey_dyn_pg_djcs Shchepetkin et al.
2004 WET DRYING -Dkey_dyn_wetdry_fct use of Flux
Corrected Transport to correct ssh and velocity
estimates and keep positive water depths
50
Before compilation choice of CPP keys (2/5)

• In Makefile.caparmor add CPP_keys in
• CPPFLAGS

TRACER ADVECTION (xy direction) default Quick /
Euler-Upwind scheme -Dkey_tssub_adv_ultimatequicke
st quickest scheme (xy direction) ultimate
limiter -Dkey_tssub_adv_ultimatequickestmacho
quickest scheme (xy direction) ultimate
limiter multidimensional extension
macho -Dkey_tssub_adv_fourthorder fourth order
centred scheme (xy direction) (along
z) default centered scheme -Dkey_tssub_wquickupwi
nd QUICK scheme along the vertical (tracer
advection) -Dkey_tssub_wcompact Upwind Compact
and Conservative scheme along the vertical
TRACER DIFFUSION default along sigma -Dkey_ts_diff
h_geo along geopotential -Dkey_ts_diffh_rho along
isopycnes
50
51
Before compilation choice of CPP keys (3/5)

• In Makefile.caparmor add CPP_keys in
• CPPFLAGS

HEAT FLUX (radiative fluxes - long and short
waves) -Dkey_sflx_solar_luyten Luyten et al,
1992 -Dkey_sflx_solar_gill Gill,
1982 -Dkey_sflx_ir_swimbank Swimbank (Agoumi
thesis) HEAT TURBULENT FLUXES (latent and
sensible fluxes and wind stresses) -Dkey_sflx_turb
_fairall Fairall et al.,2003 -Dkey_sflx_turb_ayin
a Ayina et Bentamy, 2007 -Dkey_sflx_turb_large L
arge and Yeager, 2004 -Dkey_sflx_turb_luyten Luyt
en et al, 1992 -Dkey_turb_default POL, used for
NOMADS 2 program ROTATED GRID -Dkey_grid_rotated
to use a rotated grid -Dkey_tide_saverotatedobc
to save 2D obc for a rotated child
rank AGRIF -Dkey_agrif use both keys at the
same time -Dkey_agrif -DAGRIF_MPI
-Dkey_MPI_2D use these 4 keys at the same time to
run with MPI
51
52
Before compilation choice of CPP keys (4/5)

• In Makefile.caparmor add CPP_keys in
• CPPFLAGS

TRACER -Dkey_substance -Dkey_subs_part_eqsubdt t
o save time for substance with settling
velocity SEDIMENTOLOGY --Dkey_substance
-Dkey_sedim -Dkey_sedim_mudsand to use mud/sand
sediment module -Dkey_substance -Dkey_sedim
-Dkey_sedim_mixsed to use mixed sediment
module BIOGEOCHEMISTRY AND CONTAMINANTS -Dkey_s
ubstance -Dkey_contaminant -Dkey_sedim
-Dkey_sedim_mudsand add these keys if some
contaminant are particulate -Dkey_species add
this key to grouping species of a contaminant
BIOLOGY -Dkey_substance -Dkey_biolo -Dkey_messat
add this key to read sea surface temperature
mesurement
52
53
Before compilation choice of CPP keys (5/5)
Retour au Sommaire
Return new configuration list

• In Makefile.caparmor add CPP_keys in
• CPPFLAGS

MPI -Dkey_MPI_2D MPI OMP -Dkey_MPI_2D
-Dkey_MPIOMP TRAJECTORIES -Dkey_trajec3d trajec
tories for 3D simulations (called alone without
-Dkey_ibm) Individual Based Model -Dkey_ibm
Biological processes for individual
particules (called alone without
-Dkey_trajec3d) TEST CASES -Dkey_tssub_casadvsmol
artime test tracers schemes for Smolarkievicz
configuration OFFLINE (no estimate of dynamics,
tracers advection only) -Dkey_offline_outuvflux s
ave time averaged 3d velocities and sea
level -Dkey_offline read time averaged dynamics
variables and run the code without the
dynamics part
53
54
After compilation Before runningParametrization
(1)

• ? cd RDIR/CONF/CONF-CASE
• Choice parameters and options in namelists files
  
• paramain.txt general for the configuration
  (shared with all ranks)
• paracom.txt shared with all ranks
• paraspec.txt different for each rank
• parasubs.txt need if transport substances
  (key_substance)
• parasedim.txt need if sediment using
  (key_sedim)
• parabiolo.txt need if using modules for biology
  (key_biolo)
• paraconta.txt need if using modules for
  contaminant (key_contaminant)
• Choice of options and parameters definig outputs
  files in output.dat
• Build a file mpi.txt (for MPI purpose if you want
  to divide the domain along X AND Y)

54
55
After compilation Before runningParametrization
(2)
Retour au Sommaire

• ? cd RDIR/CONF/inputs
• modify head.CONF (keep the same format !!!)
• add the input files (like bathymetry, initial
  condition, open boundary conditions, meteo )
• modify all default .dat files read by the
  simulation before any use and add the good path
  in namelists (ex river.dat, trajec2d.dat,
  outflow.dat, variables.dat)

• ? cd RDIR/CONF/CONF-CASE
• Run with mars_exe or in batch with batch_seq,
  batch_omp, runmpi_connect (which uses batch_mpi
  and batch_connect(bydate) ), or runmpt_connect
  depending of your choice (click here for details)

55
56
Return to parametrization
Parametrization (3) File head.CONF

• A default file head.CONF is automatically copied
  onto inputs directory
• DO NOT MODIFY ITS FORMAT (no more space or less
  space !)
• The first 4 letters must be equal to the variable
  nrac in routine parameters.F90
• The following number is the number of the rank
• The 4 following reals define the extension of the
  domain
• Northern latitude (at jmax)
• Southern latitude (at jmin1)
• Eastern longitude (at imax)
• Western longitude (at imin1)
• The 2 following reals set the step of latitude
  and longitude respectively (in degrees)
• The 2 following reals set the averaged step of
  latitude and longitude respectively (in meter)
  indicative values, not used by the model
• 1st integer imax
• 2nd integer jmax
• The last 4 integers are the location of the open
  boundaries of the following rank (child grid) in
  the grid of the current rank jobcmax,jobcmin,iob
  cmax,iobcmin.
• The last 4 letters is the CONF name of the child
  grid (following rank or rank of another
  configuration as a unique rank is able to save 2d
  open bounary files for several configurations at
  a same time)
• PS to save several 2d obc files, duplicate the
  line and modify the last 4 integers and the last
  4 letters.

57
Parametrization (4a) file output.dat
Return to parametrization
setting up of output files Located under
RDIR/CONF/CONF-CASE(/rank)

• THREE REQUIREMENTS
• THE MAIN OUPUT MUST BE DEFINED AT FIRST
• The user can choose l_demer.true. or
  l_out_precdate.true. (both at the same time is
  not allowed)
• The choice l_demer.true. or l_out_precdate.true.
  is possible just once inside the file
  output.dat.
• Definition of parameters are given at the
  beginning of the output.dat file
• See below the list of parameters

58
Parametrization (4b) file output.dat
Return to parametrization

• 
  
• Champs , the name of the output
  file will be champs_suffix.nc, suffix being
  defining in paracom.txt
• 06/03/2005 000000 , date_startout
  (date of the beginning output)
• 09/03/2005 000000 , date_endout
  (date to the end output)
• 1.0d0 , pasor (time step
  of saving)
• .true. , l_posit (.true.
  If horizontal extent given with indexes i j
• 
  .false. if given with longitude/latitudes)
• 0 421 0 501 , iextmin,iextmax,jextmin
  ,jextmax (horizontal extent)
• 1 30 , kextmin,kextmax
  (between 1 and kmax) (vertical extent)
• 1 1 1 ,
  i-step,j-step,k-step (space step for i,j, k)
• .false. , l_filebydate (a
  file for each date (if.true) or all records in
  the same (oif false)
• .false. , l_out_off
  (saving of 3D velocities in conservative flux
  form (for offline purpose)
• .false. , l_out_precdate
  (saving at precise dates)
• .false. , l_demer
  (filtering fields using Demerliac filter)
• .true. , l_out_ssh (saving
  of sea surface height)
• .true. , l_out_ubt (saving
  of barotropic (2D) velocities)
• .false. , l_out_u3d
  (saving of total (3D) velocities)

58
59
MPI job domain decomposition (1/2)

• PREPARATION Create a mpi.txt file if you want
  to divide your domain along X and Y
• Use decoupe program under HOMEMARS/../TOOLS/MPI2D
  _DOMAIN (see Readme.txt)
• cp bathy_CONF.txt RDIR/CONF/CONF-CASE/rankX/mpi
  .txt
• MPI2D management
• none mpi.txt, imax gt jmax
• domain
  decomposition through vertical stripes
• none mpi.txt, jmax gt imax
• domain
  decomposition through vertical stripes
• mpi.txt exist (issued from decoupe program)
• optimized
  2D domain decomposition

59
60
MPI job domain decomposition mpi.txt 2/2)
Retour au Sommaire
Return to parametrization

• 42 total
• 7 largeur
• 6 hauteur
• 42 demandé
• 32 nb actifs
• Niproc Njproc limin limax
  ljmin ljmax actif effectif
• 0 0 0
  195 0 62 T
  11251
• 1 0 196
  350 0 62 T
  7966
• 2 0 351
  487 0 62 T
  8631
• 3 0 488
  604 0 62 T
  5945
• 0 1 0
  195 63 134 T
  11209
• 1 1 196
  350 63 134 T
  11160
• 2 1 351
  487 63 134 T
  9864
• 5 1 791
  942 63 134 T
  11225
• 6 1 943 1100
  63 134 T 4644
• 0 3 0
  195 207 288 F
  0
• 1 3 196 350
  207 288 T 10977
• 2 3 351 487
  207 288 T 11234
• 3 3 488 604
  207 288 T 8792

For simplification purpose, this file is not
complete
61
Retour au Sommaire
MARS USE How to run a job

• cd RDIR/CONF/CONF-CASE/(rank)

PATHS , MODULES and batchs are specific of each
way batch_seq, batch_omp, Runmpit_connect and
batch_ompmpit etc. are created at the time of
the creation of the configuration (mkconfdir).
They are copied automatically under
RDIR/CONF/CONF-CASE/(rank)

• Sequential job
• OpenMP job
• MPI job
• Hybrid job (MPIOPENMP)

62
Sequential job
Return to run job

• RUN
• If interactive job over 10 maximum
  mars_exe
• If long run (gt 10 min under caparmor calculator)
  qsub batch_seq

• PATH MODULES (caparmor)
• source /usr/share/modules/init/csh
• module load intel-comp/11.1.073 (path for intel
  fortran and c)
• COMPILATION
• cd UDIR/CONF/CONF-CASE
• vi Makefile.caparmor_rankX
• FC ifort LINKER ifort
• DIRNC /home1/caparmor/rramel/netcdf4.1.2
• CPPFLAGS everything except key_MPI_2D
• OMP (empty)
• Compile

This sequential job is possible if you are using

63
OpenMP job
Return to run job

• RUN
• qsub ./batch_omp (queue parallel8)
• select1 to use with 1 unique node
• ncpus8 to use 8 CPUs by node

(click here to see an example of batch_omp file)
This OpenMP job is possible if you are using

• PATH MODULES (caparmor)
• source /usr/share/modules/init/csh
• module load intel-comp/11.1.073 (path for intel
  fortran and c)
• COMPILATION
• cd UDIR/CONF/CONF-CASE
• vi Makefile.caparmor_rankX
• FC ifort LINKER ifort
• DIRNC /home1/caparmor/rramel/netcdf4.1.2
• CPPFLAGS everything except key_MPI_2D
• OMP -openmp
• Compile

64
MPI job (1/3)

• RUN
• MPI linux
• runmpi_connect name_run name_outputfile nb_cpu Z
  HOME
• MPT MPI for SGI but more efficient than MPI
  under Caparmor
• runmpt_connect name_run name_outputfile nb_cpu Z
  HOME
• name_run will get name_run0.oXXX and
  name_run1.oXXX files at the end of the job, XXX
  being the job number under caparmor
• name_outputfile string_suffix, string is
  choosen at the first line in output.dat, suffix
  is defined in paracom.txt (ex champs_V9.06)
• nb_cpu from 2 to XXX
• Z0 if l_out_nc4par.true. Z1 if
  l_out_nc4par.false. And l_filebydate.false.
  Z2 if l_out_nc4par.false. and
  l_filebydate.true.
• The fifth argument home allows a faster
  creation of the concatened champs_VX.nc.all file
  under Caparmor calculator
• This script submits the mpi run and the gathering
  of output files
• (1st use of MPI
• cd HOME (off Caparmor)
• mkdir pbs touch .mpd.conf chmod 600 .mpd.conf)

(click here to see an example of batch_mpi file
which is launched by runmpi_connect)
64
65
MPI job (2/3)
This MPI job is possible if you are using

• PATH MODULES (caparmor)
• MPI linux
• source /usr/share/modules/init/csh
• module load intel-comp/11.1.073 (path for intel
  fortran and c)
• module load intel-mpi/4.0.0.028 (path for library
  MPI)
• MPT (MPI for SGI, more efficient under Caparmor)
• source /usr/share/modules/init/csh
• module load intel-comp/11.1.073 (path for intel
  fortran and c)
• module load mpt/2.01 (path for library MPT)
  (better to use the command module switch
  intel-mpi mpt/2.01)

66
MPI job (3/3)
Return to run job
This MPI job is possible if you are using for

• COMPILATION
• cd UDIR/CONF/CONF-CASE
• vi Makefile.caparmor_rankX
• FC mpiifort LINKER
  mpiifort
• DIRNC /home1/caparmor/rramel/netcdf4.1.2_par
  (for MPI use)
• DIRNC /home1/caparmor/rramel/netcdf4.1.2_mpt
  (for MPT use)
• CPPFLAGS add -Dkey_MPI_2D
• OMP (empty)
• Compile

66
67
Hybrid job (openMPMPI) (1/3)

• RUN
• MPI linux
• runmpiomp_connect name_run name_outputfile nb_cpu
  nb_omp Z HOME
• MPT MPI for SGI
• runmptomp_connect name_run name_outputfile nb_cpu
  nb_omp Z HOME
• nb_cpu is the total cpus used for the run
• nb_omp is the number of processes involved in
  the OMP parallelization. It must be smaller or
  equal to 8 (on Caparmor, since there are 8 cpus a
  node)
• The number of MPI cpus is nb_cpu / nb_cpu. nb_cpu
  must be a multiple of nb_cpu !
• For the other arguments, same signification as
  for runmpi_connect

68
Hybrid job (openMPMPI) (2/3)
This openMPMPI job is possible if you are using

• PATH MODULES (caparmor)
• MPI linux
• source /usr/share/modules/init/csh
• module load intel-comp/11.1.073 (path for intel
  fortran and c)
• module load intel-mpi/4.0.0.028 (path for library
  MPI)
• MPT (MPI for SGI)
• source /usr/share/modules/init/csh
• module load intel-comp/11.1.073 (path for intel
  fortran and c)
• module mpt/2.01 (path for library MPT) (better to
  use the command module switch intel-mpi mpt/2.01)

68
69
Hybrid job (openMPMPI) (3/3)
Return to run job
This openMPMPI job is possible if you are using

• COMPILATION
• cd UDIR/CONF/CONF-CASE
• vi Makefile.caparmor_rankX
• FC mpiifort LINKER
  mpiifort
• DIRNC /home1/caparmor/rramel/netcdf4.1.2_par
  (for MPI use)
• DIRNC /home1/caparmor/rramel/netcdf4.1.2_mpt
  (for MPT use)
• CPPFLAGS add -Dkey_MPI_2D Dkey_MPIOMP
• OMP -openmp
• Compile

69
70
Run a job under caparmor example of a batch
file (OMP)
Retour au Sommaire
Return to run job

• Batch_omp
• !/bin/csh
  ! Script in cshell
• PBS -q parallel8
  ! Choice of the queue
• PBS -l select1ncpus4
  ! 1 node, 4 cpus
• get the path for library MKL
• source /usr/share/modules/init/csh
• module load cmkl/recent
• setenv MKL_SERIAL YES
• default OMP_SCHEDULE is "static", instead use
  dynamic for mars
• setenv OMP_SCHEDULE "dynamic,1"
• cd to the directory you submitted your job
• cd PBS_O_WORKDIR
• pwd
• date
• ./mars_exe gt ./mars_out_omp

71
Run a job under caparmor example of a batch
file (MPI)
Retour au Sommaire
Return to run job

• Defined from runmpi_connect that launches
  batch_mpi
• PBS -q parallel8
  ! Choice of the queue
• PBS -l select1ncpus4mpiprocs4 ! 1
  node, 4 cpus and 4 procs a node (one is certain
  not to share the node with another job)
• Batch_mpi
• !/bin/csh
  ! Script in cshell
• get the path for mpirun
• source /usr/share/modules/init/csh
• module load intel-mpi/recent
• get the path for library MKL
• module load cmkl/recent
• setenv MKL_SERIAL YES
• cd to the directory you submitted your job
• cd PBS_O_WORKDIR
• pwd
• date

72
Run a job under caparmor (Summary 1/2)

• batch_seq, batch_omp, are created at the time of
  the creation of the configuration (mkconfdir).
  They are copied automatically under
  RDIR/CONF/CONF-CASE/(rank)
• Path and modules (caparmor)
• source /usr/share/modules/init/csh
• module load intel-comp/11.1.073
• Sequential job
• Makefile.caparmor
• FC ifort LINKER ifort
• DIRNC /home1/caparmor/rramel/netcdf4.1.2
• qsub batch_seq
• OpenMP job
• Makefile.caparmor
• FC ifort LINKER ifort
• DIR