Earth System Modeling Framework

1
Earth System Modeling Framework
ESMF Introductory Tutorial
ESMF v 3.0.1 http//www.esmf.ucar.edu
2
Introduction and Orientation
http//www.esmf.ucar.edu
3
Outline

• Administrivia
• Background
• Building ESMF
• HPC terminology
• How to create an ESMF component
• Coupling
• Setting up a parallel environment
• ESMF infrastructure Data structures
• ESMF infrastructure Utilities

4
Rough Timeline

• 830-900 Introduction and orientation
• 900-1030 Building and linking exercise
• 1030-1045 Break
• 1045-noon Terminology, superstructure,
  coupling exercise
• noon-1 pm Lunch on your own
• 100-220 Parallel environment, data
  structures, methods, and code review
• 220-235 Break
• 235-300 Utilities

5
Outline

• Administrivia
• Background
• Building ESMF
• HPC terminology
• How to create an ESMF component
• Coupling
• Setting up a parallel environment
• ESMF infrastructure Data structures
• ESMF infrastructure Utilities

6
Administrivia

• Bathrooms down the hall
• Cafeteria in next building over
• Refreshments in the hall
• No stupid questions, ask anytime
• You can tell us to speed up or slow down
• During exercises let us know if you are having
  problems or dont understand

7
Outline

• Administrivia
• Background
• Building ESMF
• HPC terminology
• How to create an ESMF component
• Coupling
• Setting up a parallel environment
• ESMF infrastructure Data structures
• ESMF infrastructure Utilities

8
What is ESMF?

• What is a framework?
• Framework vs. library
• A toolbox for modelers
• Turn model codes into components
• Couple model components
• Provide code infrastructure
• Time management
• Log management
• Etc.

9
Superstructure vs. Infrastructure

• Superstructure The means by which models are
  converted into components and coupled
• Infrastructure Tools that modelers can use
  within their own code
• Data structures Array, Field, etc.
• Utilities TimeMgr, LogErr, etc.
• These are independent, but work together

10
ESMF Structure
GridComp Land, ocean, atm, model
CplComp Xfers between GridComps
State Data imported or exported
Superstructure
Infrastructure
Regrid Computes interp weights
Bundle Collection of fields
Field Physical field, e.g. pressure
Grid LogRect, Unstruct, etc.
PhysGrid Math description
DistGrid Grid decomposition
F90
Array Hybrid F90/C arrays
DELayout Communications
Route Stores comm paths

C
Utilities Virtual Machine, TimeMgr, LogErr, IO,
ConfigAttr, Base etc.
Data
Communications
11
Whats New in ESMF 3

• Simplified build and link
• Public DistGrid - stores grid topology
• New Array Class extensions to native arrays
• Regridding w/ ESMF_ArraySparseMatMul()
• User provided interpolation weights
• Parallel sparse matrix multiply

12
Support

• Mailing lists to join
• esmf_jst_at_ucar.edu Technical discussion
• esmf_info_at_ucar.edu Quarterly newsletter
• esmf_community_at_ucar.edu Community announcements
• Location to join http//www.esmf.ucar.edu gt
  Users
• Support
• Send email to esmf_support_at_ucar.edu

13
Web

• Main site http//www.esmf.ucar.edu

14
Reference Manual

• Go to http//www.esmf.ucar.edu/
• Click Download tab
• Click View All Releases link (left nav bar)
• Scroll down to v3_0_1
• Click on the HTML version of the Reference Manual
  in the table

15
Downloading

• Go to http//www.esmf.ucar.edu/
• Click Download tab
• Latest public release here
• Click View All Releases link (left nav bar)
• Scroll down to v3.0.1
• Click to download and accept license
• Alternate Go to http//cdp.ucar.edu
• Must register and join ESMF group
• All releases are available
• Source is a gzipped tar file - must be unpacked

16
Feature Prioritization

• Issues and feature requests arrive via
  esmf_support_at_ucar.edu
• SourceForge tracker ticket is created
• CRB reviews issues and establishes release
  schedule
• http//www.esmf.ucar.edu/plans/releases/index.shtm
  l
• Prioritization can be influenced by participating
  in JST telecons

17
Future Development

• New Grid development
• Support for structured grids
• Support for unstructured meshes.
• Automated regridding
• C framework interfaces

18
Questions?
19
Outline

• Administrivia
• Background
• Building ESMF
• HPC terminology
• How to create an ESMF component
• Coupling
• Setting up a parallel environment
• ESMF infrastructure Data structures
• ESMF infrastructure Utilities

20
Building ESMFHands-On
21
Section Outline

• Requirements to build
• Set environment variables
• Build library
• Common build problems
• Directory structure
• Build and run tests

22
Requirements to Build

• Fortran 90 compiler (g95, ifort, )
• C compiler (gcc, icc, )
• MPI libraries (MPICH, LAMMPI, ...)
• gnus make gmake (not make)
• prompt make -v
• indicates type
• Perl (for running test scripts)

23
Environment Variables Review

• Typical values for Linux machines
• ESMF_DIR Top level ESMF directory path
• ESMF_BOPT g for debug, O0,1,2, optimize
• ESMF_COMM mpich, mpich2, lam, openmpi, intelmpi,
  or mpiuni for serial mode.
• ESMF_COMPILER intel, xlf, absoft,lahey, pgi,
  pathscale, g95
• ESMF_ABI (Application binary interface) System
  pointer word size/memory model 32 or 64
• See the README file and Environment Variables
  in the Quick Start section of the Users Guide
  for more info

24
Required Environment Variables

• Environment variables must be set prior to
  building the source.
• export ESMF_DIRabsolute_path
• export ESMF_BOPTg
• export ESMF_COMMmpich2
• export ESMF_COMPILERg95
• Check the settings with
• make info

25
Building the Framework

• Download the tarball (already done for tutorial)
• Unpack and build the framework
• prompt gunzip esmf_3_0_1_src.tar.gz
• prompt tar -xvf esmf_3_0_1_src.tar
• Set environment variables
• prompt more set
• prompt export ESMF_DIRabsolute_path
• prompt export ESMF_BOPTg
• prompt export ESMF_COMMmpich2
• prompt export ESMF_COMPILERg95
• Build prompt cd /esmf make

26
Review Build Commands

• Clobber any previous build
• make clobber
• Build the framework
• make
• Display build settings
• make info
• NOTE about gnumake On many Linux systems gnumake
  or gmake is renamed make. Typically this is not
  the case on non-Linux platforms.

27
Common Build Problems

• Old versions of required software
• gcc
• gnumake
• Environment variable settings
• Library paths
• 32 bit machines set to 64 bit
• Software install
• Compiler build options
• MPI build options

28
Modify Build Files

• prompt cd /esmf/build_config
• prompt ls
• Linux.absoftintel.default/
• Linux.pgigcc.default/
• Linux.g95.default/
• Linux.gfortran.default/
• Linux.xlf.default/
• Linux.intel.default/
• See the Porting to New Platforms section of the
  Users Guide

29
Directory Structure (1 of 2)

• Change to the framework directory
• prompt cd /esmf/
• README Overview ESMF of build run
• application/ Example of a standalone application
  code
• build/ Make helper file common.mk
• build_config/ Config file for building on
  different platforms
• lib/ Location of ESMF library (esmf.mk
  libesmf.a)

30
Directory Structure (2 of 2)

• makefile Top level makefile for building and
  running
• mod/ Shared modules
• quick_start/ Toy example of a coupled model
• src/ Source code
• demo/
• Infrastructure/
• Superstructure/
• system_tests/

31
Starting MPI

• Required to run
• Some flavors of MPI, such as LAM MPICH2,
  require starting an external process manager
  before running programs.
• mpdboot MPICH2
• Run the something here!!!!!
• run_something
• Halt MPI
• mpdallexit MPICH2

32
Verify the Build

• Build the unit tests
• prompt make build_unit_tests
• Run the unit tests
• prompt mpdboot
• prompt make run_unit_tests

33
What Did We Learn?

• Environment variables are key
• gnumake info can helpful in debugging
• The library build takes time
• Where to find things in the source directories
• Running the tests can confirm the build

34
Exercise Installing ESMF and Building and
Linking to a Stand-Alone Application
35
ESMF Installation

• Extracts the minimum library and helper files
  needed to link any stand-alone application to the
  framework
• Allows these files to be placed in a public space
• Eliminates need for keeping the entire source tree

36
ESMF Installation

• prompt make info
• Examine the installation environment variables
• Set installation environment variables
• ESMF_INSTALL_PREFIX/ESMF301
• ESMF_INSTALL_MODDIRmod
• ESMF_INSTALL_LIBDIRlib
• prompt more stand
• prompt make install
• prompt make installcheck
• See the Building and Installing section of the
  Users Guide for more info

37
The Stand-Alone Code

• prompt mkdir /exercise
• prompt chmod 755 -R /esmf
• prompt cd /esmf/application
• prompt ls
• Makefile
• esmf_application.f90
• prompt cp /exercise
• prompt cd /exercise
• prompt more Makefile

38
Setting up the Makefile(1 of 3)

• Step 1 Completing the Makefile path
• The Makefile requires a path to an esmf.mk
• There can be multiple versions if you have built
  with different flags
• Different versions represent different build
  choices
• Debug or Optimize
• Compiler choice
• MPI choice
• Step 2
• prompt cd /esmf/lib/
• prompt ls
• Explore the directories created by your build
  choices

39
Setting up the Makefile(2 of 3)

• Step 3 Choose the path to esmf.mk consistent
  with the desired system configuration
• Example path
• /esmf/lib/libg/Darwin.absoft.32.lam.default/
• libg gt set up in debug mode
• Darwin gt operating system
• Absoft gt compiler
• 32 gt ESMF_ABI
• lam gt MPI flavor
• What is your path?

40
Setting up the Makefile(3 of 3)

• Step 4 prompt cd /ESMF301/lib
• Step 5 prompt ls
• Notice the alternative path to esmf.mk
• Save this path
• Step 6 prompt cd /exercise/
• Step 7 Open the Makefile in an editor
• Replace the dummy path with the captured path, to
  end of the variable ESMF_INSTALL_LIBDIR_ABSPATH
  on line 14
• Step 8 Save and exit

41
Building Running the Application

• Build
• prompt cd /exercise
• prompt make
• Running
• prompt mpdboot
• prompt mpirun -np 4 esmf_application
• prompt mpdallexit

42
What Did We Learn?

• All we need to link to the ESMF library with a
  stand-alone application is the appropriate
  esmf.mk file
• We dont need the environment variables to build
  and link to the the stand-alone application once
  the framework is built.

43
Questions?
44
Outline

• Administrivia
• Background
• Building ESMF
• HPC terminology
• How to create an ESMF component
• Coupling
• Setting up a parallel environment
• ESMF infrastructure Data structures
• ESMF infrastructure Utilities

45
HPC Terminology
46
Serial vs. Parallel

• Future computing platforms will consist of
  multiple processors
• Often dont share memory
• Can have multiple threads of execution per
  processor
• MPI OpenMP are the standards
• Programs can run in a serial fashion, w/ one
  thread or,
• Parallel using multiple threads

47
PETs Persistent Execution Thread

• We call threads PETs
• In this introductory tutorial, well stick with
  one PET per processor
• Serial applications run on one PET, parallel
  applications run on multiple PETs

48
Decomposition Elements (DEs)

• A data decomposition is represented as a set of
  Decomposition Elements (DEs)
• In many applications there is one DE per PET

49
Simple Decomposition Illustration
50
Sequential Execution

• Components run one after another on the same set
  of PETs

51
Concurrent Execution

• Components run simultaneously on different sets
  of PETs

52
SPMD Single Program Multiple Datastream

• The same program runs across all PETs in the
  application
• Components may run sequentially or concurrently

53
MPMD Multiple Program Multiple Datastream

• The application consists of separate programs
  launched as separate executables
• Components almost always run concurrently

54
How does this relate to ESMF?

• ESMF supports both sequential and concurrent
  modes of execution
• ESMF supports SPMD but only very limited MPMD

55
Questions?
56
ESMF Superstructure
57
2 Reasons to Use the Superstructure

• To couple to other ESMF components
• To organize your code hierarchically into
  subcomponents

GEOS-5 AGCM
58
Definitions

• Component model A users atmosphere, land, or
  other geophysical code
• Application A collection of component models and
  the couplers that connect them

59
Simplest Coupled ESMF Application
60
Questions?
61
Outline

• Administrivia
• Background
• Building ESMF
• HPC terminology
• How to create an ESMF component
• Coupling
• Setting up a parallel environment
• ESMF infrastructure Data structures
• ESMF infrastructure Utilities

62
Creating an ESMF Component using the ESMF
Superstructure
63
What is an ESMF Component?(1 of 2)

• A component model that has ESMF callable
  standardized interfaces Initialize, Run,
  Finalize
• Has a standard interface
• call ESMF_GridCompRun(component, stateIn,
  stateOut, clock, rc)
• Not terribly complicated
• For legacy code the interfaces are created using
  wrappers

64
What is an ESMF Component?(2 of 2)

• There are two types of ESMF components
• GridComp Typically a geophysical model
  associated with a single grid.
• CplComp Arranges and executes data
  transformations, regridding, redistribution, unit
  transformations, etc.
• Each has standard methods

65
Steps to Componentization (1 of 2)

• Divide code into initialize, run, and finalize
  methods
• Describe the distribution of grids over parallel
  resources via the VM (list of PETs) and DELayout
  (arrangement of DEs)
• Pack data to be transferred into ESMF import and
  export States via Bundles, Arrays, or Fields

66
Steps to Componentization (2 of 2)

• Pack time information into ESMF time management
  data structures
• Write a set services routine that identifies each
  components initialize, run, and finalize methods
• Run using an application driver

67
Steps in Registering SetServices(1 of 2)

• Step 1 Wrap the 3 set entry point calls in a
  user created registration routine
• subroutine my_set_services(comp, rc)
• Arguments
• comp gt component name (GridComp)
• rc gt return code (integer)

68
Steps in Registering SetServices(2 of 2)

• Step 2 call ESMF_GridCompSetEntryPoint(comp,
  sub_type, sub_name, 2, rc)
• Arguments
• comp gt component name (GridComp)
• sub_type gt subroutine type (character)
• ESMF_SETINIT, ESMF_SETRUN, or ESMF_SETFINAL
• sub_name gt name of users subroutine
• phase gt EMSF_SINGLEPHASE if 1, and if gt1
  (integer)
• rc gt return code (integer)
• 3 calls needed, one for each init, run,
  finalize
• may be more calls if there are multiple phases

69
Exercise ESMF Componentization

• prompt cd /esmf/src/system_tests/
• ESMF_ArraySparseMatMul/
• prompt ls
• Open
• User_model1.F90
• ESMF_ArraySparseMatMalSTest.F90

70
What Did We Learn?

• Componentization is simplified by dividing user
  models into initialize, finalize, and run
  routines
• How to register Components with set services

71
Questions?
72
Outline

• Administrivia
• Background
• Building ESMF
• HPC terminology
• How to create an ESMF component
• Coupling
• Setting up a parallel environment
• ESMF infrastructure Data structures
• ESMF infrastructure Utilities

73
Coupling Using the ESMF Superstructure
74
3 Ways to Couple Within an Existing Application

• Convert an application into ESMF components and
  within the ESMF coupler use own transformations
  and regridding
• Use ESMF Arrays and ESMF_ArraySparseMatMul()
• User supplied weights
• Use ESMF regridding on the components
• Black box

75
Steps in Coupling

• Same as the steps for creating an ESMF component
  application except
• Also create register a coupler component
• Use the ESMF_CplCompXXX() structure
• Choose coupling method
• Run using an application driver

76
ESMF States

• All data passed between Components must be in the
  form of States
• Reference to other ESMF data objects
• Data is referenced so does not need to be
  duplicated
• Can be Bundles, Fields, Arrays, other States, or
  name-placeholders

77
Exercise Coupling Using ESMF
78
Examine the Code

• prompt cd /esmf/src/system_tests/ESMF_ArraySparse
  MatMul/
• Open
• ESMF_ArraySparseMatMalSTest.F90
• User_coupler.F90
• User_model1.F90
• User_model2.F90

79
What Did We Learn?

• Coupling requires a Coupler Component
• Creating this follows the same steps as creating
  a Gridded Component
• States are the only way to communicate between
  Components
• There are multiple coupling options
• The process is scalable additional components
  are created the same way

80
Questions?
81
Outline

• Administrivia
• Background
• Building ESMF
• HPC terminology
• How to create an ESMF component
• Coupling
• Setting up a parallel environment
• ESMF infrastructure Data structures
• ESMF infrastructure Utilities

82
VM (Virtual Machine)

• Abstraction of machine resources-elements are
  PETs
• Allows for a consistent interface to multiple
  flavors of MPI
• Handles relationship between machine resources
  and the model distribution
• Does simple MPI-lite communications between PETs

83
VM Example

• Open Web browser
• Click on Download
• Click on View All Releases
• Scroll to 3.0.1
• Open Reference Manual
• Navigate to section 40.2.1

84
DELayout

• Handles decomposition- elements are DEs
• Maintains relationship between PETs and DEs
• Default DEs PETs
• Oversubscribed DEs gt PETs
• Share resources between objects DEs lt PETs

85
DELayout Example

• Open Web browser
• Click on Download
• Click on View All Releases
• Scroll to 3.0.1
• Open Reference Manual
• Navigate to section 39.2

86
DistGrid

• Represents Grid index space and topology
• Together with VM and DELayout defines the data
  distribution of the domain decomposition across
  the computational resources

87
DistGrid Example

• Open Web browser
• Click on Download
• Click on View All Releases
• Scroll to 3.0.1
• Open Reference Manual
• Navigate to section 23.2

88
Questions?
89
Outline

• Administrivia
• Background
• Building ESMF
• HPC terminology
• How to create an ESMF component
• Coupling
• Setting up a parallel environment
• ESMF infrastructure Data structures
• ESMF infrastructure Utilities

90
Infrastructure Data Structures
91
Data Structures Overview

• Model data is contained in a hierarchy of data
  structures. The user can reference a Fortran
  array to an ESMF Array or Field, or retrieve a
  Fortran array out of an ESMF Array or Field.
• Array holds a Fortran array (with other info,
  such as halo size)
• Field holds an ESMF Array, an associated Grid,
  and metadata
• Bundle collection of Fields on the same Grid
  bundled together for convenience and data
  locality

92
Overview of ESMF Array

• Distributed representation of a multidimensional
  array
• Stores distributed data in index space
• Provides DE-local memory allocations
• Can be created, destroyed, copied, and indexed

93
Array Methods

• ESMF_ArrayCreate()
• From local F90 array
• From local array object
• ESMF_ArrayGet()
• Internals
• Local patch via F90 pointer
• Local patch via local array object
• ESMF_ArrayScatter()
• ESMF_ArrayDestroy()

94
Fields

• An ESMF Field provides methods for
• Initialization
• Setting retrieving data values
• Data redistribution regridding
• Communication
• Gather and scatter
• Manipulation of attributes

95
Bundles

• The Fields in a Bundle may be located at any
  stagger location of the associated grid
• The Fields in a Bundle may be of different rank,
  as long as the Grid dimensions are distributed
  the same
• Bundles are intended for performance
  optimization, by sharing collective
  communication, IO, regridding

96
Questions?
97
Data Structure MethodsRegridding and
Redistribution
98
Regridding via ESMF_ArraySparseMatMul()

• An Array method - so acts on distributed objects.
• Functionality underneath regridding
• Once you have the weights, regridding can be cast
  as a large sparse matrix multiplication problem.
• Does regridding given user defined weights
• See appendix for motivation

99
ESMF_ArraySparseMatMul() (1 of 2)

• Stage 1 Precompute store
• Done only once
• Scatters the interpolation weights and their
  addresses
• ESMF_ArraySparseMatMulStore( srcArray
  MySrcArray, dstArray MyDstArray, factorList
  remap_matrix, factorIndexList
  remap_addresses, rootPet0, ... )

100
ESMF_ArraySparseMatMul() (2 of 2)

• Stage 2 Conduct Parallel Multiply
• ESMF_ArraySparseMatMul(srcArray MySrcArray,
  dstArray MyDstArray, ...)

101
Redistribution

• Data movement between two distributions.
• The source destination grids must have the same
  coordinates.
• Field performs a redistribution operation of the
  data in an ESMF_Field
• Bundle performs a redistribution operation of the
  data in an ESMF_Bundle

102
Exercise Bringing it All Together

• An example of multi-component coupling using the
  ESMF_ArraySparseMatMul()

103
A Look Back at the Coupled Code

• prompt cd /esmf/src/system_tests/
  ESMF_ArraySparseMatMul/
• Open
• ESMF_ArraySparseMatMalSTest.F90
• User_coupler.F90
• User_model1.F90
• User_model2.F90

104
Questions?
105
Outline

• Administrivia
• Background
• Building ESMF
• HPC terminology
• How to create an ESMF component
• Coupling
• Setting up a parallel environment
• ESMF infrastructure Data structures
• ESMF infrastructure Utilities

106
Infrastructure Utilities
107
Utility Overview

• Time Manager
• Configuration Attributes (replaces namelists)
• Message logging - LogErr

108
Time Manager

• Time Manager Utilities include
• Calendar
• Clock needed for superstructure
• Time
• Time interval
• Alarm
• These can be used independently of other ESMF
  utilities.

109
Calendar(1 of 2)

• Keep tracks of the date as an ESMF Gridded
  Component advances in time
• Calendars can be queried for time expressed as
  various time scales
• seconds per day
• days per month
• days per year
• etc.

110
Calendar(2 of 2)

• Supported calendars are
• Gregorian The standard Gregorian calendar
• no-leap Gregorian calendar with no leap years.
• Julian Julian calendar
• Julian Day Julian days calendar
• 360-day 12 months of thirty days/ month
• no calendar elapsed model time in seconds

111
Clocks

• Clocks contain the parameters methods needed
  for time advancement
• A Clock can be queried for quantities
• start time
• stop time
• current time
• time step
• Clock methods
• incrementing the current time
• determining if it is time to stop

112
Alarms

• Identify unique or periodic events by ringing at
  specified times.
• e.g. an Alarm can be set to ring on the day of
  the year when leaves start falling from the trees
  in a climate model

113
Time and Time Interval

• A Time represents an instant in time of a
  particular calendar
• such as November 28, 1964, at 731pm EST of the
  Gregorian calendar
• Time class represents start stop times
• Time Intervals express a time period
• such as 300 milliseconds
• Time steps can be expressed as Time Intervals

114
Config

• Config is a utility for accessing ASCII resource
  files
• Config is based on NASAs Inpak 90 package
• It is optimized for minimizing formatted I/O

115
LogErr

• Methods for writing error, warning
  informational messages
• Default log created at initialization
• Additional logs can be created
• Set of standardized return codes
• Log output includes timestamps PET numbers

116
Questions?
117
Appendices
118
Appendix A

• Regridding with ESMF_SparseMatMul()

119
Motivation(1 of 7)
120
Motivation(2 of 7)
121
Motivation(3 of 7)
122
Motivation(4 of 7)
123
Motivation(5 of 7)
124
Motivation(6 of 7)
125
Motivation(7 of 7)
126
Appendix B How to convert a user model into an
ESMF component
127
Steps

• Separate code into a new module with
• initialize, run, and finalize stages
• Create register routine
• Register the initialize, run, and finalize
  routines
• Make the register routine public ?
• Include framework module

128
user_model1
129
userm1_register
130
driver routine(1 of 2)
131
driver routine(2 of 2)