Enhancements for Hydrological Modeling in ESMF

1
Enhancements for Hydrological Modeling in ESMF
Cecelia DeLuca/NCAR (ESMF) December 19, 2008 AGU
Fall Meeting
2
Outline

• ESMF overview
• Results and projects relevant to the hydrological
  community
• ESMF data services and semantic technologies
• A peek at the future

3
Earth System Modeling Framework (ESMF)

• Component-based framework that targets tightly
  coupled, computationally demanding weather and
  climate applications written in Fortran
• ESMF provides standard interfaces for model
  components
• ESMF provides common utilities and tools for
  routine modeling functions such as regridding
  between components

• Status
• Initiated by NASA in 2002 and developed and
  managed by a multi-agency consortium
• More than 76 ESMF components in the community,
  used by the NASA Modeling, Analysis and
  Prediction Program, the Department of Defense
  Battlespace Environments Institute, the National
  Weather Service, and many others

4
ESMF Application Example
Part of the GEOS-5 Atmospheric General
Circulation Modeling system

• Each box is an ESMF gridded component or coupler
  component
• ESMF State objects carry data between components
• Every component (including couplers) has a
  standard interface to facilitate exchanges
• The ESMF architecture enables the assembly of
  many different systems

5
Interoperability and standardization of interfaces

• There are only three ESMF component methods
  Initialize, Run, and Finalize (I/R/F)
• Users associate their native code I/R/F methods
  with an ESMF Component through a registration
  process
• I/R/F methods cascade down the tree
• Small set of standard arguments

call ESMF_CompRun (myComp, importState,
exportState,clock, phase, blockingFlag, rc)
6
Outline

• ESMF overview
• Results and projects relevant to the hydrological
  community
• ESMF data services and semantic technologies
• A peek at the future

7
Data representation options

• 1. Representation in index space (Arrays)
• One or more logically rectangular tilesstore
  indices and topology
• Sparse matrix multiply for regridding withuser
  supplied interpolation weights
• Very general data representation, butlimited
  interoperability since not muchsemantic info is
  encoded by the framework
• 2. Representation in physical space (Fields)
• Built on Arrays some type of Grid
• Regridding using parallel on-line interpolation
  weight generation, bilinear or higher order
• Standard high-level data structures increase
  interoperability

Supported Array distributions
8
Coupling options in ESMF
Multiple executable options

• Lots of flexibility incoupling approaches
• Single executable
• Multiple executable
• Couplingcommunicationscan be set up and
  calledin a coupler
• Coupling communications can be setup in a coupler
  but calleddirectly from within components
  useful when it is inconvenient to return from a
  component in order to perform a coupling
  operation
• Recursive components for nesting higher
  resolution regions
• Ensemble management, including sequential
  execution of ensemble members

Contributed by U Maryland
Coupler
Comp A
Comp B
Comp A
Comp B
Array send/recv
9
New Features and Applications

• ESMF v3.1.1 beta release (September 2008)
  includes
• unstructured mesh class
• observational data streams
• on-line parallel regridding
• standard metadata packets
• Consortium of U.S. operational and research
  centers (NOAA, NASA, DoD) has entered pilot phase
  of development of multi-model global ensemble
  using ESMF components

• ESMF parallel regrid
• Grid-Grid, Grid-Mesh, Mesh-Mesh
• Bilinear and higher order

The Flow-Following Finite Volume Icosahedral
Model (FIM) from NOAA GSD is converting to ESMF
to couple to National Weather Service models
NUOPC
National Unified Operational Prediction Capability
10
Targeting greater interoperability

• ESMF is a toolkit with many options for coupling,
  and ESMF compliance alone has not meant
  plug-and-play
• Several groups have been creating additional
  guidelines, conventions, and constraints for
  increased interoperability
• Example a NUOPC operational consortium
  committee has reached detailed agreements on the
  following
• Componentization level - atm, ocean, land, ice,
  atm physics and dynamics
• Clock - how time is passed between models and
  modules
• Ownership what models and modules are allowed
  to modify data held in common (e.g. aerosol
  species and concentrations)
• Level of data representation index space or
  physical space
• Metadata the description of the data, grid,
  clock, etc.
• Coupling approach - NUOPC will provide a standard
  template.
• Discussions are underway on IO, post-processing,
  and component invocation points

11
Parallel weight calculation and regridResults
for ESMF in CCSM

• dTAUx/dy - closely related to the curl of the
  wind stress, which drives the upper ocean
  circulation and is therefore important to
  represent accurately in climate simulations
• Mapped from the T62 (2-degree) Community
  Atmospheric Model (CAM) grid to the gx1v5
  (384x320 irregularly spaced)
• Noise is calculated as deviation of a point from
  the sum of itself plus four neighbors
• 33 reduction in noise globally compared to
  previous CCSM bilinear approach
• CCSM is now using ESMF for their standard
  remapping

Interpolation noise in the derivative of the
zonal wind stress
Interp. noise
grid index in latitudinal direction
Black bilinear Red higher-orderESMF
v3.1.1 Green higher order ESMF v4.0.0
12
Performance Portablity

• 30 platform/compiler combinations regression
  tested nightly, new ports include gfortran (4.3),
  Solaris, and (almost) Windows

ESMF Sparse Matrix Multiply Run-Time
Performance at the petascale The chart at right
shows scaling of the ESMF sparse matrix multiply,
used in regridding transformations, out to 16K
processors. (ESMF v3.1.0rp2) Plot from Peggy Li,
NASA/JPL Tested on ORNL XT4 in a variety of
configurations . -N1 means 1 core per node.
13
Outline

• ESMF overview
• Results and projects relevant to the hydrological
  community
• ESMF data services and semantic technologies
• A peek at the future

14
Metadata export from ESMF (v4.0)

• ESMF holds information about grids, component
  inter-relationships
• Users can associate additional metadata with ESMF
  classes (Grids, Arrays, Fields, States,
  Components) as name/value pairs in Attribute
  packages
• Attribute packages can be standard (e.g CF
  conventions for Fields) or custom
• Packages can be nested for example, metadata
  for an import State contains metadata for all the
  Fields in the State.
• Metadata can be exported in plain text or XML
  format
• then ingested into portal, where users can search
  and browse components and datasets via the
  portal, trackback to the model configuration
  that created a dataset, create dynamic comparison
  tables for specific properties, create private or
  shared workspaces, etc.
• Excellent support for component and model
  inter-comparison projects.

The path from model metadata to web interface
involves many format translations along the way
Image courtesy of Rocky Dunlap, Georgia Institute
of technology
15
Numerical Techniques for Global Atmospheric
Models Summer Colloquium

• Held at NCAR during summer 2008, about 80
  participants
• 9 atmospheric dynamical core components archived
• Running 22 test cases at up to 5 resolutions
• Search, browse, trackback, comparison tables
• Automated upload of simulation data, publishing,
  and comparison
• Data search and browse via ESG Gateway
• Demonstrates how knowledge base can combines with
  ESMF

http//dycore.ucar.edu
Curator project sponsored by
16
Work in progress

• Translation of ESMF interfaces into web service
  interfaces to support loose coupling,
  multi-language coupling, and integration of
  components into service oriented architectures
  (Curator project)
• Integration of CCSM ESMF version with Kepler
  workflow tool (PhD student at Istanbul Technical
  University)
• Automated coupler generation (PhD student at
  Georgia Tech)
• New TeraGrid award invoke CCSM ESMF version
  from an ESG-based science gateway on the TeraGrid
  - data and metadata from the run are
  automatically stored back to the portal to
  support intercomparison projects
• Working with Cray, Microsoft and NASA to finish
  port to Windows HPC 2008

Image courtesy of Rocky Dunlap, Georgia Institute
of technology
17
A peek at the future
2009 2010 2011 2012
2013 2014 2015
Interoperability Compliance definitionsand
verification tools

Integration with workflows Generation of service
interfaces for loose coupling Metadata links to
data and other services
Automated model assembly andrun
environment Based on metadata files
Tutorial development andclassroom deployment
Numerical/computational maintenance and
development