ESMF Infrastructure Layer

1
ESMF Infrastructure Layer
2nd ESMF Community Meeting, GFDL, May 15th
2003 http//www.esmf.ucar.edu, esmf_at_ucar.edu
2
Talk Outline

• What is the infrastructure layer
• What is it for
• What it contains
• ESMF infrastructure in this release
• ESMF version 1.0 function call examples
  (code!!!)
• (aka API or methods)
• Next steps.
• Infrastructure Utility Example

3
Infrastructure Layer

• A standard software platform for enabling
  interoperability (developing couplers, ensuring
  performance portability).
• Set of reusable software for Earth science
  applications. Streamlined development for
  researchers.

NCAR Atmosphere
NCAR Atmosphere
My Sea Ice
GFDL Ocean
GFDL Ocean
NSIPP Land
NSIPP Land
4
Infrastructure Layer Scope
ESMF Superstructure

• Support for
• Physical Grids
• Regridding
• Decomposition/composition
• Communication
• Calendar and Time
• I/O
• Logging and Profiling

User Code
ESMF Infrastructure
5
Location Within ESMF
1. ESMF provides an environment for assembling
components.
Application Component
Gridded Components
Coupler Components

• 2. ESMF provides a toolkit that components use to
• ensure interoperability
• abstract common services

Component run(), checkpoint()
INFRASTRUCTURE LAYER
Field halo(), import(), export() I/O
Grid regrid(), transpose() Metrics
Layout, PEList, Machine Model
6
Talk Outline

• What is the infrastructure layer
• What is it for
• What it contains
• ESMF infrastructure in this release
• ESMF version 1.0 function call examples
  (code!!!)
• (aka API or methods)
• Next steps.
• Infrastructure utility example

7
Infrastructure Internal Organization
Two tiers
Class hierarchy for data and communications
8
ESMF Infrastructure Fields and Grids
9
Creating an ESMF Field

• Fortran Array
• Array Attach
• Field Attach
• Array

real, dimension(100,100) arr
ESMF_Array
esArr ESMF_ArrayCreate(arr,)
info.
ESMF_Field
metadata
ESMF_FieldAttachArray(esFld,esArr,)
ESMF_Grid
ESMF_Array
info.
10
Creating an ESMF Grid
call ESMF_DistGridCreate()
call ESMF_PhysGridCreate()
DistGrid
PhysGrid
ESMF_Grid
11
Field and Grid Together
grid ESMF_GridCreate(,layo
ut,) field_u
ESMF_FieldCreate(grid, array)
ESMF_Field
metadata
ESMF_Array
ESMF_Grid
info.

• Create field distributed over a set of
  decomposition elements (DEs).
• Domain decomposition determined by DELayout,
  layout.
• Each object (grid and field_u) has internal
  representation.
• Other parts of the infrastructure layer use the
  internal representation e.g.
• Regrid() interpolation/extrapolation
  redistribute over DEs
• Redistribution() - general data rearrangement
  over DEs
• Halo() specialized redistribution

12
Regrid

• Function mapping fields array to a different
  physical and distributed grid.
• RegridCreate() creates a Regrid structure to be
  used/re-used
• regrid ESMF_RegridCreate(src_field,
  dst_field, method, name, rc)
• Source, Dest field can be empty of field data
  (RegridCreate() uses grid metrics)
• PhysGrid, DistGrid info used for setting up
  regrid
• Resulting regrid can be used for other fields
  sharing same Grid
• Method specifies interpolation algorithm. For
  example, bilinear, b-spline, etc

13
Regrid Interface (cont)

• RegridRun() performs actual regridding
• call ESMF_RegridRun(src_field, dst_field,
  regrid,rc)
• Communication and interpolation handled
  transparently.
• RegridDestroy() frees up memory
• call ESMF_RegridDestroy(regrid,rc)

src_field
dst_field
14
Redistribution

• No interpolation or extrapolation
• Maps between distributed grids, fields array and
  physical grid are the same i.e.
• Example layout() created two distributed grids
  one decomposed in X and one decomposed in Y.
• Redistribution() function maps array data between
  the distributions (a transpose/corner turn)
• Communication handled transparently

dst_field
15
Halo

• Fields have distributed index space of
• Exclusive E, compute C and local L region
  where
• Halo() fills points not in E or C from remote
  E e.g

call ESMF_FieldHalo(field_foo, status)
L
L
C
C
E
E
DE4
DE3
16
More functions in reference manual e.g
17
Bundle and Location Stream

• ESMF_Bundle
• collection of fields on the same grid
• ESMF_LocationStream
• Like a field but..
• unstructured index space with an associated
  physical grid space
• useful for observations e.g. radiosonde, floats
• Functions for create(), regrid(),
  redistribute(), halo() etc

18
ESMF Infrastructure Utilities

• Clock
• Alarm
• Calendar
• I/O
• Logging
• Profiling
• Attribute
• Machine model
• and comms

Ensures consistent time between components
Provides field level I/O in standard forms
netCDF, binary, HDF, GRIB, Bufr
Consistent monitoring and messaging
Consistent parameter handling
Hardware and system software hiding. Platform
customizable
19
Time

• Standard type for any component
• Calendar (support for range of calendars)

! initialize stop time to 13May2003, 200 pm call
ESMF_TimeInit(inject_stop_time,
YRint(2003,kindESMF_IKIND_I8),
MMoff_month, DDoff_day, Hoff_hour,
Moff_min,
Sint(0,kindESMF_IKIND_I8),
calgregorianCalendar, rcrc) do while
(currTime .le. inject_stop_time ) call
ESMF_ClockAdvance(localclock, rcrc) call
ESMF_ClockGetCurrTime(localclock, currtime,
rc) end
call ESMF_CalendarInit(gregorianCalendar,
ESMF_CAL_GREGORIAN, rc)
20
I/O

• Field level binary, netCDF, HDF, GRIB, bufr
• Currently I/O piped through 1 PE
• call ESMF_FieldAllGather(field_u, outarray,
  status)
• if (de_id .eq. 0) then
• write(filename, 20) "U_velocity",
  file_no
• call ESMF_ArrayWrite(outarray,
  filenamefilename, rcstatus)
• endif
• call ESMF_ArrayDestroy(outarray, status)
• Current system fixed textual.

21
Attributes

• Flexible parameter specs, configuration e.g. file
  of parameters

Code
22
Internal Classes

• Machine model
• Captures system attributes, CPU, mem,
  connectivity graph
• Useful for defining decomposition, load-balance,
  performance predictions.
• Comms
• Communication driver, allows bindings to MPI,
  shared memory, vendor system libraries

23
Comms Performance Test
Right mix (green) on Compaq gives x2 realized
bandwidth (in large message limit)
24
Talk Outline

• What is the infrastructure layer
• What is it for
• What it contains
• ESMF infrastructure in this release
• ESMF version 1.0 function call examples
  (code!!!)
• (aka API or methods)
• Next steps.
• Infrastructure utility example

25
Regrid Next Steps

• Support for all ESMF Grids
• Support for regridding methods

Bilinear Bicubic 1st-order Conservative 2nd-order
Conservative Rasterized Conservative Nearest-neigh
bor distance-weighted average
Halo Next Steps

• Support for periodicity
• More general haloing
• in tandem with distributed grid evolution
• Adjoint forms

Spectral transforms 1-d interpolations
(splines) Index-space (shifts, stencils) Adjoints
of many above
26
Distributed Grid

• Regular 2d already supported
• Next steps
• Generalized 1d decomposition
• Extend support for 2d and quasi regular
  decomposition
• Spectral grid decompositions

Physical Grid Next Steps

• Larger set of metrics, grids
• High level routines for rapid definition of
  common grids.

27
I/O Next Steps

• Broaden format set, binary, netCDF, HDF, GRIB,
  bufr
• Improve parallelization
• Full support for alarms
• Broader functionality

Time/Logging/Profiling Next Steps
28
Summary

• ESMF Infrastructure Layer
• Comprehensive class structure available in
  version 1.0
• Over the coming year significant extension of
  functionality will take place.
• Feedback and comments on version 1.0 welcome
• http//www.esmf.ucar.edu

29
Time Manager

• ESMF Infrastructure Utility detailed example
• Earl
  Schwab, ESMF Core Team, NCAR

30
What is Time Manager?

• Clock for time simulation
• Time representation
• Time calculator
• Time comparisons
• Time queries
• F90 API, C implementation

31
Clock for time simulation

• type(ESMF_Clock) clock
• call ESMF_ClockInit(clock, timeStep, startTime,
  stopTime, rcrc)
• do while (.not.ESMF_ClockIsStopTime(clock, rc))
• ! Do application work
• .
• .
• .
• call ESMF_ClockAdvance(clock, rcrc)
• end do

32
Clock (cont.)

• Clock queries/commands
• call ESMF_ClockGetCurrTime(clock, currTime, rc)
• call ESMF_ClockSetCurrTime(clock, currTime, rc)
• call ESMF_ClockGetTimeStep(clock, timeStep, rc)
• call ESMF_ClockSetTimeStep(clock, timeStep, rc)
• call ESMF_ClockGetAdvanceCount(clock,
  advanceCount, rc)
• call ESMF_ClockGetStartTime(clock, startTime, rc)
• call ESMF_ClockGetStopTime(clock, stopTime, rc)
• call ESMF_ClockGetPrevTime(clock, prevTime, rc)
• call ESMF_ClockSyncToWallClock(clock, rc)

33
Time Representation

• type(ESMF_Calendar) calendar1
• call ESMF_CalendarInit(calendar1,
  ESMF_CAL_GREGORIAN, rc)
• - ESMF_CAL_GREGORIAN (3/1/-4800 to
  10/29/292,277,019,914)
• - ESMF_CAL_JULIAN (/-
  106,751,991,167,300 days)
• - ESMF_CAL_NOLEAP
• - ESMF_CAL_360DAY

34
Time Representation (cont.)

• type(ESMF_Time) time1
• call ESMF_TimeInit(time1, YRint( 2003
  ,kindESMF_IKIND_I8),
• MM 5 , DD 15 , H 15 ,
  calcalendar1, rcrc)
• - YR, MM, DD, H, M, S F90 optional
• - D, H, M, S
  arguments
• type(ESMF_TimeInterval) timeInterval1
• call ESMF_TimeIntervalInit(timeInterval1,
• Dint( 90 ,kindESMF_IKIND_I8), rcrc)
• - D, H, M, S F90 optional arguments

35
Time Calculator

• Time differencing
• Time increment/decrement by a time interval
• Time interval arithmetic (, -, , /)

36
Time Calculator (cont.)

• call ESMF_TimeInit(time1, YRint( 2003
  ,kindESMF_IKIND_I8),
• MM 5 , DD 15
  , calgregorianCalendar, rcrc)
• call ESMF_TimeInit(time2, YRint( 2003
  ,kindESMF_IKIND_I8),
• MM 3 , DD 26 ,
  calgregorianCalendar, rcrc)
• call ESMF_TimeIntervalInit(timeInterval1,
• Dint( 90 ,kindESMF_IKIND_I8, rcrc)
• timeInterval2 time2 - time1
• time1 time1 timeInterval1 !
  Uses F90 overloaded
• timeInterval3 timeInterval1 2 !
  operators
• double precision ratio
• ratio timeInterval1 / timeInterval2

37
Time Comparisons

• gt, lt, gt, lt, , ltgt F90 overloaded operators
  between any
• 2 times or time intervals
• if (time1 lt time2) then
• end if
• if (timeInterval1 .ge. timeInterval2) then
• end if

38
Time Queries

• call ESMF_TimeGet(time1, YRyr, MMmm, DDdd,
  Hh, Mm, Ss, rcrc)
• call ESMF_TimeIntervalGet(timeInterval1, Dd,
  Hh, Mm, Ss)
• call ESMF_TimeGetDayOfYear(time1, dayOfYear,
  rc) ! double or integer
• call ESMF_TimeGetDayOfMonth(time1, dayOfMonth,
  rc)
• call ESMF_TimeGetDayOfWeek(time1, dayOfWeek, rc)
• call ESMF_TimeGetMidMonth(time1, midMonth, rc)
• call ESMF_TimeGetString(time1, string, rc)
  ! 2003-05-14T122019 (ISO 8601)
• call ESMF_TimeIntervalGetString(timeInterval1,
  string, rc) ! P1DT12H0M0S (ISO)
• call ESMF_TimeGetRealTime(time1, rc)

39
More information

• ESMF Users Guide
• ESMF Reference Manual
• ESMF Requirements Document
• Time Manager F90 API examples source code
• esmf_1_0_0_r/src/Infrastructure/TimeMgr/interface
  
• esmf_1_0_0_r/src/Infrastructure/TimeMgr/examples