Title: Integrated data management in the ESMF ESME
1Integrated data management in the ESMF (ESME)
- Steve Hankin(NOAA/PMEL IOOS/DMAC)
- ESMF Team meeting
- July 2004
2The growing importance of data integration to
modelers
- We no longer fund modeling.
- Today we fund climate prediction or coastal
processes the science topics. Modeling is
just a component. - a program manager (anonymous)
3Model outputs need to be made useful to many
classes of users
education
research community
modeling community
project
model run
4Data and products need to be made more usable for
modelers
Real time and delayed-mode observations
assimilation
data products
modeling
boundary initial conditions
validation
comparison
5An ESME must include a plan for data management.
- But, how ?
- Funds are limited
6Partnership A community of data managers has
formed
- GO-ESSP Global Organization of Earth System
Science Portals(http//esportal.gfdl.noaa.gov) - Unidata
- ESG (NCAR, LLNL)
- OPeNDAP (a.k.a. DODS)
- COLA
- NOMADS (GFDL, PMEL, NCDC, NCEP)
- NASA/GCMD
- BADC, BODC
- WMO
7Ocean data systems following similar approaches
- National Virtual Ocean Data System (NVODS)
- US Integrated Ocean Observing System
- GODAE (US and International)
- OCMIP, AOMIP,
8Data portal components
- Data discovery
- Data access/transport
- On-line browse and comparison
- (Segue to analysis)
9Data discovery Metadata search
- ? Mature standards do not exist today.
- A task for ESMF define and utilize metadata
standards for modelers
10Metadata for modelers
- Reviewed published (a standard)
- Structured (XML)
- Generated automatically in conjunction with
setting up model runs - Standardized parameter names (controlled
vocabularies) - Hierarchical
- components, grids, fields and attributes
11Data discovery Metadata search
- Others are working hard on search
- Traditional metadata partners(e.g. GCMD)
- Semantic Web (Google on steroids)(3 years off?)
12Data portal components
- Data discovery
- Data access/transport
- On-line browse and comparison
- (Segue to analysis)
13Data transport
- OPeNDAP (a.k.a. DODS)
- Network data access
- Format-independence
- Subsetting
- Aggregation (GDS, Unidata)
- Compression
- Security Grid-enabled OPeNDAPg
14OPeNDAP distributed access to data and semantic
metadata
15CF (climate and forecast)
- CF 1.0 is now a standard
- use metadata e.g. units, coords.
- curvilinear, hybrid-Z, time-dependent
- great applicability beyond modeling, too
- Discussion question
- As the use of the CF standard widens how should
the community support it?Not enough to endorse
it. Need a partnership.
16Data portal components
- Data discovery
- Data access/transport
- On-line browse and comparison
- (Segue to analysis)
17Live Access Server
18netCDF
19LAS -- an Information Product Server
Ferret, CDAT or other
- Metadata (XML) contains the intelligence
- Back end applications do the real work
- OPeNDAP provides remote data access
20(No Transcript)
21Informationaccess
Uniform data access
22A home page
Informationaccess
Uniform data access
Live Access Server
23The UI talks to LAS through an XML web service
24For example
25On-line comparison
26(No Transcript)
27ExampleAverage over a lat-long box
28Data portal components
- Data discovery
- Data access/transport
- On-line browse and comparison
- (Segue to analysis)
29(No Transcript)
30discover ? browse ? access
Metadata Standards
NASA Global Change Master Directory (GCMD)
31Collaborating groups of modelers
LAS sisters share metadata to form a unified
(virtual) site.OPeNDAP allows LAS to difference
distributed fields.
32v6.2 customizableinterfaces
33Web Service access to products
- XML request protocol implemented and documented
- XML package out implemented and documented
- XML query protocol under development
- formal SOAP interface under development
34A wealth of data products are available through
the National Virtual Ocean Data System (NVODS)
35Example Zebiac model outputlive from IRI/LDEO
36(No Transcript)
37(No Transcript)
38(No Transcript)
39Access to observations (WODB -- 9 million ocean
profiles)
40(No Transcript)
41(No Transcript)
42configurable constraints
43(No Transcript)
44Batch access to products
- Query available data sets
- Query variables in data set model_1
- Query space-time domain
- Request a subset of data as a file (asc for
ASCII format)
gtlasls http//cpu/LAS
gtlasls http//cpu/LAS model_1
gtlasls http//cpu/LAS model_1 sst
gtlasget -x 2060 -y 2060 -t 11-Dec-2000 -f
asc http//cpu/LAS model_1 sst
45(No Transcript)
46(No Transcript)
47(No Transcript)
48IOOSUS Integrated Ocean Observing System
- Detect and Predict Change
- Mitigate natural hazards
- Improve safety and efficiency of marine ops
- Ensure national security
- Reduce public health risks
- Protect and restore marine ecosystems
- Sustain marine resources
49IOOS Data Management andCommunications Subsystem
Ships
Hand Measurements
Satellites
Floats
Primary DataAssembly QC
Moorings
Metadata, Data Discoveryand Data
TransportStandards and Protocols
50IOOS Data Management andCommunications Subsystem
Ships
Hand Measurements
Satellites
Floats
Primary DataAssembly QC
Moorings
RegionalData ManagementSystems
Products
Users
InternationalData ManagementSystems
Maps
Forecasts
Terrestrial and AtmosphericData
ManagementSystems
Metadata, Data Discoveryand Data
TransportStandards and Protocols
On-line Browse
Archive Centers
Modeling
51Recommendations for ESMF
- Define ESMF metadata standard and use it
- Consider a GCMD modelers portal, too
- Endorse partnership with GO-ESSP and the emerging
tools - Data available through OPeNDAP(g)
- Live Access Servers for on-line collaborations
52Questions?
NVODS LAShttp//www.ferret.noaa.gov/nvods GCMD
DODS Portalhttp//gcmd.gsfc.nasa.gov/Data/port
als/dods