THREDDS Data Server Unidatas Common Data Model Background Summary

1
THREDDS Data ServerUnidatas Common Data
ModelBackground / Summary

• John Caron
• Unidata/UCAR
• Mar 2007

2
THREDDS Data Server
HTTP Tomcat Server
catalog.xml
Application
THREDDS Server

• WCS

• OPeNDAP

• HTTPServer

• NetcdfSubset

NetCDF-Java library
motherlode.ucar.edu
Datasets
IDD Data
3
THREDDS Catalogs

• XML over HTTP
• Hierarchical listing of online resources
  (datasets)
• Container for arbitrary search metadata
• Standard set maps to DC, GCMD, ADN
• Unidata/CDP
• Metadata can be inherited
• Design goal Make it easy for data providers
• TDS uses for configuration
• Client view vs. server view
• Data Access URLS
• Crossing the protocol boundary

4
catalog.xml
5
Motherlode catalog example
6
THREDDS WCS 1.0 Server

• Each (gridded) Dataset is WCS
• Each Grid is a Coverage
• Return formats
• GeoTIFF floating point, greyscale
• NetCDF / CF-1.0 (same as NetcdfSubset Service)
• No reprojections, resampling
• GALEON 2
• upgrade to WCS 1.1
• Try returning point datasets

7
(No Transcript)
8
THREDDS OPeNDAP Server

• Current version 2.0 NASA ESE standard
• Working on new 4.0 protocol spec
• Based on Java-OPeNDAP library
• shared development by Unidata/opendap.org
• Any CDM dataset can be served
• Server4 (Hyrax)
• latest version of opendap.org C library
• uses THREDDS catalog generation code
• THREDDS Catalogs replace dods_dir

9
Common Data Model
HTTP Tomcat Server
catalog.xml
Application
THREDDS Server

• WCS

• OPeNDAP

Then a miracle happens

• HTTPServer

• NetcdfSubset

NetCDF-Java library
hostname.edu
Datasets
IDD Data
10
Application
Scientific Datatypes
Datatype Adapter
NetCDF-Java version 2.2 architecture
NetcdfDataset
CoordSystem Builder
ADDE
NetcdfFile
I/O service provider
OPeNDAP
NetCDF-3
NIDS
GRIB
NetCDF-4
NcML
HDF5
GINI
Nexrad
DMSP

11
I/O Service Provider Implementations

• General NetCDF, HDF5, OPeNDAP
• Gridded GRIB-1, GRIB-2
• Radar NEXRAD level 2 and 3, DORADE, Chinese
  NEXRAD
• Point BUFR, ASCII
• Satellite DMSP, GINI, McIDAS AREA
• In development / tentative
• NOAA CLASS legacy files
• Barrowdale DataBlade

12
Common Data Model Layers
Coordinate Systems
Data Access
13
NetCDF-4 and Common Data Model (Data Access Layer)
14
NetCDF-4 C library

• 4.0 Beta implements CDM access layer
• complete, but waiting for HDF5 release 1.8 to
  finalize file format (Maybe this month, 1.5 years
  late!)
• Persistence format for complete CDM
• 4.1 adding Coordinate Systems
• Optional layer, focus on CF-1 (libcf)
• 4.? merge OPeNDAP access (pending funding)

15
Coordinate Systems UML
16
NcML NetCDF Markup Language

• XML representation of netCDF metadata
• Core netCDF data access model
• Coordinate System general and georeferencing
  coordinate system
• Dataset redefine, aggregate, subset
• Luca Cinquini (NCAR/SCD/ESG), John Caron, Ethan
  Davis, Bob Drach (LLNL), Stefano Nativi
  (Florence), Russ Rew

17
NcML

• NcML Coordinate Systems further developed into
  NcML-G by Stefano et al.
• NcML Core and Dataset combined into single schema
  to allow dataset modification
• Aggregation
• Union
• Syntactic join on (existing or new) outer
  dimension
• Semantic aggregation of (runtime, forecast time)
  Forecast Model Run Collection

18
NcML example

• lt?xml version"1.0" encoding"UTF-8"?gt
• ltnetcdf xmlns"http//www.unidata.ucar.edu/schemas
  /netcdf/ncml-2.2"
• location/data/nids/N0R_20041119_2147"gt
• ltattribute namecdm_datatype" valueRadial"
  /gt
• ltremove typeattribute namepassword" /gt
• ltvariable name"Reflectivity" orgNameR34768gt
• ltattribute name"units" valuedBZ" /gt
• lt/variablegt
• lt/netcdfgt

19
TDS / NcML example

• ltdatasetScan name"Ocean Satellite Data"
  path"ocean/sat" dirLocation"R/tds/netcdf/"gt
• ltnetcdfgt
• ltattribute name"Conventions"
  value"CF-1.0"/gt
• lt/netcdfgt
• lt/datasetScangt

20
TDS / NcML aggregation

• ltdataset name"WEST-CONUS_4km Aggregation"
  urlPath"satellite/3.9/WEST-CONUS_4km"gt
• ltnetcdf gt
• ltaggregation dimName"time" type"joinNew"gt
• ltscan location"/data/ldm/pub/satellite/3.9/
  WEST-CONUS_4km/" suffix".gini" /gt
• lt/aggregationgt
• lt/netcdfgt
• lt/datasetgt

21
Datasets vs. Files

• Must hide actual location of data files on your
  server
• Would like to hide actual file format
• Must encapsulate collections of files into
  logical datasets
• Homogenous metadata
• Hide arbitrary storage decisions
• Minimize number of datasets

22
Forecast Model Run Collection (FMRC)
23
Data Model Sampled Functions

• Our phenomena are continuous functions
  F Domain ? Range
• where
• Domain subset of space-time (3 spatial,
  time) (?4)
• Range Rn (product set of real numbers)
• Our measurements are sampled functions
• Domain is a point subset p, p ? ?4
• M E4 ? Rn

24
Variables

• Variable is a container for an Array of values
• dimensions
• lat 64
• lon 128
• variables
• float temperature( lat, lon)
• Domain is a set of points in Index space
• Temperature 0..63 x 0..127 ? R
• Temperature I2 ? R
• Variable Im ? Rn

25
Coordinate Systems

• Coordinate Axis Im ? R
• Axis Coordinate System Im ? E4
• V Im ? Rn
• CS Im ? E4
• V CS-1 E4 ? Rn

26
Scientific Data Types

• Trying to go beyond index-space subsetting
• Trying to satisfy V CS-1 E4 ? Rn
• I.e. support subsetting using Space, Time
  queries
• Based on datasets Unidata is familiar with
• APIs are evolving
• Intended to scale to large, multifile collections
• Corresponding standard NetCDF file format
  conventions

27
Implementations

• Datatype
• Grid
• PointObs
• RadialSweep
• Swath

• Dataset
• GridDataset
• FMRCDataset
• CollectionOfPointObs
• StationCollectionOfPointObs
• StationCollectionOfRadialSweep

28
Conclusions

• CDM is our implementation data model
• Map to data access models such as OGC
• Current work is to serve collections instead of
  individual files.
• Dataset is desired level of granularity
• Scientific data types are implementations with
  specialized access