THREDDS Data Server

1
THREDDS Data Server

• John Caron
• Unidata/UCAR
• Mar 14, 2005

2
Outline

1. THREDDS Catalog Services
2. THREDDS Data Server
3. NetCDF-4
4. NetCDF-Java 2.2

3
THREDDS

• Thematic Realtime Environmental Distributed Data
  Services
• Middleware between data providers and data users
• Enable discovery and use of scientific data
• Not a data portal
• XML over HTTP
• Java Servlets

4
THREDDS Catalogs
HTTP Server
Catalog.xml
Data Server(s) DODS, ADDE, FTP, HTTP
Client Application
Datasets
hostname.edu
5
Dynamic Catalogs, data services
HTTP Tomcat Server
Catalog.xml
Client Application
Data Server(s) DODS, ADDE, FTP, HTTP
Datasets
hostname.edu
6
Dataset Query Capability (DQC)

• Dynamic and very large datasets
• NEXRAD (level 3) from IDD
• 150 stations
• 20 products
• Irregular times
• The DQC describes the queries that the data
  server is capable of responding to.

7
Resolver Services

• Logical Dataset, eg latest ETA model run
• Dataset with Service type Resolver
• On the client, the URI of the logical dataset is
  sent to the data server
• The server finds what is available and returns an
  actual dataset URL.

8
Limitations

• Catalogs are very general data protocol
  agnostic.
• Automatic metadata extraction needs to crack
  the datasets.
• Data Servers must be separately installed,
  maintained, etc

9
THREDDS Data Server

• Add the data access!
• 100 Java single thredds.war
• requirement, not goal
• Data input is NetCDF Java 2.2 library
• Data output
• OpenDAP
• WCS (in progress)
• FTP, others (future)

10
THREDDS Data Server
HTTP Tomcat Server
Catalog.xml

• THREDDS Server
• OPeNDAP
• WCS

Application
NJ22 library
hostname.edu
Datasets
IDD Data
11
THREDDS OpenDAP Server

• Catalogs instead of dods_dir
• Next version of C library will also have
  THREDDS catalogs
• Catalogs automatically generated
• Based on DODS-Java library and the Aggregation
  Server.
• Aggregation probably done at nj22 level

12
THREDDS WCS Server

• WCS 1.0 specification
• Proposing to add NetCDF as blessed binary file
  return type
• OGC interoperability experiment
• Difficult to find clients to work with
• Working with CADCorp client
• Liping Di (George Mason)
• NASA GSFC Earth Science Gateway
• Looking for others

13
THREDDS WCS Server

• Return formats
• Geotiff floating point
• NetCDF binary
• No reprojections for now
• Must have Grid coordinate system
• Issues
• Vertical coordinates
• Time coordinates
• Large number of coverages (granularity)

14
NetCDF-4 NetCDF-Java 2.2
15
NetCDF-4

• Project funded by NASA to create new version of
  netCDF using the HDF5 file format.
• Extend and merge netCDF and HDF5
• Widespread use and simplicity of netCDF-3
• Generality and performance of HDF5
• Specifically, we are funded to create netCDF-4 C
  library API, using HDF5 library underneath.
• Russ Rew (PI), Ed Hartnett (Unidata), Mike Folk
  (co-PI), Quincey Koziol (NCSA)

16
NetCDF-4 C Library
NetCDF-4 Architecture
17
NetCDF-4 and Java

• 100 Java library for netCDF-4 files possible?
• NetCDF-Java 2.1 already a 100 Java library for
  netCDF-3 files (and OPeNDAP)
• NetCDF-Java 2.2 read HDF5 to determine what
  netCDF-4 data model should be
• Common Data Model emerged for netCDF, HDF5 and
  OpenDAP.
• OpenDAP 4.0 protocol being worked on
• The 3 groups are actively discussing

18
NetCDF-Java 2.2 (nj22)

• 100 Java library for netCDF, HDF5, OpenDAP, and
  more
• THREDDS integrated
• Under development
• Alpha release Nov 2004
• Beta release May 2005
• Release summer 2005

19
Application
Scientific Datatypes
NetCDF-Java version 2.2 architecture
Trajectory
Image
Grid
Station
Radial
NetcdfDataset
NetcdfFile
ADDE
OpenDAP
HDF5
I/O service provider
NetCDF-3
NetCDF-4
GRIB
NIDS
GINI
Nexrad

DSMP
20
I/O Service Provider

• Implement this interface
• public interface IOServiceProvider
• boolean isValidFile( RandomAccessFile raf)
• void open( RandomAccessFile raf, NetcdfFile
  ncfile)
• Array readData( Variable v2, List section)
• // only if you use Structures
• Array readNestedData( Variable v2, List
  section)

21
Goal N M instead of N M things on your TODO
List
File Format 1
Visualization Analysis
NetCDF file
File Format 2
OpenDAP Server
File Format N
WCS Service
22
Common Data Model (CDM)
23
Coordinate Systems

• Same underlying mathematics as VisAD, ASCII

24
Scientific Data Types

• Point / Station (metars)
• Unconnected
• DQCs used with IDD station/radar data
• Trajectory (floating buoys)
• 1D connected
• Radial (NEXRAD)
• elevation ,azimuth, distance
• Grid (Model output)
• cartesian x, y, z
• WCS

25
THREDDS Data Server
HTTP Tomcat Server
Catalog.xml

• THREDDS Server
• OPeNDAP
• WCS

Application
NJ22 UI library
NJ22 library
NJ22 library
hostname.edu
Datasets
IDD Data
26
Summary

• THREDDS data server combines Catalog Services and
  data servers
• As nj22 matures, data server will have more
  functionality
• OpenDAP, WCS
• NetCDF-4 simplicity of netCDF and flexibility of
  HDF5

27
(No Transcript)