THREDDS Data Server TDS and Data Discovery

1
THREDDS Data Server (TDS)and Data Discovery

• John Caron
• Unidata/UCAR
• May 15, 2006

2
THREDDS Data Server
OAI Harvester
HTTP Tomcat Server
OAI Provider
DL Records
Catalog.xml
THREDDS Server
Application
NetCDF-Java (CDM) library

• OPeNDAP

• HTTPServer

• WCS

Datasets
otherhost.gov
OPeNDAP Server
hostname.edu
3
Collection vs Inventory Datasets
Catalog
/models/ncep/NAM/ File1.grib File2.grib File3.grib
DatasetScan
Dataset
Dataset
Dataset
Dataset
Dataset
Dataset
Dataset
Dataset
Dataset
Dataset
http//motherlode.ucar.edu8080/thredds/dodsC /mod
el/NCEP/DGEX/CONUS_12km/file.grib2
4
DL Harvesting
Catalog
/models/ncep/NAM/ File1.grib File2.grib File3.grib
DatasetScan
Dataset
Dataset
Dataset
Dataset
Metadata Record
Dataset
Dataset
Metadata Record isHarvest true inherit
true
Dataset
Dataset
5
Metadata Information

• Title / Summary
• Publisher / Creator / Rights
• Lat/Lon bounding box
• Time range
• Relative time latest 7 days
• Variable names
• DLESE no (not dataset oriented)
• GCMD controlled list, required
• Unique ID/ Resource URL

6
(No Transcript)
7
(No Transcript)
8
Why not harvest Inventory?

• Too many of them, eg in IDD
• NCEP models 28 collections, 6000 files
• NEXRAD level 3 files 8M files
• Real-time datasets are never current
• DLs (GCMD, DLESE) dont want them
• Collection search in DL, browse inventory on
  server.

9
(No Transcript)
10
(No Transcript)
11
Current Work Aggregation

• Make many files into single logical dataset Make
  Collection Dataset Inventory
• Uses NcML to read into CDM, works at the
  syntactic level.
• Replaces older Aggregation Server
• Union
• Join on existing dimension
• Join on new dimension

12
TDS/NcML Aggregation

• ltdataset name"WEST-CONUS_4km Aggregation"
  urlPath"satellite/3.9/WEST-CONUS_4km"gt
• ltnetcdf xmlnshttp//www.unidata.ucar.edu/schem
  as/netcdf/ncml-2.2gt
• ltaggregation dimName"time"
  type"joinExisting"gt
• ltscan locationC/data/goes/"
  suffix".gini"/gt
• lt/aggregationgt
• lt/netcdfgt
• lt/datasetgt

13
Next DataType Aggregation

• Work at the CDM DataType level, know (some) data
  semantics
• Forecast Model Collection
• Combine multiple model forecasts into single
  dataset with two time dimensions
• With NOAA/IOOS (Steve Hankin)
• Point/Station/Trajectory/Profile Data
• Allow space/time queries, return nested sequences
• Start from / standardize Dapper conventions

14
Forecast Model Collections
15
Web services for discovery

• Latest dataset Resolver service
• Dataset Query Capability (DQC) accept query,
  return results as a collection of datasets in a
  catalog
• Future Dynamic dataset creation based on user
  query ??

16
Summary

• Expect discovery to be 2 phased
• Search for collections in DL with browser
• Use an application like the IDV (OPeNDAP) or GIS
  client (WCS) to drill down to the actual data.
• Expect aggregation / query will (eventually) tame
  the inventory problem

17
Dataset Query Capability Document

• XML document that describes the set of valid
  queries for a dataset. Queries are URLS
• http//www/dqc/radar?stnABRproductNORtime1hou
  r
• Selectors
• List of choices
• List of stations
• Numeric range (point or subrange)
• DateRange
• Latitude/Longitude Bounding Box
• Orthogonal selections (except Lists can be
  nested)
• Returns a catalog containing inventory datasets.

18
Example DQC

• ltqueryCapabilitygt
• ltquery base"http//www/dqc/radar"/gt
• ltselectList id"prod" titleParametersgt
• ltchoice namereflect value"N0R"gt
• ltdescriptiongt.5u reflectivitylt/descriptiongt
• lt/choicegt
• ltchoice name"velocity" value"N0S"gt
• ltdescriptiongt.5u storm rel. velocity
  lt/descriptiongt
• ltselectList id"time" title"Timesgt
• ltchoice name"Latest value"latest"/gt
• ltchoice name"LastHour value"1hour"/gt
• lt/selectListgt
• lt/choicegt
• lt/selectListgt

• ltselectStation id"station" title"Stations" gt
• ltstation name"AK" value"ABC"gt
• ltlocation latitude"60" longitude"161"/gt
• lt/stationgt
• ltstation name"SD" value"ABR
• ltlocation latitude"45" longitude"-98.4"/gt
• lt/stationgt
• lt/selectStationgt
• ltselectFromDateRange id"datePnt" title"Date
  selectType"point"start"2004-04-01T0000"
• end"2004-04-15T1200" /gt
• lt/queryCapability gt

19
Issues

• DQC itself doesnt deal with the query
• http//www/dqc/radar?stnABRproductNORtime1hou
  r
• Queries are expressible as paramvalue
• Extend to arbitrary URLs (token substitution), eg
  dods
• SOAP RPC?
• Returns a catalog, might be the data itself.
• Prototype/non-standard, need buy-in from clients
  to bother continuing.