DAP NETCDF Using the netCDF-4 Data Model

1
DAPNETCDFUsing the netCDF-4 Data Model
Dr. Dennis Heimbigner Unidata netCDF
Workshop August 3-4, 2009
2
Overview

• RECAP netCDF-DAP Integration
• RECAP Architecture
• DAP Elements
• Example DDS and DAS
• DAP to netCDF-4 Translation
• netCDF-3 versus netCDF-4 Translation
• Extending the URL Format
• Constraints Projections
• Constraints Selections
• Figuring Out Constraints
• Client Parameters Revisited
• Building and Installing netCDF-4 DAP
• Debugging Notes

3
RECAP NETCDF-DAP Integration

• Unidata has integrated the DAP protocol into the
  standard netCDF distribution.
• It replaces the OPeNDAP provided libnc-dap
  library
• It closely mimics the original libnc-dap
  translation
• Implemented completely in C, versus libnc-dap,
  which is C
• This talk is primarily about the support for the
  netCDF-4 data model

4
RECAP Architecture / Flow
5
DAP Elements

• DDS - describes the format of the data
• DAS - describes associated attributes
• DATADDS - the actual data aka DODS
• DATADDS contains two things
• A subset of the DDS followed by
• The actual data

6
Web Debugging

• If you are knowledgeable about DAP, you can look
  at the underlying DDS, DAS, DATADDS, and more,
  using a web browser
• Procedure
• Remove the client parameters
• Add to the base url one of the following
  extensions
• .dds, .das, .asc (or .ascii), .html, .info

7
Example DDS and DAS
Dataset Sequence String
Drifter_ID String Date_Sampled
Float64 Latitude Float64 Longitude
Float64 SST ... Drifters EOS.DBO
Attributes FacilityGlobal String
DataCenter "COAS" String DrifterType
"MetOcean" Drifter_ID String
Description Instrument unique
identifier Date_Sampled
String Description "Date/time" String
Timezone "GMT" ...
8
DAP to NETCDF-4 Translation

• The client parameter modenetcdf4 will cause
  the DAP DDS and data to be translated to the
  netCDF-4 data model
• netCDF-4 model is more powerful than DAP, so the
  translation only uses part of netCDF-4
• Versus the netCDF-3 (classic) translation
• Lossless
• All DAP primitive types handled (no upgrading)
• DAP sequences are translated to combinations of
  netCDF-4 compound types and vlen types
• Performance improvement
• No need for string or sequence dimensions

9
NETCDF-3 vs netCDF-4 translation
netcdf Drifters types compound
Drifters_record_t string Drifter_ID
double Latitude double Longitude
double SST Drifters_record_t()
Drifters_t variables Drifters_t
Drifters
netcdf Drifters dimensions Drifters 5 //
Sequence //
dimension stringdim64 64 // String
// dimension variables char
Drifters.Drifter_ID(Drifters,
stringdim64) double
Drifters.Latitude(Drifters) double
Drifters.Longitude(Drifters) double
Drifters.SST(Drifters)


10
Extending the DAP URL Format

• ltclientparamsgtltbaseurlgt?ltconstraintgt
• Additional client parameters
• Constraint tells the OPeNDAP server what subset
  of the dataset to send
• Constraint is divided into two parts
• Projections
• Selections
• Currently disabled

Data Specified By Full DDS
Data Specified By Constraints
11
Client Parameters Revisited

• Explicitly specify modenetcdf4 in order to get
  the netcdf-4 translation
• Or specify NC_NETCDF4 mode to nc_open()
• Additional show cases
• showdds (attribute _dds)
• showdas (attribute _das)
• showdatadds (attribute _datadds)
• String attribute defining the DDS, DAS, or DDS
  part of the DATADS, respectively, as obtained
  from the OPeNDAP server

12
Constraints Projections

• Projections specify array slices of underlying
  OPeNDAP variables to send
• http//test.opendap.org/dods/dts?windW0102
• Format ?ltprojectiongt,ltprojectiongt,
• Projection is DAP variable name followed by a
  slice specification firststridelast
• Note last, not count

13
Constraints Selections

• Selection constraints tell the OPeNDAP server to
  subset parts of the dataset based on the values
  in the dataset
• Can only be applied to DAP Sequences
• http//test.opendap.org/dap/data/CS02.lightgt0
• Format ltselectiongtltselectiongt
• Multiple selections are treated as logical and
• Selections always follow the projections, if any
  in the URL

14
Figuring out Constraints

• How does one know what constraints to apply?
• The problem is that they are defined in terms of
  the underlying DAP data model
• gt Breaks the netCDF abstraction
• For projections, the showprojection parameter
  can help by showing the underlying DAP variables
• For Selections, you need to know something about
  the values in the original dataset
• Look at the .info file?

15
Build and Install NETCDF4 DAP

• Add --enable-dap and --enable-netcdf-4 to your
  list of ./configure flags
• Specify the curl library (libcurl) if ./configure
  cannot find it
  --with-curlltpath to curl
  librarygt
• As usual, of course, specify --with-hdf5 and
  --with-zlib

16
Example OPeNDAP Data Sources

• http//test.opendap.org8080/dods/dts
• http//test.opendap.org
• http//test.opendap.org/dap/data/ff
• http//www.opendap.org/data/datasets.cgi
  ?xmlfilenamedatasets.xmlexfunctionnone

17
Questions?
18
Demo Sites

• http//test.opendap.org8080/dods/dts/Drifters
• http//test.opendap.org
• http//test.opendap.org/dap/data/ff
• http//www.usgodae.org/dods/GDS/coamps_cent_am
• http//www.opendap.org/data/datasets.cgi?xmlfilena
  medatasets.xmlexfunctionnone