Title: HDF OPeNDAP Project Update
1HDF OPeNDAP Project Update
- MuQun Yang and Hyo-Kyung Lee
- The HDF Group
- March 31, 2009
-
2Outline
- HDF5 OPeNDAP work review
- HDF4 OPeNDAP handler progress report
- Future work plan
3Access data via OPeNDAP
Users
Hyrax
IDV
libnc-dap
HDF HANDLER
AIRS
DAP
Visualization Tools
OPeNDAP Clients
OPeNDAP Servers
HDF4 Files
4HDF5 OPeNDAP Work
5HDF5-OPeNDAP handler
- Update with new Hyrax
- Help GES DISC serve TES and OMI data
- Finish HDF5 DAP2 mapping document and submit it
as a technical note to NASA SPG
6- HDF5-Friendly OPeNDAP Client Library
- Retrieve the key HDF5 information
- Easy to access other HDF5 data (e.g. EOS5 swath)
7Traditional OPeNDAP client library Its an
attribute that I dont understand. Ill ignore it.
Example Groups in HDF5
HDF5-Friendly OPeNDAP client library I was
waiting for this key attribute to re-construct
HDF5
8MLS swath via GrADS
CO level near Beijing, China before, during and
after Olympics
lower stratosphere
CO at these elevations is a measure of
pollution
Before
During
After
upper stratosphere
9HDF4 OPeNDAP Work
10Current HDF4-OPeNDAP handler
- Contributed by JPL in 1998
- Part of the OPeNDAP distribution
- Visualization client tools cannot display most
HDF-EOS2 and HDF4 data
11Our work
- Enhance the HDF4-OPeNDAP handler
- More NASA HDF-EOS2 and HDF4 data can be
accessed via visualization clients
12Challenges more products
Hundreds of HDF-EOS2 and HDF4 Data products
13Challenges Requirement from Clients
- CF conventions
- - Clients expect a certain data structure
- - Clients expect some key attributes
14Challenges Grid
- Various HDF-EOS2 Grid projections
- Geographic
- Sinusoidal
- Polar stereographic
Data 412 Lon12
Data 48 Lon48
Geographic
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . .
Sinusoidal
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . .
15Challenges - Swath
Data field has three columns
Longitude field has two columns
- HDF-EOS2 Swath
- - Dimension map
Longitude should have three columns
16Challenges various products
- Special HDF-EOS2 files
- Hybrid HDF-EOS2 files
- Some HDF4 objects that cannot be read by EOS
library - HDF4 data products
17Enhanced HDF4 Handler
- Follow CF conventions
- Generate the correct DAP data type.
- Add shared geo-location information.
- Insert additional key attributes.
- How Parse Metadata or Use EOS2 library
18Implementation choice
- Use Metadata parser or HDF-EOS2 library
- Metadata parser
- Pro simple
- Con limited products
- HDF-EOS2 library
- Pro maximum products
- Con install HDF-EOS2 library first
19Solution Comparison
20 Quick Implementation
- Parse Metadata hdf5_handler
- Use HDF-EOS2 library h4toh5
IDL
ncBrowse
Ferret
MATLAB
NCL
IDV
GrADS
ODC
21 Current Status
- MODIS / AIRS Grid and Swath
- Other Projections (Polar, Sinusoidal)
- Testing
IDL
ncBrowse
Ferret
MATLAB
NCL
IDV
GrADS
ODC
22 HDF-EOS2 in CF via DAP
AIRS Grid AIRS.2008.10.27.L3.RetStd001.v5.2.2.0.G
08303124144.hdf http//daac.gsfc.nasa.gov/daac-bin
/whom/mk_page_cgi.pl?PATHdatapool/AIRS/Level3_V00
5/01_AIRX3STD/2008/10
AIRS Swath ftp//airspar1u.ecs.nasa.gov/data/s4pa
//Aqua_AIRS_Level2/AIRX2RET.005//2008/330/AIRS.200
8.11.25.143.L2.RetStd.v5.2.2.0.G08331121849.hdf
MODIS Grid MOD08_D3.A2008284.005.2008289105354.hdf
http//modis-atmos.gsfc.nasa.gov/MOD08_D3/acquiri
ng.html
MODIS Swath MOD06_L2.A2001222.0025.004.20031271118
41.hdf http//modis-atmos.gsfc.nasa.gov/MOD06_L2/a
cquiring.html
23More challenges Missing Attributes
No Unit!
24 More challenges Match dimension names
- Horizontal geolocation information
- - Non-standard names ( e.g. nlon, nlat)
- How to match?
- Vertical geolocation information
- - Users provide the geolocation name and vaule
but no conventions are enforced. - How to know?
25More challenges Missing Values
- In a swath file, a data field has three
dimensions. But no dimension value for the third
dimension.
MISR_AM1_TC_CLASSIFIERS_P020_O000995_F06_0011.hdf
26A Solution to challenges
- Provide an option for the handler to generate
fully CF-compliant output - Create a separate XML input file for the handler
and feed it into the handler - Insert attributes, renaming rules, and missing
values by putting them in the XML input - This requires a knowledge about the
characteristics of data in advance
27One Solution An input file from data provider
- Example Attributes, Renaming, Values
ltxmlgt ltvariable nameRrLandRaingt ltattr
nameunit typeStringgt incheslt/attrgt lt/variabl
egt ltrename frompressStd tolevelgtlt/renamegt lti
nsert nameNAltitudeDim size10
typeInt8gt 0,1,2,3,4,5,6,7,8,9 lt/insertgt lt/xmlgt
28Future Work (Short Term)
- Expand test-suite with data pool
- Release through OPeNDAP website
- Support for Swath via Parsing
- Support for Point data type via HDF- EOS2 library
29Future Work (Long Term)
- General HDF4 Files Support
- Using an input XML file to provide full
- CF support
- HDF-EOS2 to DAP2 Mapping Document
30Thank You!
31Acknowledgements
- This work was supported by the Cooperative
Agreement with the National Aeronautics and Space
Administration (NASA) under NASA grant
NNX06AC83A, NNX08A077A and NNX06AG75A. - Any opinions, findings, conclusions or
recommendations expressed in this material are
those of the author(s) and do not necessarily
reflect the views of NASA.
32Questions/comments?