HDF OPeNDAP Project Update - PowerPoint PPT Presentation

About This Presentation
Title:

HDF OPeNDAP Project Update

Description:

HDF OPeNDAP Project Update – PowerPoint PPT presentation

Number of Views:20
Avg rating:3.0/5.0
Slides: 33
Provided by: peter1063
Learn more at: http://www.hdfeos.org
Category:
Tags: hdf | opendap | jow | project | update

less

Transcript and Presenter's Notes

Title: HDF OPeNDAP Project Update


1
HDF OPeNDAP Project Update
  • MuQun Yang and Hyo-Kyung Lee
  • The HDF Group
  • March 31, 2009

2
Outline
  • HDF5 OPeNDAP work review
  • HDF4 OPeNDAP handler progress report
  • Future work plan

3
Access data via OPeNDAP
Users
Hyrax
IDV
libnc-dap
HDF HANDLER
AIRS
DAP
Visualization Tools
OPeNDAP Clients
OPeNDAP Servers
HDF4 Files
4
HDF5 OPeNDAP Work
5
HDF5-OPeNDAP handler
  • Update with new Hyrax
  • Help GES DISC serve TES and OMI data
  • Finish HDF5 DAP2 mapping document and submit it
    as a technical note to NASA SPG

6
  • HDF5-Friendly OPeNDAP Client Library
  • Retrieve the key HDF5 information
  • Easy to access other HDF5 data (e.g. EOS5 swath)

7
Traditional OPeNDAP client library Its an
attribute that I dont understand. Ill ignore it.
Example Groups in HDF5
HDF5-Friendly OPeNDAP client library I was
waiting for this key attribute to re-construct
HDF5
8
MLS swath via GrADS
CO level near Beijing, China before, during and
after Olympics
lower stratosphere
CO at these elevations is a measure of
pollution
Before
During
After
upper stratosphere
9
HDF4 OPeNDAP Work
10
Current HDF4-OPeNDAP handler
  • Contributed by JPL in 1998
  • Part of the OPeNDAP distribution
  • Visualization client tools cannot display most
    HDF-EOS2 and HDF4 data

11
Our work
  • Enhance the HDF4-OPeNDAP handler
  • More NASA HDF-EOS2 and HDF4 data can be
    accessed via visualization clients

12
Challenges more products
Hundreds of HDF-EOS2 and HDF4 Data products
13
Challenges Requirement from Clients
  • CF conventions
  • - Clients expect a certain data structure
  • - Clients expect some key attributes

14
Challenges Grid
  • Various HDF-EOS2 Grid projections
  • Geographic
  • Sinusoidal
  • Polar stereographic

Data 412 Lon12
Data 48 Lon48
Geographic
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . .
Sinusoidal
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . .
15
Challenges - Swath
Data field has three columns
Longitude field has two columns
  • HDF-EOS2 Swath
  • - Dimension map

Longitude should have three columns
16
Challenges various products
  • Special HDF-EOS2 files
  • Hybrid HDF-EOS2 files
  • Some HDF4 objects that cannot be read by EOS
    library
  • HDF4 data products

17
Enhanced HDF4 Handler
  • Follow CF conventions
  • Generate the correct DAP data type.
  • Add shared geo-location information.
  • Insert additional key attributes.

  • How Parse Metadata or Use EOS2 library

18
Implementation choice
  • Use Metadata parser or HDF-EOS2 library
  • Metadata parser
  • Pro simple
  • Con limited products
  • HDF-EOS2 library
  • Pro maximum products
  • Con install HDF-EOS2 library first

19
Solution Comparison
20
Quick Implementation
  • Parse Metadata hdf5_handler
  • Use HDF-EOS2 library h4toh5

IDL
ncBrowse
Ferret
MATLAB
NCL
IDV
GrADS
ODC
21
Current Status
  • MODIS / AIRS Grid and Swath
  • Other Projections (Polar, Sinusoidal)
  • Testing

IDL
ncBrowse
Ferret
MATLAB
NCL
IDV
GrADS
ODC
22
HDF-EOS2 in CF via DAP
AIRS Grid AIRS.2008.10.27.L3.RetStd001.v5.2.2.0.G
08303124144.hdf http//daac.gsfc.nasa.gov/daac-bin
/whom/mk_page_cgi.pl?PATHdatapool/AIRS/Level3_V00
5/01_AIRX3STD/2008/10
AIRS Swath ftp//airspar1u.ecs.nasa.gov/data/s4pa
//Aqua_AIRS_Level2/AIRX2RET.005//2008/330/AIRS.200
8.11.25.143.L2.RetStd.v5.2.2.0.G08331121849.hdf
MODIS Grid MOD08_D3.A2008284.005.2008289105354.hdf
http//modis-atmos.gsfc.nasa.gov/MOD08_D3/acquiri
ng.html
MODIS Swath MOD06_L2.A2001222.0025.004.20031271118
41.hdf http//modis-atmos.gsfc.nasa.gov/MOD06_L2/a
cquiring.html
23
More challenges Missing Attributes
  • MODIS Grid Rain fall
  • AURA OMI Grid - Ozone

No Unit!
24
More challenges Match dimension names
  • Horizontal geolocation information
  • - Non-standard names ( e.g. nlon, nlat)
  • How to match?
  • Vertical geolocation information
  • - Users provide the geolocation name and vaule
    but no conventions are enforced.
  • How to know?

25
More challenges Missing Values
  • In a swath file, a data field has three
    dimensions. But no dimension value for the third
    dimension.

MISR_AM1_TC_CLASSIFIERS_P020_O000995_F06_0011.hdf
26
A Solution to challenges
  • Provide an option for the handler to generate
    fully CF-compliant output
  • Create a separate XML input file for the handler
    and feed it into the handler
  • Insert attributes, renaming rules, and missing
    values by putting them in the XML input
  • This requires a knowledge about the
    characteristics of data in advance

27
One Solution An input file from data provider
  • Example Attributes, Renaming, Values

ltxmlgt ltvariable nameRrLandRaingt ltattr
nameunit typeStringgt incheslt/attrgt lt/variabl
egt ltrename frompressStd tolevelgtlt/renamegt lti
nsert nameNAltitudeDim size10
typeInt8gt 0,1,2,3,4,5,6,7,8,9 lt/insertgt lt/xmlgt
28
Future Work (Short Term)
  • Expand test-suite with data pool
  • Release through OPeNDAP website
  • Support for Swath via Parsing
  • Support for Point data type via HDF- EOS2 library

29
Future Work (Long Term)
  • General HDF4 Files Support
  • Using an input XML file to provide full
  • CF support
  • HDF-EOS2 to DAP2 Mapping Document

30
Thank You!
31
Acknowledgements
  • This work was supported by the Cooperative
    Agreement with the National Aeronautics and Space
    Administration (NASA) under NASA grant
    NNX06AC83A, NNX08A077A and NNX06AG75A.
  • Any opinions, findings, conclusions or
    recommendations expressed in this material are
    those of the author(s) and do not necessarily
    reflect the views of NASA.

32
Questions/comments?
Write a Comment
User Comments (0)
About PowerShow.com