Status of HDFEOS, Related Software, and Tools

1
Status of HDF-EOS,Related Software, and Tools

• Abe Taaheri, Raytheon IIS
• Aura DSWG meeting
• October 2008

2
TOOLKIT / HDF-EOS Support

• Maintenance support was funded on EMD Evolution
  through May 2008
• One year extension until May 2009
• Current support (One full-time developer)
• HDF-EOS2 5
• SDP/MTD Toolkit
• HDF-EOS plug-in for HDFView
• HDF-EOS2 to GeoTIFF Converter (HEG)
• Other Toolkit/HDF-EOS related software
• Toolkit/HDF-EOS/HEG helpdesk

3
ECS Support of HDF-EOS

• Both HDF4 and HDF5 based flavors of HDF-EOS are
  part of ECS baseline.
• OSs Solaris (9, 10), Irix6.5 (64, n32),
• Linux (including 64-bit Opteron and Itanium),
  
• Mac OS X (Power PC and Mac Intel),
• Windows XP(MTD Toolkit, HDF-EOS2 5, HEG,
  HDFView)
• (Dropped support for Solaris 8, HP, HP11, Dec,
  IBM in 2.15 release)
• Planned full Cygwin support for late 2008
• Compilers Fortran 77/90 g77/pgf90 , C, C,
  gcc, g,
• Intel Fortran (SGI Altix), gfortran (Mac Intel)

4
Downloadable Software

• SDP/MTD Toolkit (Science Data Processing Tools)
• HDF-EOS2 (HDF4 based)
• HDF-EOS5 (HDF5 based)
• heconvert (converts HDF4 - based
  Grid/Point/Swath structures to HDF-EOS 5
  equivalents)

5
Downloadable Applications

• HEG
• - GeoTIFF conversion
• - Spatial subsetting, Field Subsetting
• - reprojection
• - stitching, sub-sampling and more .
• HDFView Plug-in, Java-based browser (HDF4,
  HDF5, HDF-EOS 2 and 5 access)
• HE5View (HDF5 - based browser)
• EOSView (HDF4 - based browser)
• Dropped support for OS updates on July 2006

6
Software Releases

• Toolkit/HDF-EOS
• Last release March 2008
• TOOLKIT 5.2.15
• HDF-EOS 2.15, HDF- EOS5.1.11
• with HDF4.2r3, HDF5-1.6.7, Szip2.1
• Next release March 2009
• TOOLKIT 5.2.16
• HDF-EOS 2.16, HDF-EOS5.1.12
• with new releases of HDF4, 5 (possibly 1.8.X)

7
Software Releases (Cont.)

• HDF-EOS Plug-in for HDFView
• Last release September 2008
• Version 2.4
• Next release
• For HDFView Version 2.5

8
Software Releases (Cont.)

• HECONVERT (HDF-EOS2 to HDF-EOS5 Converter)
• Last release 2001, 2004
• Version ?
• New release October 2008
• Version 2.0

9
Major Development/Fixes
Last Releases

• TOOLKIT/HDF-EOS
• SZIP Insatallation Default decoder,
• Choice for encoder
• Choice for configuring HDF4 with/without SZIP
• Choice for disabling HDF4s internal netCDF
• Choice for installing Toolkit/HDF-EOS in 32 or
  64-bit modes in 64-bit Linux platforms
• (except for Itanium no m32 flag)

10
Major Development/Fixes (cont.)

• Support for new releases of HDF4/HDF5/SZIP
• (HDF4.2r3, HDF5-1.6.7, SZIP-2.1)
• Installation fix for a problem caused by limits.h
  
• in some Linux platforms
• Auto Configuration capability (like HDF) added
• for both HDF-EOS2 and HDF-EOS5

11
Major Development/Fixes (cont.)

• Support for Mac Intel platform. Drop of support
  for Sun5.8, Dec, HP, HP 11, IBM
• Fixes for Lat/Lon to pixel conversions in DEM
  tools for Linux to get results similar to those
  in Unix platforms
• Added GDij2ll and GDll2ij APIs to HDF-EOS and
  HDF-EOS5
• Modified accuracy check for quaternion in CSC
  tools
• - The sum of the squares of the components of a
  quaternion should be within 1E-6 (previously was
  1E-7)

12
Major Development/Fixes (cont.)

• HDFEOSVersion string clean up in HDF-EOS5
• Added new utilities to dump lat/lon arrays for
  grids in HDF-EOS and HDF-EOS5
• Fixed several other bugs for Grid. Swath and
  Point objects (complete list in release notes)

13
Major Development/Fixes (cont.)

• HDF-EOS Plug-in for HDFView
• Current Support SUN, LINUX, WINDOWS, and
  MAC (Power and Intel)
• Auto installation of plug-in into HDFView
• running self-extracting archives.
• Adding to plug-in the fixes and enhancement
• that went into HDFView2.4
• Display image for selected pixels in the viewed
  tables

14
Major Development/Fixes (cont.)

• Display Lat/Lon for a selected pixel in the
• data field tables
• Full Mac OSX support
• Improved Help Window
• New icons for activating Lat/Lon display for
• Swath Grid
• Format conversion HDF-EOS2 -gt HDF-EOS5

15
Swath, Grid, and Point Objects
16
AMSR-E and OMI Point objects
17
AMSR-E Swath
New Features in Release Version 2.4
18
HE2 To HE5 Conversion

• New Features in Release Version
  2.4

19
Future Plans

• TOOLKIT/HDF-EOS
• Auto Configuration for TOOLKIT
• Full Support for Cygwin platform
• Support for HDF5-1.8
• Validation tool for HDF-EOS5 files
• Performance improvement
• reading HDF format eph/att files in Toolkit
• HDF-EOS5 by redesigning some functions

20
Future Plans (cont.)

• HDF-EOS Plug-in for HDFView
• More functionalities
• - Ability to modify input HDF-EOS2 granules.
• - Ability to cut/paste objects, modify/delete
  attributes.
• - Ability to create new grid/swath
• HEG
• - Support for AMSR-E Level 3 Land (CEA
  projection)
• - New MRT (MODIS Reprojection Tool) release
  updates

21
Availability

• Access to libraries, applications, Documentation
  
• TOOLKIT
• http//newsroom.gsfc.nasa.gov/sdptoolkit/toolkit.h
  tml
• HDFView
• http//newsroom.gsfc.nasa.gov/sdptoolkit/HDFView/
• HDFView_hdfeos_plugin.html
• HEG
• http//newsroom.gsfc.nasa.gov/sdptoolkit/HEG/HEGHo
  me.html
• Email
• Abe_Taaheri_at_raytheon.com
• Landover_PGSTLKIT_at_raytheon.com

22

• Thank You!

23
Tools in detail

• What are
• TOOLKIT, HDF-EOS, HEG and HDFView Plug-in ?

24
SDP Toolkit

• Science Data Production (SDP) Toolkit
• collection of tools used by
• - Science software developers who produce
• code to process instrument data.
• - HDF-EOS users who need to
• Access metadata in HDF-EOS files
• Perform Time/Date conversion
• Access Digital Elevation Model Data
• Access Ancillary Data, such as Digital Chart
• of the World database (DCW), Olson World
  Ecosystem files, DEM data files, etc.

25
SDP Toolkit (cont.)

• Tools provided in (SDP, MTD) Toolkit
• - AA (Ancillary Data Access)
• - CBP (CELESTIAL Body Position)
• - CSC (Coordinate System Conversion)
• - CUC (Constant and Unit Conversions)
• - DEM (Digital Elevation Model Access)
• - EPH (Ephemeris Data Access)
• - GCT (Geo Coordinate Transformation)
• - IO (Input Output (File I/O))
• - MEM (Memory Management)
• - MET (Metadata Access)
• - PC (Process Control)
• - SMF (Status Message File (Error/Status))
• - TD (Time and Date Conversion)

26
What is HDF-EOS ?

• HDF-EOS is an extension to HDF which standardizes
  storage of and access to common Earth Sciences
  data structures
• By providing a single interface to data
• structures common to earth science
• By providing a container for EOS inventory,
  archive and product specific metadata
• By establishing a consistent relationship
  between geolocation and science data.

27
HDF-EOS 2

• HDF4 - based, storage format for EOS standard
  products.
• Used operationally by MODIS, MISR, ASTER,
  Landsat, AIRS and other EOS instruments
• Support for Grid/Point/Swath structures
• HDF4 and HDF-EOS 2 will be supported by THG/NASA
  for the fore-seeable future.

28
HDF-EOS 5

• Based on HDF5, a complete rewrite of HDF-EOS2
  with a different user interface.
• First released in 2000.
• Format used by EOS Aura instruments
• Designed to resemble HDF-EOS 2 to the maximum
  extent possible.
• Supports same data structures
• Added prefix HE5_ to HDF-EOS 2 functions.

29
HDFView

• HDFView A Java-based browser from THG for HDF4
  and HDF5 based data.
• HDF-EOS plug-in extends HDFView to browse HDF-EOS
  data in both versions.
• The extension is a plug-in module and does not
  change THG HDFView functionality. All copy, save,
  editing, image display, etc. for HDF4 and 5
  objects remains.
• Identical look and feel when displaying HDF and
  HDF-EOS objects.
• Users can access a complete breakdown of HDF-EOS
  objects. e.g. grid information, projection info.,
  dimensions, datafields, and attributes are
  displayed. 
• Current Support SUN, SGI, LINUX, Mac, WINDOWS
  platforms

30
Swath, Grid, and Point Objects
31
AMSR-E and OMI Point objects
32
AMSR-E Swath
New Features in Release Version 2.4
33
HE2 To HE5 Conversion

• New Features in Release Version
  2.4

34
HDF-EOS to GeoTIFF Converter (HEG)

• HEG Utility that converts EOSDIS data from
• HDF-EOS format to common Geographical
• Information System (GIS)-compatible formats.
  
• Versions Available
• Stand-alone A downloadable desktop version
• Data Pool Access through NASA archive online
  storage
• Implemented also in prototypes for Stand-alone
  Data Pool, OGC (Open Geospatial Consortium) for
  Web Services Chaining, and DOWS (Deploy OGC Web
  Services) on the DAACs

35
HEG Functionality

• Conversion
• HDF-EOS (Swath or Grid ) to a single-band or
  multi-band GeoTIFFs
• HDF-EOS Swath to HDF-EOS Grid or Swath
• HDF-EOS Swath or Grid to generic Binary (with
  metadata file)
• Subsetting (spatial, field, band)
• Stitching (mosaicing) subsetting Reprojecting
• Reprojection ( to UTM, PS, TM, STP, LCC, LAMAZ,
  GEO, SIN, Albers)
• Subsampling (Subsample stacks)
• Running from Command Line (batch jobs)
• or with Java-based GUI

36
HEG Functionality

• Metadata preservation/creation.
• Currently supports MODIS, MISR, ASTER, AIRS and
  AMSR-E products on TERRA and AQUA (gt115
  products),
• Allows file selection from users local storage
• Operable on Sun, SGI, Win, Linux, MAC
• Integrated into ECS Data Pool
• Reduces the transfer time of HDF-EOS data sets
  (if subset is requested)
• Provides the end-user with the exact file
  required by their application
• User Interface
• Portable, written in Java.
• Not dependent on COTS (eg. IDL).

37
Mt. Etna Eruption MISR L1B Stitched Blocks
38
2004 Hurricane Francis, MODIS
MOD021KM, Multi-Band GeoTIFF, 3 bands
EV_1KM_RefSB, Bands 1,2,5
39
AST_L1B, 2003 California FiresGeo. projection,
Multi-Band GeoTIFF, (3 bands Visible Near
Infrared (VNIR))
.. and Subsetted
Two Swaths Stitched ..
40
Major Development/Fixes (cont.)

• HEG Release
• Spatial subsetting
• - AE-DyOcn, AE_DySno
• New Product AMSR-E and ASTER (On-Demand L2)
• AMSR-E 5 day Snow
• AMSR-E Monthly Snow
• AMSR-E Weekly Ocean
• AMSR-E Monthly Ocean
• AMSR-E Daily 12 Km Sea Ice
• AMSR-E Daily 6 Km Sea Ice
• AMSR-E L2 Rain, AMSR-E Rain Grid
• - ASTR_07XT, ASTR_09XT

41
Major Development/Fixes (cont.)

• Stitching
• MOD14A1 3-D fields (band selection)
• TM grids
• Multi-band geotif/HDF-EOS in Datapool
• Other Fixes
• Interpolation correction with AIRS products
• containing fillvalue in the data fields
• Pixel size unit consistency in conversion and
  stitch GUIs.

42
Major Development/Fixes (cont.)

• Correction for droping a column when
• stitching MODIS SIN tiles.
• Correction to ignoring of Geographic pixel size
• and processing with default pixel size
• Spatial subsetting fix for global AMSR-E
  garanules
• Correction for ignoring re-projection
  interpolation type NN after stitching