Mike Folk

1
HDF Update

• Mike Folk
• HDF-EOS Workshop IV
• Sept. 19-21, 2000

2
Topics

• What is HDF?
• HDF community
• HDF4
• past and future work
• HDF5
• HDF5-HDF5 interoperability
• Activities and growth
• Tools and utilities
• Focus for 2001

3
NCSA HDF Mission
To develop, promote, deploy, and support open and
free technologies that facilitate scientific data
storage, exchange, access, analysis and
discovery.
4
What is HDF?

• Format and software for scientific data
• Stores images, arrays, tables, etc.
• Emphasis on storage and I/O efficiency
• Free and commercial software support
• Emphasis on standards
• Users from many engineering scientific fields

5
HDF People

• NCSA
• QA, maintanance support
• Library development performance
• Tools development
• EOS ASCI
• Users, applications and tools
• Performance
• Volunteers from all over
• Users, applications tools

6
Who is supporting HDF?

• NASA/ESDIS
• Earth science applications, instrument data
• All aspects of data management
• DOE/ASCI (Accelerated Strategic Computing Init.)
• Simulations on massively parallel machines
• Emphasis on parallel I/O performance,
  functionality
• NCSA
• Grid, Vis, other RD, user support
• Others
• Applications, support, some RD

7
HDF4

• User support, tools, maintenance, QA
• JHV (Java viewer) - two update releases
• HDF4 dumper (hdp) - many improvements
• Next HDF4 Release (Oct. 2000)
• Bug fixes
• Chunking for General Raster (GR) images
• Unlimited dimensions for SDS
• GIF lt--gt HDF4 conversion tools

8
HDF4 Archiving

• New File Format and Specification Manual
• Facilitate long term maintenance of HDF4
• Necessary for bug fixing, adding new features
• Decreases learning curve for new developers
• Available soon in
• PDF, HTML and PS formats

9
HDF5
10
The growth of HDF5

• Users in 27 countries
• Users include
• Government agencies
• National labs
• Companies
• Universities

11
The growth of HDF5

• Scientific fields
• Astronomy, astrophysics, aerospace engineering
• Geophysics, remote sensing, meteorology,
• Oceanography, environmental Science
• Informational Science
• Medical Research (brain, cancer, biotech)
• Product model data

12
Facilitating interoperability between HDF4 and
HDF5

• HDF4 to HDF5 mapping specification
• Mapping HDF4 Objects to HDF5 Objects
• http//hdf.ncsa.uiuc.edu/HDF5/papers
• Rules for mapping high level HDF4 objects to HDF5
• How to describe HDF4 objects in HDF5
• How to interpret HDF5 objects as HDF4 objects
• HDF4-to-HDF5 conversion software (later)
• Space Research, Inc. Explorer 1.1
• reads both HDF4 and HDF5

13
HDF5 Activities in 2000

• HDF5 1.2.2 library release
• Fortran 90 C API
• HDF5 Abstract Data Model
• XML Document Type Definition (DTD)
• HDF5 tools
• Support for users and application developers
• HDF5 Tutorial
• Searchable, printable documentation

14
Structure of HDF5 Library

• Object API (C, Fortran 90, Java, C)
• Specify objects and transformation properties
• Invoke data movement operations and data
  transformations

• Library internals
• Performs data transformations and other prep for
  I/O
• Configurable transformations (compression, etc.)

• Virtual file I/O (C only)
• Perform byte-stream I/O operations (open/close,
  read/write, seek)
• User-implementable (I/O to stdio, network,
  memory, etc.)

15
Tools Utilities

• NCSA
• Java wrapper for HDF5
• H5View Java browser/Editor for HDF5
• H5gen XML-to-HDF5 file generator (Java)
• H5dump H5ls
• Others
• VisAD data adapter for HDF5 (Java toolkit)
• HDF Inspector/Explorer
• Open Data Explorer (IBM)
• Ensight

16
H5View
17
H5View

• Java-based tool for browsing and editing
• Display structure of file
• Display content of objects
• Create and delete objects
• Modify values of objects and attributes

18
HDF5 XML DTD

• A flexible standard language that can describe an
  HDF5 file in precise detail
• Datasets, dataspaces, datatypes
• Groups and links, structure of the file
• Values of the attributes and data
• http//hdf.ncsa.uiuc.edu/HDF5/XML

19
H5Gen XML?HDF5

• Reads an XML description of an HDF5 file
• Generates the corresponding HDF5 file
• Validates XML description vs. HDF5 DTD

20
Some uses of XML
View HDF5 files using a web browser
Archive
Data location services
XML to HDF5 (H5gen) HDF5 to XMLGenerate,
validate, reconstruct HDF5 files
Java viewers,editors,other tools
21
Focus areas for 2001

• Support Terra Aqua
• Get ready for Aura
• Support HDF-EOS 3
• Enhance HDF5View
• Refine XML DTD design tools around it
• HDF5 converters
• HDF5?XML
• HDF4-to-HDF5
• Others (e.g. GIF?HDF5)

22
Focus areas for 2001

• Expand list of applications and users
• Facilitate access to other tools software
• Get vendors on board
• Performance testing and tuning
• Extend API with ease-of-use functions
• Clusters and other new environments
• Implement a thread-safe version
• HDF5 advanced tutorial

23
Thank you!

• HDF website
• http//hdf.ncsa.uiuc.edu/
• HDF5 Information Center
• http//hdf.ncsa.uiuc.edu/HDF5/
• HDF Helpdesk
• hdfhelp_at_ncsa.uiuc.edu
• HDF users mailing list
• hdfnews_at_ncsa.uiuc.edu

5