EARTH SCIENCE MARKUP LANGUAGE Tutorial

1
EARTH SCIENCE MARKUP LANGUAGETutorial

• Define Once Use Anywhere
• INFORMATION TECHNOLOGY AND SYSTEMS CENTER
• UNIVERSITY OF ALABAMA IN HUNTSVILLE

2
Tutorial Overview

• Introduction to ESML
• Data/Application Interoperability Problem
• Interchange Technology Solution via ESML
• Writing ESML Description Files using the Editor
• ASCII Format
• Binary
• HDF-EOS
• Writing ESML reader using Python

3
ESML Vision
4
Earth Science Data Characteristics
HDF
HDF-EOS

• Different formats, types and structures (18 and
  counting for Atmospheric Science alone!)
• Different states of processing ( raw,
  calibrated, derived, modeled or interpreted )
• Enormous volumes
• Heterogeneity leads to Data usability problem

netCDF
ASCII
Binary
GRIB
5
Interoperability Accessing Heterogeneous Data
The ESML Solution
DATA FORMAT 1
DATA FORMAT 2
DATA FORMAT 3
ESML FILE
ESML FILE
ESML FILE
ESML LIBRARY
APPLICATION

• One approach Standard data formats
• Difficult to implement and enforce
• Cant anticipate all needs
• Some data cant be modeled or is lost in
  translation
• The cost of converting legacy data
• A better approach Interchange Technologies
• Earth Science Markup Language

6
What is ESML?

• It is a specialized markup language for Earth
  Science metadata based on XML - NOT another data
  format.
• It is a machine-readable and -interpretable
  representation of the structure of any data file,
  regardless of data format
• ESML description files contain external metadata
  that can be generated by either data producer or
  data consumer (at collection, data set, and/or
  granule level)
• ESML provides the benefits of a standard,
  self-describing data format (like HDF, HDF-EOS,
  netCDF, geoTIFF, ) without the cost of data
  conversion
• ESML is the basis for core Interchange Technology
  that allows data/application interoperability
• ESML complements and extends data catalogs such
  as FGDC and GCMD by providing the use/access
  information those directories lack.

7
ESML Components
ESML Schema defines syntactic metadata that
describe the structure of the file in
machine-readable and interpretable terms and are
the grammar that are used to generate the ESML
description file
ESML Description Files specify the structure of
the data file format in terms of bits and bytes
ESML Library is the middleware that applications
use to parse an ESML Description File and
retrieve data
ltaESML gt ltSyntacticMetaDatagt ltAsciigt
ltStructure instances"1"gt ltHeader
name"SizeX" format"d" symboltrue" /gt
ltHeader name"SizeY" format"d symboltrue"
/gt ltArray occursSizeX"gt
ltArray occursSizeY"gt ltField
name"BrightnessTemp" format"d"/gt
lt/Arraygt lt/Arraygt lt/Structuregt
lt/Asciigt lt/SyntacticMetaDatagt lt/aESMLgt
ESML MARKUP FOR THE DATA FILE
4 5 1 2 3 4 5 6 7 8 9 10 11
12 13 14 15 16 17 18 19 20
SIMPLE ASCII DATA FILE
8
Components of the ESML Interchange Technology
External description file for dataset or formats
DATAFORMAT1
DATAFORMAT2
DATAFORMAT3
Rules that govern the description of the data
files
OTHER FORMATS
ESMLEDITOR
ESMLFILE
ESMLFILE
ESMLFILE
ESMLSCHEMA
ESML LIBRARY
ESMLDATA BROWSER
ADaM DATA MINING SYSTEM
OTHER APPLICATIONS
Library parses and interprets the description
file and figures out how to read the data
9
Components of the ESML Interchange Technology
DATAFORMAT1
DATAFORMAT2
DATAFORMAT3
OTHER FORMATS
ESMLEDITOR
ESMLFILE
ESMLFILE
ESMLFILE
ESMLSCHEMA
ESMLFILE
ESMLSCHEMA
These three key components allow applications to
use data in a wide variety of formats
ESML LIBRARY
ESMLDATA BROWSER
ADaM DATA MINING SYSTEM
OTHER APPLICATIONS
INTERCHANGE TECHNOLOGY
10
Interchange Technology for Data Users and
Application Developers
DATAFORMAT1
DATAFORMAT2
DATAFORMAT3
OTHER FORMATS
DATA PRODUCERS OR CONSUMERS
ESMLEDITOR
ESMLFILE
ESMLFILE
ESMLFILE
ESMLSCHEMA
ESMLFILE
ESMLSCHEMA
ESML LIBRARY
ESML can be used by both scientists and
application developers
ESMLDATA BROWSER
ADaM DATA MINING SYSTEM
OTHER APPLICATIONS
INTERCHANGE TECHNOLOGY
APPLICATION DEVELOPERS
11
Advantages of using ESML

• Scientist (Data Producer/Consumer)
• ESML will let them use virtually any data format
  in their applications
• ESML files are external description files that
  can be easily created, modified and viewed by any
  text editor
• ESML has a few simple concepts which can be used
  to describe numerous data sets
• An ESML file can be seen as a set of instructions
  to the application on how to read and understand
  a data file
• If the format of the data changes for whatever
  reason (e.g., new version of data set) no
  software changes are required, just a new ESML
  file.
• Does that mean a scientist has to write an ESML
  file for every data file?
• No, in fact the beauty of ESML is that it allows
  scientist to write ONE ESML file to describe MANY
  data files that are structurally and semantically
  similar

12
Advantages of using ESML

• Data Archiving Centers (Data Producers)
• ESML files can be used to store not only the
  structural but also embed semantic information
  about the data sets
• Since ESML files are independent separate files,
  they can be generated on the fly utilizing
  metadata databases as datasets are ordered
• Centers can archive data in its native formats
  and not have to store them in any selected
  format
• Centers can now also ESMLize all their legacy
  datasets with minimal efforts
• The existing legacy datasets now become a more
  valuable data resource for scientists, because
  they can be used more efficiently and effectively
• Application Developers
• By using the ESML library, developers can build
  ESML enabled applications!
• ONE single reader component can read all the
  various data formats instead of having separate
  reader module for different formats

13
ESML IN ACTIONCollocation Algorithm
MISR/ Others
ESML file
ESML file
ESML file
MODIS
CERES

• Scientists can
• Select a variety of data in different formats for
  the collocation analysis

• Purpose
• To study the relationship between shortwave flux
  and cloud/aerosol properties
• Important for climate change studies

ESML Library
Collocation Algorithm
Analysis
14
ESML IN ACTION Ingest surface skin temperature
data in Numerical Models
MM5
GOES
Reanalysis GRIB files

• Skin temperatures come in a variety of data
  formats -
• GOES - McIDAS
• Reanalysis Data - GRIB
• MM5 Model - MM5 Binary
• AVHRR - HDF
• MODIS - EOS-HDF

ESML FILE
ESML FILE
ESML FILE
ESML LIBRARY
APPLICATION
15
ESML-Enabled Generic Subsetter
Other Formats
Binary/ ASCII
ESML file
ESML file
ESML file
HDF-EOS
Network
ESML Library
Subsetting Algorithm
For HDF-EOS data not formatted for subsetting
with the HDF-EOS library ESML file can be used
to correct the semantic tag required to subset
HDF-EOS data without the need to recreate the
data file
Subsetted Data
16
Prototype Smart Subsetter

• To demonstrate a smart application using ESML and
  ontologies, a subsetting prototype is being
  developed
• Subsetting is a frequent preprocessing step used
  by scientists to reduce the size and complexity
  of the data
• The subsetting prototype parses the semantic tags
  embedded in the ESML Description File
• The subsetting prototype then uses the linked
  ontologies to decipher meaning of these tags to
  make useful decisions
• Components of this Prototype
• Simple ontologies describing Subsetting and
  Dataset
• ESML Description Files
• Reasoning System from JTP from Stanford
  University used as an inference engine

Dataset Ontology
Subsetting Ontology
17
Smart Applications/Services using ESML and
Ontologies

• ESML Schemas focus is on providing structural
  data interoperability between data/application
• However, ESML allows embedding semantic terms for
  data fields in the Description File to provide a
  complete structural and semantic description of
  the data
• Various science communities can create their own
  ontologies (for example, SWEET) and link them
  with ESML Description Files for their data
• Application developers can add semantic parsers
  on top of the core ESML Library to build smart
  applications or services

Structural Information
ESMLSchema
ESMLSCHEMA
ESMLSCHEMA
Ontologies
Ontologies
Semantic Information
18
ESML Schema

• ESML schema defines Syntactic metadata that
  describe the structure of the file in
  machine-readable and -interpretable terms

19
Writing an ESML File (1)
ltaESML gt ltSyntacticMetaDatagt ltAsciigt
ltStructure instances"1"gt ltHeader
name"SizeX" format"d" /gt ltHeader
name"SizeY" format"d" /gt ltArray
occurs4"gt ltArray occurs5"gt
ltField name"BrightnessTemp" format"d"/gt
lt/Arraygt lt/Arraygt
lt/Structuregt lt/Asciigt lt/SyntacticMetaDatagt lt
/aESMLgt
ESML MARKUP FOR THE DATA FILE
The next slides will set describe how to write
an ESML file for a simple ASCII file described
below
4 5 1 2 3 4 5 6 7 8 9 10 11
12 13 14 15 16 17 18 19 20
SIMPLE ASCII DATA FILE
20
Writing an ESML File (2)
ltaESML gt ltSyntacticMetaDatagt ltAsciigt
ltStructure instances"1"gt ltHeader
name"SizeX" format"d" /gt ltHeader
name"SizeY" format"d" /gt ltArray
occurs4"gt ltArray occurs5"gt
ltField name"BrightnessTemp" format"d"/gt
lt/Arraygt lt/Arraygt
lt/Structuregt lt/Asciigt lt/SyntacticMetaDatagt lt
/aESMLgt
DESCRIBING ONLY THE STRUCTURE
4 5 1 2 3 4 5 6 7 8 9 10 11
12 13 14 15 16 17 18 19 20
SIMPLE ASCII DATA FILE
21
Writing an ESML File (3)
ltaESML gt ltSyntacticMetaDatagt ltAsciigt
ltStructure instances"1"gt ltHeader
name"SizeX" format"d" /gt ltHeader
name"SizeY" format"d" /gt ltArray
occurs4"gt ltArray occurs5"gt
ltField name"BrightnessTemp" format"d"/gt
lt/Arraygt lt/Arraygt
lt/Structuregt lt/Asciigt lt/SyntacticMetaDatagt lt
/aESMLgt
DESCRIBE THE FORMAT
4 5 1 2 3 4 5 6 7 8 9 10 11
12 13 14 15 16 17 18 19 20
SIMPLE ASCII DATA FILE
22
Writing an ESML File (4)
ltaESML gt ltSyntacticMetaDatagt ltAsciigt
ltStructure instances"1"gt ltHeader
name"SizeX" format"d" /gt ltHeader
name"SizeY" format"d" /gt ltArray
occurs4"gt ltArray occurs5"gt
ltField name"BrightnessTemp" format"d"/gt
lt/Arraygt lt/Arraygt
lt/Structuregt lt/Asciigt lt/SyntacticMetaDatagt lt
/aESMLgt
ENTIRE FILE CONTENTS INTO 1 LOGICAL STRUCTURE
4 5 1 2 3 4 5 6 7 8 9 10 11
12 13 14 15 16 17 18 19 20
SIMPLE ASCII DATA FILE
23
Writing an ESML File (5)
ltaESML gt ltSyntacticMetaDatagt ltAsciigt
ltStructure instances"1"gt ltHeader
name"SizeX" format"d" /gt ltHeader
name"SizeY" format"d" /gt ltArray
occurs4"gt ltArray occurs5"gt
ltField name"BrightnessTemp" format"d"/gt
lt/Arraygt lt/Arraygt
lt/Structuregt lt/Asciigt lt/SyntacticMetaDatagt lt
/aESMLgt
DEFINE THE FIRST FIELD IN THE FILE HEADER
INFORMATION
4 5 1 2 3 4 5 6 7 8 9 10 11
12 13 14 15 16 17 18 19 20
SIMPLE ASCII DATA FILE
24
Writing an ESML File (6)
ltaESML gt ltSyntacticMetaDatagt ltAsciigt
ltStructure instances"1"gt ltHeader
name"SizeX" format"d" /gt ltHeader
name"SizeY" format"d" /gt ltArray
occurs4"gt ltArray occurs5"gt
ltField name"BrightnessTemp" format"d"/gt
lt/Arraygt lt/Arraygt
lt/Structuregt lt/Asciigt lt/SyntacticMetaDatagt lt
/aESMLgt
DEFINE THE SECOND FIELD IN THE FILE HEADER
INFORMATION
4 5 1 2 3 4 5 6 7 8 9 10 11
12 13 14 15 16 17 18 19 20
SIMPLE ASCII DATA FILE
25
Writing an ESML File (7)
ltaESML gt ltSyntacticMetaDatagt ltAsciigt
ltStructure instances"1"gt ltHeader
name"SizeX" format"d" /gt ltHeader
name"SizeY" format"d" /gt ltArray
occurs4"gt ltArray occurs5"gt
ltField name"BrightnessTemp" format"d"/gt
lt/Arraygt lt/Arraygt
lt/Structuregt lt/Asciigt lt/SyntacticMetaDatagt lt
/aESMLgt
DEFINE THE DATA FIELD IN THE FILE PROVIDE
SIZE AND FORMAT INFORMATION
4 5 1 2 3 4 5 6 7 8 9 10 11
12 13 14 15 16 17 18 19 20
SIMPLE ASCII DATA FILE
26
Writing an ESML File (8)
ltaESML gt ltSyntacticMetaDatagt ltAsciigt
ltStructure instances"1"gt ltHeader
name"SizeX" format"d" /gt ltHeader
name"SizeY" format"d" /gt ltArray
occurs4"gt ltArray occurs5"gt
ltField name"BrightnessTemp" format"d"/gt
lt/Arraygt lt/Arraygt
lt/Structuregt lt/Asciigt lt/SyntacticMetaDatagt lt
/aESMLgt
CLOSE ALL THE TAGS ESML FILE IS READY
4 5 1 2 3 4 5 6 7 8 9 10 11
12 13 14 15 16 17 18 19 20
SIMPLE ASCII DATA FILE
27
Another Possible ESML Description (9)
ltaESML gt ltSyntacticMetaDatagt ltAsciigt
ltStructure instances"1"gt ltHeader
name"SizeX" format"d" symboltrue" /gt
ltHeader name"SizeY" format"d symboltrue"
/gt ltArray occursSizeX"gt
ltArray occursSizeY"gt ltField
name"BrightnessTemp" format"d"/gt
lt/Arraygt lt/Arraygt lt/Structuregt
lt/Asciigt lt/SyntacticMetaDatagt lt/aESMLgt
USE HEADER INFORMATION
4 5 1 2 3 4 5 6 7 8 9 10 11
12 13 14 15 16 17 18 19 20
SIMPLE ASCII DATA FILE
28
Using The ESML Editor

• Writing an ESML file using the ESML Editor
• Simple ASCII file
• Simple Binary file
• HDF-EOS file
• Utilizing the ESML descriptions to browse the
  data files using the ESML Data Browser
• Overview of the ESML library
• Writing a simple reader
• Writing a generic reader

29
ESML Editor

• ESML Schema Window
• Displays the ESML Schema in tree structure
• ESML File Window
• Displays the user generated ESML description
• ESML Attribute Window
• Provides the means to set the attributes for the
  selected ESML element

30
ESML Editor

• 100 Java version prototype
• Utilizes Expert System principles to give users
  correct options
• Hides the XML tags from the users
• Future Features
• Allow text editing of the XML tags also
• Incorporate feedback from users

31
Writing an ESML File Simple ASCII File (1)

• ASCII data file
• 4
• 5
• 1 2 3 4 5
• 6 7 8 9 10
• 11 12 13 14 15
• 16 17 18 19 20

32
Writing an ESML File Simple ASCII File (2)

• Understand the structure of the data file to be
  described by ESML file
• Open ESML editor (ready to create new ESML file)
  and describe this structure using the rules
  established in the ESML schema
• Add SyntacticMetadata as child of ESML and
  modify its attributes if available
• Add Ascii as child of SyntacticMetadata and
  modify its attributes if available

33
Writing an ESML File Simple ASCII File (3)

• Add AsciiStructure as child of Ascii and
  modify its attribute
• Add Field element as one child of
  AsciiStructure and modify its attributes if
  available. This element is used to describe the
  first datum in the data file
• Add semantic information to describe this
  Field
• Add Field element as one child of
  AsciiStructure and modify its attributes if
  available. This element is used to describe the
  second datum in the data file
• Add Array element as one child of
  AsciiStructure to describe the outer dimension
  of the 2D array, and modify its attributes if
  available

34
Writing an ESML File Simple ASCII File (4)

• Add Array element as child of Array to
  describe the inner dimension of the 2D array, and
  modify its attributes if available
• Add Field element as child of Array to
  describe the individual element and modify its
  attributes if available

35
Writing an ESML File Simple Binary File (1)

• The structure and content are the same as those
  of ASCII file that we just worked on, but stored
  in Binary format
• Open ESML editor
• Add SyntacticMetadata as child of ESML and
  modify its attributes if available
• Add Binary as child of SyntacticMetadata and
  modify its attributes if available

36
Writing an ESML File Simple Binary File (2)

• Add BinaryStructure as child of Binary and
  modify its attribute
• Add Field element as one child of
  BinaryStructure and modify its attributes if
  available. This element is used to describe the
  first datum in the data file
• Add Field element as one child of
  BinaryStructure and modify its attributes if
  available. This element is used to describe the
  second datum in the data file
• Add Array element as one child of
  BinaryStructure to describe the outer dimension
  of the 2D array, and modify its attributes if
  available

37
Writing an ESML File Simple Binary File (3)

• Add Array element as child of Array to
  describe the inner dimension of the 2D array, and
  modify its attributes if available
• Add Field element as child of Array to
  describe the individual element and modify its
  attributes if available

38
Writing an ESML File HDF-EOS File (1)

• Since HDF-EOS is self-describing format and
  metadata is contained in the data file, ESML file
  for HDF-EOS data file is simple
• Here are the steps to create ESML file for
  HDF-EOS
• Open ESML editor
• Add SyntacticMetadata as child of ESML and
  modify its attributes if available and needed
• Add HdfEos as child of SyntacticMetadata
• You are done!!!

39
Writing an ESML File HDF-EOS File (2)

• HDF-EOS format has rules that Latitude and
  Longitude fields have to have field names of
  Latitude and Longitude, respectively
• If a HDF-EOS file does not follow this rule, ESML
  file has to be more detailed
• Here are the follow-on from the ESML file before
• Add HdfEosStructure as child of HdfEos and
  modify its attributes if available and needed

40
Writing an ESML File HDF-EOS File (3)

• Add Field as child of HdfEosStructure
• Add semantic information as child to describe
  this Field element. If this field is latitude,
  then add a child element Latitude to indicate
  that this field is latitude field. If longitude
  field, add Longitude child element to describe
  the Field element.

41
ESML Library

• Design follows the layered cake approach where
• Lowest (core) level provides the basic
  functionality of reading the structural metadata
  from the ESML file and returning data to the user
• Additional software layers can be added to
  provide other functionalities such as using
  semantics from an ontology to use the data
  intelligently
• Includes plug-in modules for each individual
  format, allowing packaging of libraries
• Provides a simple API for easy addition of new
  formats as plug-in modules
• Provides a more intuitive user API based on the
  analogy of file access in a directory structure
• Provides the library source code via Source Forge
  repository

42
ESML v3.0 Library Layered Design

• The core ESML library provides the basic
  functionality of reading structural metadata from
  the ESML file and returning data to the user
• Intuitive user API based on the analogy of file
  access in a directory structure
• Plug-in modules for each individual format allow
  flexible packaging of libraries
• Simple Plug-in API for easy addition of new
  formats
• Additional software can be easily added to
  provide other functions such as using semantics
  from an ontology to use the data intelligently

43
PyESML Example
List all the API functions
Open a data file using ESML
List all the structures
List all the fields
Get the data for a field
Print the data
44
Current Status

• ESML data formats
• Currently supported
• ASCII, Binary, HDF-EOS, netCDF, Grib
• In work
• HDF5
• ESML Library
• Currently available
• C for Windows and Linux, Python plugin
• In work
• IDL plugin
• ESML Editor application

45
For More Information

• URL esml.itsc.uah.edu
• Become a member and post ESML related news items
  on the website
• Schema and related documents available to all
• Download the latest products
• Source code available via Source Forge
• Join the ESML mailing list