NDE XML Tailoring Service

1
NDE XML Tailoring Service

• HDF and HDF-EOS Workshop XIII
• November 4, 2009
• Tom Feroli, Peter MacHarrie

2
Goal

• Reduce the labor effort to create user-specific
  tailored products (I.e., image, gridded, map
  reprojected, and other specific file formats,
  etc.)
• Objective
• Develop (InputFormat Output Format) software
  applications instead of (Input Format Output
  Format) software applications.
• Approach
• Develop a service that provides routine data
  manipulation (i.e., sub-setting, sub-sampling,
  filtering, aggregation) and places output into a
  well-defined intermediate format.
• Create tailoring applications that accept a
  well-defined intermediate format and configurable
  output options to create a specific end-user
  product.

3
Data Selection Driver
XML
Reads
Writes
Data Selection Driver Uses Binary
Format (Subsample, Subset, Aggregate, Filter,
Scale/Offset)
HDF5
netCDF4
netCDF4
netCDF3

• Modular design, accepts drivers to accommodate
  additional input
• and output file formats (e.g., HDF-EOS5,
  netCDF3, etc.)

4
Enterprise Measure

• Enterprise Measure is a logical construct
• It is the concatenation of all related arrays in
  all files in an enterprise
• File 1 Array1 (Scan 12, Sample96,
  Channel22)
• File 2 Array1 (Scan 12, Sample 96 ,
  Channel22)
• Enterprise Measure1 (Scan24, Sample96,
  Channel22)
• Enterprise Measure reference Enterprise
  Dimensions
• A file, is a storage unit, for a partition, of
  the Enterprise Measure
• The partition is a range of values of one or more
  Enterprise Dimensions
• File 1 Scan Dimension 1 -12
• File 2 Scan Dimensions 13-24

5
Enterprise Dimension

• Enterprise Dimension is a unique logical mapping
  to dimensions within all data files.
• In VIIRS, Image and Moderate product profiles,
  the dimension AcrossTrack is defined as 6400 and
  3200
• Enterprise dimension are named uniquely for each
• AcrossTrack-I 6400
• AcrossTrack-M 3200

6
Enterprise Data Cube

• Data Cube is a structure of one or more
  Enterprise Measures. Within a data cube
• Measures share the same dimensions
• Measure names are unique
• Measures may have different data types
• In VIIRS SDRs all arrays are named Radiance
• Enterprise measure uniquely name each array
• Example
• Radiance_I1 -gt Product I1-gtRadiance Array
• Radiance_I2 -gt Product I2-gtRadiance Array

7
Enterprise Data Structures
Tailoring Service XML References Enterprise Data
Structures
8
Measures (additional information)

• Named Subset
• Array AntennaTemperature(12, 96, 22)
• Measure AntennaTemperature, references entire
  AntennaTemperature array
• Measure AntennaTemp-Channel_1, references array
  AntennaTemperature, with sub-sample specification
  of Start(1,1,1), Stride(1,1,1), Count(12,96,1)
• Named BitMask
• Array QF1_VIIRSCMIP(768, 3200)
• Measure CloudConfidencePixel
• Has Integer DataType
• References array QF1_VIIRSCMIP starting at bit 2
  for 2 bits
• Scale and Offset
• Measure defined as float, references an integer
  array and its scale and offset are applied
• Measures can be referenced in the ltselectgt or
  ltfiltergt elements of the tailoring XML.
• ltwheregtltfiltergtltmeasure nameCloudConfidencePixel
  gt
• ltoperation valueLTgtltoperand value2/gtlt/filter
  gtlt/wheregt
• (i.e., CloudConfidencePixel lt 2)