WDSSII Data formats

1
WDSS-II Data formats

• Valliappa Lakshmanan
• National Severe Storms Laboratory University of
  Oklahoma
• lakshman_at_ou.edu

2
Data types

• WDSS-II is based on types of products, not the
  products themselves.
• These are the most common WDSS-II data types
• RadialSet
• PolarGrid
• CartesianGrid2D
• CartesianGrid3D
• LatLonGrid
• LatLonHeightGrid
• DataTable
• ContourData
• WindFieldStructltLatLonGridgt
• WindFieldStructltCartesianGrid2Dgt

3
RadialSet

• A set of Radials
• In any order
• Not necessarily indexed
• At a constant target elevation angle to the
  ground
• Can cover more/less than 360 degrees
• (0,0) is the radar location
• A Radial
• A 1D array of gates
• Constant beam-width and gate-width
• Defined by starting azimuth and beam width
• The data values are scalars (float)
• Reflectivity and Velocity are two separate
  RadialSet

4
PolarGrid

• A more convenient form of a RadialSet
• Covers 360 degrees (with missing data if needed)
• At a constant target elevation angle to the
  ground
• Can be a projection of a RadialSet to the ground
• Simplifies vertical computations
• Can be indexed to start at defined azimuths
• Is a 2D array of scalars (float)
• First dimension is the radials
• Second dimension is the gates
• Goes clockwise from due-north
• (0,0) is the radar location
• Polar data are stored as RadialSets only
• Algorithms convert a RadialSet on the fly to a
  PolarGrid
• Very fast and much more flexible
• The algorithm can project/index/sub-sample as
  required.

5
CartesianGrid2D

• Not a recommended structure
• Recreates flat-earth structure of many NEXRAD
  single-radar algorithms
• Polar data projected to a plane tangent to the
  earths surface at the radar
• A 2D array of scalars
• SpaceTimeRef is the radar location
• (0,0) is the north-west corner
• First dimension is the row number
• Second dimension is the column number

6
CartesianGrid3D

• Enables vertical computations on layers of
  CartesianGrid2D products
• Constant height above the radar.

7
LatLonGrid

• A 2D array of scalars
• Cylindrical equidistant projection
• (0,0) is the north-west corner
• First dimension is latitude
• Second dimension is longitude
• Lat-long spacing is constant throughout grid
• Only earth-relative system supported by
  algorithms
• Read/write this format only
• Ingest routines will need to convert from
  Lambert, Mercator, etc. (classes exist to
  simplify this).

8
LatLonHeightGrid

• A stack of LatLonGrid
• Each LatLonGrid is at constant height
• The LatLonHeightGrid may have either uniform
  heights, or non-uniform heights
• (0,0,0) is the lowest north-west corner
• First dimension is height
• Second dimension is latitude
• Third dimension is longitude

9
Attributes

• The gridded products only have scalars.
• How do you store the VCP of a RadialSet?
• The expiry time of a product?
• Store optional things as attributes
• In source code
• setAttributeValue(vcp, 21)
• getAttributeValue(vcp) returns a DataCell
• Since attributes are optional, algorithms should
  be able to proceed if an attribute is not present
  in the data.

10
SparseGrid

• Storing complete grids can be wasteful
• For example, a running average or sum
• Need to have 50 grids in memory
• Can use SparseGrid to do this optimally
• SparseGrid2DltLatLonGridgt or SparseGrid2DltCartesian
  Grid2Dgt
• Will only store non-background values
• Useful when the grids are sparse

11
SpareGrid netcdf files

• The WDSS-II netcdf writer will try to optimize
  your write
• Choosing to write only non-missing values
• Or the entire grid
• Based on which will save you space
• Can turn this off using environment variable
  W2_SPARSE_GRID_THRESH
• -2 to always write in sparse-grid format
• 2 to always write gridded values
• Default value is 0.33

12
DataTable

• Used to store tabular data
• Column-by-column
• The values and their units only
• The colors are determined by display
  configuration
• The units of storage should be SI, not English
• Special columns
• RowName to identify unique names that will be
  matched across time
• Latitude, Longitude
• If no columns with this name exist, all rows are
  at the same location.

13
WindFieldStruct

• A data product that consists of 2 or 3 LatLonGrid
  or CartesianGrid2D
• Used to store vector quantities
• u,v,w

14
ContourData

• A product that consists of a sequence of lat-lon
  locations organized as polygons.

15
What output format?

• The output format of WDSS-II algorithms is
  determined by w2config/misc/dataformat
• The default file uses XML for DataTable
• Uses gz-compressed netcdf for all others
• Can change this to any format
• No need to change algorithms source code.

16
How to add an output format

• If some one wants you to write data in a
  different format
• Algorithm can write out in netcdf
• Use converter to convert the data to new format
• Recommended solution if the other format is not
  complete or properly geo-referenced.
• Or if you will still need the netcdf format for
  other purposes.
• You can write a DataEncoder subclass to natively
  write data in the other format
• Change w2config/misc/dataformat to write other
  format
• The algorithm source code doesnt change
• Recommended solution when deploying the
  application in other scenarios.
• Note that if you need to read this data back, you
  also need to implement a Builder and Extension
  Loader

17
Netcdf and XML data format

• The netcdf and XML data formats that WDSS-II
  algorithms use are documented online
• See http//www.wdssii.org/
• And click on Data format

18
LatLonGrid netcdf format

• dimensionsLat 650 Lon 700
• variablesfloat SHI(Lat, Lon) SHIUnits
  "dimensionless"
• // global attributesTypeName "SHI"
  DataType "LatLonGrid" Latitude 37.
  Longitude -100. Height 1000. Time
  990402843 FractionalTime 0.475000000005821
  attributes "" LatGridSpacing 0.01
  LonGridSpacing 0.01 MissingData
  -99900.f RangeFolded -99901.f

19
Index Record

• Whenever you write out a data file
• You need to also write out a Index Record
• This notifies waiting algorithms about your new
  product
• The record has three parts
• Time of product (UTC seconds since 1970)
• Integer fractional part
• Name of product (selections)
• YYYYMMDD-HHMMSS InfraredTemperature band2
• Time TypeName SubType
• In the display, the time is listed last.
• How to read the product
• FormatName format-specific-parameters
• Eg netcdf machinedirectory filename.gz
• Eg w2algs xmldata machinedirectory xmlldata

20
makeIndex

• Can automatically create XML/netcdf index records
  from a set of files
• Run makeIndex.pl or makeNetcdfIndex
• makeIndex.pl assumes file naming convention
• makeNetcdfIndex reads the data file
• Very expensive
• Better to have writing program create record also
• Use DataEncoder when writing a WDSS-II program
• Non-WDSS-II programs (C or Perl) can use lb_cat
• Open a pipe (popen) to lb_cat
• Write out XML string corresponding to record
• Close the pipe