An overview of realtime radar data handling and compression

1
An overview of real-time radar data handling and
compression
Valliappa Lakshmanan National Severe Storms
Laboratory University of Oklahoma http//cimms.o
u.edu/lakshman/
2
Real-time radar data handling and compression

• Data characteristics
• Real-time handling
• Compression
• Summary

3
Operational radar network

• Weather Surveillance Radar 1988 Doppler (WSR-88D)
• 142 radars in CONUS
• 11 in Hawaii Alaska
• S-band
• Reflectivity to 460km
• Velocity to 230km

4
Operational radar data

• Each radar operates according in one of several
  volume coverage patterns
• NWS forecast offices choose VCP for their
  radars
• All VCPs involve several elevation angles
• Resolution (current)
• 0.95 degrees
• 1km (ref)
• 0.25 km (velocity)
• Tilt in 20-60s
• Volume in 4-10 min
• Super-resolution
• 0.25km x 0.5 deg
• Faster VCPs

5
Radar data levels

• Level-I (or IQ data)
• Signal processed at radar
• 64 values at each range gate
• Not transmitted/stored routinely
• Level-II (moment data)
• Reflectivity, Velocity, Spectrum Width
• 3 scalar values at each range gate (pixel)
• Distributed via LDM (more on this later)
• Level-III (products)
• Produced by algorithms in the Radar Product
  Generator
• Distributed via NOAA Port
• Somewhat obsolete, now that Level-II is available
  in real-time
• Better products can be created in real-time
  off-site

6
Size of radar data

• Focus on Level II
• Size of radar data depends on the VCP
• Clear-air VCPs have fewer (4-5) tilts collected
  more slowly
• 7.1 MB in 990 seconds for VCP 31
• 0.6 GB per day
• Bandwidth 7.3 kB/s
• Storm-type VCPs have more tilts, therefore more
  data
• 15 MB in 460 seconds for VCP 12
• 2.8 GB per day
• Bandwidth 33.4 kB/s
• A CONUS radar processing system needs to process
  142 radars
• On a typical storm day, 1/3 of CONUS radars are
  in storm mode 200 GB
• Mean 60.6 GB/day when compressed
• Bandwidth 736 kB/s (peak 1771 kB/s)

7
Real-time radar data handling and compression

• Data characteristics
• Real-time handling
• Compression
• Summary

8
Real-time Distribution
9
Distribution components

• Radar Interface and Data Distribution System
  (RIDDS)
• Taps into radar stream at RPG
• Transmits over NWS network to regional server
• Regional servers on Abilene network
• Load dependent on geography
• Southern region average bandwidth 484 kB/s
• Western region average bandwidth 74 kB/s
• Real-time data may be requested via Local
  Directory Manager (LDM)
• Developed by Unidata
• Peer-to-peer realtime data distribution
• Top-tier providers Oklahoma, Purdue, Maryland
• Transmission is in chunks of radials (all 3
  moments)
• 50 radials at a time
• Compressed using bzip2

10
Real-time radar data handling and compression

• Data characteristics
• Real-time handling
• Compression
• Summary

11
bzip2!

• Custom compression methods can work better than
  bzip2
• Example V. Lakshmanan, Lossless coding and
  compression of radar reflectivity data, in 30th
  International Conference on Radar Meteorology,
  pp. 5052, American Meteorological Society,
  (Munich), July 2001

Not further compressible
gzip
Linear prediction, Run length encoding, Huffman
coding
bzip2
12
So why bzip2?

• Order of improvement of custom compression over
  bzip2 10
• Maintenance of compress/uncompress software has
  significant costs
• Every user of radar data would have to implement
  uncompress code
• Costs are spread out over many organizations
  really adds up
• 10 decrease in bandwidth/space not worth the
  increased cost
• Would be different if this would bring bandwidth
  below some cost point
• Example T1 line or dial-up?
• Choose among publicly available, non-proprietary
  compression formats
• gzip, bzip2, etc.
• Burrows-Wheeler (bzip2) performs best, so chosen

13
Preprocessing

• Some preprocessing done at radar site to improve
  compressibility
• Level I signal-to-noise threshold
• Level II height threshold
• Both done at radar site
• Part of radar system
• No cost on part of users
• Compression algorithm is still bzip2
• Theoretically lossy compression
• But loss is on part of image not useful to
  end-users (JPEG?)

14
Compressibility

• But
• Compression has uses beyond saving storage and
  bandwidth
• The ideas behind compressibility have wider use

15
Instrument artifacts
Mostly good data
Instrument artifact
16
Quality control of radar data

• QC needs to be done before data are fed to
  automated algorithms
• Done by means of machine intelligence algorithms
• Helps distinguish ground clutter, biological
  contamination etc.
• But
• Hard to distinguish between stratiform rain and
  instrument artifacts
• Both fill the radar volume
• With reflectivity values around 30-50 dBZ
• Velocity and spectrum width do not help
• Texture-based quality control doesnt help
• Reflectivity fields in both cases are locally
  smooth

17
Shannon entropy

• Theoretical maximum compressibility based on
  information content
• No need to actually compress file
• Simply compute Shannons information measure
  (entropy)
• The information content in the two fields
  (globally) is different
• Stratiform rain has high information content
• Even if most of the radar volume is not filled
• Instrument artifacts have low information content
• Even though most of the radar volume is filled
• Can easily detect changes in entropy
• Reject instrument artifacts even if it comes
  during a time period of rain

18
Real-time radar data handling and compression

• Data characteristics
• Real-time handling
• Compression
• Summary

19
Summary

• Radar data is transmitted routinely in real-time
  to interested users
• Compressed in chunks of 50 radials each
• Peer-to-peer sharing (software LDM)
• Compressed using bzip2
• Custom compression techniques are not
  cost-effective
• Perform better than bzip2
• But incremental (10) improvement not worth the
  increased cost
• Compression ideas have other uses
• For example in quality control of radar data