City College NOAA Compression Group

1
City College NOAA Compression Group

• Irina Gladkova
• Michael Grossberg
• Leonid Roytman

2
Acomplishments

• Multi-spectral imager algorithm for MODIS as
  GOES-R proxy
• Implemented algorithm (C/C)?
• Testing with Walter Wolf (Camp Springs)?
• Destriping Algorithm(for compression )?
• Extending compression comparison of
  state-of-the-art compression

3
Sensor Palette (MODIS)?
250m 40 sensors
500m 20 sensors
1km day 10 sensors
1km night 10 sensors
4
Data Blocks
5
Compression Comparisons
6
By Season or Hemisphere
7
Overall Compression by Block
8
Destriping for Compression
9
Stripy Image
10
Destriping Histogram Specification
11
Imager Algorithm based on Remapping Intensities
12
Prediction Based Compression
Band 1
Step1 Compress Band 1 Best Current Technique
13
Prediction Based Compression
Predicted Band 2
Band 1
Step2 Predict Band 2 from Band 1(Black Box)
Prediction
14
Prediction Based Compression
Predicted Band 2
Residuals
Actual Band 2
Step3 Compute Difference to Prediction
15
Prediction Based Compression
Residuals
Step4 Compress Residuals
16
Prediction Based Compression
Step 5 Continue Through Channels
17
Prediction Based Decompression
Band 1
Step1 Decompress Band 1 Best Current Technique
18
Prediction Based Decompression
Residuals
Step2 Decompress Residuals
19
Prediction Based Decompression
Predicted Band 2
Band 1
Step2 Predict Band 2 from Band 1Same step as in
compression
20
Prediction Based Decompression
Predicted Band 2
Band 2
Residuals
Step4 Reconstruct Band 2 losslessly
21
Prediction Based Decompression
Step 5 Continue Through Channels
22
Non-Linear Prediction

• Need to match values
• Initial Method
• Per Pixel
• Histogram Specification (non-linear)
• Completed Improvements
• Least Squares (non-linear)

23
Ongoing Prediction Improvements

• Functional Model Based Prediction
• Re-Ordering Channels Based on Correlation
• All previously decompressed channels
• Not just previous band

24
Variation 2nd Channel of 250m MODIS
Compression ratios across 7 Granules
25
Overall average Compression Ratio

26
Implementation

• Core compression implementation in C/C
• Ongoing Improvements
• Intel-Libraries for basic compression
• Possible 50x speedup
• Scientific Python based scripting
• Faster improvements to implementations
• Direct compression of hdf

27
Testing

• Developing More Complete Testing Suite
• Walter Wolf (Camp Springs)
• Have reported they are running our code and
  testing

28
Future

• Improve prediction with more bands
• Work with NPOES
• Patent
• Making Implementation Portable/Faster
• Generalize alg for many sensors
• SEVIRI, MODIS, AVIRIS, ABI-GOES-R, MTSAT
• Segmentation, Search, Fusion

29
Thanks