Service Oriented Atmospheric Radiances (SOAR)

1
Service Oriented Atmospheric Radiances
(SOAR)
AGU Presentation

• M. Halem(1),Ye. Yesha(1), C.Tilmes(2),
  D.Chapman(1),P. Nguyen(1), J. de La
  Beaujardiere(2), N. Most(3),K. Stewart(3),A.
  Bertolli(3)
• University of Maryland, Baltimore County,
• (2) NASA Goddard Space Flight Center, (3)Innovim
• 
  
• 
  halem_at_umbc.edu
• 
  
• 
  

December 11, 2007
2
OVERVIEW

• Motivation
• Gridding Problem
• SOAR Services
• SOAR AIRS/MODIS Grid Products
• Summary/Comments
• --------------------------------------------------
  --------------------------------------------
• Acknowledgements
• Francis Lindsay, NASA ACCESS Program Manager
• Andy Rindos, IBM Center for Advanced Studies,
  Manager
• Pat Gary, GSFC and Jerry Sobieski, UMCP for
  10Gb/s fiber network
• Wen Li Yang, GMU for installing WCS on the UMBC
  compute cluster
• Chris Lynnes, GSFC DAAC Manager for enabling AIRS
  data transfer

3
Motivation

• NASA, NOAA and DOD have been collecting and
  archiving satellite IR radiance data from
  temperature and moisture sounding instruments for
  the past 38 years.
• Radiance data are assimilated operationally as
  daily input to weather forecasting models
  nationally and internationally to improve their
  forecast accuracies and range of prediction.
• Radiance data have been used directly to
  determine observed global trends in temperature,
  ozone, albedo, desertification, aerosols,
  etc.(Spencer et. al. 1990)
• Radiance data are used in reanalysis of the past
  1/2 century to infer the historical record of
  global atmospheric states and to validate climate
  models derived fields (Kalnay et.al. 2000,
  Charney et. al. 1969)
• Gridded data sets for IR spectral radiance
  measurements of the AIRS and MODIS instruments
  are not products produced operationally nor are
  they available for other satellite measurements
  of IR radiances.
• IR Atmospheric Radiance Data are considered one
  of the most important and versatile climate
  measurements acquired from space ( Goody North,
  1998).

4
Why should we have gridded data sets on demand?

• More than a petabyte of satellite sounding
  radiance data collected over 3 decades and these
  data are expected to double again by 2012.
• Atmospheric radiance data products are stored at
  different spatial resolutions, spectral
  frequencies, formats, units, and archives
• Providing averaged gridded data sets is a form of
  data compression seamlessly transformed from
  orbital granules or tracks enabling scientists to
  easily subset, analyze and compare arrays of
  long records with model outputs and observations
  from multiple satellites
• The simultaneity of AIRS and MODIS IR channels on
  same platform is a unique opportunity for long
  term spectral radiance inter comparisons. NPP
  and NPOESS will provide similar multi-sensor
  radiance data from CrIS VIIRS for the
  foreseeable future
• The SOAR system, sponsored by the NASA ACCESS
  program, provides web based access and tools for
  multi-sensor gridded AIRS/AMSU and MODIS data
  records on demand and has demonstrated that
  these instruments have shown remarkable IR
  spectral radiance consistency and stability
  extending more than 5 years
• Still need to implement radiometric measurement
  algorithms for performing convolutions, limb scan
  adjustments, and statistical analysis tools for
  multi sensor data on demand
• SOAR can simulate what a multi satellite IR
  spectral radiance mission (CLARREO) might
  produce as an absolute climate record measurement
  (NRC Earth Science Decadal Report)

5
What is Gridding problem for AIRS/AMSU FOV?
6
Gridding MODIS radiances within single AIRS FOV
7
Gridding as Data Reduction

• AIRS produces 3106 obs/day for each of 2378
  spectral sounding channels or 7109
  Bytes/day
• MODIS produces 6108 obs/day for 16 spectral
  sounding channels or 10109 Bytes/day
• A 1.0 x 2.0 deg. grid with one average obs. for
  30 ch. per grid element reduces AIRS by 104
  and MODIS daily data volume of 16 IR ch by
  104. A 0.5 X1.0 deg. grid yields AIRS
  reduction of 103 and MODIS 103 resp. or
  10 MB/day
• A 0.5 X0.5 deg. grid is poor mans
  AIRS/AMSU/MODIS footprint collocation scheme
• Gridded radiances enables scientists to
  manipulate decades of atmospheric radiance data
  on demand climate analysis studies on their own
  laptops.
• (i.e. 40 GB per decade at 0.5 X 1.0 deg. is
  quite manageable today) .

8
SOAR Gridding Services

• SOAR is a publicly accessible web service that
  provides complex gridding services on-demand for
  AIRS, AMSU and MODIS radiance data
• SOAR removes tedious transformations needed for
  users to conduct the following climate analysis
• Maps orbital data sets into user specified
  spatial/ temporal/ spectral lat-lon arrays (I.e.
  gridded data sets)
• Performs arbitrary spatial-temporal subsetting
  of pre-gridded radiance arrays,
• Offers pre-gridded daily arrays of average
  radiance in grid bin, first footprint in bin,
  min/max value in bin, brightness temperatures for
  asc. desc, orbits
• Can deliver higher resolution regional gridding
  of spectral radiances on demand
• Calculates scatter plots, difference
  fields,multi-sensor radiance plots, and
  statistics of arbitrary spatial/temporal
  averaged channels
• Creates displays, animations and data delivery
  services of radiances, BTs, difference fields and
  other requested products.
• SOAR is modular and extensible, so it can add
  additional services and other IR sensors to the
  repository building on a common framework
• Still needs to include services for spectral
  convolution matching and limb scan adjustments on
  demand before gridding.

9
AIRS radiances gridded at 1.0 x 0.5 deg res

• On left, AIRS Asc gridded daily averaged radiance
  data for Jan1, 2005 for ch 2333
• On right, AIRS Desc gridded daily averaged
  radiance data for Jan 1, 2005 for ch 2333

10
MODIS gridded radiance data at 1.0 x 0.5 deg
res.

• On left, asc MODIS daily averaged gridded
  radiance data for Jan 1, 2005 for channel 32
• On right, desc MODIS daily averaged gridded
  radiance data for Jan 1, 2005 for channel 32.

11
SOAR V1.0 status Subsetting Gridding of AIRS
and MODIS Data bluegrit.cs.umbc.edu/soar/
id curt password this

• Developed a manual processing service that
  remotely accesses level 1B AIRS/AMSU and MODIS
  data on line over a 10Gb/s fiber optic net from
  UMBC through UMCP (Dragon Systems) to the G/DAAC
  and MODAPS (avg rate 440Mb/s).
• Processing service averages all granule
  observations lying in an arbitrary grid box
  resolving granules crossing poles and longitude
  boundaries.
• Generated 3 years of AIRS global gridded
  averaged radiances for 324 chs, BTs, min/max
  grid value, asc. desc., at 0.50 lat x 1.00 lon
  using IBM blade cluster at UMBC. (Grids 6 days in
  70 min/day in parallel on 12 processors or
  12min/day)
• Generated 2 months of AQUA MODIS global gridded
  averaged data at 0.50 x 1.00. and 1 week of TERRA
  MODIS ( Grids 6 days in 90 min on 12 processors
  in parallel or 15min/day) MODIS keeps only last
  month of data on line.
• Employed GMU WCS and WMS system to display a
  daily instrument resolution of AIRS and MODIS BT
  fields and their differences for processed
  gridded data fields
• SOAR v1.0 has full subsetting, averaging, anomaly
  creation and display capability for AIRS/AMSU and
  being extended to deliver MODIS data sets as
  well.

12
Global fields of AIRS and MODIS BTs

• Global AIRS BTs for Channel 2333(3.8um)
  averaged for week of Jan. 1-7,2007 on left.
• Global MODIS BTs for channel 20 (3.8um)
  averaged
• for same week of Jan. 1-7, 2007 on right.

13
Global BT fields of AIRS and MODIS

• Global AIRS BTs for channel 528 (12um) for week
  of Jan. 1-7, 2005
• Global MODIS BTs for channel 32(12um) for week
  of
• Jan. 1-7,2005

14
Sample Channel Difference Fields for MODIS-AIRS

• Difference of MODIS ch 20(3.8um) -AIRS ch
  2333(3.8um) on left for Jan1-7, 2005.
• Difference of MODIS ch 32(12um) -AIRS ch 528
  (3.8um) on right for Jan1-7,2005.
• Color Scale on right

15
AIRS/MODIS ch Radiances and BTs plotted vs
Spectral wavelengthfor Jan. 2005 showing
spectral consistency for two distinct instruments

• Plot on left shows the radiances averaged over
  20N to 20S for each of the 16 MODIS IR channels
  (wavelength) in red and the 324 AIRS operational
  channels (wavelength) in green for Jan 1 2005.
  The MODIS value is at the center of the red box.
• Plot on right shows the brightness temperatures
  for the same averaged radiances, for the same
  region and same channels on the same day and same
  resp. color codes.

16
AIRS/MODIS Channel Plots

• AIRS/MODIS Radiance and brightness temperatures
  resp. for OCT. 1, 2007
• Consistent stable instrument measurements after
  5 years!

17
3 Yr Feb 05,06,07 mean and monthlyanomalies Ch
528
18
3 Yr Feb 05,06,07 mean and monthlyanomalies Ch
2333
19
SOAR V2.0 inclusive of Science Servicesalpha.soft
warereuse.net/nsgerard/soar/ any ID Password

• SOAR v2.0 provides gridded AIRS, AMSU and MODIS
  data in a bounding box for each specified channel
  averaged over a user specified time window
• SOAR v2.0 remotely accesses L1B data on demand
  from G-DAAC or MODAPS and calculates all data
  lying in grid element for each element of granule
  and performs various statistics on data forming
  the grid.
• SOAR v2.0 provides a set of scientific web based
  services for conducting multi-sensor gridded
  analysis with arbitrary spectral and spatial
  resolutions
• SOAR v2.0 includes WMS as a mapping service for
  overlaying and differencing AIRS and MODIS data
  on global mapped Earth dataset on demand
• SOAR v2.0 provides animations of anomalies,
  difference-maps, trend-lines, and other
  statistical derivations and maintains a list of
  the user's requested visualizations
• SOAR v2.0 will capture a complete provenance
  documentation during data ingest and processing
  that is easily searched and retrieved and
  provides scientific reproducibility for on demand
  data sets

20
WMS web server AIRS/MODIS options for displays
21
MODIS ch 32 (12um), AIRS ch 528(12um), Diff.
for single day
Diff
22
The table gives the mean MODIS -AIRS BT
difference and the extrema.
TIME MODIS ch 32
AIRS ch 528 Global
MIN AV BT MAX AV BT
MinAV BT MAXAV BT Mean(M-A) 2005-01-01T00
185.209 297.947 182.843 298.529
-0.107417 2005-01-01T12 191.348 330.617
194.016 332.876 -0.103135 2005-01-02T00
186.753 298.444 184.100 299.589
-0.182602 2005-01-02T12 188.018 329.164
186.023 332.400 -0.0748267 2005-01-03T00
184.950 297.238 187.726 298.225
-0.107340 2005-01-03T12 191.853 332.165
191.066 334.149 -0.132432 2005-01-04T00
223.608 280.481 183.096 300.198
-0.166676 2005-01-04T12 184.690 331.859
182.096 334.090 -0.0385201 2005-01-05T00
186.180 297.299 191.723 298.239
-0.168761 2005-01-05T12 192.457 331.788
192.568 333.873 -0.0628987 2005-01-06T00
193.218 300.229 191.847 300.873
-0.220639 2005-01-06T12 196.028 331.110
195.323 334.096 -0.0816157 2005-01-07T00
193.182 298.362 192.672 299.223
-0.203587 2005-01-07T12 191.641 331.556
193.265 334.224 -0.0958175 2005-01-08T00
188.707 298.699 189.957 299.496
-0.274076 2005-01-08T12 190.769 329.309
191.787 332.531 -0.159123
23
Global Image Request
One-Day Radiance Average
Submit Request
View Results
24
Global Scattergram Request
Monthly Radiance Average (MODIS vs AIRS)
Submit Request
View Results
25
Global Image Request
One-Day Radiance Difference (MODIS-AIRS)
Submit Request
View Results
26
Summary

• SOAR Services
• Provides multiple gridding options
  for arbitrary spatial/spectral resolutions
• Incorporates a variety of instrument
  science analysis tools as services
• Provides transparent computing, universal data
  access and compression
• SOAR System Framework
• Offers interagency interfacing (NASA, NOAA,
  UMBC) to access multi-sensor data
• Flexible framework for integrating future
  missions, e.g. NPP,NPOESS, IASI
• Generates long term multi-instrument climate
  data records AIRS. MODIS. Etc.
• An educational and scientific tool for
  climate analytical studies.
• SOAR Scientific Findings
• Relative AIRS and MODIS IR spectral radiance
  measurements have not degraded in over five
  years
• AIRS and MODIS gridded IR spectral radiances
  have potential to provide long
• term (10 year) fundamental climate data
  record
• Maintains complete provenance for on demand
  scientific reproducibility

27
Backup

28
Compute Storage Req. of a 1.0 x 0.5 deg gridded
global data set

AIRS gridding on UMBC compute cluster bluegrit (All times shown in minutes) AIRS gridding performance on bluegrit AIRS Processing Compute time AIRS Processing Compute time
Products Avg rad, BT, min, max, Asc Desc. for 324 oper. sp. channels.
Avg Timings Storage
AIRS /MODIS Storage /daily array 0.2 GB/ 0.005 GB Blade Type Blade Type
AIRS/MIODIS Storage/month (all arrays) 40.0 GB/ 0.12 GB JS20 JS21
NumDays/ NumBlades 0.5, 1 27.01 14.09
Compute time/day 23min. 1, 2 23.27 21.65
Compute time/day 13 min 2, 4 26.45 31.08
compute time/day 8min. 3, 6 37.99 40.22
Internet Speeds(Fiber Optic 10/Gb/s,eff. Max 5Gb/s) to GSFC 440 Mb/s
29
Visualization Services
30
AIRS radiances gridded at 1.0 x 0.5 deg res.

• AIRS daily averaged gridded radiance data for asc
  and desc orbits combined for ch 2333 on Jan 3
  2005.