The Next Generation of McIDAS: McIDASV

1
The Next Generation of McIDAS McIDAS-V

• Presented at the Latin American Data Workshop
• Sao Paulo, Brazil
• August, 2008
• by
• Tom Whittaker

Space Science Engineering Center (SSEC) at
the University of Wisconsin - Madison
2
What is McIDAS ?(Man computer InteractiveData
Access System)

• Collection of user programs and libraries
• for visualizing and analyzing geophysical data
  (focus on
• environmental satellites)

• UNIX, PC Mac capable
• A synergistic tool that integrates numerous data
  types into one system
• First developed in the early 1970s
• Still in use world-wide at research, operational,
  educational, and commercial sites

3
Key McIDAS Attributes

• Access to extensive geophysical database
• Core package plus user-written applications
• Diverse functionality through software (1
  million lines of code)
• Extensive visualization capabilities
• Satellite and NOAAPORT data ingest

4
McIDAS Functionality

• Digital Image Processing
• GIS Applications
• Weather and Climate Data Analysis and
  Applications
• Graphical Displays of Data Information
• Gridded Data Processing and Analysis Tools
• Display Process Control Utilities
• Interactive and Background Processing

5
Current McIDAS Users

• NOAA NESDIS, AWC, SPC, TPC, etc.
• NASA STS, LaRC, MSFC, JPL
• Unidata 130 universities, colleges and
  international collaborators
• International EUMETSAT, Spain, Greece, Mexico,
  Australia
• Industry Honeywell, Weathernews, Universal
  Weather, Meteorlogix, Weather Central, etc.

6
Why the Change?

• Forthcoming GOES-R NPOESS operational satellite
  data cannot be optimally utilized
• great increase in data rates
• new tools for working with these large data sets
• McIDAS software (written in Fortran 77 and C) has
  a 30 year heritage resulting in limited
  extensibility potential
• New data analysis and visualization concepts
  cannot be incorporated (e.g. 4-D)

7
Direct Broadcast Continuous Evolution
MIS
SSM/I, SSMIS, SSM/T, SSM/T-2
8
Hyperspectral Altitude-Resolved Water Vapor Winds
Simulated GOES-R winds (left) versus current GOES
(right)
9
McIDAS-V Looking to the Future

• We seek to advance the functionality of McIDAS to
  meet the challenges of the future while retaining
  its current capabilities, including
  user-developed code.
• We are developing a means to transition existing
  McIDAS applications to operate in a new
  environment that supports the next generation of
  observing systems and does not have the
  limitations of the current McIDAS system.

10
McIDAS-V Functionality
McIDAS-V will be a collection of software tools,
and networked services and data designed to take
advantage of a scalable distributed computing
environment to meet user needs

• McIDAS-X
• OpenDap / OpenADDE
• Open GIS Consortium
• Database archives
• Cluster computing

11
Formal Project Requirements

• Create a powerful and versatile software system
  for environmental data processing, analysis and
  visualization
• Support existing and evolving needs of scientific
  research and algorithm/applications development
  for new programs
• Support data fusion and algorithm
  interoperability from existing and future sources
• Continue to fully support McIDAS Users Group
  (MUG) and McIDAS-X functionality as users
  transition to McIDAS-V
• Support operational users by providing frameworks
  in McIDAS-V, enabling a natural transition path
  for research results into operations
• Use system to educate students in remote sensing
  and physical sciences involve students in its
  development, evolution and use

12
What is McIDAS-V?

• McIDAS-X ? VisAD IDV HYDRA McIDAS-V

13
VisADDeveloper Bill Hibbard, UW SSEC

• Open-source, Java library for building
  interactive and collaborative visualization and
  analysis tools
• Features include
• Powerful mathematical data model that embraces
  virtually any numerical data set
• General display model that supports 2- and 3-D
  displays, multiple data views, direct
  manipulation
• Adapters for multiple data formats (netCDF,
  HDF-5, FITS, HDF-EOS, McIDAS, Vis5D, etc.) and
  access to remote data servers through HTTP, FTP,
  DODS/OpenDAP, and OpenADDE protocols
• Metadata can be integrated into each data object

14
What is the IDV?

• Unidata developed, VisAD-based, scientific
  analysis and visualization library and toolkit
• Open Source, Java framework and reference
  application
• Provides 2- and 3-D displays of geo-scientific
  data (plus, of course, animations)
• Stand-alone or networked application

http//www.unidata.ucar.edu/idv
15
(No Transcript)
16
(No Transcript)
17

• HYDRA (Hyperspectral Data Research Application)
  enables interrogation of multispectral and
  hyperspectral fields of data
• Individual pixel location and spectral band
  measurements can be easily displayed
• spectral channels can be combined in linear
  functions and the resulting images displayed
• false color images can be constructed from
  multiple channel combinations
• scatter plots of spectral channel combinations
  can be viewed
• pixels in images can be found in scatter plots
  and vice versa
• transects of measurements can be displayed
• L2 products e.g. soundings of temperature and
  moisture as well as spectra from selected pixels
  can be compared
• integrated data exploration and analysis between
  GEO and POLAR observing platforms

18
AIRS Cirrus vs Clear Sky Spectra
19
(No Transcript)
20
Mt Etna viewed by AIRS 28 Oct 2002
SO2 signal 1284-1345 cm-1
21
Inferring ash cloud height from AIRS clear sky
and in ash soundings
Ash cloud and clear sky spectra
22
HYDRA RGB composite tool applied to MSG SEVIRI
channels. In the above, Band 5 minus Band 6,
(6.7micron,7.3micron) is mapped to Red, Band8
minus Band9 (9.7,10.8) is mappped to Green and
Band5 is mapped to Red. The latter is inverted
using the histogram stretch tool. This composite
technique was developed by Eumetsat to emphasize
cyclogenesis, jet-streaks and mid vs high-level
clouds.
23
Offline-Online in LW CO2
24
Offline-Online in H2O
25
The X to V Bridge

• Interacts with a McIDAS-X remote session
• Users provides command line input in a McIDAS-V
  Data Chooser that sends commands to a server
  running McIDAS-X
• Runs all McIDAS-X commands, including status,
  text, imagery and graphics
• McIDAS-X output displayed in McIDAS-V
• Allows bi-directional interactive communication
  between McIDAS-V and McIDAS-X

26
The X to V Bridge
27
The X to V Bridge
28
McIDAS-V Transition Plan

• Build upon the existing capabilities of VisAD/IDV
• Incorporate the functionality of the
  Hyperspectral Data for Research Analysis (HYDRA)
  toolkit
• Allows McIDAS-X heritage code to be usable in the
  new environment without a need to rewrite
• Bridge software allows McIDAS-X commands to be
  submitted from the McIDAS-V display
• Provides a new environment for developing
  algorithms and new visualizations that take
  advantage of multi and hyper-spectral data from
  emerging observing systems

29
Proxy Dataset Activities

• Otkin and Huang submitted a successful proposal
  to the NCSA and received award for 192,000
  processor hours will be used to generate two
  proxy datasets covering very large geographical
  areas with fine spatial resolution
• The first simulation was configured to represent
  the anticipated GOES-R scanning regions (i.e.
  full disk, continental U.S. and a mesoscale
  domain)
• The simulation required 1 TB of memory and
  generated 10 TB of model output

30
Proxy Dataset Activities

• GOES-R research groups use model-derived proxy
  atmospheric profile and top of atmosphere (TOA)
  radiance datasets for their algorithm development
  and end-to-end processing activities.
• The Weather Research and Forecasting (WRF) model
  is used to generate physically realistic
  simulated atmospheric profile datasets with high
  spatial and temporal resolution
• Model-simulated data subsequently passed through
  a forward radiative transfer model to generate
  proxy top of atmosphere (TOA) radiances

31
Proxy Dataset Activities

• The outer domain covers the entire GOES-R
  viewing area with 6-km horizontal resolution
• The inner domain covers the CONUS area with
  2-km horizontal resolution
• The innermost domain represents a special
  mesoscale region and contains 667-m resolution

Simulation was performed at the National Center
for Supercomputer Applications
32
CIMSS GOES-R Proxy DatasetsTop of Atmosphere
Radiance Simulation
33
Proxy Dataset Activities
34
Proxy Dataset Activities
35
Oragami Demonstration A suite of software that
provides an easy workflow for processing,
analysis and visualization

• Handle otherwise unmanageable data sizes
• Utilize centralized computing, storage resources
• Provide easy workflow from search to
  visualization
• Demonstrate rapid prototyping of distributed
  scientific applications

36
Oragami Experiment Goals

• Visualization of meteorological fields from very
  large simulated model and retrieved data sets
• Submit request from web service or McIDAS-V
• Remotely query a large database to obtain the
  required data and load into an application or
  task
• Invoke the task on a cluster computer, reading
  from the database and writing results to a
  temporary file
• Inform the user where the output data resides and
  bring results into McIDAS-V

37
Viewing multiple data blocks (cubes or granules)
as part of a single visualization request across
a larger geographic area.
38
GeoCat - Background

• GeoCat was developed by the AWG Cloud Application
  Team for testing prospective GOES-R cloud
  algorithms
• Cloud algorithms require the use of many channels
  and much ancillary data. Thus, GeoCat can also
  run many non-cloud algorithms.
• In recognition of this, other AWG teams are
  already utilizing GEOCAT for their own
  development work (e.g. winds, land)
• We were directed by the AWG/AIT to incorporate
  all compatible CIMSS algorithms into GeoCat.
  This work is underway

39
GEOCAT Conceptual Model
Calibrated/navigated radiances and ancillary data
are loaded into data structures that can be
accessed by algorithms
-Satellite Images -Ancillary Data
Calculate clear sky radiances (if needed)
GEOCAT
Navigation and Calibration
Map ancillary data to pixel level
Execute lower order algorithms
Execute higher order algorithms
L1 (radiances) L2 (pixel-level products) RTM
(clear radiances)
Output from high order algorithms is available to
lower level algorithms
40
Example Usage
Command-line arguments are used to specify
run-time options.
./geocat -verbose -maxsatzen 70 -nscans 200
-use_seebor -use_snow \ -area_dir ./ -l1_dir ./
-l2_dir ./ -dumpch 2 5 14 \ -a 2 4 -f
met08_disk_1_2006_015_1200.area.gz
L2 Output
L1 Output
41
Geocat --- McIDAS-V Link

• McIDAS-V can readily access services following
  emerging standards for distributed application
  systems.
• For algorithm testing on large proxy, simulation
  and real datasets, the GeoCat test framework can
  be distributed on computing clusters.
• A toolset for submitting large GeoCat
  calculations and directly displaying output in
  McIDAS-V will allow increased productivity for
  algorithm developers.

42
Overview of Proposed Architecture
This is an example of how a GeoCat McIDAS-V web
service could operate
42
43
McIDAS-V is a collection of software tools, and
networked services and data designed to take
advantage of a scalable distributed computing
environment to meet user needs
GIS
Cluster computing
OpenDAP/ ADDE
McIDAS-V
GeoCAT
Database/ SAN
McIDAS-X
Matlab/IDL
44
McIDAS-V Future Work

• Complete HYDRA integration
• Complete development of the X to V Bridge to
  provide an evolutionary path for MUG into
  McIDAS-V (October 2007)
• Beta 1.0 release set for September 2008
• Support the development of applications for the
  NPP/NPOESS and GOES R science teams (ongoing)
• Data management and accessibility
• Broad array of formats and services
• Advanced analysis and visualization tools

45
McIDAS-V Contributors

• McIDAS-X/V
• Dave Parker
• Gail Dengel
• John Beavers
• Kevin Baggett

VisAD/IDV/Hydra Tom Whittaker Tom Rink Bruce
Flynn Don Murray Jeff Naughton Jeff
McWhirter Bill Hibbard
Origami/Geocat Ray Garcia Maciej Smuga-O Mike
Pavolonis Bob Knuteson