Realtime Data Delivery and Remote Visualization through Multilayer Interfaces

1
Real-time Data Delivery and Remote Visualization
through Multi-layer Interfaces
Vinaitheerthan Sundaram, Lan Zhao, Bedrich Bene,
Carol X. Song, Rakesh Veeramacheneni, Peter
Kristof Rosen Center For Advanced
Computing Department of Computer Graphics
Technology Purdue University
Work supported by National Science Foundation
2
NEXRAD II Data

• Next Generation Radar (NEXRAD) Level II Data
  (OR) Weather Surveillance Radar (WSR-88D) Level
  II Data
• This data contains a very fine temporal and
  spatial resolution of three attributes
  reflectivity, Doppler radial velocity and
  spectrum width
• These attributes are vital to understanding,
  monitoring and predicting severe weather
  conditions
• There are 158 Radar Stations in the country

Acknowledgment Figures are downloaded from
websites www.CCSU.edu and www.answers.com.
3
NEXRAD II Data Generation

• 3D structure in Radar Data
• Radars go through a programmed set of movements,
  which involve a continuous rotation over 360 in
  azimuth and a simultaneous increase in elevation
  by 1 to 3 per complete sweep
• Continuous NEXRAD Level II radar data stream
• The radar data files vary in size from a few
  megabytes to tens of megabytes each, depending on
  the weather conditions. The files are compressed
  using modified bzip2
• The temporal resolution is 4-5 minutes in severe
  weather vs. 9-10 minutes in calm weather

4
Availability of NEXRAD II Data in Near Real-time

• NEXRAD II Data is available in real-time on the
  TeraGrid through Purdue resource provider.
• Opportunity
• The real-time availability of high-resolution
  radar data provides an exciting opportunity for a
  wide spectrum of users ranging from basic (
  students) to expert (researchers) if the radar
  data can be accessed and visualized in 3D in a
  timely manner.
• However, catering to wide spectrum of users
  presents unique challenges as the requirements
  for each user differ.

5
Talk Outline

• Motivation
• Providing real-time data access and remote
  visualization for a wide spectrum of users
• Challenges
• A review of challenges in the state-of-the-art
  systems
• A Unique and Versatile System Design
• Multi-layer Interfaces
• Multiple Service access points
• Back-end Architecture The enabler
• Parallel Data Pre-Processing and partial-volume
  caching
• Summary and Future Work

6
Challenges

• Limitations in the State-of-the-art
• Not handle large amounts of data from many
  stations over a long time
• No direct interaction with the data for users
• Not accessible to general public because of
  complicated interfaces
• No access points to third party applications
• Challenges
• Data management issues Radar data streams at 50
  MB/secs
• Native compression format of radar data
• Data processing Computationally expensive
  processing
• Special-purpose hardware (GPU) required

7
Different User Levels and their requirements

• Expert Users ( small group )
• Perform in-depth investigation
• Examples Researchers, Emergency Management
  Personnel
• Learners/Casual Users ( large group )
• Access and visualize data for educational or
  personal purposes
• Examples K12 and College Students, Public
• Advanced Users ( small group )
• Explore and evaluate data but dont have
  resources
• Examples Graduate Students, Researchers from
  other domains
• Users levels are NOT mutually exclusive
• Expert User can be an advanced user when
  attending conferences and can be a casual user
  when teaching a class

8
System Design
Advanced Users
Casual Users
Expert Users
9
LiveRadar3D Gadget

• Web 2.0 technologies
• AJAX, Google Gadgets, Social Networking
  Applications
• Allows rapid dissemination of scientific tools to
  wide audience
• Live Radar 3D
• Shows animated Flash movie of 3D visualization of
  the region near users zip
• Scalable because movies are pre-generated and
  stored at the webserver
• Granularity 7 Regions (midwest, south, southwest
  .. )

10
LiveRadar3D Desktop

• Desktop client
• Written in C
• Runs on Linux / Windows
• Can be run on standard GPU cards
• Uses pre-processed volumes
• Leverages Teragrid processing power and local GPU
• Advantages
• Fast interactive 3D manipulation
• Scalable Supports large number of stations and
  large intervals of time

• Usage Scenario

User selects Radar- stations and time period
Tool connects to data-access interface and fetch
processed volumes
Tool renders on the local GPU
11
LiveRadar3D VNC

• For users who
• Dont want to download a client
• Dont have the resources such as GPU
• Uses VirtualGL/TurboVNC to enable remote 3D
  visualization
• A convenient way to do advanced interactive and
  collaborative visualization remotely
• Browser accessible
• Similar to LiveRadar3D Desktop in functionality
• Allows full capability available to expert users
• Disadvantage Needs server farm to scale

12
3rd Party Applications Access

• Our architecture is modular and supports
  fine-grained service access points
• Enables developing interesting 3rd party
  applications such as
• Weather prediction application can connect to
  data access interface
• Custom 3D visualizations can be built on
  pre-processed volumes

13
Services and Backend Data Architecture

• In our earlier work, we presented a system that
• Accesses NEXRAD II data
• Processes it into render-able 3D volumes using
  Teragrid
• Visualizes using Texture-based volume rendering
• Disadvantages
• On-demand processing gt Slow for large amounts of
  data
• Single access point targeted at expert users
• Extensions
• Multiple services and access points
• Preprocessing data to improve response time and
  scalability
• Volume Caching for easy access and reuse

14
Backend Data Flow Diagram
15
Parallel Data Pre-Processing

• Partial 3D volumes
• efficient data structure using Hash-maps
• spatial/temporal independence property gt
  parallel generation
• can be quickly merged to form full 3D volumes
  that can be rendered
• two orders of magnitude smaller than actual data
  and much smaller than generating full 3D volumes
• Generation of partial volumes on Teragrid
• Monitors the arrival of new data and
  pre-processes them and stores the partial-volumes
• SRB archives the past-year partial volumes

16
The fully Interactive 3D Viz. Tool for NEXRAD II
Data
17
Visualization Images Generated Using Our Tool
18
Hurricane Ike Images (September 14 2008 )
19
Related Work

• NEXRAD Data Visualization
• Integrated Data Viewer (IDV) by Unidata
• National Climate Data Center Java NEXRAD Viewer
  and Data Exporter
• CRAFT Interactive Radar Analysis System ( Java
  Viewer )
• LEAD Project ( Gateway )
• Remote Visualization
• NanoHub very small data
• Insley et al. Parallel RayTracing slow for 3D
  interactions
• Web Gadgets
• Weather.com/Floen.com simple 2D visualization
  and animation of radar data only for one station
  or whole nation

20
Summary and Future Work

• Summary of our contribution
• A hierarchical and user-oriented design
• rich and easy access to NEXRAD II data for a
  broad range of users.
• Improved response time and scalability
• parallel data pre-processing and partial volume
  caching
• An integrated end-to-end backend system
• radar data retrieval, pre-processing, remote
  rendering and 3D data visualization.
• Future Work
• Developing optimized data structures by
  exploiting the spatial and temporal
  characteristics of the data

21

• Thank you for your attention!
• Q A