Managing Storm Simulation Workflows using LEAD Gateway

1
Managing Storm Simulation Workflows using LEAD
Gateway
Suresh Marru Marcus Christie Indiana University
TeraGrid 06 June 15, 2006
2
Overview

• LEAD Introduction
• Complexity of Storm Simulation workflows
• Teragrid Resource Usage Challenges
• LEAD Gateway tools enabling to run weather
  simulations on teragrid
• LEAD Portal
• Application Service Model
• Service-oriented workflow system
• GPEL Workflow Engine

3
(No Transcript)
4
The LEAD Goal

• Provide the IT necessary to allow
• People (scientists, students, operational
  practitioners)
• and
• Technologies (models, sensors, data mining)
• TO INTERACT WITH WEATHER

5
Example Adapting Weather Prediction to
Observational Sources Using Dynamic Adaptivity
6
Dynamic Workflow in LEAD
9
Terrain data files
NAM, RUC, GFS data
3D Model Data Interpolator (lateral Boundary
Conditions)
3
1
3D Model Data Interpolator (Initial Boundary
Conditions)
Terrain Preprocessor
Surface data, upper air mesonet data, wind
profiler
2
15
11
WRF Static Preprocessor
ARPS to WRF Data Interpolator
IDV Bundle
4
Surface, Terrestrial data files
88D Radar Remapper
7
WRF
ADAS
WRF
10
Run Once per forecast Region
WRF
12
WRF
Radar data (level II)
ARPS Ensemble Generator
Radar data (level III)
13
NIDS Radar Remapper
5
8
WRF to ARPS Data Interpolator
ADAM
Satellite data
6
Visualization on users request
Satellite Data Remapper
Repeated for periodically for new data
14
ARPS Plotting Program
Data mining looking for storm signature
Triggered if a storm is detected
Static data
Initialization
Forecast
Visualization
Real time data
Analysis
Data Mining
7
LEAD Strategy

• Service Oriented Architecture (SOA)
• Virtualize access to resources (data,
  computational resources, software)
• Web access to important tools
• Users can search for data, run codes using simple
  web form interface
• Java WebStart (one-click install) when more
  advanced interface needed
• Provide workflow tools
• Users log into portal to create new workflows
  which combines the execution of community codes

8
LEAD Architecture

• Desktop Applications
• IDV
• WRF Configuration GUI

User Interface
LEAD Portal
Crosscutting Services
Control
Visualization
Workflow
Education
Browse
Portlets
MyLEAD
Monitor
Query
Control
Ontology
Client Interface
Workflow Monitor
Application Resource Broker (Scheduler)
Stream Service
Control Service
Authorization
Workflow Services
Workflow Engine/Factories
Query Service
Ontology Service
Application Configuration Services
Configuration and Execution Services
Data Services
Host Environment
Execution Description
Authentication
Decoder/Resolver Service
Transcoder Service/ ESML
VO Catalog
Application Description
Application Host
Catalog Services
WRF, ADaM, IDV, ADAS
THREDDS
GPIR
Geo-Reference GUI
Monitoring
Resource Access Services
Scheduler
OPenDAP
Grid FTP
Generic Ingest Service
OGSA-DAI
RLS
SSH
LDM
GRAM
Notification

• Observations
• Streams
• Static
• Archived

Data Bases
Distributed Resources
Steerable Instruments
Specialized Applications
Computation
Storage
9
Applications to Web Services

• We want to bring community codes into
  web-services environment
• But, we dont want to have to change them
• Solution Wrap Command line applications into
  web services using a service tool kit

10
The Generic Service Toolkit

• Start with
• A Deployed Application
• A list of all needed input files
• A list of all generated output files.
• Write a AppService Document
• Upload this to the portal Factory generator in
  the portal.
• A new Factory is started for you.
• A portal client interface to the factory is also
  automatically generated.

Upload AppService Doc
https
Portal Server
Create and Launch factory
11
Workflow System GPEL

• Now that we have wrapped our codes with service
  interfaces, how do we orchestrate their
  execution?
• Use graphical workflow composition tools and a
  workflow engine

12
XBaya Graphical Composer

• Edit to intuitively create workflows by dropping
  components (web services) and connecting them
• Deploy Takes graphical representation of
  workflow (graph) and generates BPEL workflow and
  WSDL portType for public workflow service
  interface that describes workflow input
  parameters (template)
• Manage retrieve workflow templates (with graphs)
  and deploy new templates in your workspaces
  (manage your workflows)
• Monitor subscribes to event stream and shows
  graphically workflow progress

13
Composing Workflow In the Portal

• An interactive workflow composer (WebStart)
• Component database and workflow compiler is
  provided by the grid service
• which also provides the interface tool.
• MVC pattern.
• Composer allows
• Component selectionfrom library
• Drop and drag place-ment and connectionestablish
  ment
• Save and load graphfunctions.

14
WRF Forecast with ADAS Initialization
15
Make Your Grid Flow!

• Grid Process Execution Language
• Language uses BPEL4WS / WS-BPEL 2.0
• Upcoming OASIS standard supported by all major IT
  companies (IBM, Microsoft, Oracle, )
• Execution persistent, scalable, message
  oriented workflow engine
• Process draw workflow flow (graph) and transform
  it to BPEL for execution
• Grid flexible allocation of grid resources for
  each workflow instance (including on-demand
  service creation) by using GFac

16
GPEL Goals

• Main goal of our GPEL engine is to support
  adaptive and dynamic workflows that may take very
  long to complete
• Including runtime modifications of the execution
  paths such as modifying or adding new activities
• Easy to integrate with portal and grids
• Scalable, fault tolerant, and easy to monitor and
  run with load balancing,
• Compatible with Web Services standards

17
Executing the workflow
MyLEAD Metadata Catalog
Portal Experiment Builder
2. GPEL starts executing by locating application
services and sending them requests.
MyLEAD Agent
GPEL Engine
GFac Application Factory

• Launch XBayaas workflowlistener

XBaya Workflow Monitor
Notification Broker
Application Service
Application Service
Application Service
3. Services and GPEL send notifications
about workflow progress to the broker who logs
them with the MyLEAD agent and the Monitor.
18
Interoperability

• Standard web services
• Require asynchronous invocation (WS-Addressing
  1.0)
• Prototyped integration with other workflow
  systems
• Kepler, Taverna
• WS-Eventing with pub/sub system (WSMessenger
  implementation)

19
LEAD Portal

• Building web clients for managing workflows and
  data

20
Portal Users

• Need an Identity Certificate
• Using PURSe (Portal-Based User Registration
  Service)
• Authenticates user to capability based
  authorization framework

21
Getting an Identity Certificate

• Portal Account
• Identity Certificate
• MyLEAD Personal Data Repository

22
Capabilities

• User must have requisite capability tokens to run
  applications
• Capability tokens are digitally signed policy
  documents that contain assertions about who can
  do what

23
A Tale of Two Authorization Systems
Application Service
24
Data Tools
25
Data Tools
26
Workflow Tools
27
Workflow Tools
28
MyWorkspace

• Produces PNG file, with link to view it

29
Visualizations
30
Todays case Fixed, Coarse Model Grids
31
Future Work Nested Models
20 km
32
Future Work nested models
20 km
33
Interacting with Steerable NETRAD CASA Doppler
Radars.
34
Example Adapting Weather Prediction to
Observational Sources Using Dynamic Adaptivity
35
Conclusion

• A Science Gateway Portal providing necessary
  tools and technologies to access and utilize
  Teragrid and other grid accessible HPC resources
  seamlessly.

36
Future Directions

• Have the community users try the system and
  provide feed back
• Provide functionality to various class of users
  ranging from 6th grade students to advanced
  scientists
• Schedulers Integrate with VGrADS
• and SPRUCE urgent computing
• Research on Dynamic and Adaptive workflows