LEAD Tools THRUST Group

1
LEAD Tools THRUST Group
Slides and Background Materialsfor LEAD All
Hands Meeting June 2004 Presentation at the
meeting to be given by Sara Graves and Mohan
Ramamurthy
2
Need more than high level concepts?
3
Main Issues for Tools Thrust

• Settling on the overall system architecture
  (driven by scenarios) with emphasis on component
  dependencies
• Agreeing on a priority order and schedule for
  implementing the tools at each testbed
• Defining input/output interfaces among the tools
  in order to develop an interoperable set
• Establishing a mechanism for turning tools into
  web/grid services and then creating those
  services
• Implementing the workflow orchestration that
  chains the tools together into useful end-to-end
  systems

4
Key Tools

• ADAM for data mining
• ADAS for data assimilation
• Decoders for format transformation
• IDV visualization
• LDM/IDD for real-time data delivery
• OPeNDAP, ESML/OPeNDAP, ADDE data services
• THREDDS catalog generation and services
• WRF model
• gridFTP for file transfer

5
Comprehensive Mining Testbed ComponentsDynamic
event detection and response using LEAD testbed
technologies
Mesocyclone Detection
Event
Notify
3
email
Yes
Mining
Store Events In Data Pool
Application Data Access
Local NWS radar
5
Data File Locations and other metadata
6
Online Data Pool
DODS Server
THREDDS Catalog
IDV
generate
4
1
8
ESML
Other data
LDM
METAR
NEXRAD Cache
Data Pool Order and FTP Access
Decoded
2
ESML
NCEPModels
7
Data Access
WCS
Maps
WMS
Subsetted Data
Map Access
OGC Viewer
6
Mining Testbed Component Explanations

• Test beds receive LEAD data sets via LDM from
  upstream sites.
• The LDM nodes are configured to decode and cache
  the incoming data streams to the sitess online
  data archive.
• Mesocyclone detection and other ADaM mining
  algorithms are run on the NEXRAD incoming stream
  in near-real time in an effort to quickly target
  developing weather situations. A notification
  service is triggered by the event detection to
  alert listeners and possibly automatically send
  them subscribed data sets based on the ongoing
  event.
• The UAH Data Pool provides online access to
  global passive microwave data and LEAD regional
  data. The Data Pool provides OpenDAP (DODS), FTP
  and HTTP access to the data sets.
• Applications will primarily utilize OpenDAP
  protocols for data access.

• THREDDS catalogs are generated daily (or more
  often if necessary) to provide metadata to
  applications on the location of data sets.
• The Data Visualization and Access workflow
  contains OpenGIS-compliant data services that
  provide public data access and visualization
  capabilities to data pool contents.
• Users will receive notification of detected
  events and will be able to access the data
  through usual FTP/HTTP, applications such as IDV
  using OpenDAP protocols, and through OpenGIS
  protocols such as WMS and WCS.

7
Phased Implementation NeededBut with concurrent
work on some aspects of each phase

• Define high level, comprehensive architecture(s)
  such as the mining diagram shown earlier
• Start by building minimal end-to-end system (show
  later)
• Define interfaces among tools to facilitate
  parallel work
• Add remaining key tools
• Convert tools to web services
• Develop workflow orchestration for existing tools
• Integrate tools and services into LEAD portal and
  MyLEAD
• Incorporate web services and workflow into GRID
  framework

8
Initial End-to-End SystemWalk before we try to
run
7
WRFoutput to IDVinput Decoder
Eta to WRFinput Decoder
IDV
LDM
WRF Regional Model
9
Minimal System Component Explanations

• Initial (3 month) Components
• Test beds receive LEAD NCEP ETA data set via LDM.
• The LDM nodes are configured to decode and cache
  the incoming data streams into a form suitable
  for initializing the WRF regional model and store
  the decoded datasets into the sitess online data
  archive.
• WRF regional model reads the decoded data from
  the testbed archive and runs regional model.
• WRF regional model output is stored into testbed
  data store.
• Applications will initially utilize OpenDAP
  protocols for data access.
• THREDDS catalogs are generated for all datasets
  on an ongoing basis
• Initial visualization of datasets is via the IDV.

• Next Steps (1 year)
• Building on experience with minimal system,
  define interfaces for additional tools and
  datasets
• Incorporate ADDE and ESML/OPeNDAP for serving
  other data types
• Replace NCEP ETA initialization for WRF with
  ADAS-based true data assimilation system.
• Utilize ADAM data mining for model trigger and
  guidance mechanism (as shown in the Comprehensive
  diagram.
• Develop LEAD-specific visualization facilities
  for all data types.
• Construct orchestrated workflow for minimal
  system components
• Select several tools and develop web prototype
  web services versions

10
Advantages of Minimalist Approach

• Gets end-to-end system running soon (can
  demonstrate now with Workstation ETA in place of
  WRF)
• Clarifies remaining tool interface work
• Provides examples of working tools for
  conversion to Web services
• Establishes working testbed of tools for workflow
  orchestration group
• Gives us working system to demonstrate and
  experiment with
• Allows each group to make progress in own area of
  expertise in parallel

11
Unidata Focus

• Tailor IDV for LEAD
• Replace Unidata Workstation ETA with WRF
• Modify current ETA decoder to create WRF
  initialization data
• Create WRF output decoder for CF conventions
• Support other testbeds implementing LDM,
  decoders, THREDDS, OPeNDAP, and ADDE
• Work with OU on ADAS for initialization
• Work with UAH to incorporate ADAM and
  ESML/OPeNDAP
• Work with CS experts on conversion of tools to
  orchestrated services

12
Next Steps

• Discuss and refine the suggested approach to
  tools planning
• Agree on overall plan
• Develop very specific plans and commitments for
  each set of tools developers for 3-month time
  frame
• Develop specific plans for 1-year time frame
• Get realistic tools implementation commitments
  from testbed sites
• At 3-month intervals, each tools developer and
  testbed site reports on progress and revised
  goals for next 3 months
• Based on progress reports and revised goals for
  each individual group, publish a revised overall
  plan annually