A e-Science Workflow Bus: concept and research issues - PowerPoint PPT Presentation

About This Presentation
Title:

A e-Science Workflow Bus: concept and research issues

Description:

Z. Zhao et al, VLWF-Bus: a workflow bus for multi domain e-Science applications, ... Scientific workflow systems: a new GUISE of Problem Solving Environments ... – PowerPoint PPT presentation

Number of Views:27
Avg rating:3.0/5.0
Slides: 22
Provided by: zhim
Category:

less

Transcript and Presenter's Notes

Title: A e-Science Workflow Bus: concept and research issues


1
A e-Science Workflow Bus concept and research
issues
  • Dr Zhiming Zhao
  • Faculty of Science, University of Amsterdam
  • VL-e SP 2.5

Z. Zhao et al, VLWF-Bus a workflow bus for multi
domain e-Science applications, to appear IEEE
Intl Conf. on e-Science and Grid computing,
Amsterdam, 2006
2
Outline
  • Background
  • A workflow bus and generic e-Science framework
  • Prototype and experiment results
  • Discussion
  • Conclusions
  • On going research
  • References

3
Scientific experiments and support systems
Prototype on small data scale.

Matlab
Define goal
Prototype the algorithm
Computing (Test with small data)
Vis./Int. (Validation)
Ptolemy
Refine
Experiment on full data scale.
Finding Dissemination
Apply to full size data
Data analysis
4
Scientific workflow systems a new GUISE of
Problem Solving Environments
  • In our view, a SWMS at least implements
  • A model for describing workflows
  • An engine for executing/managing workflows
  • Different levels of support for a user to
    compose, execute and control a workflow.

Workflow (based on certain model)
Composition
A SWMS
User support
Engine level control
Engine
Resource level control
resources
5
Diversity in SWMSs
  • Taverna
  • Web services based language Scufl
  • FreeFluo engine
  • Graphical viz of workflow
  • Triana
  • Components
  • Task graph
  • Data/control flow
  • Kepler
  • Actor,director
  • MoML
  • Execution models
  • Pegasus
  • Based on DAGMan
  • VDL
  • DAG

  • DAGMan
  • Computing tasks
  • DAG

6
Research mission
  • Effectively reuse existing workflow management
    systems, and provide a generic e-Science
    framework for different application domains.
  • A generic framework can
  • Improve the reuse of workflow components and the
    workflows for different experiments
  • Reduce the learning cost for different systems
  • Allow application users to work on a consistent
    environment when underlying infrastructure
    changed
  • Promotes knowledge transfer between scientists
    and between domains

7
Options
  • Abstract approach
  • Extend approach
  • Aggregate approach

SWMS1
SWMS2
SWMS3
SWMSG
SWMS1
SWMS2
SWMS3
SWMSG
SWMS1
SWMS2
SWMS3
SWMS1
SWMS2
SWMS3
SWMS G
8
Why we choose an aggregation approach?
  • Abstract approach
  • Build a perfect system
  • Difficult to find a set of systems cover all the
    required generic functionality it requires
    re-implementation of existing things
  • Extend approach
  • Incrementally development
  • The solution depends on a specific system
  • Aggregate approach
  • Maximize the reuse of the existing workflow
    systems
  • Has to handle interoperability issues provide
    customized interface existing workflow system

9
An aggregation approach a bus architecture
  • What is a bus
  • Interface specification
  • Service and management
  • Why a bus
  • Transparent and loose coupling
  • Benefit from different levels of existing
    bus/middleware e.g., object/component/agent
  • Give freedom to plug new functional components
  • Can be promoted as an integration core for
    different levels of other e-Science services
    data provenance, semantic discovery, security
    control, etc.

10
(No Transcript)
11
Architecture
  • Terminology
  • The execution of a workflow is one study, and the
    execution of a sub-workflow is called a
    sub-study, or a scenario
  • Basic idea
  • Study manager schedules sub workflows
  • Scenario managers interface third party workflow
    engines and reacts to the Study manager
  • User interface for composition, monitoring, and
    execution control.

12
Requirements
  • A distributed framework for study and scenario
    managers
  • Data input/output of a sub-workflow, description
    of the workflow can be described and recognized
    by study and scenario managers
  • Handle the user interactions which are needed in
    scenarios
  • The engine can be decoupled from a SWMS
  • Be fault tolerant

13
Considerations
  • From integration point view study and scenario
    managers can be coupled by
  • Web services
  • Object oriented middleware (CORBA, HLA, etc.)
  • Agent based middleware
  • Or an existing workflow system (Kepler, Taverna,
    Triana or others)
  • The description of meta workflow
  • The execution model of the meta workflow
  • Interface to include new services data
    provenance, knowledge infrastructure, and others.

14
A JADE/Ptolemy based prototype
Scenario Mnger
Scenario Mnger
Scenario Mnger
Study Mnger
Ptolemy
Actor
Actor
Director
Actor
User interface
15
Experiment results
  • Message delay

16
Overhead
1020 performance improvement.
17
Scalability
18
Discussion
  • Challenges in supporting scientific workflows
  • Requirements on domain specific experiments
  • Generic workflow support and domain specific
    applications
  • Existing workflow management systems are diverse
    in functionality, design and user support
  • Related work
  • Interoperability among workflow systems (sister
    Link project)
  • Resource level e.g., Kepler invokes Tavernas
    resources

19
Research focuses
  • Workflow interoperability
  • Language, component, engine and other levels
    interoperability between workflow systems
  • Abstract functionality at the bus interface
  • Knowledge infrastructure for workflow bus
  • Interface the workflow bus to the knowledge
    infrastructure of e-Science environment, e.g.,
    Data/Information/Knowledge (semantic) services
    and tools
  • Composition
  • Composing meta workflow using services,
    components or workflows from different SWMS
  • Execution
  • Using different scheduling and execution model
    based on the state of e-Science infrastructure
  • Human in the loop
  • Human interaction at both sub and meta workflow
    levels
  • Data provenance
  • Record/replay/mining data from sub and meta
    workflows
  • Integrate provenance with the knowledge backbone
    via workflow bus

20
Conclusions
  • A workflow bus is a feasible approach to realize
    generic e-Science framework
  • Multi agent technology provides a distributed
    environment for decomposing and encapsulating
    control intelligence
  • Ptolemy II provides different computing paradigms
    which give user freedom to execute workflows

21
  • References
  • Z. Zhao A. Belloum H. Yakali P.M.A. Sloot and
    L.O. Hertzberger Dynamic Workflow in a Grid
    Enabled Problem Solving Environment, in
    Proceedings of the 5th International Conference
    on Computer and Information Technology (CIT2005),
    pp. 339-345 . IEEE Computer Society Press,
    Shanghai, China, September 2005.
  • Z. Zhao A. Belloum A. Wibisono F. Terpstra
    P.T. de Boer P.M.A. Sloot and L.O. Hertzberger
    Scientific workflow management between
    generality and applicability, in Proceedings of
    the International Workshop on Grid and
    Peer-to-Peer based Workflows in conjunction with
    the 5th International Conference on Quality
    Software, pp. 357-364. IEEE Computer Society
    Press, Melbourne, Australia , September 19th-21st
    2005.
  • Z. Zhao A. Belloum P.M.A. Sloot and L.O.
    Hertzberger Agent Technology and Generic
    Workflow Management in an e-Science Environment,
    in Hai Zhuge and G.C. Fox, editors, Grid and
    Cooperative Computing - GCC 2005 4th
    International Conference, Beijing, China, in
    series Lecture Notes in Computer Science, vol.
    3795, pp. 480-485. Springer, November 2005. ISBN
    3-540-30510-6. (DOI 10.1007/11590354_61)
  • Z. Zhao A. Belloum P.M.A. Sloot and L.O.
    Hertzberger Agent technology and scientific
    workflow management in an e-Science environment,
    in Proceedings of the 17th IEEE International
    conference on Tools with Artificial Intelligence
    (ICTAI05), pp. 19-23. IEEE Computer Society
    Press, Hongkong, China, November 14th-16th 2005.
  • Z. Zhao Suresh Booms A. Belloum P.M.A. Sloot
    and L.O. Hertzberger VLWF-Bus a workflow bus
    for e-Science applications, in Proceedings of the
    2nd IEEE e-Science and Grid computing, IEEE
    Computer Amsterdam, December 46 2006.
Write a Comment
User Comments (0)
About PowerShow.com