A Workflow Bus for e-Science Applications

1
A Workflow Bus for e-Science Applications

• Dr Zhiming Zhao
• Faculty of Science, University of Amsterdam
• VL-e SP 2.5

2
Outline

• Introduction
• Scientific workflow in e-Science
• Scientific workflow management systems
• A workflow bus and generic e-Science framework
• Basic idea
• Related work
• Our approach
• Prototype
• State
• Discussion
• Conclusions
• Future work

3
Scientific experiments and e-Science

• Required services
• Managing data, computing tasks, information, and
  meta data
• Managing experiment routines (scientific workflow)

4
Scientific Workflows in e-Science
Experiment processes

• Workflows for
• Different levels
• Different function
• Different goals

Temporal workflows for administration, e.g., AAA,
and other issues.
Abstract workflows
Executable (concrete workflows)
5
Scientific workflow management systems (SWMS)

• A SWMS is able to
• Automate experiment routines
• Rapid prototype experimental computing systems
• Hide integration details between resources
• Manage experiment lifecycle
• A number of SWMSs have been developed
• Managing direct computing/data tasks CondorG
• Managing experiment tasks at an abstract level
  Pegasus
• Extending a Problem Solving Environment Kepler
• Coupling different domains software resources
  Taverna

6
Insight a Scientific Workflow Management System

• In our view, a SWMS at least implements
• A model for describing workflows
• An engine for executing/managing workflows
• Different levels of support for a user to
  compose, execute and control a workflow.

Workflow (based on certain model)
Composition
A SWMS
User support
Engine level control
Engine
Resource level control
resources
7
Diversity in SWMS

• Originated from different domains high energy
  physics, bio-informatics
• Different workflow model and description
  language DAG, MoML
• Different execution models (engines) based on
  Data flow, Continuous time, process net.
• Different types of user support composition,
  dynamic support
• Different interface for coupling resources web
  services, Ptolemy middleware, and CORBA

8
Diversity in SWMS

• Taverna
• Web services based language Scufl
• FreeFluo engine
• Graphical viz of workflow

Triana
Kepler
9
Why generic framework?

• A generic workflow system a SWMS can support
  applications from different domains.
• Improve the reuse of workflow components and the
  workflows for different experiments
• Reduce the learning cost for different systems
• Allow application users to work on a consistent
  environment when underlying infrastructure changed

10
Research context and missions

• Generic framework for different domains
• Different levels of abstraction
• Workflow services
• Short term
• Long term

11
Possible options

• Abstract approach
• Extend approach
• Aggregate approach

S1
S2
S3
SG
S1
S2
S3
SG
S1
S2
S3
S1
S2
S3
SG
12
Why we choose an aggregation approach?

• Abstract approach
• Build a perfect system
• Difficult to find a set of systems cover all the
  required generic functionality it requires
  re-implementation of existing things
• Extend approach
• Incrementally development
• The solution depends on that system
• Aggregate approach
• Maximize the reuse of the existing workflow
  systems
• Has to handle interoperability issues provide
  customized interface existing workflow system

13
A workflow bus paradigm
Workflow
Sub workflow 3
Sub workflow 1
Sub workflow 2
Triana
Kepler
Taverna
Workflow bus
14
What is a workflow bus?

• It is a special workflow engine for executing
  meta workflows, in which sub workflows will be
  executed by different other engines.
• It allows a scientist to
• design a workflow using resources from different
  workflow systems
• work on a SWMS which he is familiar with, but to
  exchange information with the other SWMSs
  transparently

15
Related work

• Interoperability among workflow systems (sister
  Link project)
• Resource level Kepler invokes Tavernas
  resources

16
An agent based prototype

• Why choosing agent technologies
• Decompose and encapsulate control intelligence
• Ontology enabled communication
• The basic idea
• Handle workflow/sub-workflow using different
  levels agents
• A study manager
• A number of scenario mangers
• Use FIPA compliant agent implementation JADE
• Use a Ptolemy as the basic framework for
  supporting workflow execution

17
Architecture
Schedule sub-workflows

• The execution of a workflow is one study, and the
  execution of a subworkflow is called a sub-study,
  or a scenario
• At one level, only one study manager active
• Study manager an agent for scheduling sub
  workflows, and coordinating their executions via
  Scenario Managers
• Scenario managers agents for interfacing third
  party workflow engines and for reacting to Study
  manager

18
User front end
Scenario Mnger
Scenario Mnger
Scenario Mnger
Study Mnger
Ptolemy
Actor
Actor
Director
Actor
19
How it works

• In user front end a user defines meta workflow,
  each actor represents a sub workflow
• At runtime, each actor initiates a scenario
  agent, and passes the workflow description to the
  scenario manager
• A scenario manager controls an engine and execute
  the sub-workflow

20
Prototype

• Ptolemy II 5.0.1
• JADE
• Tested with Taverna, Triana engines

21
Experiment results

• Communication cost

22
Cont.

• Scalability

23
Cont.

• Overhead

24
Applications

• Use case 1
• A user has workflow in Taverna
• Some functionality is missing in Taverna but can
  be provided by Triana
• He can develop the workflow in two systems, and
  run it via the workflow bus
• Use case 2
• A user wants to execute a Taverna or Triana
  workflow in multiple instances with different
  input data

25
States

• Taverna, Triana have been tested
• MoML is used for describing meta workflows

26
Discussion

• Challenges in scientific workflow support
• Domain specific application specific
  requirements on experiment scenarios
• Generic workflow support and domain specific
  applications
• Existing workflow management systems are diverse
  in functionality, design and user support

27
Conclusions

• A workflow bus is practically feasible approach
  to realize generic e-Science framework
• Multi agent technology provide a distributed
  environment for decompose and encapsulate control
  intelligence
• Ptolemy II provides different computing paradigms
  which give user freedom to execute workflows

28
Future work

• Working on developing a scenario manager for
  Kepler engine
• Synchronized data flow is currently used. More
  computing modes will be evaluated.
• Provenance for workflow bus

29
References

• Z. Zhao A. Belloum H. Yakali P.M.A. Sloot and
  L.O. Hertzberger Dynamic Workflow in a Grid
  Enabled Problem Solving Environment, in
  Proceedings of the 5th International Conference
  on Computer and Information Technology (CIT2005),
  pp. 339-345 . IEEE Computer Society Press,
  Shanghai, China, September 2005.
• Z. Zhao A. Belloum A. Wibisono F. Terpstra
  P.T. de Boer P.M.A. Sloot and L.O. Hertzberger
  Scientific workflow management between
  generality and applicability, in Proceedings of
  the International Workshop on Grid and
  Peer-to-Peer based Workflows in conjunction with
  the 5th International Conference on Quality
  Software, pp. 357-364. IEEE Computer Society
  Press, Melbourne, Australia , September 19th-21st
  2005.
• Z. Zhao A. Belloum P.M.A. Sloot and L.O.
  Hertzberger Agent Technology and Generic
  Workflow Management in an e-Science Environment,
  in Hai Zhuge and G.C. Fox, editors, Grid and
  Cooperative Computing - GCC 2005 4th
  International Conference, Beijing, China, in
  series Lecture Notes in Computer Science, vol.
  3795, pp. 480-485. Springer, November 2005. ISBN
  3-540-30510-6. (DOI 10.1007/11590354_61)
• Z. Zhao A. Belloum P.M.A. Sloot and L.O.
  Hertzberger Agent technology and scientific
  workflow management in an e-Science environment,
  in Proceedings of the 17th IEEE International
  conference on Tools with Artificial Intelligence
  (ICTAI05), pp. 19-23. IEEE Computer Society
  Press, Hongkong, China, November 14th-16th 2005.

30
A workflow bus paradigm
A scientific workflow
Sub workflow 2
Sub workflow 3
Sub workflow 1
SWMS 1
SWMS 2
SWMS 3
Workflow bus
31
Architecture
Workflow bus prototype
Study manager
Scenario manager
Taverna
Triana
32
BPEL an introductory(some slides are from
Oracle)

• Z. Zhao

33
A big picture

• Using IT technologies (software and hardware) to
  enhance (automate) business processes!

Issues Standard organizations
Developing and managing enterprise applications and services Integration enterprise applications Business process modeling and description. OMG object management group W3C world wide web consortium WfMC workflow management consortium OASIS Organization of Advanced structured information systems.
34
Business process

• What is a business process
• From business point view interaction scenarios
  among partners
• From WS point of view contracts among service
  providers
• Business processes can be described in two ways.
• Executable business processes model actual
  behavior of a participant in a business
  interaction.
• Business protocols, in contrast, use process
  descriptions that specify the mutually visible
  message exchange behavior of each of the parties
  involved in the protocol, without revealing their
  internal behavior.
• Implementing business processes using Web
  services!

35
Web Services Meet Business Processes
Web Service 1
Web Service 4
Web Service 2
Web Service 5
Web Service 3
Web Service n
36
Example Problem Space
Credit Service
Consolidate Results
PO Service
Client
Inventory Service
37
Sample Business Process Purchase Order
Sample Purchase Order
Purchase Order Request
Business A
Business B
Purchase Order Acknowledgement
Purchase Order Response
38
Business Process Challenges

• Coordinate asynchronous communication between
  services
• Correlate message exchanges between parties
• Implement parallel processing of activities
• . . .

• Manipulate/transform data between partner
  interactions
• Support for long running business transactions
  and activities
• Provide consistent exception handling
• . . .

39
Orchestration vs Choreography

• Orchestration
• An executable business process describing a flow
  from the perspective and under control of a
  single endpoint (commonly Workflow)
• Choreography
• The observable public exchange of messages, rules
  of interaction and agreements between two or more
  business process endpoints

40
From a Choreography Perspective
Public Process
Business A
Business B
PO Request
PO Acknowledgement
PO Response
Choreography The observable public exchange of
messages
41
From an Orchestration Perspective
Private Process
Business A BPEL Workflow
Send PO
Transform
PO Request
From ERP
Receive PO Ack
PO Acknowledgement
To ERP
Receive PO Response
Transform
PO Response
Orchestration A private executable business
process
42
Orchestration and Choreography Together
Generate BPEL Template
Generate BPEL Template
Business B BPEL Workflow
Business A BPEL Workflow
Receive PO
Transform
Send PO
Transform
PO Request
Send PO Ack
Receive PO Ack
PO Acknowledgement
Receive PO Response
Transform
Receive PO Response
Transform
PO Response
Two BPEL workflow templates reflecting a business
agreement
43
Recent History of Business Process Standards
BPSS (Business Process Specification Schema
) (ebXML)
BPML (Business process modeling
language) (Intallio et al)
WSCI (Web service choreography interface) (Sun et
al)
WS-Choreography (W3C)
2000/05
2001/03
2001/05
2001/06
2002/03
2002/06
2003/01
2003/04
2002/08
XLang (BizTalk) (Microsoft)
WSFL (Web Service Flow Language) (IBM)
BPEL4WS 1.0 (Business process Execution language
4 Web service) (IBM, Microsoft)
BPEL4WS 1.1(OASIS)
WSCL (Web service Conversation language) (HP)
44
Standards

• Business Process Execution Language (BPEL).
• BPEL 1.1 2003
• BPEL is promoted as a standard WSBPEL through
  Organization for the Advancement of Structured
  Information Standard (OASIS)
• BPEL was developed through the combination of
  IBM's WSFL and Microsoft's XLANG.
• Also known as BPEL4WS (the original proposal) and
  WSBPEL (the OASIS standard being developed).
• Business Process Specification Schema (BPSS) from
  ebXML.
• Business Process Management Initiative BPMI from
  OMG.
• Web Services Choreography Description Language
  (WS-CDL) from W3C focuses on describing
  peer-to-peer collaborations between Web service
  participants by defining their behaviour from a
  global viewpoint.
• XML Process Definition Language (XPDL) from WfMC
• Process Specification Language (PSL).

45
BPEL4WS

• BPEL4WS provides a language for the formal
  specification of business processes and business
  interaction protocols.
• It defines a new web service by composing a set
  of existing services
• Describe a business process
• Implement a process
• It is positioned to become the Web services
  standard for composition.

46
Business Process Execution Language for Web
Services

• Version 1.0 released by IBM, Microsoft and BEA in
  August 2002
• Accompanied by WS-Coordination, WS-Transaction
  which remain unsubmitted to standards bodies
• Version 1.1 submitted to OASIS April 2003
• XML language for describing business processes
  based on Web services
• Convergence of XLANG (Microsoft) and WSFL (IBM)
• Unprecedented industry consensus
• IBM, Microsoft, Oracle, Sun, BEA, SAP, Siebel

47
Standards Building Blocks of BPEL
BusinessProcesses
Transactions
Quality ofService
Coordination
WS-Security
WS-Reliability
Context
Management
UDDI
Discovery
Description
WSDL
Description
SOAP
Message
XML
HTTP,IIOP, JMS, SMTP
Transport
48
BPEL Depends on WSDL and WSDL Extensions
Service Implementation Definition
Service
Port
Service Interface Definition
Binding
Port types define Operations
Message
Type
49
BPEL Scenario Structure
ltprocessgt lt! Definition and roles of process
participants --gt ltpartnerLinksgt ...
lt/partnerLinksgt lt!- Data/state used within the
process --gt ltvariablesgt ... lt/variablesgt
lt!- Properties that enable conversations --gt
ltcorrelationSetsgt ... lt/correlationSetsgt lt!-
Exception handling --gt ltfaultHandlersgt ...
lt/faultHandlersgt lt!- Error recovery undoing
actions --gt ltcompensationHandlersgt ...
lt/compensationHandlersgt lt!- Concurrent events
with process itself --gt lteventHandlersgt ...
lt/eventHandlersgt lt!- Business process flow
--gt (activities) lt/processgt
50
BPEL Activities

• Primitive Activities
• ltinvokegt
• ltreceivegt
• ltassigngt
• ltreplygt
• ltthrowgt
• ltterminategt
• ltwaitgt

• Structured Activities
• ltsequencegt
• ltswitchgt
• ltpickgt
• ltflowgt
• ltlinkgt
• ltwhilegt
• ltscopegt

51
Partners

• Declare the Web services and roles used by the
  process
• Tied to WSDL of the process itself and the
  participating Web services by service link types

Partner 1(the process)
Partner 2
Partner 3
InventoryService
CreditService
PurchaseService
52
Partners in BPEL
BPEL
ltpartnerLinksgt ltpartnerLink namecustomer"
serviceLinkTypelnspurchasePLT
myRolepurchaseService/gt ltpartnerLink
nameinventoryChecker serviceLinkTypelnsinven
toryPLT myRoleinventoryRequestor
partnerRoleinventoryService/gt ltpartnerLink
namecreditChecker serviceLinkTypelnscreditPL
T myRolecreditRequestor
partnerRolecreditService/gt lt/partnerLinksgt
Purchase Process WSDL
Purchase Process PortType
ltpltpartnerLinkType namepurchasePLTgt
ltpltrole namepurchaseServicegt ltpltportType
nametnspurchasePT/gt lt/pltrolegt lt/pltpartner
LinkTypegt
ltportType namepurchasePTgt ltoperation
name"sendPurchase"gt lt/operationgt lt/portTypegt
53
Variables

• Messages sent and received from partners
• Persisted for long running interactions
• Defined in WSDL types and messages

Process
CustomerService
ltvariablegt
ltactivitygt
ltactivitygt
CustomerService
ltvariablegt
Persist/Retrieve
Persist/Retrieve
Persist/Retrieve
Persist
54
Variables in BPEL
BPEL
ltvariablesgt ltvariable namePO
messageTypelnsPOMessage/gt ltvariable
nameInvoice messageTypelnsInvMessage/gt
ltvariable namePOFault messageTypelnsorderFau
ltType/gt lt/variablesgt
Purchase Process WSDL
ltmessage namePOMessagegt ltpart
namecustomerInfo typesnscustomerInfo/gt
ltpart namepurchaseOrder typesnspurchaseOrder
/gt lt/messagegt ltmessage name"InvMessage"gt
ltpart nameIVC typesnsInvoice/gt lt/messagegt lt
message nameorderFaultTypegt ltpart
nameproblemInfo typexsdstring/gt lt/messagegt
55
How is Data Manipulation Done?

• Using ltassigngt and ltcopygt, data can be copied and
  manipulated between variables
• ltcopygt supports XPath queries to sub-select data

ltassigngt ltcopygt ltfrom variable"PO"
part"customerInfo"/gt ltto variablecreditReques
t part"customerInfo"/gt lt/copygt lt/assigngt
56
Simple Activities

• Receive
• Wait for a partner inbound message
• Can be the instantiator of the business process
• Reply
• Synchronous response to a receive activity
• Response to the inbound receive from a partner
• Invoke
• Issue a request synchronously or asynchronously
• Pick
• Specify an inbound set of messages
• Can be the instantiator of the business process
• Activity completes when one of the messages
  arrives

57
Simple Activities Combined with Structured
Activities
ltsequencegt
Receive ltPOgt
ltflowgt
Invoke ltInventoryServicegt
Invoke ltCreditServicegt
lt/sequencegt
Reply ltInvoicegt
58
Sample Activities in BPEL
ltsequencegt ltreceive partnerLinkcustomer
portTypelnspurchaseOrderPT"
operationsendPurchaseOrder variablePO
createInstance"yes" /gt ltflowgt ltinvoke
partnerLinkinventoryChecker portTypelnsinven
toryPT operation"checkINV"
inputVariable"inventoryRequest"
outputVariable"inventoryResponse" /gt ltinvoke
partnerLink"creditChecker" portTypelnscreditPT
" operation"checkCRED"
inputVariable"creditRequest"
outputVariable"creditResponse" /gt lt/flowgt ...
ltreply partnerLinkcustomer portTypelnspurcha
seOrderPT operationsendPurchaseOrder
variableinvoice"/gt lt/sequencegt
59
Links Control Flow
ltflowgt ltlinksgt ltlink name"XtoY"/gt ltlink
name"CtoD"/gt lt/linksgt ltsequence name"X"gt
ltsource linkName"XtoY"/gt ltinvoke name"A"
.../gt ltinvoke name"B" .../gt lt/sequencegt
ltsequence name"Y"gt lttarget linkName"XtoY"/gt
ltreceive name"C"/gt ltsource
linkName"CtoD"/gt lt/receivegt ltinvoke
name"E" .../gt lt/sequencegt ltinvoke
partnerLink"D"gt lttarget linkName"CtoD"/gt
lt/invokegt lt/flowgt
ltflowgt
ltXgt
ltlink XtoYgt
ltYgt
ltlink CtoDgt
lt/flowgt
60
Correlation
POCorrelationltPO_CustId 10gtltPO_OrdId
100gt
initiateyes
initiateyes
Customer
Seller

• SendPurchase
• ProcessPurchaseResponse

• AsynchPurchase
• AsynchPurchaseResponse

POResponseCorrelations ltPO_CustId
10gtltPO_OrdId 100gt ltInv_VendId
20gtltInv_InvId 200gt
initiateno patternout
initiateno
initiateyes patternout
initiateyes
61
Correlations in BPEL
ltcorrelationSetsgt ltcorrelationSet name"POCorr"
properties"corcustId corordId"/gt
ltcorrelationSet name"InvoiceCorr"
properties"corvendId corinvId"/gt lt/correlationS
etsgt ... ltreceive partnerLinkCustomer
portType"SPPurchaseOrderPT"
operation"AsynchPurchase" variable"PO"gt
ltcorrelationsgt ltcorrelation set"POCorr"
initiate"yes"gt lt/correlationsgt lt/receivegt
... ltinvoke partnerLinkCustomer
portType"SPCustomerPT" operationProcessPurch
aseResponse" inputVariable"POResponse"gt
ltcorrelationsgt ltcorrelation set"POCorr"
initiate"no" pattern"out"gt ltcorrelation
set"InvoiceCorr" initiate"yes" pattern"out"gt
lt/correlationsgt lt/invokegt ...
62
Scopes in BPEL
ltscopevariableAccessSerializable"yesno ...gt
ltvariablesgtlt/variablesgt ltcorrelationSetsgt?
... lt/correlationSetsgt ltfaultHandlersgtlt/faultHa
ndlersgt ltcompensationHandlergt? ...
lt/compensationHandlergt lteventHandlersgtlt/eventHa
ndlersgt (activities) lt/scopegt

• Provide a shared context for subset of activities
• Can contain
• fault handlers
• event handlers,
• compensation handler variables
• correlation sets
• Can serialize concurrentaccess to variables

63
Long Running Transactions and Compensation
CreditService
ltscopegt

• CheckCredit
• ChargeHoldFee
• CancelHoldFee

InventoryService

• ReserveInventory
• CancelReserveInv

lt/scopegt
64
Compensation Handlers in BPEL
ltscopegt ltcompensationHandlergt ltinvoke
partnerLink"Seller" portType"SPPurchasing"
operation"CancelPurchase"
inputVariable"getResponse"
outputVariable"getConfirmation"gt
ltcorrelationsgt ltcorrelation
set"PurchaseOrder" pattern"out"/gt
lt/correlationsgt lt/invokegt lt/compensationHandler
gt ltinvoke partnerLink"Seller"
portType"SPPurchasing"
operation"SyncPurchase"
inputVariable"sendPO"
outputVariable"getResponse"gt ltcorrelationsgt
ltcorrelation set"PurchaseOrder" initiateyes
pattern"out"/gt lt/correlationsgt
lt/invokegt lt/scopegt
65
Exception Handling in BPEL

• ltfaultHandlersgt catch exception
• Based on WSDL port defining fault
• ltfaultHandlersgt can perform activities upon
  invocation

ltfaultHandlersgt ltcatch faultName"lnscannotCompl
eteOrder" faultVariable"POFault"gt
ltreply partnerLink"customer"
portType"lnspurchaseOrderPT"
operation"sendPurchaseOrder"
variable"POFault" faultName"cannotCompl
eteOrder"/gt lt/catchgt lt/faultHandlersgt
66
Just Show Me How to Do it!

1. Compile
2. Package
3. Deploy

. . .
67
Tooling Requirements

• IDE build your Web services
• WSDL authoring model your interfaces
• Schema authoring model your messages
• Process modeling model your orchestration
• Packaging and deployment
• Debugging
• Monitoring
• Analyzing

68
What Happened to Java?

• JSR 207 - Process Definition Language for Java
• Make business processes natural for Java
  programmers

Servlet
EJB
Application Server
Based on JSR 207 Session at JavaOne 2003
69
What Happened to J2EE?

• JSR 208 Java Business Integration
• Make business processes a first class citizen in
  J2EE containers

JSR 208Machine SPI
Routing Engine
Transform Engine
BPEL Engine
. . .
Normalized Message Bus
Binding Framework
JSR 208Binding SPI
. . .
Based on JSR 208 Session at JavaOne 2003
70
Remember BPEL Does Not Solve World Hunger

• No data transformation
• No data translation (EDI, binary formats )
• No human workflow
• No trading partner agreements
• Silent on existing business protocols (ebXML,
  RosettaNet )
• Silent on non Web service interactions (e.g. java
  to java)
• . . .

71
But Remember People Are Trying to Solve World
Hunger

• W3C WS-Choreography
• Spec WS-Transaction
• Spec WS-Coordination
• Spec WS-Composite Application Framework
• OASIS WS-Reliability
• Spec ReliableMessaging
• Spec WS-Addressing
• OASIS WS-Security

72
Parting Thoughts

• Business process portability?
• Java/J2EE is portable across application servers
• BPEL is portable independent of Java
• Programming language in XML?
• Vendors, big and small, are busy building design
  times and modelers
• Is BPEL in 2003/2004 J2EE in 1998?
• Much missing but compelling foundation

73
Other BPEL based work

• BPEL4people included in new BPEL4WS 2.0.

74
Scientific workflow support in VL-e PoC
environment

• Zhiming Zhao
• The VL-e workflow team

75
Outline

• Scientific workflow and e-Science
• Workflow systems in the VL-e PoC environment
• Recommendations

76
Scientific Workflow Management Systems in an
e-Science environment
Domain specific Applications

• Functionalities
• Automating experiment routines
• Rapid prototyping of experimental computing
  systems
• Hiding integration details between resources
• Managing experiment lifecycle
• Cross different layers of middleware for
  managing
• Data
• Computing
• Information
• Knowledge

Knowledge
Information
e-Science environment
Computing tasks
Data management
Generic Grid middleware
Grid infrastructure
77
Workflow systems in PoC

• Four systems
• Kepler, Taverna, Triana, and VLAM
• Why these systems
• Based on a survey we did last year
• We learned
• None of the existing systems support all
  requirements of the VL-e use cases
• These three systems together cover most of the
  requirements,
• They provide friendly interface

78
Kepler

• Background
• The Kepler project is a cross-project
  collaboration among SEEK, SPA, Ptolemy II, GEON,
  etc.
• Current version is 1.0 alpha 8
• Java based, open source
• Usability
• Friendly GUI
• Web service compliant. It also has specialized
  architecture for resources and controller (called
  Actor and Director)
• Grid support
• Limited support for job farming and parameter
  sweeping

79
Taverna

• Background
• From myGrid, Sculf workbench V1.2
• Has a big Bio- user community
• Resources are WS based
• Usability
• It has a GUI (not very friendly)
• Flow modeling (data flow based, support limited
  control)
• Enactment and runtime control (service
  orchestration, but limited scheduling strategies
  for parameter sweeping and job farming.)
• The support for developing WS resources is not
  included

80
Triana

• Background
• Open source, in Grid lab project, at Cardiff
  university
• Version 3.2
• Usability
• Friendly user interface
• Grid support
• Not so stable

81
VLAM

• Background
• The installed version in Marix (part of the PoC)
• Usability
• GUI
• PFT and Topology
• Grid support,
• and WS support

82
Recommendations

• Web/Grid service compliant interface for resource
  invocation and integration
• Follow software engineering disciplines

83
Direction

• Short term solution
• Use these workflow systems in the use cases
• Learn lessons
• Abstract useful features
• Long term solution
• Make a new design based

84
Scientific workflow bus in an e-Science
environment

• Zhiming Zhao and Adam Belloum
• SP 2.5_at_VL-e

85
Scope and context a conceptual vision
Domain specific Applications
State of the art e-Science services.
Different levels of e-Science services.
Kepler, taverna, and triana
Existing workflow systems.
SP 2.5 effective and concrete solutions for
interfacing, Integration, and interoperability
Grid infrastructure.
86
Theme workflow bus

• Two options
• Develop a perfect system (abstract from the
  existing systems)
• the most suitable workflow language
• the most advanced execution engine
• Most sufficient user support
• Provide an integration solution (aggregate
  existing systems)
• Interface existing systems to the functional
  components which they are missing
• Allowing these systems talk to each other via the
  bus like framework

87
A basic paradigm

taverna
kepler
triana
Workflow bus
88
How the workflow bus looks like?

• Provides interface to dedicated e-Science
  services for data, computing tasks, information
  and knowledge
• Provides integration solution to different
  workflow systems

89
Question 1

• What is the proper execution models for different
  styles of Grid enabled workflows?
• Description
• Running data/computing intensive workflows on
  Grid are our interest
• These dependencies of the processes in workflow
  are different in domains
• What is the execution model for them?
• Data flow based?
• Process net?
• Continuous time?
• Or others?
• Goal to answer the following questions
• What are the existing execution models for Grid
  workflows?
• What are the requirements for an execution model
  of Grid workflows?

90
Question 2

• Can Web service based workflow support data
  intensive application?
• Description
• SOAP based WSs provide uniform interface
• But have big overhead for transmitting large
  volume data
• Is there a solution which can transparently
  choose protocol to passing different size of SOAP
  messages between WSs?
• Research goal
• At which level we can add our solution?
• Can we do it transparently?

91
Question 3

• Can Kepler benefit from our workflow bus?
• Description
• Kepler has a good GUI and sophisticated computing
  models
• But it is not really Grid enabled
• Can we execute a Kepler workflow on Grid in a
  transparently way?
• Goal
• Test how kepler works with the workflow bus.

92
Question 4

• How workflow bus supports data provenance?
• Description
• Data provenance will be an important
  functionality for workflow
• How will the workflow bus realizes this
  functionality?
• Goal
• Propose a provenance solution
• Prototype it

93
Manpower organization

• Project driven
• Decompose the questions into small steps
• Define each steps as small projects
• Assign people to concrete task
• A responsible people for each small project
• Periodically, the responsible person report the
  progress (in formal PPT way)
• Each small project should end up with a paper or
  a demonstration

94
Agenda and milestone
Question 1
Question 2
Question 3
Question 4
August
December
October
June
95
Question 1 team

• Tasks
• Survey existing models
• How existing workflow engines works
• How data and control flow are handled
• How Grid resources are utilized
• How service oriented architecture is supported
• How large volume data is handled
• Goal
• Assess if the existing engine suitable for Grid
  workflow
• Analysis the mapping between workflows and
  execution model

96
Discussion

• Developing innovative, small but usable tools
• Get close to application, and validate the design

97
WS in data intensive workflow
98
Scenarios
Engine
Service A
Service B
Service C

• Problem scenario
• Integrate legacy services in a workflow
• when the SOAP messages are big
• When one service has to give data to more than
  one services as input

99
A solution
Service A
Service B
Service C
Agent
Engine
100
A detailed scenario
Service A
Service B
Engine
Agent
Agent
Agent
Agent
101
Griddles and legacy webservices