GRIDCC: Real-time Workflow system

1
GRIDCC Real-time Workflow system

• A.Stephen McGough, Asif Akram, Li Guo, Marko
  Krznaric, Luke Dickens
• David Colling, Janusz Martyniak
• Roger Powell, Paul Kyberd
• Constantinos Kotsokalis

2
What is GRIDCC
Grid enabled Remote Instrumentation with
Distributed Control and Computation

• Adding real time control of instruments to the
  Grid
• What makes this unique
• Instruments may only beready at specific times
• Reservations
• Need to ensure other Gridservices are available
  with instruments
• Reservations / SLAs
• Real-time visualisation of live results and
  steering
• Through Virtual Control Room (VCR)
• Reservations / SLAs

3
GRIDCC Overview
GRID
Real Time
QoS
4
Motivation

• QoS is important in Workflows
• Just running a workflow and hoping is not enough
• We need more than WS-Agreement
• Though WS-Agreement is very important to us
• We want to allow the user to specify their QoS
  requirements over a workflow
• Either as hard constraints
• This must finish by time x
• As soft constraints
• This should finish by time x in y of cases

5
Aim of GRIDCC Editor

• Portal Based Editor
• Drag and Drop
• Hiding Complexities
• Ease of use
• Web Service Registry
• QoS Constraint
• XML Schema Registry (To be Implemented)
• Workflow Monitoring (To be Implemented)
• Standard Based
• Workflow Engine Independent
• Based on Flex 2 for User Interface, Java on
  Server Side

6
(No Transcript)
7
Adding QoS to a Workflow
Partner Link
XPath
8
A BPEL workflow

• lt?xml version"1.0" encoding"UTF-8"?gt
• ltprocess name"gt
• ltpartnerLinksgt lt/partnerLinksgt
• ltvariablesgt ltvariable element"hdns1SimpleTrig
  gerID" name"SimpleTriggerID"/gt lt/variablesgt
• ltflowgt
• ltsequence name"WSIIETestSequence"gt
• ltreceive createInstance"yes"
  name"WSIIETestInitiate" operation"IETest"
  partnerLink"initiate_IETest_PL/gt
• ltsequence name"WSIIETestInitialiseAndG
  etStatus"gt
• ltinvoke inputVariable"getDTInstanceR
  equestVar" name"WSIIE-getDTInstance"
  operation"getDTInstancegt
• lttarget linkName"L5"/gt ltsource
  linkName"L6"/gt
• lt/invokegt
• ltinvoke inputVariable"getDTStatusRequ
  estVargt lt/invokegt
• ltassign name"GetStatusOutput"gt
  lttarget linkName"L8"/gt
• ltcopygt ltfrom part"parameters"
  query"" variable""/gt ltto variable"StatusValue"/
  gt lt/copygt
• lt/assigngt
• lt/sequencegt
• ltswitchgt ltcase condition"bpwsgetVariabl
  eData('StatusValue')! bpwsgetVariableData('Statu
  sValueRunning')"gt
• ltsequence name"InitConfigureAndSta
  rtIESequence"gt
• ltwhile condition"bpwsgetVari
  ableData('StatusValue')! bpwsgetVariableData('St
  atusValueRunning')" name"UntilStatusIsRunning"gt

Using XPath pointers into BPEL document
9
Simple Example

• lt?xml version"1.0" encoding"UTF-8"?gt
• ltQoSRequirements xmlnsxsi"http//www.w3.org/2001
  /XMLSchema-instance" xsinoNamespaceSchemaLocation
  "C\Documents and Settings\asm\Desktop\QoSRequire
  ments.xsd"gt
• ltQoSConstraintgt
• ltXpathReferencegt/process/flowlt/XpathReferencegt
• ltCPUSpeedgt2048000lt/CPUSpeedgt
• ltMaxDurationTimegt100lt/MaxDurationTimegt
• ltReliabilitygt100lt/Reliabilitygt
• lt/QoSConstraintgt
• lt/QoSRequirementsgt

10

• lt?xml version"1.0" encoding"UTF-8"?gt
• ltQoSRequirements xmlnsxsi"http//www.w3.org/2001
  /XMLSchema-instance" xsinoNamespaceSchemaLocation
  "C\Documents and Settings\asm\Desktop\QoSRequire
  ments.xsd"gt
• ltQoSConstraintgt
• ltXpathReferencegt/process/flow/sequence1/sequenc
  e1/invoke1lt/XpathReferencegt
• ltXpathReferencegt/process/flow/sequence1/sequenc
  e1/invoke2lt/XpathReferencegt
• ltCPUSpeedgt2048000lt/CPUSpeedgt
• ltMaxDurationTimegt100lt/MaxDurationTimegt
• ltReliabilitygt100lt/Reliabilitygt
• lt/QoSConstraintgt
• lt/QoSRequirementsgt

11

• lt?xml version"1.0" encoding"UTF-8"?gt
• ltQoSRequirements xmlnsxsi"http//www.w3.org/2001
  /XMLSchema-instance" xsinoNamespaceSchemaLocation
  "C\Documents and Settings\asm\Desktop\QoSRequire
  ments.xsd"gt
• ltQoSConstraintgt
• ltXpathReferencegt/process/flow/sequence1/sequenc
  e1/invoke1lt/XpathReferencegt
• ltReservationRequiredgt
• ltStartTimegt2006-10-02T230000.0Zlt/StartTimegt
• ltEndTimegt2006-10-02T235959.0Zlt/EndTimegt
• ltEndpointNamegtcpu1.doc.ic.ac.uklt/EndpointNamegt
• lt/ReservationRequiredgt
• ltCPUSpeedgt1024000lt/CPUSpeedgt
• ltMaxDurationTimegt80lt/MaxDurationTimegt
• ltReliabilitygt100lt/Reliabilitygt
• lt/QoSConstraintgt
• ltQoSConstraintgt
• ltXpathReferencegt/process/flow/sequence1/sequenc
  e1/invoke2lt/XpathReferencegt
• ltCPUSpeedgt2048000lt/CPUSpeedgt
• ltMaxDurationTimegt20lt/MaxDurationTimegt
• ltReliabilitygt50lt/Reliabilitygt

ltReservationMadegt ltStartTimegt2006-10-02T23
0000.0Zlt/StartTimegt ltEndTimegt2006-10-02T23595
9.0Zlt/EndTimegt ltEndpointNamegtcpu3.doc.ic.ac.uklt/E
ndpointNamegt ltTokengtdveinvunvuin42235454lt/Tokengt
lt/ReservationMadegt
Make a reservation through Agreement Service
(WS-Agreement)
Store result here
12
Workflow Management Service (WfMS) Architecture
13
Architecture

• Planner
• Determines if a workflow can be executed
• Decides the best resources to use for the
  workflow
• Workflow Engine
• Engine to enact the workflow determined within
  the Planner
• ActiveBPEL
• Temporary WfWS
• Instance of a workflow deployed to the Workflow
  Engine
• Agreement Service
• Negotiation service for agreements with the CEs,
  SEs, IEs, NEs

14
Architecture

• Observer
• Monitors the temporary WfWS as it executes
  checking that it is performing within parameters
• Can be seen as an instance of a Global Problem
  Solver
• Delegation Service
• Provides a mechanism for delegating credentials
  through the WfMS
• Workload Management System
• Standard gLite WMS

15
What is the Planner?

• Makes a plan for how to execute a workflow
• Can QoS requirements be met
• Which resources to use
• Whether reservations are required in order to
  meet QoS
• Does not override the right of a user to request
  reservations
• Manipulation of the workflow
• The expectation for how the workflow will
  progress
• What will be executed and where

16
Flow of Documents
17
GRIDCC WfMS
Workflow Optimiser
Validator
Resource Selector
Resource Pruning
Reserver
SEDA based architecture
18
Pipeline Stages for WfMS

• XML to Object Stream
• Convert the XML document into a set of objects
• Object Stream to XML
• Convert Objects into XML document
• Reservation maker
• Make reservations through the Agreement Service
• Workflow fiddly bits (add in stuff we need)
• Automatically deal with adding in security
  elements to workflow
• Insert error checking into workflow, Performance
  logging of workflows
• Workflow Validator
• Workflow manipulator
• Modify a workflow to improve performance
• QoS validate
• Validate that the workflow can meet the QoS
  requirements placed on it
• Workflow Resource Selector
• Select the resources within the Grid which will
  allow QoS to be met.
• JDL Manipulator - Manipulate JDL documents to use
  resources better

19
Questions