Workflow in Astrogrid

1
Workflow in Astrogrid

• Jeff Lusted
• Dept Physics and Astronomy
• University of Leicester

2
Part One...

• A Quick Tour of Workflow Architecture

3
What is the purpose of Workflow?

• To accomplish a complex piece of work
• It takes more than one step to achieve.
• The workflow is executed asynchronously.
• Each step is executed asynchronously.

4
A Skeleton Workflow

• ltworkflow name"ThreeStepWorkflow" gt
• ltsequencegt
• ltActivity name"Step
  One" gt
• lttool
  name"query-6df"gt
• ...
• lt/toolgt
• lt/Activitygt
• ltActivity name"Step
  Two" gt
• lttool
  name"query-2Mass" gt
• ...
• lt/toolgt
• lt/Activitygt
• ltActivity name"Step
  Three" gt
• lttool
  name"federator" gt
• ...
• lt/toolgt
• lt/Activitygt
• lt/sequencegt
• lt/workflowgt

5
Expansion of Step One...

• ltActivity name"Step one"
  gt
• 
  ltdescriptiongt...lt/descriptiongt
• lttool
  name"query-6df" interface"adql"gt
• 
  ltinputgt
• 
  ltparameter name"Format"
  indirect"false"gt
• 
  ltvalue gtVOTABLElt/valuegt
• 
  lt/parametergt
• 
  ltparameter name"Query"
  indirect"true"gt
• 
  ltvalue gtivo//...jl99/query/6
  dfQuery.xmllt/valuegt
• 
  lt/parametergt
• 
  lt/inputgt
• 
  ltoutputgt
• 
  ltparameter name"Result"
  indirect"true"gt
• 
  ltvalue gtivo//...jl99/votable
  /6dfQueryResults.votlt/valuegt
• 
  lt/parametergt
• 
  lt/outputgt
• lt/toolgt
• lt/Activitygt

6
The View from 10,000 Metres

• A workflows exist at
• Design Time
• Execution Time
• When submitted for execution, a workflow
  becomes a job.

7
Design Time From 5000 Metres

• A Design
• Is held as an XML file in MySpace.
• The Design process
• Is managed by the Workflow Editor in the
  Portal.
• Is aided by holding metadata in the Astrogrid
  Registry.

8
Execution Time From 5000 Metres

• Job Execution System
• Central controlling element.
• Interprets the workflow design.
• Manages the flow of control between Steps.
• Common Execution Architecture
• Manages the local environment for each Step
  executed.
• Manages the flow of data required by each Step.

9
Overview of Workflow Architecture
10
Part Two...

• Workflow as a Programming Language

11
Back to Design...

• Steps and Tools
• Sequences and Flows.
• Variables and Scopes.
• Loops and Conditional Execution.
• Scripting and Script Objects.

12
Sequence Sequential Execution
ltsequencegt
ltActivity name"Step One" gt
lttool name"query-6df"gt
...
lt/toolgt
lt/Activitygt
ltActivity name"Step Two" gt
lttool name"query-2Mass" gt
...
lt/toolgt
lt/Activitygt
ltActivity name"Step Three" gt
lttool name"federator" gt
...
lt/toolgt
lt/Activitygt lt/sequencegt
13
Flow Parallel Execution
ltsequencegt
ltflowgt
ltActivity name"Step One" gt
lttool
name"query-6df"gt
...
lt/toolgt
lt/Activitygt
ltActivity name"Step Two"
gt
lttool name"query-2Mass" gt
...
lt/toolgt
lt/Activitygt
lt/flowgt
ltActivity name"Step Three" gt
lttool name"federator" gt
...
lt/toolgt
lt/Activitygt lt/sequencegt
14
Setting Variables

• Setting...
• Using variables to construct a filename...

ltset var"namestub" value"votable" /gt ltset
var"filecount" value"1"/gt
ltoutputgt ltparameter name"result"
indirect"true"gt ltvaluegtnamestub-
filecount 1lt/valuegt lt/parametergt lt/outpu
tgt
15
Conditional Execution
ltset var"x" value"1" /gt ... ltif test"x gt
0"gt ltthengt ltsequencegt
... lt/sequencegt
lt/thengt ltelsegt
... lt/elsegt lt/ifgt
16
While Loop
ltwhile test"results null
results.size() lt 1"gt ltstep
result-var"results"gt ...
lt/stepgt lt/whilegt
17
For Loop
ltfor var"x" items"1...10gt ltstep
result-var"results"gt ...
lt/stepgt lt/forgt
18
Scripting

• Scripting Expressions
• Script

ltset var"x" value"1" /gt ltif test"x gt
0"gt ... lt/ifgt
ltscriptgt ltbodygt print("Step
produced " count "votables")
lt/bodygt lt/scriptgt
19
Scripting Example plus Script Objects
ltstep result-var"results"gt ... lt/stepgt ltset
var"urlList" /gt ltscriptgt ltbodygt if
(results.size() ! 1)
jes.error("didn't produce expected number of
results") else votable
results.get('votable') parser new
XmlParser() nodes
parser.parseText(votable) urlList
nodes.depthFirst()
.findAllit.name() 'STREAM'
.collectit.value().flatten()
print(urlList)
lt/bodygt lt/scriptgt
20
Conclusion Astrogrid Workflow Capabilities

• Distributed workflow executing on an asynchronous
  basis
• Highly programmable...
• ... Workflows from the simple to the complex
• Can integrate astronomical tools

21
Contacts.

• Work Group Leader Jeff Lusted
  jl99_at_star.le.ac.uk
• Workflow and JES Noel Winstanley
  nw_at_jb.man.ac.uk
• Common Execution Architecture Paul Harrison
  pah_at_jb.man.ac.uk
• Workflow Editor Phil Nicolson pjn3_at_star.le.ac.uk
• www.astrogrid.org