Production Manager Tools (New Architecture)

1
Production Manager Tools (New Architecture)
LHCb Software Week 12-14 November 2003
Gennady Kuznetsov
2
Data Challenges 2001
History Controlling Monte-Carlo simulation.
In 2001, control system was based on a single
Java Servlet which in fact was a single file with
Java code. The sequence of execution was fixed
and all site dependencies had to be programmed
inside. Code had to be redistributed for each
location with the Servlet runner. Increased
complexity of the tasks and demands for
flexibility (requirement to run more then one
type of job) required architectural changes. As
a result, the Data Management group produced a
document with a new flexible architecture.
Gennady Kuznetsov
3
The Grand Design
Job Configuration, Data Production, Bookkeeping -
Design Document, M. Frank (Editor), 04/12/2001
Gennady Kuznetsov
4
Data Challenges 2003

• Advantages
• This very flexible architecture gives us freedom
  to adjust any parameters in the workflow and also
  allows us to manipulate Steps known to the
  Agent.
• Disadvantages
• Agent had to exist on the site or had to be
  submitted.
• For any change in execution, we had to update
  the agents everywhere.

Agents interpret XML file with Workflow
description
Agent
Agent
Agent
Gennady Kuznetsov
5
Components - New System
We add Module as additional component of the
architecture
This module structure allow Production Manager to
construct any algorithm as a combination of
modules.

• Levels of usage
• Module Programmer
• Step Production manager
• Workflow User/Production manager

Each step generates job as a Python program.
Gennady Kuznetsov
6
Architecture - New System
Variable
n
1
VariableCollection
edit
ModuleDefinition
ModuleInstance
Module Editor
n
n
1
1
edit
Step Editor
StepDefinition
StepInstance
n
n
1
1
edit
Workflow Editor
WorkflowDefinition
Presenter Name
7
Module/Step exampleExecution Sequence
DistributionSoftware checkand installation
Example of the Workflow 370000-Sv260r2-Bv17r4
Module
GaussSimulation
Mon. ServiceMonitoring
Data quality rawh validation
File ServiceFile transfer
Bookkeeping. ServiceBookkeeping
GaussSimulation
Mon. ServiceMonitoring
Data quality rawh validation
File ServiceFile transfer
Bookkeeping. ServiceBookkeeping
GaussSimulation
Mon. ServiceMonitoring
Data quality rawh validation
File ServiceFile transfer
Bookkeeping. ServiceBookkeeping
GaussSimulation
Mon. ServiceMonitoring
Data quality rawh validation
File ServiceFile transfer
Bookkeeping. ServiceBookkeeping
BrunelReconstruction
Mon. ServiceMonitoring
Data quality oodst validation
File ServiceFile transfer
Bookkeeping. ServiceBookkeeping
Another three lines (not enough space)
Gennady Kuznetsov
8
Module/Step exampleData propagation
Example of the Workflow 370000-Sv260r2-Bv17r4
Module
DistributionSoftware checkand installation
GaussSimulation
Mon. ServiceMonitoring
Data quality rawh validation
File ServiceFile transfer
Bookkeeping. ServiceBookkeeping
Gennady Kuznetsov
9
Module Editor
Module Name
Stored as XML file
Description
Python code of single module. Can be many classes.
Module variables.
Gennady Kuznetsov
10
Step Editor
Stored as XML file, where all modules are embedded
Definitions of Modules
Step Name
Instances of Modules
Description
Selected instance
Variables of currently selected instance
Step variables.
Gennady Kuznetsov
11
Workflow Editor
Workflow Name
Step Definitions
Stored as XML file
Step Instances
Selected Step Instance
Description
Variables of currently selected Step Instance
Workflow Variables.
Gennady Kuznetsov
12
Job Splitting
The input value for the job splitting is a Python
list object. Every single (top level) element of
this list applies to the Workflow Definition and
propagates through the code and generates single
element of the production (one or several jobs).
Python List
Gennady Kuznetsov
13
Current Status

• Done
• All three editors (Module, Step, Workflow)
  ready.
• XML schema seems stable.
• To do for the Data Challenge 2004
• Define variable types.
• Extend Python code generator to cover Workflow.
• Create appropriate module library.
• Implement list input for the Workflow Editor
  (relates to the job splitting).
• Future development
• Add graphical editing panel to Step and Workflow
  editors.
• Finish embedded Python editor with highlighting
  feature.

Gennady Kuznetsov