Data Management

1
Data Management

• Status of CVS repository
• Production databases
• Production tools

2
CVS

• Separate area for each sub-project
• Internal structure
• cmt requirements
• ltpackagegt public headers
• src sourceopt src/ltcomponentgt
• Opt python, cmds, images, etc

3
Components
Job Sub- mission
Tool
Real dataflow
Apparent dataflow
4
Components

• Job configuration Markus
• Data production Markus
• Job submission Andrei
• Bookkeeping Sebastien/Francoise

5
Job Configuration

• Database ok.
• Simple HTTP based Editor
• XML-RPC based interface
• Boost python
• Tool to import applications

6
Job Configuration Editor
7
Data Production
8
Data Production - Details
Workflow
Run

• Name
• State
• N(evt) to be processed

• Name

• Step Seq.
• Nevt(Split)
• Nevt(Merge)

Step
Step

• State
• Center
• Machine

Step
ltRecgt Job
ltTagsgt Job

• Name
• Input file types
• Output file types
• Application Name

Input File
Output File
Input File
Output File
FileType
File Type
Job Configuration DB
Workflow description
9
Data Production

• Work flows
• Database ok.
• Simple HTTP based Editor
• XML-RPC based interface
• Boost python
• Data Production
• Database ok.
• Simple HTTP based Editor
• XML-RPC based interface
• Boost python

Instantiation of work flows to production
runs Very primitive
10
Work Flow Instantiation
Pythia v2
Gauss - v5
Gauss - v5
Gauss - v5
Gauss - v5
All processes in one job Tools missing
Brunel - v12
Brunel - v12
GenTag v7
Work Flow
Execution Flow (Jobs)
11
Work Flow Instantiation

• Script generation through servlet
• Select work flow
• Instantiate work flow to production run
• Add basic information
• of events
• CPU requirements etc.
• Generate scripts
• Possibility to extend existing runs
• Scripts are then picked up by Andreis client

12
Interacting Components

• Main strategy on component failure(s)
• The system must stay operable
• Intermediately produced information items are
  cached locally until servers are up again
• Production server (see next slide)
• Production client (see Andreis talk)
• Servers/servlets to publish bookkeeping
  information(see Sebastiens talk)

13
Component Collaboration
Production Editor
Work flow Editor
Production DB
Production data
Scripts
Edit
Instantiate Workflow
Production Server

• Job request
• Status updates

Prod.Mgr
Production Client
Bookkeeping info
XML Bookkeeping
14
Script Generation

• setenv PRODUCTION_IDENTIFIER 400000067
• setenv JOB_NAME 400000369
• Target operating system linux
• Parameters of step MinBias.Dec.2002
• setenv MBIAS_QQ_USER_FILE None
• setenv MBIAS_EVTYPE 61
• Parameters of step B2pipi.Dec.2002
• setenv B2PIPI_QQ_USER_FILE B2pipi.dec
• setenv B2PIPI_EVTYPE 411300
• Parameters of work flow B2pipi.Dec.2002
• setenv SIMULATION_APP SICBMC
• setenv SIMULATION_VSN v254r2
• setenv RECONSTRUCTION_APP Brunel
• setenv RECONSTRUCTION_VSN v13r1
• setenv DBASE_PKG dbase
• setenv DBASE_VSN v248r4
• setenv XMLDDDB_PKG XmlDDDB
• setenv XMLDDDB_VSN v8r1
• setenv QQ_USER_FILE MBIAS_QQ_USER_FILE,B2PIPI
  _QQ_USER_FILE

InstantiatedWorkflow
15
Production Editor
16
Work Flow Editor
17
Production DBs - Summary

• Primitive production tools exist
• WWW based
• Basic interface for production clients exists
• Needs testing
• The environment to start developing production
  tools is in place
• Problems with MySQL
• MySQL does not support nested SELECT
• MySQL will not come in the very near future

18
Job List

• Import Brunel versions used in production run to
  job configuration database
• Construct reasonable work flow
• Test generated production script(s)
• Test production server-client collaboration