The GANGA Interface

1
The GANGA InterfaceATLAS Development

• Alvin Tan
• (Birmingham)

2
Agenda

• ATLAS
• DC2
• ATLAS/LCG Integration Tests
• Automatic Software Packaging and Installation
• Distributed Metadata Services
• ATLAS Distributed Analysis (ADA)
• GANGA
• Role in ADA
• Current status
• Job Options Editor (JOE)
• Future plans

3
ATLAS DC2

• September 2003 July 2004 Analysis phase.
• DC2 goals include
• Full use of Geant4 POOL LCG applications
• Pile-up and digitization in Athena
• Deployment of the complete Event Data Model and
  the Detector Description
• Simulation of full ATLAS and 2004 combined
  Testbeam
• Test the calibration and alignment procedures
• Use widely the GRID middleware and tools
• Large scale physics analysis
• Computing model studies (document end 2004)
• Run as much as possible of the production on
  LCG-1/2
• DC2 is essentially a slice test of the ATLAS
  Computing Model.

4
ATLAS DC2 - Schedule
September 03 Release7 Mid-November 03
pre-production release February 27th 04
Release 8 (production) April 1st 04 June 1st
04 DC2 July 15th
Put in place, understand validate Geant4
POOL LCG applications Event Data
Model Digitization pile-up byte-stream Convers
ion of DC1 data to POOL large scale
persistency tests and reconstruction Testing
and validation Run test-production Start final
validation Start simulation Pile-up
digitization Event mixing Transfer data to
CERN Intensive Reconstruction on
Tier0 Distribution of ESD AOD Calibration
alignment Start Physics analysis Reprocessing
5
ATLAS/LCGIntegration Test

• LCG-1 1.1.1 is deployed in over 20 sites across
  the world and the number is increasing.
• ATLAS-6.0.4 and some EDG WP1 bug fixes are
  included in this release.
• To test the readiness of ATLAS code on LCG,
  simulation and reconstruction were executed using
  a test suite implemented in Perl.

6
Integration Test Results (1)
7
Integration Test Results (2)
8
Integration Test Results (key)

• (1) IFIC,CERN,UAM,MSU,TOKYO Abort (Caught
  GLOBUS error 3 (I/O related) from IFIC)
• (2)KRAKOWMDS Error Invalid/ no SE registered.
• (3)IFCA,CESGA no file access to SE data from
  WN (NFS?)
• (4)CNAF Missing AtlasRelease directory
• (5) IFAEYPBINDPROC_DOMAIN Domain not bound
  messages
• (6)BNL,IMPERIALcould not connect to MDS
• (7)FZKjavax.naming.NameNotFoundexception LDAP
  Error 42
• (8)BARCELONA no local mapping for GLOBUS ID

9
Integration Test Conclusion

• With only 6 out of 19 sites passing the simul
  test, a couple of sites suffering Aborts at
  submission time and the rest experiencing various
  other problems, the tests were inconclusive and
  further investigation is needed.

10
Automatic Software Packaging and Installation

• Based on Pacman 2, CMT, tar/rpm and some shell
  scripts.
• It satisfies the requirements set out for binary
  kit distribution and developers kit
  distribution.
• Kits exist for recent releases (e.g. 7.0.2,
  7.4.0, 7.5.0).
• Early adopters have given useful feedback and it
  is now being imported into Grid production
  systems.
• At RHUL, students and RAs have been doing their
  physics studies using the 7.0.2 kit since
  November 2003.

11
Automatic Software Packaging and
Installation(cont)

• The LCG has recently confirmed the method/format
  of publishing kits.
• Future plans include (this deliverable is
  complete)
• integration with GANGA for user code
  distribution,
• full source kit distribution, and
• Pacman 3 integration.

12
Distributed Metadata Services

• Main objectives
• To understanding of the existing ATLAS metadata
  structures and its use-cases.
• Develop metadata to support navigational
  use-cases.
• Understand analysis use-cases, optimise event to
  file granularity for different types of analysis
  data and develop automated ways of obtaining
  optimal event data granularity.
• Implement a working solution that will integrate
  with existing ATLAS software, in particular, the
  ATLAS Metadata Infrastructure (AMI) product.

13
Distributed Metadata Services(cont)

• The plan
• To deliver a distributed metadata web service
  with a GANGA interface that will facilitate
  distributed analysis within the existing ATLAS
  analysis framework, consistent with ARDA, and
  function effectively when deployed as a full
  scale production service.

14
ATLAS Distributed Analysis (ADA)

• The ATLAS effort for distributed analysis in the
  context of ARDA is lead by David Adams (BNL) who
  is responsible for coordinating the development
  of software tools for distributed analysis and
  their integration into the ATLAS software
  environment. He is involved in the DIAL project
  and is currently the ATLAS coordinator of GANGA
  project.

15
ADA Scope

• Analysis (not necessarily distributed)
• Supports the manipulation and extraction of
  summary data (e.g. histograms) from any type of
  event data e.g. AOD, ESD,
• Supports user-level production of event data
  e.g. MC generation, simulation and
  reconstruction.
• Distributed analysis
• Extends the extraction and production support to
  include distributed processing, data and users.
• Natural extension of non-distributed analysis.
• Easily invoked from any ATLAS analysis
  environment including Python, ROOT and command
  line while easily ported to any other
  environments (e.g. JAS).

16
ADA - The early days

• A number of projects relevant to distributed
  analysis and production already exist in ATLAS
• DC2 production system
• Supervisor/executor model allows grid-based
  production of dataon different grid flavors
• Includes DMS file replica service combining all
  the ATLAS replica catalogs
• The AMI Database
• Just introduced a web service interface to
  bookkeeping
• DIAL Distributed Interactive Analysis of Large
  Datasets
• Running analysis web service with ROOT-based
  client
• GANGA
• User-friendly interface for distributed
  Athena/Gaudi processing
• These constitute the starting point for evolution
  to ADA.

17
Automatic Production System (Windmill)
18
ADA Strategy

• Implement ADA as a collection of grid services
• As described in ARDA document
• Use ARDA components where possible
• Add missing and ATLAS-specific pieces
• Identify high-level services users interact
  directly with
• Job submission (aka analysis service)
• Catalogs
• Define language for interaction with these
  services
• AJDL Analysis Job Description Language
• Objects exchanged between clients and services
• Service interfaces expressed in terms of these
  objects

19
ADA - AJDL
20
ADA Strategy (cont)

• Implement high-level services
• Reuse existing software where possible.
• Re-implement as requirements become clearer and
  new middleware becomes available.
• Provide clients for ATLAS analysis environments
• Python (GANGA), ROOT, command line
• Also web portal and java
• Look to common projects required pieces
• e.g. ARDA, GANGA, DIAL,
• Share as much as possible with ATLAS production
  which is also distributed and possess similar
  interfaces and code for bulk and user-level
  production.

21
GANGA in ADA

• A major player in the ADA, GANGA developers will
  work closely with DIAL developers.
• To facilitate ADA, GANGA will have an AJDL
  interface.
• GANGA and DIAL (ROOT) will be the first ADA
  clients and will provide some initial services
  based on existing functionality e.g. job
  submission and monitoring.
• GANGA will also provide an ADA scheduler service
  for other flavours of ADA clients.

22
GANGA in ADA
23
GANGA - Status Update

• Work since September 2003 has focused on
  refactoring, to create a system that is more
  modular and more flexible
• In short-term (next 1-2 months), changes will
  mainly affect developers .
• In longer term (in time for DC2) will see
  significant gains for users improvements in
  functionality, ease of use and stability.
• Have introduced PyBus software bus, developed by
  W. Lavrijsen with contributions from K.
  Harrison(released in LCG SEAL project)
• Allows dynamic association of modules and logical
  name.
• Forms the basis of a highly configurable
  framework supports ATLAS/LHCb customizations and
  user add-ons.

24
GANGA Status Update (cont)

• Moving to XML-based job description.
• In the process of defining details of XML schema
• Aim to have job description consistent with DIAL
  (and others?)
• Working to separate client and service
  functionality implemented in GANGA V1.0
• Functionality of Job Options Editor (JOE) being
  enhanced (more later).

25
GANGA - JOE

• Job-options editor (JOE) is evolving to become a
  more powerful, standalone application, which can
  be loaded by GANGA.
• Assist user in the creation/modification of
  Gaudi/Athena job options by presenting the user
  with a hierarchical view of available options
  files and helping the user with value entry.

26
GANGA - JOE (cont)

• In process of creating Job Options Information
  Resource (JOIR) database
• JOIR database of job options will facilitate
  validation by providing valid ranges, option
  choices, and option descriptions.
• Considering suggestions from LHCb for improving
  automated job-option extraction.
• Validation component of job option entry using a
  database created by automated extraction of job
  option attribute/value from Gaudi/Athena shared
  libraries.

27
GANGA Future Plans

• Work towards GANGA/DIAL integration within ADA
• More tentatively, looking at possibilities for
  interfacing to Atlantis for displaying event
  data.
• Task Editor (TEd) will build on JOE by providing
  access to other information relating to a user
  task (e.g. user code). Currently in planning
  stage.
• Demo of GANGA in the context of ADA at next ATLAS
  Software Workshop (1-5 March)
• ADA/GANGA tutorial at ATLAS Software Workshop at
  BNL (24-28 May). Full functionality foreseen for
  GANGA V2.0. (GridPP milestone for end-March).

28
End.