LCG CR : APPLICATION AREA

1
LCG CR APPLICATION AREA
P. Dauncey, M. Martinez-Perez, M. Mangano , S. D.
T. - referees

• BASED ON THE PARALLEL SESSION APPLICATION
  AREA, Monday 25/6/06

2
OUTLOOK

• APPLICATION AREA
• MANAGERIAL AND ORGANIZATIONAL ASPECTS
• MANPOWER STATUS
• STATUS AND ACHIEVEMENTS OF THE PROJECTS
• CONCLUSIVE REMARKS

3
APPLICATION AREA
4
APPLICATION AREA

• RECALL
• Deliver the common physics applications software
• Organized to ensure focus on real experiment
  needs
• Experiment-driven requirements and monitoring
• Open information flow and decision making
• Participation of experiment developers
• Frequent releases enabling iterative feedback
• Success defined by experiment validation
• Integration, evaluation, successful deployment

5
AA Projects
REPORT INCLUDED IN THE OVERVIEW

• SPI Software process infrastructure (A.
  Pfeiffer)
• Software and development services external
  libraries, savannah, software distribution,
  support for build, test, QA, etc.
• ROOT Core Libraries and Services (R. Brun)
• Foundation class libraries, math libraries,
  framework services, dictionaries, scripting, GUI,
  graphics, SEAL libraries, etc.
• POOL Persistency Framework (D. Duellmann)
• Storage manager, file catalogs, event
  collections, relational access layer, conditions
  database, etc.
• SIMU - Simulation project (G. Cosmo)
• Simulation framework, physics validation studies,
  MC event generators, Garfield, participation in
  Geant4, Fluka.

DEDICATED PRESENTATIONS
6
MANAGERIAL AND ORGANIZATIONAL ASPECTS
7

• Informal forum
• exchange of information between the AA projects
  and experiments
• Every two weeks

• Experiments participate in planning,
  management, and architectural and technical
  direction
• Every two weeks
• Public minutes

• Project milestones, proposed by AA projects
• Discussed in the AF
• Added into quarterly reports for approval
• Quarterly reports
• To monitor the progress of the projects
• Reviewed by LCG Management Board

• Reviews
• Internal AA reviews
• LHCC Comprehensive Review

8
MANPOWER, STATUS AND PERSPECTIVES
9
Manpower Table
10
Estimated Resource Needs
11
Staffing Remarks

• Reduction of manpower as anticipated for 2006
• 42 FTE -gt 39 FTE (3FTE)
• But perhaps not in the projects or work packages
  expected
• The projects most affected by the reduction
• SPI 5 FTE -gt 2.5 FTE ( EGEE contribution IT)
• COOL 2.6 FTE -gt 1.7 FTE
• Project affected positively
• PROOF 2.6 FTE -gt 4.3 FTE (DOCT, MC Fellow)
• Major reduction is expected in 2008
• End LD contracts retirement. No solution yet

12
STATUS OF THE PROJECTS
13
SPI Software Project Infrastructure 1/3

• Andreas Pfeiffer replaced Alberto Aimar as SPI
  project leader
• coping with strong (50) reduction of resources
• Outsourcing some activities ?
• SPI services (quite a diversified collection)
• External Software (Common libraries from HEP, IT,
  )
• Savannah Project Portal
• enables project members to collect all the
  information related to tracker items of their
  choice
• Build and Distribution
• LCG Software configuration management
• Builds and releases
• Software Distribution
• Quality Assurance Service
• Testing frameworks
• QA checklists and reports
• Documentation and Training
• Development of LCG policies, templates
• Code Documentation

14
SPI Software Project Infrastructure 2/3

• Examples to quantify the activity
• Evolution of configurations

15
SPI Software Project Infrastructure 3/3

• Examples to quantify the activity
• Status of Savannah

• 1944 users
• 1083 at last review 80

• 17819
• submitted bugs

• 196 projects
• 122 at last review 60

• 3539
• submitted tasks

16
SIMU - SIMULATION PROJECT 1/4

• SIMULATION SUBPROJECTS
• Simulation framework
• Interface to multiple simulation engines (Geant4,
  Fluka) and geometry models exchange (GDML), MC
  truth handling, Python interfaces
• Geant4 team participating
• Aligned with and responding to needs from LHC
  experiments, physics validation, simulation
  framework
• Fluka team participating
• Framework integration, physics validation
• Garfield team participating
• Garfield package integration and support in LCG -
  SPI
• Simulation physics validation
• Assess adequacy of simulation and physics
  environment for LHC and provide the feedback to
  drive needed improvements
• Generator services
• Generator libraries validation/test suite
  development when needed (HepMC, etc.) common
  event files

17
SIMU - SIMULATION PROJECT 2/4

• GEANT4
• 2 new releases (Jan06, Jun06)
• Improvements in simulation (EM, calorimetry, back
  scattering, etc.)
• Improved robustness
• Strong collaboration with Physics Validation
  subprojects
• FLUKA
• new realease (Sep06) 2006-3
• several physics improvements, f.i.
• Generation of primary ionisation events (request
  by ALICE)
• New high-energy hadronic generator
• Improvements in the evaporation/fission models
• First implementation of photon-muon pair
  production
• GARFIELD
• interfaces and porting
• computational methods

18
SIMU - SIMULATION PROJECT 3/4

• PHYSICS VALIDATION
• Background radiation studies
• Goal comparison of Fluka and Geant4 predictions
• Radiation studies in LHCb, CMS
• TARC benchmark test, in particular for n
• Simple benchmarks
• inclusive p production in p,K,p,p interactions
  on Mg, Ag, Au, at 100 and 320 (p-) GeV/c
  results ready!
• Test beam setups
• ATLAS EM barrel test beam 2002
• CMS HCAL test beam 2004
• ATLAS HEC test beam (Geant4 release validation)
• ATLAS barrel combined test beam 2004
• ATLAS TileCal test beam 2002
• investigations to improve hadronic shower shapes
  in the simulation
• relation between simulation developers and
  experiments has improved

19
SIMU - SIMULATION PROJECT 4/4

• GENSER - GENERATOR SERVICES subproject
• new release last summer 1.4.0
• LHC experiment requests collected in LCG
  Generator monthly meetings
• critics expressed by MC authors in recent MC4LHC
  workshop Jul06
• Procedural mistakes in releasing and fixing bugs
• Packaging issues for some MC generators
• More testing needed for core generators
• Control of feedback and support requests from
  experiments
• experiments want GENSER to continue its mandate
• ? new project leader Witold Pokorski (PH/SFT)
• field of major progress
• PYTHIA 8 (stand-alone C, with 3 year road-map,
  Sep 2004 - Sep 2007)
• MCDB (MC event Data Base) ? Ready to use !
• Generators framework General interface to any
  generator provided by GENSER (Production
  Validation)
• GDML (Geometry Description Markup Language)
• Extended to support all solids / constructs of
  Geant4
• Added support for optical surfaces and material
  properties

20
ROOT PROJECT 1/3

• ORGANIZATION
• BASE Fons Rademakers, Bertrand Bellenot(also
  SPI)
• I/O Philippe Canal, Paul Russo(FNAL), Markus
  Frank(lt10)
• DICT Philippe Canal, Axel Naumann, Stefan Roiser
• MATH Lorenzo Moneta, Anna Kreshouk
• GUI Ilka Antcheva, Bertrand Bellenot
• GRAPHICS Olivier Couet
• GEOM Andrei Gheata, Mihaela Gheata (Alice)
• PROOF Fons, Maarten Ballintijn(MIT), Gerri
  Ganis, Bertrand , Leandro Franco, Ian
  Iwaszkiewicz, Andreas Peters(arda)
• SEAL Lorenzo
• MAIN RECENT EVENTS
• ROOT Workshop at CERN October 2005
• 3 PRO versions in the last y, next Dec06
• DEV versions, next Oct06

21
ROOT PROJECT 2/3

• RECENT ACHIEVEMENTS (a selection)
• New MonaLisa monitoring plug-in
• New Users Guide
• More friendly user interfaces
• Implemented fast merging of Trees without
  unzipping the branch buffers
• storing and restoring of TTree to and from an SQL
  database
• Implemented a TTreeCache with a huge improvement
  when accessing remote files
• ROOT/CINT/Reflex integration still many unknowns
• first version in next PRO version
• several progresses in the mathematical packages
• vectors, matrices, math. Function, numerical
  calculations
• Fitting and minimization (new C MINUIT)
• Improved random numbers
• Histogram comparison
• Multivariable analysis
• Important progresses in GUI sector
• GEOMETRY PACKAGE
• Interfaces to Geant3 and Fluka validated
• Interface to Geant4 available in the coming weeks

22
Example of TTreeCache improvement

• The file is on a CERN machine connected to the
  CERN LAN at at 100MB/s.
• The client A is on the same machine as the file
  (local read)
• The client B is on a CERN LAN connected at 100
  Mbits/s with a network latency of 0.3
  milliseconds (P IV 3 Ghz).
• The client C is on a CERN Wireless network
  connected at 10 Mbits/s with a network latency of
  2 milliseconds (Mac Intel Coreduo 2Ghz).
• The client D is in Orsay (LAN 100 Mbits/s)
  connected to CERN via a WAN with a bandwith of 1
  Gbits/s and a network latency of 11 milliseconds
  (P IV 3 Ghz).
• The client E is in Amsterdam (LAN 100 Mbits/s)
  connected to CERN via a WAN with a bandwith of 10
  Gbits/s and a network latency of 22 milliseconds
  (AMD64 280).
• The client F is connected via ADSL with a
  bandwith of 8Mbits/s and a latency of 70
  milliseconds (Mac Intel Coreduo 2Ghz).
• The client G is connected via a 10Gbits/s to a
  CERN machine via Caltech latency 240 ms.
• The times reported in the table are realtime
  seconds

client latency(ms) cachesize0 cachesize64KB
cachesize10MB A 0.0 3.4
3.4 3.4 B 0.3 22.0
6.0 4.0 C 2.0
11.6 5.6 4.9 D 11.0
124.7 12.3 9.0 E
22.0 230.9 11.7 8.4
F 72.0 743.7 48.3
28.0 G 240.0 gt1800 125.4
9.9
One query to a 280 MB Tree I/O 6.6 MB
23
ROOT PROJECT 3/3

• PROOF
• an extension of ROOT to run on a distributed,
  heterogeneous system
• 4.1 FTE dedicated

24
PERSISTENCY FRAMEWORK 1/3

• 3 Software Packages
• CORAL
• Abstraction of relational database access for
  Oracle, MySQL, SQLight and FroNTier
• POOL
• C object persistency (via Root or Databases)
  and navigation (via catalogs and collections)
• COOL
• Management of versioned conditions time series
• Layered and complementing each other
• COOL and POOL use CORAL to access databases
• Experiments use COOL to reference conditions
  objects stored in POOL

25
PERSISTENCY FRAMEWORK 2/3

• RECENT ACHIEVEMENTS AND STATUS (a selection)
• CORAL
• Integration with LCG 3D services
• Database service at T0 and T1s
• Replica management via existing LFC service
• now optionally hosted in an LFC catalog (before
  implementation based on XML file)
• Next frontier consolidation
• POOL
• Consolidation and maintenance phase
• Significant work to absorb changes of the core
  infrastructure (f.i. SEAL?ROOT)
• Good news
• consolidation on fewer grid catalogs
  (CLARIFICATION WITHIN THE EXPERIMENTS!)
• LFC seems to be established as baseline
• CMS also developed uses Trivial File Catalog
• POOL did not have to invest in new catalog
  integration
• New functionality added dictionary auto-loading
• Enables the loading-on-demand of the required
  dictionary libraries at run time

26
PERSISTENCY FRAMEWORK 3/3

• RECENT ACHIEVEMENTS AND STATUS, cont.
• COOL
• 21 releases since the beginning (Nov2004)
• recently 1 FTE less !
• In general
• need to ensure continuity with expert manpower
• PF components rely on SEAL component model
• Propose to setup a table of SEAL components and
  plan their evolution

27
MORE ABOUT AA STATUS

• Release frequency
• Typically 2-3 production releases/y
• Bug fix releases - as many as needed to minimize
  TTU (time to user)
• Minimizing TTU a major goal for next year
• AA interfaces to Middleware and Fabric
• Many contact points
• Problem observed due to lack of coordination
• Milestones since last CR

28
CONCLUSIVE REMARKS
29
AA CONCLUSIVE REMARKS 1/2

• abundant amount of work
• certainly important progress and achievements
• the level of progress is not the same in the
  various projects and subprojects
• the intrinsic fragmentary nature of part of the
  projects ? managerial difficulties
• local manpower limitations (f.i. Software Process
  infrastructure)
• the Simulation project
• need to overcome the present difficulties in
  interfacing some Monte Carlo generators to the
  LCG simulation infrastructure
• ROOT project
• properly managed
• appropriate manpower resources
• important achievements, in particular
  consolidation, fast access to data
• Merging of SEAL (Shared Environment for
  Applications at LHC) progressing successfully
• important progress of PROOF, powerful tool to
  extend ROOT to run on a distributed,
  heterogeneous system
• Alice, CMS and LHCb are expressing interest in
  using of PROOF
• clear decisions by the experiments needed

30
AA CONCLUSIVE REMARKS 2/2

• persistency framework project
• key ingredient for LHC computing
• difficulty to access the progress level, limited
  help from the material presented
• important effort by AA to keep the link with the
  experiments and the users strong and effective
• Architects Forum, AA Meetings every 2 weeks
• LCG Generator monthly meetings
• Savannah portal
• manpower
• present level globally very near to the needs
• some reassignment can cure the limitations
  affecting individual projects
• possible manpower crisis in 2008 (retirements and
  contract ends)
• appropriate action be taken in2007 to guarantee
  adequate manpower level in 2008 and beyond