Applications Area Status and Plans

1
Applications AreaStatus and Plans

• PEB Meeting
• 23 April 2002
• Torre Wenaus, BNL CERN/EP
• LCG Applications Project Manager
• torre.wenaus_at_cern.ch
• CERN mobile phone 16 3062 (079 201 3062)
• http//cern.ch/lcg/peb/applications
• http//lcgapp.cern.ch

2
News

• Dirk Duellmann has accepted to be Project Leader
  for Persistency Framework project
• Positive feedback from all experiments
• Active now
• At least 60 of his time
• Important ATLAS and CMS meetings relating to LCG
  last week
• First Applications Area meeting last Friday
• Discussions with lots of good feedback on
  software process and persistency
• Architects Forum will be meeting for the first
  time tomorrow (to be bi-weekly)
• Renamed with less unappealing name (formerly
  Architects Committee)
• Rene Brun has accepted an invitation to join
• Extended meeting on software development
  infrastructure and some aspects of process May 3
• Hybrid event store workshop June 5-7
• Applications mailing list in use, 74 subscribers

3
ATLAS DB workshop, Orsay, April 15-16

• RTAG members invited (RB, DD attended)
• Interactive discussion of hybrid event store
  (HES) design developed within ATLAS
• Decision to fully align ATLAS hybrid event store
  development with common project immediately
• Intent is to synchronize hybrid event store
  schedule and deliverables with the other
  experiments
• Made possible by a 3-6 month delay in the ATLAS
  DC1 phase 2 schedule that drives the hybrid
  event store schedule
• Will deploy the common project hybrid store
• Expect to contribute 3 FTEs to common project
• Another 3 FTEs for integration and ATLAS-specific
  extensions, specializations
• Interests in both relational and object streaming
  components

4
CMS CPT Week, LCG Plenary last week

• Need for viable non-Objectivity event store
  solution later this year (summer)
• CMS now the driver for near-term hybrid store
  schedule and Im glad we have a near-term driver
• Want to avoid interim third ways will
  participate in and use LCG software
• Can identify 2-3 FTEs to participate in common
  project similar number for CMS integration,
  adaptation
• One person already committed as of yesterday
• Ready to embark on major code redesign
  everything potentially on the table
• And they clearly mean it!
• We want to get started
• And they clearly mean that too! Very proactive,
  which is very welcome
• Strong interest in propagating SCRAM, or could
  envisage putting SCRAM and CMT developers together

5
Active Project Areas

• Software Process
• Launched by SC2 following the RTAG
  recommendations as presented during Launch Week
• Starting from these recommendations, I prepared
  extensive thoughts and plans on software process
  and infrastructure
• Presented and discussed at Applications meeting
  and CMS meeting
• Persistency
• Launched by SC2 following the RTAG report that
  was recently completed and is now under review by
  SC2
• Released to PEB to initiate activity and work
  plan development, subject to any adjustments
  arising from SC2 deliberations
• SC2 will finalize the requirements it delivers to
  the PEB in the May SC2 meeting
• Project activity effectively launched last week
• In both these areas, project must develop work
  plan for submission to SC2, for their review and
  approval

6
New Active Areas in the Near Term

• New RTAGs in the pipeline
• Detector geometry and materials description
• Simulation tools
• Both were top priority in my candidate RTAG
  suggestions last month

7
RTAG Detector geometry materials description

• Write the product specification for detector
  geometry and materials description and modeling
  services
• Specify scope eg. Services to define, provide
  transient access to, and store the geometry and
  materials descriptions required by simulation,
  reconstruction, analysis, online and event
  display applications, with the various
  descriptions using the same information source
• Identify requirements including end-user needs
  such as ease and naturalness of use of the
  description tools, readability and robustness
  against errors e.g. provision for named constants
  and derived quantities
• Explore commonality of persistence requirements
  with conditions data management
• Interaction of the DD with a conditions DB. In
  that context versioning and 'configuration
  management' of the detector description,coherence
  issues..
• Identify where experiments have differing
  requirements and examine how to address them
  within common tools
• Address migration from current tools

8
RTAG Simulation Tools

• To identify Software components that are between
  GEANT4 and the experiment simulation applications
  that could benefit from a common development
  effort.
• - For example
• Components for managing geometry and sensitive
  detectors
• Magnetic field service
• Components for configuring and steering Geant4
  and installation of user actions
• Visualisation components
• Interactivity (scripting and GUI)
• To propose a decomposition of the relative
  responsibilities between the experiment
  application frameworks, the G4 toolkit and
  components in this middle layer.
• N.B. The mandate is not to overlap with the
  proposed Geant4 HEPURC (user requirements
  committee), but to address issues outside of
  Geant4

9
Top Priorities
Green done or in the works
Initiate First half 2002

• Establish process and infrastructure
• Nicely covered by software process RTAG
• Address core areas essential to building a
  coherent architecture, and develop the
  architecture
• Object dictionary
• Persistency
• Interactive frameworks, analysis tools (also
  driven by assigning personnel optimally)
• Address priority common project opportunities
• Driven opportunistically by a combination of
  experiment need, appropriateness to common
  project, and the right moment (existing but not
  entrenched solutions in some experiments)
• Detector description and geometry model
• Driven by need and available manpower
• Simulation tools

10
Software Process and Infrastructure

• Extensive material developed, based around SC2
  input, and also drawing on various discussions
  and material from the experiments, IT, and ROOT
• Posted Apr 5 but little feedback. More in
  presentations last week
• Thoughts for discussion, to develop into
  decisions
• Covering
• General policy
• Methodology (best of XP, RUP)
• Developing and releasing code
• Automated build system
• Testing, coding standards, software metrics
• Software distribution and central support
• Documentation
• Roles to be filled
• Tools
• Clear from last weeks discussions a workshop
  soon should be very productive. Planning for Fri
  May 3 afternoon.

11
(No Transcript)
12
(No Transcript)
13
Setting Up Infrastructure

• LCG Applications server set up and (since last
  night) operational
• lcgapp.cern.ch
• Dual RAID mirrored disks integrated with IT
  monitoring selective backup
• Initial host for web, CVS, MySQL, other services
• Apache, PHP, MySQL, LXR, phpMyAdmin in place,
  more to come
• Agreement with CERN IT is to migrate the services
  to official CERN IT platforms and services when
  it is practical to do so
• e.g. IT CVS services being planned
• Thanks to David Stickland and CMS for help in
  getting this going (via an offer of CMS server as
  the host)
• And to Les Robertson and David Foster for pushing
  through a well configured server
• LCG Applications AFS area is set up
• /afs/cern.ch/project/lcg/app
• Web area, documents, cvs,

14
Setting Up Tools Infrastructure

• Full time process/infrastructure support person
  from existing IT staff still in negotiation
• Experienced initial person to get things off to a
  fast start and transfer expertise to new LCG
  staff
• Need this person right away! Requested a person
  three weeks ago
• Possible help from EP
• Assignments from new LCG staff still to be
  settled as well
• Plans being developed with IT/API and Geant4 to
  share tool experts (maybe also installations)
• Again, transferring expertise to new LCG staff
• Experiment contact(s) requested to plug us into
  the tools and infrastructure available
• Will pick and choose the best to adopt or adapt
• Direct participation from the experiments is
  welcome

15
Immediate Process/Infrastructure Tasks

• Establish project AFS area, HEPix
  group/environment, general environment
  configuration
• Establish strawman repository, with ancillary
  services
• Sourceforge investigation. Trial?
• Code browsing tools LXR, cvsweb, Bonsai(?)
• Coding standards assemble rules, guidelines and
  rule set from existing experiments/projects
• Web server with Apache tomcat php/mysql
• MySQL service
• Infrastructure for uniform web based
  access/support/doc for third party tools
• Needed tools and packages ROOT, Python, Perl,
  ...
• Documentation infrastructure - e.g. doxygen
• Bug reporting system - Remedy or other? survey
  and implement
• SCRAM/CMT assessment soon
• Packaging assessment soon
• UML diagramming tool

Green done
16
Persistency Framework RTAG Outcomes

• Reasonable RTAG agreement on design criteria
• Component oriented
• Persistence for C data models
• Allow multiple persistence technologies
• No compile/link time dependencies on technology
• Transient types may have multiple persistent
  representations
• Proposed initial project
• Deliver components of a common file-based
  streaming layer sufficient to support 4
  experiments
• Able to support all kinds of data not specific
  to event data
• With management of resulting files hosted in a
  relational layer
• Nothing precludes implementation using open
  source such as MySQL

17
RTAGs First Component Diagram
18
IMCatalog
TGrid
TChain TEventList TDSet
IFCatalog
TBuffer, TMessage, TRef, TKey
IPers
ICache
IPers
TTree
One possible mapping to a ROOT implementation (un
der discussion)
IReflection
ICnv
C
TClass, etc.
TStreamerInfo
IPReflection
IPlacement
TFile, TDirectory TSocket
IReadWrite
TFile
19
Persistency project priorities

• Develop the proposed component breakdown into a
  concrete design with
• well defined interfaces on technology-neutral
  components
• prioritized implementation plan which may involve
  doing some (external) interfaces before other
  (internal) ones
• allocatable work packages
• Specify relational implementation technologies
• Cutting through oblique and cautious language
  Oracle, or MySQL, or both
• Ultimately I believe we will need both (but we
  certainly need a lightweight implementation
  early)
• Initially pursuing both, if we have the
  personnel, will make for a better-designed dual
  technology implementation
• And we may have the personnel (IT relational
  interest in the experiments)

20
Persistency priorities (2)

• Develop architecture and component infrastructure
• Component architecture inheritance hierarchy,
  factories, mapping to shared libraries, etc.
• Mapping to experiment frameworks
• Accommodation of multiple technologies
• Infrastructure and services, e.g. error handling
• Will evolve

21
Persistency Getting started

• Finish settling resource commitments
• ATLAS 3, CMS 3, LHCb , ALICE piece of Fons,
  ROOT ?, IT 2, LCG few hires
• Settle feature set(s) and requirements for
  initial prototype
• Settle who does what when
• Ultimately including maintenance and production
  services as well as development
• Start core group working on immediate priorities
• When initial component and interface design and
  implementation strategy is in place, farm out
  work packages
• Deliver by sometime in September?

22
Architects Forum

• Architecture discussions of the Persistency RTAG
  will surely carry over into discussions in the
  Architects Forum beginning with tomorrows
  meeting, I expect
• I hope that discussions in the concrete context
  of the persistency framework will aid in mutual
  understanding and convergence
• Well see!
• Top priority is turning the component breakdown
  into a concrete design with defined interfaces
  and a prioritized implementation plan
• I am confident we will deliver a persistency
  framework in a timely way, and concretely address
  many architectural issues in the process

23
Planning

• Project breakdown and schedule implemented in the
  planning tool and in early development in terms
  of content
• Accessible from the applications web page
• High level milestones for LHCC proposed and
  passed through PEB
• Need to establish one for the initial prototype
  of hybrid event store later this year sometime
  in Sept?
• June 2003 General release of hybrid event store
• Nov 2003 Distributed production environment
  using grid services
• May 2004 Distributed end-user interactive
  analysis from a Tier 3
• March 2005 Full function release of persistency
  framework
• Now developing level 2 schedule and deliverables
  in process/infrastructure and persistency

24
Schedule (in development)
25
Schedule (in development)
26
(No Transcript)
27
(No Transcript)
28
Personnel

• Still a molehill, people are trickling in
• Assignments to be settled
• Rafi Yaari established doing useful prototyping
  work, gprof and (later) TAU currently for Geant4
• Looking to new hiring round in UK for important
  augmentation
• Process and infrastructure, support roles
• Persistency, including infrastructure support,
  Tier 0 needs, maintenance and user/production
  support
• Future projects (e.g. simulation related)
• 8? expected to be in applications
• Also getting or expecting contributions at lower
  levels from several other countries

29
Conclusions

• Project is active in some important areas
• Process and infrastructure needed to do real work
• A major project in persistency, with
  participation planned by all four experiments (no
  single host experiment this time)
• Participation beginning to be quantified (IT,
  ATLAS, CMS) need to complete this
• Need to combine experiment needs with realistic
  goals to produce plan and schedule
• A first usable version later this year,
  essentially as soon as possible
• Regular meetings and activities are beginning
• Ready to engage participation by experiments, IT,
  ROOT
• Process, infrastructure, persistency
• Also need to actively engage the wider community
  in participation, collaboration
• BaBar already expressing interest in formalizing
  participation/collaboration on the persistency
  project
• PEB thoughts on how to formalize?