US CMS Software and Computing

1

• US CMS Software and Computing
• Management Breakout
• Matthias Kasemann
• Fermilab
• DOE/NSF Baseline Review
• November 14-15, 2000
• BNL

2
Talk Outline

• Roles and Responsibilities of Project managers
• US CMS Tier2 Planning
• Funding allocations in FY00, FY01 and FY02
• Total Project Request vs. Funding Profiles

3
The Goal of the SC Project

• To provide the software and computing resources
  needed to enable US physicists to fully
  participate in the physics program of CMS
• Allow US physicists to play key roles and exert
  an appropriate level of leadership in all
  stages of computing related activities
• From software infrastructure to reconstruction
  to extraction of physics results
• From their home institutions, as well as at CERN

This capability should extend to physicists
working at their home institutions
4
Participants and Management Entities

• Key Players
• CERN
• International CMS
• US CMS
• US Funding agencies
• US Universities and National Labs (FNAL)
• US Physicists
• Management Entities
• Level 1 Project Manager deputy
• Level 2 Project Manager for Core Applications
  Software
• Level 2 Project Manager for User Facilities
• Level 3 Project Managers for UF (incl. Tier2) and
  CAS
• The Advisory Software and Computing Board -- ASCB
• The Fermilab Director or Designee advised by the
  Project Management Group
• The Fermilab Computing Division
• US funding agencies and Joint Oversight Group

5
Organization of US CMS SC Project

• The project is supported by a project office
  with
• One project engineer and administrative support
• coordinating the activities at all levels
• Overseeing purchases
• Establishing MOUs, SOWs
• Preparing for Reviews
• Tracking budget
• Tracking the project progress

US Funding Agencies and Fermilab (CMS Host
Institution)
Fiscal Authority
Input and Feedback
Liaison
Support
Software and Computing Project Project Manager
US CMS Advisory Software and Computing
Board (USASCB)

Core Applications Software
User Facilities
Reconstruction Detector Software and Physics
Groups
6
Level 1 Project Manager (L1 PM)

• Level 1 Project Manager (L1 PM)
• Based at FNAL, recommended by US CMS Software
  and Computing Board, appointed by FNAL with
  the concurrence of US CMS, DOE and NSF
• Responsible for maintaining the Project within
  scope, cost schedule
• Holds fiscal authority for DOE and NSF funds
• Allocates funds and maintains a management
  organization including an integrated Cost and
  Schedule Plan and mechanisms for procurement,
  tracking, and reporting
• Executes approved project plan, conforming to
  technical and scientific policies of CMS and US
  CMS
• Make adjustments to MOUs and SOWs as required
• Reports variances in scope, cost, or schedule and
  develops plans for corrective action
• Exercise change control, with PMG as appropriate
• Develops annual budget request to DOE/NSF
  reviewed by US ASCB and approved by PMG
• Develops L1 milestones and deliverables, with US
  CMS, CMS, PMG, funding agencies
• Reports to FNAL Director or his designee
• Appoints L2 Managers in consultation with the US
  ASCB and concurrence of the PMG
• At his/her discretion, appoint a deputy or other
  supporting staff
• Liaison with CMS SCTB

7
Level 2 Project Manager for Core Applications
Software

• Defining the milestones and deliverables of the
  subproject
• Developing the technical specifications of each
  component and deliverable of the subproject
• Defining, in consultation with and subject to
  the concurrence of the L1PM, the organizational
  substructure of the subproject. This includes
  defining projects at WBS Level 3 and proposing,
  subject to the approval of the L1PM, the L3
  project leaders
• Developing, under the guidelines provided by the
  L1PM, the annual budget proposal for the
  subproject
• Allocating resources and identifying resource
  imbalances or deficiencies within the subproject
• Delivering the scope of the subproject on
  schedule, within budget, and in conformance with
  the technical specifications and quality
  assurance guidelines of the project
• Organizing and documenting internal reviews
• Being accountable for all funds allocated to the
  subproject
• Providing reports as required by the L1PM,
  USSCB, PMG, etc

8
Level 2 Project Manager for User Facilities

• developing the definition of the milestones and
  deliverables
• developing the technical specifications of each
  component and deliverable of the subproject
• subject to review by the L1PM and the US ASCB
• defining the organizational structure of the
  subproject.
• This includes defining the L3 projects and
  proposing,
• subject to the approval of the L1PM, the L3
  project leaders.
• Each Tier 2 and special purpose center will
  appoint a manager, who will be responsible for
  CMS activities at the Tier 2 or special purpose
  center.
• with the concurrence of the Level 2 manager for
  User Facilities,
• Each Tier 2 and special purpose center will be a
  Level 3 project and have its own Level 3 manager,
  who will report to this L2 manager.
• developing the annual budget proposal
• under the guidelines provided by the L1PM,

9
Level 2 Project Manager for User Facilities

• negotiating adjustments to the MOUs and SOWs
  with Fermilab and Tier 2s
• working with potential candidates for Tier 2 and
  special purpose centers to prepare their
  proposals.
• The parts of proposals for such centers which
  relate to CMS must be approved by the management
  chain described.
• Negotiating annual SOWs with Tier 2 or special
  purpose centers, and with their managing
  organizations as appropriate
• Delivering the scope of the subproject on
  schedule, within budget and within
  specifications
• Organizing internal reviews to assure that
  deliverables will meet the technical
  specifications and the quality assurance
  standards of the project
• being accountable for all funds allocated to the
  subproject
• maintaining the Cost and Schedule Plan for this
  subproject
• providing reports to the L1PM, the US ASCB, and
  the PMG as required

10
Tier 2 Centers

• Tier 2s will partner with Tier 1 which will
  provide support and liaison with CERN.
• US CMS Tier 2 Centers (or the CMS parts of
  multiple customer centers) appear as Level 3
  WBS items under the UF project, and the UF Level
  2 manager has specific responsibilities with
  respect to them
• Tier 1 interfaces to the Tier 2s in the US
• The Tier 2 CMS representative is a L3 manager of
  the US SC project

11
Organization of US CMS Projects
12
US CMS SC Data Grid

• Deploy computing resources as hierarchical grid
• Tier 0 ? Central laboratory computing resources
  (CERN)
• Tier 1 ? National center (Fermilab)
• Tier 2 ? Regional computing center (Universities)
• Tier 3 ? University group computing resources
• Tier 4 ? Individual workstation/CPU
• Data Grid to reflect the predominant role of
  data in the distributed analysis
• This concept and Tiers has been adopted
  throughout Europe by the EU Grid Project
• We are started the implementation of Tier 1 and
  Tier 2 RD systems to
• Perform RD on distributed data analysis
• Support physics and HLT studies
• Support software development

13
US CMS Tier 2 planning

• FY 2000/1 Build basic services HLT milestones
• 1-2 prototype-Tier 2 Studies with ORCA
• FY 2001 Initial Grid system File replication
  service work with Tier 1 Multi-site cached file
  access
• FY 2002 CMS Data Challenges SC TDR and
  Physics TDR
• FY 2003/4 Tier 2 centers at 5-Scale Data
  Challenge second set of sites Physics
  TDR production data grid test
• FY 2005/6 Tier 2 centers 20 Production (Dec.
  2004) at last set of sites CMS Data Challenge
• FY 2006 Production-quality Full distributed
  system Grid System Final Production Shakedown
• RD systems leverage existing resources at
  Universities
• Funding for Tier 2s to come mostly from NSF
  initiatives

14
US CMS Tier 2 Centers

• Discussed at US CMS Collaboration Board, May 20,
  2000
• Agreed start deployment of first prototype Tier
  2 center
• Query sent to each US CMS institute
• Interest and plans to participate in Tier2 RD
  now?
• Who can work on it?
• Result
• Started Caltech UC Davis UCSD 9/00
• Ready to start Florida, early 01 Univ. of
  Iowa Iowa State 4/01
• Start later for production Boston (joint
  center?), Maryland, Minnesota, Wisconsin
• Next steps
• Consolidate Tier2 plans for
• RD on distributed data analysis
• contribute to Monte Carlo production
• leverage from Universities

15
Tier 2 Center Plan
16
First Prototype Tier 2 in California

• Caltech UC San Diego UC Davis
• Hardware Plan Cost
• 80 Dual CPU Disk Linux Nodes 200 k 40 in
  Caltech, 40 in UCSD
• Sun Data Server with RAID Array 30 k partly
  from Caltech
• Tape Library 20 k in Caltech
• LAN Switches 50 k
• Collaborative Infrastructure 10 k
• Installation and Infrastructure 30 k
• Net Connect to Abilene 0 k
• Tape Media and Consumables 10 k
• Staff (Ops and System Support) 50 k
• Gigabit data access at UC Davis 30 k
• Total Estimated Cost (First Year) 430 k
• UCSD cost sharing -50 k
• UC Davis cost sharing -30 k
• Cost 350 k
• Status
• Funding allocation request submitted to start
  first Tier 2 center in California (10/03/00)

17
FY2000 funding request / allocated
Funding allocation for people prorated to match
hiring and availability, ? full load in FY2001
Funding allocation request for prototype Tier 2
submitted in October
18
Allocation so far in FY01

• The basic strategies in prioritizing use of
  limited funds
• 1) to assure the continuation of all CAS people
  on board. The current status is
• Caltech 2 people
• NEU 2 people
• Princeton 1 person
• UC Davis 1 person (3 month funded from FY00
  carry-over)
• FNAL 1 person (2nd offer accepted, starts
  January 2001)
• 2) to assure US CMS SC support by the User
  Facility.
• Four positions filled.
• 3) to provide funds to build up the compute
  resources at the Tier1 regional center.
• 4) to foresee funds to start the project office
  at FNAL
• One Project engineer/Admin support
• 5) Start operating 1. Prototype Tier2 center
  (California)
• 6) in FY01/FY02 deploy 2. Prototype Tier2 center

19
Allocation so far in FY01

• We did not receive a commitment from NSF
• This allocation does not account for several
  essential items required
• 2 additional CAS engineers
• 3.5 additional UF engineers
• Additional 140k for UF Tier1 implementation
• 10 management reserve
• The sum of the missing items is 1.57M

20
Funding FY2001 FY2002

• Goals
• Deliverables for Core Application Software
• Contributions of US share to
• Data Challenges,
• Production exercises for trigger and physics
  studies
• Test beam analyses for detector optimization
• Support US based physicists and physics analysis
  (see mission statement)
• Establish Project office to manage and control
  SC Project ramp up to 2 FTEs (FY02)
• UF Tier1
• Increase infrastructure for US based trigger
  studies, test-beam analysis and general physicist
  support
• Increase UF Tier1 Software engineers from 4(FY00)
  to 11.5(FY03)
• UF Tier2
• Test-bed for data distribution and CMS software
  to perform distributed analysis
• Deploy resources for CMS simulation effort
• Implementing first prototype Tier2 center now
  (California)
• Implement 2. prototype Tier2 center over
  FY01-FY02
• CAS need to increase Software Engineers from
  8(FY00) to 11(FY02)
• Important CMS software milestones in 2001, 2002
  and 2003
• Increase support for US based software
  development

21
Total Project Cost
22
Total Project Cost vs. Funding Profile

• Received draft funding profile from DOE
  5/30/2000
• Assumed NSF funding profile after discussion
• Although detailed planning is available, there
  is a serious shortfall for the next 2 years

23
Total Project Cost vs. Funding Profile

• Received draft funding profile from DOE
  5/30/2000
• Assumed NSF funding profile after discussion
• Implementation in danger to be delayed by 1 year

24
Evaluation of Profiles

• The US CMS SC project is managed and
  coordinated by the L1 PM, who carries budget
  authority
• Funding shared differently for the two
  subprojects
• Tier1 center and Project Office funded by DOE
• Tier2 centers expected to be funded by NSF
• CAS traditionally shared by DOE NSF

25
DOE Funding Project Request
26
NSF Funding Project Request
27
Funding vs. Project Request
28
Funding vs. Project Request
29
Evaluation of Profiles

• NSF part under-funded by 0.5M/year
• Leveraging from University sites must contribute
  to cost of Tier2 centers
• DOE part
• UF dominated by Tier 1 personnel cost
• FY2004 Tier1 center implementation starts, cost
  spread over 3 years

30
Funding Summary

• Need concrete funding profiles (DOE and NSF)
• Difficult to plan else
• (Assumed) Profile recognizes needs for operating
  SC
• Profiles heavily back-loaded towards 2005
• Result
• Funding does not allow to finish critical RD and
  implementation for 2006
• Very hard to reach milestones
• Deliverables for Core Application Software
• Contributions of US share to
• Data Challenges,
• Production exercises for trigger and physics
  studies
• Test beam analyses for detector optimization
• Essential that L1PM has flexibility to allocate
  all funds according to priorities
• Severe impact for US based CMS data analysis
• Will not be able to fully contribute to CMS
  analysis
• Will compromise readiness of CMS and US CMS for
  Physics
• Disadvantage can only slowly be corrected because
  of operational needs

31
SC Project Status/Summary - I

• Project scope, schedule and cost are well
  defined
• Resource loaded WBS exists for the two major
  subprojects
• Resources refined recently
• With input from Intl CMS planning for Hoffmann
  Review at CERN
• After first funding guidelines received from DOE
  in May
• Milestones and timelines exist and are aligned
  with CMS. They are frequent and aggressive enough
  to drive the project and its oversight
• Key management roles are identified and in place
• L1 PM on board, takes over November 2000
• ASCB elected in September 2000
• FNAL oversight Panel in place (awaiting report)
• Key interfaces are worked out in a manner
  consistent with the collaborative nature of CMS
  and US CMS and with their scientific culture
• Key interfaces are worked out with funding
  agencies and CMS
• A Project Management Plan exists
• We separated the stable (project organization
  and oversight) aspects of the plan from the more
  changeable aspects (funding dependent). We
  created two documents
• Project Management and Work Breakdown Structure
• Schedules, Milestones, and Budgets

32
SC Project Status/Summary - II

• The plan draws strength and exploits the synergy
  between
• US Universities and Fermilab
• Software Professionals and physicists
• It takes advantage of and contributes to key
  developments in the US in information technology
• Drive towards a high speed network infrastructure
• Development of ever better network software and
  applications such as grid computing concepts
• It takes advantage from experience and expertise
• through the Tier1 center and host laboratory from
  Run2 experiments for data management, analysis
  and operations
• through the Tier2 centers of the significant
  strengths of US universities in the area of
  computer science and information technology

• These are key elements to its eventual success.
• The US CMS Software Computing Project is
  ready to be baselined.