title.open ( ); revolution {execute};

1
Tony Doyle
GridPP Oversight Committee 15 May
2002
2
Document Mapping

• Exec Summary
• Goals
• Metrics for success
• Project Elements
• Risks/Dependencies (and mechanisms)
• Summary

• PMB-02-EXEC
• PMB-01-VISION
• PMB-02-EXEC
• Gantt Charts, PMB-05-LCG,
  TB-01-Q5-Report, TB-02-UKRollout,
  PMB-06-TierAstatus, PMB-04-Resources
• PMB-03-STATUS, PMB-07-INSTRUMENTS
• PMB-02-EXEC

3
Outline

• The Vision Thing
• Grid
• Scale
• Integration
• Dissemination
• LHC Analyses
• Other Analyses

• DataGrid
• LCG
• Interoperability
• Infrastructure
• Finanaces
• Summary

4
GridPP Documents
5
GridPP Vision

• From Web to Grid - Building the next IT
  Revolution
• Premise
• The next IT revolution will be the Grid. The
  Grid is a practical solution to the
  data-intensive problems that must be overcome if
  the computing needs of many scientific
  communities and industry are to be fulfilled over
  the next decade.

• Aim
• The GridPP Collaboration aims to develop and
  deploy the largest-scale science Grid in the UK
  for use by the worldwide particle physics
  community.

Many Challenges.. Shared distributed
infrastructure For all experiments
6
GridPP Objectives

• 1. SCALE GridPP will deliver the Grid software
  (middleware) and hardware infrastructure to
  enable the testing of a prototype of the Grid for
  the LHC of significant scale.
• 2. INTEGRATION The GridPP project is designed to
  integrate with the existing Particle Physics
  programme within the UK, thus enabling early
  deployment and full testing of Grid technology
  and efficient use of limited resources.
• 3. DISSEMINATION The project will disseminate
  the GridPP deliverables in the multi-disciplinary
  e-science environment and will seek to build
  collaborations with emerging non-PPARC Grid
  activities both nationally and internationally.
• 4. UK PHYSICS ANALYSES (LHC) The main aim is to
  provide a computing environment for the UK
  Particle Physics Community capable of meeting the
  challenges posed by the unprecedented data
  requirements of the LHC experiments.
• 5. UK PHYSICS ANALYSES (OTHER) The process of
  creating and testing the computing environment
  for the LHC will naturally provide for the needs
  of the current generation of highly data
  intensive Particle Physics experiments these
  will provide a live test environment for GridPP
  research and development.

• 6. DATAGRID Grid technology is the framework
  used to develop this capability key components
  will be developed as part of the EU DataGrid
  project and elsewhere.
• 7. LCG The collaboration builds on the strong
  computing traditions of the UK at CERN. The CERN
  working groups will make a major contribution to
  the LCG research and development programme.
• 8. INTEROPERABILITY The proposal is also
  integrated with developments from elsewhere in
  order to ensure the development of a common set
  of principles, protocols and standards that can
  support a wide range of applications.
• 9. INFRASTRUCTURE Provision is made for
  facilities at CERN (Tier-0), RAL (Tier-1) and use
  of up to four Regional Centres (Tier-2).
• 10. OTHER FUNDING These centres will provide a
  focus for dissemination to the academic and
  commercial sector and are expected to attract
  funds from elsewhere such that the full programme
  can be realised.
• (. WHAT WE SAID WE COULD DO IN THE PROPOSAL)

7
Grid A Single Resource
GRID A unified approach

• Many millions
• of events

• Many samples

• Various conditions

GRID A unified approach

• Peta Bytes of data storage

Distributed resources

• Many 1000s of computers required

Worldwide collaboration

• Heterogeneous operating systems

8
Grid - Whats been happening?
GRID A unified approach

• A lot
• GGF4, OGSA and support of IBM (and others)
• as opposed to .NET development framework and
  passports to access services
• Timescale? September 2002
• W3C architecture for web services
• Chose (gzipped) XML as opposed to other solutions
  for metadata descriptions and web-based
  interfaces
• linux
• as opposed to other platforms lindows??
• C (experiments) and C, Java (middleware) APIs
• mono - Open Source implementation of the .NET
  Development Framework??

9
GridPP Context
Provide architecture and middleware
Future LHC Experiments
Running US Experiments
Build Tier-A/prototype Tier-1 and Tier-2 centres
in the UK and join worldwide effort to develop
middleware for the experiments
Use the Grid with simulated data
Use the Grid with real data
10
EDG TestBed 1 Status
GRID A unified approach

• Web interface showing status of (400) servers
  at testbed 1 sites

GRID extend to all expts
11
LHC computing at a glance
1. scale

• The investment in LHC computing will be massive
• LHC Review estimated 240MCHF (before LHC delay)
• 80MCHF/y afterwards
• These facilities will be distributed
• Political as well as sociological and practical
  reasons

Europe 267 institutes, 4603 users Elsewhere
208 institutes, 1632 users
12
GridPP funded Staff at CERN
7. LCG

• IT/ADC
• Ian Neilson from 16th April. Working on
  certificate authority then scaling up EDG
  testbed.
• IT/API
• Jacek Generowicz from 1st Feb. Working on GEANT4,
  in particular, tracking visualisation
• Maria Girone from 1st April. Settling in.
• IT/FIO
• John Hearns from 1st April. Configuration
  installation management relates to WP4 effort
  and LCFG.
• Bill Tomlin from 16th April. State Management.

Little acorns..
13
RTAG Status
7. LCG

• 6 RTAGs created to date
• RTAG1 (Persistency Framework status completed)
• RTAG2 (Managing LCG Software status running)
• RTAG3 (Math Library Review status running)
• RTAG4 (GRID Use Cases status starting)
• RTAG5 (Mass Storage status running)
• RTAG6 (Regional Centres status starting)
• Two more in advanced state of preparation
• Simulation components
• Data Definition Tools

14
Applications area has momentum
7. LCG

• Torre Wenaus started as Area Manager in March.
• Activities in the Applications area are now well
  organised. See http//cern.ch/lcg/peb/applications
  .
• Weekly Applications area meetings initiated
  together with Architects Forum.
• Active work for Software Process and Persistency.
  Work plan must be developed for SC2 approval.

15
Fabrics Grid Deployment
7. LCG

• LCG Level 1 Milestone deploy a Global Grid
  Service within 1 year
• sustained 24 X 7 service
• including sites from three continents
• identical or compatible Grid middleware and
  infrastructure
• several times the capacity of the CERN facility
• and as easy to use
• Ongoing work at CERN to increase automation and
  streamline configuration, especially for
  migration to RedHat 7.2.
• Aim to phase out old CERN solutions by mid-2003.

16
LCG Timeline
1. timescale
Prototype of Hybrid Event Store (Persistency
Framework)
Hybrid Event Store available for general users
applications
Distributed production using grid services
Full Persistency Framework
Distributed end-user interactive analysis
LHC Global Grid TDR
grid
50 prototype (LCG-3) available
LCG-1 reliability and performance targets
First Global Grid Service (LCG-1) available
17
Be a part of this?

• Notes
• 1. The minimum period for LTA is 3 months. It is
  expected that a work programme will be typically
  for 6 months (or more).
• 2. Prior DataGrid and LHC (or other) experiments'
  Grid work are normally expected.
• 3. It is worthwhile reading
• http//cern.ch/lcg/peb/applications
• in order to get an idea of the areas covered, and
  the emphasis placed, by the LCG project on
  specific areas (building upon DataGrid and LHC
  experiments' developments).
• 4. Please send all enquiries and proposals to
• Tony Doyle lta.doyle_at_physics.gla.ac.ukgt and
• Tony CASS lttnt_at_mail.cern.chgt

• LCG Development
• Long Term Attachment at CERN
• This will enable Grid developments in the UK to
  be (more) fully integrated with long-term Grid
  development plans at CERN.
• The proposed mechanism is
• 1. submit a short one-page outline of current and
  proposed work, noting how this work can best be
  developed within a named team at CERN, by e-mail
  to the GridPP Project Leader (Tony Doyle) and
  GridPP CERN Liaison (Tony Cass).
• 2. This case will be discussed at the following
  weekly GridPP PMB meeting and outcomes will be
  communicated as soon as possible by e-mail
  following that meeting.

18
Summary of LCG
7. LCG

• Project got under way early this year
• Launch workshop and early RTAGs give good input
  for high-level planning
• to be presented to LHCC in July
• New plan takes account of first beam in 2007
• No serious problems foreseen in synchronising LCG
  plans with those of the experiments
• Collaboration with the many Grid projects needs
  more work
• Technical collaboration with the Regional Centres
  has to be established
• Recruitment of special staff going well (but need
  to keep the recruitment momentum going)
• Serious problem with materials funding

19
Building upon Success
6. DataGrid

• The most important criterion for establishing the
  status of this project was the European
  Commission review on March 1st 2002.
• The review report of project IST-2000-25182
  DATAGRID is available from PPARC.
• The covering letter states As a general
  conclusion, the reviewers found that the overall
  performance of the project is good and in some
  areas beyond expectations.
• The reviewers state The deliverables due for the
  first review were in general of excellent
  quality, and all of them were available on time
  All deliverables are approved. The project is
  doing well, exceeding expectations in some areas,
  and coping successfully with the challenges due
  to its size.

20
6. DataGrid
21
WP1 Workload Management (Job Submission)
6. DataGrid
1. Authentication grid-proxy-init 2. Job
submission to DataGrid dg-job-submit 3.
Monitoring and control dg-job-status dg-job-canc
el dg-job-get-output 4. Data publication and
replication (WP2) globus-url-copy, GDMP 5.
Resource scheduling use of CERN MSS JDL,
sandboxes, storage elements
Important to implement this for all
experiments
22
WP2 - Spitfire
6. DataGrid
23
WP3 - R-GMA
6. DataGrid
24
WP4 - LCFG
6. DataGrid
25
WP5 Storage Element
Data Flow Diagram for SE
6. DataGrid

• A consistent interface to MSS.
• MSS
• Castor
• HPSS
• RAID arrays
• SRM
• DMF
• Enstore
• Interfaces
• GridFTP
• GridRFIO
• /grid
• OGSA

26
WP6 - TestBed 1 Status
6. DataGrid

• Web interface showing status of (400) servers
  at testbed 1 sites

GRID extend to all expts
27
WP7 Network Monitoring
6. DataGrid
28
WP7 - EDG Authorisationgrid-mapfile generation
VODirectory
AuthorizationDirectory
6. DataGrid
29
WP8 - Applications
6. DataGrid

• 1. Realistic Large-Scale Tests
• Reliability! Need reliable dg-job- command suite
• 2. Data management
• Reliability! Need reliable gdmp- command suite,
  file-transfer commands
• 3. Mass Storage Support
• Working access to MSS (CASTOR and HPSS at CERN,
  Lyon)
• 4. Lightweight User Interface
• Put on a laptop or std. Desktop machine

• 5. Portability
• Demonstrable portability of middleware a) use
  other resources, b) debugging
• 6. Scratch Space
• Job requests X amount of scratch space to be
  available during execution, system tells job
  where it is
• 7. Output File Support
• JDL support for output files specify where
  output should go in JDL, not in job script

30
Expt. Feedback
4. and 5. Expts
31
5. Other Expts
8. Interoperability
Minimal e-Bureaucracy
32
GRID JOB SUBMISSION External User Experience
5. Other Expts
33
Things Missing, apparently
5. Other Expts
34
Expt. Feedback
4. and 5. Expts
35
GridPP Poster
3. Dissemination
36
Tier 1/A EDG Poster
3. Dissemination
37
BaBar Poster
3. Dissemination
38
LHCb Poster
3. Dissemination
39
ScotGRID Poster
3. Dissemination
40
Identifiable Progress...
3. Dissemination
t0
t1
41
WebLog
Allows every area/sub group to have its own
'news' pages
42
GridPP Core e-Science Centres
3. Dissemination
Written formally to all e-Science centres
inviting contact and collaboration with GridPP.

• NeSC
• Close ties, hosted 2nd GridPP Collaboration
  Meeting, Collaboration on EDIKT Project?
  Training...
• Belfast
• Replied but not yet up and running.
• Cambridge
• Close ties, hosted 3rd GridPP Collaboration
  Meeting. Share one post with GridPP. Will
  collaborate on ATLAS Data Challenges.
• Cardiff
• Replied - contacts through QM (Vista) and Brunel
  GridPP Group.

43
GridPP Core e-Science Centres
3. Dissemination

• London
• No formal reply but close contacts through IC HEP
  Group. IC will host 5th GridPP Collaboration
  Meeting.
• Manchester
• No collab. projects so far. Manchester HEP Group
  will host 4th GridPP Collaboration Meeting.
• Newcastle
• In contact - Database projects?
• Oxford
• Close ties, collaboration between Oxford HEP
  Group and GridPP on establishment of central
  Tier-2 centre? CS/Core-GridPP-EDG links? Probably
  host 6th GridPP Collaboration Meeting.
• Southampton
• Replied but no collaboration as yet.

44
GridPP Context (Externally)
8. Interoperability
45
GLUE
8. Interoperability

• How do we integrate with developments from
  elsewhere in order to ensure the development of a
  common set of principles, protocols and standards
  that can support a wide range of applications?
• GGF
• Within the Particle Physics community, these
  ideas are currently encapsulated in the Grid
  Laboratory Uniform Environment (GLUE).
• Recommend this as a starting point for the wider
  deployment of Grids across the Atlantic. See
  http//www.hicb.org/glue/GLUE-v0.1.doc (Ruth
  Pordes et al.)

46
8. Interoperability
47
UK Tier-A/prototype Tier-1 Centre
9. Infrastructure

• Roles
• Tier-A Centre for BaBar
• EDG testbed(s)
• LCG prototype Tier-1 Centre
• prototype Tier-1 for LHC experiments (Data
  Challenges independent of LCG development)
• Interworking with other UK resources (JIF, JREI,
  eSC) UK portal
• existing LEP, DESY and non-accelerator
  experiments
• Purchases
• First year Hardware Advisory Group (HAG1)
• Determine balance between cpu, disk, and tape
• Experts on specific technologies
• Propose more HAGs (2 and 3)..
• Needs to be successful in all roles...

48
UK Tier-A/prototype Tier-1 Centre
9. Infrastructure

• Equipment delivered and under user test
• 895K committed, 700K spent
• The 156 dual cpu 1.4GHz rack mounted systems
• integrated into the existing CSF cluster
• A special queue for testing
• BaBar have tested a number of their programs.
• 23 disk servers 2x800GB usable each
• 3 servers in use by BaBar
• CMS cannot use (due to licence problems Obj on
  RH7.2)
• 24 port Gbit switch in addition to existing one
• Extra STK 9940 tape drive
• Start this years tender much earlier.
• Need guidelines from PMB for how we release

49
Rollout of the UK Grid for PP
9. Infrastructure

• Operational stability of GridPP middleware
  Testbed team
• The gang of four Andrew McNab, Steve Traylen,
  Dave Colling (other half) and Owen Moroney
• Ensures the release of Testbed quality EDG
  software
• documentation
• lead for other system managers in terms of
  implementation
• pre-defined software cycle releases (2 months..)
• Subject of the Rollout Plan Planning for EDG
  Testbed software deployment and support at
  participating UK sites (Pete Clarke, John
  Gordon)
• LCG is the proposed mechanism by which the EDG
  testbed at CERN becomes an LCG Grid Service. The
  evolution of the EDG testbed to the LCG Grid
  Service will take account of both EDG and US grid
  technology. Need to take account of this..

50
Longer Term..
9. Infrastructure

• LCG Grid Service
• Takes account of EDG and US grid technology
• A large-scale Grid resource, consistent with the
  LCG timeline, within the UK.
• Scale in UK? 0.5 Pbytes and 2,000 distrib.
  CPUs GridPP in Sept 2004
• 50 prototype

51
17m 3-Year Project
Dave Britton
10. Finances

• Five components
• Tier-1/A Hardware ITD Support Staff
• DataGrid DataGrid Posts PPD Staff
• Applications Experiments Posts
• Operations Travel Management e Early
  Investment
• CERN LCG posts Tier-0 e LTA

52
1. Recruitment

• EDG Funded Posts (Middleware/Testbed)
• All 5 in post 1 additional
• EDG Unfunded Posts (Middleware/Testbed)
• 15 out of 15 in post
• GridPP Posts (Applications Tier1/A)
• Allocated Dec 2001
• 13 out of 15 in post
• CERN Posts
• First Round 105 Applicants, 12 Offers, 9
  Accepted
• 4 in Applications, 2 Data Management, 3 Systems
• Second Round 140 applicants, 9 Offers
• Third Round 70 Applicants
• Aim 28 posts

53
2. Monitoring Staff Effort SM
Robin Middleton
54
3. Progress towards deliverables..
Pete Clarke
55
-1. Next steps..
10. Finances

• O(100k)
• e CLRC support through to Sept 04
• Other experiments unfunded in peer review
  process
• Tier-2 centres unfunded initially
• 2.3m
• eDIKT (e-Data, Information and Knowledge
  Transformation) SHEFC Particle Physics
  application area - assignment of two (of twelve)
  FTEs in initial planning. Discussions ongoing
  with EPCC.
• O(100m)
• The first call for Framework VI will be early
  next year.
• Call out now for expressions of interest for new
  networks and integrated projects.
• Draft document led by David Williams (CERN)
  Enabling Grids and e-Science in Europe plans
  to extend the current paradigms with CERN at its
  focus as the European e-Science Centre.
• We believe this is the right approach.
• Incorporates the UKs e-Science agenda, adding a
  European dimension. It also recognises the
  central role of CERN and builds upon the recent
  successes of EDG.
• PPARC Contact Neil Geddes

56
Testbed Status Overview

• Metrics

Green Dot G1.1.3 G2.0(b) EDG-CE Babar-CE Bir
mingham y y y Bristol y y y y y Brunel y y
Cambridge y Edinburgh y y Glasgow y y Imperial
y y y Lancaster y y Liverpool y y Manche
ster y y y y y Oxford y y QMUL y y y RAL y y
y y RHUL y y UCL y
Andrew McNab - Manchester HEP - 10 May 2002
57

• Metrics

What is in place in the UK testbed? (an RB
centric view of the world) Only GridPP and Babar
VOs
Bristol
Replica Catalogue
58
Grid Support Centre
8. Interoperability

• UKHEP CA uses primitive technology
• It works but takes effort
• 201 personal certs issued (98 still valid)
• 119 other certs issued (93 still valid)
• GSC will run a CA for UK escience CA
• Uses openCA Registration Authority uses web
• We plan to use it
• Namespace identifies RA, not Project

• Metrics

• UK
• e-Science
• Certification
• Authority

• Through GSC we have access to skills of CLRC eSC
• Use helpdesk to formalise support later in the
  rollout

59
Summary

• A vision is only useful if its shared
• Grid success is fundamental for PP
• Scale in UK? 0.5 Pbytes and 2,000 distrib. CPUs
  
• GridPP in Sept 2004
• Integration ongoing..
• Dissemination external and internal
• LHC Analyses ongoing feedback mechanism..
• Other Analyses closely integrated using EDG
  tools

• DataGrid - major investment must be (and is
  so far) successful
• LCG Grid as a Service
• Interoperability sticky subject
• Infrastructure Tier-A/1 in place, Tier-2s to
  follow
• Finances (very well) under control
• Next steps on framework VI..
• CERN EUs e-science centre?
• Co-operation required with other
  disciplines/industry
• Monitoring mechanisms in place
• Emphasis on deliverables

60
Executive2 Summary

• Significant progress...
• Project is now well defined in a broad sense and
  is progressing on a series of fronts.
• We have responded and outlined our plans to
  address the concerns of the last OC concerning
• WP5
• Rollout plan
• Monitoring instruments
• Metrics for success.

• The project has demonstrated progress in
• Widespread deployment of EDG testbeds in the UK
• Integration with specific experimental areas
  (BaBar, UKDMC and LISA) and
• Demonstrating Grid deployment in the UK at the
  NeSC opening.
• We see various challenges ahead
• Development of more detailed metrics and
  monitoring of outputs
• Management of changes due to external
  developments (e.g. OGSA)
• Development of Tier-2 deployment
• Engagement of the UK HEP community and
• Future funding initiatives such as Framework VI.

61
UML Diagram
GRID A unified approach
True Vision??