HEP: Networks, Grids and Collaborative Systems for Global VOs

1

• HEP Networks, Grids and Collaborative
  Systems for Global VOs

Harvey B. Newman for the
Caltech SLAC LANL Team SC2003 Bandwidth
Challenge, PhoenixNovember 19, 2003
2
Next Generation Networks and Grids for HEP
Experiments
Worldwide Analysis Data explored and analyzed by
thousands of globally dispersed scientists, in
hundreds of teams

• Providing rapid access to event samples and
  analyzed physics results drawn from massive data
  stores
• From Petabytes in 2003, 100 Petabytes by 2007-8,
  to 1 Exabyte by 2013-5.
• Providing analyzed results with rapid turnaround,
  bycoordinating and managing large but LIMITED
  computing, data handling and NETWORK resources
  effectively
• Enabling rapid access to the Data and the
  Collaboration
• Across an ensemble of networks of varying
  capability
• Advanced integrated applications, such as Data
  Grids, rely on seamless operation of our LANs
  and WANs
• With reliable, monitored, quantifiable high
  performance

3
Large Hadron Collider (LHC) CERN, Geneva 2007
Start

• pp ?s 14 TeV L1034 cm-2 s-1
• 27 km Tunnel in Switzerland France

CMS
TOTEM
pp, general purpose HI
First Beams April 2007 Physics Runs from
Summer 2007
ALICE HI
LHCb B-physics
Atlas
ATLAS
Data Challenges Computing and Physics TDRs 2004-5
4
Four LHC Experiments The
Petabyte to Exabyte Challenge

• ATLAS, CMS, ALICE, LHCBHiggs New particles
  Quark-Gluon Plasma CP Violation

Data stored Tens of PB 2008 To 1 EB by
2015 CPU
Hundreds of TFlopsto PetaFlops
5
LHC Higgs Decay into 4 muons (Tracker only)
1000X LEP Data Rate
109 events/sec, selectivity 1 in 1013 (1 person
in a thousand world populations)
6
LHC Data Grid HierarchyDeveloped at Caltech
Emerging Vision A Richly Structured, Global
Dynamic System
7
Production BW Growth of Intl HENP Network Links
(US-CERN Example)

• Rate of Progress gtgt Moores Law. (US-CERN
  Example)
• 9.6 kbps Analog (1985)
• 64-256 kbps Digital (1989 - 1994)
  X 7 27
• 1.5 Mbps Shared (1990-3 IBM)
  X 160
• 2 -4 Mbps (1996-1998) X
  200-400
• 12-20 Mbps (1999-2000)
  X 1.2k-2k
• 155-310 Mbps (2001-2)
  X 16k 32k
• 622 Mbps (2002-3) X 65k
• 2.5 Gbps (2003-4) X
  250k
• 10 Gbps ? (2005)
  X 1M
• A factor of 1M over a period of 1985-2005 (a
  factor of 5k during 1995-2005)
• HENP has become a leading applications driver,
  and also a co-developer of global networks

8
HENP Major Links Bandwidth Roadmap (Scenario)
in Gbps
Continuing the Trend 1000 Times Bandwidth
Growth Per DecadeWe are Rapidly Learning to Use
Multi-Gbps Networks Dynamically
9
HEP is Learning How to Use Gbps Networks Fully
Factor of 50 Gain in Max. Sustained TCP Thruput
in 2 Years, On Some USTransoceanic Routes

• 9/01 105 Mbps 30 Streams SLAC-IN2P3 102
  Mbps 1 Stream CIT-CERN
• 5/20/02 450-600 Mbps SLAC-Manchester on OC12
  with 100 Streams
• 6/1/02 290 Mbps Chicago-CERN One Stream on
  OC12 (mod. Kernel)
• 9/02 850, 1350, 1900 Mbps Chicago-CERN
  1,2,3 GbE Streams, 2.5G Link
• 11/02 LSR 930 Mbps in 1 Stream
  California-CERN, and California-AMS FAST
  TCP 9.4 Gbps in 10 Flows California-Chicago
• 2/03 LSR 2.38 Gbps in 1 Stream
  California-Geneva (99 Link Utilization)
• 5/03 LSR 0.94 Gbps IPv6 in 1 Stream
  Chicago- Geneva
• TW SC2003 5.65 Gbps (IPv4), 4.0 Gbps (IPv6)
  in 1 Stream Over 11,000 km

10
Fall 2003 Ultraspeed TCP Data Stream Across the
Atlantic 10 GbE over 11,000 km

• Terabyte Transfers by the Caltech-CERN Team
• Nov 18 4.00 Gbps IPv6 Geneva-Phoenix (11.5
  kkm)
• Oct 15 5.64 Gbps IPv4 Palexpo-L.A. (10.9 kkm)
• Across Abilene (Internet2) Chicago-LA, Sharing
  with normal network traffic
• Peaceful Coexistence with a Joint
  Internet2- Telecom World VRVS Videoconference

European Commission
Juniper,Level(3)Telehouse
10GigE NIC
11
World Laboratory Experiment BW Challenge
Performance Summary

• Utilized all Three Wavelengths, from Three
  BoothsCaltech, SLAC/FNAL, Los Alamos CERN,
  U. Manchester, U. Amsterdam
• Traffic to or from CENIC, Caltech, SLAC/Palo
  Alto, TeraGrid, Starlight, Netherlight/UvA,
  Georgia Tech, CERN/Geneva, and Tokyo
• ONLY TCP traffic (FAST (primary), RENO, HSTCP,
  Scalable)
• High speed data transfer application Clarens,
  Grid-Enabled Analysis
• Disk to Disk transfers
• Two Streams of 200 MB/s each between LA (Cenic
  PoP) and Phoenix
• Memory to memory transfer
• Not enough CPU resources to run more disk to disk
  transfers
• Link utilization (Typical Single Streams 4-5
  Gbps)
• Max. rate on CENICs Wave 10.0 Gbps (output)
• Max. rate on Abilenes Wave 8.7 Gbps (output)
• Max. rate on TeraGrids Wave 9.6 Gbps (input)
• Sponsors DOE, NSF, SCINet, Cisco, Level(3),
  Nortel, Starlight, CENIC, Internet2, NLR, Intel,
  HP, ASCI
• Servers Used Dual Itanium2, Opteron, Xeon

12
Bandwidth ChallengeNetwork MAP
13
Main data transport protocol used for the
bandwidth challenge FAST TCP

• Reno TCP has poor performance w/largewindows
• FAST
• Uses end-to-end delay and loss
• Very high link utilization (gt90 in theory)
• Sender side modification only
• Fast convergence to equilibrium
• Achieves any desired fairness, expressed by
  a utility function
• Pacing reducing burstiness

95
1G
average utilization
27
10Gbps
19
txq100
txq100
txq10000
Linux TCP Linux TCP FAST
capacity 155Mbps, 622Mbps, 2.5Gbps, 5Gbps,
20Gbps 100 ms round trip latency 100 flows J.
Wang (Caltech, June 02)
Capacity 1Gbps 180 ms round trip latency1
flow C. Jin, D. Wei, S. Ravot, etc (Caltech, Nov
02)
14
Bandwidth ChallengeData Intensive Distributed
Analysis

• Analysis of particle collision events recorded by
  the CMS detector looks for needle in very large
  haystack
• For challenge event, use simulated events
  produced during a large distributed production
  run
• Multiple archive files with 767,000 events each
  stored on Clarens servers at CENIC POP in LA, and
  TeraGrid node at Caltech.
• Transferred gt 200 files at rates up to 400MB/s to
  2 disk servers on the show floor
• Decomposed archive files into ROOT data files,
  published via Clarens on disk servers
• Analysis of data performed and results displayed
  by Clarens ROOT client on kiosk machine

15
Grid Enabled Analysis View of a Collaborative
Desktop

• Building the GAE is the Acid Test for Grids
  and is crucial for next-generation experiments
  at the LHC
• Large, Diverse, Distributed Community of users
• Support hundreds to thousands of analysis
  tasks, shared among dozens of sites
• Widely varying task requirements and priorities
• Need Priority Schemes, robust authentication and
  Security
• Relevant to the future needs of research and
  industry

External Services
Storage Resource Broker
CMS ORCA/COBRA
Browser
MonaLisa
Iguana
ROOT
Cluster Schedulers
PDA
ATLAS DIAL
Griphyn VDT
Clarens
VO Management
File Access
MonaLisa Monitoring
Authentication
Key Escrow
Shell
Authorization
Logging
16
Private Grids Structured P2PSub-Communities
in Global HEP
17
HENP Lambda GridsFibers for Physics

• Problem Extract Small Data Subsets of 1 to 100
  Terabytes from 1 to 1000 Petabyte Data Stores
• Survivability of the HENP Global Grid System,
  with hundreds of such transactions per day
  (circa 2007)requires that each transaction be
  completed in a relatively short time.
• Example Take 800 secs to complete the
  transaction. Then
• Transaction Size (TB) Net
  Throughput (Gbps)
• 1
  10
• 10
  100
• 100
  1000 (Capacity of
  Fiber
  Today)
• Summary Providing Switching of 10 Gbps
  wavelengthswithin 2-4 years and Terabit
  Switching within 5-8 years would enable
  Petascale Grids with Terabyte transactions,to
  fully realize the discovery potential of major
  HENP programs, as well as other data-intensive
  fields.

18
HEP Grid Challenges Workflow Management and
Optimization

• Maintaining a Global View of Resources and
  System State
• End-to-end Monitoring
• Adaptive Learning New paradigms for
  optimization, problem resolution (progressively
  automated)
• Balancing Policy Against Moment-to-moment
  Capability
• Balance High Levels of Usage of Limited Resources
  Against Better Turnaround Times for Priority
  Jobs
• Realtime Error Detection, Propagation Recovery
• An Integrated User Environment
• User-Grid Interactions
• Emerging Strategies and Guidelines
• Including the Network as a Dynamic, Managed
  Resource

19
Dynamic Distributed Services Architecture (DDSA)

• Station Server Services-engines at sites host
  Dynamic Services
• Auto-discovering, Collaborative
• Scalable to thousands of service-Instances
• Servers interconnect dynamically form a robust
  fabric
• Service Agents Goal-Oriented, Autonomous,
  Adaptive
• Maintain State AutomaticEvent notification
• Adaptable to Web services, OGSA many platforms
  working environments (also mobile)

See http//monalisa.cacr.caltech.edu
http//diamonds.cacr.caltech.edu
Caltech/UPB (Romania)/NUST (Pakistan)
Collaboration
20
California Institute of Technology
21
Monitoring CMS farms and WAN traffic
22
Global Client / Dynamic Discovery
Monitoring Managing VRVS Reflectors
23
UltraLight Collaborationhttp//ultralight.caltec
h.edu

• Caltech, UF, FIU, UMich, SLAC,FNAL,MIT/Haysta
  ck,CERN, UERJ(Rio), NLR, CENIC,
  UCAID,Translight, UKLight, Netherlight, UvA,
  UCLondon, KEK, Taiwan
• Cisco, Level(3)

• Integrated packet switched and circuit switched
  hybrid experimental research network leveraging
  transoceanic RD network partnerships
• NLR Waves 10 GbE (LAN-PHY) wave across the US
• Optical paths transatlantic extensions to
  Japan, Taiwan, Brazil
• End-to-end monitoring Realtime tracking and
  optimization Dynamic bandwidth provisioning,
• Agent-based services spanning all layers of the
  system, from the optical cross-connects to the
  applications.

24
UltraLight
http//ultralight.caltech.edu

• Serving the major LHC experiments developments
  broadly applicable to other data-intensive
  programs
• Hybrid packet-switched and circuit-switched,
  dynamically managed optical network
• Global services for system management
• Trans-US wavelength riding on NLR
  LA-SNV-CHI-JAX
• Leveraging advanced research production
  networks
• USLIC/DataTAG, SURFnet/NLlight, UKLight,
  Abilene, CAnet4
• Dark fiber to CIT, SLAC, FNAL, UMich Florida
  Light Rail
• Intercontl extensions Rio de Janeiro, Tokyo,
  Taiwan
• Flagship Applications with a diverse traffic mix
• HENP TByte to PByte block data transfers at
  1-10 Gbps
• eVLBI Real time data streams at 1 to several
  Gbps

25
VRVS on Windows
KEK (JP)
VRVS (Version 3) Meeting in 8 Time Zones
Caltech (US)
RAL (UK)
Brazil
CERN (CH)
AMPATH (US)
Pakistan
SLAC (US)
Canada
78 Reflectors Deployed Worldwide Users in 96
Countries
AMPATH (US)
26
HENP Networks and Grids UltraLight

• The network backbones and major links used by
  major HENP projects are advancing rapidly
• To the 10 G range in 18 months much faster than
  Moores Law
• Continuing a trend a factor 1000 improvement
  per decade
• Transition to a community-owned and operated
  infrastructure for research and education is
  beginning with (NLR, USAWaves)
• HENP is learning to use 10 Gbps networks
  effectively over long distances
• Fall 2003 Development 5 to 6 Gbps flows over
  11,000 km
• A new HENP and DOE Roadmap Gbps to Tbps links
  in 10 Years
• UltraLight A hybrid packet-switched and
  circuit-switched network ultra-protocols
  (FAST), MPLS dynamic provisioning
• To serve the major needs of the LHC Other
  major programs
• Sharing, augmenting NLR and internatl optical
  infrastructures
• A cost-effective model for future HENP, DOE
  networks