ESnet Status ESCC Meeting Jan' 2004

1
ESnet StatusESCC Meeting Jan. 2004

• William E. Johnston, ESnet Manager and Senior
  Scientist
• Michael S. Collins, Stan Kluz,Joseph Burrescia,
  and James V. Gagliardi, ESnet Leads
• and the ESnet Team
• Lawrence Berkeley National Laboratory

2
Recent Accounting Difficulties

• The accounting difficulties of the past 6 months
  are behind us and had no impact on the operation
  of the network
• The subsequent Lehman review of ESnet found that
  all of the issues had been appropriately
  addressed and that the plans to move forward were
  completely appropriate
• This issue is behind us

3
William E. Johnston Bio

• Formerly Department Head of LBNL Distributed
  Systems Department
• Long history in High Performance Networking
  Community
• 1980s -1998 PI or Co-PI for
• LBL Network Advisory Group
• Advised NSF on NSF backbone transition to
  commercial service
• Chaired the ESnet Site Coordinating Committee for
  5 yrs
• Blanca/XUnet - first x-country ATM network (w/ATT
  Bell Labs)
• BAGnet first OC3 (155 Mb/s) ATM net around the
  SF Bay Area (w/ Pac Bell)
• MAGIC DARPA testbed, 1st Sprint OC48 ATM wide
  area network that worked
• NGI QoS DOE bandwidth reservation network, w/
  ESnet
• Clipper first sustained transfer of terabyte
  files for HEP, filling an OC12 circuit
• 1998-2003 NASA project manager for an 18M/yr
  Grids project
• 6M/yr in external subcontracts
• 2000-2003 PI, DOE Science Grid
• March 2002, Co-Author of LBNL/ANL, A Vision for
  DOE Scientific Networking driven by High Impact
  Science
• August 2002, Co-Author of Office of Science
  Workshop, High Performance Networks for High
  Impact Science
• June 2003, Co-Author of Office of Science
  Workshop, DOE Science Networking Challenge
  Roadmap to 2008

4
A Bit of ESnet Reorganization
ESnetManager
LBNLBusiness Services
NetworkEngineeringGroup
InfrastructureServicesGroup
ScienceServices Group
ResourceManager
accounting /bookkeeping

• NetworkEngineering
• Eng. and upgrades
• Adv. Technology
• Measurement and Monitoring
• Equip. testing

• Data management
• Asset mgmt.
• Data center
• Servers
• Eng. Email and Web
• Internal security

Audio, Video, and Data Collab.
Contracting
PKI CertificationAuthority

• NetworkOperations
• Routing and net. services
• NOC

business practices checks and balances
RD Projects
WAN Security andDisaster Recovery
new
5
Reminders

• ESnet is
• An infrastructure that is critical to DOEs
  science mission and that serves all of DOE
• Focused on the Office of Science programs, but
  also serves NNSA and other DOE Offices
• Complex and specialized both in the network
  engineering and the network management
• Extremely reliable in several dimensions
• Not a typical ISP architected to move huge
  amounts of data between a small number of sites
  and have very high-speed peering with a small
  number of other nets
• You cant go out and buy this ESnet integrates
  commercial products and in-house software into a
  complex management system for operating the
  network
• You cant go out and take a class in how to run
  this sort of network it is specialized and is
  learned from experience

6
Stakeholders

• DOEs Office of Science, scientific community
• Other DOE Offices, esp. NNSA
• DOE MICS Office, ESnet program
• ESnet Steering Committee (ESSC)
• represents the Science Offices strategic needs
• ESnet Coordinating Committee (ESCC)
• site representatives (operational issues)
• Users
• Most of the DOE Office of Science program
  participants
• NNSA / Defense Programs, EM, etc.
• DOE collaborators
• A few others (e.g. the NSF LIGO and NOAA sites)

7
ESnet is Different from a Commercial ISPor
University Network

• A fairly small number of very high bandwidth
  sites (commercial ISPs have thousands of low b/w
  sites)
• Runs SecureNet as an overlay network
• Provides direct support of DOE science through
  various science services
• ESnet owns all network trouble tickets (even
  from end users) until they are resolved
• one stop shopping for user network problems
• 7x24 coverage
• Both network and science services problems

8
ESnet is Driven by the Needs of DOE Science
August 13-15, 2002
Office of Science Organizing Committee Mary Anne
Scott, Chair Dave Bader Steve Eckstrand
Marvin Frazier Dale Koelling Vicky White
Workshop Panel Chairs Ray Bair and Deb
Agarwal Bill Johnston and Mike Wilde Rick
Stevens Ian Foster and Dennis Gannon Linda
Winkler and Brian Tierney Sandy Merola and
Charlie Catlett

• Focused on science requirements that drive
• Advanced Network Infrastructure
• Middleware Research
• Network Research
• Network Governance Model

9
Eight Major DOE Science Areas Analyzed at the
August 02 Workshop
Driven by
10
Evolving Qualitative Requirements for Network
Infrastructure
S
C
S
C
guaranteedbandwidthpaths
I
1-40 Gb/s,end-to-end
I
2-4 yrs
1-3 yrs
C
C
C
C
storage
S
S
S
compute
C
instrument
I
cache compute
CC
S
C
CC
CC
I
4-7 yrs
3-5 yrs
CC
CC
CC
C
CC
100-200 Gb/s,end-to-end
C
S
11
Evolving Quantitative Science Requirements for
Networks
12
New Strategic Directions to Address Needs of DOE
Science
June 3-5, 2003
Organized by the ESSC Workshop Chair Roy
Whitney, JLAB Report Editors Roy Whitney,
JLAB Larry Price, ANL
Workshop Panel Chairs Wu-chun Feng,
LANL William Johnston, LBNL Nagi Rao,
ORNL David Schissel, GA Vicky White, FNAL Dean
Williams, LLNL

• Focused on what was needed to achieve the science
  driven network requirements of the previous
  workshop
• Both Workshop reports are available at
  es.net/research

13
ESnet Strategic Directions

• Developing a 5 yr. strategic plan for how to
  accomplish the capabilities that the workshops
  identified were needed
• Must address bandwidth, reliability, and Quality
  of Service between DOE Labs and their major
  collaborators in the University community
• More on this later
• Current status

14
Status ESnet Connects DOE Facilities and
Collaborators
CAnet4 CERN MREN Netherlands Russia StarTap Taiwa
n (ASCC)
PNWG
SEA HUB
ESnet IP
Japan
Starlight
Chi NAP
NY-NAP
QWEST ATM
MAE-E
SNV HUB
MAE-W
PAIX-E
Fix-W
PAIX-W
Euqinix
42 end user sites
International (high speed) OC192 (10G/s
optical) OC48 (2.5 Gb/s optical) Gigabit Ethernet
(1 Gb/s) OC12 ATM (622 Mb/s) OC12 OC3 (155
Mb/s) T3 (45 Mb/s) T1-T3 T1 (1 Mb/s)
ESnet backbone Optical Ring and Hubs
peering points
hubs
SNV HUB
15
While There is One Backbone Provider, there
areMany Local Loop Providers to Get to the Sites
NY-NAP
QWEST ATM
LBNL/ CalRen2
GTN
DOE-NNSA
PANTEX
Qwest Contracted
Touch America Contracted/Owned
MCI Contracted/Owned
Site Contracted/Owned
16
ESnet Logical InfrastructureConnects the DOE
Community With its Collaborators
ESnet provides complete access to the Internet by
managing the full complement of Global Internet
routes (about 150,000)
17
Recent Changes

• Backbone Upgrade
• OC48 (4 links) DWDM southern route to be
  upgraded to OC192 - mid-late summer 04
• Hub Changes
• SNV
• ESnet to UltraNet and NLR cross-connect
• Engineering in process, goal connection by or
  before summer
• Upgraded Abilene to OC48 POS (Jan 04)
• CHI
• Engineering for ESnet to UltraNet and NLR
  cross-connect
• Engineering in process, goal connection by or
  before summer
• Upgraded Abilene to OC48 POS (Jan 04)
• NY (AOA)
• decommission NY 60 Hudson and remove ESnet
  equipment (Feb 04)
• Started process of connecting to the MANLAN at
  10G (move private peering for Abilene, SINET and
  DANTE to MANLAN) spring 04
• DC, ATL ELP
• Current routers will be upgraded to T320 in
  preparation for OC192 circuits installations
  (summer 04)
• ALT Peering w/ Abilene _at_ OC48 in near future

18
Recent Changes

• Sub-Hub Changes
• ALB
• TouchAmerica provided all of our circuits in the
  Southwest and went bankrupt
• Very dicey situation
• TA announced circuit termination by 31 Jan
• ELP-ALB, INEEL, Ames Lab
• Qwest could not take over until TA officially
  transferred the circuits, which looked like this
  would be well after 31 Jan
• We could appeal to the FCC (hah!)
• With lots of help from Qwest OC12 ELP-ALB POS has
  been replaced by Qwest OC12 POS (Jan 04)
• Seattle
• New M10 RTR installed (Dec 03)
• TouchAmerica OC3 ATM replaced by Level3 OC3 POS
  (Jan 04)

19
Site Changes and Upgrades

• DOE-GTN new RTR for MICS and OC3 POS for NNSA
  (Aug 03)
• New GigE added for JLAB (Sept 03)
• New FastEther connection for backup for DOEHQ _at_
  JLAB (Sept 03)
• New GigE added for LLNL (Sept 03)
• OC12 POS M10 installed at Equinix-San Jose (Jan
  04)
• An important peering expansion
• INEEL TouchAmerica DS3 ATM will be regroomed to
  Qwest DS3 ATM (Jan 04)
• Ames Lab (ISU) TouchAmerica DS3 will be regroomed
  to Qwest DS3 (Jan 04)
• YuccaMT DS3 ATM to be replaced by DS3 P-2-P (Feb
  04)
• NTS/Bechtel considering hubbing in Las Vegas
• OC12 POS M10 to be installed at Equinix-Ashburn
  (Feb 04)
• Move from ATM interface to OC12 POS CCC between
  SNV-NASA on shared OC12 with Abilene (2Q FY04)

20
ESnet Traffic
This is not SC.Jan. should tell.
Annual growth in the past five years has
increased from 1.7x annually to just over 2.0x
annually.
This might be SC03
Looks like it might be increasing again.
21
Who Generates Traffic, and Where Does it Go?
ESnet Inter-Sector Traffic Summary, Jan 2003
72
21
Commercial
14
DOE is a net supplier of data because DOE
facilities are used by Univ. and commercial, as
well as by DOE researchers
ESnet
17
25
DOE sites
RE
10
Peering Points
53
9
International
DOE collaborator traffic, inc.data
4
We are working on the Steve Wolff, Lehman review
challenge How much traffic do you exchangewith
AS11537 Abilene? How big are the largest site
to site / host to host flows in that exchange?
ESnet Appropriate Use Policy (AUP) All ESnet
traffic must originate and/or terminate on an
ESnet an site (no transit traffic is
allowed) E.g. a commercial site cannot exchange
traffic with an international site across
ESnet This is effected via routing restrictions
ESnet Ingress Traffic Green ESnet Egress
Traffic Blue Traffic between sites of total
ingress or egress traffic
22
SecureNet

• SecureNet connects 9 NNSA (Defense Programs)
  sites and a 10th site at DOE HQ-GTN
• The NNSA sites exchange encrypted ATM traffic
• The data is unclassified when ESnet gets it
  because it is encrypted before it leaves the NNSA
  sites with an NSA certified encrypter
• Runs over the ESnet core backbone as a layer 2
  overlay that is, the SecureNet encrypted ATM is
  transported over ESnets Packet-Over-SONET
  infrastructure by encapsulating the ATM in MPLS

23
SecureNet Mid 2003
Backup SecureNet Path
AOA-HUB
CHI-HUB
GTN
SNV-HUB
LLNL
DC-HUB
SNLL
ORNL
KCP
DOE-AL
Pantex
LANL
SNLA
SRS
Primary SecureNet Path
ATL-HUB
ELP-HUB
SecureNet encapsulates payload encrypted ATM in
MPLSusing the Juniper Router Circuit Cross
Connect (CCC) feature.
24
IPv6-ESnet Backbone
9peers
18 peers
6peers
BNL
StarLight
7peers
StarTap
Distributed 6TAP
PAIX
LBL
Chicago
Sunnyvale
New York
ANL
FNAL
DC
Albuquerque
Atlanta
SLAC
El Paso

• IPv6 is the next generation Internet protocol,
  and ESnet is working on addressing deployment
  issues
• one big improvement is that while IPv4 has 32 bit
  about 4x109 addresses (which we are running
  short of), IPv6 has 132 bit about 1040
  addresses (which we are not ever likely to run
  short of)
• another big improvement is native support for
  encryption of data

25
Science Services Teleconferencing

• Seamless voice, video, and data teleconferencing
  is important for geographically dispersed
  collaborators
• ESnet currently provides voice conferencing,
  videoconferencing (H.320/ISDN scheduled, H.323/IP
  ad-hoc), and data collaboration services to more
  than a thousand DOE researchers worldwide
• Heavily used services, averaging around
• 4600 port hours per month for H.320
  videoconferences,
• 2000 port hours per month for audio conferences
• 1100 port hours per month for H.323
• approximately 200 port hours per month for data
  conferencing

26
Science Services Teleconferencing

• There are now over 300 registered H323 users
• We have 70 ports currently and are planning for
  an expansion
• Will acquire an usage monitoring system as a
  capacity planning tool for the future
• Web-Based registration and scheduling for all of
  these services
• authorizes users efficiently
• lets them schedule meetings
• Such an automated approach is essential for a
  scalable service ESnet staff could never handle
  all of the reservations manually

27
Science Services Public Key Infrastructure

• Public Key Infrastructure supports cross-site,
  cross-organization, and international trust
  relationships that permit sharing computing and
  data resources and other Grid services
• Digital identity certificates for people, hosts
  and services essential core service for Grid
  middleware
• provides formal and verified trust management
  an essential service for widely distributed
  heterogeneous collaboration, e.g. in the
  International High Energy Physics community
• DOE Grids CA
• Have recently added a second CA with a policy
  that permits bulk issuing of certificates
• Important for secondary issuers
• NERSC will auto issue certs when accounts are set
  up this constitutes an acceptable identity
  verification
• May also be needed for security domain gateways
  such asKerberos X509 e.g. KX509

28
Science Services Public Key Infrastructure

• Certificate Authority (CA) validates users
  against the CP and issues digital identity certs.
• Certificate Revocation Lists are provided
• This service was the basis of the first routine
  sharing of HEP computing resources between US and
  Europe
• Policy Management Authority negotiates and
  manages the formal trust instrument (Certificate
  Policy - CP)
• Sets and interprets procedures that are carried
  out by ESnet
• Currently facing an important oversight situation
  involving potential compromise of user X.509 cert
  private keys
• Boys-from-Brazil style exploit gt kbd sniffer on
  several systems that housed Grid certs
• Is there sufficient forensic information to say
  that the pvt keys were not compromised??
• Is any amount of forensic information sufficient
  to guarantee this, or should the certs be
  revoked?
• Policy refinement by experience

29
Science Services Public Key Infrastructure

• The rapidly expanding customer base of this
  service will soon make it ESnets largest
  collaboration service by customer count

30
Science Services

• Directory Services for VOs is a proposed service
• Important for large-scale collaborations
• Would provide for registration of science
  collaboration members, their attributes, etc.
• Secure management interface is needed so that
  VO/collaboration appointees can enter data
• May use same infrastructure as X.509 certificate
  repository
• Will be replicated at diverse locations in the
  same way as the engineering databases

31
Science Services

• End to end monitoring
• An essential service for debugging and tuning
  high-performance distributed applications
• The exact service is not clear the general
  service is
• Provide applications with flow spec. (source,
  destination, port) level, real-time monitoring
• E.g. LBNLs Self-Configuring Network Monitor
  on-demand, passive network path monitoring
  (http//dsd.lbl.gov/Net-Mon/Self-Config.html)

32
ESnet is Monitored in Many Ways Six databases
maintain real-time state of the network
Configuration
Performance
OSPF Metrics(routing and connectivity)
SecureNet
Hardware Configuration
IBGP Mesh(routing and connectivity)
Maps Diagrams are all clickable, allowing
drilldown to finest levels of detail of the
underlying databases
33
ESnet is Monitored in Many Ways

• Real-time monitoring of traffic levels and
  operating state of some 4400 network entities is
  the primary network diagnosis tool
• Will probably work with Les Cottrell to implement
  various traceroute / iperf monitors to off-net
  beacons

34
ESnet is Monitored in Many Ways
CAnet4 CERN MREN Netherlands Russia StarTap Taiwa
n (ASCC)
Bandwidth problem suspected here
ESnet IP
Japan
QWEST ATM
International (high speed) OC192 (10G/s
optical) OC48 (2.5 Gb/s optical) Gigabit Ethernet
(1 Gb/s) OC12 ATM (622 Mb/s) OC12 OC3 (155
Mb/s) T3 (45 Mb/s) T1-T3 T1 (1 Mb/s)
35
ESnet is Monitored in Many Ways
ESnet configuration
Performance
OSPF Metrics
SecureNet
Hardware Configuration
IBGP Mesh
36
Drill Down into the Performance DB to Every
Physical and Logical Interface level for Every
Router

• 1 min, 2 hr, and daily average bandwidth reports
• hours to months of historical data are kept
  on-line to see if there is evidence of a problem
  developing gradually (which is not uncommon the
  telecom interfaces can slowly get noisier)

16 hour history
36 hour history
37
ESnet is Monitored in Many Ways
When a hardware alarm goes off here and the 24x7
operator is notified
CAnet4 CERN MREN Netherlands Russia StarTap Taiwa
n (ASCC)
ESnet IP
Japan
QWEST ATM
International (high speed) OC192 (10G/s
optical) OC48 (2.5 Gb/s optical) Gigabit Ethernet
(1 Gb/s) OC12 ATM (622 Mb/s) OC12 OC3 (155
Mb/s) T3 (45 Mb/s) T1-T3 T1 (1 Mb/s)
38
ESnet is Monitored in Many Ways
ESnet configuration
Performance
OSPF Metrics
SecureNet
Hardware Configuration
IBGP Mesh
39
Drill Down into the Configuration DB to Operating
Characteristics of Every Device
e.g. cooling air temperature for the router
chassis air inlet, hot-point, and air exhaust for
the ESnet gateway router at PNNL
40
Problem Resolution

• Lets say that the diagnoistics have pinpointed a
  bad module in a router in the ESnet hub in NYC
• Almost all high-end routers, and other equipment
  that ESnet uses, have multiple, redundant modules
  for all critical functions
• Failure of a module (e.g. a power supply or a
  control computer) can be corrected on-the-fly,
  without turning off the power or impacting the
  continued operation of the router

41
ESnet is Monitored in Many Ways
ESnet configuration
Performance
OSPF Metrics
SecureNet
Hardware Configuration
IBGP Mesh
42
Drill Down into the Hardware Configuration DBfor
Every Wire Connection
Equipment rack detail at AOA, NYC Hub(one of the
core optical ring sites)
43
The Hub Configuration Database
Equipment wiring detail for two modules at the
AOA, NYC Hub This allows, e.g., Qwest personnel
at the NYC site replace modules for ESnet
smart hands are key service
44
Qwest DS3 DCX
Sentry power 48v 30/60 amp panel (3900 list)
AOA Performance Tester (4800 list)
Sentry power 48v 10/25 amp panel (3350 list)
DC / AC Converter (2200 list)
Cisco 7206 AOA-AR1 (low speed links to MIT
PPPL) (38,150 list)
Lightwave Secure Terminal Server (4800 list)
ESnet Equipment _at_ Qwest 32 AofA HUB NYC,
NY (1.8M, list)
Juniper T320 AOA-CR1 (Core router) (1,133,000
list)
Juniper OC192 Optical Ring Interface (the AOA end
of the OC192 to CHI (195,000 list)
Juniper OC48 Optical Ring Interface (the AOA end
of the OC48 to DC-HUB (65,000 list)
Juniper M20 AOA-PR1 (peering RTR) (353,000 list)
45
Operating Science Mission Critical Infrastructure

• ESnet is a visible and critical pieces of DOE
  science infrastructure
• if ESnet fails,10s of thousands of DOE and
  University users know it within minutes if not
  seconds
• Requires high reliability and high operational
  security in the ESnet supporting infrastructure
  the systems that are integral to the operation
  and management of the network
• Secure and redundant mail and Web systems are
  central to the operation and security of ESnet
• trouble tickets are by email
• engineering communication by email
• engineering database interface is via Web
• Secure network access to Hub equipment
• Backup secure telephony access to Hub equipment
• 24x7 help desk (joint with NERSC)
• 24x7 on-call network engineer

46
Disaster Recovery and Stability

• The network operational services must be kept
  available even if, e.g., the West coast is
  disabled by a massive earthquake, etc.
• ESnet engineers in four locations across the
  country
• Full and partial engineering databases and
  network operational service replicas in three
  locations
• Telephone modem backup access to all hub
  equipment
• All core network hubs are located in commercial
  telecommunication facilities with high physical
  security and backup power

47
Disaster Recovery and Stability

• Engineers, 24x7 NOC, generator backed power
• Spectrum (net mgmt system)
• DNS (name IP address translation)
• Eng database
• Load database
• Config database
• Public and private Web
• E-mail (server and archive)
• PKI cert. repository and revocation lists
• collaboratory authorization service

• Remote Engineer
• partial duplicate infrastructure

DNS
Remote Engineer
Duplicate Infrastructure (planed full
replication of the NOC databases and servers and
Science Services databases)
Engineers Eng Srvr Load Srvr Config Srvr

• ESnet backbone operated without interruption
  through
• N. Calif. Power blackout of 2000
• the 9/11 attacks
• the Sept., 2003 NE States power blackout

48
Maintaining Science Mission Critical
Infrastructurein the Face of Attack

• A Phased Security Architecture is being
  implemented to protect the network and the sites
• The phased response ranges from blocking certain
  site traffic to a complete isolation of the
  network which allows the sites to continue
  communicating among themselves in the face of the
  most virulent attacks
• Separate ESnet core routing functionality from
  our external Internet connections by means of a
  peering router that can have a policy different
  from the core routers
• Provide a rate limited path to the external
  Internet that will insure site-to-site
  communication during an external denial of
  service attack
• Allow for Lifeline connectivity that allows
  downloading of patches, exchange of e-mail and
  viewing web pages (i.e. e-mail, dns, http,
  https, ssh, etc.) with the external Internet
  prior to full isolation of the network

49
Maintaining Science Mission Critical
Infrastructurein the Face of Cyberattack

• Normal first response is by the sites
• A second response to restore normal operation at
  least between DOE Labs (by blocking the attack
  from entering ESnet) is provided by a Phased
  Security Architecture
• The phased response ranges from blocking certain
  site traffic to a complete isolation of the
  network which allows the sites to continue
  communicating among themselves in the face of the
  most virulent attacks
• Separate ESnet core routing functionality from
  our external Internet connections by means of a
  peering router that can have a policy different
  from the core routers
• Provide a rate limited path to the external
  Internet that will insure site-to-site
  communication during an external denial of
  service attack
• Allow for Lifeline connectivity that allows
  downloading of patches, exchange of e-mail and
  viewing web pages (i.e. e-mail, dns, http,
  https, ssh, etc.) with the external Internet
  prior to full isolation of the network

50
ESnet and Cybersecurity

• ESnet protects itself and other sites infected
  ESnet sites can be blocked, partially or
  completely
• ESnet can come also come to the aid of an ESnet
  site with temporary filters on incoming traffic,
  etc., if necessary
• This is one of the very few areas where ESnet
  might participate directly in site security
• Request must come from Site Coordinator
• Not a substitute for good site security

51
Asset Management

• ESnet Asset Management System tracks all ESnet
  network and computing equipment throughout the
  country
• Approximately 270 assets at 50 locations in the
  US are tracked in a Remedy database
• Cradle-to-Grave asset movement tracking
• Received equipment is documented in Sunflower
  (LBL property database) and Remedy
• LBL Shipping Documents created electronically
• All assets tracked through carriers tracking
  system
• Set up and monitor Return Merchandise
  Authorizations with vendors
• Surplusing

52
Asset Management
E.g. first 4 locations of 50(from Remedy
database)
E.g. AOA Hub
53
Future Directions the 5 yr Strategy

• Must address bandwidth, reliability, and Quality
  of Service between DOE Labs and their major
  collaborators in the University community
• Goal A draft strategic plan by the late April
  ESSC meeting
• Elements
• University connectivity
• Scalable and reliable site connectivity
• Hi-impact science bandwidth provisioned
  circuits
• Science Services to support Grids, co
  laboratories, VOs, etc.
• Close collaboration with the network RD community

54
5 yr Strategy Near Term Goal 1

• Connectivity between any DOE Lab and any Major
  University should be as good as ESnet
  connectivity between DOE Labs and Abilene
  connectivity between Universities
• Partnership with I2/Abilene
• Multiple high-speed peering points
• Routing tailored to take advantage of this
• Latency and bandwidth from DOE Lab to University
  should be comparable to intra ESnet or intra
  Abilene
• Continuous monitoring infrastructure to verify
• Stay tuned

55
5 yr Strategy Near Term Goal 2

• Connectivity between ESnet and important RD nets
  critical issue from Roadmap
• UltraNet and NLR for starters
• Reliable, high bandwidth cross-connects
• IWire ring between Qwest ESnet Chicago hub and
  Starlight
• This is also critical for DOE lab connectivity to
  the DOE funded LHCNet 10 Gb/s link to CERN
• Both LHC tier 1 sites in the US Atlas and CMS
  are at DOE Labs
• ESnet ring between Qwest ESnet Sunnyvale hub
  and the Level 3 Sunnyvale hub that houses the
  West Coast POP for NLR and UltraNet

56
5 yr Strategy Near-Medium Term Goal

• Scalable and reliable site connectivity
• Hi-impact science bandwidth provisioned
  circuits
• Fiber / lambda ring based Metropolitan Area
  Networks
• Preliminary engineering study completed for San
  Francisco Bay Area and Chicago
• Proposal submitted
• Stay tuned

57
ESnet Future Architecture

• Migrate site local loops to ring structured
  Metropolitan Area Network and regional nets in
  some areas
• Goal is local rings, like the backbone, that
  provide multiple paths
• Dynamically provisioning of private circuits in
  the MAN and through the backbone to provide high
  impact science connections
• This should allow high bandwidth circuits to go
  around site firewalls. The circuits are secure
  and end-to-end, so if the sites trust each other,
  they should allow direct connections
• Partnership with DOE UltraNet, Internet 2 HOPI,
  and National Lambda Rail

58
ESnet Long-Term Architecture
site
one optical fiber pairDWDM
Layer 2 management equipment (e.g. 10 GigEthernet
switch)
MetropolitanAreaNetworks
corering
site
Layer 3 (IP)management equipment (router)
production IP
Optical channel (?) management equipment
provisioned circuits carriedover lambdas
site
59
ESnet Near-Term Architecture
site
one optical fiber pairDWDM
one POS flow between ESnet routers
Layer 2 management equipment (e.g. 10 GigEthernet
switch)
MetropolitanAreaNetworks
corering
site
Layer 3 (IP)management equipment (router)
production IP
provisioned circuits carriedover lambdas
Optical channel (?) management equipment
provisioned circuits carriedas tunnels through
the ESnetIP backbone
site
60
ESnet MAN Architecture - logical
CERN(DOE funded link)
StarLight
Qwest hub
Current DMZs are back-hauled to the core
router Implemented via 2 VLANs one in each
direction around the ring
ESnet core
other international peerings
Vendor neutral facility
ESnet managed? / circuit services
ESnet management and monitoring partly to
compensate for no site router

• Ethernet switch
• DMZ VLANs
• Management of provisioned circuits

ESnet managed? / circuit services tunneled
through the IP backbone
ESnet production IP service
ANL
FNAL
site equip.
Site gateway router
site equip.
Site gateway router
Site LAN
Site LAN
61
ESnet MAN Based Architecture Phase1
New York (AOA)
Chicago (CHI)
Washington, DC (DC)
The Hubs have lots of connections(42 in all)
Backbone(optical fiber ring)
Atlanta (ATL)
Sunnyvale (SNV)
El Paso (ELP)
ESnet responsibility
Site responsibility
Site gateway router
ESnet border
SiteLAN
MANoptical fiber ring
DMZ
Site
62
ESnet MAN Based Architecture Phase2-3
New York (AOA)
Chicago (CHI)
Washington, DC (DC)
The Hubs have lots of connections(42 in all)
Backbone(optical fiber ring)
MAN ? circuits are carried through the core on
dynamically provisioned MPLS paths
Atlanta (ATL)
Sunnyvale (SNV)
El Paso (ELP)
ESnet responsibility
Site responsibility
? based cross connect
Site gateway router
ESnet border
SiteLAN
MANoptical fiber ring
DMZ
Site
63
ESnet Architecture FutureEnd-to-End Optical
Transparency
The hub and border router IP production network
remains, but based on multi-? interconnected
rings that also provide dynamically provisioned,
end-to-end circuits
New York (AOA)
Chicago (CHI)
Washington
Backbone(optical fiber ring)
Atlanta (ATL)
Sunnyvale (SNV)
El Paso (ELP)
The Hubs have lots of connections(42 in all)
ESnet responsibility
Site responsibility
? based cross connect
Site gateway router
ESnet border
SiteLAN
MANoptical fiber ring
DMZ
Site
64
Future ESnet Architecture
circuit cross connect
ESnet border
Site gateway router
MANoptical fiber ring
SiteLAN
DMZ
Site
New York (AOA)
Washington
ESnetbackbone
Atlanta (ATL)
Private circuit from one Lab to another
El Paso (ELP)
circuit cross connect
Site gateway router
ESnet border
SiteLAN
MANoptical fiber ring
DMZ
Site
65
Long-Term ESnet Connectivity Goal
Japan
Europe
CERN/Europe
Japan
MANs
(using NLR as an example)
Local loops
High-speed cross connects with Internet2/Abilene
Qwest
Major DOE Office of Science Sites
NLR
66
Long-Term ESnet Bandwidth Goal

• Harvey NewmanAnd what about increasing the
  bandwidth in the backbone?
• Answer technology progress
• By 2008 (the next generation ESnet backbone) DWDM
  technology will be 40 Gb/s per lambda
• And the backbone will be multiple lambdas
• Issues
• End-to-End, End-to-End, and End-to-End

67
Science Services Strategy

• ESnet is in a natural position to be the provider
  of choice for a number of middleware services
  that support collaboration, colaboratories,
  Grids, etc.
• The characteristics of ESnet that make it a
  natural middleware provider are that ESnet
• is the only computing related organization that
  serves all of the Office of Science
• is trusted and well respected in the OSC
  community
• has the 7x24 infrastructure required to support
  critical services, and is a long-term stable
  organization.
• The characteristics of the services for which
  ESnet is the natural provider are those that
• require long-term persistence of the service or
  the data associated with the service
• require high availability, require a high degree
  of integrity on the part of the provider
• are situated at the root of a hierarchy so that
  the service scales in the number of people that
  it serves by adding nodes that are managed by
  local organizations (so that ESnet does not have
  a large and constantly growing direct user base).

68
Science Services Strategy

• DOE Grids CA that provides X.509 identity
  certificates to support Grid authentication
  provides an example of this model
• the service requires a highly trusted provider,
  requires a high degree of availability
• it scales by adding site based or Virtual
  Organization based Registration Agents that
  interact directly with the users.
• The other highly successful ESnet Science Service
  is the audio, video, and data teleconferencing
  service to support human collaboration

69
Science Services Strategy

• The Roadmap Workshop identified twelve high
  priority middleware services, and several of
  these fit the criteria for ESnet support. These
  include, for example
• long-term PKI key and proxy credential management
  (e.g. an adaptation of the NSFs MyProxy service)
• directory services that virtual organizations
  (VOs) can use to manage organization membership,
  member attributes and privileges
• perhaps some form of authorization service
• In the future, some knowledge management services
  that have the characteristics of an ESnet service
  are also likely to be important
• ESnet will seek the addition funding necessary to
  develop, deploy, and support these types of
  middleware services.

70
Conclusions

• ESnet is an infrastructure that is critical to
  DOEs science mission and that serves all of DOE
• Focused on the Office of Science Labs
• Complex and specialized both in the network
  engineering and the network management
• You cant go out and buy this ESnet integrates
  commercial products and in-house software into a
  complex management system for operating the net
• You cant go out and take a class in how to run
  this sort of network it is specialized and
  learned from experience
• Extremely reliable in several dimensions