Optical testbed

1
Optical testbed
Malathi Veeraraghavan Univ. of
Virginia mv_at_cs.virginia.edu

• Talk outline
• Item I in Yr. 1 Plan
• Existing optical testbeds centralized control
• Opportunity for creating scalable testbeds
  distributed control
• Why is scalability important?
• Plans
• Equipment to purchase
• What we plan to implement
• Wide-area circuits

2
Item I in Year 1 Plan

• Set up a Gigabit Ethernet/Ethernet over SONET
  CHEETAH testbed, develop software to demonstrate
  dynamic circuit setup and release.
• What equipment should we purchase to first create
  this testbed?
• What software should we develop?

CHEETAH Circuit-switched High-speed End-to-End
ArcHitecture
3
Any existing optical testbeds?

• Research optical testbeds
• Canarie - Canada
• Starlight Illinois
• SURFnet Netherlands
• UKLight UK

4
Canarie optical testbed
5
Canaries optical testbed
Bill St. Arnaud

• What are the network switches?
• Answer Cisco 15454 MSPPs
• Goal popularize user-controlled lightpaths
  (UCLP)
• Lightpath GbE signal - Ethernet-over-SONET (EoS)
  - GbE signal
• Experiments supported
• CRC/U. of Ottawa
• U. of Quebec and Montreal (UQAM)
• Carleton Univ.
• U. of Waterloo

6
Multi-Service Provisioning Platform (MSPP), e.g.
Ciscos 15454

• Functionality
• Cross-connect Ethernet signal to equivalent SONET
  signal
• Deployed in enterprises today multi-service -
  telephony and data

7
Canaries optical testbed

• What are the network switches?
• Answer Cisco 15454 MSPPs
• Goal popularize user-controlled lightpaths
  (UCLP)
• Lightpath GbE signal - Ethernet-over-SONET (EoS)
  - GbE signal
• Experiments supported
• CRC/U. of Ottawa
• U. of Quebec and Montreal (UQAM)
• Carleton Univ.
• U. of Waterloo

8
Research efforts provisioning

• Most of the Canarie-supported research efforts
  are focussed on provisioning
• Difficult task many steps
• Create external databases to manage routing
  information to reach a destination host, what is
  the set of switches to traverse?
• Create external databases to manage inventory
  information
• Inter-domain issues (multiple carriers) policy
  manager
• Then set up circuit with TL1 commands
• Lately, efforts at integrating provisioning
  efforts with GridFTP

9
UQAM Architecture
Figure 1 UPLA layer model Figure
2UPLA installation in multiple domains
environment

• Inter-domain layer
• End-to-end lightpath manipulation orchestrate
  the end-to-end lightpath searching and
  concatenating them.
• Intra-domain layer
• Single-domain lightpath creation and setting up.
• Respecting the domains management in allocating
  lightpath.
• Device layer
• Network equipments
• Equipment control module

Omar Cherkaoui
10
UPLA Signaling process
WSDL
WSDL
Telnet, SNMP
WSDL
WSDL
Omar Cherkaoui
11
UPLA Signaling time

• Signaling time Number of domain x 2.400
  (second)
• The causes of moderate signaling time
• Webservices signaling
• Communication is over Http
• XML processing when parse message
• Admission control processing and domain
  management verification
• Resource seeking processing

Omar Cherkaoui
12
CRC/U. of Ottawa UCLP E2E Example
Vancouver
Ottawa
SCS
SCS
SCS
JavaSpace
JavaSpace
JavaSpace
LPOS
Jini SAP
Grid SAP
Michel Savoie
13
CRC/U. of Ottawa Sequence to create e2e
lighpath using webservices and Jini

• User A accesses the Grid Service Access Point
  (GSAP) and is authenticated by a user name and
  password. The GSAP creates an instance of a User
  Functions (UF) module that uses the functions
  provided by an instance of the Jini SAP (JSAP)
  which was obtained earlier from the Jini Lookup
  Service in Federation A.
• User A sends a request to set up an E2E
  connection between User A in Federation A and
  User C in Federation C. The User Functions (UF)
  module will call the ConnectionRequest method of
  the JSAP(A) and pass the connection information
  that was entered by the user.
• The ConnectionRequest method of the JSAP(A) will
  then find the route and the LPOs required to get
  from Federation A to Federation C
• Once finished, the JSAP(A) will have all the LPOs
  required to make the connection
• The JSAP(A) will then download the Lightpath
  Object Service (LPOS) from the JLS(A) in
  Federation A and pass the information about LPOs
  and the proxies to the JavaSpaces they came from
  to the makeE2EConnection method

Michel Savoie
14
Sequence (contd)

• The LPOS(A) will then lookup the proxy to the
  class implementing the Connection Services (CS)
  interface from the JLS(A)
• The results of this lookup will be the proxy to
  the Switch Communication Service (SCS) in
  Federation A
• The LPOS(A) will then lookup the proxy to the
  Jini Lookup Service (JLS) in Federation B from
  the JLS in Federation A
• The result will be the proxy to JLS(B)
• Using the JLS in Federation B, the LPOS(A) will
  use step 5 to get the SCS for Federation B
• The LPOS(A) will then use the SCS(B) which will
  then talk TL1 to the physical switch to make the
  cross connect between LPO1 and LPO2 at the Switch
  in Federation B

Michel Savoie
15
Sequence (contd)

• Once all the cross connects are established, a
  new LPO is created (LPO3 (A-C)) to reflect the
  concatenation between LPO1 and LPO2 which is then
  put into the JavaSpace at Federation A
• The LPOS(A) will then call the SCS(A) to make the
  cross connect between User A and LPO3 at the
  Switch in Federation A
• The LPOS(A) will then use step 6 to find the
  proxy to the JLS in Federation C and steps 5 and
  10 to make the cross connect between LPO3 and
  User C in Federation C

Michel Savoie
16
Starlight

• Starlight and OMNInet optical vs. photonic
• Photonic Interdomain Negotiator (PIN) signaling
  their own design
• Domain routing database stored outside switches
• Use TL1 commands to program switches Nortel,
  Glimmerglass, Calient and IMMI
• Centralized approach

17
Problems recognizedby these efforts

• Scalability
• Using network management systems outside the
  network elements to manage
• routing data
• inventory data
• takes significant effort Operations costs
• Other problems
• Interoperability of different vendors equipment
• Hence a startup such as Elematics
• Inter-carrier issues
• My pet peeve
• Call setup delays of in the order of seconds too
  high

18
Scalability charges

• Levied against Optical/TDM networks
• Widely recognized that the current GLIF
  optical/TDM networking model does not scale
  beyond a limited number of sites Internet 2 talk
  dated 10/15/2003
• While such circuit-switched networks may not
  necessarily be suitable for deployment of the
  scale of the Internet, they are still viable
  candidates for specialized deployments for
  connecting a small number of DOE large-scale
  science nodes Report of DOE Workshop on Ultra
  High-Speed Transport Protocols and Dynamic
  Provisioning for Large-Scale Science
  Applications, April 10-11, 2003, Argonne, IL,
  dated Oct. 27, 2003.

GLIF Global Lambda Integration Facility
19
Problems recognizedby these efforts

• Scalability
• Using network management systems outside the
  network elements to manage
• routing data
• inventory data
• takes significant effort Operations costs
• Other problems
• Interoperability of different vendors equipment
• Hence a startup such as Elematics
• Inter-carrier issues
• My pet peeve
• Call setup delays of in the order of seconds too
  high

20
Inventory problem

• TL1 command to set up a crossconnect through a
  15454
• Command
• ENT-CRS ltSTS_PATHgtltTIDgtltFROMgt,ltTOgtltCTAGgtltC
  CTgt
• Example
• ENT-CRS-STS1BODEGASTS-5-1,STS-12-51162WAY
• TID unique name for the system
• From and To Access Identifiers to identify
  timeslots on interfaces
• STS-1 on the card in Slot 5
• STS-5 on the card in Slot 12
• CTAG unique identifier used to match response
  with request
• CCT Crossconnection type e.g., 1WAY or 2WAY

21
Signaling approach to connection setup
distributed

• Call setup request carries destination IP address
  D bandwidth B incoming timeslot/l
• Lookup routing data table (same function as in an
  IP router)
• find outgoing interface O to reach destination D
• Resource allocation
• Allocate bandwidth B on interface O
• Select outgoing timeslot/l
• Program switch fabric
• Map incoming timeslot/l to outgoing timeslot/l
• Send call setup request to neighbor connected by
  interface O

22
Industry answer tosupport distributed signaling
approach

• IETF GMPLS
• Routing OSPF-TE
• Routing built into network switches
• Signaling RSVP-TE
• Link Management Protocol (LMP)
• Inventory data stored in network switches
• Auto-discovery of neighbors
• OIFs UNI-C, UNI-N, NNI
• Addresses carriers inter-domain issues

23
Actually implemented!

• Not just idle specifications
• Implemented by many switch vendors
• And interoperability-tested by an OIF-sponsored
  effort led by Univ. of New Hampshire
• Demoed at OFC2003

24
Interoperability Participating Companies

• Alcatel (UNI-C, UNI-N, E-NNI, NMS/EMS)
• Avici (UNI-C)
• Ciena (UNI-N, E-NNI)
• Data Connection (UNI-C, UNI-N, E-NNI, NMS/EMS)
• Elematics (UNI-N, E-NNI)
• Mahi Networks (UNI-N, E-NNI, NMS/EMS)
• NEC (UNI-C, UNI-N, E-NNI)
• Motorola/Netplane (UNI-C, UNI-N, E-NNI, NMS/EMS)
• Nortel (UNI-N, E-NNI)
• Sycamore (UNI-N, E-NNI, NMS/EMS)
• Tellabs (UNI-N, E-NNI, NMS/EMS)
• Tellium (UNI-N, E-NNI, NMS/EMS)

25
UNI/NNI Signaling Display
26
From keynote at Opticomm, Dallas, Oct. 03

• Rajiv Ramaswami, CTO , Optical Systems Group,
  Cisco, Keynote
• UCP (Unified Control Plane) Benefits
• Superfast Provisioning
• Enables E2E circuit setup without SP intervention
  while reducing provisioning times
• Enables future bandwidth on demand applications
  as policy billing standards mature
• Enhanced Scalability
• Network level Support for thousands of nodes,
  links and circuits per inter-connected network
• Lightweight EMS Move from EMS based
  (centralized) provisioning to node level
  (distributed) provisioning using signaling

27
UCP Benefits contd.

• Interoperable vendor implementations
• Reduces EMS/NMS integration / interoperability
  issues
• UCP/GMPLS A Driver for Evolution
• Build Network as a Database
• Simplify provisioning by driving intelligence
  (topology, circuit inventory and link
  characteristics) into the NEs with updates to EMS
  (CTC/CTM)
• Enable migration from an NMS based network
  database to NEs based network database,
  retrievable on demand by NMS
• Deliver Advanced Benefits
• New services features (Ethernet,OVPN Storage)
  not possible today
• Reduce costs, increase revenues, address scale of
  growing networks
• Enable multi network/vendor/SP interoperability

28
Revisit problems identifiedby Canarie and other
groups with centralized solution

• Scalability
• Using network management systems outside the
  network elements to manage
• routing data
• inventory data
• takes significant effort Operations costs
• Other problems
• Interoperability of different vendors equipment
  
• Inter-carrier issues
• My pet peeve
• Call setup delays in the order of seconds too high

Seems like GMPLS/UCP solves ALL these problems!
Maybe we have found the Silver Bullet!
29
Open questions with distributed approach

• SONET XCs typically sold in large-size
  configurations to carriers
• Hard to get the attention of vendors for
  purchases of ones and twos
• Expensive
• What is the call setup delay?
• in the order of ms?

30
Outline

• Item I in Yr. 1 Plan
• Existing optical testbeds centralized control
• Opportunity for creating scalable testbeds
  distributed control
• Why is scalability important?
• Plans
• Equipment to purchase
• What we plan to implement
• Wide-area circuits

31
Why is scalability so important?

• Research testbeds such as ESnet, CAnet 4 may not
  see scalability as an important goal
• Reason if main application is to transfer very
  large files and support other eScience apps such
  as visualization, deployments will be small-scale

32
Why is scalability important?

• Beyond the obvious reason of growth
• Interaction between traffic load, utilization,
  costs and file sizes
• The higher the traffic load, the higher the
  utilization
• Higher the utilization, lower the costs
• The smaller the sizes of files transferred on
  circuits, the higher the load

33
Aggregate utilization in a circuit-switched
network
rho/m lt1 rho/m ua Pb 0 ua rho/m (M/M/m/inf
case) Pb0
24.8 58.2 84.6
34
should we give high-BW circuits or low-BW
circuits from util point of view?

• Per-circuit BW is high, m is small
• Rimax 10Gbps m 1
• to get high util. in this setting,
• because rimax is high, service time is low, mu is
  high
• to get rho lamba/mmu to approach 1, lambda has
  to be high. Crossover file size should be low

35

• Per-circuit BW is low, m is high
• Rimax 100Mbps m 100
• to get high util. in this setting,
• because rimax is lower, service time is higher,
  mu is smaller
• to get rho lambda/mmu to approach 1, lambda
  has to be same. Crossover file size has to be same

36
effect of m

• this means if m is small, one can run at high
  util. by increasing rho so that rho/m is close to
  1. But then Pb will be high. My guess is Pb
  increases with rho faster at low m than at high
  m.
• Compare with M/M/1/k where a packet loss is
  comparable to call blocking

37
effect of crossover file size

• the smaller this is the more it allows to build
  switches with fewer interfaces. This is the
  advantage of h/w sig. it is not being able to
  increase load. whatever the reqd. load to achieve
  high util. based on the crossover file size we
  can adjust the no. of interfaces aggregating to a
  link

38
File sizes

• Use of e2e circuits for file transfers typically
  limited to large files
• What is the drawback of using e2e circuits for
  small files?
• e.g., in a path with a 50ms r.t. propagation
  delay, if we transfer a 100KB file over a 100Mbps
  path, transfer time is only 8ms. Circuit
  utilization is 8/(508) 13.7
• Two opposing factors
• If the crossover file size (beyond which circuit
  setup is attempted) is increased
• per-circuit utilization increases
• traffic load decreases (Pareto distribution of
  file sizes), which means aggregate utilization
  decreases

39
Pareto density function
40
Impact of data rate increase

• As data rates increase, the crossover file size
  will increase
• propagation delays stay unchanged
• Therefore to achieve high utilization on the CS
  network
• decrease signaling message processing delays
• engineer large-scale networks

41
Plot of utilization u withrc 100Mbps, k20
For 50ms paths, set a crossover file size
When load is low, operate at a high blocking rate
Pb0.3
Pb0.01
42
Summary

• Background on optical testbed creation
• Circuit-switched networks
• Networking a.k.a. resource sharing ? dynamic
  circuit setup/release
• Centralized management approach or distributed
  signaling approach
• Deployment of testbed in wide-area will be
  addressed in afternoon talk

43
Why routing decision module

• Next few slides explain the need for this module

44
CHEETAH concept Leverage presence of Internet
path

• Use Internet path for initial short message
  exchanges prior to actual data transfer
• e.g., URL from client to server or get file
  request
• Run circuit-switched network in call blocking
  mode
• If call is blocked, fall back to Internet path
• Engineer network for high utilization at the cost
  of blocking
• Use Internet path for reverse direction error
  control and flow control messages and some
  retransmissions

45
Should the app. attempt a circuit setup or not?

• Mean delay if a circuit setup is attempted

Pb call blocking probability in the
circuit-switched network
If circuit setup fails, fall back to Internet path
46
Routing decision
47
Numerical resultslink rate 1Gbps
Tprop 0.1ms
Tprop 50ms
48
Crossover file sizes
When Tprop 50ms, always attempt a circuit
49
Additional background

• What is a circuit switch
• SONET hierarchy
• GFP and VC

50
Circuit switching
Unfolded view of switch
TDM or WDM multiplexers
TDM or WDM demultiplexers

• Circuit switch Position based switching

Packet switch Header based switching
51
SONET hierarchy

• OC1 51Mbps
• OC3 155Mbps
• OC12 622Mbps
• OC48 2.5Gbps
• OC192 10Gbps

52
Multi-Service Provisioning Platform (MSPP)
PC

• MSPP as a circuit switch
• Space on one side (Ethernet)
• Space and Time on the WAN side
• PLUS Physical layer modification from Ethernet
  PHY to SONET PHY
• Virtual concatenation 100Mbps Ethernet mapped to
  2OC1 instead of to an OC3