ISIS Next Generation Router

1
ISISNext Generation Router

• Shahzad Ali
• Xia Chen
• Brendan Howell
• Yu Zhong

2
Motivation

• Anybody can make a router.
• Key is to make one with
• High speed (OC-48 and up)
• High port densities
• Better performance (throughput, delay, latency)
• Cheap
• etc.
• But we always forget some requirements.
• The new network requires new services
• Guaranteed bandwidth/latency (QoS)
• Scaleable Design
• Now the task is not that simple!!

3
Possible uses for our router

• Aggregation point for large data centers
• Backbone router at a major peering point
• Backbone router for carrier facilities
• Core router for IP transit providers
• High bandwidth
• Scaleable
• QoS
• Robustness
• Major Routing Protocol support.

4
So what did we come up with?

• Design a router that meets the minimum
  specifications
• Maximize switch throughput
• Provide support for strong QoS in the switch with
  realistic assumptions
• Build an extensible simulator
• Evaluate the design using simulations
• Make the design realistic!
• Tackle the tough issue of scalability
• Result ISIS

5
Design Decisions

• 64 port OC-48 with total capacity of 160 Gb/s
• Our base design allows for 128 OC-48 (320 Gb/s)
• Uses PMC Sierra PM9311 and PM9312 crossbar and
  scheduler.
• Buffers
• PC-100 10ns access latency
• Dimension buffer by simulation
• Queuing?
• Output Queuing
• Input Queuing

6
Output Queuing

• Provides perfect QoS.
• Can order outgoing cells according to their
  priorities using FQ (WFQ, WF2Q, etc)
• Use fast sorting techniques to speedup the
  process.
• But, requires a speedup of N for switching fabric
• Not scaleable for high bandwidth switches.
• Not cheap either.

Switching Fabric
Output
Input
A simple and Fast Hardware Implementation
Architecture for WFQ Algorithms, Nen-Fu Huang
and Chi-An Su
7
Input Queuing

• Provides no support for hard guarantees (QoS).
• Difficult to maximize switch throughput while
  providing these guarantees.
• But, simple to implement, with minimal speedup
  and maximum throughput.
• Make scheduling decisions using a smart
  scheduling algorithm.
• Use iSlip or similar algorithms to achieve high
  throughput

Switching Fabric
Output
Input
8
Queuing

• Output Queuing
• No!
• Violates our design goal of scalability.
• Too Expensive
• Input Queuing
• Maybe
• If we can do proper QoS with it.
• Maybe a third option?

9
QoS with Input Queuing

• The only way we can imagine this happening is
  with buffered crossbars
• The buffers are provided at each cross-point
• Number of buffers are bounded (between 3-5 cells)
• Use FQ servers at Input, crossbar buffers and
  output to achieve QoS
• The solution is hard to implement
• Plus it only provides probabilistic guarantees
• Plus we dont even know if it actually works

10
QoS with Input Queuing

• Some recent work focuses on this aspect.
• Implementing Distributed Packet fair Queuing in
  a Scalable Switch Architecture, D. C. Stephens
• A Distributed Scheduling Architecture for
  Scalable Packet Switches, F. M. Chiussi and A.
  Francini
• Is that the best we can do?

11
CIOQ

• Combined Input output queuing combines the
  benefits of both Input and Output queuing.
• Input queuing is speedup of 1
• Output queuing is speedup of N.
• CIOQ is between 1 and N, need to buffer at input
  and output.
• Requires a speedup of only 2 to simulate an
  output queuing switch completely.
• Reasonable given the difficulty of our initial
  goal.
• Guarantees the exact same output as a OQ switch.

12
CIOQ

• Assigns a value called slackness to each VOQ at
  the input.
• Slackness is the urgency of a packet. Slackness
  of 0 means high urgency.
• Inputs and outputs select ports based on a
  priority calculated by the FQ discipline used.
• For FCFS, high priority for packet that came
  earlier.
• Uses Gale-Shapely algorithm to do a stable
  matching of inputs to outputs after they have
  been selected in Phase I.
• No analysis has been done on the throughput of a
  CIOQ switch, just an analytical bound on its
  proximity to OQ switch.

13
CIOQ

• Some recent work in this area
• Delay-Bound Guarantee in CIOQ Switches, H.
  Chao, L. Shen Chen
• Matching Output Queuing with CIOQ Switch, S.
  Chaung, A. Goel, N. McKeown, B. Prabhakar.
• College Admissions and the stability of
  marriage, D. Gale, L. Shapely.

14
ngrSim

• A simulator for ISIS Next Generation Router.
• ngrSim follows a modular design approach.
• All components are coded as modules that can be
  plugged in place of others.
• Allows for easy mix-and-match of various schemes.
• It is event driven completely.
• Everything is driven by the event queue.

15
Event handling
class handler public handler()
virtual handler() virtual void
handle(event e) 0 void set_next_handler(hand
ler h) next_handler h
protected handler next_handler

• An abstract class handler is defined.
• A class event is described which consists of an
  object of type handler.
• The handler is supposed to be the component that
  is responsible for the event.
• When the event is run, the handle function of the
  handler is called.
• Event are enqueued into an event queue.

Event
Port ID
Data
Handler
16
Event handling

• Efficient event handling capability.
• Events are queued in a structure called a
  Skip-list, developed by William Pugh at
  University of Maryland.
• A probabilistic list which allows for O(1)
  inserts, deletes and searches.
• Simulation time is kept track of in the event
  queue.
• The event at the head of the event queue is run
  (handle function called) and the simulation time
  is updated.

http//www.cs.umd.edu/pugh
17
Main Loop for ngrSim

• // Initialize the components tgen, ipp,
• // framer, sched, fab, reframer, opp
• ((crossbar)fab)-gtstart()
• ((tgen)tg)-gtstart()
• while (1)
• event e (eq.instance()).dequeue()
• if (e NULL !sim_running)
• break
• (e-gtget_next_handler())-gthandle(e)
• if ((eq.instance()).get_time() gt simTime)
• sim_running 0
• ((tgen)tg)-gtstop()
• ((crossbar)fab)-gtstop()

18
Design of ngrSim

• The design follows exactly from the design we had
  in the earlier design review.

19
Object Diagram for ngrSim
(64 Bytes, 6 Bytes Fixed Header)
(Variable Length)
Packet
Packet
Cell
Traffic
FRAMER
IPP
Fabric
Route Lookup Checksum Etc
Break packet into cells
TGEN
Scheduler
Traffic
FIFO iSlip CIOQ
Put packets on the outgoing link Use FQ here as
well
OPP
REFRAMER
Assemble cells back to packets
Crossbar
simpleOPP cioqOPP
20
Description of the components

• Packets are generated in TGEN module and are
  passed as events to IPP, Framer and Scheduler.
• Framer breaks packets into cells.
• Cells are 64 bytes with 6 bytes of header.
• 2 input output port number 4 bytes
• 1 byte for cell ID
• 1 byte for flag priority
• Scheduler is an abstract class. All schedulers
  that are implemented derive from this base class.
• We have FCFS, iSlip and CIOQ implemented.
• The scheduler does not pass the packets on to the
  fabric. The fabric is running at a fixed time
  slot which is determined by the cell size and
  line speed.

21
Description of the components

• The fabric pulls cells from the scheduler at
  these regular intervals.
• Fabric is also an abstract class. All fabrics
  will derive form this class.
• We currently have a crossbar fabric implemented.
• The fabric sends the cells to the reframer.
• Reframer reassemble cells into packets.
• If some cell is not received in a certain time
  limit, the partial packet is discarded.
• The reframer passes the reassembled packets to
  the OPP.
• The OPP queues the packets and sends them out at
  the link rate.
• We have simpleOPP and cioqOPP implemented.

22
Statistics

• All modules have statistics.
• They calculate them independently and can be
  queried.
• We keep track of
• cell and packet count
• buffer sizes
• delays
• drops
• We ran experiments for 100 million clock ticks.
• Each data point is a result of 10 runs of the
  same experiment with different random seeds.
• The values were only collected after
  steady-state.
• Due to time-constraint, we could only run for 16
  ports.

23
Throughput (FIFO)
24
Throughput (speedup 1.25)
25
FIFO with different Speedup
26
FIFO with different Speedup
27
Scheduling Algorithm Throughput
28
Scheduling Algorithm Delay
?
29
Buffer sizes with iSlip
30
Scheduling Algorithm
iSlip performs worse than CIOQ
31
To make ISIS a reality

• Physical Specifications of base design
• Chassis Height 10 in. Includes power module
  shelf (AC or DC)
• Chassis Width 17.25 in. not including rack mount
  flanges. can be rack mounted in 19 or 22 in.
• Chassis Depth 18 in. not including cable
  management system
• Chassis Weight approximately 50 lbs. depending
  on configuration
• Standards compliance
• Safety UL1950,IEC60950,IEC60825,TS001,AS/NZS
  3260
• Electromagnetic Emissions FCC Class A, ICES-003
  Class A, EN55022 Class B, VCCI Class B, AS/NZS
  3548 Class B
• NEBS SR-3580 Level 3 Compliant

32
Chassis Configuration

• Modular Design separates Line card module from
  Switch fabric module.
• Redundant Power supplies and fabrics can be used
  to ensure robustness.
• Compact design allows flexibility in rack
  placement.
• The modules are connected through proprietary
  fiber interconnect using LCS protocol.

http//www.pmcsierra.com/products/details/pm9311/
33
Thats cute but show me something big!

• The PMC9113 has 10 Gb/s channels that can be used
  to connect to other switching modules or line
  cards through fiber and LCS protocol.
• So instead of using one 320 Gb/s switch, we can
  use more than one.
• We can extend the capacity of the switch by
  connecting many switch modules in a structure.
• The easiest structure we can think of is a mesh.
• Other possibilities include a hyper-cube.

34
ISIS-A ISIS with an attitude
Switching module with 320 Gb/s capacity
10 Gp/s channel
16 such switches connected in a mesh through 10
Gb/s channels of fiber.
Full mesh of 16 equals 1024 OC-48 ports (2.5 Tb/s)
35
ISIS-A Routing

• Based on source and destination address, line
  cards route to
• ports on the card itself (through the line card)
• Other line cards on same module (through the
  fabric)
• Other switching modules (through the
  interconnect)
• Routing between switching modules based on
  modified hot-potato routing with queue lengths
• Queues only monitor bandwidth utilization of
  channels between modules.
• Built into the chip-set.
• Send to the recipient directly if queue is not
  loaded.
• Send to the least loaded queue recipient,
  otherwise.

36
ISIS-A Routing and QoS.

• Routing is more like load balancing
• Only more intelligent
• Controls delays
• For QoS,
• Some bandwidth is reserved for guaranteed traffic
  (max 10 Gb/s between two modules)
• If such traffic arrives, send directly to the
  recipient (one link) using this reserved
  bandwidth.
• If no such traffic, then use the bandwidth for
  normal traffic.
• Requires modification to the scheduler (which is
  programmable)
• Addressing has to be universal.
• Routing between modules has to be added.

37
Future Work

• Short-term
• Need to run more experiments with different
  parameters.
• Plan to follow-up on the scalability options.
• Get statistical significance for the results.
• Long-term
• Implement other scheduling algorithms.
• Implement other fabric types.
• Explore other possibilities for QoS with IQ
  switches.