Design of a High Performance PlanetLab Node

1
Design of aHigh PerformancePlanetLab Node
John DeHartjdd_at_arl.wustl.edu http//www.arl.wust
l.edu/arl
2
Revision History (10/06 - ??/??)

• 10/3/06 (JDD)
• Added info on buffer descriptor
• 10/11/06 (JDD)
• Corrections to order of Src and Dst IP Addr in IP
  packet headers
• 10/12/06 (JDD)
• Add proposed change to Lookup input and output
  data formats
• 10/24/06 (JDD)
• Updates to Egress Lookup Key
• 10/26/06 (JDD)
• Added HW definitions
• 11/02/06 (JDD)
• Added diagram of 2 code options, 4 slices, 4
  interfaces per slice.
• 11/15/06 (JDD)
• Adding November Intel Demo config info

3
Revision History (6/06 9/06)

• 6/27/06 (JDD)
• Created from Diversified Router version
• 7/5/06 (JDD)
• Still in progress
• 7/6/06 (JDD)
• Still in progress
• 7/10/06 (JDD)
• Minor modifications based on comments from Jing
  and Brandon.
• 7/17/06 (JDD)
• Updated Data between Key_Extract, Lookup and
  Hdr_format on LC projects to consistently use the
  same format for the second word of data IP Pkt
  Length (16b), Reserved(8b), Eth Hdr Len(8b)
• 8/30/06 (JDD)
• Changed fields in hdr_format to QM for IPv4 MR to
  make the agree with LC projects.
• 9/8/06 (JDD)
• Modified IPv4 Memory Map
• Clean separation between Init memory blocks and
  Dynamic memory blocks
• Modified LC Egress Lookup Key (add Sport) and
  Result (add counter index)
• 9/12/06 (JDD and BDH)
• Add Slice Memory Ptr to NN ring structures.
• Add Code option to be passed through to Header
  Format

4
Overview

• These slides are still a bit rough, but it should
  get us started
• Three Project Goals
• PlanetLab Node
• ONL Router
• Diversified Router
• First Priority PlanetLab Node for November Demo
• Phase 0.0 external GE Switch, 1 GPE, 1 LC, gt1
  NPE
• Phase 0.5 same but with Switch Blade
• What is impact on Rx, Tx, QM?
• Phase 1 same but with Multiple GPEs (each as its
  own PlanetLab Node)
• Phase 2 same but with Multiple GPEs, unified as
  one PlanetLab Node.
• Heavy emphasis in these slides is on Phase 0.0
• Probable hardware configuration
• 1 NP Blade for LC
• 1 NP Blade for NPE
• 1 GP Blade
• External GE Switch
• Possible additions
• 1 Switch Blade in place of external switch

5
Overview

• Assume we will use VLANs internally to isolate
  and identify MRs.
• Assume we can set Ethernet MAC addresses on
  blades
• 40 bits fixed,
• 8 bits variable
• View PlanetLab UDP/IP as a new Substrate Link
  Type
• No Substrate Headers used internally or
  externally
• All packets/frames are IP packets in Ethernet
  Frames
• External
• LC ? GPE, GPE ? LC
• LC ? NPE, NPE ? LC
• NPE ? GPE, GPE ? NPE
• In effect, UDP/IP Headers are the Substrate
  Header.
• MN Internal Header still used between MPEs (NPE
  ?? GPE)
• Static Shared NP implementation of MRs.
• Limited, predefined MR code options
• Statically loaded
• Different MR slices run the same code, just use
  different Filters in Lookup table.
• Queuing is slightly different on each system
  (LC_Ingress, LC_Egress, NPE)
• What about ARP?

6
System View
GPE
exception packets useinternal port numbers
NPE
VS
VS

Kernel/VNET
Switch
Default filter directs packet to GPE
LC
Filter directs packet to NPE
IPH
IPH
daddrnextNode
IPH
IPH
daddrthisNode
daddrnextNode
slicepkt
slicepkt
slicepkt
slicepkt
daddrthisNode
7
System View External Switch
PLC
GPE
Switch
/ 5 1Gb
/ 1 1Gb
Net
LC
R T M
/ 1 1Gb
R T M
NPUA
/ 5 1Gb
4 1Gb /
NPUB
Local Host x4
/ 5 1Gb

• Phase 0.0 External GE Switch (16 Ports
  connected)
• LC RTM
• 1 GE Interface connected to PLC/myPLC via Network
• 4 GE Interfaces connected directly to Local Hosts
• Extra data sources and sinks
• NP RTM
• 5 GE Interfaces used by NPUA
• 5 GE Interfaces used by NPUB
• GPE
• 1 Front panel GE Interface

8
System View External Switch
PLC
GPE
Switch
/ 5 1Gb
/ 1 1Gb
Net
LC
R T M
/ 1 1Gb
R T M
NPUA
/ 5 1Gb
4 1Gb /
NPUB
Local Host x4
/ 5 1Gb
TP GigE
Fiber GigE (max of 4 slots on switch)
9
System View Switch Blade
PLC
GPE
Switch Blade
/ 1 1Gb
Net
LC
R T M
/ 1 1Gb
NPE-1
/ 1 10Gb
/ 1 10Gb
/ 9 1Gb
Local Host x9

• Phase 05 Switch Blade
• LC RTM
• 1GE Interface connected to PLC/myPLC via Network
• 9 GE Interfaces connected directly to Local Hosts
• 1 10Gb interface via backplane
• NPE
• Will not need an RTM
• 1 10Gb interface via backplane
• GPE
• 1 GE Interface on the Fabric connector

10
PlanetLab Ingress LC Input Frame

• New PlanetLab Substrate Link Type
• Configured SL Type
• LC is told at boot/init time that this is its one
  and only SL Type.
• Similar to the way P2P-DC is handled.
• SL Type 0101b
• Port May be a dont care
• IP DAddr Verifies that packet is for our node
• IP Proto UDP
• Could be a UDP tunnel to a slice
• UDP DPort Indicates which slice
• Default route is to the GPE
• Key
• SL0101b
• Port May be a dont care.
• IP DAddr our node address

DstAddr (6B)
Ethernet Header
SrcAddr (6B)
DstAddr (6B)
Type802.1Q (2B)
SrcAddr (6B)
VLAN (2B)
TypeIP (2B)
TypeIP (2B)
Ver/HLen/Tos/Len (4B)
Ver/HLen/Tos/Len (4B)
ID/Flags/FragOff (4B)
ID/Flags/FragOff (4B)
TTL (1B)
TTL (1B)
Protocol UDP (1B)
Protocol UDP (1B)
Hdr Cksum (2B)
Hdr Cksum (2B)
Src Addr (4B)
Src Addr (4B)
IP Header
Dst Addr (4B)
Dst Addr (4B)
IP Options (0-40B)
IP Options (0-40B)
Src Port (2B)
Src Port (2B)
UDP Header
Dst Port (2B)
Dst Port (2B)
UDP length (2B)
UDP length (2B)
UDP checksum (2B)
UDP checksum (2B)
UDP Payload (MN Packet)
UDP Payload (MN Packet)
PAD (nB)
PAD (nB)
Ethernet Trailer
CRC (4B)
CRC (4B)
PlanetLab IPv4 Key(0x5) (64 bits)
11
PlanetLab Ingress LC Processing

• Ingress Processing
• Portions are similar to IPv4 MR Parse.
• IP Header checks/validation
• Check that version is IPv4
• Check IP Header checksum
• Ignore options (Leave as is and forward on)
• Drop if fragmented (What if GPE bound?)
• Extract
• IP Protocol
• IP Dst Addr
• UDP Dst Port
• Or whatever is in the 2B that would be the UDP
  port if the IP Protocol were UDP
• We shouldnt have to worry about what might be in
  this field if the IP Protocol is not UDP
• Perform Lookup
• Result contains
• Per MI Stats/Counter Index
• Ethernet DAddr for destination blade
• VLAN for destination slice (if needed)
• No changes made to IP Header

DstAddr (6B)
Ethernet Header
SrcAddr (6B)
DstAddr (6B)
Type802.1Q (2B)
SrcAddr (6B)
VLAN (2B)
TypeIP (2B)
TypeIP (2B)
Ver/HLen/Tos/Len (4B)
Ver/HLen/Tos/Len (4B)
ID/Flags/FragOff (4B)
ID/Flags/FragOff (4B)
TTL (1B)
TTL (1B)
Protocol UDP (1B)
Protocol UDP (1B)
Hdr Cksum (2B)
Hdr Cksum (2B)
Src Addr (4B)
Src Addr (4B)
IP Header
Dst Addr (4B)
Dst Addr (4B)
IP Options (0-40B)
IP Options (0-40B)
Src Port (2B)
Src Port (2B)
UDP Header
Dst Port (2B)
Dst Port (2B)
UDP length (2B)
UDP length (2B)
UDP checksum (2B)
UDP checksum (2B)
UDP Payload (MN Packet)
UDP Payload (MN Packet)
PAD (nB)
PAD (nB)
Ethernet Trailer
CRC (4B)
CRC (4B)
Indicates fields that need to be read
Indicates 8-Byte Boundaries Assuming no IP Options
12
PlanetLab Ingress LC Output Frame

• Ethernet Header is only thing that changes
• DAddr MAC Address of GPE/NPE (Result)
• 40 bits are static
• 8 bits variable and stored in Result
• SAddr MAC Address of LC (static)
• Type 802.1Q (static)
• VLAN Slice VLAN (Result)
• Type IP (static)
• Total number of 8-Byte Reads 1
• Need to read first part of IP header so when we
  do the write of last part of ethernet header we
  can fill out the 8-Byte Write.
• Total number of 8-Byte Writes 3

DstAddr (6B)
Ethernet Header
SrcAddr (6B)
Type802.1Q (2B)
VLAN (2B)
TypeIP (2B)
Ver/HLen/Tos/Len (4B)
ID/Flags/FragOff (4B)
TTL (1B)
Protocol UDP (1B)
Hdr Cksum (2B)
Src Addr (4B)
IP Header
Dst Addr (4B)
IP Options (0-40B)
Src Port (2B)
UDP Header
Dst Port (2B)
UDP length (2B)
UDP checksum (2B)
UDP Payload (MN Packet)
PAD (nB)
Ethernet Trailer
CRC (4B)
Indicates fields that need to be written
Indicates 8-Byte Boundaries Assuming no IP Options
13
PlanetLab NPE Input Frame from LC

• Ethernet Header
• DstAddr MAC address of NPE
• SrcAddr MAC address of LC
• VLAN One VLAN per MR (MR Slice)
• IP Header
• Dst Addr IP address of this node (phase 0)
• Src Addr IP address of previous hop
• Protocol UDP
• UDP Header
• Dst Port Identifies input tunnel
• Src Port with IP Src Addr identifies sending
  entity

DstAddr (6B)
Ethernet Header
SrcAddr (6B)
Type802.1Q (2B)
VLAN (2B)
TypeIP (2B)
Ver/HLen/Tos/Len (4B)
ID/Flags/FragOff (4B)
TTL (1B)
Protocol UDP (1B)
Hdr Cksum (2B)
Src Addr (4B)
IP Header
Dst Addr (4B)
IP Options (0-40B)
Src Port (2B)
UDP Header
Dst Port (2B)
UDP length (2B)
UDP checksum (2B)
UDP Payload (MN Packet)
PAD (nB)
Ethernet Trailer
CRC (4B)
Indicates 8-Byte Boundaries Assuming no IP Options
14
PlanetLab NPE Output Frame to LC

• Re-writes IP Header
• Dst Addr Next Hop
• Src Addr NPEs IP Address
• Re-writes UDP Header
• Src Port NPEs end of the tunnel to next hop
• Dst Port Other end of tunnel to next hop.
• UDP checksum?
• Re-writes Ethernet Header

DstAddr (6B)
Ethernet Header
SrcAddr (6B)
Type802.1Q (2B)
VLAN (2B)
TypeIP (2B)
Ver/HLen/Tos/Len (4B)
ID/Flags/FragOff (4B)
TTL (1B)
Protocol UDP (1B)
Hdr Cksum (2B)
Src Addr (4B)
IP Header
Dst Addr (4B)
IP Options (0-40B)
Src Port (2B)
UDP Header
Dst Port (2B)
UDP length (2B)
UDP checksum (2B)
UDP Payload (MN Packet)
PAD (nB)
Ethernet Trailer
CRC (4B)
Indicates 8-Byte Boundaries Assuming no IP Options
15
PlanetLab NPE Local Delivery Frame FROM GPE

• IP Header
• Dst Addr NPEs IP Address
• Src Addr GPEs IP Address
• No IP Options
• UDP Header
• Src Port GPEs end of the tunnel
• Dst Port NPEs end of the tunnel
• UDP checksum?
• MN Internal Header
• As defined for diversified router

DstAddr (6B)
Ethernet Header
SrcAddr (6B)
Type802.1Q (2B)
VLAN (2B)
TypeIP (2B)
Ver/HLen/Tos/Len (4B)
ID/Flags/FragOff (4B)
TTL (1B)
Protocol UDP (1B)
Hdr Cksum (2B)
Src Addr (4B)
IP Header
Dst Addr (4B)
Src Port (2B)
UDP Header
Dst Port (2B)
UDP length (2B)
UDP checksum (2B)
MN Internal Hdr (4-10B)
UDP Payload (MN Packet)
PAD (nB)
CRC (4B)
Ethernet Trailer
Indicates 8-Byte Boundaries Assuming no IP Options
16
PlanetLab NPE Local Delivery Frame TO GPE

• Re-writes IP Header
• Dst Addr GPEs IP Address
• Src Addr NPEs IP Address
• No IP Options
• Re-writes UDP Header
• Src Port NPEs end of the tunnel to GPE
• Dst Port GPEs end of the tunnel from NPE
• UDP checksum?
• MN Internal Header
• Need to look at the details of what needs to go
  in here now that we have no explicit RxMI and
  TxMI fields/values.
• Ethernet Header

DstAddr (6B)
Ethernet Header
SrcAddr (6B)
Type802.1Q (2B)
VLAN (2B)
TypeIP (2B)
Ver/HLen/Tos/Len (4B)
ID/Flags/FragOff (4B)
TTL (1B)
Protocol UDP (1B)
Hdr Cksum (2B)
Src Addr (4B)
IP Header
Dst Addr (4B)
Src Port (2B)
UDP Header
Dst Port (2B)
UDP length (2B)
UDP checksum (2B)
MN Internal Hdr (4-10B)
UDP Payload (MN Packet)
PAD (nB)
CRC (4B)
Ethernet Trailer
Indicates 8-Byte Boundaries Assuming no IP Options
17
PlanetLab NPE Exception Path Frame FROM GPE

• IP Header
• Dst Addr NPEs IP Address
• Src Addr GPEs IP Address
• No IP Options
• UDP Header
• Src Port GPEs end of the tunnel
• Dst Port NPEs end of the tunnel
• UDP checksum?
• MN Internal Header
• As defined for diversified router

DstAddr (6B)
Ethernet Header
SrcAddr (6B)
Type802.1Q (2B)
VLAN (2B)
TypeIP (2B)
Ver/HLen/Tos/Len (4B)
ID/Flags/FragOff (4B)
TTL (1B)
Protocol UDP (1B)
Hdr Cksum (2B)
Src Addr (4B)
IP Header
Dst Addr (4B)
Src Port (2B)
UDP Header
Dst Port (2B)
UDP length (2B)
UDP checksum (2B)
MN Internal Hdr (4-10B)
UDP Payload (MN Packet)
PAD (nB)
Ethernet Trailer
CRC (4B)
Indicates 8-Byte Boundaries Assuming no IP Options
18
PlanetLab NPE Exception Path Frame TO GPE

• Re-writes IP Header
• Dst Addr GPEs IP Address
• Src Addr NPEs IP Address
• No IP Options
• Re-writes UDP Header
• Src Port NPEs end of the tunnel to GPE
• Dst Port GPEs end of the tunnel from NPE
• UDP checksum?
• MN Internal Header
• As defined for diversified router

DstAddr (6B)
Ethernet Header
SrcAddr (6B)
Type802.1Q (2B)
VLAN (2B)
TypeIP (2B)
Ver/HLen/Tos/Len (4B)
ID/Flags/FragOff (4B)
TTL (1B)
Protocol UDP (1B)
Hdr Cksum (2B)
Src Addr (4B)
IP Header
Dst Addr (4B)
Src Port (2B)
UDP Header
Dst Port (2B)
UDP length (2B)
UDP checksum (2B)
MN Internal Hdr (4-10B)
UDP Payload (MN Packet)
PAD (nB)
CRC (4B)
Ethernet Trailer
Indicates 8-Byte Boundaries Assuming no IP Options
19
PlanetLab Egress LC Input Frame

• Ethernet Header addressed to LC
• IP Packet should be complete and LC does not need
  to touch it. (Phase 0)
• Should not even need to do hdr checksum
• Lookup Key
• IP Protocol (8b)
• UDP Sport (16b)

DstAddr (6B)
Ethernet Header
SrcAddr (6B)
Type802.1Q (2B)
VLAN (2B)
TypeIP (2B)
Ver/HLen/Tos/Len (4B)
ID/Flags/FragOff (4B)
TTL (1B)
Protocol UDP (1B)
Hdr Cksum (2B)
Src Addr (4B)
IP Header
Dst Addr (4B)
IP Options (0-40B)
Src Port (2B)
UDP Header
Dst Port (2B)
UDP length (2B)
UDP checksum (2B)
UDP Payload (MN Packet)
PAD (nB)
Ethernet Trailer
CRC (4B)
Indicates fields that need to be read
Indicates 8-Byte Boundaries Assuming no IP Options
20
PlanetLab Egress LC Processing

• Egress Processing
• Extract
• IP Dst Addr and UDP Sport
• Perform Lookup
• Need to generate
• MAC DAddr for next hop
• VLAN, maybe
• Assume there are a limited number of next hops
• Local Hosts
• Routers connected to the local subnet.
• Result contains
• Per MI Stats/Counter Index
• Ethernet DstAddr
• VLAN if needed
• OR
• L2 Lookup Table Index
• L2 Lookup Table Entry
• L2 Header Size (14B or 18B)
• 18 Bytes of Data

DstAddr (6B)
Ethernet Header
SrcAddr (6B)
Type802.1Q (2B)
VLAN (2B)
TypeIP (2B)
Ver/HLen/Tos/Len (4B)
ID/Flags/FragOff (4B)
TTL (1B)
Protocol UDP (1B)
Hdr Cksum (2B)
Src Addr (4B)
IP Header
Dst Addr (4B)
IP Options (0-40B)
Src Port (2B)
UDP Header
Dst Port (2B)
UDP length (2B)
UDP checksum (2B)
UDP Payload (MN Packet)
PAD (nB)
Ethernet Trailer
CRC (4B)
Indicates fields that need to be read
Indicates 8-Byte Boundaries Assuming no IP Options
21
PlanetLab Egress LC Output Frame

• Ethernet Header is only thing that changes.
• DAddr MAC Address of next hop (Result)
• SAddr MAC Address of LC (static)
• Optional
• Type 802.1Q (Result)
• VLAN Slice VLAN (Result)
• Type IP (static)
• Total number of 8-Byte Reads 1
• Need to read first 2 or 6 Bytes of the IP header
  so when we do the write of last part of ethernet
  header we can fill out the 8-Byte Write.
• Total number of 8-Byte Writes 2 or 3

DstAddr (6B)
Ethernet Header
SrcAddr (6B)
Type802.1Q (2B)
VLAN (2B)
TypeIP (2B)
Ver/HLen/Tos/Len (4B)
ID/Flags/FragOff (4B)
TTL (1B)
Protocol UDP (1B)
Hdr Cksum (2B)
Src Addr (4B)
IP Header
Dst Addr (4B)
IP Options (0-40B)
Src Port (2B)
UDP Header
Dst Port (2B)
UDP length (2B)
UDP checksum (2B)
UDP Payload (MN Packet)
PAD (nB)
Ethernet Trailer
CRC (4B)
Indicates fields that need to be written
Indicates 8-Byte Boundaries Assuming no IP Options
22
Queuing

• Assume we are using an external GE switch
• LC RTM
• 5 internal ports go to switch for traffic to/from
  NPE(s) and GPE(s)
• 5 external ports
• 1 to Internet
• 4 to local hosts
• NPE RTM
• 5 ports used by NPUA to switch
• Each port is associated with 1 of the LCs
  external ports
• 5 ports used by NPUB to switch
• Each port is associated with 1 of the LCs
  external ports
• GPE
• 1 GE port to switch
• NPE Queueing
• Needs Rate control per port

23
NPE Queuing

• Queueing on a per port basis
• Ports split across NPUA and NPUB (Phase 0.0)
• Rate control per port
• Rate control will need to be dynamically
  adjustable so we can balance bandwidth usage
  across NPs and GPE
• Each MR gets N queues per port
• MRs choose how to use the N queues
• Quantum assigned per MR for each port
• MRs can choose how to split among their N queues
• May or may not make sense to assign queues on a
  per MI basis
• Quantum being split across the queues means that
  an MR with only one active MI may not get its
  fair share.

24
NPE Queuing
Port 1(to LC Port 1)
Port 2(to LC Port 2)
N P U A
Port 3(to LC Port 3)
Port 4(to LC Port 4)
Port 5(to LC Port 5)
QM/Schd
Port 6 (to LC Port 1)
N P U B
Port 7(to LC Port 2)
Port 8(to LC Port 3)
Port 9(to LC Port 4)
Port 10(to LC Port 5)
25
LC Ingress Queuing

• 1 5-port QM support 5 GPEs
• Each port supports one GPE
• Only one queue needed per port.
• Rate control, 1 Gb/s per port
• In Phase 0.0, HW flow control will
  provide/enforce this rate.
• 1 5-port QM support all NPEs
• Each port supports one NPE
• Only one queue needed per port.
• Rate control should not be needed

26
LC Egress Queuing

• Queueing on a per MR per port basis
• Similar to NPE Queuing, but each MR gets 1 queue
  per port
• Default is equal quantum assigned to each MR on
  each port
• All get equal fair share
• Allow for changing quanta.

27
LC Functional Blocks
Lookup (2 ME)
Switch Tx (2 ME)
QM/Schd (2 ME)
Hdr Format (1 ME)
S W I T C H
Phy Int Rx (2 ME)
Key Extract (2 ME)
QM/Schd (2 ME)
Lookup (2 ME)
Key Extract (1 ME)
Switch Rx (2 ME)
Phy Int Tx (2 ME)
Hdr Format (1 ME)
28
LC Ingress Functional Blocks
Lookup
Switch Tx
Hdr Format
S W I T C H
Phy Int Rx
Key Extract
Buf Handle(32b)
RBUF
Port (4b)
Reserved (12b)
Eth. Frame Len (16b)

• Rx (2 Microengines)
• Function
• Coordinate transfer of packets from RBUF to DRAM

29
LC Ingress Functional Blocks
Lookup
Switch Tx
Hdr Format
S W I T C H
Phy Int Rx
Key Extract
Buf Handle(32b)
Buf Handle(32b)
IP Pkt Length (16b)
Reserved (8b)
Eth Hdr Len (8b)
Port (4b)
Reserved (12b)
Eth. Frame Len (16b)
Lookup Key63-32 (32b)
Lookup Key 31-0 (32b)
IP Hdr 1st Word (32b)

• Key_Extract (1 Microengine)
• Function
• Extracts lookup key.
• Peel ARP packets off and send to XScale???
• Lookup Key (64b)
• SL Type (4b) 0101b
• Port (4b) May not be needed
• IP DAddr (32b)
• IP Proto (8b)
• UDP DPort (16b)
• Notes
• Frame offset in buffer is a constant and does not
  need to be read from Buffer Descriptor
• Ethernet Hdr Length should be passed along chain
  so Hdr Format can figure out where to start
  writing its stuff.
• Ethernet Header could have different lengths
  depending on whether VLANs are present or not.

30
LC Ingress Functional Blocks
Lookup
Switch Tx
Hdr Format
S W I T C H
Phy Int Rx
Key Extract
Buf Handle(32b)
Buf Handle(32b)
IP Pkt Length (16b)
Reserved (8b)
IP Pkt Length (16b)
Eth Hdr Len (8b)
Reserved (8b)
Eth Hdr Len (8b)
Lookup Key63-32 (32b)
VLAN (16b)
Stats Index (16b)
Lookup Key 31-0 (32b)
QID (20b)
DAddr (8b)
Port (4b)
IP Hdr 1st Word (32b)
IP Hdr 1st Word (32b)

• Lookup Notes on next page

31
LC Ingress Functional Blocks

• Lookup
• Function
• Performs Lookup and passes result on to Hdr
  Format.
• Lookup Key (64b)
• SL Type (4b) 0101b
• Port (4b) May not be needed
• IP DAddr (32b)
• IP Proto (8b)
• UDP DPort (16b)
• Lookup Result (56b)
• DAddr (8b) only 8 bits of Ethernet DAddr are
  variable, other 40 are static per node.
• VLAN (12b)
• QID (20b)
• Stats Index (16b)
• Port (4b)
• For case with external switch it is the actual
  physical interface to use
• Also one port per GPE and one port per NPE
• For case with switch blade, it is just used to
  spread traffic across QM/Scheduler?
• Notes

32
LC Ingress Functional Blocks
Lookup
Switch Tx
Hdr Format
S W I T C H
Phy Int Rx
Key Extract
Buffer Handle(32b)
Buf Handle(32b)
IP Pkt Length (16b)
Reserved (8b)
Eth Hdr Len (8b)
QID(20b)
Rsv (4b)
Port (4b)
Rsv (4b)
VLAN (16b)
Stats Index (16b)
Frame Length (16b)
Stats Index (16b)
QID (20b)
DAddr (8b)
Port (4b)
IP Hdr 1st Word (32b)

• Hdr Format
• Function
• From lookup result
• re-writes just the ethernet header in DRAM to
  make frame ready to transmit.
• Extract QID, Port, Stats Index and Frame Length
  to pass on to QM/Scheduler
• May need to increment a counter based on Stats
  Index.
• Notes
• Pass Size on to QM/Scheduler so it does not have
  to read buffer descriptor for Enqueue to update Q
  Length.
• Offset to beginning of old Ethernet header should
  be constant but we dont necessarily know how
  long it was so we dont know where to put our new
  one.
• Ethernet Hdr Len is used to determine where new
  header should go

33
LC Ingress Functional Blocks
Lookup
Switch Tx
Hdr Format
S W I T C H
Phy Int Rx
Key Extract
Buffer Handle(32b)
QID(20b)
Rsv (4b)
Rsv (4b)
Port (4b)
V Valid Bit
Frame Length (16b)
Stats Index (16b)

• QM/Scheduler (See Saileshs slides for more
  details)
• Function
• Enqueue and Dequeue from queues
• Scheduling algorithm
• Drop Policy
• Notes

34
LC Ingress Functional Blocks
Lookup
Switch Tx
Hdr Format
S W I T C H
Phy Int Rx
Key Extract
TBUF
V Valid Bit

• Switch TX
• Function
• Coordinate transfer of packets from DRAM to TBUF
• Notes

35
LC Egress Functional Blocks
Lookup
Phy Int Tx
Hdr Format
S W I T C H
Key Extract
Switch Rx
Buf Handle(32b)
RBUF
Port (4b)
Reserved (12b)
Eth. Frame Len (16b)

• Rx
• Function
• Coordinate transfer of packets from RBUF to DRAM
• Notes
• Do we need port?
• May not make sense to remove it since, it is
  there for other versions of Rx.

36
LC Egress Functional Blocks
Lookup
Phy Int Tx
Hdr Format
S W I T C H
Key Extract
Switch Rx
Buf Handle(32b)
Buf Handle(32b)
Port (4b)
Reserved (12b)
Eth. Frame Len (16b)
IP DAddr (32b)
Lookup Key UDP SPort (16b)
Lookup Key IP Proto (8b)
Reserved (8b)
IP Hdr 1st Word (32b)

• Key_Extract
• Function
• Extracts lookup key
• Notes

37
LC Egress Functional Blocks
S W I T C H
Lookup
Key Extract
Switch Rx
Phy Int Tx
Hdr Format
Buf Handle(32b)
Buf Handle(32b)
IP DAddr (32b)
IP DAddr (32b)
Lookup Result 63-32 (32b)
Lookup Key UDP SPort (16b)
Lookup Key IP Proto (8b)
Reserved (8b)
Lookup Result 31-0 (32b)

• Lookup
• Function
• Performs Lookup and passes result on to Hdr
  Format.
• Lookup Key
• IP Protocol (8b)
• UDP Sport (16b)
• Lookup Result (52b)
• VLAN (12b) Value of 0x000 or 0xFFF, indicates
  invalid?
• QID (20b)
• Port (4b)
• Stats/Counter Index (16b)
• Static values for Egress Ethernet address
• Ethernet SAddr
• Types IP and/or 802.1Q
• Notes
• Lookup does no processing on the lookup result.

IP Hdr 1st Word (32b)
IP Hdr 1st Word (32b)
38
Lookup Result
Buf Handle(32b)
IP Pkt Length (16b)
Reserved (8b)
Eth Hdr Len (8b)
IP DAddr (32b)
VLAN(12b)
Stats/Counter Index (16b)
Rsvd (4b)
QID (20b)
Rsvd (4b)
Port (4b)
Rsvd (4b)
IP Hdr 1st Word (32b)
39
LC Egress Functional Blocks
Lookup
Phy Int Tx
Hdr Format
S W I T C H
Key Extract
Switch Rx
Buffer Handle(32b)
Buf Handle(32b)
QID(20b)
Rsv (4b)
Port (4b)
Rsv (4b)
IP DAddr (32b)
Lookup Result 63-32 (32b)
Ethernet Frame Length (16b)
Cntr Index (16b)
Lookup Result 31-0 (32b)
IP Hdr 1st Word (32b)

• Hdr Format
• Function
• From lookup result
• re-writes ethernet header in DRAM to make frame
  ready to transmit.
• Extract QID and frame length to pass on to
  QM/Scheduler
• Notes
• Pass Size on to QM/Scheduler so it does not have
  to read buffer descriptor for Enqueue to update Q
  Length.

40
LC Egress Functional Blocks
S W I T C H
Lookup
Key Extract
Switch Rx
Phy Int Tx
Hdr Format
Buffer Handle(32b)
QID(20b)
Rsv (4b)
Port (4b)
Rsv (4b)
V Valid Bit
Ethernet Frame Length (16b)
Cntr Index (16b)

• QM/Scheduler (See Saileshs slides for more
  details)
• Function
• Enqueue and Dequeue from queues
• Scheduling algorithm
• Drop Policy
• Memory Accesses
• DRAM None
• SRAM
• Q-Array Reads and Writes
• Scheduling Data Structure Reads and Writes
• QLength Data Structure Reads and Writes
• Dequeue Read Buffer Descriptor to retrieve
  Packet Size
• Buffer Descriptor Accesses Read packet size
• Notes

41
LC Egress Functional Blocks
S W I T C H
Lookup
Key Extract
Switch Rx
Phy Int Tx
Hdr Format
TBUF
V Valid Bit

• Switch TX
• Function
• Coordinate transfer of packets from DRAM to TBUF
• Memory Accesses
• SRAM Read Buffer Descriptor
• DRAM Transfer to TBUF
• Buffer Descriptor Accesses
• Read Size and Offset
• Notes
• Calculate DRAM address based on SRAM Descriptor
  address in buffer handle

42
NPE Functional Blocks

• Phase 0.0
• Each NPU will only support 5 LC Interfaces.
• Only need 1 Tx ME
• Only need 1 QM ME

43
NPE Functional Blocks

• Later Phases
• We may need some input queuing/buffering to
  absorb bursts at high input rates (10Gb/s,
  20MPkts/s) to keep Parse from being overloaded.
• Add an SRAM Ring between Demux and Parse instead
  of NN Ring
• If needed, add an extra ME to read from SRAM ring
  and put into NN ring

44
IPv4 MR Functional Blocks
RBUF

• Rx
• Function
• Coordinate transfer of packets from RBUF to DRAM
• Notes
• Well pass the Buffer Handle which contains the
  SRAM address of the buffer descriptor.
• From the SRAM address of the descriptor we can
  calculate the DRAM address of the buffer data.

45
IPv4 MR Functional Blocks
DstAddr (6B)
Ethernet Header
SrcAddr (6B)
Type802.1Q (2B)
VLAN (2B)
TypeIP (2B)
Ver/HLen/Tos/Len (4B)
ID/Flags/FragOff (4B)
TTL (1B)
Protocol UDP (1B)
Hdr Cksum (2B)
Src Addr (4B)
IP Header
Slice Data Ptr (32b)
Dst Addr (4B)

• SRAM Pointer/Range to static data area need to be
  added.

IP Options (0-40B)
Src Port (2B)
UDP Header
Dst Port (2B)
UDP length (2B)
UDP checksum (2B)
UDP Payload (MN Packet)

• Demux notes on next slide

PAD (nB)
Ethernet Trailer
CRC (4B)
46
Demux

• Function
• Read Pkt Header from DRAM
• Can we assume IP Header checksum is ok?
• This frame comes from LC or from GPE. Are they
  each trusted?
• Except for the possibility of options, we have to
  read all parts of the IP header anyway, so maybe
  go ahead and calculate the checksum if we have
  the cycles.
• Extract the following fields from Pkt Header and
  pass to Parse
• VLAN equivalent to Slice ID (Slice MR)
• UDP DPort equivalent to Substrate MI
• IP SAddr, UDP Sport extra interface information
  for planetlab slices.
• UDP length equivalent to MN frame length
• Calculate offset into buffer of start of MN Pkt
  Header and pass to Parse
• Based on Slice ID, determine code option.
• How do we intended to do this?
• Load a table in Local memory from SRAM at init
  time?
• Notes
• Code Identifies Code Option
• Which set of static Parse code to execute.
• We will allow a very small number of code
  options.
• How does Demux identify code option?

47
IPv4 MR Functional Blocks
Buf Handle(32b)
MN Frm Offset (16b)
MN Frm Length(16b)
Rx UDP DPort (16b)
Slice ID (VLAN) (16b)
Rx IP SAddr (32b)
Reserved (12b)
Rx UDP SPort (16b)
Code (4b)
Slice Data Ptr (32b)
Slice Data Ptr (32b)
Reserved (28b)
Code (4b)

• Parse notes on following page

48
Parse Notes

• Parse
• Function
• MR-specific header processing
• Handles IPv4 header validation
• Decrements TTL and recalculates Hdr Checksum.
• Generate MR-specific lookup key (144 bits) from
  packet
• Generate Exception bits to be passed on to Hdr
  Format (via Lookup) so Hdr Format can create shim
  fields for slow path packets going to Control
  Processor.
• Input data to Parse is MN Frame Offset and Length
• Output data from Parse is IP Pkt Offset and
  Length
• If there is a MN Internal Header, Parse basically
  consumes it.
• Hdr Format needs to start at the beginning of the
  IP Header and re-write headers upward in the
  Buffer.
• If Parse receives a !Reclassify frame/pkt from
  the CP,
• it should not decrement the TTL.
• Assume this was done on the first trip through
  the MR.
• it should not generate exceptions again.
• Code Identifies Code Option
• Which set of static Parse code to execute.
• We will allow a very small number of code
  options.
• L Flags

49
IPv4 MR Functional Blocks
Lookup
Tx
DeMux
Rx
Parse
Header Format
Buf Handle(32b)
IP Pkt Length (16b)
IP Pkt Offset (16b)
Rx UDP DPort(16b)
Slice ID (VLAN) (16b)
Cntr Index (16b)
R S V d (1b)
D (1b)
H (1b)
Exception Bits (12b)
L D (1b)
Tx IP DAddr (32b)
Tx UDP SPort(16b)
Tx UDP DPort (16b)
Port (4b)
QID(20b)
DA(8b)
Slice Data Ptr (32b)
Slice Data Ptr (32b)
Reserved (28b)
Code (4b)
Reserved (28b)
Code (4b)

• Lookup notes on next page

50
IPv4 MR Functional Blocks
Lookup
Tx
DeMux
Rx
Parse
Header Format
Buf Handle(32b)
Buf Handle(32b)
IP Pkt Length (16b)
IP Pkt Offset (16b)
IP Pkt Length (16b)
IP Pkt Offset (16b)
Lookup Key143-112 Slice ID/Rx UDP DPort (32b)
Rx UDP DPort(16b)
Slice ID (VLAN) (16b)
Lookup Key111-80 DA (32b)
Cntr Index (16b)
R S V d (1b)
D (1b)
H (1b)
Reserved (11b)
L D (1b)
R S V d (1b)
Lookup Key 79-48 SA (32b)
Tx IP DAddr (32b)
Lookup Key 47-16 Ports (32b)
Lookup Key Proto/TCP_Flags 15- 0 (16b)
Reserved (16 b)
Tx UDP SPort(16b)
Tx UDP DPort (16b)
Port (4b)
QID(20b)
DA(8b)
Slice Data Ptr (32b)
Slice Data Ptr (32b)
Reserved (12 b)
Code (4b)
L Flags (4b)
Exception Bits (12b)
Reserved (12 b)
Code (4b)
Reserved (4b)
Exception Bits (12b)

• Lookup notes on next page
• PROPOSED CHANGE TO LOOKUP INPUT AND OUTPUT DATA
  FORMATS (10/12/06)

51
Lookup

• Function
• Perform lookup in TCAM based on MR Id and lookup
  key
• Result
• IP DAddr (32b)
• UDP SPort (16b)
• UDP DPort (16b)
• Eth Daddr (8b) low order 8 bits. Top 40 are
  pre-defined
• Port (4b) Phase 0.0
• QID (20b)
• Cntr Index (16b) Used for incrementing Counters
• Output
• Buf Handle
• Exception Bits For Parse to communicate to
  Header format info about exception packets
• Slice ID (VLAN)
• Rx UDP DPort
• IP Pkt Length Length of just the IP Pkt
• IP Pkt Offset Offset from start of buffer to the
  start of IP Pkt header
• IP/UDP Header fields IP DAddr, UDP DPort, UDP
  SPort
• QID

Slice Data Ptr (32b)
Reserved (28b)
Code (4b)
52
IPv4 MR Functional Blocks
Lookup
Tx
Header Format
DeMux
Rx
Parse
Slice Data Ptr (32b)
Reserved (28b)
Code (4b)

• Header Format notes on next page.

53
IPv4 MR Functional Blocks

• Header Format
• Function
• MR specific packet header formatting
• MR specific Lookup Result processing
• Drop bit, Hit/Miss bits, NH, MAC, LD,
• LD and Exceptions may go to different UDP Ports.
• Result
• Ethernet DAddr of LC (Result)
• Constant/Static fields needed
• IP DAddr of GPE for exception and LD (configured)
• IP SAddr (configured)
• UDP Sport and UDP DPort for exception and LD
  (configured)
• QID and Port to use for exception path and LD
  (configured)
• IP Proto UDP (constant)
• Other IP Header fields calculated or constants
• Ethernet SAddr of NPE (configured)
• VLAN (passed as input)
• First Ethernet Type 802.1Q (constant)
• Second Ethernet Type IP (constant)

54
IPv4 MR Functional Blocks
Lookup
Tx
Header Format
DeMux
Rx
Parse

• QM
• Function
• CRF queue management for Meta Interface queues
• For performance reasons, QM may actually be
  implemented as multiple instances
• Each instance on a separate ME would support a
  separate set of Meta Interfaces.
• See next slide for more details

55
QM/Scheduler on Multiple MEs
QM/Schd (1 ME)
Input Hlpr (1 ME)
HeaderFormat
Tx
QM/Schd (1 ME)
Tx
NN/Scratch Rings
NN Ring

• QID(32b)
• Reserved (8b)
• QM ID (3b)
• QID(17b) 1M queues per QM
• Input Hlpr would use QM ID to select Scratch ring
  on which to put request.
• QM/Sched then sends on its output NN/scratch ring
  to its associated Tx
• With 64 entries in Q-Array and 16 entries in CAM,
  max number of QM/Schds is probably 4 (2 bits).
• Well set aside 3 bits to give us flexibility in
  the future.

56
IPv4 MR Functional Blocks
Lookup
Tx
Header Format
DeMux
Rx
Parse
TBUF

• Tx
• Function
• Coordinate transfer of packets from DRAM to TBUF

57
LC Buffer Descriptor

• Hopefully we can use the same buffer descriptor
  for the LC and the CRF Processing Engine.
• There might be some fields that are used on one
  and not on the other but thats ok (MR_ID, TxMI,
  VLAN not needed on LC)
• PE Buffer Descriptor
• LW0 buffer_next 32 bits Next Buffer Pointer
  (in a chain of buffers)
• LW1 offset 16 bits Offset to start of data
  in bytes
• LW1 BufferSize 16 bits Length of data in the
  current buffer in bytes
• LW2 reserved 8 bits reserved/unused
• LW2 reserved 4 bits reserved/unused
• LW2 free_list 4 bits Freelist ID
• LW2 packet_size 16 bits (Total packet size
  across multiple buffers)
• LW3 MR_ID 16 bits Meta Router ID
• LW3 TxMI 16 bits Transmit Meta Interface
• LW4 VLAN 16 bits VLAN
• LW4 reserved 16 bits reserved/unused
• LW5 reserved 32 bits reserved/unused
• LW6 reserved 32 bits reserved/unused
• LW7 packet_next 32 bits pointer to next packet
  (unused in cell mode)
• Leave multi-buffer fields there as a template for
  the dedicated blade implementation of a
  jumbo-frame MR.

58
Intel Buffer Descriptor
Buffer_Next (32b)
LW0
Buffer_Size (16b)
Offset (16b)
LW1
Packet_Size (16b)
Hdr_Type (8b)
Free_list (4b)
Rx_stat (4b)
LW2
Input_Port (16b)
Output_Port (16b)
LW3
Next_Hop_ID (16b)
Fabric_Port (8b)
Reserved (4b)
NHID type (4b)
LW4
FlowID (32b)
ColorID (4b)
Reserved (4b)
LW5
Class_ID (16b)
Reserved (16b)
LW6
Packet_Next (32b)
LW7
59
Our Buffer Descriptor
Buffer_Next (32b)
LW0
Buffer_Size (16b)
Offset (16b)
LW1
Packet_Size (16b)
Reserved (8b)
Free_list 0000 (4b)
Reserved (4b)
LW2
Reserved (16b)
Stats Index (16b)
LW3
Reserved (16b)
Reserved (8b)
Reserved (4b)
Reserved (4b)
LW4
Reserved (32b)
Reserved (4b)
Reserved (4b)
LW5
Reserved (16b)
Reserved (16b)
LW6
Packet_Next (32b)
LW7
60
Extra

• The next set of slides are for templates or extra
  information if needed

61
Text Slide Template
62
Image Slide Template
63
Hardware Definitions

• Blade Slot 1 Line Card
• NPUA Ingress
• NPUB Egress
• Linux Controller foghorn.arl.wustl.edu
• Windows Controller techX01.arl.wustl.edu
• Blade Slot 2 MetaRouter
• NPUA IPv4 MR 1
• NPUB IPv4 MR 2 (optional)
• Linux Controller coffee.arl.wustl.edu
• Windows Controller techX03.arl.wustl.edu
• SPI Connections
• LC Blade
• RTM04 ? NPUA04 External links to LC
  Ingress
• NPUA59 ? RTM59 LC Ingress to Internal
  Links
• RTM59 ? NPUB59 Internal links to LC
  Egress
• NPUB04 ? RTM04 LC Egress to External
  Links
• MR Blade
• NPUA04 ?? RTM04
• NPUB04 ?? RTM59

64
HW Demo Quick diagram
NPUA
6
5
MR
coffee
7
4
10
1
LC
9
2
3
8
NPUA LC Ingress
NPUB LC Egress
65
Multiple Slices, Code Options and MetaInterfaces
UDP Dport
VLAN, UDP Dport
UDP Dport
VLAN, UDP Dport
IPv4 MR
0xC100
0xC100
0x001,0xC100
0x001,0xC100
coffee 192.168.81.1 192.168.82.1 192.168.83.1 192.
168.84.1
coffee
Slice 1 (192.168.91.1) Code Option 1
0xC200
0xC200
0x001,0xC101
0x001,0xC101
0xC300
0xC300
0x001,0xC102
0x001,0xC102
0xC400
0xC400
0x001,0xC103
0x001,0xC103
0xC101
0xC101
0x002,0xC200
0x002,0xC200
foghorn 192.168.81.2 192.168.82.2 192.168.83.2 192
.168.84.2
foghorn
Slice 2 (192.168.92.1) Code Option 1
0xC201
0xC201
0x002,0xC201
0x002,0xC201
0xC301
0xC301
0x002,0xC202
0x002,0xC202
0xC401
0xC401
0x002,0xC203
0x002,0xC203
NPUA LC Ingress
NPUB LC Egress
0xC102
0x003,0xC300
0x003,0xC300
0xC102
xxx 192.168.81.3 192.168.82.3 192.168.83.3 192.168
.84.3
Slice 3 (192.168.93.1) Code Option 2
xxx
0xC202
0x003,0xC301
0x003,0xC301
0xC202
0xC302
0x003,0xC302
0x003,0xC302
0xC302
0xC402
0x003,0xC303
0x003,0xC303
0xC402
0xC103
0x004,0xC400
0x004,0xC400
0xC103
yyy 192.168.81.4 192.168.82.4 192.168.83.4 192.168
.84.4
Slice 4 (192.168.94.1) Code Option 2
yyy
0xC203
0x004,0xC401
0x004,0xC401
0xC203
0xC303
0x004,0xC402
0x004,0xC402
0xC303
0xC403
0x004,0xC403
0x004,0xC403
0xC403
66
Multi Slice IP Addresses for Test Packets
VLAN UDP Rx DPort Tunnel SA Tunnel DA IP SA IP DA Proto SPort DPort
1 0xC100 192.168.81.1 192.168.91.1 192.168.81.1 192.168.81.2 0x11 1 2
1 0xC101 192.168.81.2 192.168.91.1 192.168.81.2 192.168.81.3 0x11 1 2
1 0xC102 192.168.81.3 192.168.91.1 192.168.81.3 192.168.81.4 0x11 1 2
1 0xC103 192.168.81.4 192.168.91.1 192.168.81.4 192.168.81.1 0x11 1 2
2 0xC200 192.168.82.1 192.168.92.1 192.168.82.1 192.168.82.2 0x11 1 2
2 0xC201 192.168.82.2 192.168.92.1 192.168.82.2 192.168.82.3 0x11 1 2
2 0xC202 192.168.82.3 192.168.92.1 192.168.82.3 192.168.82.4 0x11 1 2
2 0xC203 192.168.82.4 192.168.92.1 192.168.82.4 192.168.82.1 0x11 1 2
3 0xC300 192.168.83.1 192.168.93.1 192.168.83.1 192.168.83.2 0x11 1 2
3 0xC301 192.168.83.2 192.168.93.1 192.168.83.2 192.168.83.3 0x11 1 2
3 0xC302 192.168.83.3 192.168.93.1 192.168.83.3 192.168.83.4 0x11 1 2
3 0xC303 192.168.83.4 192.168.93.1 192.168.83.4 192.168.83.1 0x11 1 2
4 0xC400 192.168.84.1 192.168.94.1 192.168.84.1 192.168.84.2 0x11 1 2
4 0xC401 192.168.84.2 192.168.94.1 192.168.84.2 192.168.84.3 0x11 1 2
4 0xC402 192.168.84.3 192.168.94.1 192.168.84.3 192.168.84.4 0x11 1 2
4 0xC403 192.168.84.4 192.168.94.1 192.168.84.4 192.168.84.1 0x11 1 2
67
November Intel Demo

• 4 MRs
• Numbered 1, 2, 3, 4
• 4 MIs per MR
• Numbered 1, 2, 3, 4
• 4 QIDs per MI
• To the User
• Numbered 1, 2, 3, 4 on each MI
• Internally to the NP filters
• np_qid 16 (MR - 1) 4 (MI - 1 )
  USER_QID
• 4 Hosts
• coffee, momcat, tabby, foghorn
• Each with a presence on each of the MRs
• Monitor
• Input BW and Pkts per MI at the LCI HF Pre-Q
  counters
• Output BW and Pkts per MI at the LCE HF Pre-Q
  counters
• Q-Length at the IPv4 MR QM
• LC Stats Index
• Np_stats_index 4 (MR - 1) MI

68
RTM Connections
SWITCH?
P10
P10
P9
P9
P8
P8
P7
P7
P6
P6
P5
P5
P4
P4
H4
P3
P3
H3
P2
P2
H2
P1
P1
H1
69
Demo Config
H1
H2
2
1
MR-2
3
4
H4
H3
H1
H2
2
1
MR-3
3
4
H4
H3
H1
H2
MR-1 MR-2 MR-3 MR-4
H1 (coffee) 192.168.81.1 192.168.82.1 192.168.83.1 192.168.84.1
H2 (momcat) 192.168.81.2 192.168.82.2 192.168.83.2 192.168.84.2
H3 (tabby) 192.168.81.3 192.168.82.3 192.168.83.3 192.168.84.3
H4 (foghorn) 192.168.81.4 192.168.82.4 192.168.83.4 192.168.84.4
2
1
MR-4
3
4
H4
H3
70
Tunnels, LCI Filters, QIDs and Stats Indices
MR M1 VLAN UDP Rx DPort Tunnel SA Tunnel DA Stats Index QID
1 1 1 0xC100 192.168.81.1 192.168.91.1 1 1
1 2 1 0xC101 192.168.81.2 192.168.91.1 2 2
1 3 1 0xC102 192.168.81.3 192.168.91.1 3 3
1 4 1 0xC103 192.168.81.4 192.168.91.1 4 4
2 1 2 0xC200 192.168.82.1 192.168.92.1 5 5
2 2 2 0xC201 192.168.82.2 192.168.92.1 6 6
2 3 2 0xC202 192.168.82.3 192.168.92.1 7 7
2 4 2 0xC203 192.168.82.4 192.168.92.1 8 8
3 1 3 0xC300 192.168.83.1 192.168.93.1 9 9
3 2 3 0xC301 192.168.83.2 192.168.93.1 10 10
3 3 3 0xC302 192.168.83.3 192.168.93.1 11 11
3 4 3 0xC303 192.168.83.4 192.168.93.1 12 12
4 1 4 0xC400 192.168.84.1 192.168.94.1 13 13
4 2 4 0xC401 192.168.84.2 192.168.94.1 14 14
4 3 4 0xC402 192.168.84.3 192.168.94.1 15 15
4 4 4 0xC403 192.168.84.4 192.168.94.1 16 16
71
Tunnels, LCE Filters, QIDs and Stats Indices
MR M1 VLAN UDP Rx DPort Tunnel SA Tunnel DA Stats Index QID
1 1 1 0xC100 192.168.91.1 192.168.81.1 1 1
1 2 1 0xC101 192.168.91.1 192.168.81.2 2 2
1 3 1 0xC102 192.168.91.1 192.168.81.3 3 3
1 4 1 0xC103 192.168.91.1 192.168.81.4 4 4
2 1 2 0xC200 192.168.92.1 192.168.82.1 5 5
2 2 2 0xC201 192.168.92.1 192.168.82.2 6 6
2 3 2 0xC202 192.168.92.1 192.168.82.3 7 7
2 4 2 0xC203 192.168.92.1 192.168.82.4 8 8
3 1 3 0xC300 192.168.93.1 192.168.83.1 9 9
3 2 3 0xC301 192.168.93.1 192.168.83.2 10 10
3 3 3 0xC302 192.168.93.1 192.168.83.3 11 11
3 4 3 0xC303 192.168.93.1 192.168.83.4 12 12
4 1 4 0xC400 192.168.94.1 192.168.84.1 13 13
4 2 4 0xC401 192.168.94.1 192.168.84.2 14 14
4 3 4 0xC402 192.168.94.1 192.168.84.3 15 15
4 4 4 0xC403 192.168.94.1 192.168.84.4 16 16
72
MR-1 Filters and QIDs
DA SA MI USER_QID NP_QID
192.168.81.1 192.168.81.2 1 2 2
192.168.81.1 192.168.81.3 1 3 3
192.168.81.1 192.168.81.4 1 4 4
192.168.81.2 192.168.81.1 2 1 5
192.168.81.2 192.168.81.3 2 3 7
192.168.81.2 192.168.81.4 2 4 8
192.168.81.3 192.168.81.1 3 1 9
192.168.81.3 192.168.81.2 3 2 10
192.168.81.3 192.168.81.4 3 4 12
192.168.81.4 192.168.81.1 4 1 13
192.168.81.4 192.168.81.2 4 2 14
192.168.81.4 192.168.81.3 4 3 15
73
MR-2 Filters and QIDs
DA SA MI USER_QID NP_QID
192.168.82.1 192.168.82.2 1 2 18
192.168.82.1 192.168.82.3 1 3 19
192.168.82.1 192.168.82.4 1 4 20
192.168.82.2 192.168.82.1 2 1 21
192.168.82.2 192.168.82.3 2 3 23
192.168.82.2 192.168.82.4 2 4 24
192.168.82.3 192.168.82.1 3 1 25
192.168.82.3 192.168.82.2 3 2 26
192.168.82.3 192.168.82.4 3 4 28
192.168.82.4 192.168.82.1 4 1 29
192.168.82.4 192.168.82.2 4 2 30
192.168.82.4 192.168.82.3 4 3 31
74
MR-3 Filters and QIDs
DA SA MI USER_QID NP_QID
192.168.83.1 192.168.83.2 1 2 34
192.168.83.1 192.168.83.3 1 3 35
192.168.83.1 192.168.83.4 1 4 36
192.168.83.2 192.168.83.1 2 1 37
192.168.83.2 192.168.83.3 2 3 38
192.168.83.2 192.168.83.4 2 4 40
192.168.83.3 192.168.83.1 3 1 41
192.168.83.3 192.168.83.2 3 2 42
192.168.83.3 192.168.83.4 3 4 44
192.168.83.4 192.168.83.1 4 1 45
192.168.83.4 192.168.83.2 4 2 46
192.168.83.4 192.168.83.3 4 3 47
75
MR-4 Filters and QIDs
DA SA MI USER_QID NP_QID
192.168.84.1 192.168.84.2 1 2 50
192.168.84.1 192.168.84.3 1 3 51
192.168.84.1 192.168.84.4 1 4 52
192.168.84.2 192.168.84.1 2 1 53
192.168.84.2 192.168.84.3 2 3 54
192.168.84.2 192.168.84.4 2 4 56
192.168.84.3 192.168.84.1 3 1 57
192.168.84.3 192.168.84.2 3 2 58
192.168.84.3 192.168.84.4 3 4 60
192.168.84.4 192.168.84.1 4 1 61
192.168.84.4 192.168.84.2 4 2 62
192.168.84.4 192.168.84.3 4 3 63
76
MR-1 Packet Flows
VLAN UDP Rx DPort Tunnel SA Tunnel DA IP SA IP DA Proto SPort DPort
1 0xC100 192.168.81.1 192.168.91.1 192.168.81.1 192.168.81.2 0x11 2 1
1 0xC100 192.168.81.1 192.168.91.1 192.168.81.1 192.168.81.3 0x11 3 1
1 0xC100 192.168.81.1 192.168.91.1 192.168.81.1 192.168.81.4 0x11 4 1
1 0xC101 192.168.81.2 192.168.91.1 192.168.81.2 192.168.81.1 0x11 1 2
1 0xC101 192.168.81.2 192.168.91.1 192.168.81.2 192.168.81.3 0x11 3 2
1 0xC101 192.168.81.2 192.168.91.1 192.168.81.2 192.168.81.4 0x11 4 2
1 0xC102 192.168.81.3 192.168.91.1 192.168.81.3 192.168.81.1 0x11 1 3
1 0xC102 192.168.81.3 192.168.91.1 192.168.81.3 192.168.81.2 0x11 2 3
1 0xC102 192.168.81.3 192.168.91.1 192.168.81.3 192.168.81.4 0x11 4 3
1 0xC103 192.168.81.4 192.168.91.1 192.168.81.4 192.168.81.1 0x11 1 4
1 0xC103 192.168.81.4 192.168.91.1 192.168.81.4 192.168.81.2 0x11 2 4
1 0xC103 192.168.81.4 192.168.91.1 192.168.81.4 192.168.81.3 0x11 3 4
77
MR-2 Packet Flows
VLAN UDP Rx DPort Tunnel SA Tunnel DA IP SA IP DA Proto SPort DPort
2 0xC200 192.168.82.1 192.168.92.1 192.168.82.1 192.168.82.2 0x11 2 1
2 0xC200 192.168.82.1 192.168.92.1 192.168.82.1 192.168.82.3 0x11 3 1
2 0xC200 192.168.82.1 192.168.92.1 192.168.82.1 192.168.82.4 0x11 4 1
2 0xC201 192.168.82.2 192.168.92.1 192.168.82.2 192.168.82.1 0x11 1 2
2 0xC201 192.168.82.2 192.168.92.1 192.168.82.2 192.168.82.3 0x11 3 2
2 0xC201 192.168.82.2 192.168.92.1 192.168.82.2 192.168.82.4 0x11 4 2
2 0xC202 192.168.82.3 192.168.92.1 192.168.82.3 192.168.82.1 0x11 1 3
2 0xC202 192.168.82.3 192.168.92.1 192.168.82.3 192.168.82.2 0x11 2 3
2 0xC202 192.168.82.3 192.168.92.1 192.168.82.3 192.168.82.4 0x11 4 3
2 0xC203 192.168.82.4 192.168.92.1 192.168.82.4 192.168.82.1 0x11 1 4
2 0xC203 192.168.82.4 192.168.92.1 192.168.82.4 192.168.82.2 0x11 2 4
2 0xC203 192.168.82.4 192.168.92.1 192.168.82.4 192.168.82.3 0x11 3 4
78
MR-3 Packet Flows
VLAN UDP Rx DPort Tunnel SA Tunnel DA IP SA IP DA Proto SPort DPort
3 0xC300 192.168.83.1 192.168.93.1 192.168.83.1 192.168.83.2 0x11 2 1
3 0xC300 192.168.83.1 192.168.93.1 192.168.83.1 192.168.83.3 0x11 3 1
3 0xC300 192.168.83.1 192.168.93.1 192.168.83.1 192.168.83.4 0x11 4 1
3 0xC301 192.168.83.2 192.168.93.1 192.168.83.2 192.168.83.1 0x11 1 2
3 0xC301 192.168.83.2 192.168.93.1 192.168.83.2 192.168.83.3 0x11 3 2
3 0xC301 192.168.83.2 192.168.93.1 192.168.83.2 192.168.83.4 0x11 4 2
3 0xC302 192.168.83.3 192.168.93.1 192.168.83.3 192.168.83.1 0x11 1 3
3 0xC302 192.168.83.3 192.168.93.1 192.168.83.3 192.168.83.2 0x11 2 3
3 0xC302 192.168.83.3 192.168.93.1 192.168.83.3 192.168.83.4 0x11 4 3
3 0xC303 192.168.83.4 192.168.93.1 192.168.83.4 192.168.83.1 0x11 1 4
3 0xC303 192.168.83.4 192.168.93.1 192.168.83.4 192.168.83.2 0x11 2 4
3 0xC303 192.168.83.4 192.168.93.1 192.168.83.4 192.168.83.3 0x11 3 4
79
MR-4 Packet Flows
VLAN UDP Rx DPort Tunnel SA Tunnel DA IP SA IP DA Proto SPort DPort
4 0xC400 192.168.84.1 192.168.94.1 192.168.84.1 192.168.84.2 0x11 2 1
4 0xC400 192.168.84.1 192.168.94.1 192.168.84.1 192.168.84.3 0x11 3 1
4 0xC400 192.168.84.1 192.168.94.1 192.168.84.1 192.168.84.4 0x11 4 1
4 0xC401 192.168.84.2 192.168.94.1 192.168.84.2 192.168.84.1 0x11 1 2
4 0xC401 192.168.84.2 192.168.94.1 192.168.84.2 192.168.84.3 0x11 3 2
4 0xC401 192.168.84.2 192.168.94.1 192.168.84.2 192.168.84.4 0x11 4 2
4 0xC402 192.168.84.3 192.168.94.1 192.168.84.3 192.168.84.1 0x11 1 3
4 0xC402 192.168.84.3 192.168.94.1 192.168.84.3 192.168.84.2 0x11 2 3
4 0xC402 192.168.84.3 192.168.94.1 192.168.84.3 192.168.84.4 0x11 4 3
4 0xC403 192.168.84.4 192.168.94.1 192.168.84.4 192.168.84.1 0x11 1 4
4 0xC403 192.168.84.4 192.168.94.1 192.168.84.4 192.168.84.2 0x11 2 4
4 0xC403 192.168.84.4 192.168.94.1 192.168.84.4 192.168.84.3 0x11 3 4
80
End of November Intel Demo Slides