Design of a Diversified Router: TCAM Usage

1
Design of aDiversified Router TCAM Usage
John DeHartjdd_at_arl.wustl.edu http//www.arl.wust
l.edu/arl
2
TCAM Issues

• Number of Databases to use
• 16 available
• Key size
• Can be different for each Database.
• Result Type
• Absolute Index
• Database Relative Index
• Memory Pointer
• TCAM Associated Data of width 32, 64 or 128 bits
• Memory Constraints
• TCAM Associated Data
• 512K x 36 bit ZBT SRAM
• Supports 128K 128-bit Results
• If used for Rx and Tx then 64K in each direction
• IXP QDR SRAM Channel
• 2 x 2Mx18 (effective 4M x 18b)
• 4 times as much as the TCAM ZBT SRAM.
• Supports 512K 128-bit Results
• If used for Rx and Tx then 256K in each direction

3
TCAM Issues

• Types of Lookups supported
• Lookup (Direct)
• Multi-Hit Lookup (Direct)
• Simultaneous Multi-Database Lookup (Direct)
• Multi-Database Lookup (Indirect)
• Simultaneous Multi-Database Lookup (Indirect)
• Re-Issue Multi-Database Lookup (Indirect)

4
TCAM Issues

• Performance Impacts
• IXP/TCAM Interface
• 16 bits wide
• 200 MHz QDR
• Effective 32bits per clock tick
• Example 128b results from ZBT SRAM via TCAM 4
  ticks ? max of 50M results/sec
• Key Size (200MHz, 16 bit Interface for commands)
• Rates are max PER LA-1 Interface, up to CAM Core
  max rate
• 32b (1W) 50M Lookups/sec (CAM Core
  constraint)
• 36b (2W) 50M Lookups/sec (CAM Core
  constraint)
• 64b (2W)100M Lookups/sec (Interface
  constraint)
• 72b (3W) 67M Lookups/sec (Interface
  constraint)
• 128b (4W) 50M Lookups/sec (Interface
  constraint)

5
TCAM Issues

• Performance Impacts (continued)
• Result Type
• Index or Pointer Types
• Key Size Max CAM Core lookup rate
• 36b 50M/sec
• 72b 100M/sec
• 144b 100M/sec
• 288b 50M/sec
• 576b 25M/sec
• Associated Data CAM Core rates
• These rates are Total across both LA-1 Interfaces
• Key Size (32b Result Rate, 64b Result Rate, 128b
  Result Rate)
• 36b 50, 50, 25
• 72b 100, 50, 25
• 144b 100, 50, 25

6
TCAM Performace Tables
7
TCAM Performace Graphs
8
TCAM Performace Graphs
9
LC Notes on TCAM Lookups

• The following slides show a way to use the TCAM
  for the lookups.
• Slight adjustments might be desirable depending
  on
• Ease of doing operations on non-byte length bit
  fields
• What we learn about methods for using the TCAM.
• Field and Identifier sizes
• MN id 32 bits
• MI id 16 bits (64K Meta Interfaces per Meta
  Router)
• MLI 16 bits (64K Meta Links per Substrate
  Link)
• Port 8 bits (256 Physical Interfaces per
  Line Card)
• QID 17 bits (128K Queues per Queue Manager)
• QM ID 3 bits (8 Queue Managers per LC or PE.)
• We probably can only support 4 QMs (2 bits)
• (64 Q-Array Entries) / (16 CAM entries) ? 4 QMs
  per SRAM Controller.

10
LC Notes on TCAM Lookups

• Lookup Key size options
• 32/36, 64/72, 128/144, 256/288, 512/576 (all in
  bits)
• Lookup Result options
• Absolute Index relative to beginning of TCAM
  array
• Database Relative Index relative to beginning of
  selected database
• Memory Pointer Index points to SRAM location of
  result data
• Associated Data 32, 64 or 128 bits of data
  associated with the lookup result.
• Associated Data is stored in ZBT SRAM attached to
  TCAM.
• TCAM Databases
• How many to use?
• 1 for TX and 1 for RX?
• 1 for TX and 1 for each of the SL Types on Rx (5
  types)?
• Other

11
LC TCAM Lookup Keys on RX

• Each SL Type will use a different Database on RX
• Thus we do not need an SL Type field in the Key
• SL Type ID will be used to identify which
  database to use
• P2P-DC(24b)
• SL Type ID 0x0
• Key Fields Port(8)/MLI(16)
• P2P-Tunnel(IPv4)(72b)
• SL Type ID 0x1
• Key Fields Port(8)/EtherType(16)/IPSrc(32)/MLI(16
  )
• P2P-VLAN0(72b)
• SL Type ID 0x2
• Key Fields Port(8)/EthSAddr(48)/MLI(16)
• MA(24b)
• SL Type ID 0x3
• Key Fields Port(8)/MLI(16)
• Legacy(24b)
• SL Type ID 0x4
• Key Fields Port(8)/EtherType(16)
• Key Fields

12
LC TCAM Lookup Keys on RX
Port(8b)
MLI(16b)
P2P-DC
24 bits
IPv4 Tunnel
72 bits
Port (8b)
MLI (16b)
IP SAddr (32b)
EtherType (16b) 0x0800
MA
Port (8b)
MLI (16b)
Ethernet SAddr (48b)
72 bits
P2P-VLAN0
Port(8b)
MLI(16b)
24 bits
Legacy
24 bits
Port (8b)
EtherType (16b) 0x0800

• Blue Shading Determine SL Type
• Black Outline Key Fields from pkt

DstAddr (6B)
DstAddr (6B)
SrcAddr (6B)
SrcAddr (6B)
Type802.1Q (2B)
Type802.1Q (2B)
TCI ? VLAN0 (2B)
TCI ? VLAN0 (2B)
TypeIP (2B)
TypeIP (2B)
Ver/HLen/Tos/Len (4B)
Ver/HLen/Tos/Len (4B)
ID/Flags/FragOff (4B)
ID/Flags/FragOff (4B)
TTL (1B)
TTL (1B)
ProtocolSubstrate (1B)
Protocol (1B)
Hdr Cksum (2B)
Hdr Cksum (2B)
Src Addr (4B)
Src Addr (4B)
Dst Addr (4B)
Dst Addr (4B)
MLI (2B)
IP Payload
LEN (2B)
Meta Frame
PAD (nB)
PAD (nB)
CRC (4B)
CRC (4B)
P2P-DC Configured
P2P-Tunnel
Legacy
13
LC Packet RX All SL Types
P2P-VLAN0
Multi-Access
P2P-DC
P2P-Tunnel
14
LC TCAM Lookup Results on RX

• Fields (68b/128b)
• VLAN (16b)
• We could probably drop this to 12b since our
  switch is supposed to only support 4K VLANs but
  we might want to leave this open to switches
  supporting larger numbers of VLANs.
• MI (16b)
• Blade Eth Hdr (8b)
• Only needs to have the DAddr.
• We can control the MAC Addresses of the Blades,
  so lets say that 40 of the 48 bits are fixed and
  8 bits are assigned and stored in the Lookup
  Result. Even 8 bits is overkill but it leaves us
  some flexibility.
• SAddr can be configured and constant per LC
• First EtherType can be constant 802.1Q
• Second EtherType can be constant Substrate
• QID (20b)
• MnFlags(8b)
• NhAddr(60b) If needed

15
LC TCAM Lookup Results on RX

• Can we say there will be no Pass Through Meta
  Links where one side will be on a Multi Access
  and hence might need a NhAddr field?
• If so then we can drop the MnFlags and NhAddr
  fields from result
• Result size then becomes 60b
• Pass Through Meta Link Fields
• MnFlags(8b)
• NhAddr(nB)
• We have 60 bits left over that could be used for
  NhAddr
• If we need more
• Do a second lookup with the following fields to
  retrieve the Next Hop bits from another Database
  for NH Bits?
• Port (8b)
• VLAN (16b)
• MI (16b)

16
LC TCAM Lookups on TX

• Key
• VLAN(16b)
• TxMI(16b)
• Result
• The Lookup Result for TX will consist of several
  parts
• Lookup Result
• Constant fields
• Calculated fields
• Fields that can be stored in Local Memory
• Some of these are common across all SL Types
• Other fields are specific to each SL Type
• Common across all SL Types (96b)
• From Result (48b)
• SL Type(4b)
• Port(8b)
• MLI(16b)
• QID (20b)
• Local Memory (48b)
• Eth Hdr SA (48b) tied to Port

17
LC TCAM Lookups on TX

• Key
• VLAN(16b)
• TxMI(16b)
• Result
• Common across all SL Types (96b)
• From Result (48b)
• SL Type(4b)
• Port(8b)
• MLI(16b)
• QID (20b)
• Local Memory (48b)
• Eth Hdr SA (48b) tied to Port
• SL Type Specific Headers
• P2P-DC Hdr (64b)
• Constant (16b)
• EtherType (16b) Substrate
• Calculated (0b)
• From Result (48b)
• Eth DA (48b)

18
LC TCAM Lookups on TX

• Key
• VLAN(16b)
• TxMI(16b)
• Result
• Common across all SL Types (96b)
• From Result (48b)
• SL Type(4b)
• Port(8b)
• MLI(16b)
• QID (20b)
• Local Memory (48b)
• Eth Hdr SA (48b) tied to Port
• SL Type Specific Headers
• P2P-MA Hdr (64b)
• Constant (16b)
• EtherType (16b) Substrate
• Calculated (0b)
• From Result (48b)
• Eth DA (48b)

19
LC TCAM Lookups on TX

• Key
• VLAN(16b)
• TxMI(16b)
• Result
• Common across all SL Types (96b)
• From Result (48b)
• SL Type(4b)
• Port(8b)
• MLI(16b)
• QID (20b)
• Local Memory (48b)
• Eth Hdr SA (48b) tied to Port
• SL Type Specific Headers
• P2P-VLAN0 Hdr (80b)
• Constant (16b)
• EtherType1 (16b) 802.1Q
• EtherType2 (16b) Substrate
• Calculated (0b)
• From Result (64b)

20
LC TCAM Lookups on TX

• Result (continued)
• Common across all SL Types (96b)
• From Result (48b)
• SL Type(4b)
• Port(8b)
• MLI(16b)
• QID (20b)
• Local Memory (48b)
• Eth Hdr SA (48b) tied to Port
• SL Type Specific Headers
• P2P-Tunnel Hdr for IPv4 Tunnel (224b)
• Constant (64b)
• Eth Hdr EtherType (16b) 0x0800
• IPHdr Version(4b)/HLen(4b)/Tos(8b) (16b) All can
  be constant?
• IP Hdr Flags(3b)/FragOff(13b) (16b) what is our
  stance on Fragments? If never used, these are
  constants, if it is possible we will have to use
  them, then this has to be calculated. Either way,
  shouldnt be in Result
• IP Hdr TTL (8b) Constant
• IP Hdr Proto (8b) Substrate
• Calculated (48b)
• IP Hdr Len(16b) needs to be calculated for each
  packet sent, so shouldnt be in Result.

21
LC TCAM Lookups on TX

• Key
• VLAN(16b)
• TxMI(16b)
• Result
• Common across all SL Types (96b)
• From Result (48b)
• SL Type(4b)
• Port(8b)
• MLI(16b)
• Ignored for Legacy Traffic
• QID (20b)
• Local Memory (48b)
• Eth Hdr SA (48b) tied to Port
• SL Type Specific Headers
• Legacy (IPv4) with VLAN Hdr (96b)
• Constant (16b)
• EtherType1 (16b) 802.1Q
• Calculated (0b)
• ARP Lookup on NhAddr (48b)

22
LC TCAM Lookups on TX

• Key
• VLAN(16b)
• TxMI(16b)
• Result
• Common across all SL Types (96b)
• From Result (48b)
• SL Type(4b)
• Port(8b)
• MLI(16b)
• Ignored for Legacy Traffic
• QID (20b)
• Local Memory (48b)
• Eth Hdr SA (48b) tied to Port
• SL Type Specific Headers
• Legacy (IPv4) without VLAN Hdr (64b)
• Constant (0b)
• Calculated (0b)
• ARP Lookup on NhAddr (Is ARP cache another
  database in TCAM?) (48b)
• Eth DA (48b)

23
SUMMARY LC TCAM Lookups

• Rx Key 72 bits
• Rx Result Size 68/128 bits
• Depends on whether we need to support MA on other
  end of pass through MetaLink and on what size the
  NhAddr needs to be.
• If we can limit NhAddr to 60 bits, then it fits
  in 128 bits.
• Otherwise we will have to do something else
• Tx Key Size 32 bits
• Tx Result Size 128 bits
• We need to watch out for the Tx Result for
  Tunnels. If we introduce anything else we want in
  there then we go beyond the 128 bits supportable
  through the TCAMs Associated memory.

24
Databases on LC

• Rx
• 0000 DC
• 0001 IPv4 Tunnel
• 0010 VLAN0
• 0011 MA
• 0100 Legacy (non-substrate) with or without VLAN
• TX
• 1000 all TX lookups use one database

25
Extra

• The next set of slides are for templates or extra
  information if needed

26
Text Slide Template
27
Image Slide Template
28
OLD

• The rest of these are old slides that should be
  deleted at some point.

29
LC TCAM Lookup Keys on RX

• P2P-DC(28b) SL_Type(4)/Port(8)/MLI(16)
• P2P-Tunnel(IPv4)(76b) SL_Type(4)/Port(8)/EtherTyp
  e(16)/IPSrc(32)/MLI(16)
• P2P-VLAN0(76b) SL_Type(4)/Port(8)/EthSAddr(48)/ML
  I(16)
• MA(28b) SL_Type(4)/Port(8)/MLI(16)
• Legacy(28b) SL_Type(4)/Port(8)/EType(16)
• Fields
• SL_Type (4b) Substrate Link Type
• 0000 DC
• 0001 IPv4 Tunnel
• 0010 VLAN0
• 0011 MA
• 0100 Legacy (non-substrate) without VLAN
• 0101 Legacy (non-substrate) with VLAN
• Port(8b) Physical Interface number
• MLI(16b) Meta Link Identifier
• Ethertype(16b) Ethernet Header Type field
• IPSrc(32b) IP Source Address
• EthSAddr(48b) Ethernet Header Source Address

30
Databases on LC

• Rx
• 0000 DC
• 0001 IPv4 Tunnel
• 0010 VLAN0
• 0011 MA
• 0100 Legacy (non-substrate) without VLAN
• 0101 Legacy (non-substrate) with VLAN
• TX
• 1000 all TX lookups use one database
• Do we really need to use the TCAM for TX Lookups?
• VLAN(16b)/TxMI(16b) is the lookup key.
• Really only 4K VLANs support 12 bits
• How many VLANs (i.e. MRs) should be supported per
  LC?
• How many MIs should be supported per MR?
• 8MB per SRAM/(16B per Lookup) ? 512K Entries

31
LC TCAM Lookup Results on RX

• Standard Fields (64b/128b)
• Type (4b) Probably dont need this since there
  is a size in MnFlags.
• 0000 Default, not Pass Through, ignore MnFlags
• 1000 Pass Through with no extra lookup needed
  for NhAddr(nB), MnFlags(8b) should be 0x00
• 1001 Pass Through with extra lookup needed for
  NhAddr(nB)
• VLAN (16b)
• MI (16b)
• Blade Eth Hdr (8b)
• Only needs to have the DAddr.
• We can control the MAC Addresses of the Blades,
  so lets say that 40 of the 48 bits are fixed and
  8 bits are assigned and stored in the Lookup
  Result. Even 8 bits is overkill but it leaves us
  some flexibility.
• SAddr can be configured and constant per LC
• First EtherType can be constant 802.1Q
• Second EtherType can be constant Substrate
• QID (20b)
• MnFlags(8b) If needed
• NhAddr(56b) If needed

32
SUMMARY LC TCAM Lookups

• Rx Key Size 128 bits
• If We dropped the SL Type field and just made
  each SL Type a separate Database then we could
  get the RX Key down to 72 bits
• Rx Result Size 64/128 bits
• Depends on whether we need to support MA on other
  end of pass through MetaLink and on what size the
  NhAddr needs to be.
• Tx Key Size 32 bits
• Tx Result Size 128 bits
• We need to watch out for the Tx Result for
  Tunnels. If we introduce anything else we want in
  there then we go beyond the 128 bits supportable
  through the TCAMs Associated memory.
• If we find we cant use the 128 bit Result size
  for Tx, we might as well include the Local Memory
  result in the TCAM lookup.
• The Local Memory result was going to be the data
  that was only dependent on the Physical Interface
  id and was going to go into Local memory to try
  to keep the TCAM result below 128 bits.

33
LC TCAM Lookup Keys on RX

• Blue Shading Determine SL Type
• Black Outline Key Fields from pkt

DstAddr (6B)
DstAddr (6B)
SrcAddr (6B)
SrcAddr (6B)
Type802.1Q (2B)
Type802.1Q (2B)
TCI ? VLAN0 (2B)
TCI ? VLAN0 (2B)
DstAddr (6B)
DstAddr (6B)
TypeIP (2B)
TypeIP (2B)
Ver/HLen/Tos/Len (4B)
Ver/HLen/Tos/Len (4B)
SrcAddr (6B)
SrcAddr (6B)
ID/Flags/FragOff (4B)
ID/Flags/FragOff (4B)
TTL (1B)
Type802.1Q (2B)
TTL (1B)
Type802.1Q (2B)
ProtocolSubstrate (1B)
TCI?VLAN0 (2B)
Protocol (1B)
TCIVLAN0 (2B)
Hdr Cksum (2B)
Hdr Cksum (2B)
TypeSubstrate (2B)
TypeSubstrate (2B)
Src Addr (4B)
MLI (2B)
Src Addr (4B)
MLI (2B)
LEN (2B)
Dst Addr (4B)
LEN (2B)
Dst Addr (4B)
Meta Frame
Meta Frame
MLI (2B)
IP Payload
LEN (2B)
Meta Frame
PAD (nB)
PAD (nB)
PAD (nB)
PAD (nB)
CRC (4B)
CRC (4B)
CRC (4B)
CRC (4B)
P2P-DC Configured
P2P-VLAN0
Multi-Access
P2P-Tunnel
Legacy