Intel IXP2XXX Network Processor Architecture Overview

1
Intel IXP2XXX Network Processor Architecture
Overview
John Morgan Infrastructure Processor
Division September 2004
2
Agenda

• IXP2400 External Features
• IXP2800 External Features
• Comparison of IXP2400 and IXP2800
• IXP2XXX Resource Overviews
• MEv2 Overview
• QDR SRAM Overview
• DDR Overview
• RDRAM Overview
• PCI Overview
• MSF Overview
• Miscellaneous

3
IXP2400 External Features

• External Interfaces
• MSF Interface supports UTOPIA 1/2/3, SPI-3
  (POS-PL3), and CSIX.
• Four independent, configurable, 8-bit channels
  with the ability to aggregate channels for wider
  interfaces.
• Media interface can support channelized media on
  RX and 32-bit connect to Switch Fabric over SPI-3
  on TX (and vice versa) to support Switch Fabric
  option.
• 2 Quad Data Rate SRAM channels.
• A QDR SRAM channel can interface to
  Co-Processors.
• 1 DDR SDRAM channel.
• PCI 64/66 Host CPU interface.
• Flash and PHY Mgmt interface.
• Dedicated inter-IXP channel to communicate fabric
  flow control information from egress to ingress
  for dual chip solution.

Host CPU (Optional)
PCI 64-bit / 66 MHz
IXA SW
QDR SRAM 1.6 GBs 64 M Byte
Classification Accelerator
IXP2400 (Ingress)
CoProc Bus
DDR DRAM 2 GByte
Micro-Engine Clusters
Customer ASICs
IXP2400 (Egress)
Flash
Slow Port
Utopia 1/2/3 or POS-PL2/3 Interface
Flow Control Bus
Utopia 1,2,3 SPI 3 (POS-PL3) CSIX
ATM / POS PHY or Ethernet MAC
Switch Fabric Port Interface
4
IXP2400 Full-Duplex OC-48 System Implementation
S D R A M
5
IXP2400 Chaining
Glueless Interface between IXP2400 Devices using
CSIX-L1
Control Plane Processor
PCI 64/66
IXP2400 Processor
IXP2400 Processor
IXP2400 Processor
2.5Gbs CSIX-L1
2.5 Gbs CSIX-L1
2.5Gbs CSIX-L1
2.5Gbs SPI3
D R A M
Q DR
Q DR
D R A M
Q DR
Q DR
D R A M
Q DR
Q DR
QDR SRAM Queues Tables
QDR SRAM Queues Tables
QDR SRAM Queues Tables
DDRPacket Memory
DDRPacket Memory
DDRPacket Memory
6
72
IXP2400
MEv2 2
MEv2 1
DDRAM
Rbuf 64 _at_ 128B
S P I 3 or C S I X
32b
MEv2 3
MEv2 4
Intel XScale Core 32K IC 32K DC
G A S K E T
Tbuf 64 _at_ 128B
PCI (64b) 66 MHz
32b
64b
MEv2 6
MEv2 5
Hash 64/48/128
Scratch 16KB
MEv2 7
MEv2 8
QDR SRAM 1
QDR SRAM 2
CSRs -Fast_wr -UART -Timers -GPIO -BootROM/Slow
Port
E/D Q
E/D Q
18
18
18
18
7
IXP2400 Bandwidths

• 600 MHz Operation 4.8 GOPs
• 2.5 Gb/s Full Duplex Media Interface
• POS-PHY
• Utopia
• CSIX-L1
• 2.4 GBs DDR Memory Bandwidth at 300 MTs
• 1.6 GBs QDR Memory Bandwidth with 200 MHz QDRII
  devices

8
IXP2400 Resources Summary

• Half Duplex OC-48 / 2.5 Gb/sec Network Processor
• (8) Multi-Threaded Microengines
• Intel XScale Core
• Media / Switch Fabric Interface
• PCI interface
• 2 QDR SRAM interface controllers
• 1 DDR SDRAM interface controller
• 8 bit asynchronous port
• Flash and CPU bus
• Additional integrated feature
• Hardware Hash Unit
• 16 KByte Scratchpad Memory,Serial UART port
• 8 general purpose I/O pins
• Four 32-bit timers
• JTAG Support

9
Agenda

• IXP2400 External Features
• IXP2800 External Features
• Comparison of IXP2400 and IXP2800
• IXP2XXX Resource Overviews
• MEv2 Overview
• QDR SRAM Overview
• DDR Overview
• RDRAM Overview
• PCI Overview
• MSF Overview
• Miscellaneous

10
IXP2800 External Features
11
10Gb/s SONET Line Card
D R A M
D R A M
D R A M
RDRPacket Memory
Control Plane Processor
QDR SRAM Queues Tables
Q D R
Q D R
Q DR
Q DR
PCI 64/66
IXP2800 Ingress Processor
Fabric Interface Chip (FIC)
IXF18101
CDR, DEMUX
15Gbs
10GbE OC-192c
10Gbs
SPI I/F
Fabric
CSIX I/F
Flow Ctl
CDR, DEMUX
15Gbs
10Gbs
IXP2800 Egress Processor
10 GbE WAN / PPP/ ATM/ OTN / SONET/ SDH
RDR Packet Memory
D R A M
D R A M
D R A M
Q DR
Q DR
Q DR
QDR SRAM Queues Tables
Q DR
12
IXP2800 System with SPI gasket
D R A M
D R A M
D R A M
Control Plane Processor
RDRPacket Memory
QDR SRAM Queues Tables
Q D R
Q D R
Q DR
Q DR
PCI 64/66
IXP2800 Ingress Processor
10Gbs
10Gbs
Dual CSIX
Utopia3
SPI4 2U3
x
SPI 2U3
SPI4 2U3
x
SPI gasket
13
IXP2800 Chaining

• Glueless interface between IXP2800 devices using
  SPI-4.2

Control Plane Processor
PCI 64/66
IXP2800 Processor
IXP2800 Processor
IXP2800 Processor
10Gbs SPI-4
10Gbs SPI-4
10Gbs SPI-4
10Gbs SPI-4
D R A M
D R A M
D R A M
Q DR
Q DR
Q DR
Q DR
D R A M
D R A M
D R A M
Q DR
Q DR
Q DR
Q DR
D R A M
D R A M
D R A M
Q DR
Q DR
Q DR
Q DR
QDR SRAM Queues Tables
QDR SRAM Queues Tables
QDR SRAM Queues Tables
RDRPacket Memory
RDRPacket Memory
RDRPacket Memory
14
18
18
18
IXP2800
Stripe
RDRAM 1
RDRAM 3
RDRAM 2
MEv2 2
MEv2 3
MEv2 4
MEv2 1
Rbuf 64 _at_ 128B
S P I 4 or C S I X
16b
MEv2 7
MEv2 6
MEv2 5
MEv2 8
Intel XScale Core 32K IC 32K DC
G A S K E T
PCI (64b) 66 MHz
Tbuf 64 _at_ 128B
64b
16b
MEv2 10
MEv2 11
MEv2 12
MEv2 9
Hash 48/64/128
Scratch 16KB
MEv2 15
MEv2 14
MEv2 13
QDR SRAM 2
QDR SRAM 1
QDR SRAM 3
MEv2 16
QDR SRAM 4
CSRs -Fast_wr -UART -Timers -GPIO -BootROM/SlowPo
rt
E/D Q
E/D Q
E/D Q
E/D Q
18
18
18
18
18
18
18
18
15
IXP2800 Bandwidths

• 1.4 GHz Operation 20 GOPs
• 10Gbs Full Duplex Media Interface
• SPI-4.2
• CSIX-L1
• 1.9 GB/s QDR SRAM Memory Bandwidth/Channel
• 2.1 GB/s RDRAM Memory Bandwidth/Channel

16
IXP2800 Resources Summary

• Half Duplex OC-192 / 10 Gb/sec Network Processor
• (16) Multi-Threaded Microengines
• Intel XScale Core
• Media / Switch Fabric Interface
• PCI interface
• 4 QDR SRAM Interface Controllers
• 3 Rambus DRAM Interface Controllers
• 8 bit asynchronous port
• Flash and CPU bus
• Additional integrated features
• Hardware Hash Unit for generating of 48-, 64-, or
  128-bit adaptive polynomial hash keys
• 16 KByte Scratchpad Memory
• Serial UART port for debug
• 8 general purpose I/O pins
• Four 32-bit timers
• JTAG Support

17
Agenda

• IXP2400 External Features
• IXP2800 External Features
• Comparison of IXP2400 and IXP2800
• IXP2XXX Resource Overviews
• MEv2 Overview
• QDR SRAM Overview
• DDR Overview
• RDRAM Overview
• PCI Overview
• MSF Overview
• Miscellaneous

18
IXP2800 and IXP2400 Comparison
IXP2400
IXP2800
600/400MHz
1.4/1.0 GHz/ 650 MHz
Frequency
1 channel DDR DRAM - 150MHz Up to 2GB
3 channels RDRAM 800/1066MHz Up to 2GB
DRAM Memory
2 channels QDR (or co-processor)
4 channels QDR (or co-processor)
SRAM Memory
Separate 32 bit Tx Rx configurable to SPI-3,
UTOPIA 3 or CSIX_L1
Separate 16 bit Tx Rx configurable to SPI-4 P2
or CSIX_L1
Media Interface
8 (MEv2)
16 (MEv2)
Number of MicroEngines
Dual chip full duplex OC48
Dual chip full duplex OC192
Performance
19
Agenda

• IXP2400 External Features
• IXP2800 External Features
• Comparison of IXP2400 and IXP2800
• IXP2XXX Resource Overviews
• MEv2 Overview
• QDR SRAM Overview
• DDR Overview
• RDRAM Overview
• PCI Overview
• MSF Overview
• Miscellaneous

20
MicroEngine v2
D-Push Bus
S-Push Bus
From Next Neighbor
Control Store 4K/8K Instructions
Local Memory 640 words
128 GPR
128 GPR
128 Next Neighbor
128 S Xfer In
128 D Xfer In
LM Addr 1
2 per CTX
B_op
A_op
LM Addr 0
Prev B
Prev A
P-Random
B_Operand
A_Operand
CRC Unit
Multiply
Lock 0-15
Status and LRU Logic (6-bit)
TAGs 0-15
32-bit ExecutionData Path
Find first bit
CAM
CRC remain
Add, shift, logical
Status
Entry
OtherLocal CSRs
ALU_Out
To Next Neighbor
Timers
128 S Xfer Out
128 D Xfer Out
Timestamp
D-Pull Bus
S-Pull Bus
21
Microengine v2 Features Part 1

• Clock Rates
• IXP2400 600/400 MHz
• IXP2800 - 1.4/1.0 GHz/ 650 MHz
• Control Store
• IXP2400 4K Instruction store
• IXP2800 8K Instruction store
• Configurable to 4 or 8 threads
• Each thread has its own program counter,
  registers, signal and wakeup events
• Generalized Thread Signaling (15 signals per
  thread)
• Local Storage Options
• 256 GPRs
• 256 Transfer Registers
• 128 Next Neighbor Registers
• 640 - 32bit words of local memory

22
Microengine v2 Features Part 2

• CAM (Content Addressable Memory)
• Performs parallel lookup on 16 - 32bit entries
• Reports a 9-bit lookup result
• 4 State bits (software controlled, no impact to
  hardware)
• Hit entry number that hit Miss LRU entry
• 4-bit index of Cam entry (Hit) or LRU (Miss)
• Improves usage of multiple threads on same data
• CRC hardware
• IXP2400 - Provides CRC_16, CRC_32
• IXP2800 - Provides CRC_16, CRC_32, iSCSI, CRC_10
  and CRC_5
• Accelerates CRC computation for ATM AAL/SAR, ATM
  OAM and Storage applications
• Multiply hardware
• Supports 8x24, 16x16 and 32x32
• Accelerates metering in QoS algorithms
• DiffServ, MPLS
• Pseudo Random Number generation
• Accelerates RED, WRED algorithms
• 64-bit Time-stamp and 16-bit Profile count

23
Intel XScale Core Overview

• High-performance, Low-power, 32-bit Embedded RISC
  processor
• Clock rate
• IXP2400 600 MHz
• IXP2800 700/500/325 MHz
• 32 Kbyte instruction cache
• 32 Kbyte data cache
• 2 Kbyte mini-data cache
• Write buffer
• Memory management unit

24
QDR SRAM Overview

• Controller Configuration
• IXP2400 - 2 channels
• IXP2800 - 4 channels
• Optional parity (support for x16 or x18 parts)
• Address up to 64 Mbytes of SRAM per channel
• Pin design supports up to 4 SRAM loads
• Supports Burst of 2 QDR Devices
• Supports byte parity bits 8, 17 for byte 0/1
• Parity can be enabled/disabled per channel in
  SRAM_control CSR

25
QDR SRAM Overview

• Peak bandwidth of 1.6 GBytes/sec per channel
• Using 200 MHz SRAMs
• Specialized SRAM operations
• Atomic swap, bit set, bit clear, add, subtract
• Hardware support for ring, queue and journal
  operations
• 64 Q_Array registers per channel
• Interface to QDR compatible TCAMs and
  CoProcessors
• Network Processor Forum LA-1 Co-Processor
  Standard Compliant
• Clamshell topology enables both Memory and
  Co-processor to share the same channel

26
IXP2400 DDR DRAM Overview

• 1 64-bit (72-bit with ECC) SDRAM channel
• DRAM sizes of 64Mb, 128Mb, 256Mb, 512Mb, or 1Gb
• Max capacity is 2GB (using 1Gb parts)
• Support x8 or x16 devices, DIMM or direct
  soldered
• Support devices with 4 banks
• Support 1 or 2 sided DIMM
• Optional ECC
• 200/300 MTS, 100MHz/150MHz respectively
• Hardware Interleaving spreads contiguous
  addresses across multiple banks

27
IXP2800 RDRAM Overview

• 3 Independent Rambus DRAM Channels which operate
  concurrently
• 1.6 GBytes/s (12.8Gbps) per channel at 800 MHz
• Maximum total of 2 GBytes
• 768 MBytes each if 3 channels are populated
• 1 GBytes each if only 2 channels are populated
• 2 GBytes if only 1 channel is populated
• Supports 64Mb, 128Mb, 256Mb, 512Mb and 1 Gb
  devices
• Supports RDRAMS with 1x16, 2x16 dependent and 4
  independent Banks
• Optional ECC and Parity Support
• Interleaving implemented in HW provides balanced
  access across all channels
• Interleave size is 128 bytes

28
PCI Interface Overview

• PCI 2.2 compliant
• PCI Bus Target
• SRAM
• DRAM
• Control and Status Registers
• PCI Bus Master to other devices
• DMA channels
• IXP2400 3 Channels
• IXP2800 2 Channels
• Doorbell and Mailbox Registers
• Loads
• 4 loads at 66MHz
• 8 loads at 33MHz

29
IXP2400 Media Switch Fabric Interface

• Protocols
• POS-PHY Levels 2 and 3
• Utopia Levels 1, 2 and 3
• CSIX-L1 for Switch Fabric Interface
• LVTTL IO (3.3V)
• 32-bit receive, 32-bit transmit
• 25133 MHz
• 8KB receive buffer and 8KB transmit buffer

30
IXP2800 Media Switch Fabric Interface

• Protocols
• SPI-4 Phase 2 for Network Device
• CSIX-L1 for Switch Fabric Interface
• LVDS IO (IEEE 1596.3, ANSI/TIA/EIA-644)
• 16-bit receive, 16-bit transmit
• 311500 MHz
• 8KB receive buffer and 8KB transmit buffer

31
Miscellaneous

• UART
• Standard RS232 primarily for debugging
• TIMER
• 4 - 32 bit timers
• Timer 4 can be used as Watchdog Timer
• GPIO
• 8 General Purpose IO pins
• Can be used as interrupt source to XScale core or
  clock to timers
• Interrupt Controller
• Provides the ability to enable or mask interrupts
  from a number of chip wide sources like timers,
  PCI devices, DRAM ECC errors, etc.
• Slow Port
• Used for Flash ROM access and 8, 16, or 32-bit
  asynchronous device access
• Allows XScale do read/ write data transfers to
  these slave devices

32
Backup
33
IXP2400 Target Application
IXP2400 will provide IXP 1200 customers a
performance upgrade for OC-12 applications and
enable multi Gigabit Ethernet platforms up to
OC-48

• WAN Edge/Access Aggregation
• Includes IP Service Switches, Multiservice
  Switches, DSLAM, Cable Head End
• Wireless Infrastructure
• Layer 4-7 Switches
• Includes Firewall, Server Offload, Content-Based
  Load Balancing

34
IXP2800 Target Application

• Metropolitan Area Network (MAN) switches and
  routers
• Internet core access switches and routers
• Multi-service switches
• 10 Gbs enterprise switches and routers supporting
  tomorrows data centers,
• Storage area networks (SAN)
• Content aware server off-load/web switches.
• Security/VPN solutions
• Wireless base stations
• Digital Subscriber Line Access Multiplexers
  (DSLAMs).

35
Edge Multi-Service Switch - WAN/LAN Solutions
IXP2400 IXP
Oahu Quad Gig Phy
Utopia3
Phy Interface
OC-48c ATM SAR TrafficManager
Amazon IXF6048 OC-48c ATM POS Framer
Vallejo 4 x 1G Ethernet MAC
1 Gig LAN or Server Farm
SPI-3 (Utopia3 Packet)
SPI-3 (Utopia3 Packet)
WAN Backbone (ATM, SONET)
Optical Ring
IXP2400 IXP
1 GigLAN Backbone or Server Farm
OR
CSIX Switch Fabric
80 Gig 1 TerabitSwitch Fabric
36
Edge Server Offload
Host CPU (IOP or iA)
PCI Bus
IXP2400 IXP
Oahu Quad Gig Phy
Phy Interface
Vallejo 4 x 1G Ethernet MAC
CSIX Switch Fabric
Server Farm
CSIX Level 1
SPI-3 (Utopia3 Packet)
80 Gig 1 TerabitSwitch Fabric
IXP2400 IXP
1 GigLAN Backbone or Server Farm
37
IXP2400 Media Configurations
D D R
D D R
D D R
Rx Tx paths each have 2 separate clock domains
for asynchronous traffic
T C A M
T C A M
T C A M
Q D R
Q D R
Q D R
Q D R
Q D R
Q D R
IXP2400
IXP2400
IXP2400
Xscale
Xscale
Xscale
Rx
Tx
Rx
Tx
Rx
Tx
32bit
32bit
32bit
16bit
8bit
32bit
D D R
T C A M
Q D R
Q D R
Utopia 1/2/3 Or SPI-3 (POS-PHY 2/3) Or CSIX_L1B
Utopia 1/2/3 Or SPI-3 (POS-PHY 2/3)
IXP2400
Xscale
Rx
Tx
Each Rx Tx path may be configured to be single
32bit, quad 8bit, dual 16bit or combination of 8
16bit wide buses
8bit
16bit
16bit
38
10Port 1Gb/s Ethernet Line Card
D R A M
D R A M
D R A M
RDRPacket Memory
Control Plane Processor
QDR SRAM Queues Tables
Q D R
Q D R
Q DR
Q DR
PCI 64/66
IXP2800 Ingress Processor
Fabric Interface Chip (FIC)
Ben Nevis
15Gbs
10x1GbE
10Gbs
SPI I/F
Fabric
CSIX I/F
Flow Ctl
15Gbs
10Gbs
IXP2800 Egress Processor
10 x 1 GbE LAN
RDR Packet Memory
D R A M
D R A M
D R A M
Q DR
Q DR
Q DR
QDR SRAM Queues Tables
Q DR
39
10Gb/s to Infiniband
D R A M
D R A M
D R A M
RDRPacket Memory
Control Plane Processor
QDR SRAM Queues Tables
Q D R
Q D R
Q DR
Q DR
PCI 64/66
IXP2800 Ingress Processor
CalypsoBen NevisLoch Lomond
InfinibandFabric
2.5Gbps
10GbE 10x1Gb OC-192c
15Gbs
10Gbs
SPI I/F
2.5Gbps
Fabric
CSIX I/F
Flow Ctl
2.5Gbps
15Gbs
10Gbs
2.5Gbps
IXP2800 Egress Processor
D R A M
D R A M
D R A M
Q DR
Q DR
Q DR
RDR Packet Memory
QDR SRAM Queues Tables
Q DR
40
10Gbs Ethernet to SONET
D R A M
D R A M
D R A M
RDRPacket Memory
Control Plane Processor
QDR SRAM Queues Tables
Q D R
Q D R
Q DR
Q DR
PCI 64/66
IXP2800 Ingress Processor
Loch LomondBen Nevis
Calypso
Serveror DiskFarms
Metro Or WAN
10GbE 10x1Gb
10Gbs
10Gbs
SPI I/F
OC-1924xOC48
SPI I/F
Flow Ctl
10Gbs
10Gbs
IXP2800 Egress Processor
QDR SRAM Queues Tables
D R A M
D R A M
D R A M
Q DR
Q DR
Q DR
Q DR
RDR Packet Memory
41
Media / Fabric Receive Logic
ThreadpushesID ontoFreelist
AutoPushStatusto Thread
Threadmovesdata
5
7
6
Rbuf 64/128 Elements 128/64B each
Status Word Per element
Rbuf Freelist
Thread Freelist
Bit vector
1
Receive State Machine
Get Free element
CreateStatus
4
3
Assignthread
Media Switch Fabric Unit
Idlebucket
2
Data Arrives
SPI-4.2 Frame
Discarded if idle packet
Pkt ctrl
Pkt payload a
Pkt ctrl
Cell payload
Pkt ctrl
Pkt payload b
Media Device
buffer
Port A
Port B
ATM Cell
packet