Highspeed backplane interconnect

1
High-speed backplane interconnect

• Vladimir Stojanovic
• (with slides from J. Zerbe, P. Desai, R.
  Kollipara)

2
Outline

• Inside the router
• Backplane channel problem
• What can backplane designer do about it
• What can IC designer do about it
• Scaling the system to 10-100Tb/s

3
Inside the Router

• Past
• OC-12
• 622 MHz LVDS parallel
• GigE
• 1.25 Gbps serial

• Present
• OC48
• 2.5 Gbps serial
• 10GigE
• XAUI (3.125 Gbps) serial

4
Serial Links in Networking Systems
Line Card
Switch Fabric IF
MAC/ Framer
NPU/ TM
Optics
CSIX Proprietary
XAUI 4, 3.125 Gbps Serial Links
SPI4.2
Backplane 8 to 16 of 1-3.2Gbps Serial Links
Switch Card
32 to 64 Backplane Serial Links (1-3.2 Gbps)
Switch Crossbar IC
5
Backplane interconnect path

• There are many components on the signal path,
  potential source of problems

6
RaSer X Link Features
7
I/O Driver Scheme (Example)

• Impedance-controlled (CML) I/Os
• Integrated terminations
• Adjustable output-voltage/common mode

8
System Issues

• Goal Increase Router Throughput
• Limitations
• Backplane channel
• Power
• Mechanical/Physical density constraints
• Backplane and linecard routing density
• Connector pin density
• Package I/O density

9
Outline

• Inside the router
• Backplane channel problem
• What can backplane designer do about it
• What can IC designer do about it
• Scaling the system to 10-100Tb/s

10
Backplane Component Effects
11
Deterministic Noise

• Inter-symbol interference
• Dispersion (skin-effect, dielectric loss) -
  short latency
• Reflections (impedance mismatches connectors,
  via stubs, device parasitics, package) long
  latency

12
XTALK and reflections

• Primary reflection sources are at the
  connector/backplane transition
• Grouped in time as a function of backplane
  length

13
Backplane channel variations

• Variability in trace length, routing layer and
  via stub
• Significantly different transfer functions even
  within the same backplane

14
Test Backplane Example
FR4 Cross Section

• Trace lengths 1.5, 9, 14, 20 and 32
• Effective number of signal layers 13
• Effective number of total layers 28

15
Backdrilling - A Solution to the Stub Effect
16
Stub Effect Eye Pattern Analysis (2.5 Gbits/sec
FR-4)
MAX STUB
MIN STUB
17
Stub Effect Eye Pattern Analysis (5.0 Gbits/sec
FR-4)
MAX STUB
MIN STUB
18
Stub Effect Eye Pattern Analysis (12.0 Gbits/sec
FR-4)
MAX STUB
MIN STUB
19
Connector design
GBX Teradyne
20
Connector Density
21
Reducing Crosstalk within the Connector
D/C Shield

• Cross talk is reduced in the mating interface by
  surrounding each pair with a ground shield

B/P Shield
Mated pair
22
Backplane Connector Considerations

• Many connector types
• Teradyne VHDM, HSD, GbX,
• Tyco HS3, HMZd,
• FCI Metral 2000, 3000, 4000,
• 3M/Harting HSHM,
• ERNI ERmetZd, ErmetXT,
• Issues
• Loss, impedance profile, crosstalk, skew
• Foot print routability, pin density, via
  impedance
• Single-ended and differential
• Press-fit and SMT

23
10Gbps Test Package Design Example

• Ceramic BGA
• Wire-bonded
• 4-Layer
• 1 mm pitch

Source Designed for Rambus by Kyocera
24
Example of a really good backplane
Works with simple OC192 xcvr
15 FR4
20 Roger
25
Outline

• Inside the router
• Backplane channel problem
• What can backplane designer do about it
• What can IC designer do about it
• Scaling the system to 10-100Tb/s

26
Loss Equalize to Flatten Response

• Channel is band-limited
• Equalization boost high-frequencies relative to
  lower frequencies

27
Receiver Linear Equalizer

• Amplifies high-frequencies attenuated by the
  channel
• Digital or Analog FIR filter
• Issues
• Also amplifies noise!
• Precision
• Tuning delays (if analog)
• Setting coefficients
• Adaptive algorithms such asLMS

28
Transmitter Linear Equalizer

• Attenuates low-frequencies
• Need to be careful about output amplitude
  limited output power
• If you could make bigger swings you would
• EQ really attenuates low-frequencies to match
  high frequencies
• Also FIR filter D/A converter
• Can get better precision than Rx
• Issues
• How to set EQ weights?
• Doesnt help loss at f

29
Transmit Linear Equalizer Single Bit
Operation
30
Decision Feedback Equalization (DFE)

• Dont invert channeljust remove ISI
• Know ISI because already received symbols
• Doesnt amplify noise
• Requires a feed-forward equalizer for precursor
  ISI
• Reshapes pulse to eliminate precursor

31
DFE Example
32
Transmit and Receive Equalization

• Transmit and receive equalizers are combined to
  make a range restricted DFE
• Tx equalizer functions as the feed-forward filter
• Rx equalizer restricted in performance of loop

33
Tx Rx Equalization Ranges
34
Pulse Amplitude Modulation

• Binary (NRZ) is 2-PAM
• 2-PAM uses 2-levels to send one bit per symbol
• Signaling rate 2 x Nyquist

• 4-PAM uses 4-levels to send 2 bits per symbol
• Each level has 2 bit value
• Signaling rate 4 x Nyquist

35
When Does 4-PAM Make Sense?

• First order slope of S21
• 3 eyes 1 eye 10db
• loss gt 10db/octave 4-PAM should be considered

36
Example 5Gbps Over 26 FR4 With No
Equalization
37
Example 5Gbps Over 26 FR4 Correct Tx
Equalization
38
Example 5Gbps Over 26 FR4 Under
Equalized
39
Example 5Gbps Over 26 FR4 Over Equalized
40
26 FR4 Bot 3.125Gbps, 2P noEQ
41
26 FR4 Bot 3.125Gbps, 2P w/EQ
42
26 FR4 Bot 6.4Gbps, 2P w/3G EQ
43
26 FR4 Bot 6.4Gbps, 2P w/EQ
44
26 FR4 Top 6.4Gbps, 2P w/EQ
45
26 FR4 Top 6.4Gbps, 4P w/EQ
46
26 Nelco6k-cb Top 10Gbps, 4P
47
26 Nelco6k-cb Top 6.4Gbps, 2P
48
Scaling the router throughput

• System (Tot. throughput 2.5Tb/s)
• 8-16 Line Cards
• lt40Gbs / LC 3-6Gbs links
• 40mW/Gbs Link power in 0.13um
• Speedup 2x
• links at switch card 200
• Limitations
• diff pairs at switch card 1640Gbs/6Gbs22
  400
• Switch card power from links 2006Gbs40mW/Gbs
  50W
• Connector density 50diff pairs/inch (tot
  length400/50 8 )
• BP/LC routing pitch 0.050
• Num. Layers (BP13, LC4) 100diff pairs/layer
  5 LC routing width
• Package ball pitch (1mm/200um)

49
Scaling the router throughput

• System (Tot. throughput 100Tb/s)
• 100 Line Cards
• 1Tbs / LC 10Gbs links
• 4mW/Gbs Link power in 0.065um
• Speedup 1x
• links at switch card 10k
• Limitations
• diff pairs at switch card 20k
• Switch card power from links in
  0.13um10k10Gbs4mW/Gbs 400W
• Connector density 50diff pairs/inch (tot
  length20k/50 400 )
• BP/LC routing pitch 0.050
• Num. Layers (BP13, LC4) 5k diff pairs/layer
  250 LC routing width
• Package ball pitch (1mm/200um)