Driver 1 Overview

1
Driver 1 High Performance Networking Chip
Krishna Saraswat Stanford University
Interconnect Focus Center
e
e
e
e
2
µ-Processor World is Changing

• Past trends
• Scaling has improved the device performance
• Processor frequency and IPC have increased
• Interconnect issues are now becoming a problem
• Power due to device and interconnects has gone up
• Conventional processor scaling is going to
  slowdown
• Design and fab costs are enormous
• Improving MIPS is getting harder
• Power is becoming increasingly a bigger problem
• For performance need to exploit parallelism
• Need new methods to tackle power problem

3
Delay Problems with Future ICs

• Even with repeaters, 7.5 clock cycles at 35nm
  node, 8X increase compared to 180nm node for a
  long signaling wire
• Future chips will require multiple clock cycles
  for communication over electrical lines
• Clock jitter and skew will become major problems

GLOBAL SIGNALING WIRE
Kapur and Saraswat, IEEE TED, April 2002
4
Chip Power Problem
Chandra, Kapur and Saraswat, IEEE IITC, June 2002

• Power dissipation rising to exorbitant
  proportions !!!
• A good chunk of it is attributed to interconnects
• Need to come up with novel schemes to reduce
  power in each department

5
A Possible Architecture of a Future µ-Processor
North Output
Logic, local memory local Wiring
W. J. Dally et. al., DAC, 2001
East Output
West Input
Tile Output
South Output
(West input shown as an example)
Partition of chip into tiles network logic

• Modular machine
• Lots of potential compute units, w/ memory
• On-chip network routers

6
Problems For Modular Machines

• Power
• Still need to get the power into the machine
• And the heat out
• Chip bandwidth
• On-chip
• Communications between the processors on die
• Off-chip
• Communication between the processors global
  memory
• Or other processors
• Locality is good
• But good machines provide large I/O bandwidth
• Current processors have 10-20GB/s I/O (80-160
  Gb/s)
• With multiple processors, numbers will be larger

7
Network Chip Problems are Worse

• Have many ports entering into switch (chip)
• Need very large I/O rates

Switch
Electronic Linecard 1
Electronic Linecard N

• Line termination
• IP packet processing
• Packet buffering

• Line termination
• IP packet processing
• Packet buffering

160Gb/s
160Gb/s
Arbitration
Request
Grant
8
Driver For High Performance Chips

• Must have high I/O demands
• Must have high power requirements
• Must have large on-chip bandwidth requirements
• Ultimately will have distributed computing too

9
Optics is Creeping Closer to the Chip

• Distances Important
• Things interesting at Inbox level traditionally
  Cu dominated, possible invasion of optics

Inbox traditionally Copper
Out of Box Optics strong-hold
On-chip
Over Backplane
Digital Systems Short interconnects
Network
10
A 40Tb/s Computing/Communication Chip

• Enables
• Scalable routers and switches
• 40Tb/s single-chip router
• Petabit router 100 chips
• Supercomputers
• 640 GFLOPS/chip
• PFLOPS machines with high global bandwidth with
  2K chips
• Prevents high-capacity systems from being limited
  by interconnect

11
While We Wont Be Building This Chip Soon

• Focuses attention on critical research problems

High-BW, off-chip links
Low Power Links
High-BW, low-power
on-chip links
Clocking and
Synchronization
Heat Removal
Power Distribution
P
M
M
On-chip networks
Optical/Electrical Packaging
Novel Optical Devices
12
On-Chip Links Alternatives in Cu Domain
Near Speed of light on-chip electrical
interconnects (Wong, Stanford)

• High frequencies travel close to speed of light
  (10 times faster than repeaters)
• Modulate electrical signal to higher carrier
  frequencies
• Possible Issues
• Requires good low-loss transmission lines
• Some overhead due to modulator/detector

Another example Low swing interconnects
(Horowitz, Stanford)
13
Clocking and Synchronization Cu Domain

• coupled standing wave oscillator for clock
  distribution.
• 10 GHz operation demonstrated, scalable to higher
  frequencies.
• Low skew jitter, lt 1ps demonstrated.
• Low power because the clock network operates in
  L-C mode.
• Coupled oscillators architecture can potentially
  reduce the number of clock tree layers, further
  reducing the power.
• Wide tuning range with the insertion of tuning
  networks.

10 GHz Standing Wave Clock
(Wong, Stanford)
14
Can Optical Interconnects help?
Chip-to-chip Optical Interconnects

• Can potentially address many aspects of the
  Driver 1 functions
• On-Chip Links
• Reduce delay
• Clocking and Synchronization
• Reduce jitter and skew
• High Bandwidth off-chip Links
• Reduce power

15
On-Chip Links Optical Vs. Electrical Wires (I)
Delay vs. Length

• Optical Interconnects are faster than repeated
  wires beyond a length well within chip size
• But we should also look at power

Kapur and Saraswat, IEEE IITC, June 2002
16
Optical Vs. Electrical Wires Delay Power
Scaling

• Scaling delay advantage increases for optics,
  power advantage diminishes
• Good for long global wires whose number is not
  large
• Even power advantage exists if switching
  activity (SA) is higher

Kapur and Saraswat, IEEE IITC, June 2002
17
Off Chip Links Photonic Components
Optical links

• Share many components with electrical links
• Need a transmitter to drive the laser or
  modulator
• Need a CDR to recover timing
• Also have some new components
• Laser/modulator, photo-diode
• Need driver for the modulator/laser, TIA for
  photo-diode
• Optical channel
• Connectors, optical wire (board, fiber, free
  space?)

Electrical links
18
I/O Link Design Issues

• Need to evaluate both electrical and optical I/O
• Determine which will scale to 40 Gb/s desired
• Since need many I/O issues is performance/power
• Not just performance
• Start with electrical links
• They are current standard (you need to beat them)
• Inside of every optical link is an electrical
  link
• Need to understand
• When optics has a chance
• What can be done to increase that chance

19
Off chip Optics Vs. Electrical
Off chip I/O

• Signal processing improves bit rates
• It also increases the cost of the link
• Area is not the issue, but power is
• More signal processing implies more flops and
  power
• Optics becomes interesting
• When cost of electrical solution is high
• And optics has a simple solution
• This will not happen below 10Gb/s

20
Specific Research Themes on Optical Interconnects

• Systems
• analyze and test proposed optical features and
  devices, find most important features of optics
• Integration techniques
• ultimate manufacturability, improved performance
• Optics
• need compact, manufacturable optics
• Short pulse sources
• exploit short pulse feature (unique to optics)
• Novel laser, modulator, and detector devices
• high-speed, low voltage, tolerant, high-yield,
  integrable devices
• Quantum dots
• possible low threshold, high speed lasers
• Photonic nanostructures
• radical opportunities in emerging field, for
  both passive opticsand active optoelectronics

21
Enabling Technology Devices

• What are the right optical devices to use?
• Need to be cheap, available in large numbers
• Operate at over 10Gb/s
• Compatible with CMOS
• Silicon devices are a long shot
• Need 3D heterogeneous integration
• Flip bond III-V to Si CMOS
• Current 3D process
• Cost, yield?
• How about germanium?
• Bandgap ideal for ? 1.5 µm
• Easily 3D integrable on Si

GaAs optoelectronic chip bonded to a Si CMOS chip
Monolithic Integration of a Ge optical receiver
22
3-D Circuit Technology
Reduced Interconnect Delay
High Bandwidth µ-Processors
Exploiting Different Process Technologies
Heterogeneous Material System Integration
Advanced Focal Planes

• Better circuit / interconnect ratio
• Unrestricted vertical interconnections between
  layers
• Low digital system power PCV2f (reduced C)
• Ideal for integration of optics with electronics

Courtesy Craig Keast, MIT Lincoln Labs
23
3D Approaches
Wafer Bonding (MIT)
Wafer Bonding (RPI)
Seeding crystallization of ?-Si (Stanford)

• Key Challenges
• Bonding Precise alignment of wafers/dies
• Crystallization Low thermal budget
  crystallization

24
The problems Caused by Increased Power
25
Integrated Microchannel Cooling

• Microfluidic networks with electroosmotic
  micropumps
• Microchannel Thermal Interconnects remove heat
  from targeted areas

Goodson, Kenny. (Stanford Univ.)
26
Driver 1 Connections

• Connects to all four tasks
• Task 1
• Electrical interconnects
• 3D
• Task 2
• Components for optical interconnects
• On-chip optical interconnects
• Optical clocks
• High speed electrical and optical I/O
• Task 3
• Power, cooling, packaging
• Thermal modeling
• Task 4
• Trade-offs between electrical and optical
  interconnections
• Internal networking, clock, etc