Implementing 100 Gigabit Ethernet: A Practical Guide

1
Implementing 100 Gigabit Ethernet A Practical
Guide

• Joel Goergen
• VP of Technology / Chief Scientist

2
Special Note Regarding Forward Looking Statements
This presentation contains forward-looking
statements that involve substantial risks and
uncertainties, including but not limited to,
statements relating to goals, plans, objectives
and future events.  All statements, other than
statements of historical facts, included in this
presentation regarding our strategy, future
operations, future financial position, future
revenues, projected costs, prospects and plans
and objectives of management are forward-looking
statements.  The words anticipates, believes,
estimates, expects, intends, may,
plans, projects, will, would and similar
expressions are intended to identify
forward-looking statements, although not all
forward-looking statements contain these
identifying words.  Examples of such statements
include statements relating to products and
product features on our roadmap, the timing and
commercial availability of such products and
features, the performance of such products and
product features, statements concerning
expectations for our products and product
features, and projections of revenue or other
financial terms. These statements are based on
the current estimates and assumptions of
management of Force10 as of the date hereof and
are subject to risks, uncertainties, changes in
circumstances, assumptions and other factors that
may cause the actual results to be materially
different from those reflected in our forward
looking statements.  We may not actually achieve
the plans, intentions or expectations disclosed
in our forward-looking statements and you should
not place undue reliance on our forward-looking
statements.  In addition, our forward-looking
statements do not reflect the potential impact of
any future acquisitions, mergers, dispositions,
joint ventures or investments we may make.  We do
not assume any obligation to update any
forward-looking statements.   Any information
contained in our product roadmap is intended to
outline our general product direction and it
should not be relied on in making purchasing
decisions. The information on the roadmap is (i)
for information purposes only, (ii) may not be
incorporated into any contract and (iii) does not
constitute a commitment, promise or legal
obligation to deliver any material, code, or
functionality.  The development, release and
timing of any features or functionality described
for our products remains at our sole discretion.
3
Per IEEE-SA Standards Board Operations Manual,
January 2005

• At lectures, symposia, seminars, or educational
  courses, an individual presenting information on
  IEEE standards shall make it clear that his or
  her views should be considered the personal views
  of that individual rather than the formal
  position, explanation, or interpretation of the
  IEEE.

4
Acronym Cheat Sheet

• CFI Call for Interest
• DWDM Dense Wavelength Division Multiplexing
• EMI Electro-magnetic Interference
• Gbps Gigabit per Second
• HSSG Higher Speed Study Group
• ITU International Telecommunications Union
• IETF Internet Engineering Task Force
• JEDEC - Joint Electron Device Engineering Council
  
• MAC Media Access Control
• MDI Medium Dependent Interface
• MSA Multi Source Agreement
• OIF Optical Internetworking Forum
• PCS Physical Coding Sublayer
• PMA Physical Medium Attachment
• PMD Physical Medium Dependent
• PHY Physical Layer Device
• SERDES Serialize / De-serialize
• SMF / MMF Single Mode Fiber / Multi Mode Fiber
• Tbps Terabit per Second

5
IEEE802.3 HSSG

• My thoughts on where we are ..

6
Current IEEE802.3 HSSG Objectives as of January
2007

• To date, the objectives within the study group
  are
• Support full duplex only
• Preserve the 802.3 / Ethernet frame format at the
  MAC Client Service Interface
• Preserve the min and max FrameSize of current
  802.3 standard
• Support a speed of 100Gbps at the MAC/PLS
  Interface
• Support at least 10km on SMF
• Support at least 100m on OM3 MMF
• Support a BER better then or equal to 10-12 at
  the MAC / PLS Service Interface
• Support at least 40km on SMF

7
Upcoming IEEE802.3 HSSG March Plenary

• Looking for data to support the Broad Market
  Potential of 100Gbps.
• Looking for data to support the market for 40km
  on SMF.

8
Shipping an IEEE802.3 HSSG Vendor Compliant
Product

• Key architectural points need to be addressed
  from the MAC to the PMD.
• Likely to be mid 2009 or later before the
  standard is complete enough to ensure
  compatibility between vendors.

MAC Media Access Control MDI Medium Dependent
Interface PCS Physical Coding Sublayer PMA
Physical Medium Attachment PMD Physical Medium
Dependent WIS WAN Interface Sublayer XGMII
10 Gigabit Media Independent Interface
9
Developing a Standard
Ideas From Industry
Industry Pioneering 1 Year
Feasibility and Research
Ad Hoc Efforts
Call for Interest
CFI July 18, 2006
Study Group
HSSG is here
Task Force
Q4 07
Working Group Ballot
Sponsor Ballot
Force10 delivers 100G Ethernet System
Standards Board Approval
09 10
Publication
10
100Gbps Architecture

• Line Card
• Switch Fabric
• Back Plane
• Power System

11
Thoughts for a 100Gbps Line Card
12
Thoughts for a 100Gbps Line Card

• 600 Mhz x 192 bit datapath is barely feasible in
  90 nm. Need to study 65nm feasibility and pick
  the right width and clock speed.
• Memory components listed in the line card diagram
  are available now or definitely before 2008.
• 32 x 10Gbps CEI serdes are possible in few 90 nm
  ASIC technologies.
• Feasibility of Mac is well documented by
  belhadj_01_1106.pdf.
• NPU Signal Pin Count 643 in this example
  including SERDES pins
• Traffic Manager Signal Pin Count 1269 in this
  example including SERDES pins. May have to be
  implemented using two chips to reduce the pin
  count.
• Die Size of the above chips are application
  dependent. For a hardwired NPU,TM 65 nm may be
  the right process.

13
Thoughts for a 100Gbps Front End

• Optics Short Reach (100m)
• 10 by 10Gbps
• Optics Long Reach (10km)
• 4 by 25Gbps
• 5 by 20Gbps
• Optics Extended Reach (40km)
• 4 by 25Gbps
• 5 by 20Gbps

14
Thoughts for a 100Gbps Front End
15
Thoughts for a 100Gbps Front End

• XFI based on 8 inches 1 connector for fr-4
  circuit boards
• Applications suggest 12 inches 1 connector for
  fr-4 circuit boards
• See proposed model

16
Thoughts for a 100Gbps Switch Fabric
17
Thoughts for a 100GbpsSwitch Fabric

• 600 Mhz x 192 bit data path is barely feasible in
  90 nm. Need to study 65nm feasibility and pick
  the right width and clock speed for a digital
  cross bar, multiple ASIC implementation.
• 32 x 10Gbps CEI serdes are possible in few 90 nm
  ASIC technologies. Need to study scaling 500 x
  10Gbps SERDES across multiple ASICs.
• Die Size of the above chips are application
  dependent. For a hardwired switch fabric, 65 nm
  may be the right process.
• Conceivable to see Master / Slave ASIC
  Architecture.

18
Thoughts for a 100GbpsBack Plane
Data Packet
Line Cards --GbE / 10 GbE RPMs SFMs Power Supplies
SERDES
Backplane
Traces

• Mid Plane applications have complexities with
  holes and connectors that make it incompatible
  for HSSG systems.

19
Thoughts for a 100GbpsBack Plane Long Reach
30in.
SDD21 -20log10(e)(b1sqrt(f) b2f b3f2
- b4f3) b1 1.25e-5
b2 1.20e-10 b3
2.50e-20 b4 0.95e-30 f 50Mhz
to 15000Mhz

• Commercially available resin-based laminates
  exist to meet these requirements (see
  OIF2006.097.00)
• Proposed CEI25 Channel Model (see OIF2006.047.00)
  Chip vendors would like to see 9dB better
  channel loss.

20
Power SystemThoughts for a 100Gbps Line Card

• Architecture
• Clean trace routing.
• Good power noise control means better than...
• Analog target 60mVpp ripple
• Digital target 150mVpp ripple
• Excellent SERDES to connector signal flow to
  minimize ground noise.
• Best choice for 100Gbps systems.

21
Power SystemThoughts for a 100Gbps Fabric

• Architecture
• Clean trace routing.
• Good power noise control means better than...
• Analog target 30mVpp ripple
• Digital target 100mVpp ripple
• Excellent SERDES to connector signal flow to
  minimize ground noise.

22
Power System300VDC to 500VDC System
Backplane
PEM-A
DC PSU-A
F1
RTNA

Vo1
Inrush and soft start circuit
CB1
VDC
Vin
F3
F2
RTN
-VA
-
V
Vo2
on/off
F1
RTNB

F4
RTN
V-
CB2
VDC
Vin
Vo3
F3
-VB
-
F5
RTN
DC PSU-B
PEM-B
System Board
23
Power System300VDC to 500VDC System

• Current DC systems require large gauge wire.
  Reducing the current capacity will allow smaller
  gauge wires.
• Smaller gauge wires allow ease of use, yet still
  can carry the 10kw to 15kw required for a 100Gbps
  system.
• Future AC systems may have to go to 480VAC to
  allow for smaller gauge wires.

24
Industry System Port Count Cycle
2006
2010
2008
2004
2002
GE
100s Ports
gt 1000 Ports
10s Ports
gt 100s Ports
100s Ports
10 GE
100 GE
Standard In Development
10s Ports
25
Conclusions

• Establish optical interfaces
• Establish electrical interfaces, including
  circuit board trace characteristics for both the
  front end optics / electrical, and the back plane
  electrical
• Study the ASIC requirements for the Network
  Processor Unit, the Traffic Manager, and the
  Switch Fabric blocks.
• Study the power subsystem.
• Cooling and EMI have to be a part of this.

26
Thank You
27
IEEE 802.3 HSSG Reflector and Web

• To subscribe to the HSSG reflector, send an email
  to
• ListServ_at_ieee.org
• with the following in the body of the message
• subscribe stds-802-3-hssg ltyour first namegt
  ltyour last namegt
• end
• SSG web page URL
• http//grouper.ieee.org/groups/802/3/hssg/index.
  html
• John DAmbrosia, Chair IEEE802.3 HSSG
• jdambrosia_at_force10networks.com