FCoE Overview

1
FCoE Overview

• IEEE CommSoc/SP Chapter
• Austin, Texas, May 21 2009
• Tony Hurson
• tony.hurson_at_ieee.org

2
Networked Storage History
3
SANs Latency, Throughput Requirement
4
SCSI Read, Write over FC
5
FC Fabric Port Terminology
6
FC Initialization Flow

• Host (Physical Machine) initiates Fabric Login
  with nearest switch FLOGIN receives a VN_Port
  ID in return
• (Additional entities on same Host e.g. Virtual
  Machines may also log into fabric using NPIV
  FDISC, each getting a unique VN_Port ID).
• (Host may use well-known FC servers to discover
  the VN_Port IDs of its Targets and their LUNs).
• Host initiates Port Login with chosen Targets

7
FC Routing
8
Ethernet Routing

• Dynamic Scheme Source Learning
• If unicast DstMAC is not in lookup table, flood
  frame to all ports except its source port.
• Note source port of SrcMAC in lookup table, if
  not already present
• Age/invalidate lookup entries
• Similar flooding behavior for multicast
• Precludes loops in fabric

9
FC Frame Format
10
FC Classes of Service

• Class 1 dedicated connection obselete
  unsupported in FCoE
• Class 2 acknowledged, with ACKs from remote
  peer. Supported in FCoE, though some link-layer
  features (F_RJT, F_BSY) irrelevant. Useful for
  large Sequences and sequential devices.
• Class 3 unacknowledged, FCoE-supported. Most
  prevalent for disk applications. Error recovery
  punted to higher (FCP) levels.
• Class F inter-switch link only. FCoE-supported

11
Protocol Stack History and Comparison
12
iSCSI SAN Pros and Cons

• For
• Runs over existing, ubiquitous Ethernet, TCP/IP
  fabrics.
• Internetworking built in.
• Lots of fabric management tools (ready for
  enterprise storage?).
• Against
• TCP Slow Start impacts I/O latency, throughput
  (but newer TCPs are tunable)
• Lossy fabrics impact I/O latency through
  re-transmission and complicate receive endpoint
  data placement.
• Bridging to legacy FC SANs slow/expensive because
  of TCP termination overhead.

13
Lossless Ethernet via PAUSE
14
FCoE Early Deployment Example
15
FCoE Frame Format
16
FCoE Endpoint Model
17
FCoE Switch Functional Model
18
Converged Ethernet

• AKA Data Center Bridging (DCB). Run up to four
  major traffic classes on single 10 GbE fabric. In
  order of market prevalence
• Networking (TCP/IP, lossy).
• Block Storage (lossless FCoE, or lossless/lossy
  iSCSI).
• Management (heartbeat traffic, low bandwidth,
  but must get through).
• Inter-Process Communication (clustered computing
  high bandwidth, low latency, lossless preferred).

19
Groundwork for DCB

• IEEE 802.1Qaz ETS DCBX bandwidth allocation
  to major traffic classes (Priority Groups) plus
  DCB management protocol.
• IEEE 802.1Qbb Priority PAUSE. Selectively PAUSE
  traffic on link by Priority Group.
• IEEE 802.1Qau Dynamic Congestion Notification.

20
IEEE 802.1Qaz Enhanced Transmission Selection

• Support at least 3 Priority Groups/traffic
  classes
• PGs identified by Priority field of existing
  802.1Q VLAN Tag
• Configured Bandwidth per PG has 1 resolution
• PG15 has limitless bandwidth (use sparingly!, for
  Management)
• Work Conservation if the wires free, use it.

21
ETS Configuration Example

• PG0 (Storage) 40 of port b/w
• PG1 (Networking) 20 of port b/w
• PG2 (IPC) 40 of port b/w
• PG15 (mgmt) limitless
• If a PG underutilizes, others can fill the space.
• Typical implementation DWRR.

22
IEEE 802.1Qbb Priority PAUSE
23
IEEE 802.1Qau Dynamic Congestion Control
Background

• Lossless fabrics are prone to congestion
  spreading (congestion trees).
• Ethernet-FC gateways with their different port
  speeds (10 GbE 8 Gbps) are natural bottlenecks.
• ETS Work Conservation model adds fuel to fire.
• Solution switches/endpoints notify traffic
  sources of incipient congestion, via feedback
  messages sources reduce rates accordingly.

24
Congestion Notification in Action
25
Congestion Control at Endpoint Transmit
26
FCoE Summary

• Presents new, but very familiar, PHY and Link
  Layers for FC.
• Core switching discipline remains FC-SW-5.
• Higher FC layers almost completely unchanged
  (thats the legacy value!)
• Biggest Ethernet-level requirement lossless
  fabric.
• Part of Converged Ethernet initiative lots of
  ancillary activity at IEEE.

27
Further Reading

• FCoE www.t11.org
• IEEE 802.1Q(azaubb) www.ieee.org
• Thank you! Questions?