Who We Are

1
Who We Are?
Detector Building Group of KFKI-RMKI(Research
Institute for Particle and Nuclear Physics),
Budapest, HUNGARY
2
Our Results

• Previous projects
• Designing Fibre Channel test equipments
  (1994-1998)
• Portable, 266 Mb/s Fibre Channel Tester
• Fibre Channel Preprocessor for Logic Analyzers
• Designing high-speed data transmission interfaces
  for CERN detectors (1996 - ...)
• Protocol design and verification
• FC, GbE, physical layer components
• Hardware design
• Software linux drivers, test programs, program
  library

3
S-LINK (CERN)

• S-LINK interface cards for CERN (mostly used at
  ATLAS)
• simple, unidirectional link interface
• first successful version was designed by
  KFKI-RMKI
• newer versions, now 2.5 Gb/s
• Also used at MPI, Garching and at several HEP and
  other scientific institutes all over the world

4
ALICE Detector Data Link (DDL)
5
DDL

• Detector Data Link (DDL) for the ALICE detector
  at CERN
• 2.5 Gb/s, duplex link
• advanced featues
• Test devices for DDL (Front-end emulator, DDL
  link emulator, etc.)

• PCI Read-Out Receiver Card (RORC) for DDL
• 1st version 33 MHz, 32-bit PCI card
• 2nd version 66 MHz, 64-bit PCI with 2
  integrated DDL interfaces

6
DDL Interfaces
DIU -RORC Interface
FEE - SIU Interface
Read-out Receiver Card (RORC) 66 MHz 64-bit PCI
DDL Destination Interface Unit (DIU)
Front-end Electronics
DDL Source Interface Unit (SIU)
Front-end Bus 32 bit
32 bit
4
T A P
optical cable, max 350 m
1
32 bit
JTAG BST lines
DDL
7
DDL Features

• Interface
• Full duplex 32-bit data path on the destination
  interface (DIU card)
• Half duplex 32-bit data path on the source
  interface (SIU card)
• Full duplex flow control (XON/XOFF)
• Interface clock up to 66 MHz (easy integration
  with PCI 66)
• 264 MB/s peak data rate, 240 MB/s sustained
  bandwidth (max.)
• Implementation
• Duplex LC optical link up to 300 m
• 2x FC or 2x GbE physical layer components
• Small Form Factor Pluggable (SFP) optical
  transceivers
• Bit error rate lt 10 -12
• Robust error detection very low undetected bit
  error rate lt 10-40
• Automatic link synchronization and management
• Extras
• Stand-by support (low power consumption)
• In-system reconfiguration /  Remote system
  upgrade
• Monitoring of the aging of laser diode of optical
  transceivers

8
Outlook to PCI Express

• PCI Express (formerly 3GIO)
• a third generation high performance I/O bus
• (1st generation ISA, EISA, VESA, 2nd
  generation PCI, PCI-X)
• PCI evolution (PCI, PCI-X, PCI-Express)
• PCI Express is software compatible to PCI and
  PCI-X

9
PCI Evolution

• PCI
• multi-drop parallel bus
• conventional PCI 33 MHz, 32-bit, max 4-5 card
  slots per bus
• newer versions 66 MHz, 64-bit, 1 (max 2) card
  slots per bus (!)
• PCI-X
• parallel bus, backward compatible (hw and sw)
• 66 MHz , 32/64-bit, max 4-5 card slots per bus
• 133 MHz, 32/64-bit, max 1-2 card slots per bus
• 266, 532 MHz versions max 1 card slot per bus
  (!)
• Buses can be bridged to each other (complex,
  expensive)
• PCI-XP
• It is still a local /IO bus, a PCI bus, but the
  connections between devices are serial,
  point-to-point interconnections
• devices are interconnected via switch(es)
• large number of devices can be interconnected
• highly scalable, hot-plug, hot-swap, QoS, etc.

10
Limitations of PCI and PCI-X

• Conventional 33 MHz PCI system
• low bandwith to nowadays needs
• 66 MHz, 133 MHz (PCI-X), ...
• only few devices can be interconnected (only 1 or
  two) on a single bus because of the strict
  electrical load and timing constraints
• Further limitations of PCI architecture
• inefficient solutions in
• data transfer cycles (wait states)
• accessing of system memory
• interrupt handling
• error handling

11
PCI Express

• However, the basic problems of a parallel bus
  system(electrical load and timing constraints,
  lack of hot-pluggability, lack of scalability,
  etc.) can be solved only by a complete redesign
  of the architecture.
• This resulted in PCI Express.

12
The PCI Express Link

• Serial point-to-point interconnect between two
  devices
• 1x, 2x, 4x, 8x, 12x, 16x, 32x type links
• 1, 2, 4, ... 32 bidirectional signal pairs
  (lanes)
• 1 lane 2.5 Gb/s now, up to 10 Gb/s later
• Low voltage, differential signaling (LVDS)
• AC coupled
• Data is encoded 8B/10B

a PCI Express link
Device A
Device B (e.g. a switch)
1 to 32 lanes
13
Bandwith

• Scalable more lanes /link higher bandwith
• 2.5 Gb/s per lane, 8B/10B encoding
• Simultaneous traffic in both directions
• 1x type link
• 500 MByte/s aggregate bandwith
• 250 MByte/s per direction
• 32x type link
• 16 GByte/s aggregate bandwith
• (8 GByte/s per direction)

14
Backward compatibility

• Supports familiar PCI transactions such as memory
  read/write, IO read/write and configuration
  read/write
• Same memory, IO and configuration address space
  as in PCI and PCI-X
• Existing OSs and driver software will run in a
  PCI Express system without any modifications
• Supports chip-to-chip interconnect and
  board-to-board interconnect via cards and
  connectors similar to the present PCI systems
• PCI Express motherboard will have a similar form
  factor to existing ATX motherboards in PCs

15
Improvements, Benefits

• Fast, highly scalable, serial point-to-point
  interconnect between two devices
• bytes striped accross the lanes ? more lanes
  per link faster transmission
• Packet-based communication protocol
• Packets are transmitted serially
• CRC embedded in each packet (auto retry link
  level error correction)
• Buffer-to buffer (link level) flow control ? no
  need the packet retry
• Message Signaled Interrupt (MSI) architecture
• No side-band signals
• hot-plug, error handling, interrupt signaling and
  else are accopmlished in-band
• Multiple devices are interconnected via switches
• Large number of devices can be connected together
  in a system
• Much fewer pins per device package
• Reduces chip and board design cost and design
  complexity

16
Outside the box?

• Initial focus of usage inside the box

• It is expected that later it will also be used
  outside the box for I/O expansion

PCI-XP links
PCI-XP links
Add-in card
Add-in card
switch
switch
opticalcable
connector
PCI-XP link
17
Mechanical Form Factors

• Connector and daughter card form factors are
  under specification
• The main type of connectors is very similar to
  the present PCI card edge connector
• A 1x type card can be inserted in a x4 type slot,
  a.s.o.
• Like with PCI, there will be other form factors
• Server I/O module
• Mini PCI Express card and connector (e.g. for
  notebooks)
• mezzanine type card
• NEWCARD (will replace CardBus PC card), etc.

18
Compete or complete?

• PCI Express will coexist with PCI / PCI-X in the
  same system
• PCI Express intends to replace AGP (graphics card
  if), but will not replace Serial ATA (hard disk,
  CD, etc)
• HyperTransport
• onboard chip-to-chip interconnect
• said to be complementary to PCI Express
• RapidIO
• as an onboard chip-to-chip interconnect
  complementary to PCI-Express
• as a system interconnect a competitive
  technology
• 1394b, USB2.0, Fibre Channel, Gb Ethernet,
  Infinband, etc.
• PCI Express can be a system interconnect that
  bridges these technologies
• Will it be?