HIPPI 6400

1
POTENTIAL HEP APPLICATIONS OF A NEW HIGH
PERFORMANCE NETWORKING TECHNOLOGY
1 A SHORT Introduction to The GSN
Specifications HIPPI 6400 PH ST Scheduled
Transfer SCSI over ST 2 Applications in High
Performance Computing 3 Applications in High
Energy Physics
Arie Van Praag Ben Segal CERN 1211 Geneva
23 Switzerland E-mail a.van.praag_at_cern.ch
http//www.cern.ch/HSI/gsn
2
HIPPI-6400 PH

• Development Project HIPPI-6400
• Commercial name GSN ( Gigabyte System Network )

Connections Full Duplex with Copper or
Fibre Cable Data Lines 164
82 Frame Signal 1 1 Clock 2
1
Principles Data Transfer in Small Micropackets 4
Multiplexed channels called Virtual channel VC0
-VC3 To avoid Congestion Operating System Bypass
with Scheduled Transfer To avoid OS Latency Full
48 bit Addressing Mode for Switching and Routing
following IEEE 802.nn To convert easily to other
network standards
3
HIPPI-6400 PH
HIPPI-6400 Standards
Document Description Status HIPPI-6400
PH Physical Layer 6400 Mbit/s ANSI NCITS
323-1998 or 800 MByte/s network ISO ISO/IEC
11518-10 HIPPI-6400 SC Switch Standard NCITS
324-1999 HIPPI-6400 OP Optical Connection NCITS
Submitted ST Scheduled Transfer NCITS
submitted SCSI over ST SCSI commands over
ST NCITS Standard SCSI T10
R-00 Sub-standards GSN ST conversions to
Fibre-Channel, HIPPI, Gigabit Ethernet,
Sonet, ATM
4
Virtual Channels
HIPPI-6400 PH
SOURCE
DESTINATION
CABLE OR FIBER
Output Buffer
Output Buffer
Virtual Channels
VC0 Maximum Size Messages of 68 Data
Micropackets ( 2176 bytes ) header
micropackets Admin Micropackets VC1 and VC2
Maximum Size Messages of 4100 Data
Micropackets ( 128 KBytes ) header
micropackets VC3 Maximum Size Messages
of 4 GBytes Data Micropackets header
micropackets
5
Flow Control
HIPPI-6400 PH
Credits are consumed as a packet moves from the
VCn Buffer to the Output Buffer.
SOURCE
DESTINATION
CABLE OR FIBER
Input Buffer
Output Buffer
TSEQ

ACK(seq)
RSEQ
ACKs are generated independent of the VC number,
and sent to the Source in the reverse direction
micropacket control information.
credit ( VCn,amount)
Credits are generated on a VC basis when data
exits from the VC buffer, and sent to the Source
in reverse direction microcontrol information.
6

Micropacket Message Format
HIPPI-6400 PH
32 DATA BYTES ( 256 Bits )
64 CONTROL BITS
MESSAGE FORMAT
7
Micropacket Structure
HIPPI-6400 PH
FIELD BITS CONTROL FONCTION VC 2 VC
Selector TYPE 4 Information Type T (AIL) 1 Last
Micropacket E (ERROR) 1 ERROR VCR 2 Virtual
Channel for credit addition CR 6 Number of
Credits RSEQ 8 ACK. Sequence number TSEQ
8 Transm. Sequence number ECRC 16 End to End
Checksum LCRC 16 Link Level Checksum
8
Control Micro Packets
HIPPI-6400 PH
Reset / Initialize
Credit Only
Null
Header
Data
Admin
Data Byte Contents 0 0 0 32 Byte
Header 32 Bytes Data Admin
Information VC 0 0 0 any any TYPE ( hex
) 2,3,4,5 7 A 9 8 F Tail 1 0 0
1 ERROR 0 0 0 1 if Erro
1 if Error 1 if Error
TSEQ xFF xFF Increments Increments
Increments Increments RSEQ 1
ACK ACK ACK ACK
ACK VCR 0 0 any any any
any CR 0 0
any any any
any LCRC single single single
single single
single ECRC single single single accumulating
accumulating accumulating
Request on VC1 Request on VC2
1 on last Micropacket
1 on last Micropacket
9
Error Correction
HIPPI-6400 PH
LINK CRC ( LCRC )
DATA IN
c47 c46 c45 c44 c43 c42
c41 c40 c39 c38 c37 c36 c35
c34 c33 c32
End to End CRC ( ECRC )
ECRC includes only Micropacket Data Bytes
including the Header Micropacket Data Bytes. X
X X X 1
16 12 3
XOR Function
10
Cable and Timing Sequences
HIPPI-6400 PH
50 METER
CONNECTORS BERG MICROPAX 100
CLOCK _2
CLOCK
Dnn or Cn
For 16 bit Macropacket
d00.0 - d00.3
d08.0 - d08.3
d16.0 - d16.3
d24.0 - d24.3
For 8 bit Macropacket
d00.0 - d00.3
d00.4 - d00.7
d08.0 - d08.3
d08.4 - d08.7
d16.0 - d160.3
d16.4 - d16.7
d24.0 - d24.3
d24.4 - d24.7
FRAME 16 bit
40 nsec
FRAME 8 bit
40 nsec
11
Fiber Connection
HIPPI-6400 OPT
2 CABLES 12 Fibers Each
Connectors MT
Connectors MT
Transmitter Launch Power -6 to -12
dBm Wavelength 850 nm Receiver
Sensitivity -18 dBm Berr 10
-12 Distance Max 1 Km Compliant to CDRH
1040.1 Class1 IEC 825-1 Class 3a
Long Distance Connection
Single Mode Fibers 2 Parallel Fibers Wavelength
1300 NM Distance 10 Km A working
Group is just formed
12
Switches
HIPPI-6400 SC
GSN - 64 SWITCH
GSN - 64 NODE
GSN - 64 NODE
Full Crossbar Non Blocking Switch with
Translation boxes for HIPPI-800 and other Data
Transporting Media
1 x
1 x
HIPPI 6400 DESTINATION
D
Translation Function
BRIDGE
4 x
8 X
HIPPI 800 SOURCE
S
HIPPI 800 Fibre Channel Gigabit Ethernet SONET
OC48c
HIPPI - 800 NODE
13
Scheduled Transfer
ST

• Specifies small control messages to pre-arrange
  Data movement
• Virtual Connection Set-up and Tear Down.
• Determine the number of operations the other side
  can accept
• Determine the Buffer Size of the other side
• Exchange Key, Port, Identifiers, Buffer Size
  Values
• Determine Maximum Transmission Unit to avoid
  overrun of the buffers at the Receiving End
• Acknowledge Partial Transfers
• Easy Transfers over Different type of High
  Performance Networks
• If used with prestored Tables an extension of a
  Scatter Gather System

14
Scheduled Transfer
ST
remote End local end Port Port Key Key Max.
Slots Max. Slots Bufsize Bufsize Max. STU Size
Max STU Size Max. Block Size Max. Block
Size Out_of_order cap. Ethertype Remote
Slots local Slots local Sync
Op_time Max_retry remote-id1 local-id1 re
mote-id2 local-id2 remote-idj local-idj
local-Port local-Key remote-Port
Virtual Connection Descriptors
Selection and Validation Criteria
Transfer Descriptor
Transfer Descriptor
Buffers
Buffer Descriptor Table
Block Descriptor
Buff 0 Bufx 1 Bufx 2 Bufx n
Address 0 Address 1 Address 2 Address n
. . . .
. . . .
15
ST
S T WRITE
Responder
Request Memory Region(RMR)
Memory Region Available (MRA)
Mxm lengthl offsetx Bufxn
Responder Buffer
Put STU Mxm
Get/FetchOp
Mxp, lengthk ioffsety, roffsetz iBufxo,
rBufxn1
Initiator Buffer
STU Mxp
FetchOp Complete
16
ST
Storage Commands
Commands are sent in STs RTS or RTR optional
payload
Byte
Content
0
type storage command
1
length 32
2-3
target number
4-31
SCSI Fibre Channel(FCP)
Command payload, bytes 0-27
17
Cannot use payload for SCSI status (can be gt30
bytes) Use STs Persistent Memory Region feature
SCSI - ST
Target
RTS I-id3 optcmd
Target Buffer
CTS
STU(data)
STU(data)
STU(data)
Status PMR
0 1 2 3 4 5
Status Put (STU)
18
S T
The Protocol Picture
NFS
FTP
Software
SCSI
...
TCP
UDP
Other protocols ...
IP
ST
DE/MUX
General Delivery Hardware
ST Hardware
Hardware
Media
19
Products as of January 2000
INTERFACES Silicon Graphics Origin
Series Available PCI Interface 64/66 Compaq
Alpha, SUN 1 Q 2000 SWITCHES ODS -
Essential 32 X 32 Available ODS - Essential
8 X 8 Available Genroco 8 X
8 Available PMR 8 X 8 Available BRIDG
ES ODS-Essential Translation Function
HIPPI-800 Available Genroco Storage Bridge
Fibre Channel Available Genroco Network Bridge
HIPPI Available Fibre Channel Available
Gigabit Ethernet Available OC48c 3 Q
2000 CABLES Berg Copper cables and
Connectors Available COMPONENTS for OPTICAL
CONNECTIONS Infineon Paroli AC Modules and
Fibres Available Gore Noptical Modules and
Fibres 1 Q 2000
20
Standards Popularity( 1995 )
Ethernet
T base 100
Gigabit Ethernet
Fibre Channel
ATM
HIPPI
HIPPI-Serial
Gigabyte System Network GSN
PCI
S-LINK
21
GSN
GSN SWITCHES
GSN
http//www.cern.ch/HSI/hippi/hug/demo99/Demo99.htm
l
SCSI/ST/GSN
GSN Storage Bridge
SCSI/FC
F.C.
SCSI/ST/GigE
SCSI/ST/ETH
FC TAPES
FC DISK ARRAYS
22
Cluster File Systems
Data Storage Networks
GFS Minnesota University
NT McIntosh Solaris etc.
LINUX IRIX
Central File Meta Data on a central
workstation Proprietary Software, e.g.
DataDirect CDNA Mercury SANergy, etc..
Symmetrical System with Distributed file system
Meta Data and Locks.
23
GSN to S-LINK ??
FPGA
Version with 6 S-LINK CANNELS Added Bandwidth
100 Mbyte/s per Interface Total Added bandwidth
800 Mbyte/s per Bridge
FIFO
FIFO
16 bit 66 MHz 132 MB/s
PROCESSOR BUS
FIFO
FIFO
FIFO
FIFO
reset
Register SET-UP
Control etc Glue Logic Synchronization etc.
PROCESSOR
FPGA
FPGA
FPGA
64 bit 40 MHz 320 MB/s
64 bit 40 MHz 320 MB/s
TO BRIDGE
RJ 12 RS232
FROM BRIDGE
24
Event Building with a Switch
1 0-100 TByte/s.
DETECTOR DATA
100- 1000 Bytes/s.
VMEbus Read Out Buffers ( ROB )
CONNECTIONS 768 (4) S-Link or 1152 (6)
S-Link or 192 HIPPI-800
10 - 100 MBytes/s
24 GSN Bridges
BRIDGE
BRIDGE
24 GSN Connections
32 X 32 GSN - 64 Switch Fabric
8 GSN Connections to Workstation Farm
FC DISK ARRAYS
To Central Data Storage or Data Analyzes
Long Distance Connection
25
NA48 Experiment
1 From Detector HIPPI raw data
CODING CONVERTING and RECODING
RECONVERTING
2 Data Distribution to Processor farm HIPPI
3 processor output in FDDI
4 Gigarouter converts data to FDDI over HIPPI
5 FDDI over HIPPI to computer centre
8 Km L.W.
6 Gigarouter converts to FDDI for processing
7 Via Gigarouter back to HIPPI for Storage
8 HIPPI to SCSI conversion for Storage
STK Tape Units
26
A FAT ST PIPE
GSN FC / ST / SCSI
Physics Data from Experiment
? ? ?
No Protocol or Format Change from DETECTOR to
Storage Analysis
Long Distance Transfer Connection
( Wavelength Multiplexing ? or Sonet or ?? )
? ? ?
GSN / Gigabit Ethernet GSN / HIPPI
Bridge GSN / FC
Central Storage in the Computer Center
FC
27
Physics Data Transportfor LHC
LHC Experiments Each experiment Transmits up
to 100 MBytes/s How to get this data to the
computer center ? OC 48c does 250 MByte/s
Atlas Alice LHCB CMS
28
GSN against Ethernet
Who makes a bet ?
Sorry Compaq that I mis-used your picture