Headlines:

1
InfiniBandTM for pedestrians

• Headlines
• ? Background
• ? Why InfiniBandTM
• ? Architecture main characteristics
• ? Physical specifications
• ? Recent InfiniBandTM products announcements
• ? Road map in time for the LHC experiments?

Disclaimer this short presentation does not
pretend in any way to give a comprehensive
overview of Infiniband and Infiniband products.
Some figures have been copied from IBA Specs 1.0
2
Future IO NGIO gt InfiniBandTM

• August 31, 1999
• Seven of the computer industry's leading
  technology companies, Compaq, Dell,
  Hewlett-Packard Company, IBM, Intel, Microsoft
  Corp., and Sun Microsystems announced today the
  intent to merge the best ideas of the Future I/O
  (FIO) and Next Generation I/O (NGIO) input/output
  architectures into one specification to be used
  by server and peripheral vendors throughout the
  industry. A new industry group is being formed
  and will adopt technologies and concepts from
  both NGIO and FIO efforts.
• October 24, 2000
• INFINIBANDSM TRADE ASSOCIATION RELEASES 1.0
  SPECIFICATION
• Volume 1, Logical specifications 880 pages
• Volume 2, Physical specifications 623 pages
• InfiniBand Trade Association around 215 member
  companies http//www.infinibandta.org/home.php3
• The official name is InfiniBandTM Architecture
  (IBA)

3
Why Infiniband?

• InfiniBandTM Architecture Specification - sect.
  1.2.1
• Existing interconnect technologies have failed
  to keep pace with computer evolution and the
  increased burden imposed on data servers,
  application processing, and enterprise computing
  created by the popular success of the Internet.
  High-end computing concepts such as clustering,
  fail-safe, and 24x7 availability demand greater
  capacity to move data between processing nodes as
  well as between a processor node and I/O devices.
  These trends require higher bandwidth and lower
  latencies, they are pushing more functionality
  down to the I/O device, and they are demanding
  greater protection, higher isolation,
  deterministic behaviour, and a higher quality of
  service than currently available.
• IBA is targeted to address the demands
  generated by the rapid growth of the Internet and
  the convergence of data and telecommunications
  (voice, data, video, and storage) on the
  Internet. It is a key technology for server
  clusters as well as I/O for remote storage and
  networking.

4
IB specs a small glossary

• The specifications are very detailed, some
  glossary
• Channel Adapter device that terminates a link
  and executes transport-level functions. Host
  Channel Adapter or Target Channel Adapter
• Channel the association of two queue pairs for
  communication
• Global Identifier (GID) a 128-bit identifier
  used to identify a Port on a Channel Adapter, a
  Port on a Router, or a Multicast Group. A GID is
  a valid 128-bit IPv6 (Internet Protocol) address
  with additional properties.
• Link a full duplex transmission path between
  any two network fabric elements
• Local Identifier (LID) a address assigned to a
  port, unique within the subnet
• Physical Lane a set of one transmit and one
  receive differential pairs. A 1X, 4X and 12X link
  is composed of one, four and twelve physical
  lanes, respectively.
• Queue Pair consists of a Send Work Queue and a
  Receive Work Queue. Send and Receive queues are
  always created as a pair. (Prevents deadlock)
• Router a device that transports packets between
  IBA (InfiniBand Architecture) subnets
• Subnet a set of InfiniBandTM Architecture
  Ports, and associated links, that have a common
  Subnet ID and are managed by a common Subnet
  Manager
• Switch a devic that routes packets from one
  link to another of the same Subnet, using the
  Destination Local Identifier field in the Local
  Route Header (LRH)

5
IBA Overview and Components

• IBA defines a System Area Network (SAN). An IBA
  system can range from a small server to a
  massively parallell cluster. The IP friendly
  nature of IBA allows bridging to internet and
  intranet.
• The devices in an IBA system are classified as
• switches
• routers
• channel adapters
• repeaters
• links that connect switches, routers, repeaters,
  and channel adapters
• The management infrastructure includes
• subnet managers
• general service adapters

6
(No Transcript)
7
IBA types of service

• Connection oriented vs. datagram.
• For connection oriented service, the Queue Pair
  (QP) is associated with exactly one other QP.
• Datagram operation allows a single QP to be used
  to send and receive messages to/from any QP on
  any node.
• Acknowledged (reliable) vs. unacknowledged
  (unreliable)
• Two types of RAW datagrams IPv6 and Ethertype.
• IBA defines both channel (send/receive) and
  memory (RDMA) semantics.

8
Layered Architecture
9
(No Transcript)
10
IBA physical layers

• Physical attach point to a port
• a Cable Connector
• a Fiber Connector
• a Backplane Connector

11
IBA physical layers

• The IB physical lane encoding will use the
  industry standard 8B/10B code, which is used by
  Fibre Channel andGigabit Ethernet. Provides DC
  balance, limited run lengths, byte
  synchronization, and the ability to distinguish
  between data and control characters.
• Release 1.0 of IBA supports only 2.5 Gbit/s
  signaling. However, provisions are made for
  potential frequency increase in future versions.
• Full options aggregate bandwidths of 500
  MBytes/s, 2 GBytes/s, and 6 GBytes/s with a
  2.5 Gbit/s wire signaling rate. (Note! 10B
  encoding gives 250 Mbytes/s, the 500 Mbytes/s
  value must refer to sum of input and output
  pair).
• Cabling both copper (differential pairs) and
  fiber-optic. Note that 1X copper comprises 4
  conductors 1 diff. pair in and 1 diff. pair out.
• Hot-pluggable modules

12
(No Transcript)
13
Infiniband products / road map

• Infiniband product marketing and chip set
  announcements are now streaming out from vendors.
  Some recent WWW information is reproduced on the
  following pages.
• On the whole, it seems that end-2001 represents
  the start-up of the Infiniband bandwagon.
• However, a real Infiniband computing paradigm
  requires an extensive software effort, the
  hardware will probably be the easiest part of it!?

14
InfinibandTM product announcements
Commercial

• Banderacom, Austin, Texas
• Feb. 06, 2001 announced availability of the
  Ibandit prototype chip set and product
  development kit
• http//www.banderacom.com
• Lane15, Austin, Texas and Netreon, Redwood City,
  Calif.
• February 6, 2001 - Lane15 Software, the leading
  developer of InfiniBandTM management software,
  and NetreonTM Inc., a provider of enterprise
  integration software for networked devices, today
  announced a collaboration to ensure that hardware
  based on the InfiniBand architecture will be
  manageable using Microsoft Active DirectoryTM
  and Windows 2000 management tools. The two
  companies expect the joint effort to result in
  implementation guidelines and management products
  that will ease the transition from current
  technologies to InfiniBand-based products for
  enterprises using Windows 2000.
• http//www.lane15.com
• http//www.netreon.com

15
InfinibandTM product announcements
Commercial

• Intel, SANTA CLARA, Calif.
• Jan. 29, 2001 - Intel Corporation today announced
  it is shipping samples of its InfiniBand silicon
  and software products, critical components that
  will help implement the InfiniBand I/O industry
  specification and enable a key advance in
  Internet servers.
• The InfiniBand architecture is a new I/O
  infrastructure technology that simplifies and
  speeds server-to-server connections and links to
  other server-related systems such as remote
  storage and networking devices. InfiniBand
  architecture's easier connectivity, reduced
  latency, improved bandwidth and enhanced
  interoperability features increase the
  performance, reliability and scalability of Intel
  Architecture-based (IA) servers to meet the
  growth needs of emerging e-Business data centers,
  service providers and server farms. Components of
  the InfiniBand architecture are expected to be
  commercially released later this year, with
  InfiniBand-enabled systems expected in late 2001.
  
• http//www.intel.com/pressroom/archive/releases/20
  010129net.htm

16
InfinibandTM product announcements
Commercial
SANTA CLARA, CALIFORNIA and YOKNEAM, ISRAEL
February 05, 2001 - Mellanox Technologies, Ltd.
today introduced its InfiniBridgeTM family of
devices supporting the new InfiniBand
architecture and announced immediate availability
of silicon. The InfiniBridge family of products
includes Switches, Host Channel Adapters (HCAs),
and Target Channel Adapters (TCAs). Mellanox is
shipping InfiniBridge silicon, development
boards, and software products to customers,
marking the first commercial availability of
devices supporting both 2.5Gb/s (1X) and 10Gb/s
(4X) links compatible with the InfiniBand 1.0
specification. The first member of the
InfiniBridge product family is the MT21108, which
combines Host Channel Adapter functionality with
a fully integrated eight-port switch, and an
InfiniBand to PCI Target Channel Adapter bridge.
17
InfinibandTM product announcements
Commercial
http//www.mellanox.com
18
IBA documents - further reading

• IBA Specifications 1.0 (1500 pages in total!),
  see http//www.infinibandta.org/home.php3
• The specifications are very detailed
• For more information on IBA, see presentations at
  the October 2000 Developers Conference, Las
  Vegas http//www.infinibandta.org/events/past/veg
  as20001024.htm These presentations, viewable and
  downloadable as PDF documents, give a
  comprehensive overview of the nuts and bolts of
  IBA, and are much easier to read than the very
  detailed specification documents.

19
InfinibandTM for the LHC timescale?

• The paradigm of the InfinibandTM Architecture
  seems to offer very interesting prospects also
  for scientific large scale computing.
• The first chip and board products are now coming
  on the market.
• However, a large software effort from IBA
  partners is required in order to offer a seamless
  and flexible integration. The long planning and
  implementation cycles in building advanced
  computer architectures will result in a slow
  adoption cycle. SGI has said (ref. IDC White
  Paper) that they plan to incorporate Infiniband
  Architecture into the next generation of Origin
  3000 that are expected sometime in the next two
  years.
• Starting to build LHC computer systems in 2004-5
  could therefore be of InfiniBand architecture.