BEE2 Hardware Design

1
BEE2 Hardware Design

• Chen Chang
• BWRC, Jan 2004

2
State-of-the-Art in FPGAs

• Xilinx Virtex-II Pro 100
• 100K logic blocks, each with 4 FFs and 4 4-luts
  ? 4 Million logic gates
• 1 Mbytes SRAM bits
• 500 18x18bit dedicated multipliers
• 20 10-Gbit/s serial communication links
• 1000 user I/Os (most with LVDS 600 Mb/s
  signalling) Good from inter-chip communication
  and memory interface.
• 2 imbedded hard PowerPC cores
• 10 GFLOPs (single precision), Itanium 2,
  1.6GFLOPs
• 200 (16-bit) GOPS

3
B2 Implementation Philosophy

• Use all COTS components
• Highly modular
• Easy system integration
• Only two types of major components
• Demonstrate SCF capabilities
• Scalable from single module to approximately 1K
  FPGA chips in a system (8 TFlops)

4
Computing node and memory

• Single Xilinx Virtex 2 Pro 70 FPGA
• 1704 package with 996 user I/O pins
• 2 PowerPC 405 cores
• 500 dedicated multipliers (18-bit)
• 1MB block SRAM on-chip
• 4 physical DDR 400 banks
• Each banks has 72 data bits with ECC
• Independently addressed with 16 logical banks
  total
• 12.8 GBps memory bandwidth, with up to 8 GB
  capacity

5
Inter-node connections

• Point-to-point connection between control node
  and processing node
• 144 bit 300 MHz DDR
• 38.4 Gbps bandwidth each branch
• Uplink connect to other modules to form a 4-ary
  tree
• Downlinks for I/O on leaf nodes and for tree
  connection on switch nodes

6
4-ary Tree connection

• 4-ary tree configuration
• High bandwidth high latency 12X Infiniband 2.5
  Gbps duplex
• Low bandwidth low latency 64 pin (32 bit) LVDS _at_
  200 MHz DDR
• Every fifth B2 module acts as a switch node
• Switch B2 modules can be used as aggregation
  computing points

7
B2 Module logical view

• 4 computing nodes, 1 control node
• 8 SATA connection up to 8 hard disks

8
B2 module physical view

• Performance
• 32 GFLOPS or 1 TOPS
• up to 32 GB DDR DRAM
• 50 GBps memory bandwidth
• 2 TByte hard drive disks
• I/O Bandwidth
• 120 Gbps high-speed I/O
• 2 Gigabit Ethernet
• Power lt 400W
• Form factor
• 1 RU (19 by 27)

9
Alternative Switch Scheme

• Specialized crossbar switch implemented as ASIC
  (Mellanox)
• 200 ns latency
• Fat tree organization with constant cross section
  bandwidth

10
Disk Storage Schemes

• Intra B2 module working storage at each module
• User disk storage schemes
• Connection to existing NAS through Gigabit
  Ethernet from all B2 modules
• Direct high bandwidth storage nodes attached to
  the main crossbar network
• SAN bridge attached to the main crossbar network
  adapting to existing SAN