Building a Beowulf: My Perspective and Experience

1
Building a BeowulfMy Perspective and Experience

• Ron Choy
• Lab. for Computer Science
• MIT

2
Outline

• History/Introduction
• Hardware aspects
• Software aspects
• Our class Beowulf

3
What is a Beowulf ?

• Massively parallel computer built out of COTS
• Runs a free operating system (not Wolfpack, MSCS)
• Connected by high speed interconnect
• Compute nodes are dedicated (not Network of
  Workstations)

4
Who uses Beowulfs?

• Pharmaceutical companies
• Investment firms
• Animation makers

5
The Beginning

• Thomas Sterling and Donald Becker CESDIS, Goddard
  Space Flight Center, Greenbelt, MD
• Summer 1994 built an experimental cluster
• Called their cluster Beowulf

6
The First Beowulf

• 16 x 486DX4, 100MHz processors
• 16MB of RAM each, 256MB in total
• Channel bonded Ethernet (2 x 10Mbps)
• Not that different from our Beowulf

7
The First Beowulf (2)
8
Current Beowulfs

• Faster processors, faster interconnect, but the
  idea remains the same
• Cluster database http//clusters.top500.org/db/Qu
  ery.php3
• Top cluster 1.920 TFLOPS peak

9
Current Beowulfs (2)
10
Why Beowulf?

• Its cheap!
• Our Beowulf, 18 processors, 9GB RAM 15000
• A Sun Enterprise 250 Server, 2 processors, 2GB
  RAM 16000
• Everything in a Beowulf is open-source and open
  standard - easier to manage/upgrade

11
Essential Components of a Beowulf

• Processors
• Memory
• Interconnect
• Software

12
Processors

• Major vendors AMD, Intel
• AMD Athlon MP
• Intel Pentium 4

13
Comparisons

• Athlon MPs have more FPUs (3) and higher peak
  FLOP rate
• P4 with highest clock rate beats out the Athlon
  MP with highest clock rate in real FLOP rate
• Athlon MPs have higher real FLOP per dollar,
  hence it is more popular

14
Comparisons (2)

• P4 supports SSE2 instruction set, which perform
  SIMD operations on double precision data (2 x
  64-bit)
• Athlon MP supports only SSE, for single precision
  data (4 x 32-bit)

15
Memory

• DDR RAM (double data rate) used mainly by
  Athlons, P4 can use them as well
• RDRAM (Rambus DRAM) used by P4s

16
Memory Bandwidth

• Good summary
• http//www6.tomshardware.com/mainboard/02q1/02031
  1/sis645dx-03.html
• DDR beats out RDRAM in bandwidth, and is also
  cheaper

17
Interconnect

• The most important component
• Factors to consider
• Bandwidth
• Latency
• Price
• Software support

18
Ethernet

• Relatively inexpensive, reasonably fast and very
  popular.
• Developed by Bob Metcalfe and D.R. Boggs at Xerox
  PARC
• A variety of flavors (10Mbps, 100Mbps, 1Gbps)

19
Pictures of Ethernet Devices
20
Myrinet

• Developed by Myricom
• OS bypass, the network card talks directly to
  host processes
• Proprietary, but very popular because of its low
  latency and high bandwidth
• Usually used in high-end clusters

21
Myrinet pictures
22
Comparison
23
Cost Comparison

• To equip our Beowulf with
• Fast ethernet 1700
• Gigabit ethernet 5600
• Myrinet 17300

24
How to choose?

• Depends on your application!
• Requires really low latency e.g. QCD? Myrinet
• Requires high bandwidth and can live with higher
  latency e.g. ScaLAPACK? Gigabit ethernet
• Embarrassingly parallel? Anything

25
What would you gain from fast interconnect?

• Our cluster Single fast ethernet (100Mbps)
• 36.8 GFLOPS peak, HPL 12 GFLOPS
• 32.6 efficiency
• GALAXY Gigabit ethernet
• 20 GFLOPS peak, HPL 7 GFLOPS
• 35 efficiency old, slow tcp/ip stack!
• HELICS Myrinet 2000
• 1.4 TFLOPS peak, HPL 864 GFLOPS
• 61.7 efficiency

26
My experience with hardware

• How long did it take for me to assemble the 9
  machines? 8 hours, nonstop

27
Real issue 1 - space

• Getting a Beowulf is great, but do you have the
  space to put it?
• Often space is at a premium, and Beowulf is not
  as dense as traditional supercomputers
• Rackmount? Extra cost! e.g. cabinet 1500,
  case for one node 400

28
Real issue 2 heat management

• The nodes, with all the high powered processors
  and network cards, run hot
• Especially true for Athlons - can reach 60C
• If not properly managed, the heat can cause crash
  or even hardware damage!
• Heatsink/fans remember to put in in the right
  direction

29
Real issue 3 - power

• Do you have enough power in your room?
• UPS? Surge protection?
• You dont want a thunderstorm to fry your
  Beowulf!
• For our case we have a managed machine room -
  lucky

30
Real issue 4 - noise

• Beowulfs are loud. Really loud.
• You dont want it on your desktop.

Bad idea
31
Real issue 5 - cables

• Color scheme your cables!

32
Software

• Well concentrate on the cluster management core
• Three choices
• Vanilla Linux/FreeBSD
• Free cluster management software (a very patched
  up Linux)
• Commercial cluster management software (very very
  patched up Linux)

33
The issues

• Beowulfs can get very large (100s of nodes)
• Compute nodes should setup themselves
  automatically
• Software updates must be automated across all the
  nodes
• Software coherency is an issue

34
Vanilla Linux

• Most customizable, easiest to make changes
• Easiest to patch
• Harder for someone else to inherit the cluster
  a real issue
• Need to know a lot about Linux to properly setup

35
Free cluster management softwares

• Oscar http//oscar.sourceforge.net/
• Rocks http//rocks.npaci.edu
• MOSIX http//www.mosix.org/
• (usually patched) Linux that comes with software
  for cluster management
• Reduces dramatically the time needed to get
  things up and running
• Open source, but if something breaks, you have
  one more piece of software to hack

36
Commercial cluster management

• Scyld www.scyld.com - founded by Donald Becker
• Scyld father of Beowulf
• Sells a heavily patched Linux distribution for
  clustering, free version available but old
• Based on bProc, which is similar to MOSIX

37
My experience/opinions

• I chose Rocks because I needed the Beowulf up
  fast, and its the first cluster management
  software I came across
• It was a breeze to setup
• But now the pain begins severe lack of
  documentations
• I have actually stripped the cluster of all Rocks
  features now -gt almost a plain RH7.1

38
Software (contd)

• Note that I skipped a lot of details e.g. file
  system choice (NFS? PVFS?), MPI choice (MPICH?
  LAM?), libraries to install
• I could talk forever about Beowulfs but it wont
  fit in one talk

39
Recipe we used for our Beowulf

• Ingredients 15000, 3 x 6 packs of coke, 1 grad
  student
• Web surf for 1 weeks, try to focus on the Beowulf
  sites, decide on hardware
• Spend 2 days filling in various forms for
  purchasing and obtaining competitive quotes
• Wait 5 days for hardware to arrive, meanwhile web
  surf some more, and enjoy the last few days of
  free time in a while

40
Recipe (contd)

• 4. Lock grad student, hardware (not money), and
  coke in an office. Ignore scream. The hardware
  should be ready after 8 hours.

Office of the future
41
Recipe (contd 2)

• 5. Move grad student and hardware to its final
  destination. By this time grad student will be
  emotionally attached to the hardware. This is
  normal. Have grad student set up software. This
  would take 2 weeks.

42
Our Beowulf
43
Things I would have done differently

• No Rocks, try Oscar or may be vanilla Linux
• Color scheme the cables!
• Try a diskless setup (saves on cost)
• Get rackmount

44
Design a 30000 Beowulf

• One node (2 processors, 1GB RAM) costs 1400,
  with 4.6 GFLOPS peak
• Should we get
• 16 nodes, with fast ethernet, or
• 8 nodes, with Myrinet?

45
Design (contd)

• 16 nodes with fast ethernet
• 73.6 GFLOPS peak
• 23.99 GFLOPS real (using the efficiency of our
  cluster)
• 16 GB of RAM
• 8 nodes with Myrinet
• 36.8 GFLOPS peak
• 22.7 GFLOPS real (using the efficiency of HELICS)
• 8 GB of RAM

46
Design (contd 2)

• First choice is good if you work on linear
  algebra applications, and require lots of memory
• Second choice is more general purpose

47
Top cluster as of August 2002

• Locus Discovery 1.92TFLOPS peak, fast ethernet
• HELICS 1.4336TFLOPS peak, Myrinet
• Who is top?