Parallel Computing What and Why

1
Parallel Computing What and Why?

• What is parallel computing?
• using several processors/execution units in
  parallel to collectively solve a problem
• loosely defined, with plenty of overlap with
  distributed computing
• Definition for our purposes
• the processors are contributing to the solution
  of the same problem
• in distributed systems the problem is often that
  of coordination (e.g. leader election, commit,
  termination detection), here the problem
  involves lots of data and computation (e.g.
  matrix multiplication, sorting)

2
Why do we need powerful computers?

• To solve much bigger problems much faster!
• Performance, performance, performace
• there are problems which can use any amount of
  computing (i.e. simulation)
• Capability
• to solve previously unsolvable problems
• too big data sizes, real time constraints
• Capacity
• - to handle a lot of processing much faster,
  perform more precise computer simulations

3
Why are powerful computers parallel?

• From Transistors to FLOPS
• by Moores law the of transistors per area
  doubles every 18 months
• how to make use of these transistors?
• more execution units, graphical pipelines, etc.
• more processors per die
• The Data Communication Argument
• for huge data it may be cheaper/more feasible to
  move the computation towards data
• The Memory/Disk Speed Argument
• parallel platforms typically yield better memory
  system performance, because they have
• larger aggregate caches
• higher aggregate bandwidth to memory system

4
A bit of historical perspective

• Parallel computing has been here since the early
  days of computing.
• Traditionally custom HW, custom SW, high
• The doom of the Moore law
• custom HW has hard time catching up with the
  commodity processors
• Current trend use commodity HW components,
  standardize SW
• Market size of High Performance Computing
• the market size for disposable diapers
• (Explicitly!) Parallel computing has never been
  mainstream.

5
A bit of historical perspective (cont.)

• Parallelism sneaking into commodity computers
• Instruction Level Parallelism - wide issue,
  pipelining, OOO
• data level parallelism SSE, 3DNow, Altivec
• thread level parallelism Hyperthreading in
  Pentium IV
• Transistor budgets allow for multiple processor
  cores on a chip.
• Most applications would benefit from being
  parallelised and executed on a parallel computer.
• even PC applications, especially the most
  demanding ones
• games, multimedia

6
A bit of historical perspective III

• Chicken Egg Problem
• Why build parallel computers when the
  applications are sequential?
• Why parallelize applications when there are no
  parallel commodity computers?
• Answers
• What else to do with all those transistors?
• They already are a bit parallel (wide issue,
  multimedia instructions, hyperthreading), and
  this bit is growing.
• Yet another reason to study parallel computing
• Principles of parallel algorithm design (locality
  of data reference) lend themselves to
  cache-friendly sequential algorithms.
• The same applies for out-of-core computations
  (data servers).

7
The beauty and challenge of parallel algorithms

• Problems that are trivial in sequential setting
  can be quite interesting and challenging to
  parallelize.
• Very simple example Computing sum of n numbers
• How would you do it in parallel?
• using n processors
• using p processors
• when communication is cheap
• when communication is expensive