Parallel Algorithms

1
Parallel Algorithms

• Lecture 34
• CS 312

2
Follow up

• Necessary and sufficient
• p is necessary for q when q gt p
• p is sufficient for q when p gt q
• For Fermats little theorem,
• n is prime is sufficient for a n-1 mod n 1

3
Details

• The last day of this class is next Wed.
• The last day on the schedule is next Mon.
• Mon will be project QA. Wed. review.
• Project 5 will be due Wed.
• Report will be due Thurs.
• Report is extra credit
• Report not accepted after Thurs. midnight

4
Objectives

• Explain what a computational model is
• Compare PRAM with fixed networks
• Describe the PRAM computational model
• Compute speedups

5
Definitions

• Sequential Algorithm
• Any algorithm designed for a single-processor
  machine
• Parallel Algorithm
• Any algorithm designed for a multiprocessor
  machine

6
Simple Example

• I have 5 loads of clothes to wash
• It takes 25 minutes to wash one load
• How long will it take me?

7
Simple Example

• I have 5 loads of clothes to wash
• It takes 25 minutes to wash one load
• How long will it take me?
• 125 minutes using 1 washing machine
• 25 minutes, using 5 washing machines (assuming
  all washing machines take the same amount of time)

8
Computational Models

• For performance evaluation
• A computational model describes how much it costs
  to perform atomic, or basic, operations.
• For today, well talk only about time cost.

9
Computational Models

• Sequential
• RAM random access machine
• Read, write, add, subtract, compare etc. all done
  in one time step.
• Parallel
• Given n machines, perform n adds, subtracts,
  compares etc in one step.
• Communication is more complicated because it
  requires a shared resource
• Shared resource implies a policy for sharing.

10
Two Communication Models

• Fixed network
• Parallel Random access
• Hybrids exist, but we wont cover them.
• Fixed network of PRAMS. Etc.

11
Fixed Network

• Processors are connected by communication links.
• Two processors can communicate in one time step
  (if there is an edge between them).

12
Fixed Network Policies

• Links simplex or duplex?
• Completely connected?
• Forwarding, bypassing, dropping?

13
Parallel RAM

• All processors share access to a common memory
• All processors have some local memory as well
• Two processors can communicate in two time steps.

14
PRAM Policies
A
B
What is A and B want to read at the same
time? What if A and B want to write at the same
time?
15
PRAM Policies

• Concurrent reads
• A and B can read at the same time
• Concurrent writes
• A and B can write at the same time
• CREW, EREW, ERCW, CRCW
• Read/write bypassing
• Dropping reads/writes
• Fences

16
Another Example

• 10 numbers to be added
• Two people (A and B)
• How do we parallelize this?
• On a fixed network
• On a PRAM
• How much time is saved?

17
In-class Demo

• 5 volunteers to add 10 numbers
• CREW PRAM
• White board is the shared memory
• Send writes to the queue. (Ill be the queue)
• Read at anytime
• Fixed connections
• Completely connected simplex network.

18
Speedup

• ? Given problem for which the best-known
  sequential algorithm has a run time of S(n)
• n problem size
• Parallel algorithm on a p-processor machine in
  time T(n, p)
• Speedup

19
Efficiency
The efficiency of a parallel algorithm is
Where S(n) bound on the sequential
algorithm, and T(n,p) bound on a parallel
algorithm with p processes n problem size p
number of processors.
20
What is the speedup for 100 numbers problem?

• Sequential time 99 units of time
• Person A can add 50 numbers in 49 units of time
• At the same time, Person B can add the other 50
  numbers
• In one more unit of time, the two sums can be
  added.
• Speedup

21
Heap Sort

• Optimal run time
• Let A be an n-processor parallel algorithm that
  sorts n keys in
• Let B be an n2-processor algorithm that also
  sorts n keys in time
• Speedup of A is
• Speedup of B is also

22
Superlinear speedup

• Is it possible?

23
Superlinear speedup

• Is it possible?
• Only when speedup is defined with runtimes
• p processors have more aggregate memory
• the cache-hit frequency might be better for the
  parallel machine w/more aggregate cache than one
  processor

24
Solve a problem in parallel

• Explore many algorithms
• Identify the one that is most parallelizable
• To get a good speedup, parallelize every
  component of the underlying algorithm

25
Amdahls law

• If a fraction ƒ of the technique cannot be
  parallelized, then the maximum speedup that can
  be obtained is limited by ƒ
• Maximum speedup achievable with ƒ and p is