Studies in Parallel and Distributed Systems

1
159.735 Studies in Parallel and Distributed
Systems Computer Science Massey University at
Albany
2
Lecturer
Dr. Christopher Messom Phone [phone] ext
9323 Office 2.58 Quad A EmailC.H.Messom_at_masse
y.ac.nz
3
Lectures
2 lectures/tutorials per week Thursday 1000
1200 IIMS LAB 1 Friday 1200 1400 IIMS
LAB 1
Office Hours One hour after each Lecture or by
appointment
4
Text and Course Material

• B. Wilkinson and M. Allen, Parallel Programming
  Techniques and Applications Using Networked
  Workstations and Parallel Computers, Prentice
  Hall.

5
Assessment

• The course will be assessed by a combination of
  practical and theoretical work.
• There will be 3 practical assignments, a research
  seminar and one three hour exam. The exam will be
  a CLOSED BOOK exam.
• The marks for this course will be divided as
  follows
• Item of Total
• Ass1 10
• Ass2 10
• Ass2 10
• Seminar 20
• Final Exam 50
• There will be several additional labs and
  exercises that will not be graded

6
Course Outline

• Parallel system architectures grand challenge
  problems, shared and distributed
• memory, interconnection networks, clusters,
  grids, Flynns classifications, performance
  measures
• Message passing programming principles, PVM, MPI
  and its uses
• Parallelization techniques and strategies
  embarrassingly parallel, partitioning,
  pipelining, synchronous and asynchronous methods
• Load balancing static and dynamic load
  balancing, termination detection algorithms
• Algorithms and applications sorting, searching,
  numerical, image processing, genetic algorithms

7
What is Parallel Programming ?

• Writing MPI programs and implement them on a
  cluster computer
• Studying the different paradigms and
  architectures of parallel and distributed systems
• Studying the different parallelization techniques
  and strategies
• Studying the various load balancing and
  termination strategies
• Analyzing a given application/problem and
  implement a parallelization strategy

8
What is not Parallel Programming ?

• Designing Parallel Computers
• Low Level Operating System design

9
Technologies that support Parallel Programming

• Open Standards
• MPI
• Message Passing Interface
• PVM
• Parallel Virtual Machine
• Openmp
• Shared memory parallelism
• Threads
• Pthreads etc
• We will focus on MPI

10
Common Parallel Programming design patterns

• Identify Problem
• Take serial algorithm if available
• Subdivide algorithm into parallel and serial
  components
• Analyse if worth making parallel
• Implement in Parallel language/ library/ tool

11
Summary

• What is Parallel Programming?
• What are the technologies and standards that
  support parallel programming?
• What is a common Parallel Programming Design
  Pattern?