Parallel Programming

1
Parallel Programming

• August 27, 2002

2
About the Course

• Lee McCauley
• 374 Dunn
• 678-2486
• t-mccauley_at_memphis.edu

3
About the Course

• Designing and Building Parallel Programs
• by Ian Foster, Addison Wesley (ISBN
  0-201-57594-9)
• Distributed Systems Concepts and Design 3rd
  Ed (NOT IN BOOKSTORE YET)
• by G. Coulouris, J. Dollimore, T. Kindberg,
  Addison Wesley (ISBN 0-201-61918-0)

4
About the Course

• Schedule (on the handout)
• Evaluation
• Class Participation and Attendance (10)
• Case Study Reports (30)
• Final Project (60)
• Proposal (5) Due 9/24
• Written Draft (15) Due 11/19
• Final Written Paper (60) Due 12/3
• Oral Presentation (20) Due 12/3

5
Trends in Parallel Computing
6
Trends in Parallel Computing
7
Trends in Distributed Computing

• Seti_at_Home
• EoN
• Evolution_at_Home
• Distributed Particle Accelerator Design
• climateprediction.com

8
Fundamental Requirements

• Locality
• Concurrency
• Scalability
• Modularity

9
A Parallel Machine Model

• Each computer executes its own program
• May access local memory or send/receive messages
• Access to same node memory is less costly than
  remote

10
Other Models

• MIMD (multiple instruction multiple data)

11
Other Models

• Shared memory multiprocessor

12
Other Models

• Local area network

13
Parallel Programming Model

• Programming von Neumann machines
• Theoretically doable in machine code
• More practical to use a high-level language
• Programming a parallel system is comparable
  only worse
• Which abstractions should we use?

14
Tasks and Channels

• A parallel computation consists of one or more
  tasks that execute concurrently. The number of
  tasks can change during execution
• A task is carried out as a sequential program
  using local memory with inports and outports to
  interface with the environment
• In addition to all local actions, a task can
  send/receive messages, create a new task, and
  terminate

15
Tasks and Channels

• A send operation is asynchronous while a receive
  is synchronous
• Outport/Inport pairs can be connected through
  channels
• Channels can be created, deleted, and referenced
  in messages
• Tasks can be mapped to physical processors in
  various ways that does not affect the semantics
  of the program

16
Lets Build a Bridge
17
Lets Build a Bridge
18
Mapping Independence

• The result of program execution is independent of
  the location of tasks
• Algorithms can be designed using tasks and
  channels regardless of the actual number of
  processors in a system scalability

19
Modularity

• The task/channel model is similar to OO
  programming
• Tasks encapsulate data and the code that runs on
  that data
• Channels are used in lieu of inter-object method
  calls

20
Other Programming Models

• Message Passing
• Only slightly different than the task/channel
  model and are, in practice, interchangeable
• Implementations tend to create a number of
  identical tasks (SPMD)

21
Other Programming Models

• Data Parallelism
• Applies the same operation to multiple elements
  of a data structure
• Compilers for this model tend to require
  additional information (such as how to distribute
  data over processors) and produce a SPMD
  executable

22
Other Programming Models

• Shared Memory
• All processors share the same memory space
• Often easier to design but introduces several
  additional worries
• Grid-lock
• Program result is not necessarily deterministic