CS Parallelism

1
CS Parallelism

• Dr. Vicki H. Allan

2
Concurrency

• Concurrency can occur at four levels
• Machine instruction level may have both an
  adder and a multiplier that are used at the same
  time.
• 2. High-level language statement level might
  have a loop (a,b,c) where c from one iteration
  and a from the next are executed at the same
  time.
• 3. Unit level several methods execute together
• 4. Program level several program execute together

3
Suppose we have two methods

• Populate marsh
• for(i0ilt 1000 i)
• create frog // high level language
  statement
• create carp
• create mosquitos
• Populate prehistoric world
• for (i0ilt10,i) create dinosaur(i)

4

• Concurrency can occur at four levels
• (termed granularity)
• Machine instruction level Create frog is
  decomposed into basic parts. If one basic
  instruction is to fold both sides into center,
  perhaps one processor folds the left side and
  one folds the right.
• 2. High-level language statement level -
• different parts of make frog happen together
• 3. Unit level populate marsh occurs with
  populate prehistoric world
• 4. Program level several programs (to do other
  things not shown here) execute together

5
What would be the advantages/disadvantages of
each type of parallelism?
6
The Evolution of Multiprocessor Architectures

• 1. Late 1950s - One general-purpose processor and
  one or more special-purpose processors for input
  and output operations
• 2. Early 1960s - Multiple complete processors,
  used for program-level concurrency
• 3. Mid-1960s - Multiple partial processors, used
  for instruction-level concurrency
• 4. Single-Instruction Multiple-Data (SIMD)
  machines. The same instruction goes to all
  processors, each with different data - e.g.,
  vector processors
• 5. Multiple-Instruction Multiple-Data (MIMD)
  machines
• Independent processors that can be synchronized
  (unit-level concurrency)

7
Making a Frog
Fold in sides
8
Take lower corner and fold up to top. Repeat
with other side.
Fold into middle
Repeat
9
Examples

• SIMD - all do the same things at the same time.
  
• All fold All Open All fold again
• Pipelined one person does fold, and then
  passes. Problems?
• MIMD all do different things

10
(No Transcript)
11

• Def A thread of control in a program is the
  sequence of program points reached as control
  flows through the program
• Categories of Concurrency
• 1. Physical concurrency - Multiple independent
  processors (multiple threads of control)
• 2. Logical concurrency - The appearance of
  physical concurrency is presented by time-sharing
  one processor (software can be designed as if
  there were multiple threads of control)

12
What would be the advantage of logical
concurrency?

• Consider the TV remote as performing context
  switch.
• Why does one switch between multiple programs?
• What is downside to switch?

13
Example Smart Remote Ads play when you are not
watching, assume program doesnt continue when
you arent watching it

• You might be an E-mail Junkie..
• You might be a computer science major
• Attraction to computer scientists

14
Concerns?

• Is switching between tasks confusing? What would
  need to be retained?
• Is switching between tasks expensive? Would
  there be a minimal size at which you spawn more
  tasks?
• What is the gain?

15

• What is the gain?
• Models actual situation better
• response time
• Use delays in processing

16
Why do we want parallelism?

• Price-performance curves
• Used to be paid more for computer - got more
  (linear relationship between price and
  performance).
• Now, for little money, get a lot of power. As you
  add more money, performance curve levels off.
  Not an efficient way to get more performance
• Parallelism is the answer string cheap
  computers together to do more work.

17
What is a Thread ?

• Just as multitasking OSs can run more than one
  process concurrently, a process can do the same
  by running more than a single thread.
• Each Thread is a different stream of control that
  can execute its instructions independently.
• Compared to a process, a thread is inexpensive to
  create, terminate, schedule or synchronize.

18
What is a Thread ?

• A process is a HEAVY-WEIGHT kernel-level entity.
  (process struct)
• A thread is a LIGHT_WEIGHT entity comprising the
  registers, stack and some other data.
• The rest of the process struct is shared by all
  threads. (address space, file desc, etc.)
• Most of the thread structure is at the user space
  allowing very fast access.

19
So for our example

• If we had two processes to populate the marsh and
  to populate the prehistoric world, each process
  would be able to stand alone.
• If we had two threads to populate the marsh and
  to populate the prehistoric world, they would
  have some shared resources (like the table or
  paper supply)

20
Concurrency Vs. Parallelism

• Concurrency means that two or more threads can be
  in the middle of executing code.
• Only one can be on the CPU though at any given
  time.
• Parallelism actually involves multiple CPUs
  running threads at the same time.
• Concurrency is the illusion of Parallelism

21
What can threads do that cant be done by
processes sharing memory ?

• Answer Nothing !... If you have
• plenty of time to kill programming,
• more time to kill processing,
• willing to burn money by buying RAM
• Debugging cross-process programs are tough.
• In Solaris creating a thread is 30 TIMES FASTER
  than forking a process.
• Synchronization is 10 time faster with threads.
• Context Switching - 5 times faster

22
What Applications to Thread?

• Multiplexing (communicate two or more signals
  over a common channel)
• Servers
• Synchronous Waiting (definition?)
• clients
• I/O
• Event Notification
• Simulations
• Parallelizable Algorithms
• Shared memory multiprocessing
• Distributed Multiprocessing

23
Which Programs NOT to thread?

• Compute bounds threads on a uniprocessor.
• Very small threads (threads are not free)
• Old Code
• Parallel execution of threads can interfere with
  each other.
• WARNING Multithreaded applications are more
  difficult to design and debug than single
  threaded apps. Threaded programming design
  requires careful preparation !

24
Synchronization

• The problem -
• Data Race - occurs when more than one thread is
  trying to update the same piece of data.
• Critical Section - Any piece of code to which
  access needs to be controlled.
• The Solution -
• Mutex
• Condition Variables
• Operations - init, lock, unlock

25
MUTEX

• A MUTual EXclusion allows exactly one thread
  access to a variable or critical section of code.
• Access attempts by other threads are blocked
  until the lock is released.

26

• Kinds of synchronization
• 1. Cooperation
• Task A must wait for task B to complete some
  specific activity before task A can continue its
  execution e.g., You cut the paper and then I fold
  it.
• 2. Competition
• When two or more tasks must use some resource
  that cannot be simultaneously used e.g., we both
  want the scissors.

27

• Liveness means the unit will eventually complete
  its execution. Im currently blocked from
  finishing my frog, but I will eventually get to
  finish.
• In a concurrent environment, a task can easily
  lose its liveness. You were supposed to wake me
  up when the scissors became available, but you
  forgot.
• If all tasks in a concurrent environment lose
  their liveness, it is called deadlock. I take the
  paper and wait for the scissors. You take the
  scissors and wait for the paper. Circular wait
  is deadlock.

28
Livelock theoretically can finish, but never get
the resources to finish.

• How do you prevent deadlock?
• How do you prevent livelock?

29
Questions?