Pointtopoint semantics

1
Lesson2

• Point-to-point semantics
• Embarrassingly Parallel Examples

2
send/recv

• What we learned last class
• MPI_Send(void buf, int count, MPI_Datatype type,
  int dest, int tag, MPI_Comm comm)
• MPI_Recv(void buf, int count, MPI_Datatype type,
  int src, int tag, MPI_Comm comm, MPI_Status
  stat)
• stat is a C struct returned with at least the
  following fields
• stat.MPI_SOURCE
• stat.MPI_TAG
• stat.MPI_ERROR

3
Blocking vs. non-blocking

• Send/recv functions in previous slide is referred
  to as blocking point-to-point communication
• MPI also has non-blocking send/recv functions
  that will be studied next class MPI_Isend,
  MPI_Irecv
• Semantics between two are very different must
  be very careful to understand rules to write safe
  programs

4
Blocking recv

• Semantics of blocking recv
• A blocking receive can be started whether or not
  a matching send has been posted
• A blocking receive returns only after its receive
  buffer contains the newly received message
• A blocking receive can complete before the
  matching send has completed (but only after it
  has started)

5
Blocking send

• Semantics of blocking send
• Can start whether or not a matching recv has been
  posted
• Returns only after message in data envelope is
  safe to be overwritten
• This can mean that date was either buffered or
  that it was sent directly to receive process
• Which happens is up to implementation
• Very strong implications for writing safe programs

6
Examples
MPI_Comm_rank(MPI_COMM_WORLD, rank) if (rank
0) MPI_Send(sendbuf, count, MPI_DOUBLE, 1,
tag, comm) MPI_Recv(recvbuf, count,
MPI_DOUBLE, 1, tag, comm, stat) else if (rank
1) MPI_Recv(recvbuf, count, MPI_DOUBLE, 0,
tag, comm, stat) MPI_Send(sendbuf, count,
MPI_DOUBLE, 0, tag, comm) Is this program
safe? Why or why not? Yes, this is safe even if
no buffer space is available!
7
Examples
MPI_Comm_rank(MPI_COMM_WORLD, rank) if (rank
0) MPI_Recv(recvbuf, count, MPI_DOUBLE, 1,
tag, comm, stat) MPI_Send(sendbuf, count,
MPI_DOUBLE, 1, tag, comm) else if (rank
1) MPI_Recv(recvbuf, count, MPI_DOUBLE, 0,
tag, comm, stat) MPI_Send(sendbuf, count,
MPI_DOUBLE, 0, tag, comm) Is this program
safe? Why or why not? No, this will always
deadlock!
8
Examples
MPI_Comm_rank(MPI_COMM_WORLD, rank) if (rank
0) MPI_Send(sendbuf, count, MPI_DOUBLE, 1,
tag, comm) MPI_Recv(recvbuf, count,
MPI_DOUBLE, 1, tag, comm, stat) else if (rank
1) MPI_Send(sendbuf, count, MPI_DOUBLE, 0,
tag, comm) MPI_Recv(recvbuf, count,
MPI_DOUBLE, 0, tag, comm, stat) Is this
program safe? Why or why not? Often, but not
always! Depends on buffer space.
9
Message order

• Messages in MPI are said to be non-overtaking.
• That is, messages sent from a process to another
  process are guaranteed to arrive in same order.
• However, nothing is guaranteed about messages
  sent from other processes, regardless of when
  send was initiated

10
Illustration of message ordering
P0 (send)
P1 (recv)
P2 (send)
11
Another example
int rank MPI_Comm_rank() if (rank 0)
MPI_Send(buf1, count, MPI_FLOAT, 2, tag)
MPI_Send(buf2, count, MPI_FLOAT, 1, tag) else
if (rank 1) MPI_Recv(buf2, count,
MPI_FLOAT, 0, tag) MPI_Send(buf2, count,
MPI_FLOAT, 2, tag) else if (rank 2)
MPI_Recv(buf1, count, MPI_FLOAT, MPI_ANY_SOURCE,
tag) MPI_Recv(buf2, count, MPI_FLOAT,
MPI_ANY_SOURCE, tag)
12
Illustration of previous code
send send
recv send
recv recv
Which message will arrive first?
Impossible to say!
13
Progress

• Progress
• If a pair of matching send/recv has been
  initiated, at least one of the two operations
  will complete, regardless of any other actions in
  the system
• send will complete, unless recv is satisfied by
  another message
• recv will complete, unless message sent is
  consumed by another matching recv

14
Fairnesss

• MPI makes no guarantee of fairness
• If MPI_ANY_SOURCE is used, a sent message may
  repeatedly be overtaken by other messages (from
  different processes) that match the same receive.

15
Send Modes

• To this point, we have studied non-blocking send
  routines using standard mode.
• In standard mode, the implementation determines
  whether buffering occurs.
• This has major implications for writing safe
  programs

16
Other send modes

• MPI includes three other send modes that give the
  user explicit control over buffering.
• These are buffered, synchronous, and ready
  modes.
• Corresponding MPI functions
• MPI_Bsend
• MPI_Ssend
• MPI_Rsend

17
MPI_Bsend

• Buffered Send allows user to explicitly create
  buffer space and attach buffer to send
  operations
• MPI_BSend(void buf, int count, MPI_Datatype
  type, int dest, int tag, MPI_Comm comm)
• Note this is same as standard send arguments
• MPI_Buffer_attach(void buf, int size)
• Create buffer space to be used with BSend
• MPI_Buffer_detach(void buf, int size)
• Note in detach case void argument is really
  pointer to buffer address, so that add
• Note call blocks until message has been safely
  sent
• Note It is up to the user to properly manage the
  buffer and ensure space is available for any
  Bsend call

18
MPI_Ssend

• Synchronous Send
• Ensures that no buffering is used
• Couples send and receive operation send cannot
  complete until matching receive is posted and
  message is fully copied to remove processor
• Very good for testing buffer safety of program

19
MPI_Rsend

• Ready Send
• Matching receive must be posted before send,
  otherwise program is incorrect
• Can be implemented to avoid handshake overhead
  when program is known to meet this condition
• Not very typical dangerous

20
Implementation oberservations

• MPI_Send could be implemented as MPI_Ssend, but
  this would be weird and undesirable
• MPI_Rsend could be implemented as MPI_Ssend, but
  this would eliminate any performance enhancements
• Standard mode (MPI_Send) is most likely to be
  efficiently implemented

21
Embarrassingly parallel examples

• Mandelbrot set
• Monte Carlo Methods
• Image manipulation

22
Embarrassingly Parallel

• Also referred to as naturally parallel
• Each Processor works on their own sub-chunk of
  data independently
• Little or no communication required

23
Mandelbrot Set

• Creates pretty and interesting fractal images
  with a simple recursive algorithm
• zk1 zk zk c

• Both z and c are imaginary numbers
• for each point c we compute this formula until
  either
• A specified number of iterations has occurred
• The magnitude of z surpasses 2
• In the former case the point is not in the
  Mandelbrot set
• In the latter case it is in the Mandelbrot set

24
Parallelizing Mandelbrot Set

• What are the major defining features of problem?
• Each point is computed completely independently
  of every other point
• Load balancing issues how to keep procs busy
• Strategies for Parallelization?

25
Mandelbrot Set Simple Example

• See mandelbrot.c and mandelbrot_par.c for simple
  serial and parallel implementation
• Think how load balacing could be better handled

26
Monte Carlo Methods

• Generic description of a class of methods that
  uses random sampling to estimate values of
  integrals, etc.
• A simple example is to estimate the value of pi

27
Using Monte Carlo to Estimate p

• Fraction of randomly
• Selected points that lie
• In circle is ratio of areas,
• Hence pi/4

• Ratio of Are of circle to Square is pi/4
• What is value of pi?

28
Parallelizing Monte Carlo

• What are general features of algorithm?
• Each sample is independent of the others
• Memory is not an issue master-slave
  architecture?
• Getting independent random numbers in parallel is
  an issue. How can this be done?

29
Image Transformation

• Simple image operations such as rotating,
  shifting, and scaling parallelize very easily
  (see Wilkinson pg. 81)
• However, image smoothing is just a step more
  involved, since the computations