Title: Parallel Computing
1Parallel Computing
- A task is broken down into tasks, performed by
separate workers or processes - Processes interact by exchanging information
- What do we basically need?
- The ability to start the tasks
- A way for them to communicate
2Communication
- Cooperative
- All parties agree to transfer data
- Message passing is cooperative
- Data must be explicitly sent and received
- Any change in the receivers memory is made with
the receivers participation - One-sided
- One worker performs the transfer of data
- Data can be accessed without waiting for another
process
3What is MPI?
- A message passing library specification
- Message-passing model
- Not a compiler specification (i.e. not a
language) - Not a specific product
- Designed for parallel computers, clusters, and
heterogeneous networks
4The MPI Process
- Development began in early 1992
- Open process/Broad participation
- IBM,Intel, TMC, Meiko, Cray, Convex, Ncube
- PVM, p4, Express, Linda,
- Laboratories, Universities, Government
- Final version of draft in May 1994
- Public and vendor implementations are now widely
available
5Point to Point Communication
- A message is sent from a sender to a receiver
- There are several variations on how the sending
of a message can interact with the program
6Synchronous
- A synchronous communication does not complete
until the message has been received - A FAX or registered mail
7Asynchronous
- An asynchronous communication completes as soon
as the message is on the way. - A post card or email
8Blocking and Non-blocking
- Blocking operations only return when the
operation has been completed - Normal FAX machines
- Non-blocking operations return right away and
allow the program to do other work - Receiving a FAX
9Collective Communications
- Point-to-point communications involve pairs of
processes. - Many message passing systems provide operations
which allow larger numbers of processes to
participate
10Types of Collective Transfers
- Barrier
- Synchronizes processors
- No data is exchanged but the barrier blocks until
all processes have called the barrier routine - Broadcast (sometimes multicast)
- A broadcast is a one-to-many communication
- One processor sends one message to several
destinations - Reduction
- Often useful in a many-to-one communication
11Whats in a Message?
- An MPI message is an array of elements of a
particular MPI datatype - All MPI messages are typed
- The type of the contents must be specified in
both the send and the receive
12Basic C Datatypes in MPI
MPI Datatype C datatype
MPI_CHAR signed char
MPI_SHORT signed short int
MPI_INT signed int
MPI_LONG signed long int
MPI_UNSIGNED_CHAR unsigned char
MPI_UNSIGNED_SHORT unsigned short int
MPI_UNSIGNED_INT unsigned int
MPI_UNSIGNED_LONG unsigned long int
MPI_FLOAT float
MPI_DOUBLE double
MPI_LONG_DOUBLE long double
MPI_BYTE
MPI_PACKED
13MPI Handles
- MPI maintains internal data-structures which are
referenced by the user through handles - Handles can be returned by and passed to MPI
procedures - Handles can be copied by the usual assignment
operation
14MPI Errors
- MPI routines return an int that can contain an
error code - The default action on the detection of an error
is to cause the parallel operation to abort - The default can be changed to return an error code
15Initializing MPI
- The first MPI routine called in any MPI program
must be the initialization routine MPI_INIT - MPI_INIT is called once by every process, before
any other MPI routines - int mpi_Init( int argc, char argv )
16Skeleton MPI Program
include ltmpi.hgt main( int argc, char argv )
MPI_Init( argc, argv ) / main part
of the program / MPI_Finalize()
17Communicators
- A communicator handle defines which processes a
particular command will apply to - All MPI communication calls take a communicator
handle as a parameter, which is effectively the
context in which the communication will take
place - MPI_INIT defines a communicator called
MPI_COMM_WORLD for each process that calls it
18Communicators
- Every communicator contains a group which is a
list of processes - The processes are ordered and numbered
consecutively from 0. - The number of each process is known as its rank
- The rank identifies each process within the
communicator - The group of MPI_COMM_WORLD is the set of all MPI
processes
19Point-to-point Communication
- Always involves exactly two processes
- The destination is identified by its rank within
the communicator - There are four communication modes provided by
MPI (these modes refer to sending not receiving) - Buffered
- Synchronous
- Standard
- Ready
20Standard Send
- When using standard-mode send
- It is up to MPI to decide whether outgoing
messages will be buffered. - Completes once the message has been sent, which
may or may not imply that the massage has arrived
at its destination - Can be started whether or not a matching receive
has been posted. It may complete before a
matching receive is posted. - Has non-local completion semantics, since
successful completion of the send operation may
depend on the occurrence of a matching receive.
21Standard Send
- MPI_Send( buf, count, datatype, dest, tag, comm )
- Where
- buf is the address of the data to be sent
- count is the number of elements of the MPI
datatype which buf contains - datatype is the MPI datatype
- dest is the destination process for the message.
This is specified by the rank of the destination
within the group associated with the communicator
comm - tag is a marker used by the sender to distinguish
between different types of messages - comm is the communicator shared by the sender and
the receiver
22Synchronous Send
- MPI_Ssend( buf, count, datatype, dest, tag, comm
) - can be started whether or not a matching receive
was posted - will complete successfully only if a matching
receive is posted, and the receive operation has
started to receive the message sent by the
synchronous send. - provides synchronous communication semantics a
communication does not complete at either end
before both processes rendezvous at the
communication. - has non-local completion semantics.
23Buffered Send
- A buffered-mode send
- Can be started whether or not a matching receive
has been posted. It may complete before a
matching receive is posted. - Has local completion semantics its completion
does not depend on the occurrence of a matching
receive. - In order to complete the operation, it may be
necessary to buffer the outgoing message locally.
For that purpose, buffer space is provided by the
application.
24Ready Mode Send
- A ready-mode send
- completes immediately
- may be started only if the matching receive has
already been posted. - has the same semantics as a standard-mode send.
- saves on overhead by avoiding handshaking and
buffering