Communication operations

1
Communication operations

• Efficient Parallel Algorithms
• COMP308

2
Communication time

• Communication requires 3 costs
• 1. Static start up time (ts)
• It is the time required to handle a message at
  the sending processor
• 2. Per-hop time (th) with l the Links that the
  message passes
• It is take a finite amount of time to reach the
  next processor in its path after a message leaves
  a processor.
• 3. Per-word transfer time (tw) with m the
  bytes
• If the channel bandwidth is r words per second,
  then each word takes time tw1/r to traverse the
  link.

3
There are 2 main communication schemes
4
store and forward vs cut-through

• In store and forward routing, when a message is
  traversing a path with multiple links, each
  intermediate node on the path forwards the
  message to the next node after it has received.
• In cut-through routing an intermediate nodes
  does not wait for the entire message to arrive
  before forwarding it.
• A tracer is first sent from the source to the
  designation node to establish a connection.
• Once a connection has been established, the flits
  are sent one after the other. All flits follows
  the same path in a dovetailed fashion.
• As soon as a flit is received at an intermediate
  node, the flit is passed on to the next node.

5
One to All Broadcast

• Initially, only the source processor has the data
  of size m that need to be broadcast. At the end
  of the termination of the procedure, there are P
  copies of the initial data, one residing at each
  processor.

6
Broadcast on ring (Store and Forward)
If the sender sends the messages consecutively to
the p-1 other processors, it takes p-1 steps. By
optimisation, we can reduce this to p/2
steps. Eg. a 8-processor ring requires 4 steps
7
NS diagram for broadcast on ring
8
Ring network, Cut-Through routing

• With cut-through routing, messages can be sent
  faster to nodes that are multiple hops away in
  the network. By using this, we send the message
  first to the outermost node.

In general, in a p-processor ring the source
processor first sends the data to the processor
at distance p/2, then both processors sends the
message to the processors at distance of p/4 in
the same direction, then to p/8, etc.
9
Broadcast on mesh (Store and Forward)
Most of the optimised communication algorithms on
a mesh are simple extensions of their ring
counterparts, by consecutively applying the ring
algorithm on each dimension of the mesh.
10
Hypercube

• The regular binary structure of the hypercube
  plays an important role in optimising
  communication.
• Here, a broadcast is performed by sending the
  message along each dimension at each step. This
  results in log p or d steps.
• It can be proved easily that log p is the minimal
  number of steps for every network.

11
Hypercube

• Important properties of the networks
• Small degree,
• Small diameter,
• Regular recursive structure,
• Easy way to embed trees, etc
• Hypercube two nodes connected if they are
  differ precisely on one bit

12
Hypercube two nodes connected if they are
differ precisely on one bit
13
(No Transcript)
14
(No Transcript)
15
(No Transcript)
16
Broadcast on hypercube (SF)
17
Broadcast on ring (Cut-Through )
18
Broadcast on mesh (C-T)
19
Broadcast on binary tree (C-T)
20
Gossiping
All-to-All Communication
21
Gossiping on Ring (Store and Forward)
22
Gossiping on Mesh (Store and Forward)
23
Gossiping on Hypercube (SF)
24
Gossiping on Ring (and Mesh)Cut-Through Routing

• Each process sends m(p-1) words of data because
  it has an m-word packet for every other processor
• The average distance that an m word packet
  travels is
• Since there are p processors, each performing the
  same type of communication, the total traffic on
  the network is
• The total number of communication channels in the
  network to share this load is p.

Hence this procedure cannot be improved by using
CT routing
25
Gossiping on Hypercube (CT routing)