Pop quiz

1
Pop quiz
Question Which one is a 2-D hypercube ?
11
01
10
11
(1)
(2)
00
10
00
01
(3)
00
11
01
10
2
Basic communication operations on a hypercube

• One-to-all broadcast
• All-to-all broadcast
• All-Reduce
• All-to-all personalized communication

3
Hypercube one-to-all broadcast
10
11
(0)
The sender
01
00
4
Step i each node with a message (0) sends (0)
to the node whose bit differs at the i th
position
10
11
Step 1
(0)
The sender
01
00
1st position
5
After step 1
10
11
(0)
(0)
01
00
The sender
ts twm time
6
Step 2 each node with (0) sends it to the node
whose bit differs at the 2nd position
10
11
ts twm time
(0)
(0)
01
00
2nd position
7
After step 2
(0)
(0)
10
11
(0)
(0)
The sender
01
00
8

• Each step takes ts twm time
• The number of steps log p
• The total time (ts twm) log p

9
Hypercube all-to-all broadcast
10
11
(2)
(3)
message
00
01
(1)
(0)
10
Step i each node sends its message to the node
whose bit differs at the ith position
1st position
10
11
(2)
(3)
Step 1
00
01
(0)
(1)
11
After step 1
10
11
(2,3)
(2,3)
ts twm time
00
01
(0,1)
(0,1)
Note ) links are bidirectional
(
12
Step 1

• The size of message m
• Time ts twm

13
Step 2 each node sends its message to the node
whose bit differs at the 2nd position
10
11
(2,3)
(2,3)
00
01
(0,1)
(0,1)
14
After step 2
10
11
(0,1,2,3)
(0,1,2,3)
00
01
(0,1,2,3)
(0,1,2,3)
15

• Step 2
• The size of message 2m
• Time ts tw(2m)
• Step i
• The size of message 2i-1m
• Time ts tw(2i-1m)

16

• Step i takes ts tw(2i-1m) time
• The number of steps log p
• The total time (ts tw(20m))
• (ts tw(21m)) (ts tw(2logp-1m))
• tslog p twm(2021 2logp-1 )
• tslog p twm(p-1)

17
All-Reduce

• Associative operation addition
• Use a communication pattern similar to that used
  in all-to-all broadcast

18
10
11
(2)
(3)
message
00
01
(1)
(0)
19
Step 1
(2)
10
11
(2)
(3)
(3)
(0)
00
01
(0)
(1)
(1)
20
After step 1
10
11
(23)
(23)
00
01
(01)
(01)
21
Time for Step 1

• The size of message m
• Time ts twm

22
Step 2
10
11
(23)
(23)
(1)
(5)
(1)
(5)
00
01
(01)
(01)
23
After step 2
10
11
(0123)
(0123)
00
01
(0123)
(0123)
24

• Step 2
• The size of message m
• Time ts twm
• Step i
• The size of message m
• Time ts twm

25

• Each step takes ts twm time
• The number of steps log p
• The total time (ts twm)logp

26
All-to-all personalized communication

• Step i each node (j) sends a message to the
  node (j XOR i )
• example ) step 3(11) on 2-D hypercube
• 00 ? 00 XOR 11 11
• 01 ? 01 XOR 11 10
• 10 ? 10 XOR 11 01
• 11 ? 11 XOR 11 00

27

• If a node needs to send a message through a path,
  then we use E-cube routing scheme.
• E-cube routing scheme
• (1) find the positions of different bits between
  the sender and the receiver
• (2) change bits from one in the lower position to
  one in the higher position

28
Example of E-cube routing

• The sender 1100, the receiver 0101
• The positions of different bits 1st, 4th bits
  differ
• 1100 ? 1101 ? 0101

change 1st bit
change 4th bit
29
2(0) 2(1) 2(2) 2(3)
3(0) 3(1) 3(2) 3(3)
10
11
1(0) 1(1) 1(2) 1(3)
0(0) 0(1) 0(2) 0(3)
00
01
30
Step 1 (01)

• The sender ? The receiver
• 00 ? 00 XOR 01 01
• 01 ? 01 XOR 01 00
• 10 ? 10 XOR 01 11
• 11 ? 11 XOR 01 10

31
Step 1
2(0) 2(1) 2(2) 2(3)
3(0) 3(1) 3(2) 3(3)
2(3)
10
11
3(2)
1(0) 1(1) 1(2) 1(3)
0(0) 0(1) 0(2) 0(3)
0(1)
00
01
1(0)
32
After step 1
2(0) 2(1) 2(2) 3(2)
3(0) 3(1) 2(3) 3(3)
2(3)
10
11
3(2)
0(1) 1(1) 1(2) 1(3)
0(0) 1(0) 0(2) 0(3)
0(1)
00
01
1(0)
33
Step 2 ( 10)

• The sender ? The receiver
• 00 ? 00 XOR 10 10
• 01 ? 01 XOR 10 11
• 10 ? 10 XOR 10 00
• 11 ? 11 XOR 10 01

34
Step 2
3(0) 3(1) 2(3) 3(3)
2(0) 2(1) 2(2) 3(2)
10
11
3(1)
1(3)
2(0)
0(2)
0(1) 1(1) 1(2) 1(3)
0(0) 1(0) 0(2) 0(3)
00
01
35
After Step 2
3(0) 1(3) 2(3) 3(3)
0(2) 2(1) 2(2) 3(2)
10
11
3(1)
1(3)
2(0)
0(2)
0(1) 1(1) 1(2) 3(1)
0(0) 1(0) 2(0) 0(3)
00
01
36
Step 3 ( 11)

• The sender ? The receiver
• 00 ? 00 XOR 11 11
• 01 ? 01 XOR 11 10
• 10 ? 10 XOR 11 01
• 11 ? 11 XOR 11 00

37

• The positions of different bits between the
  sender and the receiver 1st, 2nd bits

• The sender ? The receiver E-cube routing
• 00 ? 11 00 ? 01 ? 11
• 01 ? 10 01 ? 00 ? 10
• 10 ? 01 10 ? 11 ? 01
• 11 ? 00 11 ? 10 ? 00

38
Step 3
3(0) 1(3) 2(3) 3(3)
0(2) 2(1) 2(2) 3(2)
2(1)
10
11
3(0)
0(3)
0(1) 1(1) 1(2) 3(1)
0(0) 1(0) 2(0) 0(3)
00
01
1(2)
39
After step 3
0(3) 1(3) 2(3) 3(3)
0(2) 1(2) 2(2) 3(2)
2(1)
10
11
3(0)
0(3)
0(1) 1(1) 2(1) 3(1)
0(0) 1(0) 2(0) 3(0)
00
01
1(2)
40

• Each step takes ts twm time
• The number of steps p-1
• The total time (ts twm)(p-1)

41
Communication times on a hypercube

• One-to-all broadcast / all-to-one reduction
• ts logp twmlogp
• All-to-all broadcast / all-to-all reduction
• ts logp twm(p-1)
• All-to-all personalized communication
• ts (p-1) twm(p-1)

42
Communication times on ring, mesh, hypercube