Basic Communication Operations (cont.) Alexandre David B2-206

Transcription

1 Basic Communication Oerations (cont.) Alexandre David B-06

2 Today Scatter and Gather (4.4). All-to-All Personalized Communication (4.5). Circular Shift (4.6). Imroving the Seed of Some Communication Oerations (4.7) Alexandre David, MVP'06

3 Scatter and Gather Scatter: A node sends a unique message to every other node unique er node. Gather: Dual oeration but the target node does not combine the messages into one. M Scatter M M Gather 0 M 0 M 0 0 M Alexandre David, MVP'06 3

4 Alexandre David, MVP'06 4

5 Alexandre David, MVP'06 5 Cost Analysis Number of stes: log. Size transferred: m/, m/4,,m. Geometric sum Cost T=t s log+t w m(-). ) ( ) ( ) ( log n n n n n = = = = = =

6 All-to-All Personalized Communication Each node sends a distinct message to every other node. M 0, M 0, M, M, M,0 M, M, M, M,0 M, M, M, M 0,0 M,0 M,0 M 0,0 M 0, M 0, Alexandre David, MVP'06 6

7 Examle: Transose Alexandre David, MVP'06 7

8 Total Exchange on a Ring Alexandre David, MVP'06 8

9 Total Exchange on a Ring Alexandre David, MVP'06 9

10 Cost Analysis Number of stes: -. Size transmitted: m(-),m(-),m. T = t ( ) + s i= it w m = ( t s + t w m / )( ) Otimal Alexandre David, MVP'06 0

11 Total Exchange on a Mesh Alexandre David, MVP'06

12 Total Exchange on a Mesh Alexandre David, MVP'06

13 Total Exchange on a Mesh Alexandre David, MVP'06 3

14 Cost Analysis Substitute by (number of nodes er dimension). Substitute message size m by m. Cost is the same for each dimension. T=(t s +t w m)( -) Alexandre David, MVP'06 4

15 Total Exchange on a Hyercube Generalize the mesh algorithm to log stes = number of dimensions, with nodes er dimension. Same rocedure as all-to-all broadcast Alexandre David, MVP'06 5

16 Total Exchange on a Hyercube Alexandre David, MVP'06 6

17 Total Exchange on a Hyercube Alexandre David, MVP'06 7

18 Total Exchange on a Hyercube Alexandre David, MVP'06 8

19 Total Exchange on a Hyercube Alexandre David, MVP'06 9

20 Cost Analysis Number of stes: log. Size transmitted er ste: m/. Cost: T=(t s +t w m/) log. Otimal? NO Each node sends and receives m(-) words. Average distance = ( log)/. Total traffic = *m(-)* log/. Number of links = log/. Time lower bound = t w m(-) Alexandre David, MVP'06 0

21 An Otimal Algorithm Have every air of nodes communicate directly with each other - communication stes but without congestion. At j th ste node i communicates with node (i xor j) with E-cube routing Alexandre David, MVP'06

22 Total Exchange on a Hyercube Alexandre David, MVP'06

23 Total Exchange on a Hyercube Alexandre David, MVP'06 3

24 Total Exchange on a Hyercube Alexandre David, MVP'06 4

25 Total Exchange on a Hyercube Alexandre David, MVP'06 5

26 Total Exchange on a Hyercube Alexandre David, MVP'06 6

27 Total Exchange on a Hyercube Etc Alexandre David, MVP'06 7

28 Cost Analysis Remark: Transmit less, only what is needed, but more stes. Number of stes: -. Transmission: size m er ste. Cost: T=(t s +t w m)(-). Comared witht=(t s +t w m/) log. Previous algorithm better for small messages Alexandre David, MVP'06 8

29 Circular Shift It s a articular ermutation. Circular q-shift: Node i sends data to node (i+q) mod (in a set of nodes). Useful in some matrix oerations and attern matching. Ring: intuitive algorithm in min{q,-q} neighbor to neighbor communication stes. Why? Alexandre David, MVP'06 9

30 Circular 5-shift on a mesh. q mod on rows comensate q / on colums Alexandre David, MVP'06 30

31 Circular Shift on a Hyercube Ma a linear array with d nodes onto a hyercube of dimension d. Exand q shift as a sum of owers of (e.g. 5-shift = 0 + ). Perform the decomosed shifts. Use bi-directional links for forward (shift itself) and backward (rotation art) log stes Alexandre David, MVP'06 3

32 Or better: Direct E-cube routing. q-shifts on a 8-node hyercube Alexandre David, MVP'06 3

33 Imroving Performance So far messages of size m were not slit. If we slit them into arts: One-to-all broadcast = scatter + all-to-all broadcast of messages of size m/. All-to-one reduction = all-to-all reduce + scatter of messages of size m/. All-reduce = all-to-all reduction + all-to-all broadcast of messages of size m/ Alexandre David, MVP'06 33