COSC 6374 Parallel Computation. Communication Performance Modeling. Edgar Gabriel Fall Motivation

Transcription

1 COSC 6374 Parallel Cmputatin Cmmunicatin Perfrmance Mdeling Edgar Gabriel Fall 2015 Mtivatin Can we estimate the csts fr a parallel cde in rder t Evaluate quantitative and qualitative differences between different implementatin alternatives Understand the parameters effecting the perfrmance f the applicatin Understanding relevant hardware characteristics Restrictins: Any analytical mdel can nt replace real measurements since parallel systems are t cmplex and unpredictable 1

2 Hw t mdel cllective peratins? Eg MPI_Bcast: strngly depending n the algrithm used t implement the peratin One prcess (rt prcess) distributes the same data items t all members within a prcess grup (cmmunicatr) Linear Algrithm: the rt prcess sends ne message t each prcess in the cmmunicatr if (rank == rt ) { fr (i=0; i<size; i++ ) if ( i!= rt ) MPI_Send (buf, cnt, dat, i, TAG, cmm); } else MPI_Recv (buf, cnt, dat, rt, TAG, cmm, &stat); Linear Algrithm (I) 1 Hckney s Mdel: t( s) l s b s: message size l: latency b: bandwidth Estimate f the executin time accrding t Hckney s mdel fr p prcesses: 1 ( s, p) ( l s)*( p 1) b t (4:1) 2

3 Linear Algrithm (II) Using nn-blcking peratins: if (rank == rt ) { fr (i=0; i<size; i++ ) MPI_Isend (buf, cnt, dat, i, TAG, cmm, &req[i]); } MPI_Recv (buf, cnt, dat, rt, TAG, cmm, &stat); if (rank == rt ) { MPI_Waitall ( size, req, statuses); } Frmula (4:1) is nw arbitrarily wrng Several cmmunicatins simultaneusly nging Maximum (ptimal) number f messages depending n message size and netwrk parameters Binary and Binmial Trees Number f messages increase with every iteratin netwrk saturated starting frm a certain number f messages message segmenting can imprve the perfrmance as well 3

4 Chain Algrithms Segment a message and pass them frm ne prcess t anther Perfrms very well fr very large messages n n n n n eg k=5 k-chain Algrithm

5 Hckney s Mdel t(s) = l + s/b (14:1) l: latency f the netwrk b: bandwidth f the netwrk Hw can we determine the latency and the bandwidth? Ping-png benchmark: prcess A sends a message t prcess B, prcess B sends message back Advantage: des nt require synchrnized clcks between A and B Disadvantage: assumes symmetric cmmunicatin perfrmance ( csts (A->B) == csts (B->A) T determine latency: execute ping-png benchmark fr cnt=0 Ping png benchmark cmm = MPI_COMM_WORLD; fr (i=1; i< MAX_MSG_LEN; i*=2 ) { t1 = MPI_Wtime(); fr ( j=0; j<max_measurements; j++ ) { if ( rank == 0 ) { MPI_Send (buf, i, MPI_INT, 1, 1, cmm); MPI_Recv (buf, i, MPI_INT, 1, 1, cmm, &status); } else if ( rank == 1 ) { MPI_Recv (buf, i, MPI_INT, 0, 1, cmm, &status); MPI_Send (buf, i, MPI_INT, 0, 1, cmm ); } } t2 = MPI_Wtime(); if ( rank == 0 ) { printf( Msg len: %d avg exec%lf bandw %d \n, i, (t2-t1)/(2*max_measurements), i*sizef(int)/((t2-t1)/(2*max_measurements)); } } 14 5

6 Ping-png benchmark (II) Ping-png benchmark (II) T determine bandwidth: have t determine the saturatin pint Required message length des depend n the netwrk bandwidth 16 6

7 LgP Mdel published by Culler et al Parameters: L: upper bund n the latency Hardware parameter : verhead, defined as the length f the time that a prcess is engaged in the transmissin r receptin f a message During this time, the prcess can nt perfrm ther peratins Sftware parameter g: gap, defined as the minimum time interval between cnsecutive message transmissins r receptins The reciprcal time f g crrespnds t the per-prcess cmmunicatin bandwidth Hardware parameter P: number f prcessrs Sender Start sending LgP (II) Message enters netwrk Receiver L Message leaves netwrk End receiving Csts fr sending a messages: t L 2 (19:1) 7

8 Start sending g Sender g LgP (III) Receiver L End receiving Csts fr sending tw messages: t L g 2 (20:1) LgP(IV) Please nte: Latency in the LgP mdel is different than the latency in the Hckney mdel Hardware latency (capital L) vs end-t-end latency (l) The latency in Hckney s mdel includes the verhead f the LgP Mdel In the frmula (20:1), we assumed that < g which is typically crrect The frmulas shuld hwever be instead (21:1) t L max( g, ) 2 8

9 LgP(III) LgP assumes, that any large message can be decmpsed t a series f shrt messages eg sending a message f k bytes takes k / w 1)*max( g, L t ) ( (22:1) with w being the size f the netwrk package in bytes fr which LgP still hlds LgP assumes, that the verhead is equal fr the sender and the receiver side Mre fine grained appraches use different values, eg s and r LgGP Extensin f LgP taking int accunt, that large message can ften be transferred mre efficiently than what LgP predicts, due t special hardware supprt Additinal parameter: G: Gap per bytes fr lng messages Sending a k byte message with LgGP: cycles until the first byte enters the netwrk G cycles fr each subsequent byte cycles n the receiver side t k 1) G L ( (23:1) 9

10 LgGP g Sender GGG GGG Receiver L End receiving Csts fr sending tw k-byte messages: t ( k 1) G g ( k 1) G L 2( k 1) G g L 2 (24:1) PLgP Extensin f the PLgP mdel making the parameters g, s and r dependent n the message length m g(m), s (m) and r (m) Latency L is cnsidered t be an end-t-end latency LgP/LgGP PLgP L L+g(1)- s (1)- r (1) ( s (1)+ r (1))/2 g G P g(1) g(m)/m, fr sufficiently large m P 10

11 PLgP(III) Hw can we determine the parameters f LgP, LgGP and PLgP Design a series f measurements Determine LgP/LgGP/PLgP mdels/equatins fr each series Equatins have t lead t a a set f linearly independent equatins N equatins fr n parameters Perfrm measurements Determine the parameters frm the equatins by slving the set f linearly independent equatins PLgP(IV) Test 1: Send n very small messages (m=0) and wait fr a single acknwledgement Measure the Time t send n messages f length 0: n*g(0) (7) (8) RundTripTime (RTT n ) = 2(L+g(0)) Test 2: Send a message f length m and wait fr an ack f length 0 Measure the Time t send a message f length m : s (m) (9) RTT(m) = L+g(m)+L+g(0) (10) Test 3: send a message f length 0, wait fr Δ>RTT(m) and receive than a message f length m Since Δ>RTT(m) we knw that the message is available, and thus we really measure r (m) (11) 11

12 Example: linear bradcast g 0 GGG GGG 1 L 2 L Example: linear bradcast Executin time accrding t LgP: First message takes cycles t push int the netwrk All subsequent messages take g cycles The last message takes L+ cycles t be received t(p) = +(P-2)g+L+ Executin time accrding t LgGP: First message takes +(k-1)g cycles Subsequent messages take g+(k-1)g cycles Last message takes L+ cycles t be received t(k,p) = +(P-2)g+(P-1)(k-1)G+L+ 12

13 Example: nn-segmented chain bradcast 0 GGG 1 L GGG 2 L Example: nn-segmented chain bradcast Executin time accrding t LgP: Rt prcess takes cycles t push the message int the netwrk A prcess takes L+ cycles t receive the message and cycles t push the message int the netwrk Last prcess takes L+ cycles t receive the message t(p)=+(p-2)(l+2)+l+ = (P-1)(L + 2) Similarly fr LgGP: t(k,p)=+(k-1)g+(p-2)(l+2+(k-1)g)+l+ = (P-1)(L+2+(k-1)G) 13

14 Hw cmmunicatin really wrks Tw prtcls usually used internally by mst MPI libraries: Eager prtcl: message is sent immediately t the receiver, withut waiting fr the accrding receive t be psted Usually used fr shrt messages ( ~32KB 128KB depending n netwrk intercnnect) If the message has t be buffered n the receiver side, it is cnsidered acceptable ( since it is shrt) Rendezvus prtcl: Avids having t buffer large messages n the receiver prcess ( unexpected messages) Send a header t receiver Wait fr an acknwledgment receive peratin has started Send message data LgGPS LgP, LgGP, PLgP d nt handle prtcl switch between eager and rendezvus prtcl LGGPS: add switching pint between eager and rendezvus prtcl L: upper bund n the hardware latency : sftware verhead Separated int a cnstant part and a byte-dependent part as well as int send and receive verhead + k O r r + k O s g: gap between messages G: gap per byte, distinguished between shrt Gap G s and lng Gap G l S: threshld fr eager vs rendezvus message lengths 14

15 LgGPS shrt messages Sender + ko s G s G s G s Receiver L + k O r Csts fr sending ne shrt message f k < S bytes (uses eager prtcl): t = + ko s + L + k G s + + ko r = 2 + k (O s + O r + G s ) + L LgGPS lng messages Sender + ko s SG s (k-s)g l Receiver L L + k O r t d Csts fr sending ne lng message f k > S bytes (rendezvus prtcl): t = + max(l+, t d ) + + L ko s + L + S G s + (k-s) G l + + ko r request t send acknwledgment actual data transfer t d : time that the receiver is delayed If t d = 0 : t = 6 + 3L + ko s + S G s + (k-s) G l + ko r 15

16 MPI Cmmunicatin csts in LgGPS MPI_Send: k<= S: + ko s k > S : + max(l+, t d ) + + L ko s MPI_Isend: MPI_Recv: k <= S: max{0, +ko s +L+kG s -t d } + + ko r k > S : max{0, +L-t d } L+kO s +SG s +(k-s)g+ko r MPI_Irecv: 16