Distributed Algorithms for the Operator Placement Problem

Transcription

1 TZIRITAS ET AL.: DISTRIBUTED ALGORITHMS FOR THE OPERATOR PLACEMET PROBLEM Dstrbuted Algorthms for the Operator Placement Problem kos Tzrtas, Thanass Loukopoulos, Samee U. Khan, Senor Member, IEEE, Cheng-Zhong Xu, Fellow, IEEE Abstract Operator placement plays a key role n reducng the aggregate network overhead wthn a wreless sensor network (WS) to extend battery lfe and the longevty of the network. Consequently, optmal algorthms for the operator placement problem (OPP) are of paramount mportance to WS performance. Unfortunately, the OPP becomes P-complete when capacty constrants on the WS nodes are taken nto account. There are many algorthms n the lterature that tackle the operator placement problem; however, most of them coner tree-structured query graphs wthout lmtatons regardng the operators hosted by the WS nodes. Therefore, there s a need to propose sophstcated approaches such that the problem s solved n an effectve fashon. In ths paper, we propose a fully dstrbuted approach that takes nto account the WS node capacty constrants. The proposed approach s thoroughly evaluated through smulatons and the results reveal that the proposed approach s superor to several state-of-the-art algorthms, such as DRA, DBA, MCFA, dfs, and GRAL* found n the lterature. Index Terms operator placement, n-network processng, agent placement, dstrbuted algorthms O I. ITRODUCTIO PERATOR placement wthn the wreless sensor networks (WSs) has played a key role for mnmzng the aggregate network overhead ncurred by the nterdependences between the operators (applcaton components) and the sensory nodes. One of the most challengng ssues for the hardware engneers, protocol desgners, and software engneers s to maxmze the lfetme and the avalablty of the applcatons runnng on a WS. Gven the relatvely hgh energy cost of wreless data transmsson and recepton compared to data processng [2], reducng the frequency of communcaton and the amount of data that travels through the network s of prmary mportance. Tacklng ths challenge, we dscuss algorthms that drve the mgraton of operators. Tzrtas and C.-Z. Xu are wth the Shenzhen Insttutes of Advanced Technology, Chnese Academy of Scences, Shenzhen, Chna, E- mal: {nkolaos, cz.xu}@sat.ac.cn T. Loukopoulos s wth the Computer Scence and Bomedcal Informatcs Dept., Unversty of Thessaly, Lama, Greece, E-mal: luke@db.uth.gr. S.U. Khan s wth the Electrcal and Computer Engneerng Department, orth Dakota State Unversty, Fargo, D E-mal: same.khan@ndsu.edu C.-Z. Xu s wth the Electrcal and Computer Engneerng Department, Wayne State Unversty, Detrot, MI E-mal: czxu@wayne.edu. between the WS nodes to reduce the amount of network traffc due to the applcaton-level communcaton. The ablty to perform operator mgratons and the overhead of such operatons has receved a lot of attenton, snce the early days of WSs [4]. Indeed, performng operator mgratons brngs a lot of benefts not only to the system optmzaton but also to the clents. Specfcally, operator mgratons play a key role n several optmzatons, most notably: (a) network overhead that drectly affects the energy consumpton and (b) response tme experenced by the clents [3]. In ths paper, we study the operator placement problem (OPP) to mnmze the aggregate network overhead wthn WSs wth tree-based routng (as n ZgBee networks [0]). In prevous works, the problem has been solved n an optmal way when: (a) the network structure was a tree and (b) there were no capacty constrants on the WS nodes. Unfortunately, the optmalty does not hold when the memory or computng capacty of nodes s lmted, whch s what happens n real WSs. Specfcally, the related decson problem becomes P-complete, wth the reducton beng based to the 0- knapsack problem. Therefore, specal treatment s needed when solvng the problem n the capactated case. Cyber physcal socal systems (CPSS) are an attempt to unfy cyber and socal world wth the physcal world, through networkng. Examples of CPSS are smart ctes, smart ntellgent transportaton systems, smart grd, etc. Because WSs are the medum for the access to physcal world, they play a crucal role for the realzaton of CPSS. Consequently, the longevty of WSs s vtal to the sustanablty of CPSS. Below we gve an example to show the connecton between the operator placement problem n WSs and CPSS. Coner a socal network (such as facebook) whch sends nformaton towards applcatons nterested n specfc human behavors. Assume also a network of WSs montorng the physcal world. Applcatons send queres to WSs to get nformaton to make decsons such that they reach ther goals. The applcaton queres and ther overlap play the role of operator communcaton graph. The specfc contrbutons of ths paper are: (a) we propose an operator placement algorthm for general WS communcaton graphs takng nto coneraton capacty constrants on WS nodes; (b) we provde thorough proofs about the P-Completeness and convergence ssues of the

2 TZIRITAS ET AL.: DISTRIBUTED ALGORITHMS FOR THE OPERATOR PLACEMET PROBLEM 2 proposed approach; and (c) we present an expermental evaluaton. The remander of ths paper s organzed as follows. Secton II descrbes the related work. In Secton III, we ntroduce the system model and problem formulaton. Secton IV detals an algorthm for the un-capactated case, whle Secton V dscusses ts extenson to handle capacty constrants. Implementaton and complexty ssues are detaled on Secton VI, whle P-completeness and convergence ssues on Secton VII. Secton VIII descrbes the expermental evaluaton. Fnally, Secton IX concludes the paper. II. RELATED WORK When conerng the OPP n the context of agent-based embedded mddleware, one can fnd many mddleware/platform ntatves supportng moble code and mgratons [], [2], [6], [5]. The POBICOS [5] s an agentbased platform that s the closest to our work. It supports the automatc deployment and executon of dstrbuted montorng-and-control applcatons on top of a WS. A POBICOS applcaton s structured as a set of cooperatng moble components, called agents that are organzed as a tree. In [6] and [9] an optmal dstrbuted algorthm, called GRAL, was proposed to tackle the agent-based placement problem for tree-structured applcatons takng nto account the capacty constrants on the nodes. The man dfference between GRAL and our approach s that GRAL targets treestructured applcatons, whle our approach coners generalstructured applcatons. Onlne algorthms for the aforementoned problem are proposed n [8]. The authors n [] address the same problem wth [9]. Two objectves are addressed: (a) balance the energy consumpton wthn the WS, and (b) mnmze the total energy cost. Unfortunately, the proposed solutons of [] are centralzed. The OPP s closely related to the n-network query processng. Accordng to the n-network processng, operators process data that reach a snk node wthn the system. The problem of multple predcate queres s tackled n [4] such that to mnmze the total communcaton overhead wthn the WS network. The aforementoned problem s dfferent wth our problem n that when predcate answers ntersect a reducton takes place. The authors n [3] proposed an optmal fully dstrbuted algorthm, called dfs, for the n-network processng problem. The objectve s to mnmze the total network overhead by decdng the placement of a sngle operator wthn the WS network. The proposed algorthm s based on the Fermat node and t takes a decson to mgrate a sngle operator n a dstrbuted manner. As a result, optmalty cannot be guaranteed n the case of multple operators. The aforementoned problem s also tackled for operators organzed as a bnary tree [8] by approachng the problem n a dstrbuted fashon. The authors prove that ther approach s optmal provded that: (a) the reducton rato (the data forwarded by an operator towards ts parent dvded by the data receved by the correspondng operator from ts chldren) at each operator s no more than ½ and (b), the communcaton rato between each parent operator and ts chldren s equal to or more than 3. In [2], the proposed approach, called MCFA, tackles the problem optmally conerng general treestructured applcaton graphs. Unfortunately, the proposed soluton nvolves control messages targetng all of the nodes wthn the network, renderng t sutable only for the ntal operator placement. It must be noted that all of the aforementoned solutons do not take nto coneraton capacty constrants on nodes. There are also some other nterestng problems that are closely related to our work n the feld of vrtual machne (VM) placement. Instead of operators we can thnk of VMs that communcate wth each other. In terms of the communcaton between an operator and a WS node, we can thnk of the communcaton between a VM and a data source hosted by a server wthn the system. The VM placement problem s addressed n [5], whereby the objectve s to mnmze the network congeston wthn the system. The dynamc servce placement problem s tackled n [22], wth the objectve beng to reduce the hostng cost over tme accordng to both demand and resource prce fluctuaton. In [9] and [20], the authors target the VM placement problem wth ther objectve beng the same wth that of our problem. Unfortunately, all of the algorthms proposed n [9] and [20] are centralzed and therefore non-applcable for dynamc applcatons n WS networks. A fully dstrbuted algorthm s proposed n [4], called DBA, to solve the same problem tackled n ths paper under the context of clouds. DBA works for general-structured graphs and takes nto account capacty constrants on nodes. The dfference wth our approach s that DBA does not coner mgratng group of VMs, resultng n that way n sub-optmal placements. On the other hand, DRA [7] s an optmal fully dstrbuted algorthm workng also for general-structured applcaton graphs and takng nto coneraton capacty constrants on nodes. 2-hop awareness computng of n capacty n n 2 n 3 III. SYSTEM MODEL, OPERATOR MIGRATIO METRIC, AD PROBLEM FORMULATIO A. System Model Let a WS nfrastructure be abstracted as a graph G = (V, E), where each vertex v V represents a WS node n v and each edge e = (s, d) E represents a communcaton lnk between a WS node n s and a WS node n d. The number of hops n the shortest path between n s and n d s defned by h. The notaton W(n s ) denotes the total capacty of n v. From here onwards, when referrng to the capacty of a node, we wll mean the node s computng or processng capacty. An example network graph s depcted n Fg.. n 4 5 n 5 Fgure.etwork graph.

3 TZIRITAS ET AL.: DISTRIBUTED ALGORITHMS FOR THE OPERATOR PLACEMET PROBLEM 3 The -th operator wthn the WS s denoted by o, whle the computng requrements are captured by W(o ). Let and O denote the total number of nodes and operators wthn the WS, respectvely. The placement of operators onto nodes s encoded by a O matrx, named X. More specfcally, let X s be an entry of matrx X, then X s = ff o s hosted by n s, otherwse X s = 0. The communcaton dependences between the operators are encoded by a O O matrx denoted by C, wth C j beng the amount of data (n bytes) sent from o towards o j (C j C j ). o 9 : po 2 :0 o 0 : o 8 : 0 50 pseudo operator representng n 2 o 7 : 00 o : 7 real operator o 2 : 6 o 3 : data exchanged between o and o po : o 4 : o 5 :5 50 computng requrements Fgure 2. Operator communcaton graph 3 po 3 :0 o 6 : To capture the communcaton dependences between the operators and nodes, we extend the number of operators by pseudo operators, wth each of them representng a node. Therefore, the data exchanged between a pseudo operator and a real operator represents the communcatonal demands between the real operator and the node represented by the pseudo operator. Specfcally, o s a real operator ff 0 < O, and a pseudo one ff O < O+. By dong so, we are able to extend the matrx encodng the communcatonal dependences between operators to an (O+) (O+) matrx that ncludes the communcaton dependences between operators but also among operators and nodes. For smplcty, the pseudo operator representng n s be denoted by po s. ote that the computng requrements of a pseudo operator s zero because t does not nvolve any computaton. The placement matrx X s also extended to an (O+) matrx capturng the noton that a pseudo operator po s s hosted by n s. From here onwards, when we refer to the matrces C and X, we must understand that the reference s to the extended matrces of C and X. An example of the operator communcaton graph ncludng the dependences between the operators and nodes s llustrated n Fg. 2. ote that whte rectangles represent real operators, whle black ones represent pseudo operators. B. The Mgraton Metrc Our objectve s to perform operator mgratons to reduce the current WS overhead. Therefore, we need a metrc that can quantfy whether a mgraton contrbutes negatvely or postvely towards the mnmzaton of the total WS overhead. To derve such a metrc, we defne some extra notatons. z The varable q j records whether n z s used for the communcaton between o and o j. Specfcally, q z j = ff n z s used for communcaton between o and o j, else q z j = 0. In a specal case where o and o j are co-located, we have q z j = 0, even f the locaton s n z. The mgraton of o from the host 0 node n s (called source node) to a -hop neghbor n d (called destnaton node) s denoted by M. Let D be an O matrx encodng the new mgraton decsons as: D s = 0 ff n s dd not make a decson of whether to mgrate o or not, otherwse a mgraton decson has been made when D s =. For an operator mgraton M, we must dentfy the followng: Postve load pl : Ths load represents the gan (n terms of the total communcaton overhead) when mgratng o from n s towards ts -hop neghbor n d. Specfcally, such a mgraton wll brng o nearer by one hop to the set of operators (hereafter referred to as A) that use n d (as ether a host or routng node) to communcate wth o gven that the latter s located on n s. It s noteworthy to menton that A conssts of all the pseudo and real operators usng n d to communcate wth o. Therefore, the total communcaton overhead must decrease by an amount equal to the volume of data exchanged between o and the operators belongng to A as depcted n Eq.. Undefned load ul : Ths load concerns the communcaton load between o and ts co-located real operators, on whch n s has not yet made a decson. We term such a load undefned because when a node nvestgates n a concurrent fashon whch operators wll be mgrated to another node, t s unknown a pror whch of the operators wll reman on the current node and whch of them wll mgrate. For example, when n s decdes to concurrently mgrate o and o j to the same destnaton, the network overhead s unburdened wth the load exchanged between them. However, when n s decdes to mgrate only o (wthout o j ), then the network overhead ncreases by the volume of data exchanged between o and o j. The undefned load s captured by Eq. 2. egatve load nl : Ths load represents the network overhead when mgratng o from n s towards ts -hop neghbor n d. Specfcally, when mgratng o from n s to n d the followng take place. Operator o wll dstance tself by one hop from the set of operators (hereafter referred to as B) that do not use the destnaton node n d to communcate wth o, provded that the latter s located on n s. It s noteworthy to menton that B conssts of all the pseudo and real operators usng n d to communcate wth o, except for the real operators hosted by n s that has not made a decson on mgraton. Therefore, the total communcaton overhead must ncrease by an amount equal to the volume of data exchanged between o and the operators belongng to B (see Eq. 3). ote that the undefned load s subtracted from the frst term of Eq. 3 because t already captures the undefned load n ts summaton. Optmstc mgraton beneft OB : Ths metrc quantfes whether a mgraton of o from n s to ts -hop neghbor n d s optmstcally benefcal. The term optmstc stems from the fact that t s assumed that all of the operators communcatng and co-located wth o wll also co-mgrate from n s to n d (the undefned load s conered postve). ote that when OB (Eq. 4) s less than or equal to zero, then the network overhead cannot be ncreased by performng M. Therefore, a decson of performng M s negatve and can be made rrespectve of whether o s co-located operators mgrate or not.

4 TZIRITAS ET AL.: DISTRIBUTED ALGORITHMS FOR THE OPERATOR PLACEMET PROBLEM 4 Pessmstc mgraton beneft PB : Ths metrc quantfes whether M s benefcal n a pessmstc fashon. That s to say that operators co-located wth o do not mgrate. The mgraton M s conered pessmstcally benefcal f the postve load pl s greater than the sum of negatve load nl operators not usng n d to reach o Fgure 3. Evaluaton of M and the undefned load ul. The term pessmstc stems from the fact that n the above calculaton the undefned load s conered negatve. The usefulness of such a metrc s that when PB (Eq. 5) s greater than zero, then the network overhead decreases by performng M by at least PB unts. Therefore, a decson of performng M s postve and can be made rrespectve of whether o s co-located operators mgrate or not. pl ul nl OB PB O+ = (C j + C j ) q j d j= O Eq. = (C j + C j ) X sj ( D sj ) Eq. 2 j= O+ = (C j + C j ) ( q d j ) ul j= = pl = pl nodes usng n b to reach o + ul (ul negatve load: po s nl undefned load: ul postve load: pl 8 o 20 5 operators usng n d to reach o n b n s n d source node destnaton node nodes usng n d to reach o Eq. 3 nl Eq. 4 + nl ) Eq. 5 We set out an example to llustrate the aforementoned defntons n Fg. 3. Coner a mgraton of o from n s to n d. The dotted arrows represent the locaton of the operators. For example, the dotted arrow from o towards n s means that o s located on n s. ow: Postve load pl : The cloud at the rght e of Fg. 3 represents a group of operators that use the destnaton node n d as ether a hostng or routng node to reach o. The amount of data exchanged between o and the aforementoned operators represents the postve load pl that s equal to 20. Undefned load ul : The cloud n the mddle of Fg. 3 represents the group of operators located on n s. The amount of data exchanged between o and the aforementoned processes represents the undefned load equalng to fve. egatve load nl : The cloud at the left e of Fg. 3 represents a group of operators that do not use the destnaton node n d to reach o. The amount of data exchanged between o and the above mentoned operators, as well as po s, represents the negatve load nl equalng to nne (8+). Accordng to Eq. 4, we can also see that OB equals 6 (20+5-9). ote that f the optmstc beneft was negatve, then there could be made a decson of not mgratng o. Accordng to Eq. 5, t can be observed that PB equals to sx (20-9-5). The above means that M wll decrease the total network overhead by at least sx unts. Therefore, the decson for M can be taken. C. Problem Formulaton The total network overhead wthn the system under a placement X s captured by Eq. 6. The constrant that the computng requrements regardng the operators hosted by a node cannot exceed the capacty of the node n queston s represented by Eq. 7. The Eq. 8 captures the operator assgnment requrement of o beng assgned to exactly one node. As shown n Eq. 9, the optmal placement X * s obtaned by mnmzng Eq. 6. O+ O+ U(X) = C j X s X dj h s= d= = M j= Eq. 6 W(n s ) W(o ) X s Eq. 7 = X s = Eq. 8 s= X = argmn (U(X)) X Eq. 9 Formally, the OPP can be stated as: Gven an operator placement scheme X old, an operator communcaton graph, and a network topology, fnd a new placement scheme X new such that the total network overhead wthn the system s mnmzed wthout volatng (a) the capacty of the nodes and (b) the operator assgnment requrements. IV. DISTRIBUTED GRAPH-BASED ALGORITHM (DGA) In ths secton, we present an algorthm that decdes, n a fully dstrbuted fashon, whether (or not) to mgrate a group of operators from the current host node to another node to mnmze the total network overhead as per Eq.. We refer to the algorthm as the Dstrbuted Graph-based Algorthm (DGA). The procedure followed by DGA works for treestructured networks and s splt nto the followng parts: (a) dentfcaton of locally parttoned sub-graphs and (b) makng operator mgraton decsons. o 9 : o 0 : o 7 : 00 o : 7 o 2 : 6 o 3 : 60 Fgure 4. Parttoned sub-graphs on n 5 o 4 : o 5 :5 3 o 6 : A. Identfcaton of Locally Parttoned Sub-graphs In ths phase, each node wthn the graph s requred to dentfy all the locally parttoned sub-graphs of ts hostng

5 TZIRITAS ET AL.: DISTRIBUTED ALGORITHMS FOR THE OPERATOR PLACEMET PROBLEM 5 operators. The above s acheved by groupng the real operators that communcate locally wth each other. In the followng example we show how the dentfcaton of locally parttoned sub-graphs takes place. Coner the applcaton graph shown n Fg. 2. Assume that n hosts all of the operators shown n the graph, except for o 8. It can be observed that there are two locally parttoned sub-graphs hosted by n. The frst sub-graph (left e of Fg. 4) contans o 9 and o 2, whle the second one (rght e of Fg. 4) contans all the rest real operators except o 8. B. Makng Operator Mgraton Decsons When a node wthn the system has dentfed ts locally parttoned sub-graphs, then t s able to proceed wth the dentfcaton of whch (hosted) operators wll mgrate or not (called mgraton decsons or just decsons). The procedure of makng decsons takes place separately for each local sub-graph of the responsble node. Before proceedng wth the descrpton of the procedure, we present some new notatons and equatons. A node n a sub-graph s denoted by t Q and t may consst of more than one operator, wth Q denotng the group of operators contaned n t Q. To avod confuson wth the network nodes, a node n a sub-graph wll be called as sub-graph node. The postve, undefned, and negatve load of the operators contaned n t Q are denoted by pl Q, ul Q, and nl Q, respectvely. The equatons for the aforementoned loads are Eq. 0, Eq., and Eq. 2, respectvely. The mgraton of the whole group of operators contaned n t Q from n s to n d s defned by M Q. The optmstc beneft of M Q s represented by OB Q n Eq. 3. On the other hand, the pessmstc beneft of M Q s defned by PB Q n Eq. 4. Q d = (C j + C j ) q j Eq. 0 pl ul Q j Q Q = (C j + C j ) X sj ( D sj ) Eq. nl Q j Q Q = (C j + C j ) OB Q PB Q Q j Q = (pl nl Q = (pl Q ) ) nl ( q d Q j ) ul Eq. 2 Q + ul Eq. 3 Q ul Eq. 4 ) Algorthm descrpton Each node wthn the WS employs the aforementoned optmstc and pessmstc metrcs as well as the functonalty descrbed n Secton IV.A to make mgraton decsons for the operators t hosts. Each WS node performs the followng sx steps: Step : Choose randomly one of the locally parttoned subgraphs of n s as well as (randomly) one destnaton node among the -hop neghbors of n s to execute the followng steps. ext, proceed to step 2. Step 2: For each sub-graph node t Q n the chosen sub-graph, the followng s performed. We calculate both OB Q and PB Q. (a) In case OB Q s less than (or equal to) zero, then we predcate that M Q ncreases (or do not affect) the total network overhead rrespectve of whether Q s co-located operators mgrate or not. Therefore, the group of operators contaned n Q s decded to reman on n s. (b) In case PB Q s greater than zero, then we predcate that M Q decreases the total network overhead rrespectve of whether Q s co-located operators mgrate or not. Consequently, the decson s to mgrate Q towards n d. (c) If nether of the above holds, then no decson s made. The current step s repeated untl there are no decsons to be made. ext, proceed to step 3. Step 3: For each t Q that a decson has been made (postve or negatve), t s pruned from the sub-graph under coneraton. ote that graph-nodes that are adjacent to the pruned graph-nodes must re-calculate ther optmstc/pessmstc beneft. In case of no prunes, we proceed to step 4. Otherwse, step 2 s performed for the remanng sub-graph nodes. Step 4: For each edge n the sub-graph under coneraton, t s checked the optmstc/pessmstc beneft for mgratng concurrently the sub-graph nodes adjacent to the edge n queston. Specfcally, we check whether wth a consoldaton of two graph-nodes (called canddate consoldaton) there can be a new decson. Accordng to the above: (a) f there s a canddate consoldaton wth postve pessmstc or negatve optmstc beneft, then the respectve consoldaton s performed. In case of postve pessmstc beneft, the operators contaned n the new consoldated graph-node are decded to mgrate towards the destnaton node n d. In case of negatve optmstc beneft, the operators contaned n the new consoldated graph-node are decded to reman on the current node n s. ote that the sub-graph s updated wth a new node representng the consoldated operators. (b) If there s no case such as (a), then the canddate consoldaton wth the bggest pessmstc beneft s chosen to be performed, wthout takng any decson. ote that, f two graph-nodes t A and t B are to be consoldated resultng n t Q, then the latter contans the operators contaned n both t A and t B. The calculaton for the optmstc/pessmstc beneft of t Q s performed through Eq. 7/Eq. 8, respectvely. The current step s repeated untl there are no edges n the chosen sub-graph. Step 5. A decson s taken for the operators that remaned on the graph. ote that there s no edge between those operators. Therefore, t s guaranteed that they have ether postve pessmstc beneft, or optmstc beneft that s less than or equal to zero. Step 6. Step 2 s performed for a new -hop neghbor of n s. If for a gven sub-graph, all of the -hop nodes have been explored, then we proceed wth the next sub-graph. In case all of the pars (sub-graph, -hop neghbor of n s ) have been explored, then the procedure termnates. ote that the procedure resumes, f a new operator mgrates towards n s, or there s a change n the communcaton of the operators hosted n n s. C. Example to Illustrate DGA s Functonalty An example s gven to llustrate the functonalty of DGA. The network graph employed for ths example s shown n Fg.. We coner the rght sub-graph of Fg. 4, by choosng as destnaton node n 2 (step ). The pessmstc/optmstc

6 TZIRITAS ET AL.: DISTRIBUTED ALGORITHMS FOR THE OPERATOR PLACEMET PROBLEM 6-07/07-24/22-63/63 reman o 7 7 : o 2 : 60 o 4 : on n /-46 o : o 3 : o 5 5 :5-4/6-5/9 4/6 3 3 PB 2 OB 2 o 6 : mgrate on n 2-07/07-24/22 o 7 7 : o 2 : o : o 3 : -4/6-5/9-63/57 o 4 : mgrate on n 2 33/59 7 o 2 :, o 4 : 52/68 o :, o 7 : 6 o 3 : -5/9 mgrate on n 2 o 3 : 9/9 Fgure 5. Calculaton of benefts Fgure 6. Prunng graph-nodes Fgure 7. operator consoldatons Fgure 8. Fnal decson benefts of the graph-nodes contaned n Fg. 3 are shown n Fg. 5 (step 2). As we can see, the algorthm can make a fnal decson for o 5 and o 6 (step 2). Partcularly, because the pessmstc beneft of o 5 s postve, the decson s to mgrate o 5 towards n 2. On the other extreme, because the optmstc beneft of o 6 s negatve, the decson s that o 6 remans on n. ext, both o 5 and o 6 are pruned from the graph as shown n Fg. 6, wth o 4 and o 3 re-calculatng ther benefts (step 3). In Fg. 7, we show the consoldaton of o wth o 7, as well as the one of o 2 wth o 4 (step 4). The aforementoned consoldatons are performed after nspectng all of the possble consoldatons between adjacent sub-graph nodes. The results for all of the canddate consoldatons are shown n Table I. As observed, only {o, o 7 } and {o 2, o 4 } result n postve pessmstc beneft. Therefore, both consoldatons are performed (step 4). It must be noted that f there were no postve pessmstc beneft, then the consoldaton wth the hghest pessmstc beneft would be chosen. After performng the aforementoned, the only operator left on the graph s o 3 (Fg. 8). As can be seen ts pessmstc beneft s postve, wth o 3 beng mgrated to n 2 (step 6). ext, the algorthm proceeds n the same fashon wth the sub-graph shown at the left e of Fg. 4 (step 7). Table I. Benefts when consoldatng nodes Consoldaton PB/OB {o, o 3} -44/68 {o, o 7} 52/68 {o 2, o 3} -7/9 {o 2, o 4} 33/59 {o 2, o 7} -7/25 V. DGA O CAPACITATED ODES In the prevous secton, DGA makes decsons under the assumpton that all nodes have nfnte capacty. In ths secton, we extend DGA to capture the case where the capacty of nodes s lmted. Enablng mult-hop operator mgratons becomes an effectve soluton when dealng wth the problem of capacty constrants. To acheve the above, each node needs to ncrease ts hop-awareness from one to k (k > ). A node s conered k-hop aware, when havng the routng nformaton of how to reach any node wthn the system that s at most k hops far away from the node n queston. Before proceedng to the descrpton of the algorthm, we gve frst the requred notatons and equatons. Let f z denote the z th node along the path startng from n s and endng on n d. ote that for z = 0, f z represents n s, whle for z = h, f z represents n d. When conerng a mult-hop mgraton of a group of operators Q from a node n s to a node mpl Q mnl Q MOB Q MPB Q h Q = pl xy z= h Q = nl xy z= n d, the postve load s defned by mpl Q. The above load s expressed by Eq. 5, whch sums the ndvdual postve loads of mgratng an operator o hop-by-hop from n s to n d. In a smlar manner, the negatve mnl Q load for a mult-hop mgraton s calculated and expressed by Eq. 6. Because the undefned load remans the same for both sngle-hop and mult-hop mgratons, t s not redefned. The optmstc and pessmstc benefts for a mult-hop mgraton are denoted by Q MOB and MPB Q, respectvely (Eq. 7 and Eq. 8, respectvely). A. Algorthm Descrpton Q = mpl = mpl Q {x = f z y = f z } Eq. 5 {x = f z y = f z } Eq. 6 + ul Q Q h mnl Eq. 7 (mnl Q + ul Q h ) Eq. 8 Each node wthn the WS employs the aforementoned mult-hop optmstc and pessmstc metrcs as well as the functonalty descrbed n Secton IV.A to make mgraton decsons for the operators t hosts. Each WS node performs the followng four steps: Step. Each node n s dentfes ts locally parttoned subgraphs of operators, as descrbed n Secton IV.A. Then, step 2 s performed. Step 2. For each sub-graph node t Q, we calculate ts multhop optmstc/pessmstc beneft for mgratng the operators contaned n t G towards each n d that s wthn the awareness of n s. Intally, t G contans a sngle operator, wth ts canddate assgnments beng equal to the number of nodes wthn the awareness of n s (ncludng n s ). In case that the optmstc beneft for each assgnment contaned n t Q s less than or equal to zero, then t Q s pruned from the graph. In case a subgraph node t Q has postve pessmstc beneft, then the respectve collecton of operators (or sngle operator) contaned n t Q s not yet mgrated. If an assgnment for a par (operator, node) s not feasble due to lmtatons n computng capacty of the respectve node, then the pessmstc/optmstc beneft of that par becomes mnus nfnty. Proceed to step 3. Step 3. Each t Q n the graph contans K dfferent assgnments for the operators contaned n t, wth K denotng the number of nodes wthn the awareness of n s (ncludng n s ).

7 TZIRITAS ET AL.: DISTRIBUTED ALGORITHMS FOR THE OPERATOR PLACEMET PROBLEM 7 <{o 7 :n }> <{o 2 :n }> <{o 4 :n }> <{o 7 :n }> <{o 2 :n 3 },{o 4 :n 3 }> 0/0 0/0 0/0 best assgnment 0/0 66/8 o 7 7 : o 2 : 60 o 4 : accordng to 3 o 7 7 : {o 2 :, o 4 :} pessmstc 00 6 <{o 6 :n 6 }> beneft o 6 : 0/ o : o 3 : o 5 5 :5 o o o <{o :n }> <{o 3 :n }> <{o 5 :n 3 }> : 3 : 5 5 :5 0/0 0/0 8/28 <{o :n }> 0/0 <{o 3 :n }> 0/0 <{o 5 :n 3 }> 8/28 <{o 2 :n 3 },{o 4 :n 3 }> 66/8 <{o :n 2 }{o 7 :n 2 }> {o 2 :, o 4 :} 52/68 7 {o :, o 7 :} 6 o 3 : o 5 5 :5 <{o 3 :n }> <{o 5 :n 3 }> 0/0 8/28 Fgure 9. Prunng o 6. Fgure 0. Consoldatng o 2 and o 4 Fgure. Consoldatng o and o 7 <{o 2 :n 3 },{o 4 :n 3 }, {o :n 2 }{o 7 :n 2 }> 32/58 {o :, o 2 :, o 4 :, o 7 :} 7 o 3 : o 5 5 :5 <{o 3 :n }> <{o 5 :n 3 }> 0/0 8/28 <{o :n 2 },{o 2 :n 3 },{o 3 :n 3 }, {o 4 :n 2 },{o 7 :n 2 }> 32/32 {o :, o 2 :, o 3 :,o 4 :, o 7 :} 5 o 5 :5 <{o 5 :n 3 }> 8/28 <{o :n 2 },{o 2 :n 3 },{o 3 :n 3 }, {o 4 :n 2 },{o 5 :n 2 },{o 7 :n 2 } > 45/45 {o :, o 2 :, o 3 :, o 4 :, o 5 :5, o 7 :} Fgure 2. Consoldatng {o, o 7} wth {o 2, o 4}. Fgure 3. Consoldatng {o, o 2, o 4, o 7} wth o 3 Fgure 4. Consoldatng {o, o 2, o 3, o 4, o 7} wth o 5 <k> Each assgnment t Q s accompaned wth the respectve optmstc/pessmstc beneft, as well as wth a multdmensonal weght W(t <k> Q ) of K- dmensons (we exclude n s, wth the reason beng explaned below). The th dmenson of W(t <k> Q ) s denoted by w (t <k> Q ), whch represents the total computng requrements of the operators (contaned n t Q ) to be assgned on the th node wthn the awareness of n s (excludng n s ). For nstance, assume that t Q contans three operators {o, o 2, o 3 }, wth each of them havng sze of one <> unt. Let t Q be the frst assgnment of t Q that assgns o and o 2 on the frst node, whle o 3 on the second node wthn the awareness of n s. Gven that the number of network nodes that are wthn the awareness of n s s two (excludng n s ) n total, W(t <> Q ) s descrbed by {2, }. The knapsack sze s expressed n K- dmensons. The sze of the th dmenson represents the avalable capacty of the th node wthn the awareness of n s. ext, the K- mult-dmensonal knapsack <x> problem s solved as follows: t Q plays the role of a knapsack object, provded that <x> s the best assgnment of t Q accordng to the pessmstc beneft. The pessmstc beneft of <x> t Q corresponds to the beneft of the respectve knapsack object, whle W(t <x> Q ) sgnfes ts mult-dmensonal weght. If an object has beneft less than or equal to zero, then t s excluded when solvng knapsack. In case the objects do not volate any capacty of the nodes, then there s no reason to execute knapsack. The knapsack soluton s stored wthout performng any mgraton. In case there s an already stored soluton, then the one wth the bggest pessmstc beneft s stored. We must note that the reason n s s excluded when solvng knapsack s that n s s the current host of the operators under coneraton. Therefore, there s no capacty ssue when assgnng all of the operators under coneraton onto n s. ext, we proceed wth step 4. Step 4. For each par of adjacent graph-nodes {t A, t B } n the chosen sub-graph, we calculate ther merged optmstc/pessmstc benefts (K K n total). The procedure of mergng the optmstc/pessmstc benefts s explaned below. The optmstc/pessmstc beneft between k th canddate assgnment of t A (denoted by <k>) and k th canddate assgnment of t B (denoted by <k >) s gven by Eq. 9/Eq. 20. Specfcally, the last part of Eq. 9/Eq.20 s for recalculatng the network overhead ncurred due to the communcaton between the operators belongng to t A and t B. The above s because, when the optmstc/pessmstc beneft s calculated for t A /t B, t s assumed that the operators belongng to t B /t A are hosted by n s. ote that h sk() denotes the host of o under k-th assgnment. By takng all of the combnatons between the canddate assgnments of t A and t B, we result n a matrx of K K new canddate assgnments. ote that each such assgnment descrbes a canddate host for each operator contaned n the unon of t A and t B. The top K new canddate assgnments are kept (the rest are dscarded). The evaluaton of an assgnment s performed by conerng the summaton of the pessmstc and optmstc benefts. The above s because of avodng gettng trapped n local optmal solutons due to reducton nto the top K assgnments. ote that f a new canddate assgnment s not feasble due to lmtatons n computng capactes of nodes, then the pessmstc/optmstc beneft of the correspondng assgnment becomes mnus nfnty. In that way, each par (t A, t B ) results n at most K pessmstc benefts. The par wth the bggest pessmstc beneft s chosen to be consoldated. The algorthm termnates f there s no consoldaton to be performed, otherwse we proceed to step 3. A,B MOB s<k>,s<k > A B = MOB s<k> + MOB s<k > (C j + C j )(h sk() + h sk () + h k()k ()) A j B A,B MPB s<k>,s<k > A B = MPB s<k> + MPB s<k > (C j + C j )( h sk() h sk () + h k()k ()) A j B Eq. 9 Eq. 20

8 TZIRITAS ET AL.: DISTRIBUTED ALGORITHMS FOR THE OPERATOR PLACEMET PROBLEM 8 It must be noted that the prmary soluton s the one acheved at the end of consoldatons. The soluton of knapsack can be employed when the prmary soluton fals to result n a good placement due to the reducton of dmensons when choosng the top K assgnments. An example that the prmary soluton can fal s when the fnal consoldaton cannot be acheved due to volaton of nodes capactes n all dmensons. B. Example to Illustrate DGA s Functonalty The functonalty of DGA when conerng capacty constrants s llustrated through the followng example. We are gven the network shown n Fg. and the applcaton shown n Fg. 2, where all of the operators are hosted on n, except o 8 whch s hosted on n 2. Conerng n, we choose to dentfy the operator mgratons for the sub-graph at the rght e of Fg. 4 (step ). Then the optmstc/pessmstc beneft of each operator under K (K=3) dfferent canddate assgnments s shown n Table II. Table II. Pessmstc/optmstc benefts under dfferent assgnments wthn the awareness of n <n > <n 2> <n 3> o 0/0-4/6-202/202 o 2 0/0-24/22-48/244 o 3 0/0-5/9-30/8 o 4 0/0-63/57-26/4 o 5 0/0 4/4 8/28 o 6 0/0-54/-46-08/-92 o 7 0/0-07/07-24/24 As observed, the optmstc beneft of o 6 s less than or equal to zero for all of the canddate assgnments. Therefore, o 6 s pruned from the current sub-graph as shown n Fg. 9 (step 2). We observe that the best pessmstc beneft for each of the sub-graph nodes n Fg. 9 s zero except o 5. Therefore, there s no reason to run knapsack for only one object (step 3 teraton ). ext, we calculate the K K assgnments for each canddate consoldaton of adjacent graph-nodes n Fg. 9. As we can see there are sx combnatons of canddate consoldatons ({o, o 3 }, {o, o 7 }, {o 2, o 3 }, {o 2, o 4 }, {o 2, o 7 }, {o 3, o 5 }). The assgnments of the aforementoned canddate consoldatons are shown n Table III - Table VIII, respectvely. To explan the format of the tables, we coner Table III. The result shown n thrd column and second row s -69/75. The meanng s that when assgnng o 3 on n 3 and o on n 2, the pessmstc beneft equals -69, whle the optmstc one s 75. Table III. Pessmstc/optmstc benefts when consoldatng o and o3 under dfferent node assgnments <{o 3:n }> <{o 3:n 2}> <{o 3:n 3}> <{o :n }> 0/0-5/7-30/4 <{o :n 2}> -4/59-54/68-69/75 <{o :n 3}> -202/98-25/ /26 Table IV. Pessmstc/optmstc benefts when consoldatng o and o7 under dfferent node assgnments <{o 7:n }> <{o 7:n 2}> <{o 7:n 3}> <{o :n }> 0/0-07/-93-24/-86 <{o :n 2}> -4/-39 52/68-55/-25 <{o :n 3}> -202/2-09/-9-6/6 Table V. Pessmstc/optmstc benefts when consoldatng o2 and o3 under dfferent node assgnments <{o 3:n }> <{o 3:n 2}> <{o 3:n 3}> <{o 2:n }> 0/0-5/-3-30/-6 <{o 2:n 2}> -24/0-27/9-42/6 <{o 2:n 3}> -48/220-5/229-54/238 Table VI. Pessmstc/optmstc benefts when consoldatng o2 and o4 under dfferent node assgnments <{o 4:n }> <{o 4:n 2}> <{o 4:n 3}> <{o 2:n }> 0/0-63/-63-26/-26 <{o 2:n 2}> -24/2 33/59-30/-4 <{o 2:n 3}> -48/4 9/6 66/8 Table VII. Pessmstc/optmstc benefts when consoldatng o2 and o7 under dfferent node assgnments <{o 7:n }> <{o 7:n 2}> <{o 7:n 3}> <{o 2:n }> 0/0-07/93-24/86 <{o 2:n 2}> -24/08-7/25-224/308 <{o 2:n 3}> -48/26-4/ /430 Table VIII. Pessmstc/optmstc benefts when consoldatng o3 and o5 under dfferent node assgnments <{o 3:n }> <{o 3:n 2}> <{o 3:n 3}> <{o 5:n }> 0/0-5/- -30/-2 <{o 5:n 2}> 4/4 -/3-6/2 <{o 5:n 3}> 8/8 3/7-2/26 The top K assgnments for each par are shown n gray color n aforementoned tables. ote that the par {o 2, o 4 } s the one wth the bggest pessmstc beneft (66). Therefore, o 2 and o 4 are chosen to be consoldated as shown n Fg. 0 (step 4 teraton ). As observed n Fg. 0, the objects {o 2, o 4 } and o 5 have pessmstc beneft greater than zero under the assgnment of o 2, o 4, and o 5 on n 3. Because there s no volaton of n 3 s avalable capacty, there s no reason to nvestgate the knapsack soluton (step 3 ter. 2). The soluton s stored as best canddate soluton, wth the total beneft beng equal to 74 (66+8). ext, the par {o, o 7 } s chosen to be consoldated as depcted n Fg. (step 4 teraton 2). (The tables wth the K K assgnments for the above are omtted). We result n three objects {o 2, o 4 }, {o, o 7 }, and o 5 wth beneft greater than zero (under the assgnment of o and o 7 on n 2, whle of o 2, o 4, and o 5 on n 3 ). Because there s no volaton of n 2 s and n 3 s avalable capacty, the knapsack s not executed (Step 3 ter. 3). The total beneft of the current soluton s 26 ( ) and t s stored as the best canddate soluton (the prevous was 74). Afterwards, we coner consoldatng per pars the nodes depcted n Fg.. The best canddate s that of consoldatng {o 2, o 4 } wth {o, o 7 }, wth the assgnments and the benefts of that consoldaton beng shown n Table IX. To result n Table IX we combne the top three assgnments of {o 2, o 4 } wth the top three assgnments of {o, o 7 } (step 4 ter. 3). The total beneft of the current soluton s 26 (step 3 ter. 4). The aforementoned are shown n Fg. 2.

9 TZIRITAS ET AL.: DISTRIBUTED ALGORITHMS FOR THE OPERATOR PLACEMET PROBLEM 9 Table IX. Pessmstc/optmstc benefts when consoldatng {o, o7} and {o2, o4} under dfferent node assgnments <{o :n }, <{o :n 2}, {o 7:n 2}> <{o :n 3}, {o 7:n 3}> {o 7:n }> <{o 2:n 2}, {o 4:n 2}> 33/45 99/3 3/47 <{o 2:n 3}, {o 4:n 2}> 9/33 75/0 2/49 <{o 2:n 3}, {o 4:n 3}> 66/90 32/58 78/06 ext, we consoldate o 3 wth {o, o 2, o 4, o 7 }. The assgnments and the benefts of the consoldaton s shown n Table X (step 4 ter. 4). Afterwards, a two-dmensonal knapsack s solved for the two objects o 3 and {o, o 2, o 4, o 7 }. The weght of the former s {0, 5}, whle the weght of the latter s {2, 2}. The sze of the knapsack s {9, 6}, wth 9 and 6 representng the avalable capacty of n 2 and n 3, respectvely. ote that the avalable capacty of n 2 s 9 because o 8 s already hosted there. The knapsack soluton s to reject o 3 and accept {o, o 2, o 4, o 7 }, wth a beneft of 32 (step 5 ter. 5). Because the beneft of the prevously stored soluton was 26, the current soluton s stored as the best canddate soluton. The result s shown n Fg. 3. Table X. Pessmstc/optmstc benefts when consoldatng {o, o2, o4, o7 } and o3 under dfferent node assgnments <{o 3:n }> <{o 3:n 2}> <{o 3:n 3}> <{o 2:n 2}, {o 4:n 2}, {o :n 2},{o 7:n 2}> 99/ 98/08 83/03 <{o 2:n 3}, {o 4:n 2}, {o :n 2},{o 7:n 2}> 75/75 74/84 7/9 <{o 2:n 3}, {o 4:n 3}, {o :n 2},{o 7:n 2}> 32/32 3/4 28/48 Last, the only choce s to consoldate {o, o 2, o 3, o 4, o 7 } wth o 5, whch s depcted n Fg. 4. The assgnments and benefts of the aforementoned consoldaton are shown n Table XI. As we can see, the assgnments of o 5 on n 3 are not feasble due to the volaton of n 3 s capacty. Consequently, the benefts of those assgnments are not shown. As seen, the best pessmstc beneft that can be acheved by the respectve consoldaton s 45. On the other hand, the best soluton stored by executng knapsack was 32. Therefore, n ths example there s no reason to employ the knapsack soluton. Table XI. Pessmstc/optmstc benefts when consoldatng {o, o2, o3, o4, o7} and o5 under dfferent node assgnments <{o 5:n }> <{o 5:n 2}> <{o 5:n 3}> <{o 2:n 3}, {o 4:n 3}, {o :n 2}, {o 7:n 2}, {o 3:n }> 32/32 36/36 - <{o 2:n 3}, {o 4:n 3}, {o :n 2}, {o 7:n 2}, {o 3:n 2}> 75/75 45/45 - <{o 2:n 3}, {o 4:n 3}, {o :n 2}, {o 7:n 2}, {o 3:n 3}> 32/32 42/42 - VI. IMPLEMETATIO AD COMPLEXITY ISSUES A. Implementaton Issues To avod oscllatons, we apply the followng: (a) we do not allow mgratons that result n swappng operators and (b) for any par of operators o and o j that communcate wth each other and are hosted by dfferent nodes, we do not allow the concurrent mgraton of o and o j, provded that ther mgraton paths have overlap wth the path between the ntal hosts of o and o j. To keep updated a node wth the current avalable capacty of the nodes that are wthn ts network awareness, each tme a mgraton s performed the followng takes place. The source and destnaton nodes send each a control message to the nodes wthn ther awareness about ther current avalable capacty. When movng an operator o, ts adjacent operators are nformed for the new locaton of the operator n queston through pggy-backng. B. Complexty Issues Tme complexty: We explan analytcally the tme complexty for each step of DGA wthout conerng capacty constrants. Specfcally, the tme complexty of (a) dentfcaton of locally parttoned sub-graphs s O(E); wth E denotng the total edges, (b) Step s O(); (c) Step 2 s O(Q E ); where Q E s the total edges adjacent to the nodes of group Q; (d) the re-calculaton of optmstc/pessmstc benefts n Step 3 s O(E ); where E represent the edges between the remaned nodes and the nodes currently pruned; (e) Step 4 s O(S E ), wth S E denotng the number of edges contaned n the sub-graph under coneraton; (f) Step 5 s O(R), wth R sgnfyng the operators remaned on the graph; and (g) Step 6 s O(). ote that the aforementoned analyss can be smply extended for the case where capacty constrants are taken nto account. Message complexty: For the un-capactated case, O() messages are exchanged per operator group mgraton. For the capactated case: (a) O(H) messages are exchanged per operator group mgraton, wth H denotng the number of operators communcatng wth the operator group under mgraton; (b) O(A) messages are exchanged after performng an operator group mgraton, wth A sgnfyng the number of nodes that are wthn the awareness of source and destnaton nodes. VII. P-COMPLETEESS AD COVERGECE ISSUES Ths secton s splt nto two portons: (a) the frst part s nvolved wth the P-Complete reducton of OPP when conerng capacty constrants and (b) the second part dscusses the convergence of DGA through analytcal proofs. A. P-completeness Reducton In ths secton we formally reduce the OPP problem to the 0- knapsack problem. Theorem. The decson verson of OPP (OPP-dec) s Pcomplete by reducton to 0- Knapsack-dec. Proof. We gve the OPP-dec and Knapsack-dec statements as follows. OPP-dec: Gven a WS network and an applcaton graph together wth communcaton and capacty characterstcs, s there an assgnment of values n the placement matrx X such that the network overhead as per Eq. 6 s at most Y (for some

10 TZIRITAS ET AL.: DISTRIBUTED ALGORITHMS FOR THE OPERATOR PLACEMET PROBLEM 0 constant Y), whle respectng the constrants of Eq. 7 and Eq Knapsack-dec: Gven a set of objects, whereby the -th object (let t ) has a value of v and sze of s, a knapsack of sze K, and an szed Boolean matrx X where X = f the -th object s selected, 0 otherwse, s there an assgnment of values n F such that: = X v Y for some constant Y subject to = X s K. Frst, we show that a polynomal tme reducton exsts. Coner a 0- Knapsack-dec nstance as above. Let S and V be the aggregated szed and values of knapsack objects, respectvely. We construct an equvalent OPP-dec nstance as follows. The WS network conssts of two nodes n and n 2, wth W(n ) = S and W(n 2 ) = Q + K, where Q s a selected constant such that Q > S and K s the sze of knapsack. The applcaton graph s a star wth + operators. The frst operators correspond n a one to one bass to the knapsack objects regardng sze,.e., W(o ) = s. The last operator s the center of the star and has a sze of Q. The -th operator communcates wth the center of the star ncurrng a total load of v, where v s the related Knapsack value. It s easy to see that a vald assgnment n the X matrx nvolves allocatng the + operator at n 2 (t does not ft at n ) and the remanng at both n and n 2. Because the only communcaton takes place between the ( + )-th and the rest, the network overhead equals to = v X. The above s justfed by the fact that we count only the operators hosted by n because the ones stored at n 2 ncur zero network overhead. OPP-dec wll assgn values at X such that = v X s mnmzed. Because from Eq. 8 an operator must be assgned n exactly one WS t holds that = v X = V = v X 2. Due to the fact that V s constant, the aforementoned mnmzaton s equvalent to the maxmzaton of = v X 2. ote also that ( + )-th operator of sze Q s forcedly allocated at n 2, wth the remanng capacty of n 2 beng K. Therefore, gven a 0- Knapsack-dec nstance (whereby we ask whether an assgnment exsts of value at least Y ) we construct an equvalent OPP-dec nstance askng whether an operator assgnment of network overhead at most Y = V Y exsts. The Knapsack-dec nstance has a soluton ff the equvalent OPP-dec nstance has a soluton. Moreover, t s easy to see that the transformaton s polynomal n tme O(). To prove that OPP-dec s P-complete, t remans to show that t belongs n P. Ths s trval, snce gven an assgnment n X matrx, both the constrants of Eq. 7 and Eq. 8 as well as the network overhead as per Eq. 6 can be calculated n polynomal tme. B. DGA s Convergence Issues In ths secton we derve a proof for a theorem that guarantees the convergence of DGA. To result n the aforementoned theorem, we frst prove through lemmas that each operator mgraton strctly decreases the network overhead rrespectve of performng one hop or many hop mgratons. Lemma. The network overhead s strctly reduced when performng any sngle operator mgraton, provded that (a) swappng operators that communcate wth each other s forbdden and (b) only one hop mgratons are allowed. Proof. When a node n s mgrates an operator o towards an one hop neghborng node n d, we dscern the followng two cases: (a) the operator under mgraton does not communcate wth any operator o j that s under mgraton. (b) The operator under mgraton communcates wth an operator o j that s gong to mgrate. ote that o s mgrated towards n d ff ts pessmstc beneft PB equals PB and s strctly postve. Consequently, t s easly seen that n case of (a) any mgraton results n a decrease of the network overhead of exactly PB unts. The above s because there s no operator that communcates wth o and mgrates concurrently wth o. Therefore, PB concdes wth the network overhead reducton. In case of (b) we dscern the followng cases: (b) o j s hosted by n s and mgrates to n f ; (b2) o j s hosted by n d and mgrates to n f, under the assumpton that t dstances tself away from n s. (b3) o j s hosted by n d and mgrates to n s, (b4) o j s hosted by a node n f other than n s and n d and t mgrates to n z. j Regardng (b), t s clear that PB and PB sf reflect the network overhead reducton captured by PB + PB j sf. The above s because the same node makes both mgraton decsons. In terms of (b2) the network overhead reducton s at least PB j df. In case of (b3) ether the mgraton of o or o j s not performed due to the fact that swappng operators s not allowed. Gven that (b4) holds, the network overhead reducton s at least mn{pb, PB j fz }. It must be noted that the aforementoned are also trvally appled when conerng more than one operator such as o j. Consequently, when performng a mgraton t s guaranteed that the network overhead decreases. Lemma 2. The network overhead s strctly reduced when performng any mult-hop sngle operator mgraton, provded that there are no concurrent mgratons of two non-collocated operators o and o j that (a) communcate wth each other and (b) both mgraton paths have an overlap wth the path between the ntal hosts of o and o j. Proof. When conerng a mgraton of an operator o from n s to n d, we dscern the followng two cases: (a) there are no concurrent mgratons of operators that communcate wth each other and (b) there s a concurrent mgraton of operators o and o j that communcate wth each other. Regardng (a), by followng the same ratonale as that of Lemma, the network overhead s reduced exactly by MPB. Regardng (b), we dscern the followng cases: (b) o j s hosted by n d and mgrates to n f, under the assumpton that there s no overlap of any mgraton path wth the path between n s and n d (.e., the ntal hosts of o and o j, respectvely). (b2) o j s hosted by n d and mgrates to n f, under the assumpton that there s overlap of exactly one mgraton path wth the path between n s and n d, (b3) o j s hosted by n d and mgrates to n f, under the assumpton that there s overlap of both mgraton paths wth the path between n s and n d. Regardng (b), the network overhead s

11 etwork overhead aganst DBA umber of mgratons umber of control msgs etwork overhead umber of mgratons umber of control msgs per operator TZIRITAS ET AL.: DISTRIBUTED ALGORITHMS FOR THE OPERATOR PLACEMET PROBLEM GRAL* DGA DRA MCFA DBA dfs 0 GRAL* DGA DRA MCFA DBA dfs GRAL* DGA DRA MCFA DBA dfs Fgure 5. ormalzed network overhead Fgure 6. ormalzed number of mgratons Fgure 7. umber of control messages per operator GRAL* DRA DGA Surplus capacty GRAL* DRA DGA DBA Surplus capacty GRAL* DRA DGA DBA Surplus capacty Fgure 8. ormalzed network overhead when conerng capacty constrants reduced by exactly MPB + MPB j df Fgure 9. ormalzed number of mgratons when conerng capacty constrants. In terms of (b2), the network overhead s reduced by at least MPB j df, provded that the mgraton path of o has overlap wth the path between n s and n d. In case of (b3), the mgraton of ether o or o j s cancelled due to the restrcton that there cannot be two mgraton paths that have overlap wth the path between the ntal hosts of the correspondng operators. Therefore, gven that the mgraton of o j s cancelled, the mgraton overhead s reduced by exactly MPB. It must be noted that the aforementoned are also trvally appled when conerng more than one operator such as o j. Consequently, when performng any mgraton of more than one hop t s guaranteed that the network overhead decreases. Lemma 3. The network overhead s strctly reduced when performng any group operator mgraton. Proof. The proof becomes trval by reducng a group operator to a sngle operator. The reducton takes place by conerng a group of operators o Q as a whole, by gnorng communcaton edges between operators partcpatng n o Q. On the other extreme, for a communcaton edge e between an operator o partcpatng n o Q and an operator o j that does not partcpate n o Q we do the followng. The edge e becomes an edge between o Q and o j. In case there are more than one edge between o Q and an operator o j, then we consoldate them nto one by summng ther weghts. Theorem 2. DGA converges always n a stable placement provded that the operator communcaton patterns are stable. Proof. Accordng to Lemma, Lemma 2, and Lemma 3, there s no mgraton that can ncrease or leave ntact the network overhead. Because the network overhead cannot be negatve, DGA performs a fnte number of mgratons. Therefore, DGA converges n a stable placement. Fgure 20. ormalzed number of control messages when conerng capacty constrants VIII. EXPERIMETS The expermental evaluaton has been conducted on S-2 [23]. Fve dfferent network topologes were generated wth the number of nodes beng fxed at 00. The networks are generatng by placng randomly the nodes n a plane of dstance unts. odes are assumed to be n range of each other f ther Eucldean dstance was less than ρ dstance unts (wth ρ beng unformly dstrbuted between 20 and 40). The correspondng tree-based routng topology s obtaned by constructng a spannng tree, whereby each par of nodes s connected va a sngle path. Ten dfferent tree-structured and ten general-structured applcaton graphs were constructed, wth the number of operators rangng between 30 and 70 (unformly dstrbuted). The generaton of applcaton graphs takes place n the same way as that of network generaton. The only dfference s that n case of general-structured applcaton graphs, we do not apply the part of applyng the spannng tree. The ntal placement of operators takes place n a random fashon. The evaluaton was splt nto two parts. The frst part nvolved tree-structured applcaton graphs, whle the second one general-structured applcaton graphs. A. Tree-structured Applcaton Graphs Ths secton s splt nto two portons: (a) un-capactated nodes and (b) capactated nodes. We compare DGA wth the followng () WS algorthms, GRAL* [6], MCFA [7], and dfs [3]; and () cloud computng algorthms, DRA [7], and DBA [4]. It must be noted that all of the algorthms except dfs and MCFA are appled for one hop network awareness. The reason that dfs and MCFA are not appled for one hop network awareness s that they both flood the network n a regon of more than one hop.

12 etwork overhead aganst DRA etwork overhead aganst DRA etwork overhead umber of mgratons umber of control msgs TZIRITAS ET AL.: DISTRIBUTED ALGORITHMS FOR THE OPERATOR PLACEMET PROBLEM DGA DRA DBA dfs DGA DRA DBA dfs DGA DRA DBA dfs Fgure 2. ormalzed network overhead Fgure 22. ormalzed number of mgratons Fgure 23. ormalzed number of control messages DGA 5 DGA Surplus Capacty etwork awareness n hops Fgure 24. ormalzed network overhead when conerng capacty constrants and ncreasng surplus capacty ) Un-capactated nodes In ths secton, we do not coner capacty constrants on nodes. In Fg. 5, we show the total communcaton overhead ncurred wthn the WS when applyng the aforementoned sx approaches. As t can be seen GRAL*, DGA, DRA, and MCFA acheve the best performance. It must be noted that accordng to [7], [7], and [6], the optmal performance s always acheved when applyng DRA, MCFA, and GRAL* on tree-structured applcatons, respectvely. Even though there s no optmalty proof for DGA, t seems that ts network overhead s optmal. By delvng nto the expermental data, we notced that ndeed there was no dfference n network overhead between DGA and DRA, MCFA, and GRAL*. On the other extreme, dfs and DBA resulted n the worst performance. The above s justfed by the fact that both dfs and DBA do not coner mgratng operators n a grouped fashon. It s worth mentonng that DGA acheves about 35% network overhead reducton aganst DBA and dfs. The normalzed number of mgratons s shown n Fg. 6. It s observed that MCF acheved the least number of mgratons. The above s expected because MCF takes mgraton decsons n an almost centralzed way. Specfcally, t sends control messages towards all of the nodes wthn the network to result n the optmal placement. A floodng approach s also followed by dfs, resultng n that way n a comparably small number of mgratons. The remanng algorthms acheve almost the same performance n terms of mgratons, wth DBA beng slghtly more effcent compared to GRAL*, DGA, and DRA. The total number of control messages per operator s depcted n Fg. 7 employng a logarthmc scale. As can be seen, the worst performance s acheved by MCFA. The above s due to the fact that MCFA floods the whole network wth control messages to decde for an operator mgraton. The second worst performance s acheved by dfs, wth the Fgure 25. ormalzed network overhead when conerng capacty constrants and ncreasng network awareness reason beng that dfs does not flood the whole network but only a part of t. The rest algorthms burden the network wth only a small number of control messages. The above s explaned by the fact that when GRAL*, DGA, DRA, and DBA need to take an operator mgraton decson, they send control messages only to the nodes hostng operators communcatng wth the correspondng operator. 2) Capactated nodes In ths secton we coner capacty constrants on nodes. Intally, the capacty of each node wthn the system s fxed to the amount of the total capacty requrements needed to host the operators ntally assgned on the respectve node. ext, we ncrease by, 2, 3, 5, and 8 tmes the average capacty requrements of an operator. For ths experment, we show only the performance of DGA, DRA, and GRAL*, because the rest algorthms have not been desgned to take capacty constrants nto coneraton. In Fg.8 we present the performance of the algorthms. The plots show the network overhead acheved by the algorthms compared to DBA whch was used as a yardstck. It must be noted that the network awareness of the algorthms s fxed to four (we experment later wth dfferent awareness levels). As can be seen, DGA results n the best network overhead reducton over DBA. The second best performance s acheved by GRAL*, whle DRA has the worst performance. It must be noted that when the surplus capacty becomes 8, then all of the algorthms acheves almost the same network overhead reducton over DBA. The above s due to the fact that n case of no capacty constrants, GRAL*, DRA, and DGA acheve exactly the same network overhead. In Fg. 9, we show the normalzed number of mgratons. As observed, DGA performed the bggest number of mgratons. The above s expected because n order for DGA to result n the least network overhead, t must perform more mgratons aganst the rest algorthms. It must be noted that

13 TZIRITAS ET AL.: DISTRIBUTED ALGORITHMS FOR THE OPERATOR PLACEMET PROBLEM 3 when the surplus capacty ncreases to eght, then GRAL*, DRA, and DGA perform almost the same number of mgratons. Ths s also equvalent to the case when the nodes had no capacty constrants. Last, we observe n Fg. 20 that DGA exchanges more control messages aganst ts counterparts. evertheless, the prce pad for the ncreased performance network overhead wse s not deemed hgh. Compared to GRAL* whch s the second best overhead wse, the control messages of DGA never exceeded roughly 0%. B. General-structured Applcaton Graphs In ths secton, we show the expermental evaluaton for general-structured applcaton graphs. Because GRAL* and MCFA has been desgned to work only for tree-structured applcaton graphs, we exclude them from the evaluaton. ) Un-capactated nodes We conducted a set of experments to show the performance of DGA regardng the network overhead wthout conerng capacty constrants on nodes. In ths set of experments, all of the algorthms except dfs are appled for one hop network awareness. In Fg. 2, we can see that DGA and DRA acheve the same performance. It s noteworthy to menton DRA results always n the optmal placement when there are no capacty constrants. By delvng nto the expermental results, there was only one experment that the network overhead of DRA was lower than that of DGA by 0.% (regardng the rest experments, DRA and DGA had exactly the same performance). The superorty of DGA and DRA s clearly depcted n Fg. 2, whereby they acheve a network overhead reducton of about 30% aganst DBA and dfs. In terms of Fg. 22, dfs acheves by far the least number of mgratons aganst the rest algorthms. On the other extreme, DGA and DRA perform the bggest number of mgratons, wth DBA followng closely. The reason dfs results n such a low number of mgratons s due to ts floodng mechansm. Specfcally, dfs s floodng mechansm enables a node to nvestgate operator mgratons towards nodes that are more than one hop away aganst the node under coneraton. Of course, such a floodng mechansm comes at the expense of a bg number of control messages as shown n Fg ) Capactated nodes In ths secton, we coner capacty constrants on the nodes. The experments were desgned n the same way as that of Secton VII.A.2 by varyng the surplus capacty of WS node. Because dfs has not been desgned to handle capacty constrants, t does not appear here. We also chose not to appear the results of DBA because of ts nferorty aganst DRA and DGA. It must be noted that the network awareness of DGA and DRA was fxed to 4 hops. As seen n Fg. 24, when the surplus capacty s small there was not enough room for DGA to mprove the network overhead over DRA. Partcularly, when surplus capacty equals, the network overhead reducton of DGA over DRA s roughly 2%. When the surplus capacty equals 3, DGA acheves the bggest network overhead reducton over DRA (roughly 4%). It s worth mentonng that when the surplus capacty ncreases enough, then the performance of DGA s comparatvely equal to that of DRA. The above s justfed by the fact that when the capacty constrants are relaxed enough, DRA approaches the optmal soluton. In the last set of experments, we fxed the surplus capacty to three and vared the network awareness of WS nodes from one to fve. We can observe n Fg. 25, the performance of DGA ncreases aganst DRA when ncreasng the network awareness. It s noteworthy that DGA s mprovement over DRA remans stable for four hops and more. It must be noted that we chose to omt the fgures showng the performance of DGA and DRA n terms of the number of mgratons and control messages because results were smlar to the ones shown n VIII.A.2. IX. COCLUSIOS In ths paper we tackled the problem of operator placement problem (OPP) conerng general-structured applcaton graphs. The problem was addressed n a fully dstrbuted manner takng nto account the capacty constrants on nodes. We proved that OPP s P-complete when conerng capacty constrants on nodes, wth the reducton beng based on 0- Knapsack decson problem. The convergence of the proposed algorthm was dscussed through thorough and rgorous proofs. The expermental evaluaton showed that (a) our approach acheves almost optmal placements when capacty constrants are not conered, and (b) a network overhead reducton of up to 4% s acheved compared to the best state-of-the-art algorthm found n the lterature. Our future drectons are: (a) to coner fault tolerance ssues when executng our algorthms; and (b) to modfy DGA such that to work also for general-structured WS networks. ACKOWLEDGMETS Ths work was supported n part by the Chna atonal Basc Research Program (973 Program, o.205cb352400) and SFC under grant U kos Tzrtas work was n part supported by FSC REFERECES [] F. Aelo, G. Fortno, R. Gravna, A. Guerrer A Java-based Agent Platform for Programmng Wreless Sensor etworks, Computng, vol 54(3), pp , 200. [2] I. Ayala, M. Amor, L. Fuentes, An Agent Platform for Self- Confgurng Agents n the Internet of Thngs, Infrastructures and Tools for Multagent Systems, 202. [3] G. Chatzmlouds, A. Cuzzocrea, D. Gunopoulos,. Mamouls, A ovel Dstrbuted Framework for Optmzng Query Routng Trees n Wreless Sensor etworks va Optmal Operator Placement, Journal of Computer and System Scences, vol. 79, no. 3, pp , 203. [4] G. Chatzmlouds, H. Hakkoymaz,. Mamouls, D. Gunopoulos, Operator Placement for Snapshot Mult-predcate Queres n Wreless Sensor etworks, IEEE Internatonal Conference on Moble Data Management: Systems, Servces and Mddleware (MDM), pp. 2-30, [5] J. W. Jang, T. Lan, S. Ha, M. Chen, M. Chang, Jont VM Placement and Routng for Data Center Traffc Engneerng, IEEE IFOCOM, pp , 202. [6] E. Jung, I. Cho, S. M. Kang, otslo: The Agent Servce Platform Supportng Dynamc Behavour Assembly for Resolvng the Heterogenety of IoT, Internatonal Journal of Dstrbuted Sensor etworks, 204. [7] Z. Lu, Y. Wen, Dstrbuted and Asynchronous Soluton to Operator Placement n Large Wreless Sensor etworks, IEEE Internatonal

14 TZIRITAS ET AL.: DISTRIBUTED ALGORITHMS FOR THE OPERATOR PLACEMET PROBLEM 4 Conference on Moble Ad-hoc and Sensor etworks (MS), pp , 202. [8] Z. Lu, Y. Wen, R. Fan, S.-L. Tan, J. Bswas, Toward Effcent Dstrbuted Algorthms for In-etwork Bnary Operator Tree Placement n Wreless Sensor etworks, IEEE Journal on Selected Areas n Communcatons, vol 3, no. 4, 203. [9] X. Meng, V. Pappas, and L. Zhang, Improvng the Scalablty of Data Center etworks wth Traffc-aware Vrtual Machne Placement, IEEE Conference on Computer Communcatons (IFOCOM), pp. -9, 200. [0] M.-S. Pan, H.-W. Fang, Y.-C. Lu, Y.-C. Lu, Address Assgnment and Routng Schemes for Zgbee-based Long-thn Wreless Sensor etworks, IEEE Internatonal Conference on Vehcular Technology, [] A. Pathak, V. K. Prasanna, Energy-Effcent Task Mappng for Data- Drven Sensor etwork Macroprogrammng, IEEE Transacton on Computers, vol. 59, no. 7, pp , 200. [2] G.J. Potte, W.J. Kaser, Wreless Integrated etwork Sensors, Communcatons of the ACM, vol. 43, no. 5, pp. 5-58, [3] R. Sappd, A. Grard, C. Rosenberg, Maxmum Achevable Throughput n a Wreless Sensor etwork Usng In-etwork Computaton for Statstcal Functons, IEEE/ACM Transactons on etworkng (TO), vol. 2, no. 5, pp , 203. [4] J. Sonnek, J. Greensky, R. Reutman, A. Chandra, Starlng: Mnmzng Communcaton Overhead n Vrtualzed Computng Platforms Usng Decentralzed Affnty-Aware Mgraton, Internatonal Conference on Parallel Processng (ICPP), pp , 200. [5]. Tzrtas, G. Georgakouds, S. Lals, T. Paczesny, J. Domaszewcz, P. Lampsas, T. Loukopoulos, Mddleware Mechansms for Agent Moblty n Wreless Sensor etworks, Internatonal Conference on Sensor Systems and Software (S-CUBE), 202. [6]. Tzrtas, S. U. Khan, T. Loukopoulos, S. Lals, C.-Z. Xu, P. Lampsas, Sngle and Group Agent Mgraton: Algorthms, Bounds, and Optmalty Issues, IEEE Transactons on Computers, vol. 63, no. 2, pp , 204. [7]. Tzrtas, S. U. Khan, C.-Z. Xu, T, Loukopoulos, S. Lals, On Mnmzng the Resource Consumpton of Cloud Applcatons Usng Process Mgratons, Elsever Journal of Parallel and Dstrbuted Computng, vol. 73, no. 2, pp , 203. [8]. Tzrtas, S. Lals, S. U. Khan, T. Loukopoulos, C.-Z. Xu, and P. Lampas, "Dstrbuted Onlne Algorthms for the Agent Mgraton Problem n WSs," ACM/Sprnger Moble etworks and Applcatons, vol. 8, no. 5, pp , 203. [9]. Tzrtas, T. Loukopoulos, S. Lals and P. Lampsas, GRAL: A Groupng Algorthm to Optmze Applcaton Placement n Wreless Embedded Systems. IEEE Internatonal Parallel and Dstrbuted Processng Symposum (IPDPS), pp , 20. [20]. Tzrtas, C.-Z. Xu, T. Loukopoulos, S. U. Khan, Z. Yu, Applcatonaware Workload Consoldaton to Mnmze both Energy Consumpton and etwork Load n Cloud Envronments, IEEE Internatonal Conference on Parallel Processng (ICPP), pp , 203. [2] L. Yng, Z. Lu, D. Towsley, C. H. Xa, Dstrbuted Operator Placement and Data Cachng n Large-scale Sensor etworks, IEEE Conference on Computer Communcatons (IFOCOM), [22] Q. Zhang, Q. Zhu, M. F. Zhan, R. Boutaba, J. L. Hellersten, Dynamc Servce Placement In Geographcally Dstrbuted Clouds, IEEE Journal on Selected Areas n Communcatons (JSAC), 203. [23] etwork Smulator2 (ns2), Thanass Loukopoulos receved hs Dploma n Computer Engneerng and Informatcs from the Unversty of Patras, Greece, n 997. He was awarded a PhD degree n Computer Scence by the Hong Kong Unversty of Scence and Technology (HKUST) n Currently, he s Lecturer n the Dept. of Computer Scence and Bomedcal Informatcs, Unversty of Thessaly at Lama, Greece. He has publshed 35 papers and had the best paper award n ICPP 200. Samee U. Khan receved a BS degree n 999 from Ghulam Ishaq Khan Insttute of Engneerng Scences and Technology, Top, Pakstan, and a PhD n 2007 from the Unversty of Texas, Arlngton, TX, USA. Currently, he s Assocate Professor of Electrcal and Computer Engneerng at the orth Dakota State Unversty, Fargo, D, USA. Prof. Khan s research nterests nclude optmzaton, robustness, and securty of: cloud, grd, cluster and bg data computng, socal networks, wred and wreless networks, power systems, smart grds, and optcal networks. Hs work has appeared n over 300 publcatons. He s a Fellow of the Insttuton of Engneerng and Technology (IET, formerly IEE), and a Fellow of the Brtsh Computer Socety (BCS). He s a senor member of the IEEE. Cheng-Zhong Xu receved the PhD degree n computer scence from the Unversty of Hong Kong n 993. He s currently a professor n the Department of Electrcal and Computer Engneerng of Wayne State Unversty, and the drector of Cloud and Internet Computng Laboratory (CIC) and Sun s Center of Excellence n Open Source Computng and Applcatons (OSCA). Hs research nterest s manly n scalable dstrbuted and parallel systems and wreless embedded computng devces. He has publshed two books and more than 60 artcles n peer-revewed journals and conferences n these areas. He s a senor member of the IEEE. AUTHOR S BIOGRAPHIES kos Tzrtas receved hs B.Sc. degree from the Technologcal Educatonal Insttute of Serres, Greece, n 2004, and a M.Sc. and a Ph.D. degree from the Unversty of Thessaly, Greece, n 2006 and 20, respectvely. He s currently a research assstant professor n Shenzhen Insttutes of Advanced Technology, Chnese Academy of Scences. Hs work has appeared n over 25 publcatons.