Cloud of Things for Sensing-as-a-Service: Architecture, Algorithms, and Use Case

Transcription

1 Cloud of Thngs for Sensng-as-a-Servce: Archtecture, Algorthms, and Use Case Sherf Abdelwahab, Bechr Hamdaou, Mohsen Guzan, and Taeb Znat Oregon State Unversty, Unversty of Idaho, Unversty of Pttsburgh, Abstract We propose Cloud of Thngs for Sensngas-a-Servce: a global archtecture that scales up cloud computng by explotng the global sensng resources of the Internet of Thngs IoT) to enable remote sensng. Cloud of Thngs enables n-network dstrbuted processng of sensors data offered by the globally avalable IoT devces and provdes a global platform for meanngful and responsve data analyss and decson makng. We propose a dstrbuted sensng resource dscovery and vrtualzaton algorthms that effcently deploy vrtual sensor networks on top of a subset of the selected IoT devces. We show, through analyss and smulatons, the potental of the proposed solutons to realze vrtual sensor networks wth mnmal physcal resources, reduced communcaton overhead, and low complexty. We also desgn an uncoordnated, dstrbuted algorthm that reles on the selected sensors to estmate a set of parameters wthout requrng synchronzaton among the sensors. Our smulatons show that the proposed estmaton algorthm, when compared to conventonal ADMM Alternatng Drecton Method of Multplers), reduces communcaton overhead sgnfcantly wthout compromsng the estmaton error. In addton, the convergence tme, though ncreases slghtly, s stll lnear as n the case of conventonal ADMM. I. INTRODUCTION Remote sensng applcatons wll evolve through ondemand sensng servces provded by the global network of sensor equpped devces n our homes, factores, ctes, and bodes known as the Internet of Thngs IoT). Today n smatphones only, there are seven sensors on average per devce ncludng: magnetometer, barometer, lght, heart, humdty, and temperature sensors that one can use as partcpatory sensors to carry out applcatons lke short-term weather forecastng [], [2]. The densty of smartphones sensors n London today exceeds 4,000 sensor per square klometer. By 2020, the global number of sensor-equpped and locaton-aware devces e.g. wearable, smart home, and fleet management devces) wll reach tens of Bllons, potentally creatng dense, dynamc, locaton-aware, and onerous to Copyrght c) 202 IEEE. Personal use of ths materal s permtted. However, permsson to use ths materal for any other purposes must be obtaned from the IEEE by sendng a request to pubspermssons@eee.org. The populaton densty n London exceeds 4,000 nhabtants per square klometer and the UK smartphone penetraton reaches 55%. manage networks of devces that can realze the vson of provdng a versatle remote sensng servces, known as Sensng as a Servce [3], [4], [5]. We conjecture that employng IoT devces sensng resources n a cloud computng lke platform to support remote sensng applcatons may be an effectve approach to realze the Sensng as a Servce vson [3]. The dea s to dynamcally augment and scale up exstng cloud resources compute, storage, and network) by explotng sensng capabltes of devces through cloud agents near the network edge to form a global system named the Cloud of Thngs see Fg. ). The Cloud of Thngs s a geographcally dstrbuted nfrastructure wth cloud agent elements that contnuously dscover and pool sensng resources of IoT devces to be used by cloud users on-demand. Ths nfrastructure provdes elastc sensng resources that scale up and down accordng to remote sensng applcatons demands, provdng an optmzed and controllable sensng resource utlzaton and prcng based on measurable usage. Cloud of Thngs shfts the current, conventonal remote sensng use of cloud platforms from a collect sensor data now and analyze t later scenaro to a usage scenaro that drectly provdes meanngful nformaton from n-network processng of sensng data by IoT devces. Wthout such a conjectured nfrastructure, remote sensng users can stll gan access to sensng resources through conventonal cloud back-end systems see [6], [4]), wth less opportuntes to scale out sensng applcatons over the globally avalable sensng resources and wth ntolerable performance to applcatons that requre responsve explotaton and fuson of sensng data and agle n-network decsons e.g. localzaton [7] and estmaton [8]). A. Cloud of Thngs Infrastructure Cloud platforms near the network edge already exst n dfferent forms such as smartphones, personal computers, gateways, and servers to offer computaton offloadng to nearby devces n real-tme e.g. cloudlets [9], [0] and edge computng platforms [], [2]). We envson a new role of edge platforms as cloud agents that ncorporate IoT devces as sensng resources Fg. ) to scale up the conventonal cloud wth global and locaton specfc sensng resources. We propose

2 Fg. : Sensor network vrtualzaton n Sensng as a Servce by dfferent cloud agents near the edge. Frst ter clouds are conventonal cloud computng platforms, and cloud agents are edge computng platform wth evolved rule for Sensng as a Servce. Arrows and numbers llustrate messages flow and sequence of the proposed usage scenaro. algorthmc solutons that provde: ) fast dscovery of devces dynamc sensng resources n specfc geographcal areas, 2) optmzed devce vrtualzaton to serve as vrtual sensor networks by explotng the dscovered sensng resources, and 3) effcent n-network processng of sensng data from unrelable but dense sensors n IoT devces. Cloud agents mplement remote sensng applcatons as vrtual sensor networks to be deployed on vrtualzable IoT devces n a geographcal area. A vrtual sensor network performs dstrbuted n-network processng of sensng data such as: aggregaton, feature extracton, belef propagaton, and consensus estmaton to serve applcatons such as: dstrbuted computer vson, data analytcs, or on-demand context awareness. These vrtual sensor networks may employ devces sensng resources that are dscovered by the varous multple cloud agents. Conventonal cloud platforms provde a unfed nterface to cloud users to seamlessly use such global sensng resources from anywhere and at anytme whle hdng complextes and supportng nteracton between cloud agents. In Cloud of Thngs, IoT devces become surrogates of federated sensor networks.e admnstrated by a sngle organzaton) that can potentally reduce the total cost of ownershp for remote sensng applcatons. However, the IoT devces usually ncorporate cheap and unrelable sensors to serve specfc task that s not ntended for remote sensng applcatons. For example, augmented realty applcatons n smartphones make use of the measurements from magnetc feld sensors. a) Magnetc feld sensor readng close to a wndow low energy envronment). b) Same sensor readng close to a power source hgh energy envronment). Fg. 2: Example of energy proflng from cheap magnetc feld sensor n smartphones. The same magnetc feld sensors can be used to profle energy levels n dfferent envronment see Fg. 2 for an example). The man problem wth remote sensng applcatons based on IoT devces sensors s that ndvdual measurements from the sensors of a sngle IoT devce e.g. magnetc feld) are nsuffcent for a relable sensng task. In Fg. 2, t s hard to dstngush the real context of a magnetc sensor readng changes; 2

3 whether t s a result of user proxmty to the devce, changes n the devce s orentaton, or presence n a hgh energy envronment. Smlar unrelablty problems appear n traffc congeston estmaton n navgaton applcatons e.g. Google maps), weather predcton from smartphones barometers, or ndoor localzaton. A vrtual sensor network of a group of ndependent IoT devces can solve ths problem through dstrbuted consensus estmaton. Instead of analyzng measurements from an ndvdual sensor, the vrtual network use ndependent sensor measurements from several devces and executes an effcent n-network dstrbuted consensus estmaton algorthm to be able to effcently acheve ts goal e.g. estmatng energy level n an envronment surroundng a user). In ths paper, we focus our analyss and evaluaton on dstrbuted consensus estmaton as the sensng task under study. B. Contrbuton and Organzaton In ths paper, we propose a system to perform nnetwork analytcs, such as dstrbuted parameter estmaton, based on commodty IoT devces that act as surrogates of wreless sensor networks.e. vrtual sensor networks). We desgn a vrtualzaton algorthm that suts the use case of descrbng analytcs as on-demand vrtual sensor networks and the challenges of the conjectured archtecture n Fg.. We also propose a dstrbuted consensus parameter estmaton algorthm to be executed by the optmzed vrtual sensor network. The dstrbuted consensus algorthm provdes a relable, hgh qualty parameter estmates from the low-qualty and unrelable sensors n commodty IoT devces. We dscuss the techncal challenges and our envsoned use case of the proposed archtecture n Secton II. In Secton III, we frst propose a sensng resource dscovery algorthm that uses a gossp polcy for propagatng a sensng task requrements to devces or ther vrtual nstance at the edge cloud) and selects feasble devces to execute the task whle respondng to the dynamc changes of devces as fast as possble. Then, we propose RADV, an effcent vrtualzaton algorthm, that deploys a vrtual sensor network correspondng to the sensng task on top of a subset of the selected devces wth mnmal physcal resources. In Secton IV, we propose RADE; an effcent estmaton algorthm that reles on the vrtual sensor network, formed by our proposed vrtualzaton algorthm, to estmate a set of unknown parameters n a dstrbuted way and wthout requrng synchronzaton among the IoT devces. We dscuss several related work n Secton VI. Fnally, we numercally evaluate our proposed algorthms n Secton V and conclude ths paper n Secton VII. II. ARCHITECTURE USABILITY AND CHALLENGES The proposed Cloud of Thngs archtecture allows cloud users to run remote sensng tasks, wth certan specfcatons, vrtually on any sensor-equpped IoT devces see Fg. ). For example, a cloud user can profle polluton changes n ctes from real-tme temperature and CO2 concentraton measurements collected by sensors n vehcles wth defned precson and accuracy. The archtecture conssts of three man elements: IoT devces, frst ter clouds, and cloud agents. IoT devces are sensor-equpped devces that can serve both specfc and general purpose remote sensng applcatons. Frst ter clouds are conventonal cloud platforms that provde unfed nterfaces to users to access the system and hde complextes underlyng the realzaton of sensng servces. Throughout, we refer to a frst ter cloud as smply cloud. Fnally, cloud agents are trusted and resource-rch elements near the network edge that are well-connected to the Internet and to conventonal cloud platforms. Cloud agents can be as powerful as supercomputers, or as flexble as smartphones accordng to the types of devces they serve and the computng resources these devces demand. Throughout, we refer to a cloud agent smply as agent. Ths archtecture offers new sensng features and servce level guarantees wth several benefts. Deployng agents cloud agents) close to devces mproves responsveness to sensng task requests and enables access to a globally avalable sensng resources. From the devces vewpont, cloud resources can be splt nto local resources agents resources) and global resources clouds resources) that can mprove reslency by mgratng sensng tasks as the states of the devces - whch carry out the sensng task - change. A cloud frst ter cloud) also acts as lason to support coordnaton between dstrbuted agents, whle these agents can rapdly capture dynamcs of the devces e.g. utlzaton, connectvty, and avalablty). Ths approach smplfes analytcs and bg data wth possble drect devce access for agle n-network data processng and decson makng. The proposed archtecture fnally allows the desgn of network-aware and performanceoptmzed cloud procedures. A. Use case and System Model Fg. summarzes message sequence and flow between the dfferent archtectural elements. These are detaled as next. ) Frst ter clouds: A cloud frst ter cloud) handles sensng tasks ntated by a user wth a unfed nterface step n Fg. ). A sensng task defnes physcal parameters e.g polluton changes) that the user wshes to estmate n a defned geographcal area durng a predefned tme wth certan sensng capabltes of the IoT devces carryng out the task. The sensng task objectve can be: nformaton retreval of raw sensed data, or executon of dstrbuted algorthms on a vrtual sensor network deployed on multple nterconnected devces. We represent a sensng task by the trple, g, c, δ, where g denote the number of vrtual sensors requested to perform the sensng task and the two parameters c and δ defne the center and the radus of a geographcal area of nterest to the user s remote sensng applcaton. 3

4 2) Sensng task requests: The cloud translates a sensng task to a correspondng sensng task request that t sends to ts agents. A sensng task request defnes the vrtual sensors set, V, to be deployed on g connected devces, whch are all located wthn dstance δ from the area center c. For each vrtual sensor j V, the cloud defnes a mnmum sensng capablty, Rj). The mnmum sensng capablty represents the mnmum storage capacty, the mnmum CPU computng power, and/or the mnmum amount of tme that devces to carry out the sensng task) must fulfll. The cloud may also choose a sutable vrtual topology that nterconnects the vrtual sensors so that they can execute dstrbuted algorthms for n-network processng of devces sensed measurements. For example, for aggregaton and belef propagaton algorthms, the cloud organzes the vrtual sensor network as a spannng tree. A star topology can also be adopted for dstrbuted algorthms that are mplemented usng the map-reduce or graph-processng paradgms. Although consensus algorthms, whch are our man focus, can run wth any arbtrary topology, we show that a complete topology results n faster convergence. For our evaluaton, we focus on three common vrtual topologes: complete, cyclc, and star. For a gven topology, let E denote the set of vrtual lnks connectng the vrtual sensors and Υ = V, E) be the graph data structure that represents the vrtual sensor network of the vrtual sensors connected accordng to the gven vrtual topology). After translatng a sensng task to ts correspondng sensng task request, the cloud sends ths request.e. the graph data structure Υ) to ts agents step 2 n Fg. ). 3) The IoT devces capabltes: Agents manage a large number of nterconnected IoT devces. A devce, at tme t, mantans ts geographcal locaton denoted by loc) and ts current sensng capablty denoted by C). C) defnes the currently allowed sensng tme, avalable processng capacty, and/or avalable memory capacty that the -th devce can allocate at tme t) to fulfll the mnmum sensng capablty demanded by a vrtual sensor j.e. Rj)) to be deployed on. The sensng capablty of a devce can correspond to local devce s resources.e. CPU, memory, storage, and sensors) or to resources at the edge cloud agent) that the devce may opportunstcally use through computaton offloadng mechansms. We also assume that two devces can drectly communcate wth each other f they are wthn a transmsson radus r. We model the network of all n devces, connected to a sngle agent, as the Eucldean geometrc random graph, G = S, L), where S denote the set of n devces, and L denote the set of all lnks connectng the devces. We assume that each sensor S s capable of estmatng a vector of unknown parameters, θ R N, through nosy sensors measurements, x R M. That s, x = H θ + u, =,..., n where H R M N s sensor s sensng model typcally known to only) relatng x to θ, and u s an addtve Gaussan nose wth zero mean and varance σ 2. We assume that u and u j are ndependent from one another for all, j S. Because dfferent sensors may have dfferent sensng models and/or dfferent measurement methods, t s very lkely that dfferent sensors have dfferent estmates of θ. Also, we do not assume/requre that the sensors are synchronzed; that s, the consensus algorthms we develop n ths paper to estmate θ are asynchronous. 4) Servce Level Agreement SLA) mplcatons on agents confguratons: Agents handle sensng task requests under agreed SLAs wth users through the cloud. An SLA generally conssts of: ) a maxmum tme wthn whch the sensng task must be completed, ) a feasble selecton of IoT devces to carry out the sensng task under certan tolerances of the results accuracy, and ) a maxmum task rejecton rate defned as the rato of the number of falures to handle sensng task requests to the total number of requests. The cloud translates an SLA to parameters that agents can use n ther algorthmc solutons to dscover sensng resources and vrtualze devces effcently. Defnng all possble parameters that reflect any SLA s beyond the scope of ths work and we consder only four parameters. The frst parameter s the absolute error of the estmated parameter θ, denoted by ɛ abs. The second parameter s the relatve error, ɛ rel, of the parameter θ estmated by the dfferent vrtual sensors such that all sensors estmate θ wthn ɛ rel. The thrd parameter, defned earler, s the mnmum sensng capablty Rj) of the j-th vrtual sensor. The fourth parameter s the maxmum allowed path length, h, between any par of vrtual sensors. h lmts the number of devces/hops a message, exchanged between vrtual sensors, can go through. A vrtual lnk between two vrtual sensors may map to devces that do not necessarly deploy a vrtual sensor and the vrtual sensor network just use these devces for message forwardng. We use h to mpose an upper lmt on these ntermedate devces for two reasons. Frst, restrctng the number of ntermedate devces shall bound the sensng task performance by an SLA. Second, the sensor dscovery and vrtualzaton algorthms shall use the least number of hops and the least possble physcal resources when mappng the vrtual network, so as to maxmze the Sensng-as-a- Servce benefts step 3 n Fg. ). The mplcaton of h on the vrtualzaton desgn wll be dscussed later n ths secton. 5) Sensng task executon consensus): Consensus estmaton resembles the most commonly used sensng task relyng on a collecton of measurements from unrelable sensors. In consensus estmaton, the sensng task s to estmate a set of parameters, θ, based on the measurements sensed by the IoT devces so that the estmated parameters are at most ɛ abs away from ther actual value, and so that all the sensors consent 4

5 to the same estmate value of θ wth a tolerance of ɛ rel accordng to the SLA. Wthout loss of generalty, consder ndexng the selected g sensors n the vrtual sensor network as... g and let x = [ ] T, [ ] x T,..., x T g H = H T T,,..., Hg T and u = [ ] T. u T,..., u T g The combned measurements can then be wrtten as x = Hθ + u. One smple approach of estmatng θ s to have the cloud agent frst collect from each vrtual sensor ts measurement vector, x, and ts sensng model, H, and then solve the followng Least Squares LS) problem mnmze 2 x H ˆθ 2 ) where ˆθ s here the optmzaton varable. The unbased maxmum-lkelhood ML) estmate of θ s smply ˆθ LS = H T H ) H T x. B. Techncal Challenges and Solutons Objectves The proposed Cloud of Thngs archtecture and use case envson desgnng algorthmc solutons wth specfc objectves, gven the followng set of challenges: ) Sensng resource dscovery: In the sensor network vrtualzaton step 3 n Fg. ), an agent searches for devces wth sensng capabltes that meet the sensng task requrements specfed by the vrtual sensor network data structure Υ. For a gven Υ, the agent dscovers devces sensng capabltes and searches for a subset of devces, S S, such that a devce S, f t s geographcally located wthn δ dstance from the center c, and the dscovered sensng capablty C) satsfes the mnmum sensng capablty Rj) demanded by at least one vrtual sensor j V. We defne the vrtual doman, D), of a devce as D) = {j V : C) Rj), loc) c δ} 2) hence S = { S : D) > 0}. 3) The desgn objectve of a sensng resource dscovery algorthm s to construct the vrtual domans, D) for all S, as fast as possble and wth mnmal communcaton overhead between the agent and the devces and between the devces themselves. The challenges related to sensng resource dscovery arse from the large number of devces and ther onerous to mantan dynamcs. The large number of devces connected to an agent requres a scalable soluton to dscover devces sensng capabltes and to decde f a devce s current state e.g. connectvty to other devces) allows t to deploy a partcular vrtual sensor. Moreover, the dynamcs and rapd changes n the whole network, G, ncludng devce avalablty, moblty, connectvty, and resource utlzaton, make t too dffcult to mantan devces states n a centralzed manner. To address these challenges, we propose a dstrbuted algorthm that propagates the graph data structure Υ to devces n G usng a gossp polcy as detaled n Secton III-A. 2) Vrtualzaton: After performng the sensng resource dscovery, an agent deploys the vrtual sensor network, Υ, by means of devces vrtualzaton. The vrtualzaton task conssts of fndng: ) a set A S of exactly g connected devces accordng to the vrtual topology chosen by the cloud, and ) a set M A {, j) A V : j D)} of devce,vrtual sensor) mappng pars such that one vrtual sensor maps to exactly one devce and a devce maps to one and only one vrtual sensor n g. Also, the length h, ) of any smple path connectng two dstnct devces, A that maps a vrtual lnk j, j ) E must be less than or equal to h. We refer to a {A, M A } par that satsfes the prevous condtons as a feasble vrtualzaton of the requested vrtual sensor network Υ. Note that for any possble set A, there can exst multple mappngs, M A, and each can form a feasble vrtulzaton. The desgn objectve of a vrtualzaton algorthm s then to fnd the optmal feasble vrtualzaton, {A, M A }, that uses the least possble physcal network resources. We now defne and ntroduce what an optmal feasble vrtualzaton means. We consder that the number of vrtual sensors and the number of vrtual lnks of a gven Υ = V, E) determne the cloud cost of provdng the sensng servce, whch s gven by CostΥ) = α V + β E. The scalar α denote an ncentve pad to each devce that maps a vrtual sensor, and the scalar β denote an ncentve dvded and pad to each devce on a physcal path that maps to a vrtual lnk. An ncentve could be monetary or could be n any other form e.g., credt, servce, etc.). On the other hand, the total devces beneft from mappng the requested vrtual network, Υ, can be expressed as Beneft =,j) M A α C) Rj) + C), ) P β h h, ), h 4) where h, ) s agan the length n number of hops) of the path connectng the devce par,, ), mappng the vrtual lnk between j and j, and P = {, ) A A :, j),, j ) M A, j, j ) E}. The total devces beneft n 4) mples that the lesser the used physcal resources, the greater the beneft to the devces. The frst term of 4) captures the beneft loss of the -th devce from allocatng resources to map a vrtual sensor j. As the mnmum demanded sensng capablty Rj) becomes neglgble w.r.t. the sensng capablty C)), gets hgher beneft as t nvests lesser fracton of ts resources e.g. energy, CPU, or memory) to map j for the same ncentve α. Smlarly, the second term captures the beneft loss of devces and, whch map the vrtual sensors j and j respectvely. Such beneft loss results from mappng the vrtual lnk between j and j wth more ntermedate devces, as the same ncentve β for the vrtual lnk j, j ) s dvded on a greater number of devces.e. number of hops h, )) compared to h. Theoretcally, h can take a value up to the dameter of G. However, ths shall not work n practce as the dameter of G s assumed to 5

6 be much greater than a user desred dameter Υ. The vrtualzaton algorthm that we propose n Secton III-B conssts of fndng an optmal feasble vrtualzaton that maxmzes the total beneft gven n 4). We refer to the optmal soluton as {A, M A }. Clearly, fndng {A, M A } s hard due to the factoral sze of the soluton space n n and due to the challenges, dscussed earler, assocated wth the sensng resources dscovery task. 3) Dstrbuted consensus estmaton: The vrtual sensor network determned durng the vrtualzaton phase executes the dstrbuted sensng algorthms. The smple soluton to the LS problem, proposed n ), requres that each vrtual sensor exchanges ts measurement vector and ts sensng model wth the cloud agent, whch can create sgnfcant communcaton overhead. Therefore, we nstead propose a decentralzed approach that reles on the vrtual sensor network to provde an estmaton of the parameter vector θ. We rely on the recent results presented n [3] to develop our dstrbuted estmaton algorthm, whch reduces communcaton overhead sgnfcantly when compared to the conventonal Alternatng Drecton Method of Multplers ADMM) approach [4] n addton to not requrng synchronzaton among sensors. The proposed algorthm s presented n Secton IV. III. PROPOSED SOLUTIONS FOR SENSING RESOURCE DISCOVERY AND VIRTUALIZATION A. Sensng Resource Dscovery Although devces are drectly accessble by cloud agents, contactng the devces at fne-graned tme slots to dscover ther current sensng capabltes creates sgnfcant communcaton and computaton neffcency for large n. Such a centralzed approach requres exchangng On) messages, n each tme slot, whle constructng the vrtual domans, gven by 2), requres On) tme. Moreover, actvatng devces perodcally to update ther current sensng capabltes to ther cloud agents s power neffcent, especally f the devces are battery operated. We propose to perform sensng resource dscovery through a gossp based algorthm that requres a tme complexty of Or log n) and an average Θ) messages per devce. In ths algorthm, an agent propagates nformaton about a receved sensng task request, Υ, usng the followng gossp polcy. The agent sends Υ to a randomly chosen devce startng at t = 0. Then, any devce that receves Υ contnues sendng Υ to a random devce of ts drect neghbors untl one neghbor acknowledges that t has already double receved the same verson of Υ n a prevous step; by then the devce stops sendng Υ. The agent does not need to send Υ to each devce as the utlzed gossp polcy allows devces to dssemnate Υ autonomously, and the network of devces s guaranteed to be connected wth hgh probablty f each devce s connected to k neghbors and k log n [5]. whle True do wat t s random neghbor f Υ s then solct Υ from s else send Υ to s end f receve Υ from s f Υ = Υ then stop sendng Υ else Υ s newer than Υ Υ Υ evaluate D) end f end whle ) actve thread at devce whle True do receve Υ or solct request from s f Υ s not then send Υ to s end f f Υ s new then Υ Υ evaluate D) end f end whle ) passve thread at Fg. 3: proposed sensng resources dscovery gossp based threads at devce. Snce G s a connected network, ths smple gossp polcy guarantees that Υ reaches all the devces n Or log n) tme see [6] for tme complexty analyss of general gossp protocols n Eucldean geometrc random graphs). Hence a devce can construct D) accordng to 2) once t receves Υ and the agent can dscover sensng resources of devces that are capable of satsfyng the requrements of Υ as fast as possble wth mnmal communcaton overhead. The agent and all ts connected devces mplement the actve and passve threads shown n Fg. 3. At the k-th tme slot, let the devce be actve and contact a random neghbor devce.e.,, ) L) wth probablty T, > 0. T, denote the probablty that does not contact any other devce. Let the n n matrx T = [T, ] be a doubly stochastc transton matrx of non-negatve entres [7]. A natural choce of T, s T, = d +, f = or, ) L, 5) 0, otherwse, where d = { S :, ) L} s the degree of. When contacts, they exchange nformaton as follows see Fg. 3 ). pushes Υ to only f does not have Υ, or pulls Υ from only f does not have Υ. If contacts and both devces have receved Υ before, stops contactng any other devce. If G s connected, the proposed protocol guarantees that Υ s delvered to all IoT devces. The actual runnng tme of the proposed algorthm depends on the choce of the transton matrx T and the communcaton range of the used devce-to-devce 6

7 communcaton technology. The runnng tme s related to the mxng tme of any random walk on G [7], whch suggests that there s an optmal value of T, to mnmze the mxng tme and t s related to the second egenvalue of the transton matrx. Moreover, n case of small r, the proposed algorthm s generally slow. Practcally, ths algorthm s sutable for devce-todevce communcaton technologes that support communcaton ranges of few hundreds of meters, as n WF drect and LTE D2D and when G s suffcently dense. B. Vrtualzaton We present our proposed Randomzed and Asynchronous Dstrbuted Vrtualzaton RADV) algorthm. RADV conssts of three phases: I) prunng of vrtual domans D) for all S, II) constructon of beneft matrces n a dstrbuted manner, and III) solvng assgnment problems at vrtual sensors. RADV results n multple solutons each evaluated by a dfferent sensor devce), and the cloud agent selects the soluton wth the maxmum beneft. Phase I Vrtual Doman Prunng: Durng ths phase, we ensure that all vrtualzed sensors mantan the topology E by allowng a senor to receve the vrtual domans of other sensors and delete a vrtual sensor j from ts doman f there exsts a vrtual lnk j, j ) such that j s not ncluded n any other receved domans. Let D s {D) : S} denote the vrtual domans set that sensor s has at tme k. Intally D s = {Ds)} and h, s) = 0 for all S 2. Usng the same transton matrx, T, defned n Eq. 5), s contacts only one of ts neghbors s at tme k. Then, for all D) D s : s, s pushes D) to s only f s dd not receve D) before and h, s) < h. Also, for all D) D s : s, s pulls D) from s only f s dd not receve D) before and h, s ) < h. If no nformaton s exchanged between s and s at tme k, s stops contactng any of ts neghbors. However, s may restart contactng ts neghbors agan f t updated D s after tme k +. When s constructs ts D s, t starts by prunng Ds). The prunng s performed by deletng a vrtual sensor j Ds).e., Ds) Ds) \ {j}) f none of the vrtual sensors that are connected to j, {j V : j, j ) E}, s not ncluded n any receved D),.e. j / D) : D) D s. Ths prunng rule ensures that the vrtualzed sensors mantan the requred topology E and the constructed beneft matrces shall result n a feasble vrtualzaton. Phase II Constructon of Beneft Matrces: As mentoned earler, fndng a feasble vrtualzaton, {A, M A }, that maxmzes the total beneft gven n Eq. 4) s a hard problem due to the large sze of the soluton space. Therefore, ths phase proposes an effcent way of solvng ths vrtualzaton problem. Specfcally, we propose a method that solves ths 2 Knowledge about other sensors exstence s not needed, and h s typcally evaluated dynamcally. problem n a dstrbuted manner and wthout requrng any synchronzaton among sensors, as descrbed next. Durng ths phase, each sensor s locally constructs ts own set, A s), of g sensors that s chooses as vrtualzed sensors to assgn to vrtual sensors n V. Each sensor s also mantans g row vectors, B s) R g and A s), that we defne as the beneft vector of sensor seen by s, where the j-th element, B s),j, denotes the beneft of assgnng partcpatory sensor A s) to the vrtual senor j V as seen by s, and s gven by C) Rj) B s),j = α + β h hj, s) f j D), C) h 0 otherwse. Our objectve s then to construct, for each s S, the beneft matrx B s) = [B s) ] as fast as possble, and A s) fnd a feasble vrtualzaton, {A, M A }, that maxmzes the total beneft, B s),j,,j) M A among all s S wthout knowng the G structure. Moreover, the path length between a sensor s and any other sensor that s ncludes n ts beneft matrx must not exceed h. Fnally, a sensor s shall nclude only the beneft vectors of the g sensors wth the largest possble beneft. Each sensor s ntally sets A s) = A s) {s} f Ds) /, sets h, s) = 0 for all S, and sets B s) s,j = Cs) Rj) α + β, j Ds), Cs) 0, otherwse. Also, s mantans a scalar, b mn s, defned as the mnmum total beneft t has receved from any other sensor and wrtten as b mn s = mn B s),j. j V s also mantans the correspondng sensor, mn s = argmn B s),j. j V j V Intally, b mn s = 0 and remans so untl A s) = g. Usng the same transton matrx, T, defned n Eq. 5), s contacts ts neghbor s only once at each tme k. Then, for all A s) : s, s pushes the beneft vector B s) to s only f h, s) < h and B s),j β h ) > b mn s. Also, for all A s ) : s, s pulls the beneft vector B s ) from s only f h, s ) < h and B s ),j β h ) > b mn s. j V 7

8 If no nformaton s exchanged between s and s at tme k, s stops contactng ts neghbors at tme k +. However, s may restart contactng ts neghbors agan f B s) s updated after tme k +. When s receves B s ), s updates B s),j as B s),j = B s),j β h f j D), 0 otherwse. If / A s), then we have two scenaros. In the frst scenaro, s stll has not receved g beneft vectors, so b mn s = 0 and A s) < g, then s updates ts set of canddate sensors as A s) = A s) {}. In the other scenaro n whch A s) = g, s replaces the sensor correspondng to the mnmum total beneft, mn s, wth so that A s) = A s) \ { mn s } {}. On the other hand, f A s), then s updates B s),j f >. Fnally, s updates bmn s, mn s, and j V B s ),j j V B s),j h, s) as h, s) = h, s ) +. Fndng a feasble vrtualzaton that maxmzes the beneft B s) = [ B s) ] nstead of the beneft gven A n Eq. 4) makes the s) problem easer because every sensor has a dfferent value for the beneft B,j that depends only on the length of the physcal path between and s nstead of the path lengths of all possble combnatons of sensor pars, ) that can vrtualze a vrtual lnk. Intutvely, ths relaxaton stll leads to an optmal or near optmal vrtualzaton, because f G s very large and connected, the number of sensors that are drectly connected by a sngle physcal lnk clque) grows logarthmcally n n and hence ths number s larger than g almost surely as g n. In such a case, t s suffcent to ensure that the length of the paths between and s and between and s are the shortest possble ones to ensure that the length of the path between and s also the shortest, as n ths case, s,, and resde n the same clque wth hgh probablty. We evaluate the effectveness of ths relaxaton n Secton V and show that our vrtualzaton algorthm performs well even when the condton g n does not hold. Phase III Solvng Local Assgnment Problem: After recepton of the g beneft vectors, s proceeds to ths phase of the algorthm only f t stops communcatng and A s) = g. Each sensor s S wth A s) = g solves locally the followng assgnment problem: maxmze B s),j m j A s) j D) subject to m j =, A s), j D) 6) m j =, j V, {:j D)} m j {0, }, where m j are bnary optmzaton varables ndcatng whether the partcpatory sensor s assgned to the vrtual sensor j. The problem formulated n 6) s equvalent to the perfect maxmum weght matchng problem n a bpartte graph, and hence, we propose to use the classcal Hungaran method to solve t the worst case tme complexty s Og 3 ) [8], [9]). We can also tolerate an error ɛ > 0 of the resultng total beneft and relax the restrcton of fndng a perfect matchng for large g. Ths relaxaton s reasonable when there are enough sensors nvolved n solvng these local optmzaton problems, as n ths case we can pck the best soluton and dscard those wthout a perfect matchng. In such a scenaro, we can also use a lnear tme ɛ)-approxmaton algorthm to solve 6) [20]. In ths paper, we use the Hungaran method to solve our formulated optmzaton problems. Detals of the algorthm are omtted due to space lmtaton; readers are referred to [8], [9], [20] for detaled nformaton. Each sensor solves locally the optmzaton problem gven n 6) and sends ts obtaned soluton to the cloud agent. Ths s done asynchronously. The cloud agent then selects the soluton that leads to the maxmum total beneft, and keeps all other solutons for later use n the event that the network dynamcs nvaldate the selected soluton before the vrtual sensng task completes. Complexty and message overhead. We assume that the topology of G, devces moblty, and sensng capablty are not changed durng the executon of the vrtualzaton phase. The tme requred to spread Υ across the network s Or log n) [6]. It takes Og) worst case tme to evaluate D) locally at sensor. Also, the tme requred to spread nformaton n the prunng and beneft constructon phases s Or n log n). The prunng of the vrtual doman D) requres node to examne g receved vrtual domans, each havng at most g entres. The worst case local runnng tme of prunng s then Og 2 ). Fnally, the local runnng tme of the Hungaran method s Og 3 ). Hence, the overall complexty s Omax{r n log n, g 3 }). The average number of messages communcated per sensor durng the sensor search phase s Θ) and each message s Og) n sze. Durng prunng of vrtual domans, snce every sensor exchanges a maxmum of n domans each of sze that s also Og), the average number of messages communcated per sensor s On). However, because we restrct that messages to be communcated up to h hops for only a group of sensors that support the requrements of Υ, the average number of messages per sensor s typcally small. Fg. 4 shows the total tme and the average number of messages per sensor requred durng both the doman prunng and the beneft constructon phases. The total tme growth s lnearthmc n n when Υ s sent to exactly one sensor and when G s connected. Ths tme can, n practce, be decreased sgnfcantly f Υ s ntally sent to multple sensors. Addtonally, the average number of messages per sensor s shown to scale lnearly wth n, and s typcally a very small fracton of n. 8

9 Tme n Messages Tme Fg. 4: Tme n number of teratons) and message overhead n average number of communcated messages) resultng from constructng the beneft matrces, g = 0. IV. PROPOSED SOLUTIONS FOR DISTRIBUTED ESTIMATION After completng the sensor vrtualzaton task, usng RADV, the vrtual sensors run an n-network parameter estmaton algorthm to compute ˆθ dstrbutedly. In ths secton, we present our proposed Randomzed and Asynchronous Dstrbuted Estmaton RADE) algorthm. We frst follow the standard ADMM approach to derve prmal, dual and Lagrangan varable update equatons, then we descrbe the proposed RADE algorthm. For clarty of notaton, n what follows, we refer to the set of g selected devces, determned by RADV, smply as A. The centralzed estmaton approach gven n ) s frst decomposed nto g local estmates of θ one ˆθ for each A) whle constranng the local estmates wth the couplng constrants θ = θ j for all, j) P. Ths results n the followng optmzaton problem: mnmze 2 x H θ 2 A subject to θ θ j = 0 for all, j) P, where {θ, A} are the optmzaton varables. By ntroducng an auxlary varable, z, we decouple the constrants n 7), so that θ z = 0 for all A [2]. However, ths requres that z be shared among all the g vrtual sensors. Instead, we ntroduce g auxlary varables, z, and equvalently wrte the optmzaton problem as mnmze 2 x H θ 2 A subject to θ j z = 0 for all, j) P Average No. of Messages 7) 8) Let λ = {λ,j R N :, j) P } and ρ = {ρ,j R :, j) P } denote respectvely the set of Lagrangan multplers and the set of penalty parameters. The augmented Lagrangan s gven by L ρ θ, z, λ) = [ 2 x H θ 2 A + j A:,j) P j A:,j) P λ T,j θ z j ) ρ,j 2 θ z j 2 9) By settng the gradent w.r.t θ of Eq. 9) to zero and solvng for θ, we get ) θ = H TH + ρ,j I j A:,j) P ). H Tx + λ,j + ρ,j z j ). j A:,j) P Smlarly, we solve for z by settng the gradent w.r.t to z to zero and rearrangng the ndces of the Lagrangan multplers and the penalty parameters. It follows that z = g j A:,j) P θ j ρ j, λ j, ). The former analyss leads to the conventonal ADMM-based dstrbuted consensus estmaton algorthm gven by ) θ k+) H TH + ρ,j I = z k+). H T x + = g j A:,j) P j A:,j) P j A:,j) P λ k),j + ρ,jz k) j ) θ k) j ρ j, λ k) j, ) θ k+) j, ]. ) ), λ k+) j, = λ k) j, ρ j, z k+), 0) where the superscrpt k denotes the value of the varable at the k-th teraton. Ths conventonal ADMM algorthm, gven n 0), requres synchronzaton and varable update among the sensors [22], [23]. Moreover, at each teraton k, each sensor must send z k) and θ k) to all other sensors t s connected to, so as to evaluate ther k + prmal, dual, and Lagrangan multplers. When M s small, ths algorthm ncurs communcaton overhead that can be shown to be worse than the communcaton overhead ncurred by centralzed estmaton methods. However, when M s large, the conventonal ADMM algorthm ncurs lesser communcaton overhead than what centralzed estmaton methods ncur, but t stll remans practcally unattractve due to other weaknesses, detaled later n Secton V. Gven the absolute and relatve tolerances, ɛ abs and ɛ rel, specfed by the SLAs, we defne the prmal and dual tolerances, controllng the convergence of the algorthm at teraton k, as ɛ pr k) = gɛ abs + ɛ rel max θ k), z k) ), 9

10 and ɛ dual k) = gɛ abs + ɛ rel ρ j, λ j,. j A The tolerances, ɛ pr and ɛ dual, defne the stoppng crtera of sensor ;.e., sensor stops updatng θ and z when and θ k+) z k+) z k+) < ɛ pr k), ) z k) < ɛ dual k). 2) The stoppng crtera of RADE are dfferent from those of the conventonal ADMM. Unlke the conventonal ADMM where all sensors shall stop computatons all at the same tme usng a common stoppng crtera and common prmal and dual tolerances, the stoppng crtera Eq. ) and Eq. 2)) of RADE allow a sensor to stop ts computatons asynchronously and ndependently from other sensors. However, these crtera are not enough to ensure asynchronous mplementaton, as synchronzaton s stll requred for dual and prmal varable updates at teraton k + due to ther dependences on k. To ensure full asynchronous mplementaton, we use the doubly stochastc transton matrx, T R g g, where T,j s the probablty that a sensor contacts another sensor j at any teraton, for decdng the communcatons among sensors. We can have T,j = d + f = j or, j) P, 0 otherwse, where d = {j A :, j) P } s the degree of the vrtual sensor, n Υ, that vrtualzes. At teraton k +, sensor may need to contact only one sensor j, unless both of s stoppng crtera, Eq. ) and Eq. 2), are already satsfed. Whereas sensor j can be contacted by more than one sensor f j s not contactng any other sensor, even when both of j s stoppng crtera are satsfed. Upon contactng j, sensor pushes θ k) to j only f s prmal stoppng condton s not satsfed and pushes z k) to j only f s dual stoppng condton s not satsfed. Also, pulls θ k) j from j only f j s prmal stoppng condton s not satsfed and pulls z k) j only f j s dual stoppng condton s not satsfed. Fnally, both and j update ther k + varables usng the most recent values they receved from other sensors. Mean square error and convergence. The asynchronousness and randomzaton desgn of RADE do not mpact the Mean Square Error MSE) acheved by RADE when compared to ADMM. Ths s explaned as follows. In both ADMM and RADE, the number of necessary dual and prmal varables updates that are needed untl convergence remans unchanged, so that convergence to the same estmate s guaranteed n both algorthms. Fg. 5 shows the MSE achevable under both MSE 0. LS ADMM-Complete RADE-Complete RADE-Cyclc RADE-Star Nose Power db) Fg. 5: MSE of RADE compared to those acheved under ADMM and LS at dfferent nose power and for dfferent vrtual sensor network topologes, g = 0. RADE and ADMM when compared to LS under each of the three studed sensor network topologes: complete, star, and cycle. These results show the optmalty of RADE that we ntutvely dscussed. All approaches have the same accuracy. But of course each of them does so at a dfferent performance cost, as wll be dscussed later. On the other hand, RADE exhbts a lnear convergence rate O/k)), smlar to what the conventonal ADMM does. Fg. 6 shows the number of tme steps requred for both RADE and ADMM to converge under dfferent relatve tolerance parameters, ɛ rel. RADE convergence tends to be more restrcted by the randomzaton nature of the algorthm for smaller values of ɛ rel, whch can be seen by the ncreasng number of steps as g ncreases f ɛ rel = 0 2. ADMM generally requres a lesser number of steps to converge by relaxng the consensus constrant through reducng ɛ rel ). However, as wll be seen n the numercal results secton later, ths ncrease n the number of convergence steps s acceptable when consderng the amount of communcaton overhead that the algorthm saves. V. NUMERICAL RESULTS In ths secton, we evaluate the performance of the proposed RADV and RADE algorthms through smulatons. In our smulatons, G and Υ, are generated usng the parameters summarzed n Table I. We evaluate the performance for a complete, cyclc, or star vrtual sensor network topology, wth a randomly chosen central locaton, c. We consder recevng and servcng only one vrtual sensng task request at a tme. The G topology and connectvty can change rapdly. For a sngle Υ, we assume that the network change rate s slow enough for the completon of the sensor search and vrtualzaton phases. The absolute and relatve tolerances, ɛ abs and ɛ rel, are set to 0 4 unless specfed otherwse. Fg. 7 shows the rejecton rate encountered wth dfferent Υ topologes and n values. As we only consder 0

11 k g ADMM,0-4 RADE, 0-4 ADMM, 0-2 RADE, 0-2 Total Beneft - Cost Complete Bound - Complete Cyclc 6 Bound - Cyclc Star Bound - Star n Fg. 6: Number of tme steps k) needed untl convergence of RADE when compared to ADMM for complete topology under dfferent relatve tolerance values ɛ rel. Fg. 8: Vrtualzaton cost of RADV when compared to the upper bound under dfferent topologes. 000 TABLE I: Smulaton Parameters Parameter g r C) Rj) δ h 00 Value 0 0. U50, 00) U25, 50) k one sngle request at a tme, the results shown n ths fgure reflect manly the mpact of the vrtual sensor network topology, the number of sensors n, and the smulatons parameters gven n Table I on the rejecton rate. The denser the network of IoT devces s, the lower the rejecton rate, mplyng that the cloud s capable of grantng hgher number of requests. One way of assessng the effectveness of the vrtualzaton algorthm s by measurng the dfference between the total vrtualzaton beneft gven n 4) and the cost assocated wth the sensor vrtualzaton ntroduced n Secton III-B. For a gven number of vrtual sensors, the cost s manly determned by the choce of the topology star topology has the lowest cost and complete topology has the hghest one). For Rejecton Rate n RADV-Complete RADV-Cyclc RADV-Star Fg. 7: Rejecton rate encountered at dfferent n. 0 ADMM-Complete RADE-Complete ADMM-Cyclc RADE-Cyclc ADMM-Star RADE-Star Fg. 9: Number of tme steps untl convergence of RADE when compared to ADMM under dfferent topologes. a gven topology, the total beneft s maxmzed when each vrtual sensor s assgned to the IoT devces wth the maxmum capacty and each vrtual lnk s mapped to exactly one physcal lnk. We refer to ths maxmzed beneft as the upper bound. In Fg. 8, we evaluate the vrtualzaton effectveness acheved by RADV under dfferent vrtual topologes. As the network gets denser, RADV acheves a Total Beneft Cost that s very close to the upper bound. Snce the lowest possble vrtualzaton cost s wth star or cyclc topologes, t s desred by the cloud to arrange each vrtual sensng task n a star or a cyclc topology. Ths observaton holds true for a more general topologes. On the other hand, convergence and communcaton overhead of the dstrbuted estmaton s also mpacted by the cloud agent s choce of the vrtual topology. Ths creates a desgn trade-off, as we wll see n the next two paragraphs. Fg. 9 shows the mpact of the vrtual topology choce on the convergence performance of RADE when compared to ADMM. If g s small three to eght), the g

12 Average No. of Messages e LS ADMM-Complete RADE-Complete ADMM-Cyclc RADE-Cyclc ADMM-Star RADE-Star Fg. 0: Communcaton overhead when comparng RADE to ADMM and LS under dfferent g values. mpact of the vrtual topology on convergence of RADE and ADMM s mnmal. Ths s because the degree of parallelsm number of vrtual sensors actve at the same tme) s restrcted by the small number of vrtual sensors g. In such a scenaro, t s convenent for the cloud agent to always arrange the vrtual sensors n a star topology. However, as g ncreases, the mpact of choce of the vrtual topology becomes sgnfcant as the degree of parallelsm s hgher n a complete topology, enablng RADE to converge much faster as g gets larger. Ths convergence becomes slower wth star and cyclc topologes. Ths s because n star and cyclc topologes, only few sensors are actve at a tme, makng RADE and ADMM converge n a number of steps comparable to that of the ADMM s sequental mplementaton. In ths later scenaro, the cloud agent shall arrange the vrtual sensors as a complete topology unless the SLA permts slower convergence. Moreover, RADE converges n a hgher number of steps when compared to the conventonal ADMM. Ths s because n ADMM, all sensors are actve at each tme, and a sensor exchanges ts updated varables wth all of ts neghbors, whereas n RADE, only dsjont sensor pars are actve at a tme and varables are updated only between pars of sensors. Nevertheless, we argue that ths loss n speed of convergence for RADE s margnal when compared to the sgnfcant savngs n communcaton overhead. Fg. 0 shows the total number of ON) szed messages exchanged durng estmaton when comparng RADE, ADMM, and LS for M = 00. The number of messages exchanged by RADE s at least an order of magntude less than the number of messages generated under ADMM. Also the communcaton overhead of RADE s less than the centralzed LS especally as M becomes large. Ths savngs n communcaton overhead s attrbuted to the asynchronous desgn of RADE n whch messages among sensors are only exchanged f new values of a prmal or dual varables are changed away from ther specfed tolerances. g VI. A. Network Vrtualzaton RELATED WORK Network vrtualzaton technques proposed n the past decade consst manly of vrtual network embeddng algorthms, whch nstantate vrtual networks on substrate nfrastructures [24], [25]. Most of these vrtual network embeddng algorthms are centralzed e.g. [5], [26]) due to the ease of deployment of centralzed approaches n cloud platforms where the cloud provder desres to have full control on the physcal network resources. Dstrbuted network vrtualzaton technques are suted for Cloud of Thngs, gven the sze of the network, and are also proposed for applcatons n resource allocaton n dstrbuted clouds, wreless sensor network vrtualzaton, and cloud network as a servce [27], [28]. Beck et al. propose a herarchcal parttonng of any substrate network [29] and solve the network vrtualzaton vrtual network embeddng) problem on the scale of smaller parttons by delegatng the problem to delegate nodes. In our context, ther algorthm can progress n four steps: ) parttonng the network of IoT devces, 2) assgnng delegaton nodes among the IoT devces that actually perform the network vrtualzaton, 3) settng dstrbuted lock trees to avod nconsstent solutons among dfferent delegaton nodes, and 4) embeddng the vrtual sensor network wthn the scope of the delegaton nodes. Several assumptons n ths work prevent ths method applcablty n Cloud of Thngs. The authors requre that a centralzed node manage the IoT network topology to perform the parttonng. The centralzed node parttons the IoT devces n groups that are hghly nterconnected. Ths requrement s very hard to acheve, f not mpossble, n an Internet-scale network of IoT devces, not only because of the sze of the network but also due to ts hghly dynamc nature that prevents trackng the states of the devces and ther connectvty. Moreover, to apply the same method n cloud of thngs, the delegaton nodes of Beck s method needs to learn a sgnfcant amount of nformaton about the IoT devces n ther neghborhood, whch creates sgnfcant practcal problems such as: tmely nformaton retreval, prvacy concern, and computaton power. Esposto and Ibrahm propose to model the network vrtualzaton as a network utlty maxmzaton problem, where the utlty s a general functon that s measured on each hostng node.e. IoT devce) [30]. They solve the problem dstrbutvty usng prmal dual decomposton. To employ ther algorthm n our Cloud of Thngs context, the non-convex constrants of the network vrtualzaton problem need be relaxed. Such a relaxaton s known to have a negatve mpact on the accuracy of the soluton and may lead to false decsons [5]. Moreover, ths method requres the defnton of a sngle utlty for the IoT devce that shall not reflect the actual embeddng cost due to network condtons, and only captures network condtons seen locally by the 2