Adaptive Avatar Handoff in the Cloudlet Network

Transcription

1 Adaptve Avatar Handoff n the Cloudlet Network 2018 IEEE. Personal use of ths materal s permtted. Permsson from IEEE must be obtaned for all other uses, n any current or future meda, ncludng reprntng/republshng ths materal for advertsng or promotonal purposes, creatng new collectve works, for resale or redstrbuton to servers or lsts, or reuse of any copyrghted component of ths work n other works. Ths materal s presented to ensure tmely dssemnaton of scholarly and techncal work. Copyrght and all rghts theren are retaned by authors or by other copyrght holders. All persons copyng ths nformaton are expected to adhere to the terms and constrants nvoked by each author's copyrght. In most cases, these works may not be reposted wthout the explct permsson of the copyrght holder. Ctaton: X. Sun and N. Ansar, "Adaptve Avatar Handoff n the Cloudlet Network," n IEEE Transactons on Cloud Computng, DOI: /TCC , early access URL:

2 1 Adaptve Avatar Handoff n the Cloudlet Network Xang Sun, Student Member, IEEE, and Nrwan Ansar, Fellow, IEEE In a tradtonal bg data network, data streams generated by User Equpments (UEs) are uploaded to the remote cloud (for further processng) va the Internet. However, movng a huge amount of data va the Internet may lead to a long End-to-End (E2E) delay between a UE and ts computng resources (n the remote cloud) as well as severe traffc jams n the Internet. To overcome ths drawback, we propose a cloudlet network to brng the computng and storage resources from the cloud to the moble edge. Each base staton s attached to one cloudlet and each UE s assocated wth ts Avatar n the cloudlet to process ts data locally. Thus, the E2E delay between a UE and ts computng resources n ts Avatars s reduced as compared to that n the tradtonal bg data network. However, n order to mantan the low E2E delay when UEs roam away, t s necessary to hand off Avatars accordngly t s not practcal to hand off the Avatars vrtual dsks durng roamng as ths wll ncur unbearable mgraton tme and network congeston. We propose the LatEncy Aware Replca placement (LEARN) algorthm to place a number of replcas of each Avatar s vrtual dsk nto sutable cloudlets. Thus, the Avatar can be handed off among ts cloudlets (whch contan one of ts replcas) wthout mgratng ts vrtual dsk. Smulatons demonstrate that LEARN reduces the average E2E delay. Meanwhle, by consderng the capacty lmtaton of each cloudlet, we propose the LatEncy aware Avatar handoff (LEAD) algorthm to place UEs Avatars among the cloudlets such that the average E2E delay s mnmzed. Smulatons demonstrate that LEAD mantans the low average E2E delay. Index Terms cloudlet, moble edge computng, bg data, moble cloud computng, Avatar, handoff, replca. I. INTRODUCTION Portable User Equpments (UEs), such as smart phones, tablets, smart watches and smart glasses, come wth a rch set of embedded sensors already bult-n [1]. By utlzng these ntellgent UEs, the locaton (e.g., GPS nformaton), actvty (e.g., walkng, speakng, sttng, etc.), mood (e.g., happy, calm, alert, etc.), health nformaton (e.g., blood pressure, heartbeat rate, body temperature, etc.), and the ambent envronment nformaton of each human beng can be montored and recorded. Thus, UEs are consdered as data stream generators producng massve amounts of data, and analyzng these data s not only extremely valuable for market applcatons, but also has an ncredble potental to beneft socety as a whole [2]. For nstance, analyzng the vdeos and photos captured by UEs s crucal to localze lost chldren or terrorsts, and analyzng the users actvtes, possble events and hstorcal traffc statstcs can accurately forecast the traffc condton to help drvers selectng optmal drvng drectons. However, the value of the bg data decreases as tme passes by (for nstance, t s mportant to dentfy the terrorsts from the recent photos/vdeos quckly). Therefore, analyzng the bg data n realtme s crtcal. In a tradtonal bg data network, all the data generated by UEs are transmtted to a data center (for further analyss) va the Internet [3] [5] because the data center can provson effcent dstrbuted computng archtecture (such as MapReduce [6], Dryad [7], and Storm [8]) as well as flexble resource allocaton [9]. However, transmttng the bg data from UEs to the data center through the Internet leads to long network latency and drans the network resources. Thus, the exstng bg data networkng platform s not sutable for realtme bg data analyss. X. Sun and N. Ansar are wth the Advanced Networkng Lab., Helen and John C. Hartmann Department of Electrcal & Computer Engneerng, New Jersey Insttute of Technology, Newark, NJ 07102, USA. E-mal:{xs47, nrwan.ansar}@njt.edu. Rather than brngng the data to the computng resources, t s more effcent to brng the computng resources to the data n order to reduce the network latency and the traffc load of the network. Thus, we propose a new cloudlet network archtecture to analyze UEs data streams at the moble edge. As shown n Fg. 1, each Base Staton (BS) s attached to a cloudlet, whch can be consdered as a dstrbuted tny data center that s deployed close to UEs. Each UE s assocated wth a specfc Avatar,.e., a dedcated Vrtual Machne (VM) provdng prvate computng, communcatons and storage resources to the UE, n the nearby cloudlet. Thus, the data streams generated from UEs can be uploaded and analyzed n ther own Avatars wth low End-to-End (E2E) delay. On the top of the cloudlets, Software Defned Network (SDN) based cellular core network [10] [13] has been ntroduced n the cloudlet network archtecture to provde effcent and flexble communcatons paths between Avatars n dfferent cloudlets as well as between UEs n dfferent BSs. Moreover, every UE and ts Avatar n the cloudlet can communcate wth publc data centers (e.g., Amazon EC2) and Storage Area Networks (SANs) va the Internet n order to provson scalablty,.e., f cloudlets are not avalable for UEs because of the capacty lmtaton, UEs Avatars can be mgrated to the remote data centers to contnue servng ther UEs. The communcatons between a UE and a BS s orchestrated by the Packet Data Convergence Protocol [14], whch s specfed by 3GPP. The communcatons protocol between a BS and an OpenFlow swtch as well as between a BS and a cloudlet adopts the TCP/IP protocol. The proposed cloudlet network archtecture facltates the real-tme bg data analyss. A typcal example by utlzng the cloudlet network archtecture to analyze the bg data s the terrorst localzaton applcaton, whch s to dentfy and track terrorsts by analyzng the photos and vdeos taken by dfferent UEs. Specfcally, each UE uploads ts captured photos and vdeos to ts Avatar. The terrorst localzaton applcaton sends the terrorsts photos to Avatars, whch compare terrorsts

3 2 Fg. 1. The cloudlet network archtecture. photos to the captured photos and vdeos by runnng face matchng algorthms. If matched, the nformaton (e.g., locatons and tmestamps) of the photos and vdeos would be transmtted to the terrorst localzaton applcaton for further processng. Applyng the cloudlet network archtecture to mplement terrorst localzaton has the followng advantages. Frst, Avatars are consdered as prvate VMs for ther UEs. Avatars locally analyze the data streams generated from ther UEs and share metadata (rather than raw data) by removng UEs personal nformaton from raw data. For nstance, n the terrorst localzaton applcaton, each Avatar only provdes the locatons and the tme stamps of the matched photos and vdeos rather than the photos and vdeos to the terrorst localzaton applcaton. Ths can preserve the prvacy of each UE. Second, as compared to the tradtonal method (n whch all the captured photos and vdeos would be uploaded to a data center for analyss), analyzng each UE s captured photos and vdeos n ts Avatar (whch s placed n the local cloudlet) can sgnfcantly reduce the traffc load of the network and network delay for uploadng the photos and vdeos to the computng resources, thus speedng up the process of terrorst localzaton. In order to encourage UEs n subscrbng ther own Avatars n the cloudlet network, each Avatar can be consdered as a software clone [17] of ts UE so that the UE can automatcally offload some computatonal ntensve tasks to ts Avatar by utlzng Moble Cloud Computng (MCC) technologes [10], [15], [16]. Avatars execute the tasks and return back the results to ther UEs, and so the task executon tme and the energy consumpton of UEs can be reduced sgnfcantly [18]. Note that many MCC frameworks have been desgned to enable UEs to offload ther resource-hungry tasks to the cloud. For example, the MAUI project [19] provdes method level code offloadng based on the.net framework. Three moble applcatons,.e., a face recognton applcaton, an arcade game applcaton, and a voce-based language translaton applcaton, have been tested under MAUI and the results demonstrated by conductng code offloadng can sgnfcantly reduce the energy consumpton of moble devces and mprove the performance n terms of the delay of these moble applcatons. The CloudClone project [17] desgns an MCC framework by mgratng an applcaton thread from the moble devce at a chosen pont to the applcaton VM (whch s consdered as the applcaton-level clone of the moble devce) n the cloud. Vrus scannng, mage search, and behavor proflng applcatons have been tested under the desgned MCC framework to demonstrate the reducton of the moble devce s energy consumpton and the applcaton s executon tme by offloadng applcaton workloads from moble devces to ther clones. By applyng the mentoned MCC frameworks, t s feasble and benefcal to enable UEs to outsource ther computatonal ntensve tasks to ther Avatars n the nearby cloudlet. Therefore, Avatars play two roles n the cloudlet network,.e., the bg data analyzer and the MCC applcaton outsourcer. The rest of the paper s organzed as follows. In Secton II, we propose to hand off Avatars among cloudlets n order to mantan the low E2E delay between UEs and ther Avatars when UEs roam away. In order to avod vrtual dsk mgraton durng the Avatar handoff process, we propose to place a number of replcas of the Avatar s vrtual dsk among sutable cloudlets. In Secton III, we formulate the Avatar replca placement problem and desgn the LatEncy Aware Replca placement (LEARN) algorthm to solve the problem. In Secton IV, by consderng the capacty lmtaton of each cloudlet, we propose the LatEncy aware Avatar handoff (LEAD) algorthm to optmally place UEs Avatars among the cloudlets n each tme slot such that the average E2E delay between UEs and Avatars s mnmzed. In Secton V, we demonstrate the performance of the proposed LEARN and LEAD algorthm va extensve smulatons. In Secton VI, we brefly revew the related works. The concluson s presented n Secton VII. II. AVATAR HANDOFF UEs are roamng among BSs over tme and so the E2E delay between UEs and ther Avatars may become worse f

4 3 the Avatars reman n ther orgnal cloudlets. For nstance, as shown n Fg. 1, f UE 1 roams from BS 1 s coverage area nto BS 3 s coverage area and ts Avatar stll resdes n Cloudlet A, the communcatons path between UE 1 and ts Avatar should traverse the SDN based cellular core, whch may ncrease the E2E delay as well as the traffc load of the SDN based cellular core. Note that the E2E delay between a UE and ts Avatar comprses three parts: frst, the E2E delay between a UE and ts servng BS; second, the E2E delay between the BS and the cloudlet whch contans the UE s Avatar; thrd, the E2E delay wthn the cloudlet. Normally, the E2E delay between the BS and the cloudlet s the man factor to determne the overall E2E delay when UE roams away. Thus, we refer to the E2E delay between a UE and ts Avatar as the E2E delay between the BS (whch serves the UE) and the cloudlet (whch contans the UE s Avatar) n the rest of the paper. Long E2E delay can sgnfcantly degrade the performance of the bg data analyss as well as the MCC applcatons. Obvously, spendng less tme for uploadng the data streams to Avatars wll beneft real-tme bg data analyss to produce more valuable results. Meanwhle, the E2E delay s crtcal for MCC applcatons. It s reported that augmented realty applcatons requre an E2E delay of less than 16 ms [20] and the cloud-based vrtual desktop applcatons requre an E2E delay of less than 60 ms [21]. Thus, t s crtcal to preserve the low E2E delay between a UE and ts Avatar by mgratng the Avatar among cloudlets when the UE roams away. An Avatar s consdered as a prvate VM, whch comprses solated vcpus, memory, and vrtual dsks, and so mgratng an Avatar between cloudlets s to conduct lve VM mgraton [22] between cloudlets over the SDN based cellular core. We refer to the Avatar mgraton process as the Avatar handoff n the rest of the paper. The Avatar handoff process needs to mgrate the whole Avatar (whch ncludes the memory, the vrtual dsk 1, and the drty blocks of the memory and the vrtual dsk generated durng the mgraton process) from the source nto the destnaton cloudlet. The total Avatar handoff tme determnes the performance of the Avatar handoff [10]. Ths s because, frst, the degraded E2E delay between a UE and ts Avatar perssts untl the Avatar handoff process s fnshed, and so shorter handoff tme wll produce lower E2E delay; second, the Avatar handoff process consumes extra resources of the Avatar (especally the bandwdth resource), and thus degrades the performance of applcatons currently runnng n the Avatar. Therefore, short handoff tme wll mprove the performance of the applcatons. However, t s reported that mgratng the whole Avatar between two cloudlets over a network wth stable 10 Mbps bandwdth and 50 ms Round Trp Tme (RTT) consumes over two hours [21]; ths ndcates that handng off a whole Avatar between cloudlets cannot mantan the low E2E delay between the Avatar and ts UE but exhausts the resource of the network and the Avatar. The man reason for ncurrng 1 In order to guarantee the performance of the bg data analyss and the MCC applcatons runnng n the Avatar, the whole vrtual dsk of the Avatar should be located n the same physcal machne wth ts vcpu and memory provdng low I/O latency. the unacceptable handoff tme s to mgrate the large volume of the Avatar s vrtual dsk over the network [21]. In order to avod vrtual dsk mgraton durng the handoff process, we propose to place a number of replcas of an Avatar s vrtual dsk n the sutable cloudlets. The replcas of an Avatar 2 are synchronzed wth the Avatar s vrtual dsk durng a fxed tme perod (e.g., 5 mn). Thus, f a UE s Avatar tres to hand off to the cloudlet whch contans one of the Avatar s replcas, only the memory and the pre-handoff vrtual dsk drty blocks of the Avatar are needed to be transmtted to the destnaton cloudlet. The pre-handoff vrtual dsk drty blocks of the Avatar means the vrtual dsk drty blocks that are generated after the last replca synchronzaton process. For nstance, as shown n Fg. 2, the Avatar s replcas are synchronzed at t 1 ; meanwhle, the Avatar handoff s trggered at t 2, and thus the number of the vrtual dsk blocks, whch are modfed durng the nterval between t 1 and t 2, are defned as the pre-handoff vrtual dsk drty blocks, whch need to be mgrated durng the handoff process. Snce the drty block generaton rate of the vrtual dsk s relatvely low, only very small porton of the vrtual dsk s needed to be transmtted to the destnaton cloudlet, whch wll sgnfcantly reduce the handoff tme. Fg. 2. The llustraton of pre-handoff vrtual dsk drty blocks. III. AVATAR REPLICA PLACEMENT Mgratng the whole vrtual dsk of an Avatar durng the Avatar handoff process ncurs unbearable handoff tme and ncreases the traffc load of the SDN-based cellular core sgnfcantly. Thus, n order to avod the vrtual dsk mgraton durng the Avatar handoff process, we pre-deploy a number of the Avatar s replcas among the cloudlets. A cloudlet, whch contans one of an Avatar s replcas, s defned as the Avatar s avalable cloudlet. Thus, an Avatar can only be mgrated to ts avalable cloudlets. Note that t s unnecessary and neffcent to place the Avatar s replcas n all the cloudlets n the network because ncreasng the number of replcas for each Avatar ncreases the captal expendture (CAPEX) of the cloudlet provder (by mplementng more storage space n the cloudlets) as well as the synchronzaton traffc n the SDN based cellular core. Meanwhle, placng the Avatar s replcas n the cloudlets, whch are never vsted by the UE, cannot beneft the communcatons between the UE and ts Avatar. Therefore, t s mportant to optmally place a lmted number of replcas for each Avatar among the cloudlets so that the average E2E delay (durng a perod T, e.g., one day) between the UE and ts 2 The replcas of an Avatar are referred to as the replcas of the Avatar s vrtual dsk n the rest of the paper.

5 4 Avatar can be mnmzed (by utlzng Avatar handoff) when the UE roams n the network. x k = 1, where k K) or not (.e., x k = 0). Meanwhle, let t jk be the average E2E delay between BS j and cloudlet k. The value of t jk (j k) can be measured and recorded by the SDN controller [26], [27]. Note that f j = k, we say that cloudlet k s BS j s attached cloudlet. Moreover, denote y jk as a bnary varable ndcatng UE s Avatar s located n cloudlet k (.e., y jk = 1) or not (.e., y jk = 0) when UE s n BS j s coverage area. Let τ j be the average E2E delay between UE and ts Avatar when UE s n the BS j s coverage area, then we have: τ j = t jk y jk. (1) Fg. 3. The llustraton of the optmal Avatar replca placement. Normally, the Avatar s replcas wll be placed where ts UE wll commonly vst (t has been demonstrated that about 10% to 30% of all human movement can be explaned by ther socal relatonshp, whle 50% to 70% s attrbuted to perodc behavors [23]; thus, we beleve that the dynamcs of future human movement can be relably predcted based on the mathematcal models [23] [25]), such as home and workplace. However, ths s not the optmal Avatar replca placement strategy. For nstance, suppose the cloudlet network topology s shown n Fg. 3, whch contans 7 BS-Cloudlet (BSC) combnatons 3, and two replcas of UE s Avatar need to be placed. Meanwhle, suppose the occurrence probablty of UE n BS j s coverage area, denoted as p 4 j (where j = 1, 2,, 7), s also shown n Fg. 3. Thus, tradtonally, two replcas wll be placed n BSC-1 and BSC-2 (because p,1 and p,2 are the two largest values, mplyng that UE wll most commonly vst BSC-1 and BSC-2). Yet, deployng the two replcas n BSC-1 and BSC-7 may be the optmal soluton for UE. Ths s because, frst, the value of p,2 and p,7 are close; second, BSC-1 and BSC-2 are adjacent to each other, and so the E2E delay between UE and ts Avatar s low even f UE s n the BSC-2 s coverage area and ts Avatar s n BSC-1. On the contrary, snce BSC-7 s far away from BSC-1, the E2E delay may be unbearable f UE s n the BSC-7 s coverage area and ts Avatar s n BSC-1, and thus placng the 2nd replca n BSC-7 may mprove the average E2E delay sgnfcantly. Therefore, we conclude that the value of p j s not the only determnant to affect the performance of the Avatar replca placement. The E2E delay between dfferent BSCs can also affect the performance of Avatar replca placement. A. System model Let I, J and K be the set of UEs, BSs and cloudlets, respectvely. Denote x k as a bnary varable ndcatng one replca of UE s Avatar ( I) s located n cloudlet k (.e., 3 A BSC combnaton ndcates that a BS s attached to a dedcated cloudlet. 4 p j = the total tme that UE stays n the BS j s coverage area the total tme perod (.e., one day) Denote τ as the average E2E delay between UE and ts Avatar durng the perod T (e.g., one day); meanwhle, let p j be the predcted occurrence probablty of UE n BS j s coverage area durng the perod T ; then, we have: τ = p j τ j = p j t jk y jk. (2) The optmal Avatar replca placement for each UE ( I) s to mnmze ts average E2E τ durng the perod T. Thus, we formulate the problem as follows: P 0 : arg mn p j t jk y jk (3) x k, y jk s.t. x k = κ, (4) j J, y jk = 1, (5) j J k K, y jk x k, (6) k K, x k {0, 1}, (7) j J k K, y jk {0, 1}, (8) where κ s the total number of replcas that can be deployed among the cloudlets for each UE s Avatar. Constrant (4) requres that exactly κ replcas are placed for UE. Constrant (5) ndcates that UE Avatar should be located n exactly one cloudlet when UE s n BS j s coverage area. Constrant (6) mples that UE Avatar can only be located n a cloudlet f and only f the cloudlet contans one of ts replcas (.e., n order to satsfy Constrant (6), y jk could equal to 1 ff x k = 1; otherwse, y jk should be 0 f x k = 0). Constrants (7) and (8) mples x k and y jk (j J and k K) are bnary varables. Lemma 1. The Avatar replca placement problem (.e., P 0) s NP-hard when κ > 1. Proof: The formulaton of the Avatar replca placement problem s equvalent to the p-medan problem [28] where κ = p > 1, and the p-medan problem has been proved to be NP-hard on a general network topology (note that t has been demonstrated that the p-medan problem can be solved n polynomal tme O(n 2 p 2 ) only f the network s a tree [29]). Therefore, we need to demonstrate the topology of the proposed cloudlet network s not a tree.

6 5 Based on the cloudlet network proposed n Sec. I, each BS can communcate wth all the cloudlets over the SDN based cellular core. Thus, the cloudlet network can be consdered as a complete graph n whch every vertex represents the BScloudlet par. Every par of dstnct vertces s connected by a unque edge, whch represents a communcatons lnk wth a dedcated cost n terms of the E2E delay. Therefore, the Avatar replca placement problem s NP-hard. B. LatEncy Aware Replca placement (LEARN) Inspred by the Lagrangan relaxaton algorthm for solvng the p-medan problem [30], we desgn the LatEncy Aware Replca placement (LEARN) algorthm to optmally place the replcas among cloudlets for each UE. The basc dea of LEARN s to teratvely obtan the lower bound (LB) and upper bound (UB) of the Avatar replca placement problem through Lagrangan procedure untl the dfferece between the LB and UB s less than a predefned value ψ. Specfcally, we relax Constrant (5) n P 0 to obtan the followng Lagrangan problem: P 1 : max mn L= p j t jk y jk + ( λ j 1 ) y jk λ j x k y jk = (p j t jk λ j ) y jk + λ j, (9) s.t. Constrants (4), (6), (7), (8), where λ j ( j J, λ j 0) are the Lagrangan multplers. For fxed values of the Lagrange multplers λ j, the above relaxed problem (.e., P 1) wll yeld an optmal objectve value that s an LB on any feasble soluton of the orgnal Avatar replca placement problem (.e., P 0). Lemma 2. Defne vector = { k k K}, where k = mn (0, p j t jk λ j ); defne the cloudlet set K (K { } K), where K = κ and k k K are the κ number of the smallest values n vector. Then, for any gven set of multplers λ = {λ j j J }, the optmal soluton of the Lagrangan problem, denoted } as X = {x k k K} and Y = {yjk j J, k K, can be expressed as follows: k K, x k = { 1, k K. 0, otherwse. { j J, k K, yjk 1, pjt = jk λ j < 0 & x k =1. 0, otherwse. (10) (11) Proof: Obvously, n order to mnmze the objectve functon of the Lagrangan problem (.e., P 1), y jk should be chosen ts maxmum value f p j t jk λ j 0 (j J, k K) for any gven set of Lagrangan multplers λ = {λ j j J }, otherwse, y jk = 0. Thus, by consderng Constrant (6), the optmal soluton of yjk s gven by: y jk = x k, f p j t jk λ j 0; yjk = 0, otherwse. By substtutng the optmal soluton of yjk nto L (Eq. (9)), the Lagrangan problem s transformed nto: mn L = k x k + λ j (12) x k s.t. Constrants (4), (7), where k = mn (0, p j t jk λ j ). For any gven set of Lagrangan multplers λ, the above problem s trval to solve,.e., x k = 1, f k K (where K s the set of κ number of the cloudlets, whch have the smallest values of k n = { k k K}); x k = 0, otherwse. Thus, Eq. (10) and Eq. (11) have been proved. Note that solvng the relaxed problem can provde the LB of the orgnal Avatar replca problem,.e., LB = k x k + λ j. (13) However, the soluton of the Lagrangan relaxaton problem may not be the feasble soluton wth respect to the orgnal Avatar replca problem (P 0),.e., Constrant (5) may } not be satsfed for the soluton Y = {yjk j J, k K. In order to obtan a feasble soluton of the orgnal problem, denoted as Y = { y jk j J, k K }, we can smply allocate the Avatar of UE to the cloudlet, whch has the lowest E2E delay among the cloudlets contanng one replca of the Avatar, when UE s n BS j,.e., for each j J, we have: k K, y jk = { } 1, t jk = mn {t jk k K 0, otherwse. (14) where K s the set of avalable cloudlets (whch contan one replca of UE s Avatar) of UE s Avatar,.e., K = {k x k = 1, k K}. { Substtutng the feasble soluton (.e., Y = yjk j J, k K } ) nto the objectve functon of the orgnal problem (.e., Eq. (3)), we have the UB of the orgnal problem: UB = p j t jk y jk. (15) Note that the orgnal problem always chooses ts UB as ts objectve value because the U B can guarantee the exstence of the feasble soluton. However, selectng dfferent values of Lagrange multpler vector (.e., λ) may generate dfferent values of the UB. Thus, by applyng the subgradent method [31], we adjust the values of Lagrange multplers n each teraton n order to obtan the smaller value of U B. The teraton termnates untl UB opt LB ψ, where UB opt ndcates the best (.e., smallest) value of UB that has been found n the prevous teratons. In the n th teraton (n > 1), the values of the Lagrangan multplers λ n j (j J ) are calculated based on the followng expresson: { }} j J, λ n j =max 0, λ n 1 j θ n { y jkn 1 1, (16)

7 6 where λ n 1 j are the Lagrangan multplers generated n the prevous teraton; y jkn 1 (j J, k K) are the optmal soluton of P 1 (.e., the relaxed problem) n the prevous teraton, whch can be calculated based on Eq. (11) and θ n s the step length adopted n the n th teraton, whch can be calculate based on the followng expresson [32]: θ n = α ( UB opt LB n 1) ( ) 2, (17) y jkn 1 1 where α (0 < α < 2) s a decreasng adaptaton parameter and LB n 1 s the value of LB n the prevous teraton (.e., LB n 1 = k x n 1 k + λ n 1 j ). The detal of the LEARN algorthm s shown n Algorthm 1. C. One example to llustrate the LEARN algorthm Suppose there are three BSs n the network and each BS s attached to one cloudlet. Assume the average E2E delay vector s T = There s a UE n the network and the occurrence probablty of the UE n the respectve BSs durng the day s P= [0.5, 0.3, 0.2]. If we need to place two replcas (.e., κ = 2) for ts Avatar s vrtual dsk, then LEARN shall apply the followng procedure to obtan the optmal replca placement for the UE: Steps 1-2 n Algorthm 1: randomly select the ntal values of Lagrangan multplers, e.g., λ= [5, 5, 5]; ntalze LB = 0 and UB opt = + ; Steps 4-5 n Algorthm 1: gven the value of λ, calculate the values of X and Y for the UE based on Lemma 2; n ths example, X = [1, 1, 0] and Y = Then, update the value of LB based on Eq. (13); n ths example, LB = 0; Steps 6-7 n Algorthm 1: calculate the value of Y based on Eq. (14). In ths example, Y = Then, update the value of UB based on Eq. (15); n ths example, UB = 3.5; Steps 8-11 n Algorthm 1: compare UB wth UB opt ; n ths example, UB < UB opt, and thus X opt = X = [1, 1, 0] and UB opt = UB = 3.5; Steps n Algorthm 1: update the value of Lagrangan multplers,.e., λ, based on Eq. (16), and goes back to Steps 4-5 untl UB opt LB ψ. The LEARN algorthm s executed offlne,.e., for a fxed perod T, LEARN wll update the replca placement for dfferent UEs durng the off peak hours. Also, the replca placement updatng perod T can also vary among dfferent UEs. For nstance, f UEs have smlar behavors durng the workdays, LEARN only needs to update the replca placement of the UEs durng the weekends; otherwse, t s preferred to update the replca placement daly. It s worth to note that a centralzed controller (or a control functon runnng n the SDN controller) s used to predct the occurrence probablty for each UE, obtan the average E2E delay vector among dfferent cloudlets and BSs from the SDN controller, and generate the replca placement vector for each UE by executng the proposed LEARN algorthm. Algorthm 1 LEARN algorthm Input: 1) The occurrence probablty vector for UE among BSs,.e., P = {p j j J }. 2) The average E2E delay vector T = {t jk j J, k K}. Output: { The replca placement vector for UE,.e., X opt = x opt k k K}. 1: Intalze the set of Lagrangan multplers λ = {λ j j J }. 2: Intalze LB = 0 and UB opt = +. 3: whle UB opt LB > ψ do 4: Calculate X and Y based on Lemma 2; 5: Update the value of LB based on Eq. (13); 6: Calculate Y based on Eq. (14); 7: Calculate the value of UB based on Eq. (15); 8: f UB < UB opt then 9: X opt = X ; 10: UB opt = UB; 11: end f 12: Update step length θ n based on Eq. (17); 13: Update Lagrangan multplers λ based on Eq. (16); 14: end whle 15: return X opt. IV. ADAPTIVE AVATAR HANDOFF After the replcas of each UE s Avatar beng deployed among cloudlets, the Avatar can be handed off among ts avalable cloudlets based on ts UE s locaton. Optmally, the Avatar wll be handed off to the avalable cloudlet, whch ncurs the lowest E2E among ts avalable cloudlets, when the UE roams nto a new locaton. However, each cloudlet has ts CPU and memory capacty, and so the Avatar may not be handed off to the optmal cloudlet because the optmal cloudlet may not have enough resdual capacty to host the Avatar. Therefore, t s necessary to desgn an adaptve Avatar handoff strategy to determne the locaton of each UE s Avatar n each tme slot n order to mnmze the average E2E delay between all the UEs and ther Avatars durng the tme slot by jontly consderng the capacty lmtaton of each cloudlet. Note that dfferent from the Avatar replca placement problem (whch tres to generate the replca placement soluton for each UE s Avatar based on the statstcs for a long tme perod (e.g., one day)), the adaptve Avatar handoff problem tres to obtan the locaton of each UE s Avatar (rather the Avatar s replcas) based on real tme nformaton (e.g., the current locatons of all the UEs) and the problem should be solved n real tme. A. Problem formulaton Let l j be a bnary ndcator to dentfy UE n BS j s coverage area (.e., l,j = 1) or not (l,j = 0) n the current

8 7 tme slot. Meanwhle, let z k be a bnary varable to ndcate whether UE s Avatar s n cloudlet k (z k = 1) or not (z k = 0) n the current tme slot. X opt = { x opt k k K}, whch s generated by the LEARN algorthm, s the optmal replca placement vector for UE. In order to avod the vrtual dsk mgraton, UE s Avatar can only be allocated to ts avalable cloudlet k (.e., z k could equal to 1 ff k K, where K { = k x opt k = 1, k K} ). Thus, the average E2E delay between UE and ts Avatar n the current tme slot,.e., τ, can be expressed as follows: τ = l j t jk z k. (18) Suppose all the UEs Avatars are homogeneous,.e., all the Avatars have the same CPU, memory and bandwdth confguratons. Denote q k as the capacty of cloudlet k,.e., the total number of Avatars can be hosted by cloudlet k. Thus, we formulate the Avatar handoff problem as follows: P 2 : arg mn z k s.t. I, k K, I l j t jk z k (19) z k = 1, (20) z k q k, (21) I I k K, z k {0, 1}, (22) where the objectve s to mnmze the average E2E delay between all the UEs and ther Avatars n the current tme slot. Constrant (20) ndcates each Avatar should be hosted by one cloudlet, whch contans one replca of the UE s Avatar; Constrant (21) mples that each cloudlet has ts capacty lmtaton; Constrant (22) ndcates z k s a bnary varable. Note that t s not easy to solve P 2 snce the avalable vares among dfferent UEs Avatars that makes the summaton ndex k n the object functon of P 2 to vary among dfferent UEs Avatars. The followng lemma facltates a dual problem of P 2, whch can be readly solved because the summaton ndex k n the objectve functon of the new problem does not vary among dfferent UEs Avatars. cloudlet set K Lemma 3. Let τ be τ = l j t jk x opt k + εz k, (23) where ε s a very small postve value close to zero. Then, P 2 can be equvalently transformed nto: l j t jk P 3 : arg mn z k x opt I k + εz k, s.t. I, z k = 1, l jt jk x opt k k K, z k q k, I I k K, z k {0, 1}. Proof: For each I, f x opt k = 0 (.e., k K\K ), +, and thus z +ε k should be set to zero n order to mnmze the value of τ ; f xopt k = 1 (.e., k K ), the expresson of τ s approxmately equal to τ. Thus, P 2 and P 3 are equvalent. By applyng Lemma 3, the problem (.e., P 2) can be transformed nto: arg mn z k I c k z k (24) s.t. Constrants (20), (21), (22), l jt jk where c k = s the weghted E2E delay between UE x opt k and ts Avatar located +ε n cloudlet k. The proposed Avatar handoff problem s consdered as a specal case of the generalzed assgnment problem (n whch Constrant (21) s depcted as k K, a k z k q k, where I a k > 0), whch s proven to be NP-hard [1]. Recently, many heurstcs have been desgned to fnd the suboptmal soluton of the generalzed assgnment problem and each of them has ts tradeoff between the complexty and the performance. In the proposed network, we need to allocate tens of thousands of UEs nto tens of thousands of cloudlets n each tme slot. Thus, we desgn a novel LatEncy aware Avatar handoff (LEAD) algorthm to effcently solve the proposed Avatar handoff problem. B. LatEncy aware Avatar handoff (LEAD) Frst, we buld a relaxed Avatar handoff problem (.e., P 4) by relaxng Constrant (21),.e.: P 4 : arg mn c k z k z k I s.t. I, z k = 1, I k K, z k {0, 1}. { The optmal soluton of P 4, denoted as Z = z k }, k K can be easly derved,.e.: { { 1, k = arg mn ck k K }. I, z k = k (25) 0, otherwse. The optmal soluton of P 4 generates the optmal cloudlet (whch ncurs the mnmum weghted E2E delay among the cloudlets) for each Avatar. However, t may not be the feasble soluton of the orgnal Avatar handoff problem,.e., the total number of Avatars that are hosted by some cloudlets may exceed{ ther capacty. Denote these set of cloudlets as K 1,.e., K 1 = k } z k > q k, k K ; denote the set of cloudlets, I whch have enough{ resources to host at least one Avatar, as K 2,.e., K 2 = k } z k < q k, k K ; denote the set I of UEs, whose Avatars { are hosted by cloudlet } k K 1, as I 1,.e., I 1 = z k = 1, k K 1, I. The basc dea of the LEAD algorthm s to choose a sutable UE s Avatar, whose optmal cloudlet has volated ts capacty lmtaton (.e., I 1 ), and reallocate the Avatar nto ts suboptmal cloudlet,

9 8 whch has enough space for hostng at least one Avatar, for each teraton. The suboptmal cloudlet, denoted as k, of UE s defned as the cloudlet that ncurs the mnmum weghted E2E delay among the cloudlets, whch have enough { resources } to host at least one Avatar,.e., k = arg mn ck k K 2. Denote c ( I 1 ) as the weghted E2E delay degradaton by reallocatng UE s Avatar from ts optmal cloudlet k nto ts suboptmal cloudlet k,.e.: I 1, c = c k c k, (26) { } where k = arg mn ck k K 1 and k = { } k arg mn ck k K 2. Thus, the LEAD algorthm wll k select to reallocate { a sutable } UE s Avatar, where = arg mn c I 1, n each teraton n order to mnmze the weghted E2E delay degradaton. The teraton contnues untl K 1 =. The detal of the LEAD algorthm s shown n Algorthm 2. The complexty of each teraton n LEAD (.e., from Step- 5 to Step-9 n Algorthm 2) s O( I K ) + O( I ) + 3 O(1) = O( I K ), where O( I K ) s the complexty of Step-5, O( I ) s the complexty of Step-6, and 3 O(1) s the complexty for executng Step-7 to Step-9. In the worst case scenaro, the total number of teratons of LEAD s I. Thus, the complexty of LEAD s O( I 2 K ). Note that the LEAD algorthm cannot guarantee that all the Avatars can be placed n one of ts avalable cloudlets,.e., t mght happen that some Avatars, all of whose avalable cloudlets are full, cannot be hosted by one of ts avalable cloudlets. Then, these Avatars wll be placed n the central cloud (by default, the central cloud contans at least one replca of each UE s Avatar). Lemma 4. The LEAD algorthm termnates after a fnte number of teratons, producng an feasble soluton to the orgnal Avatar handoff problem. ( Proof: Let ξ = z k q k 1 I ntally, and so ξ > 0. Then, n each teraton, the value k ). Assume K 1 of ξ s decreased by one because LEAD wll reallocate UE s Avatar from cloudlet k (.e., set z { } k = 0), where k = arg mn ck k K 1, nto cloudlet k, where k = { k } arg mn ck k K 2. Thus, ξ wll be reduced to zero n a k fnte number of steps, and hence K 1 =. Smlar to LEARN, the LEAD algorthm s also executed n a centralzed manner to determne the locatons of Avatars for ther UEs n each tme slot. V. SIMULATION RESULTS In order to evaluate our proposed replca placement algorthm and Avatar handoff algorthm, we have obtaned data traces of more than 13,000 UEs and extracted ther moblty n one day n Helongjang provnce n Chna 5. The whole area 5 The authors acknowledge the Center for Data Scence of Bejng Unversty of Posts and Telecommuncatons for provdng these nvaluable data traces. Algorthm 2 LEAD algorthm Input: 1) The replca placement vector for each UE,.e., X opt = { x opt k k K}, I. 2) The average E2E delay vector,.e., T = {t jk j J, k K}. 3) The locaton ndcator vector for all UEs n the current tme slot,.e., L = {l j I, j J }. Output: Avatar locaton ndcator } vector for all UEs,.e., Z = {z k I, k K. 1: Intalze z k by solvng the relaxed problem (.e., P 4) based on Eq. (25). 2: Intalze the cloudlet sets K 1 and K 2 based on ther defnton. 3: Intalze the UE set I 1 based on ts defnton. 4: whle K 1 do 5: I 1, calculate c based on { Eq. (26); } 6: Fnd UE, where = arg mn c I 1 ; 7: Reallocate UE s Avatar from ts optmal cloudlet k (.e., set z k = 0) nto ts suboptmal cloudlet k, (.e., set z k = 1); 8: Update the cloudlet sets K 1 and K 2 ; 9: Update the UE set I 1 ; 10: end whle 11: return Z. contans 5,962 BSs and each UE s locaton s montored durng one day perod. Specfcally, each packet that s transmtted to/from a UE s montored, and the packet analyzer extracts the BS nformaton (.e., the BS s ID and locaton) from each packet and consders the BS s locaton to be the current locaton of ths UE (for nstance, f a packet from the UE contans the nformaton of BS-A, then we say the UE s currently assocated wth BS-A and the current locaton of the UE s BS-A s locaton). We apply these UE moblty traces to obtan the occurrence probablty vector for each UE among BSs durng the day (.e., the values of P = {p j j J }, where the total amount of tme UE s assocated wth BS j one day perod ). p j = Meanwhle, we assume that each BS s attached to a cloudlet and the average E2E delay between a BS and a cloudlet (.e., the value of t jk, where j k) s a functon of the geographc dstance between the BS and the cloudlet [33],.e., t jk = 0.016d jk , where d jk s the dstance between BS j and cloudlet k n unt km and the unt of t jk s n ms. Each BS s geographc locaton (.e., the longtude and the lattude of the BS) s gven by the UE moblty traces and each cloudlet s attached to a BS, and thus the value of d jk s known; consequently, the value of t jk can be calculated for all j J and k K based on the E2E delay model. A. Performance of the LEARN algorthm Frst, we smulate the proposed LEARN algorthm based on the mentoned UE moblty trace. Specfcally, we frst calculate the value of p j based on the mentoned UE moblty trace. Then, by takng P and T as nput parameters, we further obtan the replca placement vector for each UE by applyng the LEARN algorthm. Consequently, the average

10 9 E2E delay for all the UEs durng the day s derved gven the replca placement vector for each UE. For comparsons, we consdered the scenaro that all the UEs Avatars are located n the central data center (.e., UEs Avatars cannot mgrate when UEs are roamed among BSs), whch s placed n the southeast pont of the area. In ths scenaro, we also calculate the average E2E delay between UEs and the data center durng the day. As shown n Fg. 4, gven the number of replcas (.e., the value of κ), the average E2E delay acheved by LEARN s sgnfcantly reduced as compared to that of the tradtonal bg data network (n whch the UEs access ther Avatars n the remote cloud/data center va the Internet). Moreover, as the number of replcas ncreases, the average E2E delay s reduced accordngly. Specfcally, as compared to the tradtonal data center network, the average E2E delay acheved by LEARN s mproved by 75.79%, 93.12%, 97.27%, and 98.59% when the value of s selected to be 1, 2, 3, and 4, respectvely. Note that κ = 1 (.e., there s one replca for each UE) ndcates the locaton of each Avatar s fxed (.e., the Avatar s placed n the locaton where ts UE most vst) and the Avatar cannot handoff among the cloudlets because no extra replcas are placed n other cloudlets. Also, when the value of κ s greater than 4, the decrement of the E2E delay by ncreasng the value of κ s not sgnfcant. We further analyze the moblty trace by calculatng at least how many locatons,.e., the BSs coverage area, that each UE wll stay over 90%, 95% and 99% of the tme durng the day, respectvely. As shown n Fg. 5, 92.22%, 86.93% and 75.65% of the UEs spend 90%, 95% and 99% of the tme durng the day (n terms of 21.6, 22.8 and hours) to stay at only four locatons, respectvely. Thus, placng fve replcas for the UEs may not sgnfcantly beneft the average E2E delay durng the day as compared to placng four replcas. Note that placng more replcas for each Avatar may ncrease the CAPEX to the cloudlet network provder by deployng more storage resources. Also, allocatng more replcas for each Avatar may generate more synchronous traffc, and thus ncrease the traffc load of the cloudlet network. B. Performance of the LEAD algorthm We further evaluate the performance of our proposed LEAD algorthm. Each Avatar s replcas have already been placed to the correspondng cloudlets, whch are calculated by the LEARN algorthm. Stll, UEs mobltes and the locatons of the BSs follow the moblty traces that we have sampled from the real world. By applyng the moblty traces, the locaton ndcator vector for all UEs (.e., the values of L) n dfferent tme slots durng the day can be obtaned. Intally, we setup the capacty of all the cloudlet to be the same,.e., k K, q k = 10 (each cloudlet can host at most 10 Avatars n each tme slot). We frst test the average E2E delay between UEs and ther Avatars durng the day. As shown n Fg. 7, as compared to the results of the LEARN algorthm 6 (whch generates the optmal average E2E delay between UEs and 6 We consder the average E2E delay of the LEARN algorthm as the lower bound n terms of the best case scenaro of the LEAD algorthm. ther Avatars wthout consderng the capacty constrants), the average E2E delay generated by the LEAD algorthm s ncreased because some Avatars cannot handoff to ther optmal cloudlets (due to the cloudlet capacty lmtaton) when the UEs roam away. Consequently, these Avatars need to handoff to ther suboptmal cloudlets or to the remote data center. Note that as the number of replcas ncreases, the average E2E delay decreases accordngly. Ths s because as the number of replcas ncreases, the Avatars, whose optmal cloudlets exceed ther capacty lmtaton, have hgher probablty to handoff to ther suboptmal cloudlets wth lower E2E delay. For nstance, as shown n Fg. 6, assume there are two Avatar s replcas that have been optmally placed n cloudlet A and cloudlet B. Suppose the optmal locaton of the Avatar s cloudlet A at the current tme slot but the cloudlet A s full, and so the Avatar needs to handoff to the suboptmal cloudlet, whch s cloudlet B; ths may ncrease the E2E delay between the Avatar and ts UE, and we denote the E2E delay ncrement as t A B. Then, f there are one more Avatar s replcas that have been placed n the cloudlet C, and so the Avatar can be handed off to the suboptmal cloudlet, whch can be cloudlet B or cloudlet C. Thus, the E2E delay ncrement by handng off the Avatar to the suboptmal cloudlet s mn{t A B, t A C }, where t A C s the E2E delay ncrement by handng off the Avatar to cloudlet C. Obvously, t A B mn{t A B, t A C }. Therefore, the average E2E delay between UEs and ther Avatars decreases when the number of the replcas ncreases. Note that, as mentoned n Sec. IV-B, the LEAD algorthm cannot guarantee that all the Avatars can be handed off to ther avalable cloudlets,.e., some Avatars need to be handed off to the remote data center, whch s the man factor of ncreasng the average E2E delay because the average E2E between the UEs and the remote data center reaches ms, whch s sgnfcantly longer than the average E2E delay produced by the LEAD algorthm. Thus, we further test the average number of the remote Avatars, whch are defned as the Avatars that have to be handed off to the remote data center, durng the day. As shown n Fg. 8, the average number of the remote Avatars decreases as the number of each Avatar s replcas ncreases. Ths s because that the probablty (that all the Avatar s avalable cloudlets are full) decreases as the number of the Avatar s replcas ncreases. As mentoned prevously, the average E2E delay of the LEAD algorthm s longer than that of the LEARN algorthm because LEARN does not consder the capacty lmtaton of the cloudlets. In other words, owng to the capacty constrants, some Avatars cannot be handed off to ther optmal cloudlets, thus resultng n the average E2E delay growth n LEAD. In order to study how the cloudlet capacty mpacts the performance of the LEAD algorthm. We further record the average E2E delay of the LEAD algorthm by changng the capacty of each cloudlet. As shown n Fg 9, when the cloudlet capacty ncreases (from 6 to 16), the average E2E delay of the LEAD algorthm mproves tremendously. As the cloudlet capacty reaches 16, the average E2E delay of the LEAD algorthm does not mprove sgnfcantly. Ths s because most of the Avatars can be handed off to ther optmal cloudlets as ther UEs roam away. Although ncreasng the cloudlet

11 10 Fg. 4. The average E2E delay of the cloudlet network by applyng LEARN and that of the tradtonal bg data network. Fg. 5. The statstcal results of the UE moblty trace. Fg. 6. The llustraton of the average E2E delay reducton. capacty can mprove the average E2E delay, the average cloudlet utlzaton 7 s reduced accordngly. As shown n Fg. 10, the average cloudlet utlzaton drops from 35% to 10.9% as the cloudlet capacty ncreases from 6 to 26. Therefore, there s a tradeoff between the average E2E delay and the cloudlet utlzaton. In order to optmze the tradeoff, the capacty of dfferent cloudlets should be vared. Specfcally, the cloudlets, whose connected BSs have hgher UE densty (such as shoppng malls and publc transportatons), should have larger capacty, and vce versa. Thus, n the future, we wll try to desgn a cloudlet deployment strategy to determne the capacty of each cloudlet such that the average cloudlet utlzaton can be maxmzed and the average E2E delay s guaranteed. VI. RELATED WORKS Recently, telecommuncatons vendors have shown the great nterest on the concept of moble edge computng (MEC). European Telecommuncatons Standards Insttute (ETSI) created an ndustry ntatve on MEC to standardze the MEC platform by utlzng the concept of cloudlet. Also, n the academc area, many works [35] [39] have proposed to utlze the cloudlet to reduce the E2E delay between users and computng resources, and thus mprove the performance of MCC applcatons as well as bg data networkng. Chen et al. [40] mplemented a cogntve assstance applcaton (whch 7 cloudlet utlzaton = total number of Avatars hosted by the cloudlet the cloudlet capacty provdes step-by-step vsual gudance for users to mplement a complex task) runnng n a wearable devce (such as Google Glass) wth the help of cloudlet processng. The cloudlet s deployed one wreless hop away to guarantee the strngent E2E delay requred by the applcatons. Quwader and Jararweh [41] developed a cloudlet-based wreless body area network. The data streams generated by the users are transmtted to the nearest cloudlet through WF. The cloudlet stores and processes the data streams locally to reduce the latency as well as the communcatons power consumpton as compared to the tradtonal cloud-based wreless body area network. Xu et al. [42] proposed an effcent algorthm to optmally place a fxed number of cloudlets among the wreless access ponts n the wreless metropoltan area network whle mnmzng the average access delay between moble users and the cloudlets. Cesell et al. [43] also tred to optmally deploy a number of cloudlets among the aggregaton nodes and core nodes n the cellular network n order to mnmze the captal cost (.e., nstallaton costs) of the cloudlet provders whle consderng the latency between users and ther VMs n the cloudlets. All the above papers do not consder how to optmally handoff users Avatars/VMs among cloudlets when users are roamng n the network. Dynamcal Avatar/VM handoff s crtcal n the cloudlet network because the E2E delay between the users and ther Avatars/VMs may become worse when users roam away. Our prevous work [15], [44] presented a green cloudlet network archtecture n whch all the cloudlets are powered by both green energy and on-grd energy. In order to mnmze the on-grd energy consumpton, we proposed to mgrate the Avatars to the cloudlets wth more green energy generaton and less energy demands whle guaranteeng the E2E delay requrement between users and ther Avatars. However, ths work dd not take the vrtual dsk mgraton nto consderaton durng the Avatar mgraton process and the vrtual dsk mgraton may ncur long E2E delay as well as the network congeston as explaned n Sec. II. In order to avod the vrtual dsk mgraton durng the handoff process, Ha et al. [21] proposed that a number of base VM mages, whch contans a set of wdely used software/content, are pre-

12 11 Fg. 7. The average E2E delay of LEAD and LEARN. Fg. 8. The average number of remote Avatars. Fg. 9. The average E2E delay over dfferent cloudlet capactes. Fg. 10. The average cloudlet utlzaton over dfferent cloudlet capactes. deployed n each cloudlet. Thus, once the user s VM s handed off to another cloudlet, only the VM overlay (whch s defned as the dfference between the user s VM and the base VM mages n the cloudlet) need to be mgrated over the network. However, the VM handoff tme s stll rather long f the user s VM has a huge amount of prvate VM overlay. Moreover, n our cloudlet network, each Avatar should store and process ts UE s prvate data (such as photos, GPS nformaton as well as sensng data streams), whch would ncur larger VM overlay. As compared to these prevous efforts, ths paper presents several enhancements. Frst, we have proposed the cloudlet network archtecture by brngng the computng and storage resources close to UE n order to reduce the E2E delay between UEs and computng resources. Second, each UE s assgned a dedcated VM n terms of the Avatar to process ts own data and applcatons. Meanwhle, each UE can access ts Avatar seamlessly based on the cloudlet network. Thrd, n order to facltate Avatar handoff (by avodng the vrtual dsk mgraton) and mantan low average E2E delay between UEs and ther Avatars, we have desgned the LEARN algorthm to place a number of replcas nto sutable cloudlets. Fourth, consderng the capacty lmtaton of each cloudlet, we have proposed the LEAD algorthm to optmze the locatons of all the Avatars n each tme slot n order to mnmze the average E2E delay for all the UEs and ther Avatars durng each tme slot. VII. C ONCLUSION In ths paper, we have proposed the cloudlet network archtecture to facltate bg data networkng as well as moble cloud computng. Specfcally, each UE can access ts own Avatar, whch s consdered as prvate computng and storage resources for the UE, wth the low E2E delay. In order to mantan the low E2E delay when UEs roam away, ther Avatars should be handed off among cloudlets accordngly. However, mgratng the hgh volume of the Avatar s vrtual dsk durng the handoff process ncurs unbearable handoff tme, whch may sgnfcantly degrade the E2E delay as well as the performance of the Avatar. Also, mgratng the Avatar s vrtual dsk durng the handoff process may tremendously ncrease the traffc n the SDN based cellular core. Thus, n order to avod the vrtual dsk mgraton durng the handoff process, we have proposed to place a number of replcas of the Avatar s vrtual dsk among the cloudlets so that the Avatar can be handed off among ts avalable cloudlets (whch contan one of the Avatar s replcas) based on ts UE s locaton. We have

13 12 desgned the LEARN algorthm to optmally place the replcas among the cloudlets for each Avatar so that the average E2E delay between the Avatar and ts UE s mnmzed durng the day. Moreover, after optmally deployng the replcas for each Avatar, we have desgned the LEAD algorthm to determne the locatons of all the Avatars n each tme slot so that the average E2E delay between all the UEs and ther Avatars s mnmzed n each tme slot, whle the capacty of the each cloudlet s not volated. The smulaton results demonstrate that applyng the LEARN algorthm n the cloudlet network archtecture can sgnfcantly reduce the average E2E delay between UEs and ther Avatars durng the day as compared to the tradtonal centralzed cloud archtecture (.e., all the UEs Avatars are located n the central cloud). Furthermore, the LEAD algorthm can stll mantan the low average E2E delay by selectng sutable parameters n terms of the number of the replcas for each Avatar and the capacty of each cloudlet. Note that handng off a UE s Avatar among ts avalable cloudlets when the UE roams away may stll generate an extra overhead [10], e.g., the extra traffc for mgratng Avatar s memory to the destnaton cloudlet. Thus, there s a tradeoff between mnmzng the E2E delay (between a UE and ts Avatar) and mnmzng the extra overhead. To optmze ths tradeoff, varous aspects shall be consdered,.e., the memory utlzaton of the UE s Avatar, the network condton n the SDN-based cellular core network, etc. We have demonstrated that the tradeoff between the average E2E delay and the average cloudlet utlzaton through the smulatons. Thus, n the future, we wll study the cloudlet placement problem,.e., determnng the sutable capacty for each cloudlet, to optmze ths tradeoff. Moreover, the number of the replcas can also be vared among dfferent UEs and the sutable number of replcas can be selected based on the UE s behavor. REFERENCES [1] Q. Han, S. Lang, and H. Zhang, Moble Cloud Sensng, Bg Data, and 5G Networks Make an Intellgent and Smart World, n IEEE Network, vol. 29, no. 2, pp , Mar. Apr [2] M. Musoles, Bg Moble Data Mnng: Good or Evl?, n IEEE Internet Computng, vol. 18, no. 1, pp , Jan. Feb [3] X. Y, F. Lu, J. Lu, and H. Jn, Buldng a Network Hghway for Bg Data: Archtecture and Challenges, n IEEE Network, vol. 28, no. 4, pp. 5 13, July August [4] J. Lu, F. Lu, and N. Ansar, Montorng and Analyzng Bg Traffc Data of a Large Scale Cellular Network wth Hadoop, n IEEE Network, vol. 28, no. 4, pp , July August [5] E. Baccarell, et al., Energy-effcent Dynamc Traffc Offloadng and Reconfguraton of Networked Data Centers for Bg Data Stream Moble Computng: Revew, Challenges, and a Case Study, n IEEE Network, vol. 30, no. 2, pp , March Aprl [6] J. Dean and S. Ghemawat, MapReduce: Smplfed Data Processng on Large Clusters, n Communcatons of the ACM, vol. 51, no. 1, pp , [7] M. Isard, et al., Dryad: Dstrbuted Data-Parallel Programs from Sequental Buldng Blocks, n ACM SIGOPS Operatng Systems Revew, vol. 41, no. 3, pp , [8] Apache storm. [Onlne]. Avalable: date of access: May 1, [9] X. Sun, N. Ansar, and R. Wang, Optmzng Resource Utlzaton of a Data Center, n IEEE Communcatons Surveys & Tutorals, vol. 18, no. 4, pp , Fourthquarter [10] X. Sun and N. Ansar, PRIMAL: PRofIt Maxmzaton Avatar placement for Moble Edge Computng, n Proceedngs of IEEE Internatonal Conference on Communcatons (ICC), Kuala Lumpur, Malaysa, May 23 27, 2016, pp [11] X. Jn, L. E. L, L. Vanbever, and J. Rexford, Softcell: Scalable and Flexble Cellular Core Network Archtecture, n Proceedngs of the nnth ACM conference on Emergng networkng experments and technologes, Santa Barbara, CA, Dec , 2013, pp [12] X. Sun and N. Ansar, EdgeIoT: Moble Edge Computng for the Internet of Thngs, IEEE Communcatons Magazne, vol. 54, no. 12, pp , December [13] X. Sun and N. Ansar, Latency Aware Workload Offloadng n the Cloudlet Network, n IEEE Communcatons Letters, do: /LCOMM , early access. [14] Evolved Unversal Terrestral Rado Access (E-UTRA); Packet Data Convergence Protocol (PDCP) Specfcaton. [Onlne]. Avalable: date of access: May 1, [15] X. Sun, N. Ansar, and Q. Fan, Green Energy Aware Avatar Mgraton Strategy n Green Cloudlet Networks, n 7th IEEE Internatonal Conference on Cloud Computng Technology and Scence (Cloudcom)), Vancouver, Canada, Nov. 30 Dec. 3, 2015, pp [16] C. Borcea, et al., Avatar: Moble Dstrbuted Computng n the Cloud, n 3rd IEEE Internatonal Conference on Moble Cloud Computng, Servces, and Engneerng (MobleCloud 15), San Francsco, CA, Mar. 30- Apr. 3, 2015, pp [17] B. G. Chun, et al., Clonecloud: Elastc Executon between Moble Devce and Cloud, n Proceedngs of the sxth conference on Computer systems, Salzburg, Austra, Apr , 2011, pp [18] X. Sun and N. Ansar, Energy-Optmzed Bandwdth Allocaton Strategy for Moble Cloud Computng n LTE Networks, n 2015 IEEE Wreless Communcatons and Networkng Conference (WCNC), New Orleans, LA, 2015, pp [19] E. Cuervo, et al., MAUI: Makng Smartphones Last Longer wth Code Offload, n Proceedngs of the 8th nternatonal conference on Moble systems, applcatons, and servces, San Francsco, CA, Jun , 2010, pp [20] S. R. Ells, K. Mana, B. D. Adelsten, and M. I. Hll, Generalzeablty of Latency Detecton n a Varety of Vrtual Envronments, n Proceedngs of the Human Factors and Ergonomcs Socety Annual Meetng, New Orleans, LA, Sep , 2004, vol. 48, no. 23, pp [21] K. Ha, et al., Adaptve VM Handoff Across Cloudlets, Techncal Report CMU-CS , CMU School of Computer Scence, [22] W. Zhang, K. T. Lam and C. L. Wang, Adaptve Lve VM Mgraton over a WAN: Modelng and Implementaton, n 2014 IEEE 7th Internatonal Conference on Cloud Computng, Anchorage, AK, 2014, pp [23] E. Cho, S. A. Myers, and J. Leskovec, Frendshp and Moblty: User Movement n Locaton-based Socal Networks, n Proceedngs of the 17th ACM SIGKDD nternatonal conference on Knowledge dscovery and data mnng, San Dego, CA, Aug , 2011, pp [24] W. Su, S. J. Lee, and M. Gerla. Moblty Predcton n Wreless Networks, n 21st Century Mltary Communcatons Conference Proceedngs(MILCOM), Los Angeles, CA, Oct , 2000, pp [25] A. Nadembega, A. Hafd, and T. Taleb, A Destnaton and Moblty Path Predcton Scheme for Moble Networks, n IEEE Transactons on Vehcular Technology, vol. 64, no. 6, pp , June [26] N. L. M. Adrchem, C. Doerr, and F. Kupers, Opennetmon: Network Montorng n OpenFlow Software-Defned Networks, n 2014 IEEE Network Operatons and Management Symposum (NOMS), Krakow, Poland, May , pp [27] C. Yu, et al., Software-Defned Latency Montorng n Data Center Networks, n th Internatonal Conference on Passve and Actve Measurement (PAM), New York, NY, Mar , 2015, pp [28] Glbert Laporte, Stefan Nckel, and Francsco Saldanha da Gama, Locaton Scence, Sprnger, ISBN: , [29] O. Karv and S. L. Hakm, An Algorthmc Approach to Network Locaton Problems II: The p-medans, SIAM Journal on Appled Mathematcs, vol. 37, no. 3, pp , [30] Mark S. Daskn, Network and Dscrete Locaton: Models, Algorthms, and Applcatons, John Wley & Sons, ISBN: , [31] M. Held, P. Wolfe, and H. P. Crowder, Valdaton of Subgradent Optmzaton, Mathematcal programmng, vol. 6, no. 1, pp , [32] M. Held and R. M. Karp, The Travelng-Salesman Problem and Mnmum Spannng Trees: Part II, Mathematcal programmng, vol. 1, no. 1, pp. 6 25, 1971.

14 13 [33] R. Landa, et al., The Large-Scale Geography of Internet Round Trp Tmes, IFIP Networkng Conference, Brooklyn, NY, May 22 24, 2013, pp [34] S. Sahn and T. Gonzalez, P-Complete Approxmaton Problems, n Journal of the ACM (JACM), vol. 23, no. 3, pp , [35] G. A. Lews, et al., Cloudlet-based Cyber-Foragng for Moble Systems n Resource-Constraned Edge Envronments, n Companon Proceedngs of the 36th Internatonal Conference on Software Engneerng, Hyderabad, Indan, May 31 Jun. 07, 2014, pp [36] Z. Xu, et al., Effcent Algorthms for Capactated Cloudlet Placements, n IEEE Transactons on Parallel and Dstrbuted Systems, vol. 27, no. 10, pp , Oct [37] Y. Zhang, D. Nyato, and P. Wang, Offloadng n Moble Cloudlet Systems wth Intermttent Connectvty, IEEE Transactons on Moble Computng, vol. 14, no. 12, pp [38] T. G. Rodrgues, et al., Towards a Low-Delay Edge Cloud Computng through a Combned Communcaton and Computaton Approach, n 2016 IEEE 84th Vehcular Technology Conference (VTC-Fall), Montreal, QC, Canada, 2016, pp [39] T. G. Rodrgues, K. Suto, H. Nshyama, and N. Kato, Hybrd Method for Mnmzng Servce Delay n Edge Cloud Computng through VM Mgraton and Transmsson Power Control, IEEE Transactons on Computers, vol. 66, no. 5, pp , May [40] Z. Chen, et al., Early Implementaton Experence wth Wearable Cogntve Assstance Applcatons, n Proceedngs of the 1st ACM Workshop on Wearable Systems and Applcatons (WearSys), Florence, Italy, May 19 22, 2015, pp [41] M. Quwader and Y. Jararweh, Cloudlet-based Effcent Data Collecton n Wreless Body Area Networks, Smulaton Modellng Practce and Theory, vol. 50, pp , [42] Z. Xu, W. Lang, W. Xu, M. Ja and S. Guo, Capactated Cloudlet Placements n Wreless Metropoltan Area Networks, 2015 IEEE 40th Conference on Local Computer Networks (LCN), Clearwater Beach, FL, Oct , 2015, pp [43] A. Cesell, M. Premol, and S. Secc, Cloudlet Network Desgn Optmzaton, n IFIP Networkng Conference (IFIP Networkng), Toulouse, France, May 20 22, 2015, pp [44] X. Sun and N. Ansar, Green Cloudlet Network: A Dstrbuted Green Moble Cloud Network, IEEE Network, vol. 31, no. 1, pp , January/February Nrwan Ansar [S 78,M 83,SM 94,F 09] s Dstngushed Professor of Electrcal and Computer Engneerng at the New Jersey Insttute of Technology (NJIT). He has also been a vstng (char) professor at several unverstes such as Hgh-level Vstng Scentst at Bejng Unversty of Posts and Telecommuncatons. Professor Ansar has authored Green Moble Networks: A Networkng Perspectve (Wley-IEEE, 2017) wth T. Han, and co-authored two other books. He has also (co-)authored more than 500 techncal publcatons, over 200 n wdely cted journals/magaznes. He has guest-edted a number of specal ssues coverng varous emergng topcs n communcatons and networkng. He has served on the edtoral/advsory board of over ten journals. Hs current research focuses on green communcatons and networkng, cloud computng, and varous aspects of broadband networks. Professor Ansar was elected to serve n the IEEE Communcatons Socety (ComSoc) Board of Governors as a member-at-large, has chared ComSoc techncal commttees, and has been actvely organzng numerous IEEE Internatonal Conferences/Symposa/Workshops. He has frequently delvered keynote addresses, dstngushed lectures, tutorals, and nvted talks. Some of hs recogntons nclude IEEE Fellow, several Excellence n Teachng Awards, some best paper awards, the NCE Excellence n Research Award, the IEEE TCGCC Dstngushed Techncal Achevement Recognton Award, the ComSoc AHSN TC Techncal Recognton Award, the NJ Inventors Hall of Fame Inventor of the Year Award, the Thomas Alva Edson Patent Award, Purdue Unversty Outstandng Electrcal and Computer Engneer Award, and desgnaton as a COMSOC Dstngushed Lecturer. He has also been granted over 30 U.S. patents. He receved a Ph.D. from Purdue Unversty n 1988, an MSEE from the Unversty of Mchgan n 1983, and a BSEE (summa cum laude wth a perfect GPA) from NJIT n Xang Sun [S 13] receved a B.E. degree n electronc and nformaton engneerng and an M.E. degree n technology of computer applcatons from Hebe Unversty of Engneerng, Hebe, Chna. He s currently workng towards the Ph.D. degree n electrcal engneerng at the New Jersey Insttute of Technology (NJIT), Newark, New Jersey. Hs research nterests nclude moble edge computng, bg data networkng, green edge computng and communcatons, Internet of Thngs, and cloud computng.