Learning with handoff cost constraint for network selection in heterogeneous wireless networks

Transcription

1 WIRELESS COMMUNICATIONS AND MOBILE COMPUTING Wirel. Commu. Mob. Comput. 16; 16: Published olie 16 October 14 i Wiley Olie Library (wileyolielibrary.com). DOI: 1.1/wcm.55 RESEARCH ARTICLE Learig with hadoff cost costrait for etwork selectio i heterogeeous wireless etworks Zhiyog Du *, Qihui Wu ad Palog Yag College of Commuicatios Egieerig, PLAUST, Najig 17, Chia ABSTRACT I heterogeeous wireless etworks, etwork selectio algorithms provide the user with the optimum etwork access choice. The optimal etwork is evaluated accordig to etwork parameters. Cosiderig that the etwork parameters are dyamic ad uavailable for the user i realistic heterogeeous wireless etwork eviromets, most existig etwork selectio algorithms caot work effectively. Learig-based algorithms ca address the problem of ucertai etwork parameters, while they commoly eed cosiderable etwork hadoff, resultig i ubearable hadoff cost. I order to tackle the ucertaity of etwork parameters, we formulate the etwork selectio problem as a multi-armed badit problem. Moreover, two olie learig-based etwork selectio algorithms with a special cosideratio o reducig etwork hadoff cost are proposed. By updatig i a block maer, both algorithms achieve optimal logarithmic-order regret ad limited etwork hadoff cost. The simulatio idicates that the two algorithms ca sigificatly reduce the etwork hadoff cost ad improve the trasmissio performace compared with existig algorithms, simultaeously. Copyright 14 Joh Wiley & Sos, Ltd. KEYWORDS etwork selectio; etwork hadoff; etwork hadoff cost; olie learig *Correspodece Zhiyog Du, College of Commuicatios Egieerig, PLAUST, Najig 17, Chia. duzhiyog1@gmail.com 1. INTRODUCTION I heterogeeous wireless etworks where differet etworks are available cocurretly [1], etwork selectio is of great importace for improvig users commuicatio quality [,3]. Most of existig approaches compare the etworks performace ad select the best oe to access [4 6], kow as always best coected. Note that the etworks performace is evaluated o the basis of etwork parameters such as badwidth, delay, ad price. However, i realistic etwork eviromets where there is o additioal protocol or etwork ifrastructure available for iformatio sharig [7], these etwork parameters are commoly dyamic ad uavailable for users, which imposes great difficulties o etwork selectio. Recetly, some etwork selectio or vertical hadoff algorithms have bee proposed [8 13] to partially tackle this problem. I the literature, etwork selectios are slot based, ad the preferred etwork is leart by dyamic iteractios with the eviromet. However, the dyamic iteractio process may lead to cosiderable etwork hadoff cost. As is kow, the etwork hadoff process icludes system discovery, hadoff decisio, ad vertical hadoff executio, which will icur additioal sigalig cotrol cost. Whe a learig-based algorithm is used, sufficiet samplig i each etwork is required, which results i frequet hadoff amog etworks, kow as the pig-pog effect. For this reaso, the learig process may icur excessive etwork hadoff cost, while few existig approaches cocer with the etwork hadoff cost problem. Although etwork hadoff ca affect the reward fuctio i [8], the log-term hadoff cost is ucertai. I [1] ad [13], Q-learig-based algorithms are adopted to lear the Nash equilibriums for etwork selectio i a multiple-user case. But these studies maily focus o users reward, ad oe of them pays attetio to the etwork hadoff cost i the learig process. Motivated by this fact, we aim to ivestigate olie learig-based etwork selectio algorithms with hadoff cost cosideratio. Reducig the etwork hadoff cost i learig-based algorithms is difficult. I olie learig theory, the user faces the iheret trade-off betwee exploratio ad exploitatio, that is, keepig eough sampligs i suboptimal actios to explore the eviromet ad selectig the potetially optimal actio as possible. Balacig this Copyright 14 Joh Wiley & Sos, Ltd. 441

2 Learig with hadoff cost costrait for etwork selectio Z. Du, Q. Wu ad P Yag trade-off i etwork selectio problem icurs etwork hadoff. The hadoff cost cosideratio further makes the desig of learig algorithms complex. I this paper, etworks are selected for each slot. O this basis, we formulate the etwork selectio as a olie learig problem ad propose two learig-based etwork selectio algorithms. The algorithms reduce the etwork hadoff cost by updatig i a block maer, rather tha by updatig i each slot. The startig poit of this desig is that with the progress of learig, the optimal etwork is selected with icreasig probability, which idicates that the expected performace loss of blockbased updatig algorithms approaches to that of slot-based updatig algorithms, ad the hadoff cost ca be sigificatly reduced. It is iterestig to fid that compared with some existig algorithms, the proposed algorithms ca reduce the hadoff cost as well as improve the trasmissio performace, simultaeously. Our mai cotributios ca be summarized as follows: The etwork selectio problem without prior etwork parameter iformatio is formulated. May existig etwork selectio algorithms assume that the etworks are static ad etwork parameters are kow. Although some works cosider the sceario where istat etwork iformatio is ukow, some prior statistical iformatio is eeded. We cosider the more realistic sceario that both the prior istat ad statistical iformatio o etwork parameters are ukow. We reformulate the etwork selectio problem i dyamic heterogeeous wireless etworks as a multiarmed badit (MAB) problem. The MAB model eables the user to optimize the log-term reward without prior istat ad statistical iformatio o etwork parameters. Two olie learig-based etwork selectio algorithms with hadoff cost cosideratio, block upper cofidece boud (UCB)1-based algorithm ad UCB-based algorithms, are proposed. Both algorithms ca reduce the hadoff cost i terms of rate of hadoff cost (RHC) compared with existig algorithms, where RHC is a ewly defied metric for measurig the etwork hadoff cost. The block UCB1-based algorithm geeralizes the origial UCB1 algorithm suitable for switchig cost-sesitive scearios. We derive the upper bouds of the regret ad RHC for the proposed two algorithms. These bouds ca give us some guides for hadoff costsesitive scearios. I the rest of this sectio, we review some related work. We maily focus o etwork selectios i the dyamic etwork eviromet, where dyamic selectio or Q-learig algorithms are used. I [8], the dyamic etwork parameter resultig from user arrival ad departure is formulated as a Markov process, ad Q-learig is used for maximizig users quality of service. A weighted bipartite graph-based algorithm for multiple flows i heterogeeous wireless etwork is proposed i [9]. Game theory [1,11] is commoly used for multiple-user scearios. The authors i [1,13] focus o resource sharig amog multiple users i a heterogeeous etwork. The iteractio amog users is formulated as a game, ad Q-learig-based algorithms are used to lear the Nash equilibriums. The hadoff cost is ot icluded i the aforemetioed works. Some existig works take ito accout the hadoff cost i differet ways. The authors i [14] icorporate the hadoff cost ito the reward ad formulate the etwork selectio problem as a Markov decisio process. Based o [14], the authors further cosider the hadoff costrait i [15]. However, they assume that the state trasitio probabilities o etwork parameter are kow ad use the policy iteratio algorithm, rather tha learig algorithms. I [16] ad [17], the sigalig cost is cosidered. The hadoff latecy is treated as the hadoff cost i [18]. I these works, the hadoff cost is a attribute i the utility for makig the selectio decisio, while the log-term hadoff cost is ucertai. The MAB problem has bee widely studied i various fields i which desigig olie learig algorithms is oe of the focuses. The reprehesive algorithms iclude the idex-based policy i [19] that ca asymptotically achieve the lower boud of the regret, while it is limited to some families of reward distributios with a sigle real parameter. The authors i [] proposed several classical idex-based algorithms that achieve logarithmic-order regret for ay reward distributio with bouded support uiformly over time, rather tha oly asymptotically. There are also some works cosiderig the switchig cost i MAB. Authors i [1] proposed a policy achievig asymptotic efficiet performace for reward distributios with a sigle parameter, which is the same with the optimal rule for the MAB problem without switchig cost. This policy was exteded to a multiple-user case i []. This policy ispires our work, while algorithms we cosidered here achieve logarithmic-order regret uiformly over time rather tha asymptotically. The orgaizatio of this paper is as follows. The system model is described i Sectio. The problem formulatio is preseted i Sectio 3. We propose two olie learig-based etwork selectio algorithms i Sectio 4. The proposed algorithms are verified by simulatios i Sectio 5. Some implemetatio issues are discussed i Sectio 6. Fially, we coclude the paper i Sectio 7.. SYSTEM MODEL We cosider a heterogeeous wireless etwork cosistig of multiple wireless access etworks, such as wireless local area etworks (WLANs) with Third-geeratio Partership Project Log-term Evolutio (LTE) etwork. Deote the etwork set as N, jn j D N. A multiplemode user termial locates i the overlapped coverage area of N etworks. Differet from most existig works, 44 Wirel. Commu. Mob. Comput. 16; 16: Joh Wiley & Sos, Ltd. DOI: 1.1/wcm

3 Z. Du, Q. Wu ad P Yag Learig with hadoff cost costrait for etwork selectio Figure 1. A sample of the cosidered system. we cosider a much more practical situatio where etwork parameters such as badwidth ad delay are ukow ad eve dyamic, which may result from the chagig traffic load i the etworks. Note that i this iformatioucertai situatio, most of existig approaches caot work effectively. I order to tackle the ucertaity of etwork parameters, we assume a slotted system where the user dyamically selects ad adjusts its etwork access decisio ı i each slot. A illustratio of the system model with three etworks ad slot legth T is show i Figure 1. Explicitly, at the begiig of the t-th slot, the user selects a etwork ı.t/ N to trasmit. If ı.t/ ı.t 1/, a etwork hadoff happes; otherwise, the user keeps o trasmittig i the curretly associated etwork. Deote vector x D Œx 1, x, :::, x L as the etwork parameter vector whose elemets x l, l D 1,, :::, L, are cocered L etwork parameters, such as badwidth ad delay. At the ed of the t-th slot, users obtai a reward r.x ı.t/.t//, wherex.t/ is the dyamic etwork parameter vector of etwork i the t-th slot; the utility fuctio r.x/ maps the experieced etwork parameters vector x.t/ ito the users reward. Note that the selectio of ı.i/ is based o the observatio o the access actio history A.t/ D fı.1/, ı./, :::, ı.t 1/g ad reward history U.t/ D fr.ı.1//, r.ı.//, :::, r.ı.t 1//g. A etwork selectio policy makes the etwork selectio decisio for each slot accordig to the observatios A.t/ ad U.t/. I additio, as it is kow that the etwork hadoff icurs hadoff cost, we icorporate the etwork hadoff cost ito cosideratio. The hadoff cost we cosider is the sigalig cost. I a etwork hadoff, the user should first iform the curretly associated etwork ad discoect from it. The, the user must discover ad sychroize with the target etwork to fiish the ew associatio. Both these stages icur sigalig cost. Furthermore, differet types of wireless etworks may have distict radio techology or protocols. Thus, we believe that the hadoff cost should deped o the ivolved two etworks. Explicitly, defie the hadoff cost matrix as C D fc m, g, m, N,wherec m, is the sigalig cost whe the user is haded off from etworks m to i two successive slots. Moreover, the cost is zero whe o hadoff occurs; that is, c m, D ifm D. 3. PROBLEM FORMULATION I this sectio, we first formulate our etwork selectio problem with a special cosideratio o the etwork hadoff cost. Because solvig the formulated problem is itractable, we the reformulate the problem as a olie learig problem ad resort to olie learig-based algorithms Hadoff cost cosideratio We assume that the slot duratio is appropriately chose such that the ivolved etwork parameters i x are fixed i a slot, while i differet slots, they are idepedet radom variables. Because of the dyamics i etwork parameters, the reward i each slot is radom. I this cotext, the etworks performace is evaluated by the expected rewards D EŒr.x /, Naturally, if, N, is kow, the optimal etwork for the user is the oe with maximal expected reward, that is, D arg max N (1) However, this paper cosiders the practical sceario that both the prior istat ad statistical iformatio o etwork parameters are ukow. This is ideed the case because the real-time etwork parameters x l, l D 1,, :::, L, of differet etworks caot be provided for the user without ay additioal protocol or etwork ifrastructure. Wirel. Commu. Mob. Comput. 16; 16: Joh Wiley & Sos, Ltd. 443 DOI: 1.1/wcm

4 Learig with hadoff cost costrait for etwork selectio Z. Du, Q. Wu ad P Yag Give a etwork selectio policy, deote the accumulative reward i t slots as S.t/ D t r.x ı.i/.i//; the P id1 expected accumulative reward is defied as " t # N EŒS.t/ D E r.x ı.i/.i// D E ŒV.t/ id1 D1 P where V.t/ D t Ifı.i/ D g is the umber of slots, id1 etwork is selected i the first t slots, Ifg is the idicator fuctio, ad E Œ meas takig expectatio uder policy. Meawhile, the etwork selectio policy icurs a etwork hadoff cost. Deote the expected" etwork hadoff tp cost up to the t-th slot as EŒH.t/ D E c ı.i 1/,ı.i/ #. id1 By treatig the etwork hadoff cost as a performace loss i the total reward, the log-term reward is EŒS.t/ EŒH.t/, where 1 is the weight for the hadoff cost. Frequet etwork hadoff i a limited time would produce a severe egative effect, which is kow as the pig-pog effect i etwork hadoff. I order to avoid excessive etwork hadoff, we defie a additioal hadoff cost metric, RHC, as.t/ D EŒH.t/ t The RHC is ideed the average hadoff cost per slot. We are goig to fid a etwork selectio algorithm or policy (we will use etwork selectio algorithm ad etwork selectio policy iterchageably i the rest of the paper) maximizig the log-term reward ad miimizig the RHC,.P/ ( max EŒS.t/ EŒH.t/ mi.t/ Geerally, fidig the solutio for.p/ may ot be very difficult i optimizatio problems. However, i this paper, the etwork parameter iformatio is dyamic ad ukow, which makes the optimal solutio impossible to obtai. Fortuately, we fid that the classical MAB problem ca well model the cosidered problem, where olie learig algorithms for the MAB problem may be used for the etwork selectio. 3.. Olie learig problem reformulatio I MAB problems, there are N arms for a player to play. I each decisio istat, the player chooses oe of the arms to play. Playig ay arm, the player will obtai a radom reward r, which follows some ukow distributio ˆ. The player s goal is to fid a policy maximizig the log-term expected reward i successive t plays. We ca reformulate our etwork selectio problem as a MAB problem: the N available etworks are the arms of the badit, the user is the player, ad accessig oe of the etworks i each slot is equivalet to playig a arm. As a result, the goal is to maximize the log-term reward. Uder this MAB problem framework, we resort to olie learig-based algorithm. We aim to desig olie learig-based etwork selectio algorithms with the followig properties: (1) Achievig logarithmic-order regret i maximizig EŒS.t/ EŒH.t/ () Achievig cotrolled RHC Property 1 cocers the received log-term reward. Note that the regret is a metric reflectig a learig algorithm s performace, which is defied as the loss of the expected rewards betwee a policy ad the ideal case that the optimal actio is kow ad selected i each decisio. I our problem, the ideal policy is stickig to the etwork defied i (1). Note that for the ideal policy, there is o hadoff cost. The, a policy s regret is defied as EŒR.t/ D EŒS.t/ feœs.t/ EŒH.t/ g D. /E ŒV.t/ C EŒH.t/ () Maximizig the log-term reward is equivalet to miimizig the regret. I learig theory, fidig the optimal policy i miimizig the regret is hard, eve impossible. Actually, people usually seek for policies achievig logarithmic-order regret with the umber of plays, which is demostrated as the order optimal growth of regret []. For this reaso, we also aim at olie learig algorithms with logarithmic-order regret as stated i property 1. O the other had, property requires cotrolled hadoff cost whe the logarithmic-order regret is achieved. This is a additioal requiremet to olie learig algorithms. I the followig, we have to desig olie learig-based etwork selectio algorithms with costraied hadoff cost. 4. ONLINE NETWORK SELECTION ALGORITHMS WITH CONSTRAINED NETWORK HANDOFF There exists several olie learig algorithms that achieve the logarithmic-order regret, while limited work explicitly focused o the accumulative switchig cost (hadoff cost i our problem). I most existig learig algorithms, actio decisio is made for each play (slot), which ievitably icurs excessive actio switch or etwork hadoff i our cotext. Our idea i reducig etwork hadoff cost is keepig a decisio for multiple successive plays. We call this kid of learig as block-based olie learig. We evisio two types of block patter: oe is costat block legth, ad the other oe is variable block legth. For costat block legth, we ca keep each actio decisio 444 Wirel. Commu. Mob. Comput. 16; 16: Joh Wiley & Sos, Ltd. DOI: 1.1/wcm

5 Z. Du, Q. Wu ad P Yag Learig with hadoff cost costrait for etwork selectio for costat successive plays. For variable block legth, we cotrol the block legth to gradually grow with time. These two types of block patters are expected to brig about differet impacts o the hadoff cost ad regret. Fortuately, we ca derive such etwork selectio algorithms based o classical learig algorithms. A questio arises o how to set the block patter to reduce the switchig cost without violatig the logarithmic regret property? I the followig, we preset ad validate two algorithms, block UCB1-based algorithm ad UCB-based algorithm Costat block legth: block UCB1-based etwork selectio algorithm UCB1 [] is a widely used olie learig algorithm. However, the origial UCB1 does ot cosider the switchig cost; as a result, there are high probabilities to switch actio especially at the early stage of the algorithm. We modify the UCB1 to obtai a block UCB1-based etwork selectio algorithm as show i Algorithm 1. The block UCB1-based algorithm works i a block form; that is, whe a etwork is selected, it will be fixed for m successive slots, where m is a iteger larger tha 1. Note that m is a iput parameter to cotrol the block legth. The proposed algorithm works as follows: at the begiig, each etwork is selected for oe q block of slots. After that, the etwork maximizig Or C m log k is selected for each block, where k is the block idex, Or is the sample mea of the reward for the trasmissio i etwork, ad is the umber of blocks i which etwork is selected. The upper bouds of the regret ad RHC of the block UCB1-based etwork selectio algorithm is provided by Theorem 1. Algorithm 1. Block UCB1-based etwork selectio algorithm Iitiate: Set block legth as m. Select each etwork for oe block of slots for the first N blocks, k D N. Loop k D k C 1 q Select ı.k/ D arg max Or C m log k N for successive m slots. Update ı.k/ D ı.k/ C 1 ad Or ı.k/ D kc1 P gmp Ifı.t/Dı.k/gr.t/ gd1 td.g 1/mC1 ı.k/. Ed loop its RHC is upper bouded as block.t/ c C c t : < Œ1C 8logt m # C 3 where D ad c D max c m, is the maximal m hadoff cost to the optimal etwork. Remark 1. The proof of Theorem 1 ca be foud i the Appedix. Theorem 1 idicates that the block UCB1-based algorithm ca achieve the logarithmic-order regret, while the RHC boud is i the form of t. Moreover, m is a adjustable parameter for both the regret ad RHC. Give a fixed t, it seems that the expected regret ad RHC decrease with the icrease of m. Our simulatio results idicate that the property of these bouds accords with that of the actual regret ad RHC. This gives us some isights to adjust the algorithm s performace. 4.. Variable block legth: UCB-based etwork selectio algorithm Aother learig algorithm i [] is UCB. Although the origial UCB aims at the order optimal regret without the hadoff cost cosideratio, we foud it works i a block-similar way with variable block legth, which may cotribute to reducig switchig cost. We the derive the UCB-based etwork selectio algorithm as show i Algorithm. The algorithm works as follows: at the begiig, the first N slots are trasmitted i each etwork oce to esure that all the etworks are accessed by the user. From the o, the algorithm is updated by icreasig legth blocks. At the ed of each block, the etwork for the ext block is determied by Algorithm. UCB-based etwork selectio algorithm Iitiate: D, N, i D. Select each etwork i N oce for the first N slots. Loop i D i C 1 Select ı.i/ D arg max N for C a i, g for successive. ı.i/ C 1/. ı.i/ / slots. Update ı.i/ D ı.i/ C 1; Or is the sample mea of reward for trasmittig a slot i etwork. Ed loop (4) Theorem 1. The expected regret of the block UCB1- based etwork selectio algorithm is upper bouded as EŒR block.t/ : < m C c C c Œ1C 8logt m # C 3 (3) D arg max N for C a i, g (5) where Or is the sample mea of the reward by trasmittig aslotietwork, is the umber of blocks i which etwork is selected, i is q the total block idex,./ D d.1 C / e,ada i, D.1C / logœei=./,< <1,./ is a parameter cotrollig the growth patter of the block legth. Note that. ı.i/ C 1/. ı.i/ /is the block legth Wirel. Commu. Mob. Comput. 16; 16: Joh Wiley & Sos, Ltd. 445 DOI: 1.1/wcm

6 Learig with hadoff cost costrait for etwork selectio Z. Du, Q. Wu ad P Yag that geerally icreases with ı.i/ ; that is, the more a etwork is selected, the larger is the block legth i which this etwork is selected. The origial regret of the UCB algorithm, which is equivalet to P. / E ŒV.t/ i (), has bee prove i the literature as E E ŒV.t/ : < c D 1C.1 C /.1 C 4 / log e t C c (6).1 C /e.1 C /1C C 1 C 11.1 C / 5 log.1 C / We derive the regret boud ad RHC boud of the UCB- based etwork selectio algorithm i Theorem. Theorem. The RHC of UCB-based etwork selectio 1 algorithm for ay t max : < is upper bouded as UCB.t/ c C c,t (7) t : < " # 1.1 C 4 / log et,t D 1 C log.1 C / log ( ".1 # C / C log exp log.1 C / 4 C exp. 1 / C Its regret is upper bouded as 11.1 C / exp. 1 / 5 log.1 C / (8) EŒR UCB.t/ c C c,t : < C : <.1 C /.1 C 4 / log et C c (9) 5. SIMULATION RESULTS The simulatio sceario cosists of two WLANs represeted as WLAN1 ad WLAN, ad a LTE etwork, as show i Figure. We assume that a user locates i the overlappig area of three etworks; that is, WLAN1 + WLAN + LTE are available simultaeously. Followig the same way with [14], we maily focus o the impact of the badwidth ad the delay o users reward ad defie the utility fuctio r.b, d/ as r.b, d/ D wf 1.b/ C.1 w/f.d/ (1) where b ad d are the experieced badwidth ad delay, respectively, f 1.b/ is a badwidth-related reward, ad f.d/ is a delay-related reward, 8 ˆ< f 1.b/ D ˆ: 8 ˆ< f.d/ D ˆ: 1,.b b mi /.b max b mi /,, 1,.d max d/.d max d mi /,, b b max b mi < b < b max b b mi (11) < d d mi d mi < d < d max d d max (1) w 1 is the weight determiig the importace of the badwidth reward ad the delay reward. I f 1.b/, b mi ad b max are two badwidth thresholds represetig the required miimal ad maximal badwidths, respectively. Obviously, whe the achieved badwidth is larger tha the maximal badwidth b max, the user caot beefit ay more. O the other had, for the achieved badwidth smaller tha the miimal badwidth b mi,theyare meaigless because the user caot obtai ay reward i such low badwidths. The badwidth reward icreases liearly with the achieved badwidth whe it is i the rage of Œb mi, b max. Similarly, d mi ad d max are the two delay thresholds i f.d/. The badwidth thresholds, delay where c D max kn c k, is the maximal hadoff cost to etwork, D. Remark. See the proof i the Appedix. It is see that the order optimal regret is achievable for the UCB- log./ based algorithm. The RHC boud is i the form of t, which evetually decreases as t icreases. Compared with the block UCB1-based algorithm, it seems that the RHC i the UCB algorithm has a faster decreasig speed. is the key parameter for both the regret ad RHC. Although it is difficult to aalyze the relatioship of ad the bouds, our simulatio results reveal that both the bouds ad the actual regret ad RHC decrease as icreases. Figure. The cosidered simulatio sceario. 446 Wirel. Commu. Mob. Comput. 16; 16: Joh Wiley & Sos, Ltd. DOI: 1.1/wcm

7 Z. Du, Q. Wu ad P Yag Learig with hadoff cost costrait for etwork selectio Table I. Network parameters. b uit (Mb/s) B max B mi d uit (ms) D max D mi WLAN1.3 1b uit 3b uit 5 6d uit d uit WLAN.3 8 b uit 3b uit 5 1d uit d uit LTE.5 8 b uit 4b uit 5 6d uit 4d uit LTE, Log-term Evolutio; WLAN, wireless local area etwork WALN1 WLAN LTE.7.6 Ratio Slot idex Figure 3. A sample ru of the block UCB1-based algorithm with m D 1. thresholds, ad w ca be set accordig to either some recommedatios [3] or user prefereces. I the simulatio, b mi D.5 Mb/s, b max D 8Mb/s, d mi D 1 ms, d max D 6 ms, ad w D.5. We stress that the cocered parameters ca be differet ad the utility ca be defied i differet ways, but this is ot the mai cocer of this paper. We adopt the discrete model similar with [14] to represet the badwidth ad delay dyamics i etworks. Explicitly, the badwidth ad delay i etworks are approximated to be several levels, with the uit badwidth b uit ad uit delay d uit as the miimal iterval, respectively. The maximal badwidth B max, miimal badwidth B mi, maximal delay D max, ad miimal delay D mi are show i Table I. I each slot, the joit badwidth delay state i each etwork ca be oe of possible values with some ukow distributio. Without loss of geerality, we assume that the joit badwidth delay state follows a uiform distributio o its state space. The etwork is statioary if the statistical distributio of the badwidth delay state remais uchaged i differet slots. The slot legth isassumedtobe1s. The omial hadoff cost matrix amog these etworks is C D WLAN1 WLAN LTE WLAN1..3 WLAN..5 LTE.3.5 where the elemet i the i-th row ad j-th colum of the 3 3 matrix is the correspodig hadoff cost whe the user switches from the i-th etwork i the first colum to the j-th etwork i the first row. Because WLAN1 ad WLAN have similar radio techology, the hadoff costs betwee them are smaller. O the other had, because LTE ad WLAN have totally differet radio techologies, the correspodig hadoff costs are larger. The followig simulatios are preseted from two aspects. O oe had, we have to see the impact of the parameters m ad o the performace of block UCB1-based ad UCB-based etwork selectio algorithms, respectively. O the other had, we compare the performace of differet etwork selectio algorithms i differet scearios Impact of parameters First of all, we have to test the covergece of the proposed algorithms. Figures 3 ad 4 show the sample rus of the block UCB1-based algorithm ad UCB-based algorithm. The Y-axis is the accumulative selectio ratio of a etwork. We ca observe that for both algorithms, WLAN1 is selected icreasigly frequetly with time, while the other two etworks are selected less ad less. Note that although the covergece treds are the same, the evolutio patters of the two algorithms are differet because of differet update rules ad block legths. The selectio Wirel. Commu. Mob. Comput. 16; 16: Joh Wiley & Sos, Ltd. 447 DOI: 1.1/wcm

8 Learig with hadoff cost costrait for etwork selectio Z. Du, Q. Wu ad P Yag WLAN1 WLAN LTE.7.6 Ratio Slot idex Figure 4. A sample ru of the UCB-based algorithm with D.5. Ratio of selectig the optimal etwork (WLAN1) UCB1(m=1). m= m=4.1 m=8 m= Slot idex Figure 5. The covergece of the block UCB1-based algorithm. ratio of WLAN1 i the UCB-based algorithm with D.5 is geerally larger tha that i the block UCB1-based algorithm with m D 1. Accordig to the descriptios of the algorithms, m ad are the key parameters i the block UCB1-based algorithm ad UCB-based algorithm, respectively. Hece, we adjust these two parameters ad study their impacts o the algorithms. We show the ratio of selectig the optimal etwork, WLAN1, as a reflectio of the covergece speed of the algorithms i Figures 5 ad 6. The results are averaged by 5 rus. We ca fid that after about 1 slots, the covergece speed icreases with the icrease of the block legth for block UCB1, while the covergece speed icreases with the decrease of for UCB. Also, the UCB-based algorithm seems to have a faster covergece speed tha the block UCB1-based algorithm i this simulatio settig. We further focus o the regret ad RHC of the algorithms. I Figure 7, we vary m to see its impact o the regret of the block UCB1-based etwork selectio algorithm. Each simulatio lasts for 1 slots ad are averaged by 5 rus. It is see that the expected regrets with differet m grow sub-liearly with the slot idex icrease. Compared with UCB1, the block UCB1-based algorithms achieve sigificat smaller regret. Especially, a larger m results i a smaller regret. The results validate both the logarithmic-order regret ad the superiority of the block UCB1 algorithm revealed i (3). However, the fluctuatio of the regret i time becomes more serious whe m is larger tha 8. This is because whe m is too large, the short-term liear growth i regret emerges at times. For this reaso, m should ot be too large. We the preset the RHC of UCB1-based algorithms i Figure 8. We ca 448 Wirel. Commu. Mob. Comput. 16; 16: Joh Wiley & Sos, Ltd. DOI: 1.1/wcm

9 Z. Du, Q. Wu ad P Yag Learig with hadoff cost costrait for etwork selectio 1 Ratio of selectig optimal etwork (LWAN1) =.1 =.5.1 =.1 = Slot idex Figure 6. The covergece of the UCB-based algorithm upper boud, m=1 upper boud, m= simulatio, m=1 simulatio, m= simulatio, m=4 simulatio, m=8 simulatio, m= Regret 4 3 upper boud, m=4 upper boud, m=8 1 upper boud, m= Slot idex Figure 7. The regrets of the block UCB1-based etwork selectio algorithms. obtai the observatios that the RHC shows a peak i the first 1 slots ad gradually decreases with the icrease of the slot idex for all five cases. This ca be explaied by the fact that i the early stage of learig algorithms, frequet etwork hadoff occurs for explorig the eviromet. After that, the optimal etwork is selected more ad more ofte as the learig algorithm progresses; thus, hadoff becomes more ad more rare. Geerally, the peak RHC ad the stable RHC decrease as the m icreases, ad UCB1 icurs the largest RHC, i accordace with (4). For m 4, the RHC is almost costat after about 4 slots. Figures 7 ad 8 idicate that the hadoff cost reductio does ot lead to reward loss. Geerally, i case of the short-term liear growth i the regret, a media m such as 1 < m < is recommeded. Moreover, we ca observe that the upper bouds of the regret ad RHC are cosiderable loose i Figures 7 ad 8. The reaso is that the upper bouds are built o the properties of the order optimal etwork selectio policy rather tha the optimal oe. Nevertheless, the growth treds of the upper bouds with regard to the slot idex ad m geerally accord with the actual regrets ad RHC. Figures 9 ad 1 show the expected regret ad RHC for the UCB-based etwork selectio algorithm with differet. As expected i (9), for a give, the expected regret grows sub-liearly as the slot idex icreases, ad alarger geerally leads to a smaller regret. Also, the regrets show fluctuatio whe exceeds some threshold, for example,.4. Similar with block UCB1-based algorithm, the RHC of UCB-based algorithm has a peak i the first 1 slots ad decreases with the icrease of slot idex. Sigificatly differet from block UCB1-based Wirel. Commu. Mob. Comput. 16; 16: Joh Wiley & Sos, Ltd. 449 DOI: 1.1/wcm

10 Learig with hadoff cost costrait for etwork selectio Z. Du, Q. Wu ad P Yag RHC upper boud, m=1 upper boud, m= upper boud, m=4 upper boud, m=8 upper boud, m= simulatio, m=1 simulatio, m= simulatio, m=4 simulatio, m=8 simulatio, m= Slot idex Figure 8. The RHC of the block UCB1-based etwork selectio algorithms. 6 Regret upper boud, α=.5 upper boud, α=.1 simulatio, α=.1 simulatio, α=.5 simulatio, α=.1 simulatio, α=.4 1 upper boud, α= Slot idex Figure 9. The regrets of the UCB-based etwork selectio algorithms. RHC simulatio, α=.1 simulatio, α=.5 simulatio, α=.1 simulatio, α=.4 upper boud, α=.5 upper boud, α=.1 upper boud, α= Slot idex Figure 1. The RHC of the UCB-based etwork selectio algorithms. 45 Wirel. Commu. Mob. Comput. 16; 16: Joh Wiley & Sos, Ltd. DOI: 1.1/wcm

11 Z. Du, Q. Wu ad P Yag Learig with hadoff cost costrait for etwork selectio algorithm, the peak RHC almost remais uchaged for differet. This is because the block legth is small at the early stage of the UCB-based algorithm idepedet from. From this poit of view, the peak RHC seems to be uavoidable. We ca coclude that both the expected regret ad RHC of UCB-based algorithm decrease as icreases, i accordace with (7) ad (9);.1.4 is feasible i this cotext. Similar with the block UCB1- based algorithm, although the upper bouds of the regret ad RHC are quite loose, their growth treds with the slot idex ad geerally accord with the actual regrets ad RHC. 5.. Performaces compariso We further compare the performaces of differet etwork selectio policies. Five additioal algorithms are cosidered here. Oe is the stochastic policy that radomly selects a etwork for each slot. Aother three algorithms are fixed etwork access policy, which stick to WLAN1, WLAN, ad LTE all the time. The fifth algorithm is a ideal vertical hadoff algorithm where the istat badwidth ad delay iformatio are assumed to be available ad the etwork maximizig the et reward is selected i each slot, that is, ı.i/ D arg max r.x.i// c ı.i 1/,. This algorithm is the optimal oe. Note that whe oly the statistical distributios of the badwidth delay state of all etworks are kow, the optimal etwork selectio policy is defied as i (1), that is, stickig to WLAN1 i this sceario. I case of cofusio, we call it a statistically optimal policy. We set m D 1 ad D.4 i this subsectio for the block UCB1-based algorithm ad UCB-based algorithm, respectively. Both the log-term reward ad RHC are averaged by 5 rus. The log-term rewards of these algorithms are show i Figure 11. It ca be see i Figure 11 that with the slot idex icrease, the log-term rewards of all algorithms icrease. The vertical hadoff algorithm has the largest reward, slightly better tha that of the statistically optimal policy (WLAN1). The block UCB1-based algorithm ad UCB-based algorithm have early the same results, which is smaller tha the WLAN1 ad vertical hadoff algorithm ad larger tha the other three algorithms. Without learig ability, the stochastic policy ca oly achieve a log-term reward slightly larger tha the poorest oe (stickig to WLAN) ad sigificatly poorer tha that of the block UCB1-based ad UCB-based algorithms. We tur to the RHC as show i Figure 1. As ca be see, the stochastic policy has a costat RHC of. after about 1 slots, which is the largest of all. O the other had, the vertical hadoff algorithm has the smallest RHC. Most of all, both the block UCB1-based algorithm ad the UCB- based algorithm have a RHC smaller tha.5 after slots. The RHC of these two algorithm keeps decreasig ad approaches a stable value smaller tha.. The precedig result idicates that our proposed block UCB1- based ad UCB-based algorithms ca effectively couter the ucertaity i etwork parameters ad have cotrolled etwork hadoff cost. I additio, otice that the peak RHC of the UCB-based algorithm i its early stage is fairly large, which may be oe primary cosideratio i practical implemetatios. The parameter plays a key role i the log-term reward. A larger idicates that the user is more sesitive to the egative effect of etwork hadoff cost. Therefore, we vary the etwork hadoff cost weight to see its impact o the the proposed algorithms. I Figure 13, the ratios of the log-term rewards i differet algorithms are show. The log-term rewards are sampled i the 1th slot for each algorithm. As ca be see, the performace gais of the two proposed algorithms over the stochastic policy grow as icreases. Whe the hadoff cost is igored, that is, D, the log-term reward ratio is aroud 1., while for a large weight, that is, D.8, the ratio approaches. This pheomeo illustrates that the proposed algorithms outperform the stochastic algorithm with differet Log term reward vertical hadoff UCB block UCB1 WLAN LTE stochastic statistically optimal (WLAN1) Slot idex Figure 11. The log-term rewards of differet algorithms. Wirel. Commu. Mob. Comput. 16; 16: Joh Wiley & Sos, Ltd. 451 DOI: 1.1/wcm

12 Learig with hadoff cost costrait for etwork selectio Z. Du, Q. Wu ad P Yag.35.3 UCB block UCB1 stochastic vertical hadoff.5 RHC Slot idex Figure 1. The RHC of differet algorithms block UCB1 vs WLAN1 UCB vs WLAN1 block UCB1 vs stochastic UCB vs stochastic UCB1 vs vertical hadoff UCB vs vertical hadoff Ratio φ Figure 13. The ratio of log-term rewards of differet algorithms with differet weights.. I particular, the performace gais icrease with the icrease of. O the other had, the log-term reward ratios with the optimal policy WLAN1 ad vertical hadoff algorithm are larger tha.9 ad.8 for differet, respectively. This idicates that the proposed algorithms perform well for differet etwork hadoff cost weights compared with algorithms with prior etwork parameter iformatio. Furthermore, we simulate the log-term reward ratios i a more geeral sceario where the badwidth ad delay i two successive slots chage accordig to some Markov chai model. We ca obtai a similar result with Figure 13 for this sceario. Summarily, both the proposed etwork selectio algorithms ca achieve a fair log-term reward with cotrolled etwork hadoff cost i differet scearios. I additio, the block UCB1-based algorithm outperforms the UCB-based algorithm cosiderig that the block UCB1-based algorithm has a much smaller peak RHC tha the UCB-based algorithm. We also cosider two o-statioary scearios. The first sceario focuses o the case where the traffic loads of etworks chage ad hece etworks performace chages. Figures 14 ad 15 show the log-term rewards ad RHC curves i the situatio where WLAN1 becomes overloaded i the 3th slot. We assume that i WLAN1, the probabilities o better states (with smaller delay ad higher badwidth) after the 3th slot are decreased. As a result, the LTE becomes the statistically optimal etwork. Comparig Figures 11 ad 1, we foud that the growth rates i the log-term rewards of stickig to WLAN1 ad stochastic algorithms slow dow otably. Although the block UCB1- based ad UCB-based algorithms could be affected by WLAN1 s load chage, they ca still achieve fair performace, slightly worse tha stickig to LTE. This idicates that these two algorithms ca adjust themselves to the etwork load chage to some extet. O the other had, the RHC of the UCB-based algorithm ad stochastic 45 Wirel. Commu. Mob. Comput. 16; 16: Joh Wiley & Sos, Ltd. DOI: 1.1/wcm

13 Z. Du, Q. Wu ad P Yag Learig with hadoff cost costrait for etwork selectio Log term reward WLAN1 UCB block UCB1 WLAN LTE stochastic vertical hadoff 5 WLAN1 s load chages Slot idex Figure 14. The log-term rewards of differet algorithms whe WLAN1 becomes overloaded UCB block UCB1 stochastic vertical hadoff.5 RHC WLAN1 s load chages Slot idex Figure 15. The RHC of differet algorithms whe WLAN1 becomes overloaded. algorithm seems to be ot affected by WLAN1 s load chage, while the RHC of the block UCB1-based algorithm icreases to a high level immediately after the 3th slot. This results ca be illustrated as follows. The stochastic algorithm is ot aware of the loads of etworks; thus, its RHC is ot affected by WLAN1 s load chage. Because the block legth i the UCB-based algorithm icreases with time, the etwork hadoff itself happes very rarely after hudreds of slots. As a result, the chage i the hadoff cost caused by the etwork load chages is egligible i the UCB-based algorithm. O the cotrary, the block UCB1-based algorithm has a costat block legth, ad the chage i the hadoff cost caused by the etwork load chages ca be otable for a medium m. Moreover, the RHC of vertical hadoff algorithm decreases after the 3th slot because of less etwork switch. The results i this sceario idicate that the proposed algorithms ca adapt to the load chage of etworks, while their reactio i RHC ca be slightly differet. The secod o-statioary sceario cosiders the mobility of the user. We assume that the user travels from poits A to B ahe red route as show i Figure. The simulatio eviromet ca be divided ito three stages, where i the first 3 slots, the available etworks are WLAN1 + LTE, betwee the 3th ad 6th slots, the available etworks are WLAN1 + WLAN + LTE, ad after the 6th slot, the available etworks are WLAN + LTE. The statistically optimal etworks i these three stages are WLAN1, WLAN1, ad LTE, respectively. The user moves i a relatively slow velocity such that the eviromet i these three stages is statioary. We ca see i Figure 16 that the log-term reward growth rates of the statistically optimal policy, block UCB1- based algorithm, ad UCB-based algorithm chage at Wirel. Commu. Mob. Comput. 16; 16: Joh Wiley & Sos, Ltd. 453 DOI: 1.1/wcm

14 Learig with hadoff cost costrait for etwork selectio Z. Du, Q. Wu ad P Yag Log term reward block UCB1 UCB stochastic vertical hadoff statistically optimal WLAN1+LTE 1 5 WLAN1+WLAN+LTE WLAN+LTE Slot idex Figure 16. The log-term rewards of differet algorithms i the mobility sceario WLAN1+LTE block UCB1 UCB stochastic vertical hadoff.5 WLAN1+WLAN+LTE WLAN+LTE RHC Slot idex Figure 17. The RHC of differet algorithms i the mobility sceario. the 6th slot. WLAN1 is the statistically optimal etwork before the 6th slot, ad the reward growth rates of the two proposed algorithms decrease slightly after the 3th slot. The reaso is that the chage i available etworks from WLAN1 + LTE to WLAN1 + WLAN + LTE leads to adaptio of the algorithms ad fially results i performace loss. Nevertheless, the two proposed algorithms achieve larger log-term rewards tha the stochastic algorithm. I additio, we observe that the RHC of these algorithms i Figure 17 chages to some extet compared with that i Figure 1. Naturally, the RHC of the stochastic algorithm chages i differet stages because of the differet available etworks. The RHC of the block UCB1- based algorithm ad UCB-based algorithm i the mobility sceario is slightly smaller tha those i Figure 1. This is maily because the average available etworks i the mobility sceario are less tha that i the statioary sceario, which idicates less etwork hadoff. The precedig results demostrate that i the mobility sceario, the proposed algorithms ca achieve comparable log-term reward as i the statioary sceario, while icurrig limited etwork hadoff cost. Fially, we briefly evaluate the algorithms from the system s perspective. We compare the blockig probabilities of differet algorithms. I the cosidered sceario, the etwork parameter settigs are the same with those i Sectio 5.1 at the first 3 slots, while after the 3th slot, the dyamic traffic loads of WLAN1 ad LTE chage to a relatively high level. As a result, the radom badwidth is uiformly distributed i the rage of oe to six badwidth uits, ad the radom delay is uiformly distributed i the rage of 8 14 delay uits i WLAN1. For the LTE, the radom badwidth is uiformly distributed i the rage of oe to eight badwidth uits, ad the radom delay is uiformly distributed i the rage of 4 14 delay uits. The user is blocked if the miimal badwidth ad delay 454 Wirel. Commu. Mob. Comput. 16; 16: Joh Wiley & Sos, Ltd. DOI: 1.1/wcm

15 Z. Du, Q. Wu ad P Yag Learig with hadoff cost costrait for etwork selectio vertical hadoff block UCB1 UCB stochastic Blockig probability Slot idex Figure 18. The blockig probabilities of differet algorithms. requiremets caot be met simultaeously. The blockig probabilities averaged by 5 rus are show i Figure 18. We ca observe that before the 3th slot, all algorithms icur o blockig. O the other had, whe the traffic load of WLAN1 ad LTE icreases after the 3th slot, the blockig evet rises i all algorithms except for the vertical hadoff algorithm. Nevertheless, as the slot idex icreases, the blockig probabilities of block UCB1 ad UCB geerally decrease. Whe we compare the earcostat ad larger blockig probability i the stochastic algorithm with that i the proposed algorithm, we ca coclude that the proposed algorithms ca improve the system capacity to some extet. 6. IMPLEMENTATION ISSUES The proposed olie learig-based etwork selectio algorithms are simple ad feasible for practical implemetatios. Oe of the mai cosideratios o the algorithms is the covergece speed. The theoretical performace requiremet of the algorithms leads to a relatively log covergece time. However, the covergece time makes limited differece o its practical implemetatio because the olie learig idicates that the learig process accompaies the ormal trasmissio ad has limited impact o the user. Furthermore, we ca also cut dow the covergece time for practical cosideratios: ispired by the fact that the optimal etworks are selected icreasigly frequetly with time, we ca stop the olie learig ad stick to the most frequetly selected etwork. This simplificatio ca be performed oce there is a domiatig etwork i the selectio decisio history. Aother issue is the choice of the utility fuctio. The proposed algorithms do ot specify the utility fuctio. Actually, the utility fuctio ca icorporate both techical ad o-techical aspects. The techical aspect icludes quality-of-service parameters, such as delay, throughput, ad packet loss rate. The o-techical aspect may iclude ecoomical parameters, user preferece, ad so o. The choice of utility fuctio depeds o the specific sceario ad requiremet, which are beyod the focus of this paper. For more details o the utility fuctios i etwork selectio, we refer iterested readers to [4,5]. 7. CONCLUSION I this paper, we studied the etwork selectio problem i heterogeeous etworks with a dyamic etwork eviromet. We formulated the etwork selectio with dyamic ad ukow etwork parameter as a MAB problem ad proposed two olie learig-based etwork selectio algorithms: block UCB1-based algorithm ad UCB-based algorithm. Cosiderig the egative effect of frequet etwork hadoff, the two algorithms are specially desiged for reducig the hadoff cost. Simulatio demostrates that both algorithms ca sigificatly reduce the hadoff cost ad improve the expected accumulative reward compared with some existig algorithms, simultaeously. ACKNOWLEDGEMENTS This work was supported by the Natioal Sciece Foudatio of Chia uder Grat ad , ad i part by the Jiagsu Provice Natural Sciece Foudatio uder Grat BK APPENDI A: A.1. Proof of Theorem 1 Proof. Give the slot idex t, the block umber, K D tm i the block UCB1-based algorithm..k/ is the Wirel. Commu. Mob. Comput. 16; 16: Joh Wiley & Sos, Ltd. 455 DOI: 1.1/wcm

16 Learig with hadoff cost costrait for etwork selectio Z. Du, Q. Wu ad P Yag umber of blocks that etwork is selected i the first K blocks,.k/ D 1 C K I fı.km/ D g kdnc1 K l C I fı.km/ D,.k 1/ lg kdnc1 ( s K m log k l C I Or.k/ C kdnc1.k/ s ) m log k Or.k/ C.k/,.k 1/ l ( s K Or.k/ D l C I m C logk m kdnc1.k/ s ) Or.k/ m C logk m.k/,.k 1/ l Note that Or.k/ m is the sample mea of r, based o the Cheroff Hoeffdig boud [], we ca have E Œ.K/ 8logK m C 1 C 3 8Cm m m C 1 C 3 8log.t C m/ 8logm D m C 1 C 3 < 8logt m The hadoff cost boud is C 1 C 3 (A. 1) E ŒH block.t/ E 4 3 c.k/ 5 : < C c E 4 3.K/ 5 : < D c E Œ.K/ : < C c E Œ.K/ : < c C c " # 8logt m : < C 1 C 3 (A. ) where D, c D max m c m, is the maximal hadoff cost for a hadoff to the optimal etwork ad K D t m is the block umber for t. Note that the precedig case correspods to the worst sceario where ay two suboptimal etwork selectios are separated by oe optimal etwork selectio. The expected regret ca be bouded as E ŒR block.t/ m E Œ.K/ C E ŒH.t/ : < m Cc " # 8logt Cc m : < C 1 C 3 (A. 3) This completes the proof. A.. Proof of Theorem Proof. Similarly, the hadoff cost ad regret for the UCB-based algorithm are as follows: E ŒH UCB.t/ 3 3 E 4 c 5Cc E 4 5 : < : < D : < c C c E Œ (A. 4) E ŒR UCB.t/ E Œ CE ŒH UCB.t/ : < (A. 5) where c D max kn c k, is the maximal hadoff cost whe the selected etwork is. We boud H UCB.t/ ad R UCB.t/ by fidig the boud for E Œ,,which is the expected umber of blocks i which etwork is 1 selected. For ay t max <, D, deote Q as the largest iteger satisfyig.q 1/.1 C 4 / log et Accordig to the defiitio of./,. / t, for8; the log.1 C / Therefore, 1 C Q C log.1c / >1 log.1c / >Q Ifetwork has started the -th blockg Ifetwork has started the -th blockg (A. 6) 456 Wirel. Commu. Mob. Comput. 16; 16: Joh Wiley & Sos, Ltd. DOI: 1.1/wcm

17 Z. Du, Q. Wu ad P Yag Learig with hadoff cost costrait for etwork selectio Accordig to [], the evet etwork has started the -th block implies 9i such that at least oe of the followig two evets holds Thus, E ΠQ C C Or,. 1/ C a t, 1 Or.i/ C a. 1/C.i/,i log.1c / >Q P log.1c / >Q i Or,. 1/ Ca t, 1 P Or.i/ C a. 1/C.i/,i At last, usig the Cheroff Hoeffdig boud, BD log.1c / >Q log.1c / >Q i i P Or.i/ C a. 1/C.i/,i exp.i/.i/ exp i.i/.i/ exp i.i/.1c / log log.1c / e exp. 1 / >Q. 1/ C.i/.i/.i/. 1/ C.i/ exp. 1 / log.1 C / From [], (.i/ ).i/ exp < 1 C i 11.1 C / 1C 5 log.1 C / I the followig, we boud E Πh by boudig its i.1c4 / log.et compoets. Because.Q 1/ /,.1 C / Q 1 d.1c / Q 1 ed.q 1/, the, The, B 1 C 11.1 C / 5 log.1 C / e.1c / log.1 C / (A. 1) Deote Q 1 C " # 1.1 C 4 / log et log.1 C / log (A. 7) A./ D P Or,. 1/ C a t, 1 B D log.1c / >Q Accordig to [], P Or.i/ Ca. 1/C.i/,i i A./ D P Or,. 1/ C a t, 1 C o exp. 1/ = (A. 8) I additio, t max < 1= ; thus,. 1/ >.1 C 4 / log et.1 C 4 / (A. 9) Combiig (A. 7), (A. 9), ad (A. 1), we have E Π1 C C " 1.1 C 4 / log et log.1 C / log ( " # log.1 C / Cexp. 1 /C which completes the proof. REFERENCES exp.1 C / C / exp. 1 / 5 log.1 C / 1. Lioh A, Magagula H, Cha A, Falowo OE. Hadover approaches for seamless mobility maagemet i ext geeratio wireless etworks. Wireless Commuicatios ad Mobile Computig 1; 16(1): Kwo O, Oh H, Lee Z, Lee G, Park Y, Sog H. Etire etwork load-aware cooperative routig algorithm for video streamig over mobile ad hoc etworks. Wireless Commuicatios ad Mobile Computig 13; 13(1): Ako M, Islam M, She, Sigh A. OUR: Optimal Update-based Replacemet policy for cache i wireless data access etworks with optimal effective hits ad badwidth requiremets. Wireless Commuicatios ad Mobile Computig 13; 13(15): ) # Wirel. Commu. Mob. Comput. 16; 16: Joh Wiley & Sos, Ltd. 457 DOI: 1.1/wcm