Q-learning Based Adaptive Zone Partition for Load Balancing in Multi-Sink Wireless Sensor Networks

Transcription

1 Q-learning Based Adapive Zone Pariion for Load Balancing in Muli-Sink Wireless Sensor Neworks Sheng-Tzong Cheng and Tun-Yu Chang Deparmen of Compuer Science and Informaion Engineering, Naional Cheng Kung Universiy, Tainan, R.O.C. Absrac - In many researches on load balancing in Muli- Sink WSNs, sensors usually choose he neares sink as desinaion for sending daa. If all sensors in his area all follow he neares-sink sraegy, sensors around neares sink called hospo will exhaus energy early and his sink is isolaed from nework. In his paper, we propose a load balancing scheme for muli-sink WSNs. A mobile anchor wih direcional anenna is inroduced o adapively pariion he nework ino several zones so he raffic load in he region can be assigned o he sink. Besides, o adap o differen daa raffic paern, we apply machine learning o mobile anchor and implemen a Q-learning agen. Through ineracions wih environmen, he agen can discovery a near-opimal conrol policy for movemen of mobile anchor and achieve minimizaion of residual energy s variance among sinks, which preven he early isolaion of sink and prolong he nework lifeime. Keywords: Wireless Sensor Nework, Muli-Sink, Load Balancing, Machine Learning, Q-learning 1 Inroducion The wireless sensor neworks (WSNs) are widely used in a large variey of applicaions such as miliary, ocean and wildlife monioring. The WSNs consis of a large number of low-cos devices called sensors, which monior curren saus of environmen and send sensing daa o he sink node. Because of he limiaions of he energy supply, sorage space and compuaion of sensor nodes, he daa ransmied beween a sensor node and he sink node mus been forwarded by oher sensor nodes. The muli-hop WSNs wih one sink have been developing for a long ime, bu many limiaions exis in he kinds of archiecure: robusness, scalabiliy and reliabiliy due o heir complee dependence on he only one sink in large-scale WSNs. As all he sensors around sink exhaus energy, he sink will be isolaed from he WSNs. I means ha he WSNs lose is funcionaliy. All hese limiaions make single WSNs infeasible in real applicaions. For his reason, he archiecure of muli-sink WSNs is proposed. The muli-sink wireless sensor nework is a WSN wih muliple sink nodes. Compared wih single-sink, a muli-sink WSN has some advanages as follow: (1) i can avoid he breakdown of he whole neworks in a single-sink WSN; (2) i can decrease he lengh of he communicaion pah and prolong he lifeime of sensor neworks; (3) i can balance he nework raffic load and improve nework performance. Therefore, los of researchers have been working on energy efficien rouing in muli-sink WSNs. Many energy efficien rouing algorihms [4][5][6] have been proposed o prolong he lifeime of sensor nework. Sensors usually choose he neares sink as desinaion for sending daa. However, in WSNs, evens ofen occur in specific area. If all sensors in his area all follow he Neares- Sink sraegy, sensors around neares sink called hospo will exhaus energy early. Algorihms ha only consider sensor nodes will lead o early isolaion of specific sink (EISS) problem in asymmerical daa generaion environmen. I means ha his sink is isolaed from nework early and numbers of rouing pahs are broken. Sink Node (a) Asymmerical raffic load Sensor Node Sensor Node wih Energy Exhaused (b) Increase delay ime and hop disance Fig. 1. The Early Isolaion of Specific Sink Problem More specifically, sensors around specific sink will exhaus energy earlier due o a large number of daaforwarding. In Fig. 1.a, daa generaion rae in some area is higher han ohers. If all sensors inside he area send daa o sink depend on he neares sink sraegy, sensors around sink in he lower righ corner rapidly exhaus heir energy due o he considerable imes of daa-forwarding. The lower righ sink will be isolaed from WSNs once hese sensors near he sink exhaus baery. Furhermore, in Fig. 1.b, he EISS leads

2 o ha a lo of sensors swich he desinaion o farher sink. This siuaion considerably increases he energy consumpion for overall nework and acceleraes he isolaion rae for oher sinks. For solving he above-menioned problem, we propose he Q-learning based adapive zone pariion (QAZP) scheme. By he way of QAZP scheme, he whole nework is pariioned ino numbers of zones for each sink, and he size of zones is adjused for balancing power consume according o he residual energy of sensor nodes nearby each sink. The res of his paper is organized as follows. In secion 2, we inroduce some relaed work regarding energy efficien rouing algorihms in muli-sink WSNs. In secion 3, we propose he sysem archiecure for QAZP scheme and formulae he decision making problem. In secion 4, we elaborae he deails of he learning agen implemened by QAZP scheme. In secion 5, we compare our new QAZP scheme wih he previous mehods. We conclude our work in secion 6. 2 Relaed Work Several auhors have developed daa-cenric rouing algorihms in WSNs [1][2][3][4][5][6]. Unlike hierarchical rouing algorihms, daa-cenric rouing algorihms can suppor a large scale nework wih muliple sink nodes as well as muli-hop rouing while sensing daa communicaions. These algorihms are based on eiher he finding he nodes wih minimal hop coun or he compuaion he nodes and pahs wih leas energy consumpion o prolong he lifeime or increase capaciy of he nework. In [1][2][3], sensors will choose he neares sink in geomeer as heir desinaion for sending sensing daa. Since ha he sensing daa can arrive a neares sink hrough minimum hop disance, his proocol can achieve he purpose of leas energy consumpion for he whole nework. However, he geographically asymmeric generaion of evens may lead o asymmeric energy consumpion. In [4], auhors propose a PB (pah boleneck-oriened) and EC (energy cos-based) rouing algorihm (PBEC) o opimize he balance of nework-wide energy consumpion. The auhors invesigae he effec of choosing differen values for weigh coefficiens beween PB and EC on he nework lifeime performance. However, he radeoff beween PB and EC could no sui for kinds of environmen, especially enduring o ransmi sensing daa when some nodes exhaus heir energy. In [5], auhors propose he rouing algorihms ELBR (Energy Level Based Rouing) and PBR (Primary Based Rouing) in muli-sink sensor neworks. The energy level of nodes and energy cos of pahs are defined o help rouing packes: ELBR choose he pah wih he maximum energy level in order o ransmi more imes, PBR akes boh he energy level and he energy cos of he rouing pah ino consideraion. Through he energy consumpion is more balanced and he nework lifeime is more prolonged, i sill has high overhead while each sensor node acquires residual energy of is neighbors. In [6], he auhors presen a MSLBR (muli-sink and load-balance rouing) algorihm o balance he loads among he neighbors of sink nodes and prolong he nework lifeime. Each node s packes consider shores communicaion hops from iself o a neighbor of one sink node in a round-robin fashion. Wih he desinaion selecion sraegy, he raffic load can be uniformly disribued among neighbors of sink nodes. However, i does no consider he locaion informaion abou sensing node and neighbors of sink nodes so he sensor ofen selecs a desinaion which is farhes and he overall energy consumpion and daa delay ime will increase. 3 Sysem Archiecure and Problem Formulaion 3.1 Sysem Archiecure The sysem archiecure of Q-learning based adapive zone pariion (QAZP) for load balancing in muli-sink wireless sensor neworks we considered is shown in Fig. 2. The archiecure ypically consiss of a ask manager, a base saion, a mobile anchor, several sink nodes and a large number of sensor nodes. Is has he following characerisics: (1) here are a large number of sensors wih a unique ideniy in large scale WSNs and hey have he same iniial energy and communicaion range; (2) muliple sinks are deployed in WSNs, and all sinks have infinie amoun of energy; (3) he exisence of a conrollable mobile anchor (MA), which is equipped wih direcional anenna and GPS device. Base Saion Mobile Anchor Sink Node Sensor Node Fig. 2. Sysem Archiecure of QAZP Inerne Saellie Task Manager The sensor node esablishmens he several roue pahs o ransmi sensing daa o one of sink nodes. By he way of he MA is inroduced o adapively pariion he nework ino several zones and he direcional anenna are powerful User

3 enough o send beacon signals ha can be heard by all sensors in WSNs, each zone is responsible for collecing sensing daa o he sink. While he MA moves, a sensor node receives new broadcas message and hen ransmis follow-up sensor daa o new assigned sink node. The sysem archiecure of QAZP can adapively disribue he raffic load among hospos around sinks and avoid he EISS problem via he movemen of he MA. Moreover, we apply a learning agen o he MA for learning under unknown and sochasic environmens: The movemen of he MA is a reinforcemen learning (RL) problem, and we resolve by a heurisic algorihm described in nex. 3.2 Problem Formulaion The agen inside he MA may choose an acion o le he MA be moved o viciniy or remain a curren locaion. The MA can move a fixed disance oward eigh differen direcions. By he movemen of he MA, we adapively pariion he nework ino numbers of daa collecion zones for each sink. Afer a period of operaion ime in WSNs, he residual energy of hospos may change ino unbalance due o asymmerical daa generaion. If he residual energy of hospos around upper lef corner sink is he highes, he agen shifs he MA o he lower righ corner. Then he MA rebroadcass sink-assignmen packes o each sensor again from he new locaion. Afer he movemen, he daa collecion zone of upper lef corner sink is spread. Relaively, he oher hree sink s collecion zone began o shrink. We ake ino accoun he following wo facors for definiion of sae: residual energy of hospos for each sink and locaion of he MA in Fig.3. The sae of n sinks WSNs in ime is defined as s Eavg S1, Eavg S2,..., Eavg Sn, X, Y where E avg ( S n ) denoes he average residual power of one hop neighbors (hospos) of sinks n. X and Y respecively denoe he x-coordinae and y-coordinae of he MA. r N 1, (1) N i1 2 s ak Eavg Si Eavg Si where N is he number of sinks in WSNs, E avg ( S i ) denoes he average residual energy among hospos around sink i, E and avg S i denoes he average number among every E avg ( S i ). Acually, r(s,a k ) represens he sandard derivaion among every E avg ( S i ). Moreover, his expression yields a negaive quaniy ha resuls in lower magniude values depicing a large reward and higher magniude values represening a smaller reward. Specifically, he higher sandard derivaion represens ha large differences of energy among hospos, hen he agen can only obain less reward value from environmen. On he conrary, i ges more reward value ha represens ha he residual energy of hospos around sinks is well balanced. The agen in MA learns o find an opimal policy hrough ineracions wih whole wireless sensor nework and discovery opimal policy from his experience. The policy is o find he maximum reurn in saes. The reurn is he sum of he rewards: R r( s1, a1) r( s2, a2 ) r( s3, a3 ) r( st, at ) where T is a final ime sep. In infinie-horizon processes, a mechanism known as discouning is applied o conrol he rae a which rewards are accrued. The addiional concep ha we need is ha of discouning. In paricular, he agen maximizes he expeced discouned reurn: 2 k R r( s, a1) r( s2, a2 ) r( s3, a3 ) r( si, ai k 1 ) where is a parameer, 1, called he discoun rae. Finally, we inroduce an evaluaion funcion known as Q-funcion [8] o formulae he objec of QAZP. I is defined as Q(s,a ) which denoes he oal discouned reward couning from he saring (s,a) sae-acion pair over an infinie ime. The evaluaion funcion is represened as (2) (3) Q s, a E rs, a s s, a a (4) Fig. 3. Average Residual Energy in Curren Sysem Sae The reward funcion r(s,a) represen he immediae reward from environmen due o a seleced acion a a sae s. I is defined as where E{.} denoes for he expecaion operaor and γ <1 is a discouned facor which represens he imporance of reward value in he fuure. The final goal of QAZP is o obain an opimal policy: According o curren sae, he QAZP agen can choose an opimal acion a* which maximizes he evaluaion funcion. In his Reallocaion of he MA problem, he maximizaion of evaluaion funcion also represens ha minimizaion of energy consumpion s variance among hospos.

4 Nex, he evaluaion funcion can be expanded below, Q s, a E rs, a s s, a a (5) Signal Sae Consrucion Sae s-1 Sae s Acion Se A Q-funcion ( Q-Table ) Targe Q Q s, A Acion Selecion Sraegy Reallocaion of Mobile Anchor Reassign Sink Q s, a Er s, a s s, a a E rs, a 1 s s, a a (6) Reward Compuaion Acion a-1 Reward r-1 Esimaion of Q-value Q 1 s, a Er s, a Ps, a, E rs, a 1 s, a rs, a Ps, a, Q, a' Q s 1, a 1 a' where p(s,a,s ) denoes ha aking acion a in sae s will ransform ino sae s wih p probabiliy. The resul of his expansion means ha he evaluaion funcion of he curren sae-acion pair can be represened as he sum of he curren sae-acion s immediae reward and he expeced value of evaluaion funcion for all he possible nex sae-acion pairs. There is a sandard equaion o compue he opimal policy π*. The equaion is called Bellman Opimaliy Equaion [9] which obains he opimal acion a* by following wo kinds of operaions: Q * s, a rs, a P( s, a, ) max Q *, a' a' A The firs operaion is o find Q value in each (s,a) pair. The Q value is sum of immediae reward in (s,a), and expeced Q value for every possible nex sae-acion pairs Q +1 (s,a) as follow: Q ( s, ) s a rs, a maxq, b (7) (8) (9) Q, a (1) b s, a Q s, a Q s a Q 1, (11) 4 QAZP Scheme The agen in MA learns Q-values hrough several ineracions wih environmen. Wih he learned Q-values being sored in Q-able, he ask of nework pariion is carried ou by using he learned Q-values. Then we elaborae he deails of he learning agen implemened by QAZP scheme. Fig. 4. shows he srucure of he QAZP learning agen. 4.1 Sae Consrucion Consruc he saes s and s -1 by acquiring he residual energy of hospos around sinks. We assume ha powerful sink can direcly send residual energy noificaion packes o he MA hrough one-hop communicaion. Fig. 4. Srucure of Learning Agen 4.2 Reward Compuaion The agen can deec he coordinaes of he MA hrough GPS receiver. Based on his informaion, he sae is consruced for Q-funcion block. Moreover, he residual energy of hospos is used for reward calculaion by Eq. (1). However, he reward calculaed a curren decision poin is assigned o sae-acion pair a previous decision poin. 4.3 Q-funcion Wih he inpu of quaniaive saes which are fed ino he Q-funcion called Q-able, he agen can obain Q-values for all possible sae-acion pairs. Based on hese Q-values, agen will decide a moving acion. 4.4 Acion Selecion Sraegy Q values for possible sae-acion pairs can be obained by execuing he Q-funcion. However, if he Q values have no convergence, i means ha he agen is sill in learning procedure. In his siuaion, if he agen always chooses he bes acion, i probably will resul in a local opimal acion. For his reason, here is a rade-off beween exploiaion and exploraion: To learn an opimal policy, he agen should ry a number of wrong decisions in early learning o improve overall performance. However, afer a long ime, he policy will approach o a near-opimal policy. I means ha he agen should no spend cos and ime in raining. There is a well-known sraegy called ε-greedy [7] o resolve his radeoff beween exploiaion and exploraion. In his paper we adop he ε-greedy sraegy o our acion selecion sraegy block, in which he agen chooses he curren opimal acion wih 1 probabiliy, on he oher hand, i learns he ohers acion wih ε probabiliy where 1. Besides, he ε value will decrease over ime, i means ha he agen will explore as far as possible in he beginning of early learning. A he end of learning, agen will exploi he learned knowledge o execue he opimal acion wih high probabiliy. 4.5 Reallocaion of he MA The agen guilds he MA in which direcion o move. If he MA is reallocaed, i rebroadcass he sink assignmen

5 packe and repariions he Muli-Sink WSNs ino number of zones which equals o he number of sinks. 4.6 Esimaion of Q-value Based on hese informaion: sae s -1, seleced acion a - 1, curren sae s, and reward r -1, we updae he Q-values by Eq. (8). In any learning epoch, since he only one acion is chosen for curren sae, i means ha he Q-value of he chosen acion pair is updaed, while ohers remain. Then deailed procedure is described as Table 1. Table 1. Pseudo Code of QAZP Program sar Se he MA s locaion and spreading angle All sensors follow neares-sink sraegy emporarily Iniialize,ε and Q(s,a) Iniialize s, s -1, a, a -1 While ( all hospo is alive and decision-making) do The agen collecs sysem sae from sinks S -1 =s, a -1 =a Selec an acion a according o ε -greedy sraegy The agen obain reward r -1 Updae Q(s -1,a -1 ) depend on s -1, a -1, s and r -1 The seleced acion a rigger anchor s movemen If( a == one of moving direcion) Anchor is reallocaed o a new posiion Rebroadcas sink-assignmen packe Sensor choose a rouing pah o forward daa End 5 Performance Evaluaion The effeciveness of our proposed scheme called QAZP is validaed hrough simulaion. This secion describes simulaion environmen, performance merics and simulaion resuls. The resuls are compared wih performance of he Neares-sink (NS) and Round-Robin (RR) [6] rouing algorihms. We implemen our proposed scheme by C++ programming language. We pariion simulaion ime ino numbers of slos, he decision-making agen choose acion a he beginning of each slo. The agen periodically collecs sysem sae from each sink, and Q-values are sored in able. As he simulaion goes on, he Q-values are improved by employing expression (1) (11) and approach heir rue value. Parameer is iniialized o 1 and is iniialized o.9, and hey are linearly decremened unil hey reach a he end of learning. The simulaion environmen is se in Table 2. There are 6 sensors uniformly deployed in a square-shaped 1x 1 area. The communicaion range of sensor is se o en unis. A each round, sensors ransmi one packe o sink wih specific probabiliy depended on differen concenraion model: Linear and Complicaed concenraion model, which are proposed by [1]. Sensors in ho area generae 1 packe per round wih 1. or.6 probabiliies, while ohers wih.3 probabiliies. The performance evaluaion of hree sraegies is based on wo concenraion model represening various daa generaion rae. We will show ha our proposed scheme is able o adap o kinds of concenraion environmen. Table 2. Simulaion Parameers Number of sensors 6 o 8 Sensing area 1 x 1 Number of sinks 2,3,4,5 Iniial energy of sensor 5 unis nodes Energy consumpion for 1 uni per packes ransmission Packe generaion rae Linear / Complicaed concenraion model Communicaion range 1 unis disance Decision-Making Cycle Moving disance per decision-making 1 rounds 1, 1 2 unis disance Use he following merics for comparing he performance of NS, RR and QAZP: 1) Average hop coun: The average hop coun from source o sink represens ha he number of forwarding imes for packes. The higher i is, he larger he aggregae energy consumpion. Besides, he long disance also means high daa delay ime. 2) Nework lifeime: The lifeime is defined from he deploymen o he insan when he firs hospo exhauss baery. This is a good indicaor for he expeced lifeime of nework as i shows how well he load balancing scheme avoid EISS problem. Fig. 5. shows he average hop coun vs. numbers of sink node for differen schemes. The number of sensor node is 6. The average hop coun o sink of NS scheme is abou four hops, i is he shores among hree schemes because sensors always choose he neares sink; he average hop coun o sink of RR scheme is abou en hops, i is he longes obviously because he round-robin way lead o ha many sensors near a sink ofen choose anoher sinks which is much far away from hem; he average hop coun o sink of QAZP

6 scheme is abou 6 hops. The average hop coun of QAZP scheme is beween NS and RR. I means ha QAZP scheme can balance load wih lile effec on hop disance. Sensors can choose sink which is as close as possible o hem. Besides, we can observe he average hop coun of RR scheme increase slighly as he numbers of sink node. I means ha he number of sink node of RR scheme may be resriced; herefore RR scheme may be no suiable in more sink nodes environmen. (b) Complicaed model Fig. 7. Average Lifeime vs. Numbers of Sink Nodes (b) Complicaed model Fig. 5. Average Hop Coun vs. Numbers of Sink Nodes Fig. 6. shows he average hop coun vs. numbers of sensor node for differen schemes. The number of sink node is 4. We observe he average hop coun varies inconspicuously because he sensing area is he same and he lengh of rouing pah is changeless. Fig. 8. shows he average lifeime vs. numbers of sensor node for differen schemes. The number of sink node is 4. We observe ha he average lifeime increases as he numbers of sensor node because he daa can selec more nodes o ransmi and hen he inerval of ransmis is exended. As menion above, we can also observe he average lifeime of RR scheme in complicaed concenraion is more insable han QAZP scheme. I shows ha he heurisic algorihms like QAZP scheme are more adapive han seady algorihms. (b) Complicaed model Fig. 8. Average Lifeime vs. Numbers of Sensor Nodes (b) Complicaed model Fig. 6. Average Hop Coun vs. Numbers of Sensor Nodes Fig. 7. shows he average lifeime vs. numbers of sink node for differen schemes. The number of sensor node is 6. NS scheme derives leas lifeime in boh linear and complicaed concenraion model because i selecs neares node o ransmi sensing daa. However, i exiss longer lifeime in complicaed concenraion model because mass sensor nodes gaher mass daa in linear concenraion model. RR scheme has maximum lifeime in linear model because all hospos exhaus energy equally, bu i is suscepible in complicaed concenraion model. The average lifeime of QAZP scheme is close o RR scheme bu more sable han RR scheme in complicaed concenraion model. I means QAZP scheme can balance he load under differen raffic paern environmen. 6 Conclusions In his paper, we propose a load balancing scheme called QAZP. To resolve he EISS problem in Muli-Sink WSNs, he cenralized he MA adapively pariions he nework ino numbers of zones according o he residual energy of hospos around sinks. Besides, we apply Machine Learning o he MA for adaping o any raffic paern. Sensors around sinks are defined as hospos, and source nodes can choose differen hospos as heir desinaion. The residual energy of hospos and hop disance are used o choose he pah for rouing a packe. From performance evaluaion, we show ha he proposed QAZP scheme prolongs he WSNs lifeime under asymmerically daa generaion environmen. Besides, daa packe can achieve sink hrough shorer pah han RR scheme, i means ha overall energy consumpion is lower han RR.

7 7 Acknowledgemen The auhors would like o hank he Naional Science Council of he Republic of China, Taiwan, for financially supporing his research under Conrac No. NSC P References [1] Soyurk, M. and Alilar, T., A Novel Saeless Energy- Efficien Rouing Algorihm for Large-Scale Wireless Sensor Neworks wih Muliple Sinks in Proc. of he IEEE Annual Wireless and Microwave Technology Conference, 26. [2] Hyunyoung Lee, Klappenecker, A., Kyoungsook Lee, and Lan Lin Energy efficien daa managemen for wireless sensor neworks wih daa sink failure in Proc. of he IEEE Inernaional Conference on Mobile Adhoc and Sensor Sysems Conference, Nov. 25. [3] Yuichi Kiri, Masashi Sugano, Masayuki Muraa, "Self- Organized Daa-Gahering Scheme for Muli-Sink Sensor Neworks Inspired by Swarm Inelligence," saso, pp , Firs Inernaional Conference on Self-Adapive and Self- Organizing Sysems (SASO 27), 27. [4] Min Meng, Xiaoling Wu, Hui Xu, Byeong-Soo Jeong, Sungyoung Lee, and Young-Koo Lee, Energy efficien rouing in muliple Sink sensor neworks, The fifh Inernaional Conference on Compuaional Science and is Applicaions, pp , 27. [5] Yunyue Lin, Qishi Wu, Energy-Conserving Dynamic Rouing in Muli-Sink Heerogeneous Sensor Neworks. 21 Inernaional Conference on Communicaions and Mobile Compuing (CMC), pp , 21. [6] Chunping Wang, Wei Wu, "A Load-balance Rouing Algorihm for Muli-sink Wireless Sensor Neworks," in Inernaional Conference on Communicaion Sofware and Neworks, 29. ICCSN 29. [7] L. P. Kaelbling, M. L. Liman, and A. P. Moore, Reinforcemen learning: A survey, Journal of Arificial Inelligence Research, vol. 4, pp , [8] C. J. C. H. Wakins, and P. Dayan, Technical noe: Q learning, Machine Learning, vol. 8, no. 3, pp , [9] R. Bellman, Dynamic Programming. Princeon, NJ: Princeon Univ. Press, [1] T. Suzuki, M. Bandai and T. Waanabe, DispersiveCas: Dispersive Packes Transmission o Muliple sinks for Energy Saving in Sensor Neworks, in Proc. of Personal, Indoor and Mobile Radio Communicaions (PIMRC) 26.