Smart Duty Cycle Control with Reinforcement Learning for Machine to Machine Communications

Transcription

1 IEEE ICC Workshop on Smart Communcaton Protocols and Algorthms (SCPA 2015 Smart Duty Cycle Control wth Renforcement Learnng for Machne to Machne Communcatons Yun L, Kok Keong Cha, Yue Chen, Jonathan Loo School of Electronc Engneerng and Computer Scence, Queen Mary Unversty of London School of Scence and Technology, Mddlesex Unversty {yun.l, mchael.cha, yue.chen@qmul.ac.uk, J.Loo@mdx.ac.uk Abstract Machne to machne (M2M communcatons s one of the key underpnnng technologes for Internet of Thngs (IoT applcatons n 5G networks. The large scale of M2M devces mposes challenge on conventonal medum access control protocols. In ths paper, we propose a renforcement learnng (RL based duty cycle control for domnant short-range technology IEEE to provde hgh performance and relable M2M communcaton. We frst model a practcal mult-hop M2M communcaton network that takes varous network dynamcs nto consderaton. Then, we mathematcally derve the dstrbuted optmal duty cycle control polcy to optmse the energy effcency, end-to-end delay and transmsson relablty. Fnally, a RL based practcal duty cycle control s developed to learn the optmal polcy drectly wthout pror network nformaton, whch contrbutes to the smart duty cycle control under varous network dynamcs. Smulaton results show that the proposed RL based duty cycle control acheves the best balance between optmalty and stablty, compared wth the optmal and the exstng IEEE duty cycle controls. I. INTRODUCTION Wth the fast development of Internet of Thngs (IoT applcatons, machne to machne (M2M communcatons are consdered to add value to the emergng 5G networks [1]. However, t also mpose challenge for medum access control protocols to provde hgh performance and relable M2M communcatons [2]. To accommodate mllons of M2M devces, a hybrd M2M communcaton archtecture s proposed to support M2M communcatons n 5G networks by European Telecommuncatons Standards Insttute [3]. In the proposed hybrd archtecture, M2M gateways act as traffc aggregaton and protocol translaton ponts between M2M networks and LTE networks [4]. As the massve access of M2M devces s ensured va M2M gateways, smart access controls for M2M gateways and routers are necessary to ensure the network performance n terms of the energy effcency, end-to-end delay and transmsson relablty. The domnant short-range technology IEEE n M2M networks mproves the energy effcency by applyng duty cycle [5]. However, the duty cycle ntroduces delay as the sender may have to wat untl the recever becomes actve [6]. Ths problem s even severe n mult-hop networks as t wll accumulate hop by hop. What s more, the unform duty cycle wth centralsed control n current standard may not meet the varous requrements of applcatons and devces. In addton, t s dffcult or mpossble for M2M gateways/routers to obtan the perfect network nformaton, such as current traffc, specfc applcaton requrements and devce capablty of each devce n practcal envronments. Some exstng work [7] [10] focus on delay and energy effcency for duty cycle based MAC protocols. The delay reducton n [7] and [8] s acheved by adjustng sleep/actve schedules. However, the unformed duty cycle control for all devces s not flexble when devces generate tme-varyng traffc. In addton, contenton reports, pggyback flags or modfcatons of the packet header are needed for these controls. Dstrbuted duty cycle controls for mult-hop networks are proposed n [9] and [10]. The end-to-end delay s guaranteed by assgnng a local delay requrement to each sngle hop. Then a dstrbuted feedback controller s desgned to adapt sleep ntervals to meet the local sngle-hop delay requrement. However, ths approach requres sgnfcant amount of sgnallng from the neghbour devces to compute the delay. To address the non-perfect nformaton problem, renforcement learnng (RL s employed n [11] [13] to learn the behavours of devces and the change of networks. However, there s a lack of theoretcal support about the performance of the learned duty cycle control. More recently, an adaptve optmal duty-cycle algorthm s proposed wth the am of mnmsng energy consumpton whle meetng the relablty and delay requrements [14]. However, the theoretcal analyss of ths algorthm s based on non-beacon-enabled mode. In our prevous work [15], [16], we focus on low complexty duty cycle controls wth perfect network nformaton to optmse the relablty, delay and energy effcency for IEEE mult-hop networks. To further extent our work n more practcal secnaros, we develop a RL based duty cycle control for M2M networks wth unavalable network nformaton, varous network dynamcs, and tme-varyng traffcs n ths paper. The contrbutons of the paper are summarsed as follows. Frst, we model a mult-hop M2M network where varous network dynamcs, ncludng dynamc traffc generaton, dfferent applcaton requrements and devce capabltes are taken nto consderaton. Secondly, we formulate a dstrbuted duty cycle control problem for M2M gateways and routers n IEEE M2M networks. The gateways and routers wll decde ther own duty cycle locally wth the am of optmsng the transmsson relablty, energy effcency and end-to-end delay. Thrdly, we derve the optmal duty cycle control polcy by applyng Dynamc Programmng(DP technque. Last but not least, a RL duty cycle control s developed to learn the optmal polcy drectly wthout pror network nformaton. Specfcally, the developed RL duty cycle control s based on Q-learnng, a well known model-free RL technque, whch also contrbute to the balance between optmalty and stablty. The remander of the paper s organsed as follows. In secton II, we gve the system model and the background /15/$ IEEE 1438

2 IEEE ICC Workshop on Smart Communcaton Protocols and Algorthms (SCPA 2015 of IEEE MAC protocol. Problem formulaton s gven n Secton III. In secton IV the derved optmal polcy and the proposed Q-learnng based duty cycle are presented. Smulaton results and concluson are gven n secton V and secton VI, respectvely. II. SYSTEM MODEL It has been shown n [17] that the cluster-tree based networks have better performance and scalablty compared to the flat networks. Thus, we focus on the hybrd networks wth cluster-tree based M2M networks, as shown n Fg 1. The M2M gateways, routers and end-devces are three types of devces n the network. There s one M2M gateway wthn each sub-network, the devces that can partcpate n multhop routng are referred to as routers. The devces that do not partcpate n routng are referred to as end-devces. The routers forward the packets generated n the capllary network to the M2M gateway, whch then forward the collected packets to cellular base staton. Cellular Network Base Staton Gateway Router M2M Devce Cellular Lnk Lnk Fg. 1: Network Model. We descrbe the network n levels whch s defned by the number of hops the devce needs to reach the cellular base staton. The level of devces n s denoted as l n. For each M2M capllary network, the M2M gateway n 0 s at level-1, thus l n0 = 1; there are N routers n connected wth n 0 at level-2 and devce m connected to the router n are at level-3 and so forth. The end-devces are the chld devces at the last level of the network. We denote the chld devce set of router n as ch n, and the number of the devces n ch n s M n. A. IEEE Duty Cycle Control The duty cycle s appled n IEEE beacon-enabled mode to ncrease energy effcency. The duraton between two consecutve beacons s called Beacon Interval (BI, whle the duraton of an actve perod s called Superframe Duraton (SD. Specfcally, BI = abasesuperf rameduraton 2 BO, (1 SD = abasesuperf rameduraton 2 SO, (2 where Beacon Order (BO and Superframe Order (SO are two ntegers rangng from 0 to 14 (0 SO BO 14, and abasesuperf rameduraton = 15.36ms at 2.4 GHz wth 250 kbps data rate. Duty cycle, the rato of the actve porton over each tme perod BI, s equals to SD/BI = 2 SO BO. In IEEE (2011 [18] mult-hop transmsson, each gateway/router dvdes ts BI nto two superframes, called ncomng superframe and outgong superframe. The router n receves the beacon from ts M2M gateway n the ncomng superframe, and transmts ts beacon to ts chld devces n the outgong superframe. Whle the ncomng superframe duty cycle s decded by the parent router of the devce and enclosed n the receved beacon, each M2M gateway/router can control ts outgong superframe duty cycle (refered as duty cycle n ths paper. To smplfy the synchronsaton, we assume all routers have same BO. Thus, the duty cycle control of devce n s acheved by settng ts outgong SO. B. Queue and Traffc Models We assume all generated packets are avalable at the begnnng of each tme perod. All the packets are forwarded to the M2M gateway n 0 for uplnk transmsson and qn max s the maxmum queue length of devce n. The new arrved packets wll be dropped f the queue length of the devce reaches ts maxmum. The change of queue length of devce n over tme s gven as ( = mn [qn k + rn k fn k + gn k, qn max, (3 q k+1 n where 0 k K 1, [ = max(0,, gn k s the number of packets beng generated by devce n n tme perod k; fn k s the number of packet transmtted by devce n n tme perod k; and rn k s the number of packets receved by devce n n tme perod k. We assume each chld devce generates and sends ndependent Posson dstrbuted number of packets wthn the actve perod of each BI, whch means fn k and gn k are ndependent random varables. C. Channel Model We assume a dual-slope model of path loss wth dstance, Nakagam frequency-flat small-scale fadng, and lognormal shadowng based on [19]. The overall channel propagaton loss s expressed as L c,db =L 0,dB + X s,db + X f,db (4 { 10n 0 log(d d d n 0 log(d n 1 log( d. d 1 d > d 1 where X f,db = 10 log(x f and X f s a unt-mean gammadstrbuted random varable wth varance 1/m (where m s the Nakagam fadng parameter, X s,db s a zero mean Gaussan random varable wth standard devaton σ s, d s the dstance between the sender and recever, and all logarthms are base 10. We assume that the fadng and shadowng are constant durng each tme perod. The condton for the successful transmsson s that the receved sgnal power Prec,n k s above the senstvty threshold Psens,n k of the devce. We denote the successful transmsson probablty of devce n at tme perod k as φ k n,m = { 1 P k rec,n P k sens,n 0 P k rec,n < P k sens,n (5 III. PROBLEM FORMULATION In ths secton, we formulate the duty cycle control as a DP nventory control problem [20] to mnmse the total expected jont-cost of energy consumpton and end-to-end delay. We enable the ACK transmsson to ncrease the relablty of transmsson. 1439

3 IEEE ICC Workshop on Smart Communcaton Protocols and Algorthms (SCPA 2015 To ensure the costs of energy consumpton and end-to-end delay are addtve wth same unt, we defne the transmttng energy cost E t (fn k, recevng energy cost E r (rn k, dle lstenng energy cost E l (rn k, and end-to-end delay cost D(rn k of the devce n n terms of number of packets. As the energy consumpton of ACK packets transmsson exsts only when the router receves packets. Thus a fxed ACK transmsson energy cost A s ntroduced along wth the recevng energy consumpton. The specfc defnton of each cost s gven as: r E r (rn k A + c = r k n qn max l n f rn k > 0, (6 0 f rn k = 0. E f (f k n = c f f k n q max n l n, (7 E l (q k n = c l [f k n g k n q k n r k n q max n l n, (8 D(qn k = c d [qk n + rn k + gn k fn k qn max, (9 l n where A = c f M n ; c f, c r, c l and c d are the cost coeffcents of transmttng, recevng, dle lstenng and delay, respectvely. Note that c r < c l, as f c r were greater than c l, t would never be optmal to receve new packet at the last perod and possbly n earler perods. Dfferent devce buffer sze qn max and level of the devce l n have also been ntegrated nto the problem formulaton to dfferentate routers and chld devces. We further ntroduce α and β to assgn the weghtngs of energy effcency and end-to-end delay for dfferent applcaton requrements. Generally, the weghts of a cost functon wll be normalsed as one,.e.,α+β = 1. Then, the expected weghtedsum jont-cost functon for devce n at tme perod k s { ( = E α E f (fn k + E r (rn k + E l (qn k + βd(qn k. J k n (10 IEEE beacon-enabled mode apples slotted carrer-sense multple access wth slotted collson avodance (CSMA/CA for frame transmsson. We assume devces need to perform two clear channel assessments (CCAs before frame transmsson. The beacon transmsson duraton s D bcn. For each router, the total frame transmsson duraton s gven as P D = SD D bcn = M n =1 D m + δ + D ACK, where D m s the frame transmsson duraton of chld devce m ch n, δ and D ACK are watng tme and transmsson duraton of the ACK packet, respectvely. Then the number of packets that can be receved by devce n at k th tme perod s rn k = M n =1 η D m /D p, where D p s transmsson duraton per packet and η s the throughput lmtaton coeffcent [21], whch shows mpact of the backoff and contenton durng CSMA/CA transmsson. Accordng to (2, the relatonshp between the mnmum SO and rn k s gven as r SO(rn k k = log 2 ( n D p + δ + D ACK + D bcn. η (11 Our objectve s to fnd the optmal duty cycle control πn for each devce n, whch mnmse the overall expected jontcost. If we denote Φ k n as the set of chld devces m ch n wth φ k n,m = 1. then, the jont optmsaton problem s: { K 1 P n : mn E φ k n π n D,m Jn k (12 s.t. k=0 m Φ k n q K n = 0, r k n r max n. where D s vald duty cycle sets of devce n and rn max s the maxmum number of packets devce n could receve. Accordng to (1 and (2, the range of D s restrcted by the maxmum vald SO. IV. DUTY CYCLE CONTROL FOR IEEE In ths secton, we solve the problem P n by decomposng t nto a sequence of subproblems S(rn k by applyng the DP algorthm. Based on (10, the cost-to-go functons U(qn k of S(rn k s gven as { ( U(qn k = mn E α E r (rn k π n D + E f (fn k + E l (qn k + βd(qn k + E{U(qn k+1. (13 Then, the objectve of each subproblem S(r k n s to mnmse the cost-to-go functon U(q k n from tme perod K back to tme perod k. Thus, the optmal soluton to S(r 0 n s the soluton to P n. To reduce the number of notatons n the equatons, we denote m k n = qn k + rn k and n k n = fn k gn k. If δ(0 = 0, δ(rn k = 1 for rn k > 0, based on (4 - (8, we have { U(qn k = mn E Aδ(rn k π n D + W (m k n αc r qn k qn max, l n (14 where W (m k n = αe f (f k n + αe r (r k n + αc l [nk n m k n A. Optmal Duty Cycle q max n l n + βc d [mk n n k n q max n l n + U([m k n n k n. (15 It s not trval to solve S(r k n as functon U(q k n s not a convex functon due to the lmted buffer sze of devces. However, t has been proved by Scarf that an optmal mult perod(s, S soluton exsts, f U(q k n s A convex functon [22]. Defnton 1. The real-valued functon f s an A convex functon, f A 0, for all z 0, b > 0, f satsfes the A convexty property ( f(y f(y b A + f(z + y f(y + z. (16 b The dffcult part of ths problem s that the A convexty of W (m k n does not mply the A convexty of U([m k n n k n, whch s necessary to the recursve 1440

4 IEEE ICC Workshop on Smart Communcaton Protocols and Algorthms (SCPA 2015 process of provng the A convexty of W (m k n from the A convexty of W (m k+1 n. To show the property between U([m k n n k n and W (m k n, we rewrte (12 as W (m k n = (17 ( α E f (fn k + E r (rn k + βc d [m k n n k n qn max + R(m k n l, n where R(m k n = αc l [nk n m k n qn max l n + U([m k n n k n. The A convexty of W (m k n holds f the A convexty of W (m k+1 n mples A convexty of R(m k n. Now we wll gve suffcent condtons to the A convexty of the cost-togo functon U(qn k and then derve the optmal soluton. Proston 1. The cost-to-go functon U(q k n s A convex functon, f 1 W (m k n s an A convex functon. 2 R(m k n s an A convex functon. Theorem 1. If the cost-to-go functon U(q k n s A-convex, the optmal duty cycle control s a mult-perod polcy: when the queue length q k n s smaller than the T k n, SO k n s set based on (10, otherwse, SO equals to zero: SO k n = { SO(r k n, f q k n < T k n, 0, f q k n T k n, (18 where r k n s the optmal number of packets the devce should receved at each tme perod. Proof: The detaled proof s omtted due to the space lmtaton. Please refer to [16] for related nformaton. B. Renforcement Learnng Based Control The optmal duty cycle control can be found by runnng DP. However, n many practcal scenaros ether the characterstcs of the packet arrval processes change over tme, or t s not possble for M2M gateways/routers to have relable statstcal network nformaton such as the current traffc, specfc devce capablty, applcaton requrements and current channel condton. In ths secton, we present the Q-learnng procedure to fnd the optmal duty cycle control. In Q-learnng, the devce tres to learn the optmal polcy from ts hstory of nteracton wth the envronment. In other words, the Q-learnng does not requre perfect nformaton of the problem as t wll try to learn the optmal duty cycle drectly by updatng the state-acton value (Q-value whle nteractng wth the network, thus t s also called model-free RL. A hstory of the devce s a sequence of state-acton-cost set {qn k, rn k, Jn k, whch means that the devce was n state qn k and dd acton rn k, whch resulted n t recevng an mmedate cost Jn k and beng n state qn k+1. For a gven polcy π, a Q- value s the expected dscounted cost when executng acton rn k at state qn k and then followng polcy π thereafter, and t s defned as Q π (qn k, rn k = Jn k + δ mn Q π (qn k+1, rn k+1, where δ s the dscount factor. Then gven the learnng rate γ, Algorthm 1 Q-Learnng based Duty Cycle Control Requre: number of routers N, control tme perod K and maxmum teraton tmes T 1: for each n N do 2: Intalse Q 0 (qn 0, rn 0 = 0 3: for t = 1 T do 4: for k = 0 K do 5: f rand < ɛ then 6: [Exploraton step]: 7: select rn k arbtrarly 8: else 9: [Exploraton step]: 10: Q t (qn k, rn k = Jn k + δ mn Q t (qn k+1, rn k+1 11: rn k arg mn Q t (qn k, rn k 12: end f 13: Set SOn k accordng to (11 14: end for 15: Update Q(qn k, rn k accordng to (19 16: end for 17: end for the devce wll update ts estmate for Q(q k n, r k n at teraton t (0 < t < T as Q t (qn k, rn k = Q t 1 (qn k, rn k (19 { + γ Jn k + mn Q t (qn k, rn k Q t 1 (qn k, rn k. The learnng rate γ (0, 1] specfes how far the current estmate Q(qn k 1, rn k 1 s adjusted toward the update target Jn k + mn Q(qn k, rn k. The learnng rate s typcally tme varyng, decreasng wth tme. Separate learnng rates may be used for each state-acton par. The expresson nsde the curly bracket n (19 s the temporal dfference, whch s the dfference between the estmates of Q (qn k, rn k at two successve tme perod, k + 1 and k. Fndng a balance between explotaton of the devce s current knowledge, and exploraton, nformaton-gatherng actons taken to mprove current knowledge s crucal to the effcency of Q-learnng algorthm. One of the condtons the sequence Q(q k n, r k n provably converges to Q (q k n, r k n s that the devce keeps tryng all actons n all states wth nonzero probablty. Ths means that the devce must sometme explore,.e., perform other actons than dctated by the current polcy. As shown n Algorthm 1, we apply ɛ-greedy, a varaton on normal greedy selecton n our proposed Q-learnng duty cycle control. In ɛ-greedy, the devce dentfes the best move accordng to the state-acton values. However, there s a small probablty ɛ that, rather than take the best acton, the devce wll arbtrarly select an acton from the remanng actons to perform the exploraton. V. SIMULATION RESULTS AND ANALYSIS In ths secton, the performance of proposed Q-learnng based duty cycle control s evaluated n a two hop clustertree network wth Matlab. Smulaton parameters are gven n TABLE I. We appled ON/OFF traffc model n the smulaton. When the devce s actve (ON, the dstrbuton of the traffc generaton follows Posson dstrbuton. When the devce s nactve (OFF, t s dle and does not generates any packets. 1441

5 IEEE ICC Workshop on Smart Communcaton Protocols and Algorthms (SCPA energy effcency (kb/j benchmark control DP optmal Q Learnng cost functon value DP optmal Q Learnng packet arrval rate (kbps Fg. 2: Energy effcency versus packet arrval rate. average end to end delay per packet (s DP optmal Q Learnng 0 packet arrval rate (kbps Fg. 3: End-to-end delay versus packet arrval rate. Energy consumpton parameters are based on CC2420 data sheet [23] and MAC layer parameters are based on IEEE BI s 0.49s (BO = 5, and the servce rate fn k follows posson dstrbuton and the number of observaton tme perods K s 100. The results are the averaged values of 1000 runs of the devce n. Packets are dropped only when the queue length of the devce reaches ts buffer sze qn k fn k +rn k > qn max. The maxmum queue length of routers s 50 packets and that of the chld devces s 20 packets. The performance of proposed Q-learnng based control s compared wth a benchmark control, DP optmal and DE- DutyCon [16]. Specfcally, the benchmark control ams at mnmsng the end-to-end delay by maxmsng the number of transmtted packets at each tme perod, thus we set a fxed SO equals to 3, whch s suffcent large to transmt all generated packets. The DP optmal control processes DP exhausted search for entre observed tme perods to fnd the optmal duty cycle. DE-DutyCon, a rollout algorthm based control whch searches the mnmum cost of current tme perod and estmates future costs based on a heurstc control. TABLE I: Smulaton Parameters Parameter Value Parameter Value frequency 2.4 GHz α 0.2 data rate 250kbps β 0.4 transmt power 36.5 mw packet sze 100 bytes receve power 41.4 mw CCA sze 8 symbols dle lsten power 41.4 mw ACK packet sze 10 symbols sleep power mw unt backoff perod 20 symbols learnng rate 0.9 dscount factor packet arrval rate (kbps Fg. 4: Cost functon value versus packet arrval rate. packet drop rato DP optmal Q Learnng 10 4 packet arrval rate (kbps Fg. 5: Packet drop rato versus packet arrval rate. The energy effcency s shown n Fg. 2. Snce SO can only be nteger, the change of actve perod length s radcal. The dle lstenng energy consumpton caused by the change of SO lead to the non monopoly change of the energy effcency curves of Q-learnng and benchmark control between kbps. The energy effcency curves of benchmark control, DP optmal control and DE-DutyCon are flat after the packet arrval rate s hgher than 60 kbps, as SO values became stable. Fg. 3 shows the end-to-end delay performance of the observed controls. The proposed Q-learnng based duty cycle control has the lowest end-to-end delay compared to other controls. The change of the end-to-end curve of DP optmal control between kbps s because wth the ncreased SO due to more packets were able to be transmtted. It s observed from Fg. 2 and Fg. 3 that the ncrease of energy effcency s always came wth the ncrease of end-to-end delay for all compared controls. Ths observaton llustrates the trade-off between energy effcency and end-toend delay. Fg. 4 shows the averaged jont-costs per BI of the evaluated control mechansms. The cost functon value of the Q- learnng based control s very close to that of the benchmark control, but hgher than that of the optmal control and DE- DutyCon. Ths s because the learnng process of Q-learnng based control wll gradually approachng the optmal soluton, thus the average jont-cost wll be hgher. The performance of overall two-hop packet drop rato s shown n Fg. 5. Although the packet drop s not consdered n the jont-cost functon, Q-learnng based control s able to reduce the packet drop rato by nteractng wth the network. 1442

6 IEEE ICC Workshop on Smart Communcaton Protocols and Algorthms (SCPA 2015 cost functon value DP optmal 0.1 Q Learnng tme perod Fg. 6: Instant cost over tme. The Q-learnng based control has lower packet drop rato compared to the DP optmal and DE-DutyCon. In addton, the packet drop rato of Q-learnng based control only started to ncrease when the packet arrval rate s hgher than 30kbps, whle optmal control and DE-DutyCon started around 20kbps. It s shown n Fg. 4 and Fg. 5 that Q-learnng based control reduced the packet drop rato compared to other controls but ncreased the jont-cost value. Ths observaton llustrates that there s a trade-off between the jont-cost and packet drop rato. Furthermore, Fg.6 shows how the jont-costs of the compared control mechansm evolved over tme, takng the packet arrval rate of 100bps as example. Due to the learnng process, Q-learnng based control took about 10 BIs to become stable, whle other controls took less than 5 BIs. However, t can be seen that once the Q-learnng reached ts stable state, the cost value has much lower varance compared to other controls. In summary, the proposed Q-leanng based duty cycle control acheved hgher energy effcency, smlar end-to-end delay and packet drop rato compare to the benchmark control. When compared to the DP optmal and DE-DutyCon, the proposed Q-leanng based control strkes a balance between optmalty and algorthm stablty. VI. CONCLUSION In ths paper, we formulated a duty cycle control problem for mult-hop IEEE M2M networks. Based on the mathematcal analyss, we developed a Q-learnng based duty cycle control for M2M gateways/routers to learn the optmal duty cycle control. Trade-offs between dfferent performance ndcators have been nvestgated, n partcular energy effcency and end-to-end delay, cost and packet drop rato, and algorthm stablty and optmalty are analysed thoroughly n smulaton. Smulaton results shown that the proposed Q- learnng based duty cycle control acheved the best balance between optmalty and stablty, compared wth the optmal and the exstng IEEE duty cycle controls. REFERENCES [1] D. Lu, Y. Chen, K. K. Cha, T. Zhang, and M. Elkashlan. Opportunstc User Assocaton for Mult-servce HetNets Usng Nash Barganng Soluton, IEEE Communcatons Letters, vol.18, no.3, pp , Mar [2] AG. Gotss, AS. Loumpas, and A. Alexou Analytcal Modellng and Performance Evaluaton of Realstc Tme-controlled M2M Schedulng Over LTE Cellular Networks, Transactons on Emergng Telecommuncatons Technologes 2013; vol.24, no.4, pp [3] ETSI, ETSI TR : Machne-to-Machne communcatons (M2M; Applcablty of M2M Archtecture to Smart Grd Networks; Impact of Smart Grds on M2M platform, September [4] Zheng K, Hu F, Wang W, Xang W and Dohler M. Rado Resource Allocaton n LTE-advanced Cellular Networks wth M2M Communcatons, Communcatons Magazne, IEEE, vol.50, no.7, pp , [5] VB. Msc and J. Msc, Evaluatng Effectveness of IEEE Networks for M2M Communcatons, n Machne-to-Machne Communcatons: Archtectures, Technology, Standards, and Applcatons, Boca Raton, FL: CRC Press, [6] C. Buratt. Performance Analyss of IEEE Beacon-Enabled Mode, Vehcular Technology, IEEE Transactons on, vol. 59, no. 4, pp , Jan [7] S.H. Yang, H. W. Tseng, E. Wu, and G.-H. Chen, Utlzaton Based Duty Cycle Tunng MAC Protocol for Wreless Sensor Networks, n Proc. IEEE Global Telecommuncatons, GLOBECOM. Dallas, Texas, pp , Nov [8] Z. Yuqun, F. Chen-Hsang, I. Demrkol, and W.B. Henzelman, Energy- Effcent Duty Cycle Assgnment for Recever-Based Convergecast n Wreless Sensor Networks, n Proc. IEEE Global Telecommuncatons Conference, GLOBECOM, pp. 1-5, Dec [9] X. Wang, X. Wang, G. Xng, and Y. Yao, Dynamc Duty Cycle Control for End-to-End Delay Guarantees n Wreless Sensor Networks, n Proc. IEEE Internatonal Workshop on Qualty of Servce, IWQoS, Bejng, Chna, Jun [10] H. Byun and J. Yu, Adaptve Duty Cycle Control wth Queue Management n Wreless Sensor Networks, IEEE Transactons on Moble Computng, vol. 12, no. 6, pp , Jun [11] Z. Lu, I. Elhanany, RL-MAC: a Renforcement Learnng based MAC Protocol for Wreless Sensor Networks, Internatonal Journal of Sensor Networks vol. 1, pp , [12] J. Mao, F. Xang, H. La, RL-based Superframe Order Adaptaton Algorthm for IEEE Networks, n Chnese Control and Decson Conference, pp ,2009. [13] R. Alberola and D. Pesch, Duty Cycle Learnng Algorthm (DCLA for IEEE Beacon-enabled Wreless Sensor Networks, Journal of Ad Hoc Networks, vol. 10, no. 4, pp , [14] P. Park, S. C. Ergen, C. Fschone and A. S. Vncentell, Duty-Cycle Oprmzaton for IEEE Wreless Sensor Networks, ACM Transactons on Sensor Networks, vol.10, no.1, 12:1 12:32, Nov [15] Y. L, K. K. Cha, Y. Chen and J. Loo, Duty Cycle Control wth Jont Optmsaton of Delay and Energy Effcency for Capllary Machneto-Machne Networks n 5G Communcaton System, Transactons on Emergng Telecommuncatons Technologes, vol.26, no.1, pp. 5669, Jan [16] Y. L, K. K. Cha, Y. Chen and J. Loo, Optmsed Delay-Energy Aware Duty Cycle Control for IEEE wth Cumulatve Acknowledgement, n Proc. IEEE Internatonal Symposum on Personal, Indoor and Moble Rado Communcatons, PIMRC 14, Sep [17] A. Gupta, A. D. de Clauzade de Mazeux, and M. Becker, Effect of Topology on the Performance of Moble Heterogeneous Sensor Networks, n Proc. 6th Annu. Medterranean Ad Hoc Netw. WorkShop, pp , [18] IEEE std , Part. 15.4: Wreless Medum Access Control (MAC and Physcal Layer (PHY Specfcatons for Low-Rate Wreless Personal Area Networks (LR-WPANs, IEEE Std., [19] Natonal Insttute of Standards and Technology, NIST Prorty Acton Plan 2 Gudelnes for Assessng Wreless Standards for Smart Grd Applcatons, 2010 [20] D. P. Bertsekas, Dynamc Programmng and Optmal Control, 3rd Edton, Athena Scentfc, [21] T. R. Park, T. H. Km, J. Y. Cho, S. Cho and W.H. Kwon, Throughput and energy consumpton analyss of IEEE slotted CSMA/CA, Electroncs Letters, vol. 41, no. 18, [22] H. Scarf, The Optmalty of (s,s Polces for the Dynamc Inventory Problem, n Proc. of the 1st Stanford Symposum on Mathematcal Methods n the Socal Scences, Stanford, CA, [23] CC GHz IEEE Complant and ZgBee Ready RF Transcever, onlne: