To Relay or Not to Relay: Learning Device-to-Device Relaying Strategies in Cellular Networks

Transcription

1 To Relay or No o Relay: Learning Device-o-Device Relaying Sraegies in Cellular Neworks Nicholas Masronarde, Viral Pael, Jie Xu, Lingia Liu, and Mihaela van der Schaar Absrac- We consider a cellular nework where mobile ransceiver devices ha are owned by selfineresed users are incenivized o cooperae wih each oher using okens, which hey exchange elecronically o buy and sell downlink relay services, hereby increasing he nework s capaciy compared o a nework ha only suppors base saion-o-device (B2D) communicaions. We invesigae how an individual device in he nework can learn is opimal cooperaion policy online, which i uses o decide wheher or no o provide downlink relay services for oher devices in exchange for okens. We propose a supervised learning algorihm ha devices can deploy o learn heir opimal cooperaion sraegies online given heir experienced nework environmen. We hen sysemaically evaluae he learning algorihm in various deploymen scenarios. Our simulaion resuls sugges ha devices have he greaes incenive o cooperae when he nework conains (i) many devices wih high energy budges for relaying, (ii) many highly mobile users (e.g., users in moor vehicles), and (iii) neiher oo few nor oo many okens. Addiionally, wihin he oken sysem, self-ineresed devices can effecively learn o cooperae online, and achieve over 20% higher hroughpu on average han wih B2D communicaions alone, all while selfishly maximizing heir own uiliies. Keywords- Cellular neworks, device-o-device relaying, oken exchange sysem, Markov decision process, online learning I. INTRODUCTION D2D communicaions has recenly emerged as a candidae echnology for supporing cooperaive relaying in mobile broadband cellular neworks. By aking advanage of user diversiy in space and ime, cooperaive relaying can simulaneously increase nework capaciy and decrease delay [1], which can improve he performance of imporan applicaions such as wireless video sreaming [3][4]. In addiion o he aforemenioned benefis, D2D relaying is appealing because i does no increase infrasrucure coss for he nework operaor. This is in conras o infrasrucure relay nodes, which are included as a par of he curren LTE-Advanced sandard [5]. Alhough here are many benefis associaed wih D2D relaying, i requires coordinaion across muliple The work of Mihaela van der Schaar and Jie Xu was suppored by NSF gran no N. Masronarde is wih he Deparmen of Elecrical Engineering, Universiy a Buffalo, Buffalo, NY USA ( nmasron@buffalo.edu). M. van der Schaar and J. Xu are wih he Deparmen of Elecrical Engineering, Universiy of California, Los Angeles, Los Angeles, CA, USA ( mihaela@ee.ucla.edu and iexu@ucla.edu). L. Liu is wih he Deparmen of Elecrical Engineering and Compuer Science, Universiy of Kansas, Lawrence, KS USA (lingialiu@ic.ku.edu). 1

2 nodes in he nework, which is especially difficul o realize when nodes are cell-phones, lapops, or ables (hereafer, User Equipmens, or UEs) ha are owned by self-ineresed users, who aim o maximize heir own uiliies. In paricular, mos exising work on cooperaive communicaions/relaying implicily assumes ha all nework devices are obedien, i.e., hey will ac as relays whenever requesed. In pracice, however, since relaying coss energy and provides no angible benefi o he relay, obliging UEs o serve as relays could quickly lead o widespread user dissaisfacion and, ulimaely, loss of cusomers and revenue for he nework operaor (for insance, a UE s baery can quickly drain while relaying daa for oher users, making i so ha he owner canno use her UE when she needs i). Indeed, according o he J.D. Power and Associaes 2012 U.S. Wireless Smarphone Cusomer Saisfacion Sudy [6], poor baery life reduces boh cusomer saisfacion and loyaly. Therefore, o realize he poenial of D2D relaying in cellular neworks, a sysem mus be developed ha incenivizes UEs o cooperae while allowing hem o ac as relays only when i is in heir self-ineres. A. Relaed Work There is a lo of grea lieraure on cooperaive relaying in wireless and cellular neworks, e.g., [2]- [4][7][8], including work on D2D relaying [9][10][11]. However, his work does no consider selfineresed users who require incenive o cooperae. Neverheless, many of he echniques and soluions in his lieraure can be implemened in conuncion wih differen incenive mechanisms; in his sense, work on cooperaive/d2d relaying echniques is complemenary o work on incenivizing relaying. Various mechanisms have been proposed o incenivize relaying in ad-hoc, cellular, and peer-o-peer neworks. These mechanisms can be roughly grouped ino hree classes: bandwidh exchange mechanisms, repuaion-based mechanisms, and oken-based mechanisms. In bandwidh exchange mechanisms, when device A receives relay service from device B, device A immediaely delegaes a porion of is bandwidh o device B as compensaion for relaying [12][13]. Varians of bandwidh exchange mechanisms have been proposed based on exchanging ransmission ime [14] or direcly exchanging relay services [15]. Unforunaely, hese mechanisms are subopimal because hey require device A o immediaely and direcly reciprocae o device B, even hough device B may no need assisance a he curren ime or, imporanly, may no need assisance from device A. Meanwhile, oken-based mechanisms and many repuaion-based mechanisms exploi indirec reciprociy [16], which is based on he idea ha device C will relay for device B because device B relayed for device A in he pas. Due o his flexibiliy, indirec reciprociy can achieve beer nework performance han direc reciprociy. Repuaion-based mechanisms enable devices o idenify noncooperaive devices so ha hey can be eliminaed as poenial relays and/or punished by dropping heir packes [16][19][20][21]. To idenify noncooperaive devices, each device racks he oher devices repuaions based on is firs-hand ineracions [21] and, in some implemenaions, also uses second-hand experience obained hrough 2

3 repuaion propagaion [16][19][20]. Unforunaely, repuaion-based mechanisms are no suiable in cellular neworks for wo reasons. Firs, hey do no scale well o large-scale cellular neworks because firs-hand experience mus be racked for many devices and reliable second-hand repuaion informaion canno be propagaed in a imely fashion. Second, mos repuaion-based mechanisms rely on he broadcas naure of wireless neworks o enable devices o monior each oher's ransmissions [16][19][20]; however, omnidirecional broadcasing will become less common in maor 4G mobile broadband cellular sandards [17][18] because hey suppor MIMO and beamforming. Token-based mechanisms have been proposed o incenivize cooperaion in ad-hoc [22][23][24][25][26], cellular [27][28], and peer-o-peer neworks [29]. Elecronic okens work as a virual currency: users pay okens o oher users in exchange for providing services. Alhough okens have been used o incenivize cooperaion in cellular neworks [27][28], opimal and pracical soluions are sill far from being developed. For insance, [27] focuses on developing a suiable sysem model and archiecure, bu assumes ha each UE deploys an arbirary fixed hreshold cooperaion policy, which does no depend on is experienced nework environmen, hereby resuling in subopimal performance. In our prior work [28], we use a oken sysem o incenivize UEs o provide relay services o each oher. In paricular, we focus on how a sysem planner can design a oken sysem offline o maximize is efficiency (i.e., he probabiliy ha UEs ha need help from a relay will ge i) wihin an idealized nework operaing in equilibrium. We show ha, if he expeced fuure benefi of having an addiional oken ouweighs he immediae cos of relaying, hen a self-ineresed UE will be willing o ac as a relay; we prove ha hreshold sraegies are he only sraegies ha a self-ineresed UE, which wans o maximize is own uiliy, will adop; and we deermine he opimal oken supply ha should be deployed in he nework o maximize he sysem efficiency. This heory is furher developed in [30]. However, in [28][30], we do no address how individual UEs can dynamically opimize heir cooperaion policies in a non-idealized nework environmen, operaing ou of equilibrium, which is of paramoun imporance in real sysems. B. Our Conribuions In his repor, we adop a oken-based approach similar o [28]. However, raher han focusing on incenive design from he planner s perspecive, we invesigae how he UEs can learn o adap heir cooperaion sraegies online, which hey use o decide wheher or no o provide downlink relay services for oher devices in exchange for okens. Specifically, we sudy how an individual UE in he nework can learn is opimal 1 cooperaion policy online based on is experienced nework environmen, and how his sraegy impacs and is impaced by he oher UEs in he nework, which are simulaneously learning. Our conribuions are as follows: 1 We mean opimal in he bes response sense [38]: ha is, a UE s opimal cooperaion policy maximizes is long-run uiliy given is experienced nework environmen and he aggregae behavior of he oher UEs. 3

4 We formulae he decision problem faced by each UE, namely, he problem of deciding wheher or no o relay, as a Markov decision process (MDP). Each UE s obecive is o maximize is long-erm uiliy, which is defined as he difference beween (i) he benefis i gains over ime by receiving daa hrough a relay and (ii) he energy coss i incurs over ime by relaying daa for oher UEs. We formulae he MDP such ha a UE s opimal cooperaion policy no only depends on is curren geographic locaion, is disance from he neares base saion, and he channel condiions, bu also on he oher UEs locaions and cooperaion sraegies. In oher words, each individual UE s opimal cooperaion policy is coupled wih boh he nework environmen and he oher UEs. We assume ha each UE has a relay energy budge ha specifies how much energy i is willing o consume relaying daa for oher users. We consider a UE s remaining relay energy budge when deermining is opimal cooperaion policy. We show experimenally ha opimal cooperaion sraegies are hreshold in a UE s oken holding sae, and ha he hreshold decreases as he UE s relay energy budge decreases. We propose a simple, low-complexiy, supervised learning algorihm ha each UE can deploy o learn is opimal cooperaion policy online. In he proposed algorihm, a UE esimaes wo parameers from is ineracions wih he nework environmen, i.e., he frequency wih which i is asked o relay daa for oher UEs and he frequency wih which oher UEs relay daa for i. Using is esimaes of hese wo parameers, he UE selecs is curren cooperaion policy from a pre-compued look-up able. We sysemaically evaluae he proposed learning algorihm in various deploymen scenarios involving users wih high and low mobiliy and UEs wih high and low relay energy budges. Our simulaion resuls sugges ha UEs have he greaes incenive o cooperae when he nework conains many UEs wih high relay energy budges, many highly mobile users (e.g., users in moor vehicles), and neiher oo few nor oo many okens. The remainder of his repor is organized as follows. In Secion II, we presen he sysem model. In Secion III, we formulae an individual UE s opimizaion problem. In Secion IV, we propose he supervised learning algorihm ha each UE can deploy o learn is opimal cooperaion policy online. In Secion V, we presen our simulaion resuls. Finally, we conclude he repor in Secion VI. II. SYSTEM MODEL A. Downlink Nework Model In his secion, we presen our downlink nework model. For illusraion, we use an OFDM-based nework model similar o, e.g., LTE/LTE-Advanced or IEEE (WiMax); however, he opimizaion and learning soluions proposed in Secions III and IV, respecively, can be applied in oher neworks such as 3G or ad-hoc neworks. We consider a cellular nework conaining N mobile ransceiver devices, which 4

5 we refer o as User Equipmens (UEs). We assume ha ime is sloed ino discree ime inervals indexed by N. The ime slo duraion (seconds) is greaer han he amoun of ime required o deermine if a relay ransmission is needed, idenify an appropriae relay, and complee a relay ransmission, bu no so long ha he UEs geographic locaions or channel condiions change significanly during he ime slo. In an LTE nework, for example, he ime slo duraion could be se o he lengh of one frame (10 ms). In any given ime slo, some UEs are scheduled o receive daa from he BS on he downlink; however, some of hese UEs may experience bad channel condiions due o pahloss, mulipah effecs, and shadowing, which will limi heir downlink daa raes. In his siuaion, inermediae UEs can ac as relays o help deliver daa from he BS o he desinaion UEs. To enable device-o-device (D2D) relaying, we exploi D2D communicaions echnology [9][10][11]. Noe ha D2D relays are no he same as infrasrucure relay nodes, which are included as a par of he curren LTE-Advanced sandard [5] and do no require incenives o cooperae. For illusraion, we consider a wo-hop amplify-and-forward relaying sraegy using a single inermediae relay (i.e., a ransmission from he BS o he relay followed by a ransmission from he relay o he desinaion). Addiionally, we use a simple mode and relay selecion sraegy for deciding when a UE will reques a relay and which relay i will selec. Imporanly, he sraegies considered in his repor are for illusraion: he oken sysem and opimizaion/learning framework proposed in Secions III and IV can work wih any oher relaying, mode, and relay selecion sraegies wihou significan modificaion. 2 Before we presen our illusraive mode and relay selecion sraegy in Secion II.A.2, we provide expressions for he achievable ransmission raes wih and wihou relay ransmission in Secion II.A Direc and Relay Assised Transmission Raes Le i Γ denoe he received Signal-o-Inerference plus Noise Raio (SINR) on he link beween node i and UE in ime slo, where node i could be he BS or anoher UE: i.e., where i i i gp i W N0+ I i P (W) is he ransmission power on link i, Γ =, (1) bandwidh, N 0 (W/Hz) is he noise power specral densiy, and i g is he channel gain, W (Hz) is he link s i I is he iner-cell inerference experienced by UE. Equipped wih capaciy achieving error conrol codes, he ransmission rae on link i in ime slo can be wrien as i i g2( i r = W lo 1 + Γ ) (bis/s). (2) 2 Our framework can be applied wihou significan modificaion o any wo-hop relaying scheme ha uses a single inermediae relay (regardless of he relaying, mode, and relay selecion sraegies); however, he oken sysem and opimizaion/learning framework would have o be adaped o work wih eiher wo-hop relaying sraegies ha use muliple relays or muli-hop relaying sraegies. 5

6 Suppose ha, in ime slo, UE {1,, N} is scheduled o receive daa from he BS (indexed by 0). UE can eiher receive daa direcly over he B2D link (direc mode) or hrough an inermediae relay (relay mode). In direc mode, he BS ransmis o UE a he rae r = W lo g (1+ Γ ) (bis/s) for seconds. In relay mode, following sandard relay channel analysis for amplify-and-forward cooperaion [8], we obain he following received SINR over he cooperaive link: 0i i 0i ΓΓ 0i i Γ =, (3) Γ +Γ + 1 such ha he effecive ransmission rae from he BS o UE, hrough UE i, is 0 i 0 0 lo 2(1 i + Γ r = W g ). Noe ha he relay-o-desinaion link uses he same physical resources as he BS-o-relay link. We discuss his in more deail in he nex subsecion. 2. Illusraive Mode Selecion and Relay Selecion Sraegies In pracice, mode selecion, relay selecion, and scheduling should be opimized oinly in order o bes uilize he available downlink resources; however, for simpliciy, we assume ha he BS firs schedules he UEs for downlink ransmission, and hen each scheduled UE deermines is ransmission mode and, if necessary, is desired relay. Furhermore, we assume ha he scheduler operaes under he assumpion ha all ransmissions are performed in he direc mode. In his way, we can inegrae D2D relaying ino he sysem wihou requiring any modificaion o he scheduler. Le and le DL U and UL U denoe he ses of UEs ha are scheduled on he downlink and uplink, respecively, IDLE U denoe he se of idle UEs in ime slo. Noe ha, since UEs canno ransmi and receive a he same ime, DL U, UL U, and IDLE U pariion he se of UEs U = {1, N, }. The ses of scheduled users, as well as heir corresponding resource allocaions, can be deermined using any scheduling policy IDLE (proporional fair, round robin, resource fair, ec.). We le U ( ) denoe he se of idle UEs ha are IDLE associaed wih he same BS as UE. Only UEs in U ( ) can serve as relays for UE in ime slo. Consider UE 0 DL U, which is scheduled o receive daa from he BS in ime slo over a link wih bandwidh W Hz. We assume ha, if he received SINR on he BS o device (B2D) link, arge hreshold 0 0 Γ, is above a arge Γ, hen he BS will ransmi o UE in he direc mode. Noe ha, given he allocaed arge 0 arge 2 bandwidh W, he arge SINR corresponds o a arge rae r = W log (1 + Γ ). However, if IDLE he B2D link s SINR is below he arge hreshold, hen UE will aemp o find a relay i U ( ) hrough which i can mee he arge rae while using he leas power. UE achieves his by solving he following mode and relay selecion problem: 6

7 s.. = arg min ( ) 0i 0 arge W 2 + Γ 0 arge < Γ i max 0i 0 i 0 = W = W Rae consrain: r log (1 ) SNR consrain: Power consrain: i Γ P P i U IDLE BW consrains: W W and. P i (4) If here is a relay i ha is a soluion o problem (4), and is willing o provide relay services, hen UE will receive daa from he BS, hrough UE i, in relay mode; oherwise, UE will receive daa from he BS in direc mode. In problem (4), he rae consrain requires ha he BS can mee he arge ransmission rae hrough he relay; he SINR consrain requires ha he B2D link s SINR is below he arge hreshold; he power consrain requires ha he relay s ransmission power is below is maximum ransmission power; and he bandwidh (BW) consrains require ha he BS-o-relay and relay-o-desinaion links use he same resources ha he scheduling policy originally allocaed o he B2D link. Imporanly, since he relay ransmission is performed in wo phases, he BW consrain W = W implies ha he D2D link is given dedicaed (orhogonal) resources; herefore, we do no have o worry abou he D2D communicaions causing inerference o he BS or oher UEs in he cell. This is in conras o a lo of work on D2D communicaions, which assumes ha D2D links reuse uplink resources, e.g., [10]. In he case of downlink relaying, we believe ha i makes more sense for he D2D relays o reuse downlink resources because hey are helping he base saion. B. Virual Token Exchange Sysem Performing a relay ransmission coss energy and provides no benefi o he relay. Consequenly, wihou proper incenives, no self-ineresed user would wan her UE o relay daa for oher users. To overcome his problem, we incenivize relaying hrough he use of a virual oken exchange sysem. In our proposed sysem, UEs are iniially endowed wih a se of okens (e.g., by he device s manufacurer or by he nework operaor). A UE mus expend one oken in order o receive daa hrough a relay, and a UE can only earn addiional okens by relaying daa for oher UEs. Imporanly, if a UE does no have any okens, hen i canno receive daa hrough a relay, so i will defaul o receiving daa direcly from he BS. In his way, no user is worse off in he proposed sysem han hey would be in a nework wih only B2D communicaions. Consider UE 0 arge DL U, which is scheduled o receive daa from he BS in ime slo. Suppose ha Γ < Γ and ha, afer solving problem (4), i selecs UE i as a candidae relay. A his poin, UE i 0 7

8 sends a relay reques o UE i, which replies wih eiher a posiive or negaive relay acknowledgemen (R- ACK/R-NACK) indicaing if i is willing o relay or no (in Secions III and IV, we describe how UE i makes his decision). If UE s reques is R-ACK d, hen i receives daa from he BS, hrough UE i, in relay mode, and a virual oken ransfer akes place (from UE o UE i ); oherwise, UE receives daa direcly from he BS. Similar o [24], we assume ha each UE is equipped wih a secure and amper proof hardware module, which is locaed beween he UE s daa link and physical layers. The secure hardware module keeps rack of how many elecronic okens he UE currenly holds and helps manage he virual oken exchange process. 3 We say ha he oken exchange is virual because okens are no acually exchanged beween UEs; insead, upon receiving daa hrough a relay, a UE s oken holding is auomaically decreased by one, and upon compleing a relay ransmission, a UE s oken holding is auomaically incremened by one. Similar o [22], in order o preven malicious users from cheaing he oken sysem for heir own benefi, we assume ha he BS uses public-key crypography o creae digial signaures, which he relay and desinaion UEs can use o verify ha messages were in fac generaed by he BS and, furhermore, verify ha he messages were no ampered wih. 4 Our proposed virual oken exchange sysem works as follows: 1. All base saions owned by a nework operaor share a common privae key and an associaed public key. All UEs know he public key. This can be achieved by having he device s manufacurer or he nework operaor load he public key on each UE before i is sold o he end user. A benefi of his approach is ha he public key is very unlikely o be ampered wih by a malicious hird pary and, herefore, we can be confiden ha i is auhenic. 2. Before a base saion sends daa hrough a relay, i uses he privae key o digially sign he daa as shown in Fig. 1(a). 3. The relay UE s secure hardware module uses he public key o auhenicae he daa i ransmis (i.e., deermine ha i was generaed by he BS) as shown in Fig. 1(b). If he daa is auhenic, hen he relay s hardware module incremens is oken holding by one. Since he secure hardware module is locaed beween he daa link and physical layers, i will be very challenging for a malicious user o modify he daa afer i has been auhenicaed; herefore, his sysem prevens malicious UEs from earning okens if hey amper wih he daa before relaying i. 3 Noe ha, since UEs will already have o be designed o suppor D2D relaying, inegraing he secure hardware module will require very lile addiional engineering and cos overheads. Moreover, since consumers frequenly upgrade heir mobile handses, UEs wih hese capabiliies can be widely deployed in a reasonably shor ime frame (e.g. 1-2 years). 4 Digial signaures are analogous o sealing a message in an envelope wih a personal wax seal: he envelope can be opened by anyone, bu he unique seal auhenicaes he sender; moreover, if he seal is broken, hen he receiver knows ha he message inside he envelope may have been ampered wih. 8

9 4. The desinaion UE s secure hardware module uses he public key o auhenicae he daa i receives and verify is inegriy (i.e., deermine ha i was no alered by he relay) as shown in Fig. 1(c). If he daa is auhenic and has no been ampered wih, hen he desinaion UE s hardware module decremens is oken holding by one. This prevens desinaion UEs from losing okens if hey receive daa ha has been ampered wih. Daa from Link Layer Daa o Link Layer Daa Privae Key Digial Signaure Secure Module Public Key Verificaion success Tokens++ Secure Module Public Key Verificaion success Tokens-- Signed Daa o Relay UE (a) Base Saion Daa o Physical Layer (b) Relay UE Daa from Physical Layer (c) Desinaion UE Fig. 1. Virual oken exchange sysem archiecure. (a) Digial signaure generaion a he BS. (b) Daa verificaion in he relay UE s ransmi pah. (c) Daa verificaion in he desinaion UE s receive pah. C. UE Model We now describe our model of UE {1,, N}. Token holding sae: A any given ime, UE holds k K = {0,1,..., T} okens, where T is he oal number of okens in he nework. As described in Secion II.B, a UE mus expend one oken in order o receive daa hrough a relay, a UE can only earn addiional okens by relaying daa for oher UEs, and, if a UE does no have any okens, hen i canno receive daa hrough a relay. Relay energy budge sae: Since UEs are mobile, hey have limied baery energy. We le p max represen he oal amoun of baery energy ha he user is willing o consume relaying daa for oher users over he course of one baery charge. p max can be se based on user preferences. Wih each relay ransmission ha UE provides, i expends some energy, and herefore reduces is relay energy budge max p [0, p ]. When is relay energy budge reaches 0, UE will no longer relay daa for oher UEs and can no longer receive relay service from oher UEs; however, i will coninue o receive daa over he direc B2D link. In oher words, p = 0 does no mean ha he UE s baery is compleely drained; i only means ha he UE will no longer paricipae in he oken sysem. We refer o p = 0 as he dead sae and assume ha p is rese o p max when he UE s baery is recharged. Imporanly, he UE keeps any okens ha i holds when i eners he dead sae and can use hem once is baery is recharged. Noe ha, in our 9

10 problem formulaion, we assume ha p is coninuous (see Secion III); however, as in [34], we quanize p in our simulaion resuls o make our proposed opimizaion problem racable. The hree parameers described in he following hree paragraphs are used o characerize he nework environmen experienced by UE. These parameers are unknown a priori and ime-varying because, as summarized in Table 1, hey depend on various environmenal facors including UE s geographic locaion, is disance from he neares BS, he locaions of oher UEs in he nework, he channel condiions, and he cooperaion sraegies, oken holdings, and relay energy budges of he oher UEs. The definiions of each of hese parameers are illusraed in Fig. 2. Table 1. Parameers ha characerize UE s experienced nework environmen, and heir dependence on various environmenal facors ( indicaes dependence). Environmenal Facors Environmenal Parameers ORDR IRDR RRE λ i e Geographic locaion Disance o neares BS Channel condiions Locaions, cooperaion sraegies, oken holdings, and relay energy budges of oher UEs µ X Oubound relay demand rae (ORDR): We le λ denoe he probabiliy ha UE wans o ge help from a relay o receive daa from he BS. Given our illusraive mode and relay selecion sraegy [see Secion II.A.2], in which a UE seeks a relay if is SINR is below a arge hreshold, we have λ ( 0 a rg e Γ < Γ ) = Pr, (5) which means ha λ is equivalen o he ouage probabiliy of he B2D link o UE. Imporanly, λ is he probabiliy ha UE wans help from a relay, bu i is no he probabiliy ha i acually receives help from a relay. For his reason, we refer o λ as UE s oubound relay demand rae (ORDR). Inbound relay demand rae (IRDR): We le daa for anoher UE. We refer o µ denoe he probabiliy ha UE is asked o relay µ as UE s inbound relay demand rae (IRDR). Relay recruimen efficiency (RRE): Le e denoe he h UE s relay recruimen efficiency (RRE), which is defined as he following condiional probabiliy: e 0 arge Γ k p = Pr(R-ACK received <Γ, > 0, > 0). (6) In words, e is he condiional probabiliy ha UE ges help from a relay (i.e., receives a R-ACK), given ha i requires a relay ransmission, has a leas one oken o pay a relay, and has a non-zero relay energy 10

11 budge. The uncondiional probabiliy ha UE receives a R-ACK can be wrien as λe where I A is an indicaor variable ha is se o 1 when he even A occurs and is se o 0 oherwise. I, { k > 0, p > 0} Oubound Relay Reques Inbound Relay Reques UEs In Ouage (a) Idle UEs Relay Oubound Requesλ R-ACK e UE (Receiver) UE (Receiver) (c) Inbound Requesµ UE (Idle UE) UE (Relay) R-ACK σ ( k, p ) Receiver ( k, p ) ( k 1, p ) Fig. 2. Illusraion of oubound and inbound relay requess in a nework wih cooperaive downlink ransmission assuming ha k > 0 and p > 0. (a) UE seeks a relay from he pool of idle UEs wih probabiliy λ and is reques is R-ACK d wih probabiliy e. (b) Afer is relay reques is R-ACK d, UE ges help from a relay in exchange for one oken. (c) UE receives a relay reques wih probabiliy µ and R-ACK s he reques based on is cooperaion acion σ ( k, p ). (d) Afer i R-ACK s he reques, UE acs as a relay and expends c unis of energy in exchange for one oken. Cos and benefi: We le b be he benefi gained by UE when i receives daa hrough a relay and le c be he energy cos incurred by UE when i provides a relay ransmission o anoher UE. We assume ha c (b) 2 arge b = αlog (1 + Γ ) bis/s/hz, which is fixed for all relay ransmissions, and ha i i = P Joules, where P is he ransmission power used by UE o relay daa o UE i [recall from (4) ha he seleced relay can mee he arge rae using he leas ransmission power]. The consan α is se such ha b > c ; if his condiion does no hold, hen UE s have no incenive o cooperae [30]. Cooperaion acions: When a UE receives a relay reques, i mus decide if i will R-ACK he reques. We define he acion se A ( p ) as a funcion of he UE s relay energy budge p : i.e., {0,1}, p > 0 A ( p ) = (7) {0}, p = 0, where a = 1 A ( p ) (R-ACK) means ha he UE is willing o ac as a relay and a = 0 A ( p ) (R- NACK) means ha i is no. 5 If he UE is in he dead sae, hen i will no ac as a relay. (d) ( k, p ) ( k + 1, p c ) 5 One may argue ha A ( p ) = {0,1} only if p > c, where c is he amoun of energy required o relay. However, since p is he relay energy budge and no he acual baery sae, and c pmax, we allow relay ransmissions as long as p > 0. This simplifies he model. 11

12 Cooperaion policy: UE s cooperaion policy σ ( k, p ) A ( p ) is a funcion, which maps is curren oken holding sae k and relay energy budge p o a cooperaion acion a A ( p ). Given is IRDR, oken holding sae, relay energy budge, and cooperaion policy, he probabiliy ha UE relays daa for anoher UE in a ime slo is (, µσ k p ) I. The role of he cooperaion policy is illusraed in { p > 0} Fig. 2(c). Noe ha, a UE will only be able o ge relay service as ofen as i provides relay service because i mus earn as many okens as i spends. We discuss his in more deail in Secion III.B. Sae evoluion: We denoe he UE s sae by s = ( k, p ) S, where k K = {0,1,..., T} is is oken holding sae and max p [0, p ] is is relay energy budge. When he UE acs as a relay, i gains one oken and is relay energy budge reduces by c : i.e., Ac as a relay: ( k, p ) ( k + 1, p c ). (8) When he UE uses a relay, i loses one oken and is relay energy budge remains he same: i.e., Use a relay: ( k, p ) ( k 1, p ). (9) Transiion probabiliy funcion: We le ([, P k p ] [ k, p ], a ) denoe he sae ransiion probabiliy funcion, which gives he probabiliy ha UE ransiions from sae s = ( k, p ) o sae s = ( k, p ) afer aking cooperaion acion a. Based on (8) and (9), as well as he definiions of he ORDR λ, IRDR µ, and RRE e, he sae ransiion probabiliy funcion is defined as follows: λei, if k = k 1 and p = p { k > 0, p > 0} µ ai, if k = k + 1 and p = p c ([, ] [, ], ) { p > 0} = 1 λ ei µ ai, if k = k and p = p { k > 0, p > 0} { p > 0} P k p k p a 0, oherwise, Inuiively, in each ime slo, a UE can provide help as a relay, ge help from a relay, or do neiher: if UE has non-zero okens and a non-zero relay energy budge (i.e., I k > 0, p > 0 { } (10) = 1 ), hen wih probabiliy λ e i ges help and pays one oken [line 1 in (10)]; if UE has a non-zero relay energy budge (i.e., I p { > 0} = 1), hen wih probabiliy µ a i provides help, gains one oken, and loses c unis of baery energy [line 2 in (10)]; if UE has non-zero okens and a non-zero relay energy budge, hen wih probabiliy 1 λ e µ a i neiher ges help nor provides help [line 3 in (10)]; if UE has zero okens or is in he dead sae (i.e., I > 0, > 0 I p { > 0} { k p } = 0 ), hen i canno receive help; if UE is in he dead sae (i.e., = 0 ), hen i canno provide help; and, all oher cases occur wih probabiliy 0. Expeced uiliy: Le u ( k, p, a ) denoe UE s expeced uiliy in sae s = ( k, p ) when i akes cooperaion acion a. The expeced uiliy is defined as follows: (,, ) I λ eb I µ ac { k > 0, p > 0} { p > 0} u k p a =. (11) 12

13 Inuiively, if UE has non-zero okens and a non-zero relay energy budge, hen i ges help from a relay wih probabiliy λ e and receives benefi b ; if UE has a non-zero relay energy budge, hen i provides help wih probabiliy µ a and incurs cos c ; and, if UE has a zero relay energy budge, hen i can neiher provide help nor receive help, herefore, is uiliy wihin he oken sysem is 0. III. A. Problem Formulaion OPTIMAL COOPERATION POLICY FOR A SINGLE UE In his secion, we formulae he problem of deermining a UE s opimal cooperaion policy as an MDP [31]. We assume ha is ORDR λ, IRDR µ, and RRE e are known and fixed, and herefore is ransiion probabiliy and uiliy funcions are known and fixed; however, in Secion IV, we consider he case when hese parameers are unknown a priori and ime-varying, and propose a learning algorihm ha each UE can deploy o dynamically adap is cooperaion policy o is experienced nework environmen. Le u denoe UE s uiliy in ime slo and le E [ u ] = u ( k, p, a ) denoe is expecaion. Each UE aims o maximize is infinie horizon discouned uiliy 6 in which benefis and coss ha are received ime seps in he fuure are discouned by he facor β, where β [0,1) : ha is, E βu. (12) = 0 We assume ha he discoun facor β is a common parameer among all UEs and ha i is se by he nework operaor. If β= 0, hen each UE only considers is immediae uiliy, and will herefore never choose o ac as a relay (because i will incur cos c wihou any immediae benefi o iself). In general, larger values of β compel each UE o look farher ino he fuure o deermine is opimal cooperaion acion a he presen ime. If he nework environmen is saionary, hen discoun facors closer o 1 will always lead o beer long run performance; however, since he ORDR λ, IRDR µ, and RRE e are ime-varying in pracice, forecass of he fuure may be inaccurae, so discoun facors oo close o 1 may 6 There are oher opimizaion obecives ha one migh hink o use; however, hey are no well suied for he problem considered in his repor. h For example, a finie horizon undiscouned uiliy obecive [31], i.e., E u = 0, resuls in a non-saionary cooperaion policy, which requires a large amoun of memory o sore due o he long ime horizons ha are required in our opimizaion. This is furher exacerbaed by he fac ha he learning soluion we propose in Secion IV requires pre-compuing and soring many cooperaion sraegies. Alernaively, we could use receding-horizon conrol [32]. However, due o he long ime horizons required in our opimizaion, compuing he opimal decision in each ime sep would be compuaionally infeasible. h As anoher example, an infinie horizon undiscouned uiliy obecive [31], i.e., lim E u 0 h h =, canno be used for our problem because he Markov chain induced by any non-rivial cooperaion policy (in which he UE chooses o cooperae) is non-ergodic. This is due o he fac ha here is an absorbing sae in he MDP. We have chosen o use an infinie-horizon discouned formulaion because i is guaraneed o have a saionary opimal policy regardless of he srucure of he MDP, and i is less compuaionally complex han he finie-horizon case for long ime horizons (corresponding o discoun facors close o 1). 1 13

14 acually lead o worse long run performance. Deermining he opimal discoun facor is ou of he scope of his repor, bu we will consider he problem in fuure research. We define UE s opimal sae value funcion V ( k, p ) as he expeced infinie horizon discouned uiliy ha i will gain from each sae if i execues he opimal policy σ ( k, p ) : ha is, V ( k, p ) = maxe βu (13) σ = 0 The opimal sae value funcion is unique and saisfies he following Bellman opimaliy equaion: V ( k, p ) = max u ( k, p, a ) + β P([ k, p ] [ k, p ], a ) Vk (, p ) a ( p ). (14) A ( k, p ) S Q ( k, p, a ) In (14), we also defined he acion-value funcion Q ( k, p, a ), which is he value of aking cooperaion decision a in sae s = ( k, p ) and hen following he opimal policy hereafer. Given he acion-value funcion, i is easy o deermine he opimal cooperaion policy: σ ( k, p ) = arg max Q ( k, p, a ). (15) a A( p ) Imporanly, he opimal value funcion can be compued using he well-known value ieraion algorihm: V n 1( k, p ) max u ( k, p, a ) P ([ k, p ] [ k, p ], a ) V (, ) n k p + = + β a ( p ), (16) A ( k, p ) S where 0 (, V k p ) = 0 for all s = ( k, p ) S and V ( k, p ) converges o he opimal sae-value funcion V ( k, p ) as n [31]. n In pracice, he cos and ransiion probabiliy funcions are unknown a priori and ime-varying (due o he fac ha he ORDR λ, IRDR µ, and RRE e are unknown and ime-varying), so UE canno direcly apply value ieraion o find he opimal policy; insead, i mus learn is opimal cooperaion policy online based on is experience. We propose a soluion o he online problem in Secion IV. B. Balance of Earned and Expended Tokens We noed in Secion II.C ha, on average, a UE will only be able o ge relay service as ofen as i provides relay service because i mus earn as many okens as i spends. In his secion, we characerize his, balance under some simplifying assumpions. Le k +, 0 and k 0 denoe he cumulaive number of okens earned and expended by UE from ime slo 0 o ime slo, respecively, and le k0 0 denoe is iniial oken allocaion. Wihin he proposed oken sysem, he following condiion mus always hold:,, k k k, (17) 14

15 which indicaes ha he number of expended okens canno exceed he number of earned okens plus he number of iniial okens. Suppose ha UE has an unlimied relay energy budge (and an unlimied baery capaciy), and ha is ORDR λ, IRDR µ, and RRE e are fixed so ha we can sudy he seadysae behavior of (17). 7 If we divide boh sides of (17) by and ake he limi as, hen we ge Pr(UE receives help from a relay) Pr(UE provides help as a relay), (18) 1 Token + 1 Token because UE expends one oken every ime ha i receives help from a relay and i earns one oken every ime ha i provides help as a relay. The condiion in (18) can be rewrien as: [ ] [ (, λ ee )] I µ E σ k p k { k > 0} k, (19) 1 Token + 1 Token where he lef hand side is he probabiliy ha UE uses a relay (and expends one oken), he righ hand side is he probabiliy ha UE acs as a relay (and earns one oken), E [ ] denoes he expecaion over he UE s seady-sae oken holding disribuion, E [ I ] is he probabiliy ha UE has non-zero k { k > 0} okens, and E [ σ ( k, p )] is he probabiliy ha UE will R-ACK an inbound relay reques. Noe ha k equaliy holds in (19) if σ ( k, p ) is he opimal policy; if equaliy does no hold, hen UE earns more okens han i can spend, and herefore consumes more energy relaying han necessary. C. Threshold Sraegies UE s cooperaion decision depends on is oken holding and relay energy budge saes. Our simulaion resuls in Secion V.B show ha he opimal cooperaion policy σ ( k, p ) is hreshold in he k oken sae k and ha he hreshold depends on he relay energy budge p : specifically, 1, if k Kh( p ), σ ( k, p ) = 0, oherwise, where he hreshold Kh ( p ) is non-decreasing in (20) p. In oher words, as a UE s relay energy budge drains, i will no wan o earn as many okens because i will have less opporuniies o use hem before i eners he dead sae; herefore, i uses a lower hreshold. A rigorous proof of his resul is lef as fuure work; however, in [28], we show ha policies are hreshold if UEs have unlimied relay energy budges. IV. LEARNING THE OPTIMAL COOPERATION POLICY Each UE experiences differen environmenal dynamics depending on is geographic locaion, is disance from he neares BS, is channel condiions, and, imporanly, he locaions, cooperaion sraegies, oken holdings, and relay energy budges of he oher UEs. The dynamics experienced by UE are 7 If a UE s relay energy budge is finie, hen here will be no oken exchanges in seady-sae. This is because, in seady-sae, he UE s relay energy budge will be zero. By assuming ha he relay energy budge and baery lifeime are boh unlimied, we can use he seady-sae behavior o approximae he behavior we observe while he UE s relay energy budge is non-zero, bu finie. 15

16 capured by hree parameers, namely, he ORDR λ, he IRDR µ, and he RRE e, which are all defined in Secion II.C. In his secion, we propose a simple, low-complexiy, supervised learning algorihm ha a UE can deploy o learn is opimal cooperaion policy σ ( k, p ) online, despie he fac ha hese hree parameers are unknown and ime-varying. 8 Our proposed learning algorihm comprises an offline phase and an online phase. In he online phase, UE esimaes is ORDR λ, IRDR µ, and RRE e. Then, in each ime slo, UE selecs is cooperaion policy based on he esimaed values. In principle, UE could use hese esimaed values o populae he ransiion probabiliy and uiliy funcions defined in (10) and (11), respecively, and hen use value ieraion [see (16)] o compue he corresponding opimal cooperaion policy; however, recompuing he opimal policy in every ime slo would be compuaionally prohibiive. For his reason, we propose o firs compue a collecion of cooperaion policies offline, which correspond o a represenaive se of discreized environmenal parameers, and hen use a simple look-up able o selec UE s cooperaion policy online in each ime slo. To reduce he size of he look-up able, insead of esimaing he ORDR λ and RRE e independenly, we direcly esimae π = λe [0,1 / 2], which we refer o as he oubound relay success rae. Noe ha he maximum oubound relay success rae is ½ because, on average, a UE can only receive relay service as ofen as i provides relay service (see Secion III.B). Addiionally, since he opimal policy is hreshold in he oken sae (see Secion III.C), each policy can be represened compacly wih only one (hreshold) value per relay energy budge sae. We provide pseudo-code for he offline and online phases in Table 2 and Table 3, respecively, and describe each phase in more deail in Secions IV.A and IV.B, respecively. A. Offline Phase 2 Le 1 X Π= { π, π,, π } 1 2 Y 1 2 Z, Μ= { µ, µ,, µ }, and C = { c, c,, c } be finie ses conaining represenaive values of he oubound relay success rae, IRDR, and relay cos, respecively. In he offline phase of he proposed learning algorihm, we compue he collecion of policies { σ( kp, πµ,, c) : π, µ, c } Π Μ C. To compue he policy σ( kp, πµ,, c), we firs populae he ransiion probabiliy and uiliy funcions defined in (10) and (11), respecively, wih π, µ, and c in place of λ e, µ, and c (lines 3-4 in Table 2). Subsequenly, we use value ieraion o compue he opimal sae-value funcion and he opimal cooperaion policy corresponding o he represenaive parameers (lines 5-6 in Table 2). 8 We do no use a more convenional reinforcemen learning based approach, such as a Q-learning [31], because our problem does no saisfy he required convergence condiions. Specifically, Q-learning requires ha all sae-acion pairs are visied infiniely ofen : ha is, in he limi as, every acion mus be aken in every sae an infinie number of imes. This condiion does no hold for our problem because he relay energy budge is non-increasing in ime. We refer he ineresed reader o he previous version of his echnical repor [39] for an in deph sudy of he shorcomings of radiional reinforcemen learning algorihms in he conex of our problem. 16

17 Since he ses of possible oubound relay success raes, IRDRs, and relay coss are discreized ino X, Y, and Z values, respecively, a oal of XYZ policies mus be compued offline and sored in a look-up able. Assuming ha he relay energy budge is discreized ino W values, he oal amoun of memory required o sore he look-up able is OWXYZ ( ) (because each policy can be compacly represened wih one hreshold value per relay energy budge sae). For example, if each variable is discreized ino 10 values, and he hreshold is represened by one bye, hen he lookup able will require 10 KB of memory. Imporanly, he offline phase can be performed before he consumer purchases her cellular device and he collecion of opimal policies can be preloaded ono he device (e.g., by he device s manufacurer or by he nework operaor). Addiionally, he offline phase can be performed again whenever he consumer changes her relaying preferences, i.e., changes her maximum relay energy budge p max. B. Online Phase Table 2. Learning algorihm: Offline phase 2 Inpu: 1. 1 X Π= { π, π,, π } 1 2 Y 1 2 Z, Μ= { µ, µ,, µ }, C = { c, c,, c }, p max, and he number of quanized relay energy budge saes 2. For each ( πµ,, c) Π Μ C 3. Populae he ransiion probabiliy funcion ([ k, p ] [ k, p], a) 4. Populae he uiliy funcion ukpa (,, ) as defined in (11) P as defined in (10) 5. Calculae opimal sae-value funcion Vkp (, ) using value ieraion (16) 6. Calculae he opimal cooperaion policy σ ( kp, ) using (15) 7. Record he opimal cooperaion policy σ( kp, πµ,, c) σ( kp, ) 8. End 9. Oupu: Collecion of policies { σ( kp, πµ,, c) : π Π, µ Μ, c C } In he online porion of he algorihm, each UE mainains online esimaes of is oubound relay success rae π, denoed by π ˆ, and is IRDR µ, denoed by µ ˆ. These esimaes can be deermined using an exponenial moving average of successful oubound relay requess and inbound relay requess, respecively (lines in Table 3). Upon receiving an inbound relay reques (line 6 in Table 3), UE evaluaes he energy cos c i will incur if i provides a relay ransmission (line 8 in Table 3). UE hen akes he acion a = σ ( k, p f( π ˆ ), g( µ ˆ ), hc ( )), where f : [0,1 / 2] Π maps he esimae π ˆ o he neares value in Π, g : [0,1] Μ maps he esimae µ ˆ o he neares value in Μ, and h : [0,1] C maps he cos c o he neares value in C (line 9 in Table 3). 17

18 Table 3. Learning algorihm: Online phase. 1. Iniialize: k 0, p0 = pmax, π ˆ 0, µ ˆ 0, f (), g (), h (), w 2. For = 0,1, 3. oubound _ success 0 and inbound _ reques 0 4. If k > 0 and p > 0 and UE has a successful oubound relay reques 5. oubound _ success 1 6. Else if p > 0 and UE receives an inbound relay reques 7. inbound _ reques 1 8. Deermine energy cos c for relay ransmission 9. Take cooperaion acion a σ ( k, p f( π ˆ ), g( µ ˆ ), hc ( )) 10. End w 1 πˆ + 1 oubound _ success+ πˆ w w w 1 µ ˆ ˆ + 1 inbound _ reques+ µ w w 13. Deermine he nex sae as describe in Secion II.C 14. End V. SIMULATION RESULTS In his secion, we presen our simulaion resuls. In Secion V.A, we describe he simulaion seup ha is used in Secions V.C-V.F. In Secion V.B, we presen several numerical resuls o highligh he srucure of he opimal cooperaion policies. Our focus in Secions V.B is on he behavior of a single UE s opimal (bes response) cooperaion policy given he aggregae behavior of he oher UEs in he nework. For his reason, he resuls in Secions V.B are generaed ouside of he nework simulaion so ha we have full conrol over he UE s environmen. In Secions V.C-V.F, we simulae a large-scale cellular nework in which all users simulaneously deploy he supervised learning algorihm proposed in Secion IV. Specifically, in Secion V.C, we highligh he key differences beween he proposed approach o cooperaion (in which users are self-ineresed and mus be incenivized o cooperae using he oken sysem) and he convenional approach o cooperaion (in which users are obedien and are obliged o cooperae even if i is no in heir self-ineres) in erms of individual user performance and overall nework performance. Then, in Secions V.D and V.E, we invesigae how UEs mobiliy and relay energy budges impac heir own performance, he performance of oher UEs, and he overall nework performance wihin he proposed framework. Finally, in Secion V.F, we sudy he impac of he nework-wide oken supply on he overall nework performance. A. Cellular Nework Simulaion Seup In our illusraive simulaion seup, we assume ha N= 1500 mobile ransceivers are uniformly and randomly disribued in a 10 km x 10 km square area consising of 100 cells wih size 1 km x 1 km. There 18

19 is one BS a he cener of each cell. In each ime slo, each UE moves o a nearby locaion according o a random waypoin mobiliy model ha is commonly used in he simulaion of mobile neworks [35][36] and may need o receive daa from he BS in he corresponding cell. We assume a ime slo duraion of = 5 s, which implies ha downlink resources are allocaed every 5 s and ha he channel is invarian over his allocaed ime. In pracice, he ime slo duraion would be much shorer o boh mach he imescale a which mos cellular neworks schedule resources (e.g., 10 ms) and o ensure channel coherence in each ime slo; however, in order o observe he effec of mobiliy on sysem performance, we need o operae on a larger ime scale. Noe ha his is no a limiaion of our proposed framework, bu is insead a limiaion of our abiliy o simulae a large-scale sysem wih a small ime slo duraion. In paricular, using a 10 ms ime slo duraion o simulae, e.g., one hour of real-ime sysem operaion, requires 500 imes more compuer ime han using a 5 second ime slo duraion. We divide users/ues ino wo mobiliy classes and wo relay energy budge classes: 1. High mobiliy users move a speeds beween 50 and 120 km/hour (similar o [37] which uses a maximum speed of 30 m/s = 108 km/hour in is experimens). For insance, users in moor vehicles are considered high mobiliy users. These users play differen roles in he nework over ime because someimes hey will be far from a base saion where hey will have high ORDRs, and someimes hey will be closer o a base saion where hey will have high IRDRs. 2. Low mobiliy users move a speeds beween 0 and 8 km per hour. For example, users in offices, resaurans, or on foo are considered o be low mobiliy users. Due o heir limied mobiliy, hese users ypically do no swich roles over ime and heir ORDRs and IRDRs are relaively saic. 3. High relay energy budge UEs can R-ACK many relay requess before enering he dead sae. 4. Low relay energy budge UEs can R-ACK fewer relay requess before enering he dead sae. The relay energy budges of UEs in he high and low relay energy budge classes are specified separaely for each of our simulaions. For compuaional reasons (i.e., so we can compue he opimal policy wih value ieraion), we quanize he relay energy budge sae ino eleven bins (one represening he dead sae) and we limi he maximum number of okens ha he UE can hold o We consider pah loss and shadow fading for he channel model such ha [33], P = P PL( d ) 10 η log( d / d ) χ, (21) rx x 0 0 where P rx and P x are he receive and ransmi powers (in db), respecively, PLd ( 0) is he pah loss of he reference disance d 0, d is he disance beween he source and desinaion, η is he pah loss facor, and χ is a Gaussian disribued random variable represening he effec of shadow fading. We assume ha he 9 Noe ha maximum oken holding is se o be above he highes cooperaion hreshold so ha i does no affec he opimal policy. 19

20 maximum ransmission power of he BS and UEs is 15 dbm, he oal sysem bandwidh is 50 MHz, a maximum of 10 MHz can be allocaed o each downlink ransmission, and he arge daa rae is 1 bi/s/hz. If he arge downlink daa rae canno be achieved, hen he UE requess a relay ransmission. Using he above parameer values, he average ORDR hroughou he nework is approximaely λ= 0.1. All UEs adap heir cooperaion policies using he supervised learning algorihm proposed in Secion IV and all UEs use a policy look-up able ha is compued using he represenaive oubound relay success raes, IRDRs, and coss ha are given in Table 4. Finally, we assume ha here are T= 9000 okens in he nework ha are uniformly and randomly disribued among he UEs a he sar of he simulaion. We discuss he impac he oken supply in Secion V.F. The parameers used in our simulaions are summarized in Table 4. Table 4. Simulaion parameers. Parameer Noaion Value Number of UEs N 1500 Time slo duraion 5 s Maximum ransmi power max P 15 dbm (32 mw) Toal BW / Max. BW allocaion N/A 50 / 10 MHz Pah loss facor η 3 Reference disance / pahloss / PLd ( ) 100 m / -5 db Noise power specral densiy N 0 Sd. dev. of Gaussian fading var( χ ) -10 dbm 2 db Targe SINR arge Γ 0 db Token supply T Variable Discoun facor β 0.99 Oubound relay success rae se Π { 0.05, 0.1,,0.45} Inbound relay demand rae se Μ { 0.05, 0.1,,0.45} Cos se C { 0.025, 0.05,,0.225} J Benefi b 0.5 Learning window size w 50 No. quanized relay energy budge N/A saes 11 (including dead sae) Relay energy budge d0 0 p max Variable User mobiliy N/A Variable Before discussing our resuls, we need o inroduce wo new definiions: R-ACK rae: The R-ACK rae is he probabiliy ha, upon receiving a relay reques, a UE sends a R-ACK. Under he assumpions in Secion III.B, he R-ACK rae is equivalen o E [ σ ( k, p )]. Throughpu gain: The hroughpu gain is he raio of he acual hroughpu o he direc ransmission hroughpu. Since he acual hroughpu is always greaer han or equal o he direc k 20

21 hroughpu, he minimum hroughpu gain is 1. Noe ha a hroughpu gain of 1 is achieved if users are self-ineresed and he oken sysem is no in place o incenivize cooperaion. B. Srucure of he Opimal Policy In his secion, we presen several figures o highligh he srucure of he opimal cooperaion policy. Throughou his secion, we fix cerain parameers o illusrae he behavior of he opimal cooperaion policy as we vary oher parameers; imporanly, he general behavior of he opimal cooperaion policy wih respec o he variable parameers does no depend on he specific values of he fixed parameers. We specify he fixed parameers in each figure s capion or ile. We assume ha each UE has a relay energy budge ha allows i o R-ACK 1000 relay requess on average before enering he dead sae. In Fig. 3, we illusrae he opimal cooperaion policy σ ( kp, ). As we discussed in Secion III.C, he opimal policy is hreshold in he oken sae k, and he hreshold decreases as he relay energy budge p decreases. All policies ha we have observed have his hreshold srucure. π=λe=0.45, µ=0.45, c/b=0.45 Opimal Policy [σ(k,p)] Fig. 3. Srucure of he opimal cooperaion policy σ ( kp, ) wih discoun facor β= Fig. 4(a) illusraes how he opimal cooperaion hreshold varies wih respec o he discoun facor β and he cos-o-benefi raio c / b (noe ha he policy depends on he cos-o-benefi raio, raher han he absolue values of he coss and benefis [30]). The opimal cooperaion hreshold is non-decreasing in β because, if a UE looks farher ahead, hen i anicipaes more opporuniies o use addiional okens, and herefore has more incenive o cooperae in order o earn more okens. Addiionally, as we proved in our prior work [28], he opimal hreshold decreases as he cos-o-benefi raio c / b increases. Fig. 4(b) illusraes how he opimal decision hreshold varies wih respec o he cos-o-benefi raio c / b and IRDR µ. The opimal decision hreshold increases as he IRDR decreases because, if a UE is asked o relay infrequenly, hen i has incenive o earn okens whenever i has he opporuniy. Fig. 4(c) illusraes how he opimal decision hreshold varies wih respec o he cos-o-benefi raio c / b and he oubound relay success rae Tokens (k) Baery (p) π= λe. We observe ha he opimal decision hreshold 21

22 decreases as he oubound relay success rae decreases. This happens because, if a UE is unable o successfully recrui relays, hen i does no have incenive o gaher many okens. Fig. 4. Srucure of he opimal cooperaion policy. (a) Opimal hresholds for various values of β and c / b. (b) Opimal hresholds for various values of µ and c / b. (c) Opimal hresholds for various values of π and c / b. The resuls in (b) and (c) use discoun facor β= C. Comparison o Convenional Cooperaive Relaying The foundaional assumpion in his repor is ha UEs are self-ineresed. In oher words, hey aim o maximize heir own uiliies and do no care abou he overall performance of he nework. Since relaying coss energy and provides no immediae benefi o he relay, self-ineresed UEs will never cooperae unless hey are incenivized o do so. Consequenly, wihou he proposed oken-based incenives, selfineresed UEs will never achieve more han he direc B2D hroughpu. In conras, he ypical assumpion in he cooperaive communicaions lieraure (see, e.g., [2]-[4][7]-[11]) is ha UEs are obedien, i.e., hey will always cooperae even when i is no in heir self-ineres. To highligh he key differences beween a nework of self-ineresed users and a nework of obedien users in erms of individual user performance and overall nework performance, we compare he following hree scenarios: 1. All UEs are self-ineresed, have finie relay energy budges, and are incenivized o cooperae by using okens (i.e., he proposed soluion). 2. All UEs are obedien and have unlimied relay energy budges (and unlimied baery energy). 3. All UEs are obedien and have limied relay energy budges. We use he simulaion es-bed described in Secion V.A. We assume ha 70% (30%) of users have high (low) mobiliy and ha 70% (30%) of users have high (low) relay energy budges. We assume ha UEs wih high (low) relay budges can relay 100 (40) imes on average before enering he dead sae. These choices of high and low relay energy budges allow us o highligh how he nework performance changes as increasing numbers of UEs wih low relay energy budges ransiion o he dead sae while UEs wih high relay energy budges remain acive. Imporanly, we have verified hrough numerous simulaions (no repored here due o space limiaions) ha any oher choice of iniial high (low) relay energy budges naurally leads o similar ransiory nework behavior. 22

23 (a) (b) Fig. 5. Performance saisics for heerogeneous users (70% high mobiliy and 70% high relay energy budge) in hree differen scenarios. (Scenario 1): Self-ineresed users wih finie relay energy budges operaing wihin he proposed oken sysem. (Scenario 2): Obedien users wih infinie relay energy budges. (Scenario 3): Obedien users wih finie relay energy budges. (a) Average hroughpu gains. (b) Cumulaive frequencies of users average uiliies. In Fig. 5(a), we compare he average hroughpu gain obained in he hree aforemenioned scenarios. In Fig. 5(b), we compare he cumulaive frequencies of he users average uiliies for each of he scenarios in Fig. 5(a). From Fig. 5(a), i is clear ha he overall nework performance is bes in he case when all users are obedien and have infinie relay energy budges. However, Fig. 5(b) shows ha over 250 of he individual UEs in he nework (over 16%) acually obain negaive average uiliies, and are herefore being obliged o cooperae even hough i is no in heir self-ineres. These poor performing UEs ypically have low mobiliy and are in he core of he nework so hey end up providing relay service much more frequenly han hey can exploi relay service hemselves. These low mobiliy users are responsible for he relaively low 25 h percenile hroughpu gain illusraed in he cener of Fig. 5(a). In conras, he low mobiliy users a he periphery of he nework end o perform very well because hey end up exploiing relay service more frequenly han hey need o provide i. In effec, he low mobiliy users a he periphery are free-riders: hey benefi from cooperaion wihou bearing is coss. These low mobiliy users are responsible for he relaively high 75 h percenile hroughpu gain illusraed in he cener of Fig. 5(a) and are herefore largely responsible for he average nework-wide hroughpu gains. When all users are obedien, bu have finie relay energy budges, he overall nework performance degrades relaive o he infinie budge case [see Fig. 5(a)], bu is sill beer han he case wih selfineresed users. The performance degradaion relaive o he infinie relay budge case occurs for wo main reasons: (i) many low relay budge and low mobiliy users in he core of a cell provide relay service very frequenly; consequenly, hey ener he dead sae before making i o he periphery of a cell where hey could balance ou heir average uiliy by exploiing relay service more frequenly (as some low mobiliy users do in he infinie relay budge case); and (ii) as he low relay budge UEs ener he dead sae, he 23

24 densiy of acive UEs in he nework decreases leading o higher relay coss for all remaining acive UEs. While he overall nework performance is beer han he case wih self-ineresed users, Fig. 5(b) shows ha over 700 of he individual UEs in he nework (over 46%) acually obain negaive average uiliies, and are herefore being obliged o cooperae even hough i is no in heir self-ineres. When all users are self-ineresed and have limied relay energy budges, he overall nework performance is lower han when all users are obedien [see Fig. 5(a)]; however, Fig. 5(b) shows ha over 1490 of he individual UEs in he nework (over 99%) obain posiive average uiliies. This is because selfineresed users will only relay if he marginal value of having an addiional oken ouweighs he cos of obaining i. Noe ha he few self-ineresed UEs ha obain (very small) negaive average uiliies each provided relay service a few more imes han hey exploied relay service us before he simulaion ended; imporanly, unlike in he cases wih obedien users, he negaive average uiliies do no reflec a fundamenal absence of incenive o relay, bu are simply an arifac of he finie simulaion duraion. D. Impac of Mobiliy In his secion, we use he simulaion es-bed described in Secion V.A o invesigae how each UE s mobiliy impacs is own performance, he performance of oher UEs, and he overall nework performance. We consider five mobiliy mixures in which he nework comprises 10%, 30%, 50%, 70%, and 90% high mobiliy users, and he remaining users have low mobiliy. In oal, here are N= 1500 users. We assume ha all users sar wih he maximum relay energy budge, which is defined such ha each user can R-ACK an average of 1000 relay requess before enering he dead sae. We use a relaively high relay energy budge compared o he 3000 ime slo simulaion duraion so we do no conflae he impac of low relay energy budges wih he impac of mobiliy (we invesigae he impac of heerogeneous relay energy budges in Secion V.E). We use a discoun facor β= In Fig. 6, we plo he average ORDR λ, average IRDR µ, average RRE e, average relay lifeime (i.e., he amoun of ime a UE is no in he dead sae), average R-ACK rae, and average hroughpu gain over 3000 ime slos for users in each mobiliy class (i.e., high and low mobiliy) and for all users combined. The lower and upper error bars in Fig. 6 indicae he 25 h and 75 h perceniles, respecively, over he considered se of users. ORDR and IRDR: Fig. 6(a) and Fig. 6(b) illusrae he average ORDR and average IRDR, respecively, under he various user mobiliy mixures. Recall ha, a he beginning of he simulaion, users are uniformly disribued hroughou he nework. Since he low mobiliy users do no significanly deviae from heir saring posiions, and he high mobiliy users move hroughou he nework, he average ORDR and IRDR for users in each class is approximaely he same; however, he variaion of ORDRs and IRDRs across users wihin each class varies significanly. In paricular, he low mobiliy users have a much larger 24

25 range of ORDRs and IRDRs because, if hey are a he periphery of a cell, hen hey will have large ORDRs and small IRDRs, and if hey are in he core of a cell, hen hey will have small ORDRs and large IRDRs. In conras, he high mobiliy users move beween he peripheries and cores of various cells over ime, and herefore experience less deviaion in hese parameers across he populaion. Fig. 6. Impac of he mobiliy mixure. (a) Average ORDR λ. (b) Average IRDR µ. (c) Average RRE e. (d) Average R-ACK rae. (e) Average relay lifeime. (f) Average hroughpu gain. RRE and R-ACK rae: Fig. 6(c) and Fig. 6(d) illusrae he average RRE and average R-ACK rae, respecively, under he various user mobiliy mixures. Imporanly, hese parameers are inimaely ied o each user s ORDR and IRDR. On average, low mobiliy users have lower relay recruimen efficiencies, lower R-ACK raes, and more variaion of hese parameers across he populaion. These effecs emerge because low mobiliy users have srongly imbalanced ORDRs and IRDRs unless hey are lucky enough o be siuaed beween he core and periphery of a cell. If a UE s ORDR is larger han is IRDR, i.e., λ > µ, hen i ends o run ou of okens [i.e., E k[ I { k> 0} ] in (19) will be small]. Wihou okens, i canno recrui relays, which reduces is RRE. Noe ha his also hurs oher UEs because i reduces opporuniies for hem o earn okens. On he oher hand, if a UE s ORDR is smaller han is IRDR, i.e., λ < µ, hen i will end o collec a surplus of okens and no have any incenive o R-ACK incoming relay requess, which in urn reduces he RREs of he oher UEs in he nework. 25

26 Average relay lifeime: Fig. 6(e) illusraes he average relay lifeime of he UEs under various mobiliy mixures. As we noed earlier, we inenionally seleced he relay energy budges such ha no UEs would ener he dead sae, i.e., p= 0, in hese simulaions. Throughpu gain relaive o direc ransmission: Fig. 6(f) illusraes he average hroughpu gain under he various user mobiliy mixures. As expeced, having a higher fracion of high mobiliy users in he nework improves he average nework hroughpu (relaive o direc ransmission only) because hese users play differen roles in he nework over ime, which enables a consisen exchange of okens beween UEs in he periphery and core of each cell. E. Impac of he Relay Energy Budge Users will selec differen relay energy budges based on heir preferences. In his secion, we sudy how each UE s relay energy budge impacs is own performance, he performance of oher UEs, and he overall nework performance. We use he simulaion es-bed described in Secion V.A wih discoun facor β= As in Secion V.C, we assume ha UEs wih high (low) relay budges can relay 100 (40) imes on average before enering he dead sae. We consider five mixures in which he nework comprises 10%, 30%, 50%, 70%, and 90% users wih high relay energy budges, and he remaining users have low relay energy budges. In oal, here are N= 1500 users. We assume ha all users are high mobiliy users so we can clearly separae he impac of users wih low relay energy budges from he impac of users wih low mobiliy. In Fig. 7, similar o Fig. 6, we plo he average ORDR λ, average IRDR µ, average RRE e, average R-ACK rae, average relay lifeime, and average hroughpu gain for users in each relay energy budge class (i.e., high and low) and for all users combined. Noe ha hese merics are measured over he lifeime of each user (i.e., for all ime slos before a user eners he dead sae, p= 0 ). The oal simulaion duraion is 3000 ime slos. The error bars in Fig. 7 show he 25 h and 75 h perceniles over he considered se of users. ORDR and IRDR: Fig. 7(a) and Fig. 7(b) illusrae he average ORDR and average IRDR, respecively, under he various relay energy budge mixures. We observe ha he relay energy budge has an insignifican impac on he ORDR and a minor impac on he IRDR of he high relay budge users. Specifically, as he number of UEs in he dead sae increases, here are less UEs in he nework ha require help from relays, and herefore he high relay energy budge UEs IRDRs decrease. 26

27 Avg. ORDR % of High Baery Devices Avg. IRDR Low Budge Baery All High Budge Baery % of High Baery Devices (a) % of High Budge UEs (b) % of High Budge UEs (c) Avg. Relay Recruimen Efficiency % of High Baery Devices % of High Budge UEs Avg. ACK Rae Avg. Lifeime Avg. Throughpu Gain % of High Baery Devices % of High Baery Devices % of High Baery Devices (d) % of High Budge UEs (e) % of High Budge UEs (f) % of High Budge UEs Fig. 7. Impac of differen relay energy budge mixures wih T= 9000 okens. (a) Average ORDR λ. (b) Average IRDR µ. (c) Average RRE e. (d) Average R-ACK rae. (e) Average lifeime. (f) Average hroughpu gain. RRE and R-ACK rae: Fig. 7(c) and Fig. 7(d) illusrae he average RRE and average R-ACK rae, respecively, under he various relay energy budge mixures. When here are many low UEs wih low relay budges in he nework, he average RRE and average R-ACK rae are low. This is because UEs wih low relay budges have lower hresholds in heir opimal cooperaion policies [see Fig. 4(a)]; herefore, when here are many low relay budge UEs, mos of hem will have sufficien okens and, consequenly, have no incenive o R-ACK incoming relay requess. Ineresingly, when he low relay budge UEs carry less han he average number of okens, he high relay budge UEs mus carry more han he average number of okens; herefore, mos of he high relay budge UEs will also have sufficien okens and, consequenly, have no incenive o R-ACK incoming relay requess. Since he R-ACK rae is small, and he ORDR and IRDR are similar, i follows from he balance equaion (19) ha he RRE mus also be small. Anoher ineresing phenomenon in he nework is ha he UEs wih high relay energy budges have significanly lower average RREs and R-ACK raes han he UEs wih low relay energy budges. This is because, as he low relay budge UEs drain heir budges, hey furher reduce heir oken holdings, which in urn increases he oken holdings of UEs wih high relay energy budges. Since he high relay budge UEs are flooded wih okens, hey have no incenive o R-ACK incoming relay requess. As illusraed in 1 27

28 Fig. 8, hese ineresing phenomena can be compensaed for by reducing he oken supply in he nework so ha UEs hold fewer okens and have incenive o R-ACK more incoming relay requess [28]. We furher discuss he impac of he oken supply on he overall nework performance in Secion V.F. Average relay lifeime: Fig. 7(e) illusraes he average relay lifeime of he UEs under he various relay energy budge mixures. The average relay lifeime (for UEs wih boh high and low relay budges) decreases as he number of high relay budge UEs increases because he R-ACK rae is higher. Throughpu gain relaive o direc ransmission: Fig. 7(f) illusraes he average hroughpu gain under he various relay energy budge mixures. The hroughpu gain is high (low) when he RRE and R- ACK rae are high (low). Fig. 8. Impac of oken supply on he hroughpu gain under differen relay energy budge mixures. Compare o Fig. 7(f). (a) T= 7000 okens. (b) T= 8000 okens. F. Impac of he Token Supply The number of okens in he nework is an imporan design parameer ha can significanly impac he average nework hroughpu as illusraed in Fig. 8. In Fig. 9, we illusrae he impac of he oken supply on he average hroughpu gain using oken supplies ranging from 0 o in 1000 oken incremens. The resuls in Fig. 9 were generaed under ideal condiions where all 1500 users have high mobiliy and high relay energy budges; however, he general rend does no depend on he specific mixure of users. The bes overall nework performance in Fig. 9 is achieved when here are 8000 okens in he nework. If here are oo few okens in he nework, hen many UEs will have zero okens and herefore be unable o buy relay service when hey need i. As illusraed in Fig. 9, his will significanly reduce he average nework hroughpu. In he exreme case ha here are zero okens in he nework (i.e., here is no oken sysem), here is no incenive for he self-ineresed UEs o provide relay services, so he average nework hroughpu will be equivalen o ha of a nework wih only direc B2D ransmission. If here are oo many okens in he nework, hen many UEs will have sufficien okens (i.e., h ( k > K p ) ) and will herefore 28