Distributed Resource Management in Multi-hop Cognitive Radio Networks for Delay Sensitive Transmission

Transcription

1 1 Distributed Resource Maagemet i Multi-hop Cogitive Radio Networs for Delay Sesitive Trasmissio Hsie-Po Shiag ad Mihaela va der Schaar Departmet of Electrical Egieerig (EE), Uiversity of Califoria Los Ageles (UCLA) Los Ageles, CA {hpshiag, ABSTRACT this paper, we ivestigate the problem of multi-user resource maagemet i multi-hop cogitive radio etwors for delay-sesitive applicatios. Sice the tolerable delay does ot allow propagatig global iformatio bac ad forth throughout the multi-hop etwor to a cetralized decisio maer, the source odes ad relays eed to adapt their actios (trasmissio frequecy chael ad route selectios) i a distributed maer, based o local etwor iformatio. We propose a distributed resource maagemet algorithm that allows etwor odes to exchage iformatio ad that explicitly cosiders the delays ad cost of exchagig the etwor iformatio over the multi-hop cogitive radio etwors. The term cogitive refers i our paper to both the capability of the etwor odes to achieve large spectral efficiecies by dyamically exploitig available frequecy chaels as well as their ability to lear the eviromet (the actios of iterferig odes) based o the desiged iformatio exchage. Note that the ode competitio is due to the mutual iterferece of eighborig odes usig the same frequecy chael. Based o this, we adopt a multi-aget learig approach, adaptive fictitious play, which uses the available iterferece iformatio. We also discuss the tradeoff betwee the cost of the required iformatio exchage ad the learig efficiecy. The results show that our distributed resource maagemet approach improves the PSNR of multiple video streams by more tha 3dB as opposed to the state-of-the-art dyamic frequecy chael/route selectio approaches without learig capability, whe the etwor resources are limited. dex Terms: distributed resource maagemet, cogitive radio etwors, multi-hop wireless etwors, multi-aget learig, delay sesitive applicatios.. NTRODUCTON The demad for wireless spectrum has icreased ad will eep icreasig rapidly i the foreseeable future with the itroductio of multimedia applicatios such as YouTube, peer to peer multimedia etwors, ad distributed gamig. However, scaig through the radio spectrum reveals its iefficiet occupacy i most This wor was supported by ONR.

2 2 frequecy chaels. Hece, the Federal Commuicatios Commissio (FCC) suggested i 2002 [1] improvemets for spectrum usage, which eable more efficiet allocatios of frequecy chaels to licese-exempt users without impactig the primary licesees. Based o this, cogitive radio etwors [2][3] were proposed which eable wireless users to sese ad lear the surroudig eviromet ad correspodigly adapt their trasmissio strategies. such cogitive wireless eviromets, two mai challeges arise. The first challege is how to sese the spectrum ad model the behavior of the primary licesees to idetify available frequecy chaels (spectrum holes) 1. The secod challege is how to maage the available spectrum resources amog the licese-exempt users to satisfy their QoS requiremets while limitig the iterferece to the primary licesees. this paper, we focus o the secod problem, i.e. the resource maagemet, ad rely o the existig literature for the first challege [4][5]. The majority of the resource maagemet research i cogitive radio etwors has focused o a sigle-hop wireless ifrastructure [6]-[10]. this paper, we focus o the resource maagemet problem i the more geeral settig of multi-hop cogitive radio etwors. A ey advatage of such flexible multi-hop ifrastructures is that the same ifrastructure ca be re-used ad recofigured to relay the cotet gathered by various trasmittig users (e.g. sources odes) to their receivig users (e.g. sis odes). These users may have differet goals (applicatio utilities etc.) ad may be located at various locatios. For the multi-hop ifrastructure, there are three ey differeces as opposed to the sigle-hop case. First, the users have as available etwor resources ot oly the vacat frequecy chaels (spectrum holes or spectrum opportuities [2][6]) as i the sigle-hop case, but also the routes through the various etwor relays to the destiatio odes. Secod, the trasmissio strategies will eed to be adapted ot oly at the source odes, but also at the etwor relay odes. cogitive radio etwors, etwor odes are geerally capable of sesig the spectrum ad modelig the behavior of the primary users ad thereby, idetifyig the available spectrum holes. multi-hop cogitive radio etwors, the etwor odes will also eed to model the behavior of the other eighbor odes (i.e. other secodary users) i order to successfully optimize the routig decisios. other words, etwor relays also require a learig capability i the multi-hop cogitive radio etwor. Third, to lear ad efficietly adapt their decisios over 1 the wireless eviromet without primary licesees, such as the SM bad, there is o such problem. The mai challege is the resource maagemet problem.

3 3 time, the wireless odes eed to possess accurate (timely) iformatio about the chael coditios, iterferece patters ad other odes trasmissio strategies. However, i a distributed settig such as a multi-hop cogitive radio etwor, the iformatio is decetralized, ad thus, there is a certai delay associated with gatherig the ecessary iformatio from the various etwor odes. Hece, a effective solutio for multi-hop cogitive radio etwors will eed to tradeoff the value of havig iformatio about other odes versus the trasmissio overheads associated with gatherig this iformatio i a timely fashio across differet hops, i terms the utility impact. this paper, we aim at learig the behaviors of iteractig cogitive radio odes that use simple iterferece graph (similar to the spectrum holes used i [6][8]) to sequetially adjust ad optimize their trasmissio strategies. We apply a multi-aget learig algorithm the fictitious play (FP) [15] to model the behavior of eighbor odes based o the iformatio exchage amog the etwor odes. We focus o delay-sesitive applicatios such as real-time multimedia streamig, i.e. the receivig users eed to get the trasmitted iformatio withi a certai delay. Due to the iformatioally decetralized ature of the multi-hop wireless etwors, a cetralized resource maagemet solutio for these delay-costraied applicatios is ot practical [14], sice the tolerable delay does ot allow propagatig iformatio bac ad forth throughout the etwor to a cetralized decisio maer. Moreover, the complexity ad the iformatio overhead of the cetralized optimizatio grow expoetially with the size of the etwor. The problem is further complicated by the dyamic competitio for wireless resources (spectrum) amog the various wireless odes (i.e. source odes/relays). The cetralized optimizatio will require a large amout of time to process ad the collected iformatio will o loger be accurate by the time trasmissio decisios eed to be made. Hece, a distributed resource maagemet solutio, which explicitly cosiders the availability of iformatio, the trasmissio overheads ad icurred delays, as well as the value of this iformatio i terms of the utility impact is ecessary. The paper is orgaized as follows. Sectio, we discuss the mai challeges of the dyamic resource maagemet i multi-hop cogitive radio etwors ad the related wors. Sectio provides the multi-hop cogitive radio etwor settigs ad strategies ad Sectio V gives problem formulatio of the distributed resource maagemet for delay sesitive trasmissio i such etwors. Sectio V, we determie how to quatify the rewards ad costs associated with various iformatio exchages i the multi-hop cogitive radio

4 4 etwors. Sectio V, we propose our distributed resource maagemet algorithms with the iformatio exchage ad itroduce the adopted multi-aget learig approach adaptive fictitious play i the proposed algorithms. Simulatio results are i Sectio V. Fially, Sectio V cocludes the paper.. MAN CHALLENGES AND RELATED WORKS A. Mai challeges i multi-hop cogitive radio etwors To desig such a distributed resource maagemet i multi-hop cogitive radio etwors, several mai challeges eed to be addressed: Dyamic adaptatio to a time-varyig etwor eviromet Multi-hop cogitive radio etwors are geerally experiecig the followig dyamics: 1) the primary users directly affect the spectrum opportuities available for the secodary users, 2) the mobility of the etwor relays that affects the etwor topology, 3) the traffic load variatio due to multiple applicatios simultaeously sharig the same etwor ifrastructure, ad 4) the time-varyig wireless chael coditios. Give the dyamic ature of the cogitive radio etwors, wireless odes eed to lear, dyamically self-orgaize ad strategically adapt their trasmissio strategies to the available resources without iterferig the primary licesees. Due to these time-varyig dyamics, the outcomes of these iteractios do ot eed to coverge to a equilibrium, i.e., disequilibrium ad perpetual adaptatio of strategies may persist, as log as the performace of the delay sesitive applicatio is maximized [15]. Hece, repeated iformatio exchage amog etwor odes is required for odes to efficietly lear ad eep adaptig to the chagig etwor dyamics. formatio availability i multi-hop ifrastructures Due to the iformatioally-decetralized ature of the multi-hop ifrastructure, the exchaged etwor iformatio is oly useful whe it ca be coveyed i time. The timeliess costrait of the iformatio exchage depeds o the delay deadlie of the applicatios, the iformatio overhead, ad the coditio of the etwor lis, etc. Hece, the value of iformatio i terms of its impact o the users utilities will eed to be quatified for the differet settigs of the multi-hop cogitive radio etwor. This iformatio will impact the accuracy with which the wireless odes ca model the behavior of other odes (icludig the primary users) ad hece, the efficiecy with which they ca respod to this eviromet by adequately optimizig their trasmissio strategies.

5 5 B. Related wors Distributed dyamic spectrum allocatio is a importat issue i cogitive radio etwors. Various approaches have bee proposed i recet years. [8], a decetralized cogitive MAC protocols are proposed based o the theory of Partially Observable Marov Decisio Process (POMDP), where a secodary user is able to model the primary users through Marovia state trasitio probabilities. [9], the authors ivestigated a game-theoretic spectrum sharig approach, where the primary users are willig to share spectrum ad provide a determied pricig fuctio to the secodary users. [10], a o-regret learig approach is proposed for dyamic spectrum access i cogitive radio etwors. However, these studies focus o dyamic spectrum maagemet for the sigle-hop etwor case. Exploitig frequecy diversity i wireless multi-hop etwors has attracted eormous iterests i recet years. [11], the authors propose a distributed allocatio scheme of sub-carriers ad power levels i a orthogoal frequecy-divisio multiple-access-based (OFDMA) wireless mesh etwors. They proposed a fair schedulig scheme that hierarchically decouples the sub-carrier ad power allocatio problem based o the limited local iformatio that is available at each ode. [12], the authors focus o the distributed chael ad routig assigmet i heterogeeous multi-radio, multi-chael, multi-hop wireless etwors. The proposed protocol coordiates the chael ad route selectio at each ode, based o the iformatio exchaged amog two-hop eighbor odes. However, these studies are ot suitable for cogitive radio etwors, sice they igore the dyamic ature of spectrum opportuities ad users (etwor odes) eed to estimate the behavior of the primary users for coexistece. To the best of our owledge, the dyamic resource maagemet problem i multi-hop cogitive radio etwors has ot bee addressed i literature. summary, the paper maes the followig cotributios. a) We propose a dyamic resource maagemet scheme i multi-hop cogitive radio etwor settigs based o periodic iformatio exchage amog etwor odes. Our approach allows each etwor odes (secodary users ad relays) to exchage their spectrum opportuity iformatio ad select the optimal chael ad ext relay to trasmit delay sesitive pacets. b) We ivestigate the impact of the iformatio exchage collected from various hops o the performace of the distributed resource maagemet scheme. We itroduce the otio of a iformatio cell to explicitly idetify

6 6 the etwor odes that ca covey timely iformatio. mportatly, we ivestigate the case that the iformatio cell does ot cover all the iterferig eighbor odes i the iterferece graph. c) The proposed dyamic resource maagemet algorithm applies FP [15], which allows various odes to lear their spectrum opportuity from the iformatio exchage ad adapt their trasmissio strategies autoomously, i a distributed maer. Moreover, we discuss the tradeoffs betwee the cost of the required iformatio exchage ad the learig efficiecy of the multi-aget learig approach i terms of the utility impact. Next, we preset our etwor settigs of the multi-hop cogitive radio etwors.. MULT-HOP COGNTVE RADO NETWORKS - SETTNGS AND STRATEGES A. Networ etities this paper, we assume that a multi-hop cogitive radio etwor ivolves the followig etwor etities ad their iteractios: Primary Users (PUs) are the icumbet devices that possess trasmissio liceses for specific frequecy bads (chaels). Without loss of geerality, we assume that there are M frequecy chaels i the cosidered cogitive radio etwor. We also assume that the maximum umber of primary users that ca be preset i the etwor equals M. Note that these primary users ca oly occupy their assiged (licesed) frequecy chaels ad ot other primary users chaels. Sice the primary users are licesed users, they will be guarateed a iterferece-free eviromet [2][4]. Whe a primary user is ot trasmittig data usig its assiged frequecy chael, a spectrum hole is formed at the correspodig frequecy chael. Secodary Users (SUs) are the autoomous wireless statios that perform chael sesig ad access the existig spectrum holes i order to trasmit their data. The secodary users ca occupy the spectrum holes available i the various frequecy chaels. this paper, the secodary users are deployig delay sesitive applicatios. Specifically, we assume that there are V delay sesitive applicatios simultaeously sharig the cogitive radio etwor ifrastructure, havig uique source ad destiatio odes. These secodary users are able to deploy their applicatios across various frequecy chaels ad routes. Networ Relays (NRs) are autoomous wireless odes that perform chael sesig ad access the existig spectrum holes i order to relay the received data to oe of its eighborig odes or SUs. Hece, ulie i the SUs case, there is o source or destiatio preset at the NRs. Note that multiple applicatios ca use

7 7 the same NR usig differet frequecy chaels. B. Source traffic characteristics Let V i deote the delay sesitive applicatio of the i -th SU. Assume that the applicatio V i cosists of pacets i K i priority classes. The total umber of applicatios is V. We assume that there are a total of K V = Ki + 1 priority classes (i.e., C = { C 1,, CK }). The reaso for addig a additioal priority class is i= 1 because the highest priority class C 1 is reserved for the traffic of the primary users. The rest of the classes C, > 1 ca be characterized by: λ, the impact factor of a class C. For example, this factor ca be obtaied based o the moey paid by a user (differet service levels ca be assiged for differet SUs by the cogitive radio etwor), based o the distortio impact experieced by the applicatio of each SU or based o the tolerated delay assiged by the applicatios. The classes of the delay sesitive applicatios are the prioritized based o this impact factor, such that λ λ if < ', = 2,..., K. The impact factor is ecapsulated i the header (e.g. RTP header) ' of each pacet. D, the delay deadlie of the pacets i a class sesitive applicatios oly whe it is received before its delay deadlie. L, the average pacet legth i the class C. C. this paper, a pacet is regarded useful for the delay A variety of delay sesitive applicatios ca use the cogitive radio set-up discussed i this paper. Multimedia trasmissio such as video streamig or video coferecig ca be examples of such applicatios [14]. We assume i this paper that a applicatio layer scheduler is implemeted at each etwor ode to sed the most importat pacet first based o the impact factor ecapsulated i the pacet header. C. Multi-hop cogitive radio etwor specificatio We cosider a multi-hop cogitive radio etwor, which is characterized by a geeral topology graph G ( MNE,, ) that has a set of primary users M = { m1,..., m M }, a set of etwor odes N = { 1,..., N } (iclude SUs ad NRs) ad a set of etwor edges (lis) E = { e1,..., e L } (coectig the SUs ad NRs). There are a total of N odes ad L lis i this etwor. Each of these N etwor odes is either a secodary user (as a source or a destiatio ode) or a etwor relay.

8 We assume that F = { f1,..., f M } is the set of frequecy chaels i the etwor, where M is the total umber of the frequecy chaels. To avoid iterferece to the primary users, the etwor odes ca oly use spectrum holes for trasmissio. Hece, to establish a li with its eighbor odes, each etwor ode N ca oly use the available frequecy chaels i a set F F. Note that these wireless odes i a cogitive 8 radio etwor will cotiuously sese the eviromet ad exchage iformatio ad hece, F may chage over time depedig o whether the primary users are trasmittig i their assiged frequecy chaels. The etwor resource for a etwor ode N of the multi-hop cogitive radio etwor icludes the routes composed by the various lis ad frequecy chaels. We defie the resource matrix [ ] {0,1} L R = R M for the etwor ode as follows: ij R ij 1, if li ei is coected to the ode = ad the frequecy chael fj is available. 0, otherwise. (1) Whether or ot the resource R ij is available to ode N depeds ot oly o the topology coectivity, but also o the iterferece from other traffic usig the same frequecy chael. Next, we discuss the iterferece from other users (icludig the primary users). D. terferece characterizatio Recall that the highest priority class C 1 is always reserved i each frequecy chael for the traffic of the primary users. The traffic of the SUs ca be categorized ito K 1 priority classes ( C2,..., C K ) for accessig frequecy chaels. The traffic priority determies its ability of accessig the frequecy chael. Primary users i the highest priority class C 1 ca always access their correspodig chaels at ay time. The traffic of the SUs ca oly access the spectrum holes for trasmissio. Hece, we defie two types of iterferece to the secodary users i the cosidered multi-hop cogitive radio etwor: 1) terferece from primary users. practical cogitive etwors, eve though primary users have the highest priority, secodary users will cause some level of iterferece to the primary users due to their imperfect awareess (sesig) of the primary users. The primary users iterferece depeds o the locatio of the M primary users. We rely o methods such as i [5] that cosider the power ad locatio of the secodary users to esure that the

9 9 secodary users do ot exceed some critical iterferece level to the primary users. We also assume that the spectrum opportuity map is available to the secodary users as i [6][10]. Sice the primary users will bloc all the eighbor lis usig its frequecy chael, a etwor ode will sese the chael ad obtai the Spectrum Opportuity Matrix (SOM) of the primary users: Z 1, if the primary user is occupyig frequecy chael f L M = [ Z ] {0,1}, with Z = ad the li ei ca iterfere with the primary user. 0, otherwise. ij ij j (2) A simple example is illustrated i Figure 1, which idicates the SOM of the primary users ad the resource matrix of each etwor ode i the multi-hop cogitive radio etwor. F = { f, f } 1 2 N = {,, } E = { e, e, e } e 1 2 m 1 e 3 f 1 m 2 1 e 3 2 Fig. 1. A simple multi-hop cogitive radio etwor with three odes ad two frequecy chaels. 2) terferece from competig secodary users. We defie [ ] {0,1} L = M as the terferece Matrix (M) for the traffic i priority class C, 2. Spectrum opportuity matrix of the primary users: f 1 f 2 e Z 2 = e e ij Resource matrix at each ode: R f 1 f 2 e = e 1 1 R e f 1 f 2 e = e 0 0 R e f 1 f 2 e = e 1 1 e ij 1, if li ei usig frequecy chael fj ca be = iterfered by the traffic of priority class C. 0, otherwise. (3) The iterferece caused by the traffic i priority class C ca be determied based o the iterferece graph of the odes that trasmit the traffic (as i [10]). The iterferece graph is defied as the correspodig lis that are iterfered by the trasmissio of the class C traffic 2. The M ca be computed by the iformatio 2 a wireless eviromet, the trasmissio of eighbor lis ca iterfere with each other ad sigificatly impact their effective trasmissio time. Hece, the actio of a ode ca impact ad be impact by the actio of the other relay odes. order to coordiate these eighborig odes, we costruct the iterferece matrix with biary 1 ad 0.

10 10 exchage amog the eighbor odes. The available resource matrix ca be mased out by the SOM ad M of the higher priority classes, i.e. ( ) R = R 1... Z, where the otatio represets elemet-wise multiplicatio of the matrixes ad deotes the iverse operatio, which turs 1 ito 0 ad 0 ito 1. The resultig resource matrix R ( ) represets the available resource aroud the etwor ode for the class C traffic uder the iterferece of other higher priority traffic (classes). Next, we defie the actios available to the etwor odes i a multi-hop cogitive radio etwor. E. Nodes actios We defie the actio of the etwor ode i order to relay the delay sesitive applicatio V i as A = ( e E, f F ). We assume that a etwor relay ca select a set of lis to its eighbor odes (lis coected to ode ) E E. Correspodig to the actios, we defie the trasmissio strategy vector of the etwor ode as s = [ sa A = ( e E, f F )], where s A represet the probability that the etwor ode will choose a actio A. We refer to a actio at a ode as a feasible actio for trasmittig a class C traffic, if A = (, e f) is a available resource i ( ) R (i.e. elemet R = 1 i R ( ) ), sice i this ef case the selected li ad frequecy chael do ot iterfere with the traffic i the higher priority classes. That is, ˆ () { (, ) ( ) [ ] L M = A = e f = Ref, Ref = 1} A R. (4) We deote the set of all the feasible actios for ode as A ˆ () for class C traffic. We ext determie the correspodig delay based o differet actios, which cosiders the deployed cross-layer trasmissio strategies i order to compute the Effective Trasmissio Time (ETT) [19] over the trasmissio lis. Each etwor ode computes the ETT ETT ( e, f ), with e E, f F for trasmittig delay sesitive applicatios i priority class C : ETT L (, e f ) =. (5) T (, e f) (1 p (, e f)) T( e, f ) ad p (, e f ) represet the trasmissio rate ad the pacet error rate of the etwor ode usig the frequecy chael f over the li e. T (, e f ) ad p (, e f ) ca be estimated by the MAC/PHY layer li

11 11 adaptatio [20]. Specifically, we assume that the chael coditio of each li-frequecy chael pair ca be modeled usig a cotiuous-time Marov chai [17] with a fiite umber of states S (, ). The time a chael coditio speds i state i S (, ) is expoetially distributed with parameter ν i (rate of trasitio at state i ef ef i trasitios/sec). We assume that the maximum trasitio rate 3 of the etwor is ν ad the variatio of the chael coditios i a time iterval τ 1/ ν is regarded egligible. Defie the actio vector A = [ A σ ] as the vector of the actios of all the etwor relay odes for trasmittig i i V i. Assume that the i th delay sesitive applicatio V i are trasmitted from the source ode s i N to the destiatio ode i d N with a total of q i pacets. The routes of V i are deoted as σ i = { σij j = 1,..., qi}, where σ ij is the route of the j th pacet i pairs that the pacets flow through, i.e. V i. A route σ ij is a set of li-frequecy σ = {( ef, ) the jth pacet of V flows through li eusig frequecy chael f}. (6) ij i Note that if the actio of a certai relay ode chages, the correspodig route σij( A i ) of relayig V i also chages. We deote the ed-to-ed delay of the pacets trasmitted usig the route σ ( A ) as d ( σ ( A )). ij i ij ij i Based o the topology, each etwor relay ode receivig a pacet ca decide where to relay the pacet to ad usig which frequecy chael, i order to miimize its ed-to-ed delay dij ( σij ( A i )). Fially, to calculate dij ( σij ( A i )), the source ode eed to obtai the delay iformatio from other odes accordig to the actios tae by the relay odes, i.e. d ( σ ( A )) = ETT ( A ), for V C. (7) ij ij i i i σij V. RESOURCE MANAGEMENT PROBLEM FORMULATON OVER MULT-HOP COGNTVE RADO NETWORKS By examiig the cumulated ETT values, the objective of a delay sesitive applicatio is to miimize its ow ed-to-ed pacet delay. The cetralized ad proposed distributed problem formulatios are subsequetly 3 case that some of the chael coditios chage severely i the etwor, a threshold th ν ca be set by protocols to avoid these fast-chagig odes ad the ν is hece selected as the maximum trasitio rate below this threshold value.

12 12 provided. Cetralized problem formulatio with global iformatio available at the sources f we assume that the global iformatio 4 G i is available to the source ode i s for the delay sesitive applicatio V i, the route σij ( Ai, G i ) ca be determied for each pacet j of V i. The cetralized optimizatio ca be performed at every source ode i order to maximize the utility u i. Hece, for applicatio V i we have: A opt i = arg max u ( A, G ) i i i subject to A Aˆ, (8) for all A A i where q i u ( A, G ) = λ Prob{ d ( σ ( A, G )) D }, Dij = D ad λij = λ if j C. (9) i i i ij ij ij i i ij j = 1 However, due to the limited wireless etwor resource, the ed-to-ed delay costrait d ( σ ( A, G )) D ca ij ij i i mae the optimizatio solutio ifeasible. Hece, a sub-optimal greedy algorithms that perform optimizatios sequetially from the highest priority class to the lowest priority class are commoly adopted [25][14]. Specifically, for class C, the followig optimizatio is cosidered: A opt i = arg mi d ( σ ( A, G )) j C ij ij i i subject to d ( σ ( A, G )) D,, (10) ij ij i i A Aˆ for all A A. i where A = [ A σ, j C ]. i ij Due to the iformatioally decetralized ature of the multi-hop wireless etwors, the cetralized solutio is ot practical for the multi-user delay sesitive applicatios, as the tolerable delay does ot allow propagatig the global iformatio G i bac ad forth throughout the etwor to a cetralized decisio maer. For istace, the optimal solutio depeds o the delay d ij icurred by the various pacets across the hops, which caot be timely relayed to a source ode. For istace, whe the etwor eviromet is time-varyig, the gathered global iformatio G i ca be iaccurate due to the propagatio delay for this iformatio. Moreover, the complexity of the cetralized optimizatio grows expoetially with the umber of classes ad odes i the etwor. The problem is further complicated by the dyamic adaptatio of the trasmissio strategies deployed 4 The word global iformatio meas the iformatio gathered from every ode throughout the etwor. We discuss the required iformatio i Sectio V.

13 13 by the wireless odes, which impacts their spectrum access ad hece, implicitly, the performace of their eighbor odes. The optimizatio will require a large amout of time to process ad the collected iformatio might o loger be accurate by the time trasmissio decisios eed to be made. summary, i the studied dyamic cogitive radio etwor, the decisios o how to adapt the aforemetioed actios at sources ad relays eed to be performed i a distributed maer due to these iformatioal costraits. Hece, a decompositio of the optimizatio problem ito distributed strategic adaptatio based o the available local iformatio is ecessary. Proposed distributed problem formulatio with local iformatio at each ode: stead of gatherig the etire global iformatio G i at each source, we propose a distributed suboptimal solutio that collects the local iformatio L at ode to miimize the expected delay of the various applicatios sharig the same multi-hop wireless ifrastructure. Note that at each ode, the ed-to-ed delay for sedig a pacet j C i equatio (10) ca be decomposed as: P d ( σ ) = d ( σ ) + E[ d (, σ )], (11) ij ij ij ij P where d ( σ ) represets the past delay that pacet j has experieced before it arrives at ode ad ij Ed [ (, σij)] represets the expected delay from the ode to the destiatio of the pacet j C. The sedig pacet j C is determied by the applicatio layer scheduler accordig to the impact factor λ. The P iformatio about λ ca be ecapsulated i the pacet header ad d ( σ ij) ca be calculated based o the timestamp available i the pacet header. The priority scheduler at each ode esures that the higher priority classes are ot iflueced by the lower priority classes (see equatio (10)). Sice at the ode the value of d P ( σ ) is fixed, the optimizatio problem at the ode becomes: ij opt A = arg mi E[ d (, σij( A, L))] P subject to Ed [ (, σij( A, L ))] D d ( σij) ρ, j C, (12) A Aˆ where Ed [ (, σ ( A, L ))] represets the expected delay from the relay ode to the destiatio of the ij pacets i class C. ρ represets a guard iterval such that the probability Prob{ Ed [ (, σ ( A, L ))] + d P ( σ ) > D } is small (as i [22]). To estimate the expected delay ij ij

14 Ed [ (, σ ( A, L ))] i equatio (12), each etwor ode maitais a estimated trasmissio delay ij Ed [ ( )] from itself to the destiatio for each class of traffic usig the Bellma-Ford shortest-delay routig algorithm [17]. We assume that each ode maitais ad updates a delay vector d = [ Ed [ (2)],..., Ed [ ( K)]] (ote that the first priority class is reserved for the primary users) with elemets for each priority class. Each 14 etwor ode exchages such iformatio to its eighbor odes ad selects the best actio opt A for the highest priority pacet i the buffer of the etwor ode. We will discuss the miimum-delay routig/chael selectig algorithm i Sectio V. Note that a group of pacets i the buffer of a ode ca tae the actio A, sice the actio is determied based o local iformatio L. Sice i the cogitive radio etwors, the available chael is time-variat, the iformatio eeds to be timely coveyed to the etwor ode for the distributed optimizatio. Compared to the cetralized approach i equatio (8), the distributed resource maagemet i equatio (12) ca adapt better to the dyamic wireless eviromet by periodically gatherig local iformatio. Next, we discuss the distributed resource maagemet with iformatio costraits i more detail. V. DSTRBUTED RESOURCE MANAGEMENT WTH NFORMATON CONSTRANTS A. Cosidered medium access cotrol this paper, we assume that the required local iformatio L is exchaged usig a desigated coordiatio cotrol chael similar to [13]. Such a coordiatio chael ca be selected from the existig SM bads, sice there is o primary licesee i these bads to iterfere with. The trasmissio is time slotted ad the time slot structure of a ode is provided i Figure 2. We deote the time slot duratio as t. The actio are selected at each ode, durig each time slot, after the coordiatio iterval (that icludes the chael sesig for SOM ad the iformatio exchage for M). We deote the coordiatio iterval at the etwor ode as d( L ). The goal of the coordiatio iterval at each time slot is to provide the feasible actio set A ˆ for the chael access ad the relay selectio of the pacet trasmissio. We will discuss how to obtai A ˆ based o the SOM ad the M amog the eighborig odes whe we itroduce the proposed algorithm, i Sectio V. A

15 15 Coordiatio iterval Pacets trasmissio d ( L ) Decisio maig d ( L ) Chael sesig Time slot t Fig. 2. Trasmissio time lie at the ode with local iformatio L. Besides the SOM ad M, the iformatio required i the coordiatio iterval should also iclude the delay vectors d ad the cotrol messages for RTS/CTS coordiatio [8][12]. Note that the local iformatio L does ot eed to iclude all these iformatio i each time slot (except the cotrol messages). For example, the SOM ad M ca be collected i a differet period, depedig o the sesig ad iformatio exchage mechaism. Hece, the coordiatio duratio d ( L ) will vary for differet time slots, which will be discussed i more detail i Sectio V.C. Next, we ivestigate the beefit of acquirig iformatio from differet h -hop eighbor odes, which also affects the duratio of the coordiatio iterval d ( L ). B. Beefit of acquirig iformatio ad iformatio costraits For the etwor ode, the local iformatio L gathered from differet etwor odes has differet impact o decreasig the objective fuctio Ed [ (, σ ( A, L ))] i equatio (12). Let ij () x = { (, A ), A, d N } deote the set of local iformatio gathered from the eighbor odes, x x x x x x which is x hops away from ode, where N x represets a set of odes that is x hops away from ode. We defie L ( x) = { ( l) l = 1,..., x} as the local iformatio gathered from all of these eighbor odes. Give the local iformatio L ( x), we defie the optimal expected delay as K(, x) = E[ d (, σij( A opt, L ( x)))]. The larger x will has a smaller expected delay K(, x ). The beefit (reward) of the iformatio ( x) for the class C traffic is deoted as J (, ()) x. a static etwor case, J (, ()) x is defied as: J (, ()) x K (, x 1) K (, x), if x > 1. (13) We defie J (, (1)) = K (,1) sice L (1) = (1). The reward of iformatio J (, ()) x ca be regarded as the beefit (decrease of the expected delay) i terms of the expected delay Ed [ (, σ )] if the iformatio ( x) is received by ode. Note that the optimal expected delay K(, x ), give the ij

16 16 iformatio L ( x) : x K (, x) = K (,1) J (, ()) l. (14) l = 2 Equatio (14) states that the optimal expected delay is a decreasig fuctio of x, meaig that smaller expected delays ca be achieved as more iformatio is gathered. The improvemet is quatified by the reward of the iformatio J (, ()) l. Here, we igore the cost of exchagig such iformatio, which will be defied i the ext subsectio. Figure 3 shows a simple illustrative example of reward of iformatio at ode, which is five hops away from the destiatio ode of class C traffic. The more iformatio () available from x odes that is x hops away, the smaller optimal expected delay K (, x ) ca be obtaied. 200 K (, x ) (msec) Static J (, ()) x (msec) Dyamic 200 d J (, ()) x (msec) ( x ) ( x ) x = 1 5 (1) 1 (2) L(3) L x = 3 x = 4 x = (3) (4) 4 d (5) ( x ) L ( x ) = { ( l ) l = 1,.., x } ( x ) = { (, A ), A, d } x x x x Fig. 3. Example of the static reward of iformatio J(, ()) x, dyamic reward of iformatio J(, ()) x ad optimal expected delay K(, x ) (where the iformatio horizo h (, ν ) = 3, average pacet legth L =1000 bytes, ad average trasmissio rate T = 6Mbps over the multi-hop etwor). Let J() = [ J(, ()), x for 1 x H] deote the reward vector from 1 -hop iformatio to H -hop d d iformatio, where H = max{ H, H}. H represets the shortest hop couts from the ode to the destiatio ode of the class C traffic ad H represets the iterferece rage i terms of hop couts for d ode. We also eed to cosider the hop cout H i case that the destiatio ode is close to the ode withi the iterferece rage. We assume that the reward vector J () is obtaied whe the etwor is first deployed ad oly updated ifrequetly, whe SUs joi or leave the etwor. Note that all the elemets i J () are oegative, i.e. J (, ()) x 0, for 1 x H, due to the fact that owig additioal iformatio caot icrease the expected delay Ed [ (, σ )] i a static etwor. However, if we cosider the ij

17 propagatio delay of such iformatio exchage across the etwor i the dyamic etwor, the dyamic d reward of iformatio J (, ()) x decreases as the hop cout x icreases. Whe the iformatio of the further odes reaches the decisio ode, the iformatio is more liely to be out-of-date (i.e. the iformatio caot reflect the exact etwor situatio i a dyamic settig, sice the etwor coditios ad traffic d characteristics are time-varyig). Oce the iformatio is out-of-date, J (, ()) x = 0, i.e. there is o beefit d from gatherig iformatio that is out-of-date. Note that i a dyamic etwor, oce J (, ()) x = 0, J d (, ( x ')) = 0 for x x' H. Therefore, i the dyamic etwor, we defie the iformatio horizo hν (, ) such that 17 h (, ν) arg max x d subject to J (, ( x)) > φ(, ν),1 x H. (15) where φ(, ν) 0 represets a miimum delay variatio specified by the applicatio which determies the miimum beefit of receivig local iformatio for class C traffic. fact, h (, ν ) depeds o the variatio speed ν of the wireless etwor coditio (i.e. the trasitio rate of the Marovia chael coditio model, see Sectio.E). a dyamic etwor with higher variatio speeds ν (e.g. with high mobility), a higher threshold φ(, ν ) is eeded to guaratee that the iformatio () is still valuable ad it should be exchaged. This results i a smaller iformatio horizo h (, ν ). We illustrate this mobility issue i Sectio V. Note that the iformatio horizo h (, ν ) varies for differet classes of traffic at differet locatios i the etwor. Sice higher priority class traffic has more etwor resources tha the lower priority class (i.e. they are scheduled first for optimizatio i equatio (12)), the threshold value φ(, ν) φ( ', ν), if < ' ad thereby, h(, ν) h( ', ν), if < '. other words, the iformatio horizo h (, ν ) of a higher priority class C is larger tha the iformatio horizo h ( ', ν ) of a lower priority class C '. Although the iformatio horizo h (, ν ) ca vary at differet locatios for differet priority classes depedig o the applicatios, the complexity of such implemetatio is high ad the adaptatio of the iformatio horizo itself ca be a iterestig topic. Hece, we will leave the iformatio horizo adaptatio problem to our future research. For simplicity, we assume i this paper that the iformatio horizo is oly a fuctio of the etwor variatio speed ν, i.e. h (, ν) = h() ν. The iformatio horizo h( ν ) is determied x

18 for the most importat class amog the SUs i the etwor. This defiitio of the iformatio horizo h( ν ) is aliged with [14], i which h( ν ) is defied as the maximum umber of hops that the iformatio ca be coveyed i τ, such that the etwor is cosidered uchaged (recall that ay etwor chages withi the iterval τν ( ) 1/ ν ca be regarded egligible). Based o this iformatio horizo h( ν ), we assume that the etwor odes withi the h( ν ) hops form a iformatio cell. Oly the local iformatio L ( h) withi the iformatio cell is useful to the ode, sice the reward of iformatio is zero, i.e. J (, ()) x = 0 for x > h( ν). the dyamic etwor, etwor ode determies its actio at time slot t based o the acquired iformatio at the previous time slot t 1. The optimizatio problem i equatio (12) ca be writte as: opt A ( t) = arg mi E[ d (, σij( A, L( h, t - 1)))] P subject to Ed [ (, σij( A, L ( ht, - 1)))] D d ( σij) ρ, j C. (16) A Aˆ ( t 1) Recall that the eighbor odes of the ode are defied as the odes that ca iterfere or ca be iterfered by 18 the ode (withi H hops), which may ot alig with the rage of the iformatio cell (withi h( ν ) hops). f all eighbor odes are withi the h -hop iformatio cell, all ecessary iformatio are timely coveyed to the ode. Otherwise, the eighbor odes that are too far away caot covey the iterferece iformatio to the ode i time. Sice the required iformatio caot be acquired i time, the solutio i equatio (16) becomes suboptimal. We refer to this problem as iformatio exchage mismatch problem. Figure 4 illustrates two simple etwor examples with ad without the mismatch problem. Note that i Figure 4(b), sice the iformatio cell does ot cover all the iterferig eighbor odes, the ceter ode 2 will still be iterfered by other secodary users. fact, due to the ature of the multi-hop wireless eviromet, the etwor odes that are far away from the ode have limited iterferece impact o ode 2. Hece, eve though the iformatio horizo h does ot match the iterferece rage, the performace degradatio of the optimizatio problem i equatio (16) usig the local iformatio L ( h) is limited.

19 19 terferece rage of 2 formatio horizo (a) (b) 1 m 1 A, ( 1 1), E[ d ( 1)] 1 m 1 A, ( 1 1), E[ d ( 1)] 2 A, 4 ( 4), E[ d ( 4)] A, 5 ( 5), E[ d ( 5)] 4 5 A, ( 3 3), E[ d ( 3)] A, 4 ( 4), E[ d ( 4)] A, ( 3 3), E[ d ( 3)] 4 A, ( 6 6), E[ d ( 6)] 6 6 Fig 4. (a) 2-hop iformatio cell etwor without iformatio exchage mismatch problem. (b) 1-hop iformatio cell etwor with iformatio exchage mismatch problem. C. Cost of iformatio exchage the previous subsectio, we discuss the reward of iformatio i a h -hop iformatio cell while igorig the egative impact of the iformatio exchage. this sectio, we discuss the cost (icrease of the expected delay) due to this iformatio exchage. Recall that the duratio of the time slot is t ( ν ), which is also the iterval betwee the repeated iformatio exchages i the etwor. We defie there are c time slots i τ secods, i.e. t τν ( ) ( ν ) =. (17) c c defies the frequecy of the decisio maig as well as the learig process, which will be discussed i detail i Sectio V. Note that decisios ca be made every t ad this time slot duratio is short eough compared to τ. Hece, the etwor chages i t is also egligible. Recall that the coordiatio duratio i a time slot for the etwor ode is d ( L ( h)). Assume the iformatio uit for the required iformatio is U ( ), ( A) d U, ad U ( ) per class, respectively. Assume the average umber of odes i a h -hop iformatio cell is Nh (). The iformatio time overhead of L ( ) is o h average ( d) ( ) ( A) d ( L ( h)) = N( h)[( K 1)( U + U ) + U ]. Note that eve though the iformatio exchage is implemeted i a desigated coordiatio chael [13], a etwor ode with a sigle atea caot trasmit both the data ad the cotrol sigals at the same time. This

20 iformatio exchage time overhead decreases the effective trasmissio rate at ode usig the lie e ad frequecy chael f : t( ν) d( L( h)) T (, e f) = T(, e f). (18) t ( ν) Hece, the effective trasmissio time at a ode usig the li e ad frequecy chael f to trasmit a pacet i class C becomes: t ( ν) ETT (, e f ) = ETT (, e f ). (19) t ( ν) d ( L ( h)) coclusio, the icrease of the effective trasmissio time degrades the performace of the delay sesitive applicatios. The degradatio depeds o the cotet of the local iformatio exchage L ( h), ad the etwor d variatio speed ν. Hece, the beefit J (, ()) x i equatio (15) will decrease due to this cost of the c iformatio. Hece, we deote the value of iformatio with this cost cosideratio as J (, ()) x : c J(, ()) x = K (, x 1) K (, x) t( ν) t( ν). (20) = K(, x 1) K(, x) t ( ν) d ( L ( x 1)) t ( ν) d ( L ( x)) Ad the optimal iformatio horizo h (, ν ) i equatio (15) also decreases due to the cost. Next, we discuss the proposed distributed resource maagemet algorithm based o the iformatio exchages ad learig capabilities to tacle the optimizatio problem i equatio (16). 20 V. DSTRBUTED RESOURCE MANAGEMENT ALGORTHMS Figure 5 provides a system diagram of the proposed distributed resource maagemet. First, a pacet j C is selected from the applicatio scheduler at the ode based o the impact factor λ of the pacet ad a actio A is tae for that pacet. The applicatio layer iformatio icludig C, L, D is coveyed to the etwor layer for this actio decisio. Networ coditios T (, e f), p (, e f ) are the coveyed from the MAC/PHY layer for computig the ETT values usig equatio (5). additio to the T (, e f), p (, e f ), the actio selectio is impacted by the iterferece iduced from the actio of these eighbor odes ad hece, the iformatio received from the eighbor odes i the iformatio cell. Recall that L ( h) = { ( l) l = 1,..., h}. We use the otatio h ( ) to represet the set of the eighbor odes

21 of the etwor ode i the h -hop iformatio cell. Hece, the local iformatio exchaged L ( h) = { ( ( h), A ), A, d } across the etwor odes is required. Hece, the ode ows the h ( ) h ( ) h ( ) 21 estimated delay d ( h) from its eighbor odes to the destiatios, so as the actios A h ( ) of its eighbor odes ad their M ( ( h), A ( h) ). Based o the delay iformatio from the eighbor odes d ( h), a etwor ode ca update its ow estimated delay to the various destiatios ad determie the miimum-delay actio based o Bellma-Ford algorithm [17]. L ( h ) = { ( A, ), A, E [ d ( )]} Upstream ode Node pacets formatio exchage iterface Data trasmissio ter ode iformatio exchage Cross-layer message passig Applicatio layer pacet schedulig A ( R ) ( ) Ed [ ( )] V C, L, D Networ layer miimumdelay route/chael selectio Pacet trasmissio Z Tef (, ), pef (, ), e E, f F i C A Dowstream ode MAC/PHY layer adaptatio ad chael sesig Fig. 5. System diagram of the proposed distributed resource maagemet. We separate the distributed resource maagemet ito two blocs at the ode as i Figure 5 the iformatio exchage iterface bloc that regularly collects required local iformatio ad the route/chael selectio bloc to determie the optimal actio. We ow discuss the role of the exchaged iformatio ad the two algorithms implemeted i these blocs, respectively. A. Distributed resource maagemet algorithms The ext algorithm is performed at etwor ode at the iformatio exchage iterface i Figure 5. Algorithm 1. Periodic iformatio exchage algorithm: Step 1. Collect the required iformatio the ode first collects the required iformatio the SOM Z from chael sesig ad L () h = { ( (), h A h ( )), A h ( ), d h ( )} from the eighbor odes i the iformatio cell. Step 2. Lear the behavior of the eighbor odes by cotiuously moitorig the actios of the eighbor odes, ode ca model the behavior of the eighbor odes or lear a better trasmissio strategy usig strategy vectors s( ) = [ s ( ) A = ( e E, f F )], ( h), where s ( ) represets the probability A A

22 22 (strategy) of selectig a actio A by the ode, which will be discussed i the ext subsectio. Step 3. Estimate the resource matrix from the SOM ad the M (, A ) gathered from the eighbor ode, the resource matrix ca be obtaied for each class of traffic by ( ) R = R 1... Z, which will be explaied i Sectio V.A i more details. The the available resource ( R ) ( ) are provided to the etwor A layer route/chael selectio bloc stated i the Algorithm 2. Step 4. Update iformatio { ( A, ), A, d } based o the recetly selected actio A, the latest delay vector d, ad the M ( A, ). Two types of iterferece model are cosidered i this paper whe costructig the M ( A, ) from equatio (3): 1) A etwor ode ca trasmit ad receive pacets at the same time Note that a ode caot reuse a frequecy chael f F used by its eighbor odes. f a frequecy chael is used by its eighbor odes, all the elemets i the colum of the iterferece (, A) that is associated with the frequecy chael are set to 1. The the M is exchaged to the odes withi the pre-determied iformatio horizo h. 2) A etwor ode caot trasmit ad receive pacets at the same time this case, if the frequecy chael f F is used, all the elemets i the colum of the M (, A) associated with the frequecy chael are set to 1. additio, if a etwor li e E is used by its eighbor odes, all the elemets of the M ( A, ) that is associated with the ode are also set to 1, o matter what frequecy chael it uses. The the M is exchaged to the odes withi the pre-determied iformatio horizo h. Step 5. Broadcast the iformatio { ( A, ), A, d } ad repeat the algorithm periodically i every t ( ν ) secods. The ext algorithm is performed at the etwor ode at the etwor layer miimum-delay route/chael selectio bloc i Figure 5. Algorithm 2. Miimum-delay route/chael selectio algorithm: Step 1. Determie the pacet to trasmit based o the impact factor, oe pacet j i the buffer at the ode is scheduled to be trasmitted. Assume the pacet j C, ad the iformatio of C, L, D P d are extracted or computed from the applicatio layer. Step 2. Costruct the feasible actio set costruct the feasible actio set A ˆ () from the resource matrix

23 ( ) R give from the iformatio exchage iterface for the priority class C at the ode (see equatio (4)). Step 3. Estimate the chael coditio the trasmissio rate T (, e f ) ad pacet error rate p (, e f ) for each li-frequecy chael pair ( e E, f F ) are provided from the PHY/MAC layer through li adaptatio [20]. Step 4. Calculate the expected delay toward the destiatio for each actio A A ˆ () of the traffic class C : Ed [ (, A )] = ETT ( A ) + Ed [ ()], for A Aˆ (), (21) '( A ) 23 where Ed [ '( )( )] represets the correspodig elemet for the class C i the delay vector A d from the eighbor ode '( A ). ETT ( A ) ca be calculated based o L, T (, e f ), ad p (, e f ) usig equatio (5). P Step 5. Chec the delay deadlie if Ed [ ( )] D d ρ, drop the pacet. P Step 6. Select the miimum delay actio if Ed [ ( )] < D d ρ, fid the miimum-delay route ad frequecy chael selectio, i.e. determie the optimal actio A from the feasible actio set A ˆ (). other opt words, the goal here is to solve equatio (16) at ode : A arg mi E[ d (, A )]. (22) opt = A ˆ A ( ) Note that the feasible actio set A ˆ () i equatio (22) depeds o the actios of other eighbor odes A. t is importat for the etwor odes to adopt learig approaches for modelig the behaviors of these etwor odes to decrease the complexity of the dyamic adaptatio. This will be discussed i the ext subsectio. Step 7. Sed RTS request after determiig the ext relay ad frequecy chael, sed RTS request idicatig the determied actio iformatio opt A to the ext relay. Step 8. Wait for CTS respose ad trasmit the pacets. Step 9. Update the delay ad the curret actio iformatio after selectig the optimal actio, update the estimated delay Ed [ ( )] usig expoetial movig average with a smoothig factor α : old opt Ed [ ( )] = α Ed [ ( )] + (1 α) Ed [ (, A )], (23) ad provide the updated delay vector d = [ Ed [ (2)],..., Ed [ ( K)]] to Algorithm 1 at the iformatio exchage iterface. Figure 6, we provide a bloc diagram of the proposed distributed resource maagemet. For the

24 blocs that beyod the scope of this paper, we refer to [4][5] for chael sesig, [8][12] for RTS/CTS 24 coordiatio, ad [17] for the delay vectors. : blocs that are ot covered i this paper Periodic iformatio exchage algorithm Miimum-delay route/chael selectio algorithm Chael sesig for primary users fo. exchage amog secodary users terferece matrix Delay vectors RTS/CTS coordiatio Z ( ), A Determiig resource matrix usig AFP ( ) d RTS ( A ) CTS ( A ) R Priority scheduled pacet buffer C Miimum-delay Select a feasible Route/chael actio that miimizes selectio Ed [ (, A )] RTS/CTS coordiatio A Pacet trasmissio { ( A, ), A, d } formatio update Fig. 6. Bloc diagram of the proposed distributed resource maagemet at etwor ode. B. Adaptive fictitious play (AFP) We ow provide a learig approach for the SUs to lear the feasible actio set A ˆ () i equatio (22) for our distributed resource maagemet algorithms. Specifically, based o the iformatio exchage L ( h), the behaviors of the eighbor odes i the iformatio cell ca be leared (Step 2 of Algorithm 1) ad based o the behaviors, the feasible actio set A ˆ () is determied. This motivates us to apply a well-ow learig approach fictitious play [15], applied whe the SUs are willig 5 to reveal their curret actio iformatio ad thereby, they are able to model the behaviors (strategies) of other SUs (a model-based learig [18]). However, due to the iformatio costrait discussed i the previous sectio, oly the iformatio from the eighbor odes i the iformatio cell is useful. Hece, we adapt the fictitious play learig approach to our cosidered etwor settig. Figure 7(a) provides a bloc diagram of the proposed distributed resource maagemet algorithm usig the adaptive fictitious play. Note that oly part of the SUs ca be modeled via the learig approach depedig o the iformatio horizo. Specifically, a ode maitais a strategy vector over time s(, t) = [ s (, t) A = ( e E, f F )] for each of its eighbor odes ( h) i the iformatio cell. A 5 f the actio iformatio is ot provided by the other secodary users, a ode ca lear its ow strategy from its actio payoffs the estimated delay Ed [ ( )]. The learig approach refers to the reiforcemet learig (a model-free learig [18] or a payoff-based learig).