Jamming-resistant Multi-radio Multi-channel Opportunistic Spectrum Access in Cognitive Radio Networks

Transcription

1 JOURNAL OF L A T E X CLASS FILES, VOL. 6, NO., JANUARY 27 Jammig-resistat Multi-radio Multi-chael Opportuistic Spectrum Access i Cogitive Radio Networks Qia Wag, Member, IEEE, Kui Re, Seior Member, IEEE, Peg Nig, Member, IEEE ad Shegsha Hu, Studet Member, IEEE Abstract For achievig optimized spectrum usage, most existig opportuistic spectrum sesig ad access protocols model the spectrum sesig ad access problem as a partially observed Markov decisio process (POMDP) by assumig that the iformatio states ad/or the primary users (PUs) traffic statistics are kow a priori to the secodary users (SUs). While theoretically soud, the existig solutios may ot be effective i practice due to two mai cocers. First, the assumptios are ot practical, as before the commuicatio starts, PUs traffic statistics may ot be readily available to the SUs. Secodly ad more seriously, existig approaches are extremely vulerable to malicious jammig attacks. By leveragig the same statistic iformatio ad stochastic dyamic decisio makig process that the SUs would follow, a cogitive attacker with sesig capability ca sese ad jam the chaels to be accessed by SUs while ot iterferig PUs. To address the above cocers, we formulate the atijammig multi-chael access problem as a o-stochastic multi-armed badit (NS-MAB) problem, where the SU seder ad the SU receiver adaptively choose chaels to operate respectively. By leveragig probabilistically-shared iformatio betwee the seder ad the receiver, the proposed spectrum sesig ad access protocol eables them to hop to the same set of chaels with high probability while gaiig resiliece to jammig attacks without affectig PUs activities. We aalytically show the covergece of the learig algorithms ad derive the performace boud based o regret. We further discuss the problem of trackig the best adaptive strategy ad characterize the performace boud based o a ew regret. Extesive simulatios are coducted to validate the theoretical aalysis. The results show that the probabilistic spectrum sesig ad access protocol ca overcome the limitatio of existig solutios ad is highly resiliet to various jammig attacks eve with jammed ACK iformatio. Idex Terms Ati-jammig, cogitive radio etworks, multi-radio multi-chael. INTRODUCTION COGNITIVE radio is a emergig advaced radio techology i wireless access, with may promisig beefits icludig dyamic spectrum sharig, robust cross-layer adaptio ad collaborative etworkig. Opportuistic spectrum access (OSA), which is at the core of cogitive radio techologies, has recetly received icreasig attetio due to its great potetial to improve the spectrum utilizatio efficiecy ad reliability [2] [6]. The basic idea of OSA is that idividual secodary users (SUs) dyamically search ad access the spectrum vacacy to maximize the spectrum utilizatio while itroducig limited iterferece to the primary users (PUs). I existig literature, the optimality of the chael sesig ad access problem has bee extesively studied from Qia Wag ad Shegsha Hu are with the School of Computer Sciece, Wuha Uiversity, Chia. qiawag@whu.edu.c. Kui Re is with the Departmet of Computer Sciece ad Egieerig, The State Uiversity of New York at Buffalo, NY 426, USA. kuire@buffalo.edu. Peg Nig is with the Departmet of Computer Sciece, North Carolia State Uiversity Raleigh, NC 27695, USA. pig@csu.edu. A prelimiary versio [] of this paper was preseted at the 9th IEEE Iteratioal Coferece o Network Protocols (ICNP ), Vacouver, BC Caada, 2. the sigle-chael access settig to the multi-chael access settig ad from perfect sesig to imperfect sesig usig various optimizatio tools. Most of existig solutios, however, ievitably assumed that traffic statistics are pre-kow to SUs. I practice, such assumptio may ot always hold ad more seriously, solutios based o this assumptio are vulerable to malicious jammig attacks. First, PU s traffic statistics (i.e., iitial iformatio states, trasitio probabilities ad the order of trasitio probabilities) may ot be readily available to the SUs prior to the start of sesig actios. Without a priori iformatio o the traffic patters, those opportuistic spectrum sesig ad access protocols caot work. Moreover, a cogitive jammer with sesig capabilities ca choose chaels to sese by leveragig the same statistic iformatio ad stochastic dyamic decisio makig process. Based o the sesig results, the attackers the jam the idle chaels potetially used by SUs without affectig activities of PUs. This is due to the fact that the structure of those sesig policies is fixed ad the chael selectio procedure that SUs follow is publicly kow. Therefore, a jammer ca predict which chaels the SUs are goig to use i each timeslot ad prevet the spectrum from beig utilized efficietly.

2 JOURNAL OF L A T E X CLASS FILES, VOL. 6, NO., JANUARY 27 2 Traditioal ati-jammig schemes, icludig both frequecy hoppig spread spectrum (FHSS) ad direct-sequece spread spectrum (DSSS) [7], commoly rely o some pre-shared secrets (i.e., hoppig sequeces ad spreadig codes) to achieve jammigresistat commuicatio. However, they are ot directly applicable to CRNs due to the fact that the presharig of secrets are ot applicable i a dyamic SU etworice SUs may ever meet each other before the start of commuicatio. Recetly, ucoordiated frequecy hoppig (UFH) ad ucoordiated direct-sequece spread spectrum (UDSSS) ad their variatios were proposed to elimiate the reliace o the pre-shared secrets [8] [3]. The major problem with UFH ad UDSSS is that they are both very expesive. For UFH, it takes a log time for a SU seder to trasmit a message to a SU receiver. This is ot practical for CRNs where SUs eed to fiish trasmissio quickly to yield the chael to PUs. O the other had, UDSSS may take less time to deliver a message, but the message decodig process at the receiver side will icur a large cost. Moreover, applyig UDSSS directly to the ati-jammig problem i CRNs results i a problem. UDSSS is commoly used i a broadcast commuicatio settig where the commuicatio chael is publicly kow ad SUs are usig radomly-selected spreadig codes to defed agaist jammig. I CRNs, it will cause large iterferece to PUs whe they are also active o the same commuicatio chael. I [4], [5], the problem of defedig jammig attacks i cogitive radio etworks was ivestigated usig game-theoretic approaches. However, they oly explored the siglechael case ad assumed that the SU receiver ca always commuicate with the secodary seder (i.e., they are cosidered as a sigle player) ad sesig is perfect. I [6], the spectrum sesig problem was formulated uder time-varyig chaels as a adversarial badit problem. Similar to [4], [5], the authors oly cosidered the case of sigle sesig chael ad assumed that the SU receiver ad the SU seder were cosidered as a sigle player. I [7], [8], ati-jammig games were ivestigated i CRNs. However the SU seder ad the SU receiver are still cosidered as a sigle player, i.e., they are assumed to stay coordiated by iitializatio with the same radom seed. To address the above limitatios, i this paper we propose a decetralized ad robust ati-jammig multi-chael spectrum access protocol for ad hoc CRNs, which ca accommodate both the eviromet dyamics ad the strategic behaviors of the jammers. Compared to existig UFH protocols, i a CRN settig, our protocol ca adaptively choose the most likely free chaels with high probability istead of radomly sesig ad accessig chaels. That is, the trasceivers will selectively sese chaels with high probability of o-occupacy by the jammer ad the PUs, based o the history iformatio of sesig ad access. However, if the SU seder detects the presece of a PU o a sesed chael, it will remai silet ad does ot access that chael i the curret timeslot. The sesig results together with the immediately followig access results will be feedbacked to the sesig actios i the future timeslots. Therefore, commuicatio efficiecy ca be sigificatly improved without affectig PU s activities. Differet from existig determiistic dyamic spectrum access protocols, we adopt a probabilistic spectrum sesig ad access approach, where the seder ad the receiver sese/access chaels i a upredictable way ad their kowledge of chaels ca coverge due to shared iformatio the jammer does t have. As will be show, by leveragig probabilistically-shared iformatio betwee the seder ad the receiver, the commuicatio efficiecy ca be retaied while gaiig resiliece to jammig attacks. Compared to the prelimiary versio [], i this paper we have made substatial improvemets icludig both the theoretical performace aalysis ad experimets. New experimetal results ad full proofs of performace bouds are provided. We also discuss the problem of trackig the best compoud (adaptive) strategy ad characterize the bouds o the ew regret. The mai cotributios of this paper are: We propose the first olie adaptive multi-chael jammig-resistat spectrum access protocol for ad hoc CRNs by formulatig the ati-jammig problem as a o-stochastic MAB problem. We aalytically show the covergece of the learig algorithms as T goes to ifiity, i.e., the time-averaged performace differece betwee the SU seder ad thesu receiver s optimal strategies is o more tha 2k Tε l, where k = max{k 2 s,k r }, k r ad are the umber of chaels the receiver ad the seder ca access simultaeously i each timeslot, respectively, ε is the probability of sesig ad is the total umber of chaels. The proposed algorithm ca be efficietly implemeted i polyomial time. We further cosider the problem of trackig the best adaptive strategy ad preset a extesio of our costructio for ati-jammig spectrum access. We aalyze the performace boud o the ew regret defied based o the adaptive optimal strategy. We aalytically show the time-averaged performace differece betwee the SU seder ad the SU receiver s optimal strategies is upper bouded by O(2k l), where k = max{,k r }. Sice,k r ad are pre-set system parameters, the performace boud is costat as T goes to ifiity. The exteded algorithm for trackig the best compoud strategy ca also be implemeted i polyomial time. We preset a thorough quatitative performace characterizatio of the proposed scheme. The performace is evaluated by aalyzig a practical metric the expected time for message delivery with high

3 JOURNAL OF L A T E X CLASS FILES, VOL. 6, NO., JANUARY 27 3 probability. We also perform a extesive simulatio study to validate our theoretical results. It is show that the proposed algorithm is efficiet ad highly effective agaist various jammig attacks eve with jammed ACK iformatio. 2 MODELS AND ASSUMPTIONS 2. System Model ad Assumptios I a typical cogitive radio etwork (CRN), there exist a primary user (PU) etwork ad a secodary user (SU) etwork. To facilitate dyamic spectrum access, the spectrum is divided ito chaels, each of which evolves idepedetly ad has the same total badwidth. Differet from most existig works, i our model we assume the chael statistics are ot ecessarily the same for chaels. I the system, PUs occupy ad vacate the spectrum followig a discretetime Markov process, where chael i trasits from busy state ( ) to idle state ( ) with probability p i ad stays i idle state ( ) with probabilityp i. I the SU etwork, SUs seepectrum opportuities amog chaels. Specifically, SUs reserve a sesig iterval i each timeslot to detect the presece of a PU. Based o the outcomes of sesig, the SU seders decide whether to take the opportuity to access the curretly idle chaels or ot, ad vacate the spectrum wheever PUs reclaim them. At the ed of a timeslot, the SU receiver seds a short ackowledgemet to the SU seder o the chael where a packet trasmissio is successful. It is worth otig that we ivestigate the problem of robust spectrum sesig ad access i a ad hoc SU etwork without a cetral cotroller for coordiatig the SUs. Therefore, each autoomous SU aims to maximize its ow performace by sesig ad accessig the spectrum idepedetly [2]. Differet from most existig opportuistic spectrum access (OSA) protocols [2] [6] where traffic statistics are kow a priori, we cosider a more geeral ad practical sceario where traffic statistics are ot available to SUs before the start of commuicatio. For ease of expositio, i the followig discussio we term oe pair of commuicatig SUs as the seder ad the receiver. I a multi-radio settig, the seder ad the receiver are equipped with < ad k r < radios, respectively, eablig them to access multiple chaels simultaeously i each timeslot. Sice SUs must ot iterferece with active PUs i each timeslot, a SU seder thus seses < ad accesses oly k a chaels sequetially. At the receiver side, various efficiet message verificatio schemes ca be used for packet verificatio to defed agaist pollutio attacks, ad fragmets that have passed itegrity checks are reassembled to recostruct the origial message. To relax the strict sychroizatio betwee the seder ad the receiver, we ca let the hoppig frequecy of the receiver be much slower tha the hoppig frequecy of the seder, so packet losses caused by the lack of sychroizatio betwee seder ad receiver ca be eglected. Note that i our model, we do ot cosider ode autheticatio ad message privacy, which are orthogoal to the security problems this work addresses. 2.2 Threat Model ad Assumptios I CRNs, PUs such as TV users are licesed users (i.e., beig protected by law) ad usually well physically protected. From the jammer s perspective, it is very difficult to lauch effecitve attacks, ad there will be heavy pealties o the attackers if beig detected [7]. Therefore, we assume the jammer does ot have high icetive to attack PUs ad risk itself i jammig the licesed bads whe PUs are active. Istead, the jammer s target is o the secodary users (SUs), who are ulicesed users ad oly permitted to access the spectrum whe ot iterferig with PUs. The SUs access to the spectrum is opportuistic i ature without clear legal protectio. Besides, SU etworks are usually dyamic ad hoc etworks formed by radomly deployed self-orgaizig wireless devices, where it is difficult to implemet effective security coutermeasures. A stealthy attacker ca choose to jam ay targeted SUs ad prevet the targets from usig the spectrum for commuicatio. Note that such attacks agaist SUs by the jammer are stealthy ad do ot affect PUs commuicatios. That is, the attacker utilizes the same sesig iterval to detect (sese) the activity of the PUs ad oly jam the idle chaels (which are potetially used by SUs) based o the sesig outcomes. We assume the jammer has similar radio capabilities as SUs. That is, i each timeslot, the jammer is capable of sesig ad jammig k j (k j < ) chaels simultaeously. Assumig the jammer kows the whole spectrum access protocol, his objective the is to prevet the spectrum from beig utilized efficietly by the legitimate SUs. Specifically, we cosider four types of jammers with differet jammig strategies: Static jammer. A static jammer is a oblivious attacker, who selects the same set of chaels i each timeslot to sese. Based o the sesig results he emits jammig sigals o the sesed idle chaels. Note that the jammig actio is made idepedet of the sesig history the jammer may have observed i the past. Radom jammer. A radom jammer is also a oblivious attacker, who selects a set of chaels uiformly at radom from the public set of chaels i each timeslot to sese. Based o the sesig results he emits jammig sigals o the sesed idle chaels. Similar to the static jammer, the jammig actio is made idepedet of the sesig history he may have observed i the past. Myopic jammer. A myopic jammer is a powerful cogitive attacker ruig the myopic algorithm, which

4 JOURNAL OF L A T E X CLASS FILES, VOL. 6, NO., JANUARY 27 4 is a well-kow OSA strategy ad ca achieve suboptimal performace (The priciple of myopic policy will be show later i Sectio 3). Iitially, the jammer selects k j chaels to sese i each timeslot util all the chaels have bee sufficietly sesed. The he ca make a accurate estimatio of the traffic statistics usig the sesig results, based o which he utilizes myopic policy to predict PUs chael occupacy patter ad emits jammig sigals o the most likely idle chaels. Obviously, i each timeslot the jammig strategy is selected based o the sesig history ad pre-kow chael occupacy statistics. Adaptive jammer. A adaptive jammer is also a cogitive attacker ruig a multi-armed badit (MAB) algorithm, which is a olie learig protocol (The MAB based learig protocol will be show i sectio 4). The jammer selects k j chaels to sese i each timeslot ad jams the sesed idle chaels based o his sesig history ad past observatios. Note that, i the power adaptive jammig attack model we assume the jammer ca adjust his sesig ad jammig strategies by leveragig the outcomes of jammig. I other words, we assume that the jammer kows whether he succeeds i jammig the trasmittig chaels (where both the seder ad the receiver reside o i a timeslot) for all the past timeslots. We emphasize that it is almost impossible to implemet such a powerful jammer i practice. However, for the purpose of performace compariso we show that SUs equipped with our ati-jammig spectrum sesig ad access protocol are still resiliet to such adaptive jammig attacks. 3 VULNERABILITY ANALYSIS OF MULTI- CHANNEL OPPORTUNISTIC SPECTRUM AC- CESS PROTOCOLS I this sectio, we aalyze the weakess of the existig multi-chael opportuistic spectrum access protocols uder jammig attacks due to their determiistic feature, which motivates us to develop a probabilistic spectrum sesig ad access approach i the ext sectio. For ease of illustratio, i the followig we cosider a SU etwork with a sigle seder-receiver pair, but the same ideas ca also be applied ad exteded to a multi-user settig. is sesed idle. The objective of the seder the is to maximize the rewards that it ca gai over a (potetially ifiite) umber of timeslots. It has bee show that this optimizatio problem ca be solved by a stochastic dyamic programmig (SDP) approach [9] to obtai optimal performace. To reduce the computatio complexity of SDP caused by the expesive backward iductio procedure, may researches has bee focused o idex policies ad myopic policy that maximizes the coditioal expected reward acquired attwas first proposed ad explored i [2], [3]. By cocetratig oly o the preset ad completely igores the future, myopic approaches achieves suboptimal performace i geeral. I myopic policy, it has also bee show that a sufficiet statistic or the iformatio state of the system for the optimal decisio makig is the belief vector Ω(t) = [ω (t),ω 2 (t),...,ω (t)], where ω i (t) is the coditioal probability that chael i is idle i timeslot t. I timeslot t, a sesig actio a(t) deotes the chaels to be sesed. Let K i (t) {,} deote whether a ACK o chael i is received or ot i timeslot t. Give a(t) ad K i (t), the belief state i timeslot t+ is give by [2] ω i (t+) = p i, i a(t),k i (t) = p i, i a(t),k i(t) = ω i (t)p i +( ω i(t))p i, i / a(t). Assume all chaels have the same trasmissio rate B i, the myopic policy uder Ω is defied as â(t) = argmax a(t) i a(t) ω i(t)b i. Recetly, the dyamic multi-chael access problem was studied uder a special class of restless multi-armed badit problems (RMBP) i [6], based o which a idex policy called Whittle s idex policy has also bee applied i the dyamic spectrum access. Similar to myopic policy, the proposed Whittle s idex policy eables the SU seder to choose those chaels whose curret states have the largest idices to sese ad access. However, a strict costrait which requires the activatig of exact m = arms/chaels at each time step may cause the optimality to be lost, but eve so the Whittle s idex policy has the ear optimal performace. Aother iterestig observatio is that the Whittle s idex policy has the same structure as the myopic policy whe chaels are stochastically idetical. 3. Opportuistic spectrum access with kow chael traffic statistics I the cotext of cogitive radio for opportuistic spectrum access, a sigle-chael access problem withi the framework of POMDP was ivestigated, ad myopic policies uder both perfect ad imperfect sesig cases have bee ivestigated i [2] [6]. The mai idea of these schemes is that the seder chooses a subset of chaels to sese based o its past observatios ad gais a fixed reward if a chael 3.2 Aalysis of OSA Uder Malicious Jammig Attacks Although theoretically soud, almost all the existig OSA protocols (icludig idex based policies) oly work well i o-malicious eviromets. Amog others, oe key assumptio made by the existig solutios is that the traffic statistics should be kow a priori. Take idex based policies for example, it is required that the iitial belief vectors Ω() ad the order of state trasitio probabilities (i.e.,p i is greater

5 JOURNAL OF L A T E X CLASS FILES, VOL. 6, NO., JANUARY 27 5 or less tha p i ) o all chaels be pre-kow to SUs. I practice, however, these statistics may ot be readily available [5]. More seriously, due to the determiistic ature of the chael/frequecy selectio procedure, those OSA protocols are vulerable to malicious jammig attacks. That is, a itelliget jammer, who kows the traffic statistics of all chaels or lears them through sesig ad estimatio by observig all chaels, ca leverage such iformatio to predict which chael to be used. Sice the idex policies always choose the first chaels with the largest idices for sesig ad accessig, the jammer ca use the same dyamic decisio process to perform effective jammig attacks. I the worst case, the commuicatio ca be completely jammed as the jammer maitais the same updates iformatio for chael idex as SUs i each timeslot. From a theoretical perspective, most of OSA protocols are formulated as optimizatio problems with determiistic solutios. For example, the idex policies are established based o the stochastic model of the chael statistics. Cosider the Whittle s idex policy developed uder the restless multi-armed badit problems (RMBP) [2]. Sice the evolvemet of iformatio state (belief vector) is kow, the players ( the seder ad the receiver) ca compute ahead of time exactly what payoffs (rewards) will be received from each arm (chael). Based o the above aalysis, it is ecessary ad importat to develop probabilistic OSA protocols that are resistat to various jammig attacks ad ca accommodate the special characteristics of CRNs. To ehace the robustess of OSA, the problem of defedig jammig attacks i cogitive radio etworks was ivestigated usig game-theoretic approaches [4], [5]. However, they oly explored the sigle-chael case ad assumed that the SU receiver ca always commuicate with the secodary seder (i.e., they are cosidered as a sigle player) ad sesig is perfect. I [6], the spectrum sesig problem was formulated uder time-varyig chaels as a adversarial badit problem. Similar to [4], [5], the authors oly cosidered the case of sigle sesig chael ad assumed that the SU receiver ad the SU seder were cosidered as a sigle player. I this paper, we cosider a more practical model ad make a step towards the developmet of robust multi-radio multi-chael OSA protocols for CRNs. 4 JAMMING-RESISTANT MULTI-RADIO MULTI-CHANNEL OPPORTUNISTIC SPECTRUM ACCESS 4. Scheme Overview Based o the above aalysis, we ca see that whe a attacker lauches malicious jammig attacks to disrupt legitimate commuicatios i SU etworks, the chael statistics (which are determied by activities of PUs whe there exists o jammig) caot correctly reflect the true state (idle or busy) of the chael. That is, the rewards (i.e., idicatios of successful packet receptios) associated with each chael caot be modeled by a statioary distributio or o statistical assumptios ca be made about the trasitio of iformatio state ad the geeratio of rewards. This is due to the dyamic behaviors of both PUs ad jammers, i.e., PUs occasioally occupy ad free the chaels ad a jammer may adjust his sesig ad jammig strategy to maximize the effect of jammig. These effects will make the geeratio of rewards arbitrarily chage o chaels i each timeslot. Motivated by this observatio, it is ecessary to keep a exploratio of the best possible set of chaels for trasmissio to adapt the dyamics of jammers ad PUs. Meawhile, it is also ecessary to exploit the previously-chose favorable set of chaels as too much exploratio will potetially uderutilize them. Obviously, the proposed ati-jammig problem is thus the oe balacig betwee exploitatio ad exploratio, rather tha oly optimizatios. 4.2 Problem Formulatio: A Multi-player Game I this paper, we cosider a jammig ad atijammig game amog a SU seder, a SU receiver ad a jammer uder dyamic PU behaviors. To fully utilize the vacat spectrum, the objective of the SU seder-receiver is to choose the sesig, access/receivig actios i each timeslot to maximize the total expected rewards (i.e., successfully received packets) over T timeslots. O the cotrary, the jammer s objective is to miimize the total expected rewards to disrupt the legitimate SU commuicatios. Sice chael states (idle or busy) are ot directly observable before chael sesig, the seder chooses chaels to sese durig the sesig iterval, where the sesig actio is made based o all the past decisios ad observatios. Due to PUs dyamic actios o a chael, the seder oly chooses k a (k a ) idle chaels to access. At the receiver side, the receiver idepedetly chooses k r chaels to receive, where the selectio is also made based o all the past decisios ad observatios. Durig the same timeslot, the jammer chooses k j chaels to sese ad jam the sesed idle chaels based o his chose jammig strategy. Note that, although we cosider a sigle SU pair i our ati-jammig problem, the proposed scheme ca be directly applied to a SU commuicatio etwork with multiple SU seder-receiver pairs. This is because each SU, which is autoomous i a ad hoc SU etwork, ca utilize our proposed scheme to maximize its ow performace by takig iterferece/collistios caused by other SU pairs as jammig sigals. It is easy to see that whe the umber of other SU pairs i the eighborhood of the receiver which use the same chaels is much less tha, the impact of uitetioal iterferece ca be egligible.

6 JOURNAL OF L A T E X CLASS FILES, VOL. 6, NO., JANUARY 27 6 We ext formalize the jammig ad ati-jammig game usig mathematical otatio. We first umber the chaels/frequecies from to ad costruct the vector space {,}. Obviously, the seder s sesig ad access strategy space ad the receiver s receivig strategy space are deoted as S s {,} of size ( ) ad Sr {,} of size ( ) k r, respectively. I a SU s strategy/vector, the value of the f- th (f {,...,}) etry of a vector is if the f-th chael is chose for sedig ad access or receivig; otherwise. Accordigly, the jammig strategy space ( for the jammer is deoted as S j {,} of size ) k j. Differet from a SU s strategy, the value i the f-th etry deotes that the jammer chooses the f-th chael to sese ad jam ad the value is otherwise. Differet from the above three parties, PUs activities o the chaels are idepedet of other parties s actios, ad a PU s actio/strategy ca also be deoted as a vector s p {,}, where the value deotes the chael is idle ad the value deotes the chael is occupied. Durig each timeslot, the seder, the receiver ad the jammer choose their ow respective strategies s s S s, s r S r ad s j S j, respectively. I each timeslot, assume the PU s strategy or activity is s p. From the receiver s perspective, s s s p s j ca be cosidered as a joit decisio made by the seder, the PU ad the jammer, where deotes bitwise AND operatio. We say that i timeslot t the seder, a reward g f,t = is itroduced for chael f if the f-th etry of s s s p s j is ; otherwise o reward is received, i.e., g f,t =. O the receiver side, the receptio of a reward depeds o the state of the chael f the receiver has chose for packet receptio. I additio, we use erasure codig combied with short sigatures to verify/autheticate the received packets, reassemble message ad defed agaist pollutio-based DoS attacks [9]. Note that, we do ot differetiate betwee packet jammig ad packet collisios as they both cause iterferece to the legitimate packets, ad packet codig ca be used to recover bit errors i received packets. After the receiver chooses a strategy s r, a reward o chael f is revealed to the receiver if ad oly if f is chose as a receivig chael. There are four possible cases: Case : No packet is received o f. This is because f has ot bee selected by the seder for trasmissio. I this case, reward is obtaied. Case 2: A packet is received of. If the received packet fails to pass the verificatio, reward is obtaied. Case 3: A packet is received o f. Jammed or collided packets that caot be recovered will be discarded, resultig i reward. Case 4: A packet is received o f. If o jammig is detected or corrupted packets due to jammig ca be recovered via packet codig, a reward is obtaied. Similarly, after the seder chooses a strategy s s, a reward o chael f is revealed to the seder if ad oly if f is chose as a sedig chael. A reward is obtaied if a ACK is received o f, otherwise the reward is. I this paper, we formally formulate the jammigresistat spectrum sesig ad access problem as a o-stochastic MAB problem (NS-MAB) [2] [23], where each chael ca be cosidered as a arm of a multi-arm badit. Due to the jammig effect ad dyamics behaviors of PUs, each chael f is the associated with a arbitrary ad ukow sequece of rewards, which ca be obtaied o a chael if the seder ad the receiver choose f for sedig ad receivig simultaeously. For ease of aalysis ad presetatio, we first defie some importat otatio. I each timeslot t {,...,T}, the seder (receiver) idepedetly selects a strategy I t from his strategy sets. We write f i if chael f is chose i strategy i, i.e., the value of the fth etry of i is. Note that a strategy is a vector of dimesio, I t deotes a particular strategy chose for timeslot t, ad i deotes a geeral strategy i the strategy set. The total rewards of a strategy i durig timeslot t is g i,t = f i g f,t, ad the cumulative rewards up to timeslot t of each strategy i is G i,t = t s= g i,s = t f i s= g f,s. The total rewards over all chose strategies up to timeslot t is thus Ĝt = t s= g I s,s = t s= f I s g f,s, where I s is chose radomly accordig to certai distributio over the strategy set. To quatify the performace, we use the followig metric called regret: max i Sx G i,t Ĝx T, x {s,r}, where the superscript is used to differetiate the seder from the receiver, ad the maximum is take over all strategies available to the seder or the receiver. The regret is defied as the accumulated rewards (or successfully received packets) differece over T timeslots betwee the proposed strategy ad the optimal static oe. The static optimal strategy deotes the best fixed solutio (i.e., the best set of chaels that if keepig to use them largest rewards will be geerated.) for message receptio i the presece of jammig. Note that the seder ad the receiver will adaptively choose their ow strategies i each timeslot based o the updated probability distributios over the strategy set. As for the seder (receiver), the updates of the probability distributio are determied by the outcomes of joit actios of PU, the jammer ad the receiver (seder). Thus, the accumulated rewards of the seder (receiver) alog the time deped o the actios of the other three parties i each timeslot. 4.3 Our Costructio I this subsectio, we preset our jammig-resistat spectrum sesig ad access protocol. Our algorithm is a probabilistic oe that ca accommodate the

7 JOURNAL OF L A T E X CLASS FILES, VOL. 6, NO., JANUARY 27 7 Algorithm A Jammig-resistat Multi-radio Multichael Spectrum Sesig ad Access Protocol. Iput:,k r,, T, ε (,], δ (,), β s,β r (,], γ s,γ r (,/2], η s,η r >. Iitializatio: Iitialize all system parameters, settig the chael weight wf, x = f [,], the strategy weight wi, x = i [,N x ], ad the total strategy weight W x = N x = ( ), where x = s,t. For timeslot t =,2,...,T : Select a strategy It x accordig to p x i,t ( i [,N x ]), with p x i,t computed followig Eq. (5). 2: Compute chael selectio probability qf,t x ( f [,]) as qf,t x = i:f i px i,t. 3: Trasmit a packet if ad oly if the chael is sesed to be idle. 4: Perform verificatio ad jammig detectio oce a packet is received o chael f. Trasmit back a ackowledgemet o f if the received packet passes the check. 5: Compute rewards gf,t x ( f It x ) ad virtual rewards gf,t x with the revealed g f,t ( f [,]), followig Eqs. (3) ad (4). 6: Update chael weight wf,t x ad strategy weight wi,t x followig Eqs. () ad (2), respectively. Update the total strategy weight as Wt x = N x i= wx i,t. Ed chages of chael status caused by a (potetially) malicious jammer. The dyamic property of the proposed solutio lies i the trade-off betwee exploratio actio ad exploitatio actio, which will both affect the system performace. As show i Algorithm, the algorithm comprises two subalgorithms:a s at the seder side ada r at the receiver side. I Algorithm, the system parameters β,γ ad η are determied by the regret boud, ad the derivatio of them will be show i proof of Theorem. Let N x (x {s,r}) deote the total umber of strategies. Each strategy is assiged a strategy weight, ad each chael is assiged a chael weight. Durig each timeslot, the chael weight is dyamically adjusted based o the virtual chael rewards revealed to the seder ad the receiver: w x f,t = w x f,t eηx g x f,t, x {s,r}. () We use expoetially weighted forecasters which follow the Exp3 ( Expoetial-weight algorithm for Exploratio ad Exploitatio ) first proposed i [2]. I a multi-armed badit settig, at time t, a expert is chose with probability that icreases with the past performace of the expert. I practice, the most popular choice of such kid of fuctio is expoetial fuctio. It is easy to see that the icrease of the virtual chael rewards leads to larger chael weights. A strategy idicates the choices of chaels for use, so we defie the weight of a strategy as the product of the weights of all chaels: w x i,t = Π f i w x f,t = wx i,t eηs g x i,t, x {s,r}. (2) where gi,t x = f i gx f,t. Here, the reaso to estimate reward for each chael first istead of estimatig rewards for each strategy directly is that the reward of each chael ca provide useful iformatio about the other uchose strategies cotaiig the same chaels. The parameter β is used to cotrol the bias i estimatig the chael reward g s g s f,t = g r f,t = f,t gf,t s +βs εqf,t s ad gr f,t, which are computed as: R t if f I s t, β s εq s f,tr t otherwise, gf,t r +βr qf,t r β r qf,t r if f I r t, otherwise, where qf,t x (x {s,t}) deotes the chael f s probability distributio, ad R t is a radom variable uder Beroulli distributio satisfyig P{R t = } = ε. The parameter β is a fixed value that will be determied before the executio of the protocol (see the proof of Theorem ). Based o the true rewards revealed to the seder ad the receiver, we defie the virtual rewards to icrease weight of good chaels, i.e., icrease the access probabilities of good chaels which have bee less ofte sesed. I Algorithm, at the begiig of each timeslot, the trasceiver chooses a strategy based o the probability distributio p x i,t (x {s,t}) as: p x i,t = ( γ x ) wx i,t W + γx t x C x i C x (5) ( γ x ) wx i,t W otherwise, t x where x {s,r}. The parameter γ x is used to balace betwee wx i,t W ad t x C. x I the calculatio of the strategy probability distributio, the first part is a distributio which assigs to each actio a probability mass expoetial i the estimated cumulative reward for that actio, ad the secod part is the uiform distributio. If ot mixed with the uiform distributio, the algorithm might have large deviatios with high probability, i.e., from time to time it may cocetrate o the wrog strategy for too log ad the icur a large regret. So the mixig is doe to make sure that the algorithm tries out all strategies ad gets good estimates of the gais for each chael [2]. Note γ x is a fixed value that will be determied before the executio of the protocol (see the proof of Theorem ). The coverig strategy set C x is defied to esure that each chael/frequecy is sampled sufficietly ofte. The coverig set has the property that for each chael f, there is a strategy i i the coverig set such that f i. Based o the defiitio of strategy, each strategy icludes k x (x {s, r}) active chaels. Thus, we ca costruct oe typical ad simple coverig set with size C x = (x {s,r}). (3) (4)

8 JOURNAL OF L A T E X CLASS FILES, VOL. 6, NO., JANUARY 27 8 Discussios. I practice, the trasceiver (i.e., the seder ad the receiver) may ot have the same sesig outcomes due to sesig errors. So, i our desig we let the seder perform sesig i each timeslot, ad the receiver oly selects chaels to liste o. Note that the operatig poit of the spectrum sesor is set as the probability of the collisio with PUs [2], which icludes two types of sesig errors: false alarm probability ad miss detectio probability. Without loss of geerality, we use τ to deote the sesig error probability i the followig aalysis, where τ = P{false alarm}( P{PU active}) + P{miss detectio}p{p U active}. To elimiate the iformatio asymmetry betwee the seder ad the receiver, the seder uses the ackowledge iformatio to update the probability distributio over strategy set. Thus, the accumulated rewards for the seder ad the receiver are equivalet, i.e., Ĝ s t = Ĝr t (Note that, we make this assumptio to obtai the upperboud performace of the proposed ati-jammig scheme. I Sectio 6, we evaluate the case where ACKs are radomly jammed by the attacker, showig the strog resiliece of our proposed scheme). I additio, because the seder ad the receiver are ot perfectly sychroized, it is ecessary ad importat to evaluate how close the seder s ad the receiver s strategies are as time goes. Sice the closer the trasceivers chose strategies, the more rewards geerated. This is equivalet to sayig that how well the learig based algorithm proceeds to maximize the system throughput. The spectrum sesig usually cosumes more eergy compared to receptio, i.e., it is costly to obtai the sesig results [24]. I certai applicatio scearios, legitimate odes may oly have a limited umber of sesig times due to eergy costrait. Let ε deote the proportio of timeslots whe sesig is performed. For T timeslots, the umber of sesig times is approximately Tε. I Algorithm, we itroduce a Beroulli radom variable with P{R t = } = ε at the seder side. Thus, the seder seses the chael with probability ε. There are two possible cases whe the seder does ot perform sesig i a timeslot. I the first case, the seder remais silet i this timeslot without trasmittig ay packets. Due to the radom sesig ad access strategy, it is hard for the adversary to predict the behaviors of the seder. However, as o packets are trasmitted, the trasmissio delay may be icreased. I the secod case, the seder still accesses the most possibly free chaels based o the previous probability distributio. I this case, there is a tradeoff betwee the collisio probability with PUs ad the umber of sesig times. 4.4 Theoretical Aalysis I this subsectio, we aalyze the performace of Algorithm i terms of both optimality ad efficiecy. For ease of aalysis, we first give the followig two importat defiitios. Defiitio : A algorithm A is α-static (or α- adaptive) approximatio of the static (or adaptive) optimal solutio if ad oly if it ca trasmit the message successfully i time αt with high probability (w.h.p) l ǫ whe the static (or adaptive) optimal solutio ca trasmit the same message successfully with the same probability l ǫ i time T, where ǫ > is a costat ad l is the umber of packets i the message. Defiitio 2: The regret of a algorithm A is the differece betwee the accumulated rewards usig the static optimal strategy ad that usig A over T timeslots, i.e., G max T ĜA T, where Gmax T = f i T max i S G i,t = max i S s= g f,s ad ĜA T = T s= g I s,s = T s= f I s g f,s. The first defiitio is used to characterize the approximatio ratio betwee the proposed algorithm ad the static ad adaptive optimal solutios. The secod defiitio is used to characterize the throughput performace betwee the proposed algorithm ad the optimal solutio. I the followig aalysis, we will write G max istead of G max T wheever the value of T is clear from the cotext. I additio, we will write G max T (s) ad G max T (r) to deote the rewards of the static optimal strategies for the seder ad the receiver, respectively. Due to the probabilistic strategy selectio, the seder ad the receiver are ot perfectly sychroized i each timeslot. However, we show that the seder s sesig strategy ad the receiver s receivig strategy will coverge to their ow optimal strategies. The followig theorem measures how close their optimal strategies are as T. Theorem : The ormalized reward distace T Gmax T (s) G max T (r) coverges to at rate O(/ T) as T. Proof: We first prove that at the receiver side, with probability at least δ, the regret G max T (r) ĜAr T is at most 6k r Tl, while β r k = r T l δ, γr = 2η r ad η r = l 4T ad T max{kr l δ,4l}. We use a superscript for η, γ, β to differetiate betwee the seder ad the receiver. However, for ease of expositio, we do ot differetiate betwee the other otatio sice they are idepedet i the proofs for the seder ad the receiver. Now we itroduce some otatio for performace aalysis: G i,t = T t= g i,t ad G i,t = T t= g i,t for all i N, where G i,t (G i,t ) deotes the total gai (virtual gai, respectively) of strategy i i T timeslots, ad G f,t = T t= g f,t ad G f,t = T t= g f,t for all f, where G f,t (G f,t ) deotes the total gai (virtual gai, respectively) o chael f i T timeslots. The relatioship betwee gai ad virtual gai is derived as follows.

9 JOURNAL OF L A T E X CLASS FILES, VOL. 6, NO., JANUARY 27 9 The proof is applicable for ay chael f. u > ad c >, by usig the boud of Cheroff, we get P[G f,t > G f,t + u] e cu E[e c(g f,t G f,t ) ]. Let u = l δ /β ad c = β, we obtai e cu E[e c(g f,t G f,t ) ] = δ E[eβ(G f,t G f,t ) ]. Thus, we ca show that e β(g f,t G f,t ) for all T. Let Z t = e β(g f,t G f,t ). By showig E[Z t ] Z t for all t 2 ad E[Z ], it is easy to prove δ (,), β < ad f, P[G f,t > G f,t + β l δ ] δ Next we show the regret boud by usig l WT W. First, we directly ca obtai the lower boud by the defiitio l WT W = l N G i= eηr i,t ln η r max i N G i,t ln. The we derive the upper boud as follows: η r g i,t = η r f i g f,t +β r q f,t η r f i, where the secod iequality term holds due to q f,t γr C ( f) by the defiitio. Usig the fact that e x + x + x 2 for all x, for all t =,2,,T we have l Wt l N i= (η r ) 2 g 2 w i,t W t e ηr g i,t ηr k r(+β r ) C γ r i,t )) l( + N η r N γ r i= p i,tg i,t + (ηr ) 2 (6) W t = l( N w i,t i= W t ( + η r g i,t + p i,t i= γ (η r g r i,t + (ηr ) 2 g 2 i,t )) N γ r i= p i,tg 2 i,t. We derive the iequalities usig the followig two commo facts: N i= p i,t γ r ad iequality l(+x) x for all x >. Let N deote the strategy set {,...,N}. O the oe had, we have N i= p i,tg i,t = N i= p i,t f i g f,t = f= g f,t i N:f i p i,t = f= g f,t q f,t = g It,t + β r. O the other had, N i= p i,tg 2 i,t = N i= p i,t( f i g f,t )2 N i= p i,tk r f i g 2 f,t = k r f= g 2 f,t i N:f i p i,t = k r f= g 2 f,tq f,t k r ( + β r ) f= g f,t, which holds the fact that g f,t +βr q f,t. Note that for clearly differetiatig betwee the regret bouds for the seder ad the receiver, i the derivatio we loose the bouds a little bit by choosig k r istead of mi{k r, ε( τ), k j }. Therefore, l Wt W t ηr γ (g r It,t +β r )+ (ηr ) 2 k r(+β r ) γ r f= g f,t. Summig for t =,,T, we have the followig iequality l WT η r γ r (ĜT + β r T) + (η r ) 2 k r(+β r ) W γ r f= G f,t γ r (ĜT + β r T) + (η r ) 2 k r(+β r ) γ C max r i N G i,t. Here, Ĝ T deotes the expected total gai of the proposed algorithm i T timeslots. By usig the upper ad the lower bouds together, we get Ĝ T ( γ r η r k r ( + β r ) C )max i N G i,t γr η ln β r T. r Applyig Eq. (6), we ca show with probability at least δ, Ĝ T ( γ r η r k r ( + β r ) C )(max i N G i,t kr β l γr r δ ) η ln β r T. r Here, we used the fact γ r η r k r ( + β r ) C > which follows the assumptios of the theorem. η r By doig some traspositios ad usig the followig fact max i N G i,t Tk r, we have max i N G i,t ĜT (γ r + η r ( + β r )k r C )Tk r + ( γ r η r (+β r )k r C ) kr β l r δ ++ γr η ln +β r T r with probability at least δ. Let K = mi{, k j,k r }. Sice G ˆ T = KT L ˆ T ad max i N G i,t = KT mi i N L i,t, we have L ˆ T KT(γ r +η r (+ β r )k r C )+( γ r η r (+β r )k r C )mi i N L i,t + ( γ r η r ( + β r )k r C ) kr β l r δ + γr η ln + β r T r with probability δ. By simplifyig the iequality, we have L ˆ T mi i N L i,t k r Tγ r + 2η r Tk r + k r β r l δ + γr k r η r k r l+β r T with probability δ Settig β r = T l δ ad γ r = 2η r k r C, we ca get G max T (r) ĜAr T = max i N G i,t ĜT 4η r Tk 2 r C + ln η + 2 k r r T l δ which holds with probability δ if T kr l( δ ). Fially, by usig the facts C = k r ad N kr, ad settig η r = ln 4kr 2T C, we ca prove max i N G i,t ĜT 6k r Tl by properly choosig δ. Similarly, at the seder side we first show the coectio betwee the true ad the estimated cumulative rewards. The oly differece is that the computatio of estimated chael rewards is ivolved with a radom variable ε. We prove that with probability at least δ, the regret G max T (s) ĜAs T is 2 bouded by 4 β s = Tl ε by property choosig δ, εl 4T ad Tε l 2 δ, γs = 2ηs ε ad η s = T max{ ks l2 2 δ εl,4l}. To clearly differetiate the regret bouds for the seder ad the receiver, we loose the bouds a little bit by choosig k r ad istead of mi{k r, ε( τ), k j }. Hece, the sesig error probabilityτ does ot appear i the fial expressio of performace boud. Fially, as ĜAs T 6k r Tl + 4ks 2 2 l δ, = ĜAr T, Gmax T Tl ε 2k (s) G max T (r) ε Tl, where k = max{k 2 s,k r }. Thus, T Gmax T (s) G max T (r) at rate O(/ T) as T. Theorem 2: Algorithm has time complexity O(k x T) ad space complexity O(k x ), where x {s,r}. Proof: I the proposed algorithm, the computatio of probability distributios of strategy ad chael are the most time-cosumig steps due to the expoetial umber of possible strategies. I this proof, we show that the time complexity ca be reduced by usig dyamic programmig. We use S( f, k) to deote the strategy set of which each strategy selects k chaels from f, f +,,. Also, we use S( f, k) to deote the strategy set of which each strategy selects k chaels from,2,, f. We defie W t ( f, k) = i S( f, k) f i w f,t ad W t ( f, k) = i S( f, k) f i w f,t. Note W t ( f, k) = W t ( f +, k) + w f,t W t ( f +, k ) ad W t ( f, k) = W t ( f, k) + w f,t W t ( f, k ), which implies both W t ( f, k)

10 JOURNAL OF L A T E X CLASS FILES, VOL. 6, NO., JANUARY 27 ad W t ( f, k) ca be calculated i O(k x ) (lettig W t ( f,) =, W( +, k) = W(, k) = ) by usig dyamic programmig approach f ad k k x. I step, a strategy should be draw from ( ) k x strategies. Istead of drawig a strategy, we choose chael for the strategy oe by oe. Here, we choose chaels oe by oe i the icreasig order of chael idices, i.e., we determie whether the chael should be selected, ad the chael 2, ad so o. f, if k k x chaels have already bee selected from chaels,,f, we select a chael f with probability w f,t W t (f+,k x k ) W t (f,k x k) ad ot select f with probability Wt (f+,kx k) W t (f,k x k). Let w(f) = w f,t if chael f i; w(f) = otherwise. Obviously, w(f) is actually the weight of f i the strategy weight. I our algorithm, w i,t = f=w(f). Let c(f) = if f is selected i i; c(f) = otherwise. f f=c(f) deotes the total umber of chaels selected from chaels,2,, f ii. By this implemetatio, the probability of choosig i is w( f)w t ( f+,k x f= f f= c(f)) f= w( f) W = t ( f,k x f f= c(f)) W = wi,t t (,k x) W t. This probability is equivalet to that i Algorithm, which implies the implemetatio is correct. Because we do ot maitai w i,t, it is impossible to compute q f,t as we described i Algorithm. The, q f,t ca be calculated i O() as q f,t = ( γ) kx k= +γ {i C:f i} C W t (f,k)w f,t W t (f+,k x k ) W t (,k x) for each roud. I practice, the trasmitted messages, which may have much larger size tha the legth of timeslots, have to be split ito small fragmets to fit the timeslots. As show above, the proposed jammig-resistat spectrum sesig ad access protocol is probabilistic i ature, so we caot guaratee the trasmitted message is delivered i certai umber of timeslots with probability oe. So, to evaluate the trasmissio efficiecy, we cosider the expected time for a message delivery with high probability, which implies the probability goes to oe whe the total umber of packets goes to ifiity. Based o the ackowledgemet iformatio, i each timeslot the seder will pick up a packet that has ot bee delivered. Without loss of geerality, assumig a message M is partitioed ito l packets M,M 2,,M l, each of which has size M i = M /l ( i l). The, the trasmitted message M ca be recostructed at the receiver if ad oly if all l packets are successfully received. The followig theorems characterize the approximatio factors for the static optimal ad adaptive optimal solutios. Theorem 3: Whe l 36(+cǫ)k r l/(c ) 2 ǫ 2, our algorithm is ( + cǫ)-static approximatio for ay costat c >. Proof: See []. 3 lk(+cǫ) Theorem 4: Whe l 36ε( τ)( k j)(c ) 2 ǫ, our 2 algorithm is 2 k rε( τ)( k j) K( + cǫ)-adaptive approximatio for ay costat c >, where K = mi{k r, ε( τ), k j }, ε is the probability of sesig a chael ad τ is the sesig error probability. Proof: See []. Discussios. As ca be see i the proof of Theorem, the parameters β, η ad γ are fixed values ad they are all pre-computed before the protocol executio. If we aim at esurig that with probability at least δ the regret boud ca be achieved, we ca set a preferable value for δ. The parameter selectio process is as follows. We have β x k = x T l δ, γx = l 4T. Here, ad k r are pre-selected 2η x ad η x = system parameters. Oce T is obtaied, the specific values of β x, η x ad γ x ca be determied such that the regret boud holds (or asymptotic optimality is achieved). To determie T, i our protocol desig, we let the seder determie a feasible T ad ecode it i each packet for trasmissio. The receiver obtais T by successfully decodig ay received packet ad begis to ru the algorithm. Assume p is the probability of message delivery, the seder determies T by first estimatig a lower boud k r of k r ad a upper boud k j of k j. It the calculates ǫ such that l = p ǫ ad determies the costat c > such that l = 36(+cǫ)k r l/(c ) 2 ǫ 2. Fially, the expected time k for message delivery is T = (+cǫ)l/(ε( τ) k j r ). By theorem 3, with probability at least p the message M ca be successfully recovered at the receiver. 5 TRACKING THE ADAPTIVE COMPOUND STRATEGY FOR ANTI-JAMMING SPECTRUM ACCESS I the above discussios, regret is computed as the accumulated reward differece betwee the proposed ati-jammig strategy ad the static optimal strategy. We have show that the proposed jammig-resistat i Algorithm ca track the static optimal strategy ad coverge to it as time goes. Accordig to the defiitio, the static optimal strategy is selected as the fixed best strategy used for all timeslots. However, for each timeslot there always exists the best strategy agaist the joit strategy of the other parties ivolved i the ati-jammig game. Likig these strategies from all timeslots together, the best compoud strategy is formulated, ad this is so-called the adaptive optimal strategy. So, a iterestig questio ca be raised here: at each timeslot, the good strategy may chage, is it possible to select a sequece of strategies to approximate the adaptive/compoud strategy? 5. The Proposed Costructio I Algorithm, the size of the static optimal strategy set is ( k r ) ( ( ) ). However, for all possible compoud