/13/$ IEEE

Transcription

1 A Game-Theoretical Anti-Jamming Scheme for Cognitive Radio Networks Changlong Chen and Min Song, University of Toledo ChunSheng Xin, Old Dominion University Jonathan Backens, Old Dominion University Abstract Cognitive radio networks are a promising solution to the spectrum scarcity issue. However, cognitive radio networks are vulnerable to various kinds of security attacks, among which the jamming attack has attracted great attention as it can significantly degrade spectrum utilization. In this article we model the jamming and anti-jamming process as a Markov decision process. With this approach, secondary users are able to avoid the jamming attack launched by external attackers and therefore maximize the payoff function. We first use a policy iteration method to solve the problem. However, this approach is computationally intensive. To decrease the computation complexity, Q-function is used as an alternate method. Furthermore, we propose an algorithm to solve the Q-function. The simulation results indicate that our approach can achieve better performance than existing approaches to defend against the jamming attack. C ognitive Radio (CR) has received increasing attention due to its potential to solve the conflict between limited spectrum supply and spectrum demand from ever-increasing wireless applications and services []. Being capable of utilizing the spectrum in an opportunistic manner, cognitive radio enables secondary users (SU) to sense the portion of the spectrum that is available, select the appropriate channel for access, and vacate from the channel when the primary user (PU) returns. Cognitive radio networks (CRNs) are subject to various kinds of security attacks, among which the denial of service (DoS) attack is one of the most serious threats. By launching a DoS attack over communication channels, the attacker can severely degrade network performance. The channel jamming attack is one of the DoS attacks that are simple to launch, and difficult to be countermeasured. The jamming attack is a security threat where the attacker interferes a set of communication channels by injecting a continuous jamming signal or non-continuous short jamming pulses. As a result, the communication channels either cannot be accessed or the signal to noise ratio (SNR) in these channels is heavily deteriorated. In traditional wireless networks, the jamming attack has been studied extensively and there is a rich literature of antijamming approaches []. For example, spread spectrum [3] and channel hopping [4] are two defense schemes that are widely used in the physical layer and link layer, respectively. The research of Min Song is supported in part by NSF CAREER Award CNS-489 and NSF IPA Independent Research and Development (IR/D) Program. However, any opinion, finding, and conclusions or recommendations expressed in this material; are those of the author and do not necessarily reflect the views of the National Science Foundation. The research of Chunsheng Xin is supported in part by NSF under grants CNS- 77, CNS-7668, and ECCS However, these approaches cannot be directly applied to cognitive radio networks since spectrum availability is dynamic in cognitive radio networks. Recently, several anti-jamming approaches for cognitive radio networks have been proposed, e.g. the stochastic game framework by Wang et al. [5], and the decentralized anti-jamming multi-channel spectrum access protocol by Wang et al. [6]. In this article we focus on jamming attacks in cognitive radio networks and propose a game-theoretical anti-jamming scheme () scheme to countermeasure this attack. Compared with the previous work, the anti-jamming scheme we develop has a low computation complexity, while achieving very good throughput. To avoid jamming, SUs proactively hop among accessible channels. We formulate the jamming-hopping process as a Markov Decision Process and propose a Policy Iteration scheme, with a complexity analysis of Policy Iteration scheme. Although this scheme is effective, it may be computationally prohibitive. To reduce the computation complexity, we further propose a Q-function based approach. Through extensive simulations, we demonstrate that our proposed scheme can achieve a high payoff. In addition, our scheme has a low probability of being jammed by an attacker. The remainder of the article is organized as follows. We review the related work. We describe the system model. We discuss the proposed anti-jamming scheme. We present simulation results. Finally we conclude the article. Related Work In this section, we discuss the state-of-the-art work on antijamming schemes for cognitive radio networks and traditional wireless systems. In [7] Li and Han applied game theory and Markov decision process theory to study the jamming and anti-jamming problem in cognitive radio networks. For the one-stage case, the jamming and anti-jamming are modeled as /3/$5. 3 IEEE IEEE Network May/June 3

2 a zero-sum game, and the Nash Equilibrium strategy was calculated. For the multi-stage case, the problem was modeled as a stochastic game. The game was analyzed based on the framework of partially observable Markov Decision Process. However, several assumptions of this article are strong. For example, the channel availability was assumed perfectly known to both benign and malicious users, the attacker was assumed always rational, i.e. always following the best strategy, and the attacker was assumed to know the benign user s strategy. In [5] the authors also used game theory to formulate channel hopping. The problem was formulated as a zerosum problem. At each time slot, the SUs observe spectrum availability, the quality of the channel, and the attacker s strategy. Using the minimax-q learning method, an SU can learn the optimal policy and maximizes throughput. In [6] Wang et al. proposed an anti-jamming protocol, where the sender and the receiver hop to the same set of channels for communication with a high probability. The network is time-slotted. In each time slot, the sender chooses a set of channels with high weights to sense, transmits on the detected idle channels, and waits for ACK. The receiver also chooses a set of channels to receive packets based on the weights of channels. The weight of a channel is adjusted based on the channel reward, i.e. whether the sender successfully receives an ACK. In [8] the authors studied the vulnerability of rate adaptation algorithms (RAA) to the jamming attack, and the weakness of IEEE8.. Several algorithms were proposed to determine optimal jamming strategies against RAA for a given jamming budget. System Model Channel Model In this article, we consider a cognitive radio network with a number of PUs, where each PU is on a different channel. The spectrum is divided into N channels. The statistics of each channel are independent of each other. The channel activity is modeled as a discrete-time Markov process. Each channel has two states: State (idle), indicating that the PU is inactive on this channel. State (busy), indicating that the PU is active on the channel. The transition probability from state to state is denoted as p, while the transition probability from state to state is denoted as p. We assume that initially all N channels are idle. The probability that the PU is idle/busy on each channel at different time slots can be calculated recursively. Secondary User Model Let H denote the number of SUs in the network. We assume that all SUs operate in a time-slotted mode. To avoid jamming, SUs periodically change their operation channels. Each SU uses a small interval at the beginning of each time slot for spectrum sensing. During this sensing interval, each SU detects the presence of the PU in all of the N channels and chooses up to L accessible channels for communication based on our anti-jamming scheme. We assume that all SUs can detect whether the channels they have accessed are jammed by the attacker at the end of each time slot, and every SU keeps a history of spectrum sensing and successful/jammed communication. Due to space limitations, we will not discuss details on how to detect that a channel is jammed, as this itself is a large topic. The interested reader is referred to [9]. In each time slot, SUs decide whether to transmit on each channel. For each SU, its activity on a channel is one of the following three possibilities: A successful transmission. An unsuccessful transmission, i.e. being jammed. No transmission. Since these three activities yield different payoffs, SUs need to choose an action in each channel that maximizes the system s expected payoff. This payoff is based on the history of the channel. A successful packet transmission pays U to SUs for each channel, where U is the utility for a single channel, while when jamming occurs in a channel, it costs C to SUs. The payoff for no transmission is always zero. We assume that the value of the utility and the cost of each channel are constant and the same for every channel. Attacker Model Without loss of generality, we consider one external malicious attacker. This attacker is not authenticated by and associated with the cognitive radio network. The attacker can launch a jamming attack to degrade spectrum utilization. Due to the jamming attack, a channel either cannot be accessed, or the SNR on this channel is heavily deteriorated. There are other possible reasons for reducing SNR. However, we focus on the jamming attack in this article. How to distinguish other factors for the reduced SNR from the jamming attack will be one of our future directions. Both SUs and the attacker are assumed to follow a time-slotted access scheme. The attacker s scheme is characterized as follows. First it chooses a set of channels to perform spectrum sensing for a certain duration, called spectrum sensing duration in this article. If it has detected that the PU is active in the channel, or the SUs do not occupy the channel (no signal on the channel) after the spectrum sensing duration, the attacker hops to a new set of channels and performs sensing again, until the attacker finds some channel(s) that are occupied by one or several SUs, and launches the jamming attack on these channels. Note that while the attacker tries to find the channels of SU(s) through spectrum sensing and hopping, the SU is also dynamically hopping channels. Therefore, the attacker may not be able to find the channels occupied by SU(s) within a certain time to launch the jamming attack. In each time slot, the attacker can jam up to J channels. Game-Theoretical Anti-jamming Scheme Game Formulation We formulate the jamming and anti-jamming between the attacker and the SUs as a game. There are two types of players in this game: SUs and the attacker. The strategy set of this game denotes the set of channels for access, which is finite. Therefore, according to [], this game has a Nash equilibrium. Moreover, in our model the two-player game is a zerosum game, since the gain of one player results in a corresponding loss of the other player. Let s denote the state of all channels in a time slot k, indicating whether each channel is accessible by the SUs. Let S denote the set of all possible states. Both the SUs and the attacker have the knowledge of the state information since they all sense the channels at the beginning of each time slot. The SUs have a finite action set A(s), which depends on the state s of the channels. At time slot, each SU randomly chooses L accessible channels for communication. After that, in each time slot, the SU selects an action from A(s) based on the state s of the channels, e.g. stay on certain channels or hop to other channels. At a given time slot, let M denote the number of channels being occupied by the PUs. Each SU selects L (L N M) channels for communication from the N M accessible channels. Therefore, there are C L N M possible selections in total, and for each selection, there is an action. IEEE Network May/June 3 3

3 Hence, A(s) = {a,, a i,, ac L N M}, where a i indicates the action for the ith selection of L channels. At a given time slot, the SUs select an action a based on state s of the channels, and at the next slot, the SUs select another action a based on state s of the channels in this slot. The probability of transiting from action a in state s to action a at state s is denoted as Pr(s, a Ís, a). The SU s transition probability depends on the state s at a time slot. For example, when L =, in time slot k, an SU accesses channel l. In time slot k +, if channel l is occupied by PU, the SU must hop to one of the N M accessible channels. In this situation, the transition probability of the SU is Pr ( s, a s, a) =. N M On the other hand, if channel l is still accessible in slot k +, we set the transition probability to stay in channel l in slot k + as p stay, i.e. the SU stays in this channel with Pr(s, a Ís, a) = p stay, which means that the SU hops to another accessible channel with probability Pr( s, a s, a) = ( pstay ) N M. As described earlier, each SU selects L channels for communication. The utility earned by the SUs successful communication subtracting the cost due to jammed communication in L channels is the gain for the SUs in a time slot, denoted as G(s, a i ) based on the state s of the channels and the action a i. Hence we have G(s, a i ) = S L l=(x l (s, a) U y l (s, a) C), where x l (s, a) and y l (s, a) are switching functions, which depend on the state s and the corresponding action a of the SUs: x l (s, a) = when communication in channel l is successful under action a at state s, while x l (s, a) = when communication in channel l is jammed under action a at state s. Similarly, y l (s, a) = when communication in channel l is jammed under action a at state s, while y l (s, a) = when communication in channel l is successful under action a at state s. For example, we consider a special condition when there is only one SU in the network and during each time slot, the SU accesses only one channel. The SU s communication can either be successful or jammed. Therefore, G(s, a) = U if the communication is successful, and G(s, a) = C otherwise. Policy Iteration Scheme We use Markov Decision Process (MDP) to formulate the anti-jamming process. An MDP has four components: Finite state set S. Finite action set A. Transition probability. Gain. Since the anti-jamming process discussed above contains these four components, it can be formulated as an MDP. We can solve the MDP to obtain the optimal strategy. We define the stationary policy: p i : s Æ a i, where p i ŒP(s) = {p, p,, p C L N M }. P(s) corresponds to the action set A(s) at state s. For example, given two PUs, four channels, and two SUs, in the first time slot, suppose that the first channel is accessed by an SU, the second channel is idle, the third channel is occupied by the PU, and the fourth channel is jammed by the attacker. Then the state s for the four channels is {SU access, Idle, PU occupy, Attacker jamming}, the action set A(s) = {stay, hop}, the policy set P(s) = {channel Æstay, channel Æhop, }. Since the communication is successful for the SU at this slot, G(s, a) = U. Let V p (s) denote the value function for p, which is the ly choose a policy p ŒP(s) at time slot Calculate V p (s) using Eq. while time slot k π final time slot do for all policy p Œ P(s ) do {traverse all actions} Calculate V p (s ) using Eq. if V p (s) < V p (s ) then V p (s) = V p (s ) p = p end if end for policy update: calculate p(s) with updated p using Eq. end while Algorithm. Policy iteration algorithm. expected total reward, i.e. k k Vπ () s = E G( s, a ) a = π ( s ), s= s i k= = E Gsa (, ) s, a + Pr s, a sa, V ( s ) s ( ) where G(s k, a k ) is the total gain of SU to choose action a k at state s k since the beginning slot, a is a specific action at the current state s, G(s, a) is the immediate gain, and S s Pr(s, a Ís, a) V p (s ) is the expected future gain. The optimal value function has an additional property, i.e. satisfying the Bellman optimality equation. Hence we have V* () s = max G(, s a) + Pr( s, a s, a) V* ( s ), π s π* ( s) = arg max Gsa (, ) + Pr( s, a sa, ) V* ( s ). π s We can use a Policy Iteration algorithm, described in Algorithm, to solve the Bellman optimality equation to obtain the optimal policy. Complexity Analysis We now briefly analyze the computation complexity of Algorithm. Inside the while loop of Algorithm, line 5 is the value determination phase and line is the policy improvement phase. In each iteration, for the value determination phase, the complexity is O(ÍSÍ 3 ) from solving the linear equations. On the other hand, the policy improvement can be performed in O(ÍAÍÍSÍ ) steps. Therefore, in each iteration of Algorithm, the cost is O(ÍAÍ ÍSÍ + ÍSÍ 3 ). Algorithm will run T time slots. Based on the above analysis, the computation complexity of Algorithm is O(ÍTÍ (ÍAÍ ÍSÍ + ÍSÍ 3 )). Q-Function Scheme The policy iteration algorithm may be computationally intensive. Hence, we propose a Q-learning algorithm as an alternative approach to solve the problem. We adopt the Q function from [], and it can be approximated as follows, Qk+ (, s a) = G(, s a) + Pr ( s, a s, a) max { Qk ( s, a )} k k = ( ) α + α k+ k+ Qs, a ( ) Gs (, a ) k+ k+ + max { Qs (, a )}, π (3) 4 IEEE Network May/June 3

4 Initialize: Q(s, a) =, G(s, a) =, ly choose a start action for k = to final time slot do Calculate the learning rate a for all actions a i Œ A(s) do Calculate Q(s, a i ) using Eq. 3 if Q(s, a i ) > Q(s, a) then Q(s, a) = Q(s, a i ) a = a i end if end for SU s action is a end for Algorithm. Q-learning algorithm. where a Œ [, ] is the learning rate. In this article, we use a linear learning rate defined as a = /k+. We describe how to calculate the Q function in Algorithm. Simulation Results Figure. Jamming probability under static attack, scenario Figure. Jamming probability under random attack, scenario. In this section, we evaluate the performance of the proposed through simulations. We compare the performance of to two existing schemes under three different types of jamming attacks. The first existing scheme is called the random scheme, where the SU has no idea of the history of each channel and randomly chooses one accessible channel at the beginning of each time slot. When choosing the channel, the SU would not consider the channel condition or the attacker s strategy. The second existing scheme is called the intelligent scheme, where the SU keeps a history of accessing/jamming states for all channels, and chooses the channel that is least jammed. We consider three types of jamming attacks. The first one is called static attack, where the jammer chooses J accessible channels at the beginning of each time slot. The second type of attack is called random attack, where the jammer chooses J accessible channels at the beginning of each time slot. These two types of attacks do not have the information of the state history of each channel. The third type of attack is called intelligent attack. The attacker keeps a history of the channel state, i.e. jammed or not, and picks the channels that have the highest probabilities of being jammed. The performance metric we use is the jamming probability, i.e. the probability that the SU is jammed. The presented results are the average values from dozens of simulations using different seeds. We consider two scenarios. We first examine the performance of under scenario. In this scenario we assume one PU in the network. We assume that the transition probabilities of each channel from busy to idle and from idle to busy, i.e. p and p, are both equal to.5. There are channels in the licensed band, i.e. N =. We first assume that there is one SU in the network, i.e. H =. During each time slot, the SU accesses one channel, i.e. L =. We set p stay =.37. We assume that the PU is occupying its channel all the time, i.e. M =. The SU stays in a channel with p stay when the channel is accessible in the next time slot, while it hops to each of the other accessible channels with Pr(s, a Ís, a) =.63 /N M. As we assumed earlier, there is one attacker and the attacker can jam up to three channels at each time, i.e. J = 3. Figure plots the jamming probabilities of the random scheme, intelligent scheme, and under the static attack in scenario. At the beginning time slots, these three schemes have similar jamming probabilities. However, when the time progresses, the jamming probabilities of both the intelligent scheme and decrease. In particular, the jamming probability of becomes negligible after time slot 8, while the jamming probability of the random scheme does not change much. This is because the random scheme does not take any feedback into consideration from previous actions. The jamming probabilities of the three schemes under the random attack are plotted in Fig.. Similar to the case under the static attack, the jamming probabilities of the three schemes are around the same at the beginning. As time elapses, both the intelligent scheme and have lower jamming probabilities. However, the performance of is better than that of the intelligent scheme. Figure 3 illustrates the jamming probabilities of the three schemes under the intelligent attack. Comparing the results in Fig. 3 with the results in Fig. and Fig., we can see that the performance of the intelligent scheme under intelligent attack is worse than the performance under the static attack or random attack. The reason is that the attacker is also intelligent, i.e. it also learns from the accessing/jamming history of each channel. Hence the performance of the intelligent scheme degrades. However, the performance of is still very good even under the intelligent attack. Next we examine the performance of in the second scenario. In this scenario, the transition probabilities of each channel from busy to idle and from idle to busy are both assumed as.5. There are 5 channels in the licensed band, IEEE Network May/June 3 5

5 Figure 3. Jamming probability under intelligent attack, scenario Figure 5. Jamming probability under random attack, scenario Figure 4. Jamming probability under static attack, scenario. Figure 6. Jamming probability under intelligent attack, scenario. i.e. N = 5, and three SUs in the network, i.e. H = 3. During each time slot, each SU accesses three channels, i.e. L = 3. p stay is still set as.37. There are two attackers in the network and each attacker can jam up to three channels at a time, i.e. J = 3. Figure 4 plots the jamming probabilities of the three schemes in scenario. We can see that the jamming probabilities are higher than the ones in the first scenario shown in Fig.. For example, the jamming probability of the random scheme is around.385, while in Fig. it is around.3. The reason is that there are more jammers in the network. Therefore, it is more difficult for the SUs to defend the jamming attack. However, still achieves the best performance. Figure 5 and Fig. 6 illustrate the simulation results under the random attack and the intelligent attack. achieves better performance than the other two anti-jamming strategies under these two attacks too. Conclusion and Future Directions In this article, we have discussed the challenges in defending jamming attacks in cognitive radio networks. During the antijamming process, the SUs proactively hop among accessible channels to defend against jamming attacks. This jamminghopping process is formulated as a Markov Decision Process. We have presented a game-theoretical anti-jamming scheme, called. The probability of jamming is chosen as the performance metric to evaluate our proposed scheme. Performance evaluations demonstrate that can achieve higher payoff than existing approaches and lower the jamming probability. As one possible future direction, we will study how to distinguish other factors for the reduced SNR from the jamming attack, to design a more robust anti-jamming scheme. References [] M. Song et al., Dynamic Spectrum Access: From Cognitive Radio to Network Radio, IEEE Wireless Commun., vol. 9, no.,, pp [] W. Xu et al., The Feasibility of Launching and Detecting Jamming Attacks in Wireless Networks, Proc. 6th ACM Int l. Symp. Mobile Ad Hoc Networking and Computing, 5, pp [3] R. Pickholtz, D. Schilling, and L. Milstein, Theory of Spread-Spectrum Communications A Tutorial, IEEE Trans. Commun., vol. 3, no. 5, 98, pp [4] V. Navda et al., Using Channel Hopping to Increase 8. Resilience to Jamming Attacks, Proc. IEEE Infocom, 7, pp [5] B. Wang et al., An Anti-Jamming Stochastic Game for Cognitive Radio Networks, IEEE JSAC, vol. 9, no. 4,, pp [6] Q. Wang, K. Ren, and P. Ning, Anti-Jamming Communication in Cognitive Radio Networks with Unknown Channel Statistics, Proc. IEEE Int l. Conf. Network Protocols,, pp IEEE Network May/June 3

6 [7] H. Li and Z. Han, Dogfight in Spectrum: Jamming and Anti-Jamming in Multichannel Cognitive Radio Systems, Proc. IEEE Globecom, 9. [8] G. Noubir et al., On the Robustness of IEEE 8. Rate Adaptation Algorithms Against Smart Jamming, Proc. 4th ACM Conf. Wireless Network Security,, pp [9] W. Xu et al., The Feasibility of Launching and Detecting Jamming Attacks in Wireless Networks, Proc. ACM MobiHoc, 5. [] J. Nash, Non-Cooperative Games, Annals of Mathematics, vol. 54, no., 95, pp [] M. Lagoudakis and R. Parr, Value Function Approximation in zero- Sum Markov Games, Proc. 8th Conf. Uncertainty in Artificial Intelligence,, pp Biographies CHANGLONG CHEN received his BS degree and MS degree in Telecommunication Engineering from Jilin University, Changchun, China, in 5, and in 9, respectively. He is currently a Ph.D. candidate in Electrical Engineering and Computer Science at the University of Toledo, Toledo, OH. His research interests include network security, cognitive radio networks, and performance analysis. MIN SONG [SM] (min.song@utoledo.edu) is a Professor in the Electrical Engineering and Computer Science Department at the University of Toledo. Professor Song is currently serving the NSF as a Program Director. He received his Ph.D. in Computer Science from the University of Toledo in. Over the years, he has secured more than $ million research funding from NSF, DOE, and NASA. Dr. Song is the recipient of CAREER Award. CHUNSHENG XIN [M] (cxin@ieee.org) is an Associate Professor in Electrical and Computer Engineering Department of Old Dominion University. He received his Ph.D. in Computer Science and Engineering from the State University of New York at Buffalo in. His research interests include cognitive radio networks and network security. His research is supported by several NSF grants and published in leading technical journals and conferences. JONATHAN BACKENS (jback6@odu.edu) received his B.S. degree in Computer Engineering and Computer Science from Christopher Newport University in 4. He is currently a Ph.D. candidate in Electrical and Computer Engineering at Old Dominion University in Norfolk, Virginia. His research interest include rural wireless mesh networking, game theory in communication systems, cognitive radio networks and performance analysis. IEEE Network May/June 3 7