Distributed Learning for Multi-Channel Selection in Wireless Network Monitoring

Transcription

1 Distributed Learning for Muti-Channe Seection in Wireess Network Monitoring Yuan Xue, Pan Zhou, ao Jiang, Shiwen Mao and Xiaoei Huang Department of Computer Science and Engineering, Lehigh University, Bethehem, PA, USA Schoo of Eectronic Information and Communications, Huazhong University of Science & echnoogy, Wuhan, China Department of Eectrica and Computer Engineering, Auburn University, Auburn, AL, USA Emai: Abstract In this paper, we address an important probem in the wireess monitoring, i.e., how to choose channes with best or worst quaities timey and accuratey. We consider both scenarios of one or more sniffers simutaneousy monitoring mutipe channes in the same area. Since the channe information is initiay unknown to the sniffers, we sha adopt earning methods during the monitoring to predict the channe condition by a short time of observation. We formuate this probem as a nove branch of the cassic muti-armed bandit MAB probem, named exporation bandit probem, to achieve a trade-off between monitoring time/resource budget and the channe seection accuracy. In the mutipe sniffer cases, incuding party-distributed with imited communications and fuy-distributed without any communications scenarios, we take communication costs and interference costs into account, and anayze how these costs affect the accuracy of channe seection. Extensive simuations are conducted and the resuts show that the proposed agorithms coud achieve higher channe seection accuracy than other exporation bandit approaches, hence it proves the advantages of the proposed agorithms. I. INRODUCION o dea with the compex environment of wireess networks, dedicated devices such as sniffers [] are impemented to monitor the wireess channe activities e.g., transmission rate and quaity of service QoS. hese devices coect detaied information incuding PHY/MAC characteristics of the wireess network, which are critica for network management and diagnosis. However, due to the imitation of hardware devices, one sniffer cannot monitor a channes in its vicinity at one time. Network monitoring is very important in many type of scenarios, for exampe, the IEEE 802. protoco divides spectrums into mutipe channes, for which mutipe sniffers are needed. Generay, mutipe monitoring devices are depoyed in the same area to cooperativey monitor channes in this area. Unike previous works focusing on the channe assignment probem [2], [3] where different sniffers are in charge of different channes, we concentrate more on the channe seection in wireess monitoring, i.e., how to effectivey choose an optima or worst for network diagnose coection of channes by each sniffer independenty. Channe seection is an essentia probem of wireess monitoring, for instance, choosing channes Sniffer is a passive monitoring device. In this paper, for simpicity, we use the term sniffer to refer to both active and passive monitoring devices. with the highest transmission rates are hepfu for resource aocation and coud avoid wasting time/energy on bad channes. Meanwhie, identifying channes with unusua behaviors which may be caused by hardware faiures or externa attacks in a timey and accurate fashion pays an indispensabe roe in some mission-critica scenarios such as voice over IP VoIP, disaster recovery and miitary appications. We investigate the probem of distributed wireess monitoring in this paper. When mutipe sniffers coexist in the same area, our goa is to figure out how to coordinate these sniffers and make the optima seection of channes without a centra controer. Initiay, the channe activities are unknown to the sniffers, i.e., they have to coect channe information constanty by themseves during a sequentia earning process. Intuitivey, if we spend more time on monitoring, we wi obtain more accurate resuts. Unfortunatey, in most cases, we do not have an infinite time for channe seection, therefore a time budget is needed. As a resut, there is a trade-off between competing the monitoring quicky versus making more accurate seections. his non-trivia probem inspired us to formuate it as a distributed exporation bandit or pure exporation probem, which is a subcass of the cassic mutiarmed bandit MAB probem []. In contrast to standard MAB agorithms such as UCB [5], which are evauated in terms of regret, 2 pure exporation methods focus on finding the arms with the maximum expected rewards rather than maximizing arm rewards during the earning process. Firsty, we study the channe seection probem with a singe sniffer. Athough the sniffer can ony monitor one channe at a time, it can switch its target constanty. hus, if we coud predict channe condition by a short time of observation, we coud use one sniffer to monitor mutipe channes. We assume that there is a time budget for the sniffer, so it has to make decisions within the given time horizon. We then mode this probem as a mutipe identification probem with a fixed budget []. We propose an eimination-based earning agorithm, which aows the sniffer to reduce the number of channes to monitor constanty during the iterations. We show that the proposed singe sniffer agorithm has a negative- 2 Regret is defined as the difference between the expected rewards of the optima strategy made by a genie and that of the given poicy //$ IEEE

2 exponentia error probabiity 3 over time. Next, we consider the case where mutipe sniffers coexist in the same area. If they work for the same administrator, they can share some information to obtain more accurate monitoring resuts. Nevertheess, due to the geographica separation of the sniffers, this cooperation incurs a communication cost. For such a scenario, we propose a distributed exporation bandit agorithm with imited communications. For passive sniffers, there wi not be any inferences when they observe the same channe, but they sti have communication cost then they communicate with each other. As for depoying active monitoring devices to monitor the network, interferences between them may hurt their abiity of getting correct monitoring resuts. We name this kind of interference as a coision. Both communication cost and coision are taken into consideration in our anaysis. In another practica scenario, different sniffers might be operated by different administrators. So they cannot communicate with each other, and sti have to suffer the oss of accuracy caused by coisions if they are active devices. hus, we propose fuy distributed agorithms for this scenario, in which sniffers have no need to communicate with each other. he main contributions of this work are described beow: We address the channe seection in wireess monitoring by formuating it as an exporation bandit probem. Both singe sniffer and mutipe sniffer agorithms are proposed for different scenarios. We investigate the distributed exporation bandit probem, mainy in the fixed budget setting. his technique coud have ots of appications but hardy any work have been done in this area. Communication costs and coision oss are aso taken into consideration in this paper. wo we-designed agorithms are proposed to mitigate the negative impact of coisions by aowing imited communications. he rest of this paper is organized as foows. In Section II, reated works on wireess monitoring and distributed earning are discussed. Section III introduces the probem formuation of distributed channe seections in wireess monitoring. Agorithms and anaysis are presented in Section IV and V, respectivey. Simuation resuts are avaiabe in Section VI. We concude this paper in Section VII. II. RELAED WORKS Many previous works have been done on wireess monitoring, most of which attempted to design a compete monitoring system. In [], for the first time, Yeo et a. introduced passive monitoring for the IEEE 802. based wireess networks with mutipe sniffers. Deshpande et a. aso studied the mutichanne wireess monitoring probem in [7], where they mainy focused on achieving higher observed frame rates with the proposed channe samping strategies. In [2], Zheng et a. 3 Error probabiity is defined as the probabiity of choosing any suboptima channes during the monitoring. proposed onine earning agorithms to sove the channe assignment probem of wireess monitoring in unknown environments. We aso adopt onine earning techniques in our work but the probem is totay different. Unike [2], we address the channe seection probem of the wireess monitoring. We aso propose a channe assignment agorithm to sove the distributed mutipe channe seection probem. Chen et a. in [3] aso studied the channe seection probem in wireess monitoring. However, they choose channes for different sniffers to aow them to monitor mutipe channes together. In their optimization probem, a centra controer is required and each sniffer can aways communicate with other sniffers. he centra controer is not needed in our work, moreover, we consider the communication cost in distributed wireess monitoring. In the area of onine earning, muti-armed bandit MAB probem has drawn a ot of attention in recent decades. MAB is a cassic exampe of tradeoff between exporation and expoitation, which aims to achieve the maximum cumuative sum of rewards in the earning process. Lai & Robbins proposed an index poicy in [] with a ogarithmic regret bound and Auer et a. [5] introduced the we-known UCB strategy which achieves Oog regret uniformy over time. Consider the arge amount of channes and dynamic environment, we introduce an innovative scheme with exporation bandit techniques in this paper. Exporation bandit is a new branch of MAB and it can be mainy divided into two categories: fixed confidence setting and fixed budget setting. In the fixed budget setting, payers shoud seek for a singe best arm or a best subset of arms with a fixed time budget. In another setting of the exporation probem named fixed confidence setting, such as Even-Dar et a. in [8] and Kayanakrishnan et a. in [9], payers aim at reducing the number of sampes i.e., simpe regret in [0] to satisfy the specific constant of finding near-optima arms. Compared with cassic MAB methods, exporation bandit methods have more accurate resuts in the channe seection of wireess monitoring because they spend more time on bad channes. hus we adopt exporation bandit agorithms in this paper. In the exporation bandit probem, many works on the mutipe identifications probem EXPLORE-m in [] has been done such as [] and [9]. Nevertheess, none of them has considered subset seection with different payers. Different from the existing iterature, we study the chaenging issue of mutipe identifications in the distributed setting. here have aso been considerabe efforts on distributed earning techniques of MAB. In area of distributed earning, Liu and Zhao [2] introduced the time-division fair sharing DFS poicy for a centraized time sharing schedue for mutipe users. ekin and Liu [3] utiized the regenerative property of markov chain to sove the probem of rested and restess MAB probems with mutipe payers. Kaathi et a. in [] proposed an agorithm based on the Bertsekas auction agorithm, which has Oog 2 t regret bound due to the communication cost. Distributed exporation bandit was studied by Hie et a.

3 in [5]. heir work is most reated to this paper. However, our work is competey independent with respect to [5]. Hie et a. studied the probem of distributed exporation in the fixed confidence setting whie we mainy focus on the fixed budget setting. hey speed up the earning process via communications among sniffers. In contrast to their work, both agorithms with and without communications are proposed in this paper. III. PROBLEM FORMULAION In this section, we present the probem formuation of channe seection in wireess monitoring. First, we discuss the formuation of the singe sniffer and mutipe sniffer probems. A. System Mode Consider one or more sniffers monitors K wireess channes in some area. When the scae of the network K is very arge, the administrator needs to focus on a subset of best or worst channes to manage the network more efficienty. hus, the sniffers have to choose M best M <K channes out of K channes within the time budget. According to different usage, the time budget can be different and the sniffers can finish the channe seection with or without communications with each other. We assume that the sniffer can ony monitor one channe at a time. In the singe sniffer scenario, the sniffer is given a time budget firsty, then it has to choose M channes within time. After finishing the channe seection, the sniffer wi transmit the resuts to the administrator of this network for further monitoring work. In the mutipe sniffers scenario, each sniffer wi yied a compete outcome of M channes independenty. Coisions wi happen if sniffers are active monitoring devices. Assume there are n sniffers and n M, if a the sniffers are operated by the same administrator, they can communicate during the monitoring process to avoid coision among themseves. Communication costs wi hurt the reward achieved by sniffers, and further affect the accuracy of seection. Under the fuydistributed scenario, the sniffers do not communicate with each other. hey wi make decisions based on their past monitoring resuts. Aso, they wi choose M channes within the time budget independenty. Note that in fuy distributed monitoring cases, coisions wi happen. Next, we introduce some notations and compexity measurements in this paper. B. Notations and Compexity Measurements Consider K channes to be monitored in a wireess network, where K = {,...,K} is the set of channes. For simpicity, we assume that each channe j s activity in wireess network foows an i.i.d. distribution with density function fx; θ j, whie the parameter θ j is not known apriori. Each time the sniffer measures the channes, it wi obtain a reward. Reward For simpicity, we ony use the optima channes seection assumption in this paper. Note that a agorithms and anayses can be easiy extended to the worst channe seection scenario. here contains information of channe activities measured by the sniffer. When there are mutipe sniffer in the system, et N = {,...,n} denote the set of sniffers. Let φ be the monitoring poicy adopted by the sniffers. During the channe seection process, a the sniffers have the same time budget and their cocks are synchronized. Error probabiity e φ i is the probabiity of choosing suboptima channes by sniffer i under poicy φ. Assume that each channe j has a mean reward μ j, which is the mean of random variabe X j t. We rank them in a descending order, i.e., μ >...>μ K. We aso introduce the notation of hardness H M in this subsection for preparation of the proof. We define the gaps and the compexity measures of the distributed channe seection in wireess monitoring probem as foows. Δ M j = μ M μ j, Δ min = min μ k μ j, 2 j<k K H M = K j= Δ M j 2, 3 H2 M j =max, j K Δ M j 2 where the notation j {,...,K} is determined by order of Δ M... Δ M K. hese notations decide the hardness of finding the optima channes during the channe seection process, we wi discuss more detais in the foowing sections. IV. SINGLE SNIFFER MONIORING In this section, we introduce a nove singe sniffer agorithm named Sequentia Mutipe Eimination SME for channe seection in wireess monitoring. Detais of the proposed agorithm are given in Agorithm. In SME, different from previous eimination-based agorithms, we aow sniffers to eiminate mutipe channes in each round where each round divide the time budget equay. he number of eiminated channe is a quarter of ast round, to et sniffers observe channes more times when it is harder to distinguish distinguish good channes from bad channes, because the reward gap between different channes become smaer as exporation time goes by. We pick a quarter because after severa experiments, we found that this is a reativey good choice which is neither too big nor too sma, a parameters are designed according to this principe. First, we divide the channe seection process into rounds and et = og 3K M+. hus, in round, the sniffer wi remove K = 3K M+/ channes with the owest empirica rewards from the remaining active channe set A, and put them into set K.Itfoowsthat K 3 3K M + =K M. 5 = Let A = A and A = K M M 3, for a. After rounds, the sniffer wi provide the resut of M chosen channes. SME aows sniffers to reduce the number of sampes constanty during the process of channe monitoring, and guarantees each channe has been samped enough times before been dropped. o cacuate the error probabiity of SME, we introduce a emma first.

4 Agorithm Sequentia Mutipe Eimination for Singe User SME : Input : K channes, M chosen channes, time budget, t = A, = og 3K M+, A = A, A 0 = K, K = 3K M+. 2: Output : M channes with highest empirica rewards. 3: for each =, 2,..., do : Sampe a channes in A for t times; 5: Rank these channes according to their empirica rewards, et ˆμ >...> ˆμ A ; : Eiminate a the channes in K = {j :ˆμ j ˆμ A K }, A + = A /K. 7: end for 8: Return A. Lemma. In SME, assume a channe p outside M is not eiminated before round. hen in round, for channe j M, the probabiity of ˆμ j < ˆμ p satisfies P[ˆμ j < ˆμ p ] 2 t Δ M j +Δ M p 2. Proof: Let Δ jp =Δ M j +Δ M p when j M and p>m. For α>0 and β>0, by the Chernoff-Hoeffding inequaity, we have 2 t αδ jp 2 P[ˆμ p >μ p + αδ jp ] exp P[ˆμ j <μ j βδ jp ] exp 2 t βδ jp 2. Since Δ jp =Δ M j +Δ M p = μ j μ p,wehave P[ˆμ j < ˆμ p ] 2 t αδ jp 2 +βδ jp 2 t Δ M j +Δ M p 2, where the ast inequaity is due to the fact that if α + β, then α 2 + β 2 2. From Lemma we know that under the poicy of SME, when the mean reward gap between good channes and bad channes Δ jp is very arge, the probabiity for the sniffer of identifying the optima channes becomes very high. Besides, if the sniffer spends more time on monitoring, it wi achieve ower probabiity of choosing bad channes. hen we can derive an upper bound for the error probabiity of SME. heorem. he error probabiity of SME is upper bounded as e 3 M K M og 3K M+. 7 H 2 og 3K M+ Proof: Assume that channe j Mis not eiminated by SME in the first rounds. hen in round, the probabiity for channe j of being eiminated satisfies e P[ˆμj < ˆμp] j M A p K j M A t Δ 2 jp p K j M A M t Δ 2 ja MA Mmaxexp MA Mexp H 2 hen for a rounds, we have e MA M =0 H 2 = 3 MK M H 2 Δ M A A M K M og 3K M+. 9 H 2 og 3K M+ heorem shows that the error probabiity of SME is Oe with respect to time budget. So as the time budget grows, the error probabiity of SME decreases exponentiay. Aso, SME needs at most Oog K ogk og K times of observation with respect to number of channes K to identify the optima channes, which is much smaer than previous agorithms such as SAR []. hus, SME has better performance in the channe seection of wireess monitoring. V. DISRIBUED MONIORING In this section, we examine the scenarios where mutipe sniffers monitor the channes in the same area. Assume there are n sniffers monitoring K channes simutaneousy, each channe s information is initiay unknown to a the sniffers. Each of the sniffers wi provide a compete set of M chosen channes after a period of time. A new chaenge for this mutipe sniffer scenario is that when more than one active sniffers are on the same channe, the interference between them wi hurt the rewards observed by these sniffers. Such coisions shoud be considered in the design of mutipe active monitoring devices schemes. Based on our singe sniffer agorithm, we first introduce two coision-free agorithms which are for sniffers with imited communications. hen we study the fuy distributed circumstance in which sniffers do not change information with each other. A. Party Distributed Agorithms with Limited Communications We assume that there are n sniffers monitoring the same K channes simutaneousy for the same administrator. hey can communicate with each other to avoid the coision among sniffers. However, the communication cost wi degrade the accuracy of the the resuts gained by the sniffers. So it woud be costy for the sniffers to keep on exchanging information with each other. During the earning process, each sniffer has to

5 Fig.. Channe Seection Mode of Agorithm 2. make their own decision with imited hep from other sniffers. We proposed two distributed exporation bandit agorithms in this section to sove this probem. In the proposed agorithms, communication cost for each sniffer is taken into account. First, we propose an agorithm named Distributed Sequentia Mutipe Eimination with Virtua Channes DSME-VC in Agorithm 2. In DSME-VC, we aso eveny divide the time budget into rounds. he eimination process is the same as the SME agorithm. Next, the sniffer wi put its chosen channes into a channe set V, which we ca the virtua channe set. he sniffer wi spend time on virtua channes but they wi not coect any channe information. Meanwhie, the sniffer wi broadcast its chosen channes to a other sniffers. If one channe is chosen by a the sniffers, then the sniffer wi remove this channe from V. As iustrated in Fig., with a round-robin fashion time aocation poicy, when the sniffer 2 is assigned to monitor a virtua channe 2, it won t coect any information about channe 2 and just rest for time t. his strategy can competey avoid the potentia coisions among sniffers. For different sniffers, they may choose different channes in the same round, so we shoud cacuate the probabiity for a channe of being chose by a the sniffers. hen we can compute the expectation of the remaining channes in each round. In DSME-VC, we competey avoid the potentia coisions among sniffers by introducing the virtua channes. A sniffer never reay drops a channe uness it thinks the channe is seected by a sniffers simutaneousy. Compared with the singe sniffer agorithm, DSME-VC wastes some time on virtua channes, which is an inevitabe cost for avoiding coisions. At round, assume the communication cost for each sniffer is c. Since we divide into rounds, the tota communication cost of rounds is C = c. Actuay, the communication cost wi hurt the resuts observed by the sniffers, and wi affect the accuracy of monitoring resuts. o cacuate the communication cost of DSME-VC, we firsty define the compexity communication cost as c 0 = max {c i,j}, 0 i N,j K where c i,j is the communication cost for sniffer i of exchanging information about channe j with another sniffer. c 0 here refers to the maximum communication cost for a singe sniffer and singe channe. Consider n sniffers monitor mutipe channes simutane- Agorithm 2 Distributed Sequentia Mutipe Eimination with Virtua Channes DSME-VC : Input : K channes, M chosen channes, n sniffers, time budget, = og 3K M+, A = A, A 0 = K, K = 3K M+, k =0, V =. 2: Output : M channes with highest empirica rewards. 3: for each =, 2,..., do : for each sniffer i N do 5: whie k K do : if the channe [i + + k modk ] beongs to A then 7: Sampe it for t = A V times; 8: end if 9: k := k +; 0: end whie : Rank these channes according to their empirica rewards, et ˆμ >...> ˆμ A ; 2: Choose a the channes in K = {j : ˆμ j ˆμ A K }, A + = A /K. Broadcast chosen channes to other sniffers; 3: Eiminate channes chose by a sniffers, put others back to V as virtua channes. : end for 5: end for ousy, then sniffer i s communication cost in round is c i, = n K j= c i,j. hus, the tota communication cost for sniffer i is upper bounded as K C i = c i, = n c i, n K c 0 = nk Mc 0. = = j= = In the foowing, we cacuate an upper bound for the probabiity for any channe j to be seected in round. hen we wi prove the expected number of channes chosen by a the sniffers. Lemma 2. In DSME-VC, he probabiity for channe j outside K to be chosen by one sniffer in round satisfies P[j A i, ] K exp Δ 2 min K exp Δ 2 min. Proof: Based on Lemma and a union bound, we have P[j A i, ] P[ ˆμ j < ˆμ k ] A <k A A P[ˆμ j < ˆμ k ] k>a A exp Δ jk 2 t k>a A A exp Δ ja 2 t. For any j<k,wehave P[j A i, ] A A exp Δ ja 2 t A A exp Δ A Δ A 2 t A A exp Δ 2 mint A exp Δ 2 mint A exp Δ 2 mint. 2 With the concusion of Lemma 2, we can now prove the

6 expected number of channes chosen by a the sniffers. heorem 2. In DSME-VC, the expectation of number of channes chosen by a sniffers in the round satisfies n E[N] A A A exp Δ 2 min A A K N. 3 Proof: For a channe inside K, the probabiity of being chosen by one sniffer is at east A j= P[j A i, ]. hen we have n A P[j A i, ] E[N] A A A A n A A A exp Δ 2 mint, A A where foows from the ast inequaity in 2. Since t = /K r= N r /K, we have heorem 2. In heorem 2, the probabiity for a the sniffers choose the same channe grows exponentiay as the increase of the number of sniffers n. his fact guarantees that when there are many sniffers in the same area, DSME-VC wi not have too many virtua channes. Next, we derive an upper bound on each sniffer s error probabiity for Agorithm 2. heorem 3. he error probabiity of DSME-VC for each sniffer satisfies M e MA M H = 2K N + N, 5 where N is A /A A A exp Δ 2 min/k n. Proof: First, we wi modify the time aocation poicy with respect to Agorithm. = t r K, r Nr r= r= from heorem 2 we have that n N r K 3K M+ A exp Δ2 min K r n A A exp Δ 2 min A A K. With 8 in heorem, the error probabiity in round is upper bounded as A e MA M H 2K r Nr M MA M H 2K N N. 7 hen, the tota error probabiity of Agorithm 2 is e = e M MA M = H 2K N + N. Compared with heorems, heorem 3 shows that the upper error probabiity bound of DSME-VC is bigger than SME. However, it competey circumvents the coisions among sniffers and when K is not very arge, the error probabiity of DSME-VC is quite cose to SME. So the DSME-VC agorithm is appicabe in the distributed monitoring when the number of sniffers is reativey sma. Fig. 2. he negotiation process between sniffers in Agorithm 3. Agorithm 3 Distributed Auction-based Channe Assignment Agorithm DACA : Input : K channes, M chosen channes, n sniffers, time budget, t = K, = og 3K M+, A = A, A 0 = K. 2: Initiaization : For a random channe j, sniffer i provides apricep ij randomy, communicate p ij with other sniffers. 3: whie <do : whie k<a do 5: For sniffer i, etj =argmax j Sk, ˆμ i,j ; : if i =argmax i N p ij then 7: Sniffer i sampes channe j for t times, broadcast to a other sniffers that it has finished this round, then waits for other sniffers; 8: ese 9: Move to another channe s S k, randomy, et p is = ˆμ i,s, communicate with other remaining sniffers, go back to step ; 0: end if : Let S k, := S k, /j, k := k +; 2: end whie 3: Update the empirica reward ˆμ i,j for a j A. Eiminate channes in K = {j : ˆμ j ˆμ A K }, A + := A /K,k :=, := +, S k, := A. : end whie 5: Return A. We aso propose a distributed agorithm without using virtua channes in Agorithm 3 named Distributed Auctionbased Channe Assignment DACA. Athough DSME-VC soves the potentia coision probem, when K becomes very arge, DSME-VC may waste too much time on virtua times. So Agorithm 2 is not efficient enough when K is very arge. DACA soves this probem without using the virtua channes. Assume there is an undirected bipartite graph GS, U, E, where S and U are the set of sniffers and channes, respectivey. E stands for the connection between sniffers and channes. When the sniffer i eiminates channe j, the edge Ei, j wi aso be removed from set E. So the sniffer wi ony provide a price to the channe in its active set. During the communication with others, the sniffer wi decide whether to observe the channe or not. If sniffer i is not the highest bidder of channe j, it wi move to another channe randomy. For exampe, in Fig. 2, three sniffers are monitoring three channes. In the first communication round, both sniffer and

7 2 provide price to channe and sniffer 3 provides its price to channe 3. After communication with each other, sniffer and 3 finds itsef to be the highest bidder of channe and 3, respectivey. In the second round, sniffer 2 gives its price to channe 2, after communicate with sniffer, the sniffer 2 starts to coect information of channe 2. his process asts unti each of the sniffers finds a channe. After a channe is observed by channe, it be wi removed from set E temporariy before the next round. Since the channe seection process of DACA is the same as SME, the error probabiity of Agorithm 3 wi aso be MK M 2 og 2K M+ /3 exp /H 2 og 3K M+. hus, compared with heorem 3, the performance of DACA wi be better than DSME-VC. However, to better evauate DACA, we shoud take communication cost among sniffers into consideration. hen we wi derive the communication cost of DACA. First, we introduce a emma to bound the number of communications in DACA. Lemma 3. Number of communications for each sniffer in DACA is at most K M + M n 3 n og 3 3K M+. Proof: We consider a worst case of Agorithm 3. When the sniffer i communicates with other sniffers in the negotiation phase of th round, if he aways fai to provide the highest price, he has to keep communicating with a remaining users, then the number of communications is at most n n 2 A = n3 n A. 8 hen the tota number of communications is n 3 n A = K M + M n 3 n 3 = K M + M n 3 n og 3 3K M+. With the resut of Lemma 3, we come to a concusion that the communication cost of the DACA agorithm is at most c 0 K M + M 3 og 3K M+ n 3 n. Lemma 3 shows that when the number of sniffers become very arge, the DACA agorithm has much more communication cost than the DSME-VC. he numerica resuts of communication cost caused by DACA and DSME-VC can be found in section VI. Since the channe seection process of DACA is the same as SME, the error probabiity is aso the same as SME in heorem. Compared with heorem 3, DACA has ower error probabiity bound than DSME-VC. his is because DACA does not use virtua channes so sniffer can sampe each channe for more time than DSME-VC, thus it yieds more accurate resuts. B. Fuy Distributed Agorithms without Communications Under the fuy distributed scenario, a the sniffers cannot have any communications during the monitoring process. Agorithm Fuy Distributed Sequentia Mutipe Eimination FDSME : Input : K channes, M chosen channes, n sniffers, time budget, t = K, = og 3K M+, A = A, A 0 = K, K = 3K M+. 2: Output : M channes with highest empirica rewards. 3: for each =, 2,..., do : whie m< A do 5: Choose one channe in A randomy, sampe it for t times; : Choose another channe randomy, m := m +; 7: end whie 8: Rank these channes according to their empirica rewards, et ˆμ >...> ˆμ A ; 9: Eiminate a the channes in K = {j :ˆμ j ˆμ A K }, A + = A /K, m :=. 0: end for : Return A. his maybe because they are on a miitary mission where communications are dangerous or they just work for different administrators so cannot exchange information with each other. In this subsection, we propose two fuy distributed agorithm without any communications. We first propose an agorithm named Fuy Distributed Sequentia Mutipe Eimination FDSME in Agorithm. he idea of FDSME is intuitive: just adapt SME to the distributed setting. When n sniffers monitor K channes of some network, they just pick channes in their own active channe set foowing a random order. In FDSME, when mutipe sniffers are active monitoring devices, coisions among them may happen. hen we prove the upper bound of coision probabiity given in Lemma : Lemma. In FDSME, the probabiity of at east one channe is chose by mutipe sniffers simutaneousy satisfies P[c] C j + n C j n nc j n, 9 where C j is K exp Δ 2 min t M exp Δ 2 min t. Proof: First, assume that there are no coisions before round. In the th round, when some channe j outside K is chosen by more than one sniffer, a coision happens. According to Lemma 2 and a union bound, the probabiity for each sniffer to choose channe j outside K satisfies P[j A i, ] A exp Δ 2 mint A exp Δ 2 mint. hen we have that each sniffer s probabiity of choosing channe j at east once in a rounds is [ ] P j A i, P[j A i, ] < K exp Δ 2 mint M exp Δ 2 mint C i,j. 20 Next, we cacuate the coision probabiity for channe j in a rounds: P[j] P[ j k Ai] n k N i=2 i=2 Ci nc i,j i =C j + n C j n nc j n.

8 Agorithm 5 Fuy Distributed Sequentia Mutipe Eimination with Virtua Channes FDSME-VC : Input : K channes, M chosen channes, i N, time budget, t = 3/K+, = og 3 K +, k =, A = K, A = A, K = K M, V =.. 2: Output : M channes with highest empirica rewards. 3: for each =, 2,..., do : whie k K do 5: Set j =[i + + k modk 2] : if j A then 7: Sniffer i sampes channe j for t times. 8: ese 9: Sniffer rests for t time. 0: end if : end whie 2: Rank these channes according to their empirica rewards, et ˆμ >...> ˆμ A. 3: Eiminate a the channes in K = {j :ˆμ j ˆμ A K }, A + = A /K, V := V K. : end for 5: Return A. Now we can cacuate the error probabiity of FDSME. heorem. In FDSME, the error probabiity for each user under fuy distributed scenario satisfies e MK M H 2 C j + n C j n nc j n. 2 Proof: When a coision happens, we assume that it wi hurt the channe rewards observed by the sniffer. According to Lemma, the number of conficting channe in round is at most K K exp Δ 2 min K exp Δ 2 min, so there are at most KC j + n C j n nc j n conficting channes in the whoe process. hen we modify heorem to get that e MA M MA M H 2C. H 2C j + n C j n nc j n hus, the tota error probabiity is e =0 e MA M =0 H 2C j + n C j n nc j n 3 M K M og 3K M+. 22 H 2C j + n C j n nc j n When K and n is sma, the FDSME agorithm has amost the same performance as SME. However, as the number of channes K becomes considerabe, the coision probabiity of FDSME cannot be ignored anymore. Sti, the error probabiity of FDSME is acceptabe even when K is very arge. We aso proposed another fuy distributed agorithm with virtua channes in Agorithm 5 named Fuy Distributed Sequentia Mutipe Eimination with Virtua Channes FDSME- VC. he channe choosing process is simiar to DSME-VC but the sniffer won t broadcast its monitoring resuts after each round. After a channe is chosen by the sniffer, it wi be abeed as a virtua channe. ime budget for each round is not the same. Assume the sniffer sampe a channes for t times in the th round, t = 3/K + and t = t /. hen we have that K = t, so we can deduce that = og 3/K +. Inth round, when the sniffer is assigned to monitor a virtua channe, it wi rest for t time and does nothing. After rounds, the sniffer yieds a compete resut of M chosen channes. heorem 5. he error probabiity of FDSME-VC for each sniffer is upper bounded as e og 3/K + K MM 2 K Δ2 max, 23 where Δ max =max j K {Δ M j }. Proof: he proof of heorem 5 is simiar to heorem. We omit the proof due to the space imitation. Compared with FDSME, the upper error probabiity bound of FDSME-VC is higher when K is not considerabe. However, FDSME-VC wi waste much time on virtua channes when K is very arge, so it has worse performance in the arge scae network depoyments. VI. SIMULAION RESULS In this section, we show the simuation resuts of the proposed agorithms in wireess monitoring. We compare our singe sniffer agorithm with the SAR agorithm in [], then we iustrate the performance of the proposed distributed agorithms. We consider a few different setups where number of channes and users varies from one to another. With out oss of generaity, we assume that each channe s reward is associated with an i.i.d. Bernoui distribution. When K = 9,M = 0 and K = 99,M = 5, the parameters of each distribution are Θ = 0.02, 0.0, 0.0,...,0.98 and Θ=0.0, 0.02, 0.03,...,0.99, respectivey. he regret here is defined as the difference between true means of optima M channes and that of channe chosen by the agorithms. In Fig. 3, SME has much smaer regret when compared with SAR, especiay when K is arge. When K =99, compared with SAR, SME improves the accuracy and efficiency for more than one hundred times. Even when K =9, the proposed SME agorithm is ten times better than SAR. In Fig., we can see that the DACA agorithm has ower regret than DSME-VC in both scenarios. When K = 99, compared with DSME-VC, DACA improves the accuracy of channe seection for more than 30%. hen we compare the communication cost for two agorithms in Fig. 5.

9 Regret 0 SME,K=9,M=0 9 SME,K=99,M=5 SAR[],K=9,M=0 8 SAR[],K=99,M= ime Budget x 0 Regret 0.7 DSME VC,K=9,M=0 DSME VC,K=99,M=5 0. DACA,K=9,M=0 DACA,K=99,M= ime Budget x 0 ABLE I SUMMARY OF HE PROPOSED ALGORIHMS Agorithm Distributed? Communications? Coisions? SME DSME-VC DACA FDSME FDSME-VC Fig. 3. Regret performances comparison for singe user. parison for mutipe users with com- Fig.. Regret performances communications. Number of Communications DSME VC,n=5 DACA,n=5 DSME VC,n=0 DACA,n= Number of Channes ime Budget x 0 Fig. 5. Expected communication Fig.. Regret performances comparison for mutipe users without cost for each sniffer. communications. Regret FDSME,K=9,M=0 FDSME,K=99,M=5 FDSME VC,K=9,M=0 FDSME VC,K=99,M=5 In Fig. 5, when the number of sniffers is 5, the different between two agorithms is not very arge, which indicates that the communication cost of DACA is acceptabe for reativey sma number of sniffers. However, when the n equas to 0, compared with DSME-VC, the communication frequency of DACA becomes very arge, which aso shows the advantage of DSME-VC for arge scae monitoring system. In Fig., we assume there are 0 sniffers monitor K channes simutaneousy. When K is reativey sma, the FDSME-VC agorithm outperforms the FDSME agorithm sighty. However, when K becomes arger, FDSME wins. When K =99, the accuracy of FDSME is about 0% higher than FDSME-VC. his is because FDSME-VC wastes too much time on virtua channes when K is very arge. he numerica resuts suggests that as the time budget grows, error probabiities for a agorithms decrease exponentiay, which is competey in conformity with our theoretica anaysis. More importanty, each agorithm has its advantages in specific scenarios. In summary, the simuation resuts prove the advantages of the proposed agorithms in the channe seection of wireess network monitoring. VII. CONCLUSION By modeing channe seection probem in wireess monitoring as an exporation bandit probem, we studied both singe sniffer and mutipe sniffer monitoring scenarios. As iustrated in abe I, a few singe or distributed exporation bandit agorithms are proposed in this paper for different practica scenarios. Simuations are conducted and the resuts iustrated the performance of the proposed agorithms in the mutipe channe seection. Both theoretica anaysis and simuation resuts show that the we-designed agorithms proposed in this paper have exceent performances in different scenarios of channe seection in wireess monitoring. As far as we know, no simiar work has been done before, so the proposed agorithms coud have great potentia appications in practice. ACKNOWLEDGMENS his research was supported by the Nationa Science Foundation of China under Grant 09 and NSF ECCS REFERENCES [] J. Yeo, M. Youssef, and A. Agrawaa, A framework for wireess LAN monitoring and its appications, in Proc. 3rd ACM Workshop on Wireess security, Phiadephia, PA, Oct. 200, pp [2] R. Zheng,. Le, and Z. Han, Approximate onine earning for passive monitoring of muti-channe wireess networks, in Proc. IEEE INFO- COM 203. urin, Itaey: IEEE, Apr. 203, pp [3] X. Chen, Y.-A. Kim, B. Wang, Y. Song, H. Dinh, and G. Chen, Sniffer channe seection for monitoring wireess ans, Esevier Computer Communications, vo. 35, no., pp , 202. []. L. Lai and H. Robbins, Asymptoticay efficient adaptive aocation rues, Esevier Advances in Appied Mathematics, vo., no., pp. 22, 985. [5] P. Auer, N. Cesa-Bianchi, and P. Fischer, Finite-time anaysis of the mutiarmed bandit probem, Machine Learning, vo. 7, no. 2-3, pp , May [] S. Bubeck,. Wang, and N. Viswanathan, Mutipe identifications in muti-armed bandits, in Proc. 30th Internationa Conference on Machine Learning ICML-3, Atanta, GA, June 203, pp [7] U. Deshpande,. Henderson, and D. Kotz, Channe samping strategies for monitoring wireess networks, in Proc. WiOP 200. Boston, MA: IEEE, Apr. 200, pp. 7. [8] E. Even-Dar, S. Mannor, and Y. Mansour, Action eimination and stopping conditions for the muti-armed bandit and reinforcement earning probems, Journa of Machine Learning Research, vo. 7, pp , Dec [9] S. Kayanakrishnan, A. ewari, P. Auer, and P. Stone, Pac subset seection in stochastic muti-armed bandits, in Proc. 29th Internationa Conference on Machine Learning ICML-2, Edinburgh, Scotand, June 202, pp [0] S. Bubeck, R. Munos, and G. Stotz, Pure exporation in muti-armed bandits probems, in Agorithmic Learning heory. Springer, 2009, pp [] S. Kayanakrishnan and P. Stone, Efficient seection of mutipe bandit arms: heory and practice, in Proc. 27th Internationa Conference on Machine Learning ICML-0, Haifa, Israe, June 200, pp [2] K. Liu and Q. Zhao, Distributed earning in muti-armed bandit with mutipe payers, IEEE ransactions on Signa Processing, vo. 58, no., pp , 200. [3] C. ekin and M. Liu, Onine earning of rested and restess bandits, IEEE ransactions on Information heory, vo. 58, no. 8, pp , 202. [] D. Kaathi, N. Nayyar, and R. Jain, Decentraized earning for mutipayer mutiarmed bandits, IEEE ransactions on Information heory, vo. 0, no., pp , Apr. 20. [5] E. Hie, Z. S. Karnin,. Koren, R. Lempe, and O. Somekh, Distributed exporation in muti-armed bandits, in Advances in Neura Information Processing Systems, 203, pp