Calibrating And Comparing Simulators for Wireless Sensor Networks

Transcription

1 211 Eighth IEEE International Conference on Mobile Ad-Hoc and Sensor Systems Calibrating And Comparing Simulators for Wireless Sensor Networks Andriy Stetsko, Martin Stehlík, Vashek Matyas Faculty of Informatics, Masaryk University, Brno, Czech Republic {xstetsko, xstehl2, Abstract In this paper, we present our findings from the calibration and comparison of selected simulators for wireless sensor networks. This work is motivated by our current research on a framework that optimizes a network-based intrusion detection system for a given application. For this purpose, we need a simulator that supports realistic models for topology, antenna, radio propagation, noise, radio, medium access control and energy consumption factors that can influence the performance of an intrusion detection system, which is intended to be run on the medium access control layer. In the paper, we consider four open-source simulators,, and. We compare these simulators and run a set of experiments on MICAz sensor nodes in the indoor and outdoor environment. Based on the data gathered from the real experiments, we calibrate the radio propagation and noise models of,, and. Also, we calibrate the energy consumption model of, and according to the MICAz datasheet. We present the results from the simulations and compare them between each other. Even though the simulators are set in the same way, their results significantly differ from each other. In the paper, we discuss possible reasons of the differences. Keywords-calibration; comparison; simulator; wireless sensor network. I. INTRODUCTION A wireless sensor network (WSN) consists of sensor nodes standalone devices equipped with a microcontroller, a radio, sensors and a set of batteries. The device measures a physical phenomenon and sends the measurements to a base station through its wireless interface. When a sensor node cannot directly communicate with a base station it uses hop-by-hop communication. WSNs are assumed to be used in different scenarios (e.g., emergency response, wildlife or battlefield monitoring) since they can be easily deployed in an area without additional investments to a cable installation. However, the advantage turns into a disadvantage when energy resources are considered. A common sensor node has only two AA batteries. When a WSN is assumed to be left unattended, which is a common case, energy saving becomes a tricky task. A lot of new protocols that take the specifics of WSNs into account has been proposed, e.g., medium access control (MAC) and routing protocols, key distribution and intrusion detection schemes. However, only a small number of them were tested on real hardware. When a testbed includes dozens or hundreds of sensor nodes, the testing becomes a significantly more time consuming process in comparison to simulations, which provide an easy and relatively fast evaluation of proposals. However, the simulation results are inaccurate (often, even misleading) when realistic models are not used or they are not properly calibrated. In [1], Kurkowski et al. surveyed 2-25 proceedings of the MobiHoc conference and presented shortfalls found in the simulation methodology. Model validation and verification was one of them according to the authors, many researchers download a simulator, compile it and begin to run simulations. The authors emphasized that a simulation model must be validated before any experiment takes place. In [2], Hurni et al. calibrated and validated the energy consumption model of OMNeT++ Mobility Framework for the WiseMAC protocol. In order to evaluate a protocol that runs at a certain layer, we need to calibrate and validate the layers underneath. In our research, we focus on intrusion detection systems (IDSs), which collect data at the MAC layer by overhearing the communication between the neighbors. Currently, we are working on a framework that optimizes an IDS for a given application. For more information on the conceptual design of the framework, see [3]. The optimization process requires to repeat the evaluation of an IDS with different settings. The optimization on real hardware is not considered due to several reasons. First, the evaluation on real hardware takes significantly more time than in a simulator. Second, the purchase of a large network is not always feasible due to the current price of a sensor node (about e8). As optimization metrics we choose accuracy (expressed by a number of false positives and a number of false negatives) and energy consumption. For more information on the metrics see [4]. Since the accuracy of a network-based IDS is highly dependent on its ability to overhear packets, its evaluation results depend on the proper calibration of the wireless channel, antenna and radio. Although we can find an accurate IDS, it may consume so much energy that nobody will deploy such an IDS in a real scenario. Hence, energy consumption should be taken into account and it should be properly modeled as well. To our best knowledge, the selection of proper models, calibration of these models remains an open issue in wireless sensor networks. In Section II, we present a theoretical comparison of four selected open-source simulators (,, and ), focusing on topology, an /11 $ IEEE DOI 1.119/MASS

2 tenna, radio propagation, noise, radio, medium access control and energy consumption modeling. In Subsection III-A, we proceed with a description of two experiments, which we run on MICAz sensor nodes in the indoor and outdoor environment in order to calibrate the radio propagation and noise models. The calibration process is described in the same subsection. In Subsection III-B, we describe an experiment in which we measure energy consumption of two sensor nodes running two different applications. We calibrate the energy consumption model of, and according to the MICAz and CC242 radio datasheets. In Section IV, we present and compare the results from the calibrated simulators. The possible reasons of the differences are mentioned in the same section. We conclude the work in Section V. II. THEORETICAL COMPARISON In this section, we present the theoretical comparison of the selected simulators with focus on topology, wireless channel, antenna, radio, MAC layer and energy consumption modeling. We selected the simulators based on the following criteria. First, it should support the simulation of WSNs. Second, it should be open-source. Third, the simulator should be actively supported by its developers. Although more simulators meet our criteria, due to the limitation on human/time resources, we consider only four of them (the most promising ones from our point of view) (version 3.), (version 2.1), (version that comes with TinyOS 2.1.1) and (version 9.7). emulates the TinyOS operating system and provides models for a network simulation. and are frameworks that are built on top of the OMNeT++ platform and extend it with the models for a simulation of WSNs. and are network simulators that do not emulate hardware or any of the existing operating systems. is also a network simulator but it can be used together with WSim a platform simulation tool which can run a native code generated for the target microcontroller (currently, only TI MSP43 is supported). A. Antenna All of the considered simulators only provide the model for an omni-directional antenna., however, provides a possibility for a user to implement own antenna model with a non omni-directional radiation pattern. Sensor nodes may integrate either an internal or an external antenna. For an external monopole or dipole antenna (e.g., a quarter-wave whip antenna used in a MICAz sensor node [5]), the horizontal radiation pattern is omnidirectional. However, for an internal inverted-f antenna used in a tmote sky sensor node, the horizontal radiation pattern is non omni-directional [6]. The simulation of a non omnidirectional antenna with a simplified omni-directional model might be inaccurate. B. Topology In, and a node location can be modeled by 3-dimensional Euclidean coordinates. supports only 2-dimensional coordinates. All of the considered simulators allow to set a topology manually or generate it automatically. can generate a topology where nodes are placed randomly (following a uniform distribution), into a grid or into a grid with variations. can generate a topology by setting node coordinates to randomly generated numbers. comes with the wsnet-topogen a tool intended for a topology generation. The tool allows to generate a topology with uniformly distributed nodes (for a given density), a static number of nodes, nodes placed in a form of grid. Additionally, the topology can be shaped into a circle or rectangle and it can contain holes or clusters. In, a topology creation is significantly different from other simulators. The topology is represented as a weighted oriented graph, where each edge specifies the received signal power (gain) between two nodes and the coordinates are not considered. offers the Javabased tool called LinkLayerModel which calculates the gains between each two nodes based on a topology (specified in 2-dimensional Euclidean metric space) and a wireless channel model. This tool allows to generate a grid, a grid with variations or randomly distributed topologies. C. Wireless Channel In our work, we consider only static sensor nodes. Hence, we do not examine models (or extensions) that take sensor nodes mobility into account.,, and implement the widely used log-normal shadowing model. The model calculates an average received power P r (d) at distance d from a transmitter using the following formula: P r (d)[dbm] = P r (d )[dbm] 1 n log 1 ( d d )+X δ (1) where P r (d ) is an average power at reference distance d, n is a path loss exponent, and X δ is a random variable that follows N(, δ 2 ) distribution. For more information on the model see [7]. extends the log-normal shadowing model with the temporal variation model [8]. In addition to the log-normal shadowing model, provides the ITU Indoor radio propagation model, Rayleigh and Nakagami fading models. D. Noise The noise modeling has a high impact on the reception of packets., and model noise as a constant value that does not change with time. uses the Closest Pattern Matching (CPM) algorithm that calculates a statistical model capturing noise and interferences appearing in the wireless environment. The input of the algorithm is a noise trace file based on empirical measurements. 734

3 E. Radio and MAC In the experiments, we used MICAz sensor nodes (see Section III). The nodes are equipped with a 24 MHz CC242 radio. The radio is compatible with the IEEE standard, which specifies PHY and MAC layers of a networking stack. We check whether the considered simulators provide models for CC242 PHY and MAC layers., and directly support a CC242 radio. provides the radio model with possibility to select between 8 MHz, 9 MHz and 24 MHz bands (as defined in the standard). If more than one signal is received by a node in, or, a signal-to-interference-plus-noise-ratio (SINR) is calculated. Based on the SINR and modulation scheme, the bit error rate (BER) is calculated. Based on the BER and packet length, the decision about a packet reception is made. Additionally, takes the encoding into account for the computation of packet error rate (PER). The comes with another approach where PER is calculated based on the empirical measured data with a CC242 radio chip., and have a model for the CSMA-CA unslotted protocol. provides Configurable CSMA, which offers several functions for controlling parameters of back-off behavior. In TinyOS, Configurable CSMA is set up to correspond to the behavior of a CC242 radio chip by default. F. Energy Consumption, and model the energy consumption as the linear depletion of a battery. Specific node operations can draw energy, which is subtracted from the remaining battery capacity. and allow to specify values for energy consumptions of a radio chip in different modes (receiving, transmitting and sleeping). also models energy consumption of the transitions between any of these radio modes and energy consumption of a microcontroller. models energy consumption of only two transitions between receiving and transmitting modes. allows to specify the energy consumption of a node in two modes when transmitting or receiving a packet. does not have a direct support for energy consumption. The development of its extension Power- was terminated in 25 and is no longer supported. III. CALIBRATION We ran a series of indoor and outdoor experiments on MICAz sensor nodes to calibrate a wireless channel model. The energy consumption model was calibrated according to the MICAz and CC242 datasheets. Our testbed consists of a base station, transmitter and receiver. In order to automate the process of experiments setting and data collection, we implemented four applications. The first application is written in Java and it is intended to be run on a laptop. Using it, we were able to set the transmission power of the transmitter, a number of packets to be sent, a time interval between two consecutive packet transmissions. Using the same application, we collected data from a receiver. The second application is run on a base station, which is connected to a laptop through the USB and serves as a gateway between the laptop and sensor nodes. The application simply forwards packets with requests from the laptop to sensor nodes and packets with responses from the sensor nodes to the laptop. The third application is run on a transmitter, it receives settings from the base station and transmits packets according to these settings. The last application is intended to be run on a receiving sensor node. Its primary goal is to record a number of received packets together with their received signal powers. When a request to send statistics comes from a base station, the receiver calculates average signal power, its standard deviation and send them together with a number of received packets to the base station. A. Radio Propagation Model We calibrated the log-normal shadowing model for a specific topology in two different environments indoor and outdoor. In order to calibrate a simulator for the environment (not only a specific topology), more extensive experiments are required. Sensor nodes should be placed in different locations of the environment to get the average signal power and deviation for a certain distance. The first set of experiments was run at an old football pitch, the second one in a corridor of the building where our laboratory is situated. The corridor is over 5 m long and 2.14 m wide. 3 cm high paper boxes with sensor nodes at top of them were put along the corridor at the same distance from the both walls. Our testbed consists of 3 MICAz sensor nodes. One of the sensor nodes was set up as a transmitter, another one as a base station. The remaining one was set up as a receiver. The transmitter sent 1 packets, each packet every 5 ms. The packet size was 11 B. The transmission power was set to minimum possible one, i.e., 25 dbm. The sensor nodes were configured to use a channel number 26. The experiment was repeated i times, each time putting the receiver at a different distance d i (i N) from the transmitter. At a distance d i, the receiver calculated the average signal power value P i for all n i (n i N, n i 1) successfully received packets. The values of P i for different distances d i are depicted in Figure 1. As expected, the graph for the outdoor environment is smoother than the graph for the 735

4 *log (a) Outdoor *log (b) Indoor Figure 1: Relationship between an average signal power and a distance from the transmitter. indoor environment. The most likely reason is signal reflections from the walls as well as a higher number of networks operating on the same or adjacent frequencies. At the old football pitch, there were only several networks using the channels 7, 11 and 12 (specified in the standard), which do not overlap with the channel 26 (specified in the standard) [9]. In the building, there were more than fifteen networks using the channels 1, 3, 5, 7, 11 and 13, where the last one overlaps with the channel 26 [9]. The log-normal shadowing model involves three parameters, a signal power at a reference distance P (d ), a path loss exponent n and a deviation δ (see Subsection II-C). We took 1 m as the reference distance. The signal power at the reference distance was set to dbm 1. X δ was set to zero. Since it follows N(, δ 2 ) distribution, after a high number of repetition (1 in our case) the mean value should be zero. Using the non-linear least squares method (implemented in the Curve Fitting Toolbox of MATLAB R21b), we found the optimal value of the parameter n, so the model fits the real data in the best possible way. For the indoor environment, n = 1.56, while for the outdoor environment n = The fitting curves are depicted in Figure 1. The other possibility was to set P (d ) to the value measured during the experiment ( 7 dbm for both environments, see Figure 1) or to optimize it together with the path loss exponent n. Both approaches may result into more accurate simulation results. However, only and 1 It is calculated using the Friis equation for a 248 MHz wave frequency (corresponds to channel number 26), setting antenna gains of a tranceiver and a receiver to 1. allow to set the signal power at a reference distance without their source code modification. and calculate it automatically using the Friis equation. B. Energy Consumption Model We measured energy consumption of two MICAz sensor nodes. One of them was running the transmitter application, the second one the receiver application. The transmitter sent 6 packets, each packet every 1 ms. The packet size was 11 B. The transmitter power was set to 25 dbm. Both sensor nodes were equipped with two AA batteries. In order to measure the energy consumption, we used the widely accepted approach which utilizes a shunt resistor. We put a 1 Ω resistor into a current path. Using the PicoScope 4224 oscilloscope (sampling at the 1 MHz frequency), we measured the supply voltage and voltage over the resistor. For both sensor nodes, the average supply voltage (during the experiment) was the same and equal to V. Using MATLAB for processing of the gathered data, we obtained the following results. The transmitter consumed.3373 J, while the receiver consumed.3483 J. The energy consumption model in the simulators was set in accordance to the MICAz and CC242 datasheets. A CC242 radio chip consumes 18.3 ma (8.5 ma) in a receiving (transmitting) mode when a supply voltage is 3.3 V [1]. We converted these values for the case when a supply voltage is V. The radio chip consumes ma ( ma) in a receiving (transmitting) mode when the supply voltage is V. The power consumption (multiplication of the current and supply voltage) is then equal to mw (18.75 mw) in a receiving (transmitting) mode. In, there is also a possibility to set the energy consumption of a microcontroller. We set it to mw (according to [11]) which is ma * V. IV. VALIDATION In this section, we present the results from,, and simulators set in way we described in Section III. A. Radio Propagation Model A topology was set manually. Nodes formed a line. First of them was a transmitter, others were receivers. The first receiver was at the distance of 1 m from the transmitter. The distance between the receivers was 1 m. The total number of nodes varied for different simulators (simulations). It was set in such a way that the last node did not receive any packets from the transmitter. The transmitter sent 1 packets, each packet every 5 ms. The transmission power was set to 25 dbm. The size of packet was 11 B. Each node recorded the number of received packets, signal power of the received packets and calculated the average value. 736

5 *log (a) Outdoor *log (b) Indoor Figure 2: Relationship between the average signal power and distance (a) Outdoor (b) Indoor Figure 3: Relationship between the number of received packets and signal power. 95 The file with the noise floor sampled by a sensor node during the experiments was used as the input for the noise model of. In and, we set the noise floor to dbm and dbm for the outdoor and indoor environment, respectively (i.e., we used the mean values of the measured noise floor). The deviation δ was set to.57 and.48 for the outdoor and indoor environment, respectively. These values come from the experiments, which we described in Subsection III-A. In, and we used the CC242 radio. In, we utilized the radio for the 24 MHz band. The simulation was repeated only once, which helped us to reveal some interesting facts about the simulators. The relationship between the average signal power and distance is depicted in Figure 2. For both environments, the results from perfectly match the fitting curves from the calibration. The results from and are different since they generate X δ for a certain link and keep it unchanged during the whole simulation. On the contrary, and generate X δ for every packet and every second, respectively. The relationship between the number of received packets and the averaged signal power is depicted in Figure 3. In and the nodes received packets up to 95 dbm, while in and sensor nodes stop receiving packets at approximately 9 dbm threshold. It might be caused by differently implemented mechanisms which translate SINR into a packet reception probability. Finally, the relationship between the number of received packets and distance is depicted in Figure 4. Even though the simulators were calibrated and were set in the same way, their results are significantly different. Again, this might be caused by differently implemented mechanisms, which translate SINR into a packet reception probability. B. Energy Consumption Model We simulated only two sensor nodes, at the distance of 1 m from each other. The receiver has been receiving all of (a) Outdoor, both simulations and real measurements (b) Indoor, simulations (c) Indoor, real measurements Figure 4: Relationship between the number of received packets and distance. the transmitted packets. The packet size was set to 11 B, 6 packets were sent, each every 1 ms. The results from the simulators are listed in Table I. The results from were significantly different from the results from, and the real measurements. A deeper investigation showed that counts energy consumption only when a node is receiving or transmitting a packet. Although in the real experiment, a node was always in the receiving mode, the does not take this into account. Also, the code analysis shows that the packet transmission takes.176 ms, which does not correspond to reality. In the reality, a packet transmission (or reception) takes between 1 and 1.5 ms (see Figure 5). 737

6 Voltage, V Table I: Energy consumption. Simulator Transmitter Receiver (radio) J J (radio + CPU) J J J J.1975 J.4369 J Real measurements.3373 J.3483 J Time, ns (a) Transmitting Voltage, V Time, ns (b) Receiving Figure 5: Voltage over resistor. V. CONCLUSION In this work, we compared,, and simulators in both theoretical and experimental ways. The theoretical comparison revealed the following. None of these simulators provide a model for non omni-directional antenna. These simulators provide more than one model for radio propagation, one of which is the widely accepted log-normal shadowing model. The energy consumption is modeled as a linear depletion of a battery in, and. does not model the energy consumption. We ran a set of experiments on MICAz sensor nodes and calibrated the simulators. We simulated the same scenarios as in the real experiments and compared the results. The simulations revealed that even though the simulators were set in the same way, their results (e.g., an energy consumption, a number of received packets) significantly differ from each other. In order to add both more realism and credibility to simulation results, researchers should calibrate and validate the models (e.g., for radio propagation, energy consumption) used in simulators based on the data collected from (smallscale) real experiments. Also, a continuous comparison of simulators should be undertaken. It would help to understand the differences in simulators, to reveal their design and implementation mistakes, to unify the simulators in the sense that they will provide the same results when set in the same way. Further, we plan to undertake a deeper investigation on the differences of simulators and to examine their impact on the performance of simulated protocols by implementing and simulating the same intrusion detection system on different simulators set in the same way. ACKNOWLEDGMENT We are grateful to Petr Hanáček, Filip Jurnečka, Jiří Kůr, Michal Král, Luděk Smolík, Tobiáš Smolka, Ihor Stetsko, Petr Švenda for the discussions and suggestions. Also, we thank the anonymous reviewers for their comments and suggestions that improved the paper. This work was supported by the project GAP22/11/422 and Andriy Stetsko was additionally supported by the project GD12/9/H42 of the Czech Science Foundation. REFERENCES [1] S. Kurkowski, T. Camp, and M. Colagrosso, MANET simulation studies: The incredibles, ACM Mobile Computing and Communications Review, vol. 9, pp. 5 61, 25. [2] P. Hurni and T. Braun, Calibrating wireless sensor network simulation models with real-world experiments, in IFIP-TC 6 Networking Conference. Springer-Verlag, 29, pp [3] A. Stetsko and V. Matyas, One size does not fit all how to approach intrusion detection in wireless sensor networks, in Annual Doctoral Workshop on Mathematical and Engineering Methods in Computer Science (MEMICS9). Schloss Dagstuhl Leibniz-Zentrum fuer Informatik, 29. [4] A. Stetsko and V. Matyas, Effectiveness metrics for intrusion detection in wireless sensor networks, in European Conference on Computer Network Defense. IEEE Computer Society, 29, pp [5] Memsic. Mote processor radio & mote interface boards user manual: Document part number: Rev A. [Online]. Available: [6] Moteiv. Tmote sky: Ultra low power IEEE compliant wireless sensor module. [Online]. Available: [7] T. Rappaport, Wireless Communications: Principles and Practice, 2nd ed. Upper Saddle River, NJ, USA: Prentice Hall PTR, 21. [8] A. Boulis. : A simulator for wireless sensor networks and body area networks: Version 3.2: User s manual. [Online]. Available: [9] IEEE Standard for Information technology Telecommunications and information exchange between systems Local and metropolitan area networks Specific requirements. Part 15.4: Wireless Medium Access Control (MAC) and Physical Layer (PHY) Specifications for Low-Rate Wireless Personal Area Networks (WPANs), 26. [1] TI. CC GHz IEEE / ZigBee-ready RF transceiver. [Online]. Available: [11] Memsic. MICAz wireless measurement system. [Online]. Available: 738