Device-Free Localization and Activity Recognition using Array Sensor. Jihoon Hong

Transcription

1 Device-Free Localization and Activity Recognition using Array Sensor by Jihoon Hong Adissertationsubmittedinpartialsatisfactionofthe requirements for the degree of Doctor of Philosophy in Engineering in the Graduate School of Science and Technology of the Keio University, Yagami Committee in charge: Professor Tomoaki Ohtsuki, Chair Professor Iwao Sasase Professor Fumio Teraoka Professor Hiroshi Shigeno September 2014

2 Device-Free Localization and Activity Recognition using Array Sensor Copyright 2014 by Jihoon Hong

3 1 Abstract Device-Free Localization and Activity Recognition using Array Sensor by Jihoon Hong Doctor of Philosophy in Engineering Keio University, Yagami Professor Tomoaki Ohtsuki, Chair In recent years there has been a growing interest in localization and activity recognition of people indoors. Most of the existing systems such as the Global Positioning System (GPS) and wearable sensors, which require wearing sensing devices on the person to estimate their locations and activities, might be inconvenient from the users perspective. This is particularly relevant for users who have some physical or mental disabilities. There are other scenarios where it might be impossible to attach the sensors, since the person that is being monitored is not expected to cooperate with the system, such is the case with intruders in an alarm system. Device-free sensing technology is a novel concept to estimate location and to recognize activity of people using radio frequency (RF) as the sensor, where the people do not need to carry any sensing device. It can be applied in various scenarios such as intruder detection in security and elderly monitoring in healthcare. Recently, many researchers have proposed state-of-the-art device-free sensing methods using received signal strength (RSS) measurements. These methods show high localization accuracy and provide reliable performance in real environments. However, RSS-based systems have some limitations, because the RSS itself contains noise and fluctuates even in a static environment. In this dissertation, we discuss novel methods for device-free localization and activity

4 2 recognition using signal subspace features such as signal eigenvectors and eigenvalues measured by an antenna array on the receiver, referred to as array sensor. There are significant challenges to use the array sensor for localization and activity recognition. To achieve accurate localization and activity recognition, we need as many signal features as possible. However, the conventional array sensor approach provides only a pair of signal eigenvectors and eigenvalues due to the assumption of a single channel transmitter. Moreover, a binary classifier used in that approach, cannot classify locations and activities. In particular, for activity recognition, the temporal changes of a person should also be considered. In Chapter 1, we provide the motivations and objectives of this dissertation. In Chapter 2, we describe the related work in device-free sensing technologies including conventional sensor technologies such as computer vision, passive infrared (PIR), and ultrasound sensors. We then review the state-of-the-art in RF-based technologies using ultra wideband (UWB), Doppler radar, RSS and signal subspace. We also briefly describe the array sensor and support vector machines (SVM), which were used for the proposed methods. In Chapter 3, we present a localization method based on multiple channels and multiple subarrays using the array sensor. The proposed method uses spatial smoothing processing (SSP) to obtain radio wave features such as the signal eigenvectors and eigenvalues from multiple channels. The localization is based on an SVM exploiting the signal subspace features. The experimental results show that the localization accuracy is improved using the proposed method. In Chapter 4, we propose a multiple SVM-based localization method to enhance localization accuracy. Unlike the method in Chapter 3 which requires multiple channels, the proposed method can extract more reliable features from a single channel transmitter, using a probabilistic-based feature selection via multiple SVMs. Using outlier detection and error mitigation methods based on spatial-temporal averaging, the localization error can be reduced. The experimental comparisons with state-of-the-art methods show the efficiency of

5 3 the proposed method. In Chapter 5, we introduce a state classification method that can be used to estimate the activities of a person, such as walking, standing, sitting, and falling. We show feature extraction and selection methods for several activities, and apply the SVM to classify the target entity s activities. We then demonstrate using experimental data obtained in a bathroom and laboratory environments. In Chapter 6, we conclude the dissertation with key findings in this study and future work.

6 To my family i

7 ii Contents List of Figures List of Tables v vii 1 Introduction Motivations and Objectives Applications Contributions Outline of the Dissertation Related Work and Fundamental Technology Sensor Technologies Computer Vision Passive Infrared (PIR) Ultrasound RF-based Sensing Technologies Ultra Wideband (UWB) Doppler Radar Received Signal Strength (RSS) Nuzzer Signal Subspace Array Sensor based on Signal Subspace Overview of Array Sensor Array Data Model Subspace-Based Method Cost Functions of Signal Subspace Support Vector Machine (SVM) Separable Case Non-Separable Case Kernel-based Technique

8 iii Multiclass Support Vector Machines Conclusion Device-free Localization with Multiple Channels and Multiple Subarrays using Array Sensor Introduction Proposed Localization Method Multiple Channels Spatial Smoothing Processing (SSP) Proposed Localization Algorithm Training Phase Localization Phase Experimental Results Experiment 1: Detection of Person s Activities, Intruding, Walking, and Stopping Experiment 2: Localization with SSP Cost Function in the Testing Phase Comparison of Localization Accuracy and RMSE Probability Map Experiment 3: Impact of the Array Antenna Placement on Localization Performance Conclusion Device-Free Localization using Multiple SVMs Introduction The PLAS System Model Overview Feature Extraction Localization with Error Mitigation Scheme Experimental Results Experimental Environment Comparison between RSS and Signal Eigenvector Impact of Parameters on Localization Accuracy Impact of Subset Size (δ) Impact of Outlier Threshold (κ) Comparison with Other Algorithms Conclusion State Classification Method for Human Activity Recognition Introduction Proposed State Classification Method

9 iv Feature Extraction Training State Classification Experimental Evaluation Experiment Classification Results of Proposed Method State Classification based on RSS Comparison of Kernel Functions Experiment Conclusion Conclusions Contributions Future Work Bibliography 92 Appendix A List of Papers by Author 99 A.1 Journals A.2 International Conferences A.3 Domestic Conferences A.4 Newsletter A.5 Patents

10 v List of Figures 1.1 Structure of this dissertation Illustrations of the change of signal subspace L-element uniform linear array An illustration of support vectors (blue circles and yellow squares) and optimal hyperplane for the case of two classes. x i and x j are ith and jth features of x, respectively.w is a weight vector, which defines a direction perpendicular to the hyperplane. w is the Euclidean norm of w. ξ i,i =1,,l is the margin slack variable and it allows some misclassification samples Examples of the different scenarios of the array sensor in an indoor environment. (a) Single channel, (b) Multiple channels L-element array divided into M-element subarray L L correlation matrix and M M sub-correlation matrix Proposed localization algorithm The room used for experiment An example of the changes of the signal eigenvector in the room due to the movement of a person, such as entering, walking, stopping, and leaving The room used for the experiment The change of cost function P i (t) (i =1, 2, 3) without SSP The change of cost function P i (t) (i =4, 5, 6) without SSP The change of cost function Pi SSP (t) (i =4, 5, 6) with SSP (N =3,M =6) Examples of the localization probability maps. Note that P is the human position number shown in Fig (a) Localization accuracy without SSP (F =2),(b)LocalizationaccuracywithSSP(F =20) The room used for the experiment 3. The numbers in circle show the points to localize human s position Algorithm of proposed localization system Experimental environment

11 vi 4.3 Measurements comparison between (a) RSS and (b) P (t) fornoeventanda person standing. A person stands at location 8 (see Fig. 4.2) Measurements comparison between (a) RSS and (b) P (t) fornoeventanda person standing. A person stands at location 9 (see Fig. 4.2) Impact of system parameters on RMSE during centralized antenna in LOS and NLOS cases. (a) Subset size δ varies from 1 to 5 while k =4, w =100, κ = 1.0. (b) Outlier threshold κ varies from 0.2 to 2 while k =4,w=100,δ= CDFs of the random estimation in LOS, Nuzzer, and PLAS: k = 4,w = 100,δ =3,κ=1.0. (a) Centralized antenna in LOS, (b) Distributed antenna in LOS CDFs of the random estimation in NLOS, Nuzzer, and PLAS: k =4,w = 100,δ = 3,κ= 1.0. (a) Centralized antenna in NLOS, (b) Distributed antenna in NLOS Proposed state classification algorithm An illustration of feature vector with its label Experimental environment Average classification accuracy with different frame length τ values An example of classification results. Label 1: No event. Label 2: Walking. Label 4: Standing while showering. Label 6: Falling down Comparison of kernel functions The setup for the office room of the array sensor. Note that the transmitter Tx is placed under the desk to make NLOS environment

12 vii List of Tables 1.1 Examples of applications of the proposed system Problems of existing researches and main contributions of this dissertation Comparison of different RF-based localization systems Comparison of different RF-based activity recognition systems Experimental parameters Transmission frequencies Comparison of localization accuracy and RMSE. D s =Dimensionofthesignal subspace, N = Number of subarrays, F = Number of features RMSE results Description of Experiments Experimental Parameters Comparison of All Experiments Experimental parameters Descriptions of seven states used in Experiment Descriptions of six scenarios used in Experiment Accuracy comparison between RSS (conventional) and signal subspace feature using the proposed method Descriptions of five states used in Experiment Confusion matrix of classification results

13 viii Acknowledgments First of all, I would like to acknowledge to my advisor Prof. Tomoaki Ohtsuki. This work would not have been possible without him. His guidance and support are extremely helpful from the beginning to the end of this work. I have learned a lot of things including research skills, knowledge, and attitude from him. Besides my advisor, I would like to thank the rest of my dissertation committee: Prof. Iwao Sasase, Prof. Fumio Teraoka, and Prof. Hiroshi Shigeno, for their encouragement, insightful comments, and challenging questions to improve the quality of this dissertation. IwasveryluckytoworkwithamazingpeopleatOhtsukiLab.Iwouldliketothankall of Ohtsuki Lab members for all the help and great support. I would also like to thank my former lab mates, Dr. Kazuhiko Mitsuyama and Dr. Chinnapat Sertthin, for their advice and guidance throughout writing this dissertation. My sincere thanks also go to Shun Kawakami, Yusuke Inatomi, and Shoichiro Tomii, for their collaboration and support of this work. I am in debt to my lab mates, Juan C. Corena, Hiroki Hayashi, Anthony Beylerian, and Mondher Bouazizi, for their great help in this dissertation work. I would like to express my appreciation to Japanese Government Scholarship Global 30 Project (Monbukagakusho), Rotary Yoneyama Memorial Scholarship, Yamaoka Memorial Scholarship, Tamura Memorial Scholarship, and JASSO Honors Scholarship for their kind support. I am also thankful to GCOE, KLL, Ph.D. Student Research Support programs at Keio University, for the research grants to present my work at conferences and to conduct experiments. I am grateful to have received the JSPS Research Fellow (DC2) for this work. Last but not least, I would like to express my appreciation to my family. My parents and sister have given me opportunities to go abroad for my Master s and Ph.D. study. Most especially, I am very grateful to my lovely wife, Sangha Kang, who has always been a great support and encouragement to finish my Ph.D. study.

14 1 Chapter 1 Introduction Estimating the location and the activity of a person is increasingly important in our lives. From the estimated information, people can find valuable solutions for their problems such as finding friends, monitoring babies, helping patients, and so on. Both communications and sensors have been gradually developed and improved together in wireless networks, to become ordinary technologies that assist us to find these information. This dissertation focuses on the sensors to estimate the location and the activity of a person, particularly who does not carry any sensor device, using radio frequency (RF) in wireless networks. 1.1 Motivations and Objectives Localization is a key application in sensing technologies, owing to the capabilities for a wide range of services. The most well known technology for localization is the Global Positioning System (GPS) [1, 2]. The GPS consists of at least twenty four satellites in a variety of six different orbits [2]. GPS-based localization uses at least four GPS satellites, which transmit time and location information; a GPS receiver measures the information and determines its position. GPS provides coarse-grained location information (about 5 m of accuracy) using time-of-arrival

15 2 (TOA), also called as time-of-flight (TOF), for outdoor use. However, due to limitation of TOA, the position of GPS receiver in indoor environments cannot be reliably measured. Another limitation is that the target object should carry the GPS receiver to be tracked. This limitation is critical for many applications, such as intruder detection and perimeter defense systems. Another important application of sensing technologies is the activity recognition, such as walking around, standing still, and falling on a floor. For activity recognition, wearable sensors have become emerging technologies with an increasing number of applications owing to low cost, small size, and high accuracy [3]. For example, their potential applications include elderly and patient monitoring in home or hospital, supporting sports and fitness training, and entertainment such as gaming. Various types of sensor devices such as accelerometers, gyroscopes, thermometers, and electrodes, are used for wearable sensor systems. Nonetheless, like the aforementioned limitation of GPS, wearable sensor systems also require the target object to carry sensors; it might be inconvenient to set the sensors, particularly for the user who has some physical/mental disabilities. We call the aforementioned technologies device-based sensing, because those require setup devices such as sensors on the target object. Since RF signals have characteristics such as diffraction, reflection and refraction caused by conductive objects, we can use those characteristics to estimate the location and activity of people, even if they do not have any sensing device. We call this device-free sensing. In conventional device-free sensing technologies, computer vision-based sensing systems such as closed-circuit television (CCTV) [38] are commonly used to monitor people or areas of interest, owing to their price and ease of use. However, a significant problem is the infringement of user s privacy. For example, it is difficult to install in private areas, such as a bathroom. Another problem is the limited detection area; such systems can not localize behind obstacles. In recent years, many researchers have proposed state-of-the-art DFL methods using re-

16 3 ceived signal strength (RSS) measurements [11, 23, 24, 29, 32]. Radio tomographic imaging (RTI)-based DFL method adopts a linear model for imaging of moving persons in wireless sensor networks [23, 24]. On the other hand, WLAN-based DFL method in [29, 32] use a fingerprinting technique with a probabilistic method. These methods show high localization accuracy and provide reliable performance in real environments. However, RSS-based systems have some limitations, because the RSS itself contains noise and fluctuates even in a static environment. As a result, the conventional device-free sensing techniques based on the line-of-sight (LOS) condition are not valid in such scenarios. Using micro-doppler features, an approach for human activity recognition has been proposed [19]. The approach uses micro-doppler radar to measure human activities by extracting features from the Doppler spectrogram. To classify human activities, a support vector machine (SVM) is adopted. The classification accuracy of the approach was found to be above 90 %. However, the detection range of the micro-doppler radar is narrow to cover all the area of interest. In addition, more performant hardware is needed for non-line-of-sight (NLOS) environments. To overcome the aforementioned problems, a signal subspace-based approach has been proposed for device-free sensing in recent years [34, 35]. In the approach, the system uses an antenna array at the receiver side, referred to as array sensor, to obtain the signal subspace features such as signal eigenvectors. The system uses the changes of signal subspace spanned by signal eigenvectors of the data correlation matrix of the received signal from the array sensor. For example, when a human enters the room where the system is installed, the signal subspace in the room changes by the human. Unlike the general applications for direction of arrival (DOA) of transmitted signals using antenna array, the array sensor does not require DOA estimation. The array sensor also uses any frequency band demanding on the application. Main advantage over the conventional RSS-based systems is the noise mitigation. Based on a subspace based method, received signals can be divided into signal subspace and noise subspace spanned by eigenvector and its corresponding eigenvalue. Since

17 4 the array sensor uses only the signal eigenvectors spanning signal subspace as a signal feature, it is more robust to noise than RSS. The work of [34, 35] focused on human presence based on a binary classifier such as a simple threshold-based decision. Moreover, they confined to a single channel transmitter to obtain the first eigenvector of the data correlation matrix. In many sensing applications, particularly localization and activity recognition which we focused on, only statistical learning approaches such as machine learning can be used to quantify RF features due to the complexity of the relationship between the input (i.e., features) and the output (i.e., locations and activities) of the system. Thus, problems of localization and activity recognition can be essentially be solved as an estimation problem. The main objective of this dissertation is to develop a more accurate and applicable device-free sensing system using the array sensor, to estimate the location and activity of aperson,whichcanbeusefulforwiderrangeofemergingapplications. Therearesignificant challenges to use the array sensor for localization and activity recognition. To achieve accurate localization and activity recognition, we need as many signal features as possible. However, the conventional array sensor approach [34, 35] provides only a pair of signal eigenvectors and eigenvalues due to the assumption of a single channel transmitter. Moreover, the binary classifier used in that approach, cannot classify locations and activities. In particular, for activity recognition, the temporal changes of a person should also be considered. 1.2 Applications Our target in this dissertation is the localization and the activity recognition of a single person. Though multiple people detection enables also interesting applications such as people counting, this is out of the scope of this dissertation. The target applications are mainly monitoring a person in a specific area in emergency scenarios such as healthcare monitoring for the elderly people who are living alone [36, 37]. The potential applications of the proposed

18 5 Table 1.1: Examples of applications of the proposed system Capabilities Shortcomings Applicability Presence Location Activity Tracking Identity Home security Theft detection Fall detection Health monitoring Home automation Navigation Advertisement Monitoring in hospital - - system depending on the capabilities and shortcomings are of many, including home security, theft detection, fall detection, health monitoring, and home automation as summarized in Table Contributions This dissertation has contributions in both device-free localization and device-free activity recognition. For device-free localization, this dissertation has two major contributions: The first contribution is a multiple channel-based localization method that takes into account the advantage of all possible radio channels to obtain signal eigenvectors and eigenvalues as signal features. Fingerprint-based localization method using a support vector machine is also proposed; it uses the signal features extracted from multiple channel and multiple subarrays.

19 6 The second contribution is a robust localization algorithm which uses a probabilistic model to estimate the location of a target object using outputs of multiple support vector machines. Unlike the aforementioned localization method using multiple channels and multiple subarrays, for which the cost of high installation of the system is high, the proposed method does not require multiple channels and multiple subarrays. The challenge now is how to obtain reliable features without increasing channels. We can obtain not only the signal eigenvector and eigenvalue but also each RSS measurements from each antenna of the array sensor. To use as many signal features as possible, we also use the RSS measurements with the signal eigenvector. However, as mentioned earlier, the RSS values are less reliable than the signal subspace features due to the noise and fading. This problem creates a need for a process that find relevant feature subset among the set of features. We present an enhanced localization method using multiple SVMs to find relevant features. We also propose a spatial-temporal error mitigation scheme based on the output of multiple SVMs, to reduce localization error. Therefore, the proposed algorithm, referred to as passive localization using array sensor (PLAS), enables estimating the location of a target object more accurately. We evaluate the localization performance of PLAS compared to state-of-the-art DFL method Nuzzer [32] in different propagation environments: line of sight (LOS) and non line of sight (NLOS). In addition, we analyze two types of receive antenna placements: centralized and distributed antennas. The evaluations of receive antenna placement show that centralized antennas at the receiver side can greatly improve the localization accuracy, making it more suitable for real-word applications. The major contribution of this dissertation in device-free activity recognition is to apply the array sensor in the device-free sensing system. Unlike the localization methods mentioned above, here we take into account temporal changes of activities using a time window to select signal feature. For example, when a person walks around the room, the temporal change of propagation environment is different from that when standing, sitting, falling and others. To find a best fit time window, we validate the length of the time windows experimentally.

20 7 An SVM is applied to recognize human activity, and is experimentally validated. Results of extensive experiments including real life situations, (e.g., standing in a bathroom), show that the proposed method can recognize the target states with high accuracy. In addition, the proposed method confirms that the signal eigenvector as a feature has higher classification accuracy than the RSS in activity recognition cases. To summarize, this dissertation has contributions in both device-free localization and device-free activity recognition. The problems of existing approaches and main contributions of this dissertation are summarized in Table 1.2. The publications from this work are also listed in Appendix A.

21 8 Table 1.2: Problems of existing researches and main contributions of this dissertation Chapter 3 Topic Device-free localization with multiple channels and multiple subarrays using array sensor Problems of existing In [34, 35], the array sensor considers only human approach [34, presence in a room with the assumption that the 35] available channel is one. Thus, it provides only a pair of signal eigenvector and eigenvalue. Moreover, athreshold-basedbinaryclassifiercannotclassifylocations of a person. Proposed method We use multiple channels and multiple subarrays to increase reliable features for estimating locations. The proposed method takes into account the advantage of all possible radio channels to obtain signal subspace features such as signal eigenvectors and eigenvalues. To estimate locations, an SVM exploiting the signal subspace features is proposed. Effect of proposed method The experimental results show that the proposed method can estimate the location of a person with a root mean square error (RMSE) of 1.47 m, when all available features are used. We also found that when the array elements are placed lower than the target object could improve the localization accuracy.

22 9 Chapter 4 Topic Device-free localization using multiple SVMs Problems of existing approach [47] The conventional method in [47], requires multiple channels and multiple subarrays, which increase hardware cost. Moreover, the localization accuracy is degraded when the target object is located at untrained locations. Proposed method The proposed method can extract more reliable features even from a single channel transmitter, using a probabilistic-based feature selection via multiple SVMs. To reduce localization errors, we present outlier detection and error mitigation methods based on spatial-temporal averaging. Effect of proposed method The experimental results show that the localization accuracy can be improved by the proposed method, particularly in the case of centralized antenna. Chapter 5 Topic State classification method for device-free activity recognition Problems of existing The event detection method proposed in [34, 35] uses approach [34, athresholdvaluetoclassifybetweennoeventand 35] an event. Although the threshold-based method can classify two states, it is difficult to classify more states such as standing, sitting, and falling. Proposed method We propose a state classification method for the array sensor, using multiclass SVMs to classify more than two states. Effect of proposed method We conduct experiments to prove the effectiveness of our proposed method. We also compare signal subspace feature and RSS using the proposed method, to determine which is a promising method for wireless monitoring. The SVM-based method with signal subspace features outperforms the RSS-based method. Chapter 6 Conclusion Key finding and future work

23 Outline of the Dissertation This dissertation focuses on device-free localization and activity recognition using array sensor. The structure of this dissertation is summarized in Fig Chapter 2 describes the related work in device-free sensing technologies including conventional sensor technologies such as computer vision, passive infrared (PIR), and ultrasound sensors. We then review RF-based technologies using ultra wideband (UWB), Doppler radar, RSS and signal subspace. We also briefly describe the array sensor and SVMs, which were used in the proposed methods. Chapter 3 presents a localization method based on multiple channels and multiple subarrays using the array sensor. The proposed method uses spatial smoothing processing (SSP) to obtain features such as the signal eigenvectors and eigenvalues from multiple channels. The localization is based on an SVM exploiting the signal subspace features. The experimental results show that the localization accuracy is improved using the proposed method. Chapter 4 proposes a multiple SVM-based localization method to enhance localization accuracy. Unlike the method in Chapter 3 which requires multiple channels, the proposed method can extract more reliable features from a single channel transmitter, using a probabilistic-based feature selection via multiple SVMs. Using outlier detection and error mitigation methods based on spatial-temporal averaging, the localization error can be reduced. The experimental comparisons with state-of-the-art methods show the efficiency of the proposed method. Chapter 5 introduces a state classification method that can be used to estimate the activities of a person, such as walking, standing, sitting, and falling. Unlike the localization methods in Chapters 3 and 4, we take into account the temporal changes of activities using atimewindowtoselectsignalfeature.weshowafeatureextractionmethodandapplythe SVM to classify the person s activities. We then demonstrate the efficiency of the proposed method using experimental data obtained in a bathroom and laboratory environments. Finally, this dissertation concludes with key findings and future work in Chapter 6.

24 11!"#$%&'()( *+%',-./0,+(((((!!"#$%&'(F( D&8#%&-(G,'H(((((!!"#$%&'(1( =%#%&(!8#<<4C/#0,+(;&%",-(6,'(( Figure 1.1: Structure of this dissertation.

25 12 Chapter 2 Related Work and Fundamental Technology This chapter presents the previous research related to the work in this dissertation. Section 2.1 starts with an introduction to sensor technologies, including computer vision, passive infrared, and ultrasound. Section 2.2 overviews the RF-based device-free sensing technologies. Section 2.3 presents the signal subspace-based sensing technique using an antenna array, referred to as array sensor, used to detect changes in the propagation environment. Finally, section 2.4 explains support vector machines. 2.1 Sensor Technologies Various technologies including computer vision, passive infrared (PIR), ultrasound, and pressure, have been proposed for the localization and activity recognition of people. These technologies are usually costly, including infrastructure, deployment and maintenance, and may have some restrictions, depending on the environments in which they are applied. For example, computer vision based on real-time image processing has become one of most common technologies in recent years, yet it does not work in dark environments and the concern for

26 13 the privacy of users is always a major issue. PIR sensors can perceive human activities via infrared rays emitted by the human body. One of the drawbacks of PIR sensors is that the event detection area of the sensors is limited, due to their narrow sensing range. Moreover, this approach requires numerous PIR sensors to sense a wide area, thus increasing the cost of installation. Ultrasound and pressure sensors do not invade users privacy and have low cost of installation. However, ultrasound sensors are too sensitive in noisy conditions, and pressure sensors have a limited sensing area. More detailed descriptions of the three major non-rf based sensing technologies are discussed below Computer Vision Computer vision is based on image processing using cameras. Many applications using computer vision are already available in our daily lives, e.g., road surveillance, home security, tracking, and activity recognition [5]. Traditional cameras such as optical cameras, e.g., charge-coupled devices (CCD) and complementary metal-oxide-semiconductor (CMOS), are used in image processing. Optical cameras capture images using an optical lens, then convert the images to electromagnetic signals. With signal processing techniques, the signals are extracted as features of target objects. The computer vision-based sensing approach is a powerful solution since it can detect target objects accurately and passively. However, privacy concerns are the biggest drawback of any camera-based system. In particular, no user may want to install cameras in his/her private rooms such as bathroom. Another drawback of cameras is having very limited detection range. The optical camera can only capture the surface of target objects, i.e., it cannot detect target objects in non-line-of-sight (NLOS) situations; thus if the target objects exist behind obstacles such as walls, it cannot detect them. Infrared cameras based on thermal imaging can be a key solution for the problem of detecting through walls [6]. Although the infrared cameras can sense thermal signatures even if the target is behind the walls, they require temperature differences between the

27 14 target objects and surrounding materials Passive Infrared (PIR) Passive infrared (PIR), also called as Pyroelectric infrared, sensors use the infrared signals reflected by the target objects. The wavelength of infrared signals is longer than visible light, which is invisible to human eyes, but can still be sensed. In recent years, many motion detection systems [7] are using the PIR sensors, since they use a simple detection method with an on/off switch. Moreover, PIR sensors are inexpensive and small compared to the other sensors. Similar to camera-based systems, however, PIR sensors cannot detect the target objects in NLOS situations, because the infrared signals cannot pass through obstacles. Moreover, the detection range of a PIR sensor is very limited. In addition, PIR sensors are sensitive to thermal noise which often causes a high number of false alarms Ultrasound Ultrasound signals are also an attractive source for device-free localization and activity recognition systems [8, 9]. Like PIR sensors, a typical human cannot hear ultrasound frequencies around 40 khz. In other words, ultrasound-based sensing systems require line-of-sight (LOS), because ultrasound signals reflect off obstacles. Therefore, most of ultrasound-based systems are installed on ceilings to expand detection coverage in indoor environments. Room conditions such as temperature, air pressure, and humidity may also affect ultrasound signals. Unlike optical cameras, ultrasound-based systems do not use optical images from the target surface, but use the ultrasound echos reflected off the target surface. Thus, the color of target surfaces, e.g., clothes, is not a main concern in ultrasound-based systems. However, the material of target surfaces, e.g., metal and wood, influences the detection performance of the systems.

28 RF-based Sensing Technologies In recent years, radio frequency 1 -based sensing technologies have received attention, because of their applications to many scenarios. The device-free RF based systems take advantage of radio wave characteristics such as the reflection, diffraction and scattering of the RF by aperson. AvarietyofRFbasedsensorsindevice-freeapproacheshavebeensuccessfully developed for localization and activity recognition. Most common RF sensing technologies are described in detail in the following subsections Ultra Wideband (UWB) Like Morse code communication, the ultra wideband (UWB) techniques use pulse waves, yet extremely short-pulse (impulse) waves on the order of nanoseconds in communications. The UWB techniques can be used not only for communications between a transmitter and areceiver,butalsoforsensingapplicationse.g.,fine-grainedindoorlocalizationandevent detection [15, 16, 17]. The other advantages of the UWB techniques are low-power consumption and robustness in rich multipath environments such as in indoors. Although most UWB systems should satisfy the standards such as the Federal Communications Commission (FCC) restrictions on bandwidth, interference to/from narrowband-based system is still an ongoing issue Doppler Radar The Doppler radar measures the Doppler shift, that is the shift in frequency of a source signal in response to a moving target object, thus corresponds to the speed of the target object. We consider a narrowband continuous wave (CW) Doppler radar. When a target object is moving, the Doppler shift is changed by the reflected wave emitted from the Doppler radar. The Doppler shift f d is proportional to the carrier frequency f c of the signal, which is defined 1 In this dissertation, the term radio frequency uses as a synonym for radio wave.

29 16 as follows: f d = 2vf c c v, (2.1) where c is the speed of light, about m/s. Since the speed of light c is much faster than that of a target c v, thetargetvelocityv can be calculated as follows: v = cf d = f d λ, (2.2) 2f c 2 where λ is the wavelength of the carrier frequency. Doppler radars are very effective for detecting the movement of a person because they can isolate the signatures coming from stationary objects such as furniture. Another advantage of Doppler radars is that low cost commercial off-the-shelf (COTS) devices are already available in market. In addition, periodic vibrations or rotations caused by breathing and heart beating can be measured by using the micro-doppler features, that can be extracted from a timefrequency analysis based on the theory of electromagnetic back-scattering fields [18, 19]. An approach for human activity classification method based on micro-doppler features of a radar using support vector machine (SVM) was also evaluated in [19]. Measured data of different activities were collected using a Doppler radar and then their features were extracted from the Doppler spectrogram. An SVM was then trained using the measurement features to classify the activities such as running, walking, crawling, boxing, and sitting still. The classification accuracy based on the micro-doppler features was found to be above 90 %. However, the limitation of Doppler radars is that only moving targets can be detected. Hence, stationary or slow moving targets cannot be detected. Moreover, multiple Doppler radars might be needed to cover a wide range, because a Doppler radar is generally sensitive to the angle between the movement of the target and the radar. In addition, there is a tradeoff between the detection coverage and range resolution, relative to the carrier frequency. While a Doppler radar in a high frequency band, e.g., 24 GHz, may estimate the movement

30 17 of target objects with high accuracy, it may not detect the target behind obstacles because the obstacles reflect radio waves Received Signal Strength (RSS) In device-free RF based systems, RSS is a well-known metric and is the most common measurement metric because of the fact that it can be extracted from any standard wireless device. RSS values are available in received packets on the MAC layer, and the measured value represents the relative signal strength from the transmitter to the receiver. Unlike timeof-arrival (TOA) and time-difference-of-arrival (TDOA), RSS-based systems do not require time synchronization between the transmitter and the receiver. Many researchers have studied and discussed RSS-based device-free localization and/or activity recognition [23] [32]. RSS-based device-free techniques can be categorized into two groups: model-based methods and fingerprint-based methods. The model-based methods take into account the effect of human body on the log-distance path loss model in a wireless sensor network, such as a radio tomographic imaging (RTI) technique [24]. The fingerprintbased methods use a database containing measured RSS values, which is constructed in a training (or offline) phase. Then, the location or activity of the target object is estimated using a pattern matching algorithm, such as machine learning, during testing (or online) phase [29]. Only a few works have discussed the activity recognition using the RSS measurements. Broadcasting waves such as FM and TV, can be used to detect human movements by using the changes of RSS in broadcasting waves from outside [10, 11]. The main advantage of broadcasting based systems is that there is no need to install a transmitter. The systems need only a receiver to detect human movements, thus the cost for installation of the system is lower than that of transmitter-receiver pair systems. Unfortunately, there are fundamental problems with the RSS-based sensing techniques. The RSS is very sensitive to interference and noise; hence the changes of RSS over time are

31 18 large. Thus, the performance of RSS-based systems including localization/activity recognition accuracy and stability might degrade due to those problem [33] Nuzzer A probablistic DFL system, Nuzzer, has been proposed in recent years [32]. The Nuzzer system uses fingerprint-based probabilistic techniques for estimating locations. Such as general fingerprint-based systems, Nuzzer also has two phases: offline and online phases. In the offline phase, a passive radio map is constructed by measuring the effect of a person standing at radio map locations within wireless links. In the online phase, the system estimates the person s location with two estimators: the discrete space estimator (DSE) and the continuous space estimator (CSE). The DSE is based on a naive Bayes classifier that classifies the estimated location with the maximum probability given the radio map. Like other classifiers, the naive Bayse classifier also outputs only limited locations constructed in the radio map. Therefore, if the radio map locations are sparse, the localization accuracy may be degraded. Thus, the CSE based on spatial and temporal averaging processes is used for enhancing localization accuracy. Nevertheless, the Nuzzer system has some limitations as the following: The measurement metric for localization is RSS-only. Moreover, the accuracy of Nuzzer is dependent on the deployment of transmitters and receivers. If the transmitters and/or receivers are closed to each other, i.e., centralized antenna, each wireless link would have similar RSS, and thus the localization performance would be degraded. However, the Nuzzer has not been considered in the centralized antenna case. Compared to the distributed antenna deployment, the centralized antenna deployment has more advantages in terms of deployment, management, and portability in practical use Signal Subspace Compared to the RSS measurements, the signal subspace has been shown to be more reliable measurement metric [34, 35, 46, 79, 80, 81]. The signal subspace-based method [59]

32 19 decomposes the data correlation matrix of a received signal into the orthogonal signal and noise subspaces via the eigenvalue decomposition (EVD) method. The main advantage of the signal subspace is being more robust to noise since it is orthogonal to the noise subspace. In addition, unlike the information of RSS values which includes a signal magnitude only, the signal subspace contains information about the magnitude and phase of source signal. Thus, the signal subspace is a powerful metric for sensing, that can be used to estimate the location and the activity of a person. Using the signal subspace, fingerprint-based methods have been proposed for indoor localization [79, 80, 81]. They use an antenna array receiver to obtain the signal subspace from the data correlation matrix of received signal. Like other fingerprint-based localization methods, the unknown location is estimated by matching between the measured signal subspace at unknown locations and the stored signal subspace at known locations. However, the target of localization in these works is the transmitter, not a human being; these works are device-based sensing approaches. Athreshold-baseddevice-freesensingmethodhasbeenproposedin[34,35]. Themethod in [34, 35] use a single antenna transmitter and an antenna array receiver, and assume that both of them are fixed in the environment of interest. When the environment of interest is changed by some events, e.g., a person entered the room, the signal subspace is also changed. Using an optimal threshold value, results in [34, 35] showed that the classification accuracy for two events, e.g., no event and a person entered the room, was 100 %. The installation and maintenance costs of the system [34, 35] are low compared to the model-based system [24], which requires a dense deployment of sensor nodes, because it only uses a pair of a transmitter and a receiver. Unlike the multiple-input multiple-output (MIMO) radar-based system [60], the system [34, 35] can be used without time and/or phase synchronization and calibration, because it only uses the change of signal subspace. Thus, the use of signal subspace is a promising approach to overcome the limitations of RSS-based methods. The motivation of this dissertation is to use the signal subspace for device-free sensing,

33 20 which includes localization and activity recognition. Thus, a more detailed description of the signal subspace-based device-free sensing [34, 35] will be explained in the following section. 2.3 Array Sensor based on Signal Subspace This section describes a signal subspace-based approach using an antenna array, also referred to as array sensor [34, 35]. The array sensor uses the signal subspace spanned by eigenvectors and eigenvalues obtained from the spatial-temporal data correlation matrix of the received array signal vector. In this dissertation, we call the eigenvectors and eigenvalues of the data correlation matrix corresponding to the signal subspace, as signal eigenvectors and signal eigenvalues, respectively. When an event (e.g., a person enters a room) occurs, the propagation environment of interest changes as well, so the signal eigenvectors and eigenvalues change. Thus, the signal subspace represents the propagation environment of interest. The rest of this section is organized as follows: We first overview the array sensor in subsection then describe the array data model of the array sensor in subsection Next, we explain the subspace-based method in subsection Finally, the cost functions of signal subspace used in the array sensor are described in subsection Overview of Array Sensor We depict the changes of the signal subspace of the propagation environment caused by a person between no event and an event, as shown in Fig When a person moves, the signal subspace in the environment is also changed. On the other hand, the signal subspace of interest is stationary while there is no event such as an empty room. Based on this, the correlation of the two signal eigenvectors and eigenvalues during different events and/or at different locations, can be used as a spatial feature for localization. The spatial feature represents the change of multipath fading, shadowing, and reflection of interest. Thus, the

34 21 (a) No event (b) An event Figure 2.1: Illustrations of the change of signal subspace. signal subspace can be exploited to significantly increase system performance. To measure the signal subspace, the system requires an antenna array. In current wireless communication systems, however, the antenna array is used in various systems rather than a traditional single antenna system. Also the antenna array has many advantages over the traditional one. The greatest advantage of the antenna array is that it can capture spatial features such as signal subspace and noise subspace. We discard the noise subspace and use the signal subspace only. Then we take into account the signal subspace as a feature in our proposed system.

35 22 Signal source s k (t) θ k Reference point Wavefront d (l 1)d (L 1)d #0 #1 #l #L 1 Figure 2.2: L-element uniform linear array Array Data Model Consider the L-element uniform linear array (ULA) as shown in Fig We assume far field narrowband sources. The source signal s k (t), k =1, 2,,K (K L) isarrivingfrom direction θ k at time t, asplanewaves,owingtothefarfieldassumption.thereceivedsignal vector x(t) isrepresentedas K x(t) = a(θ k )s k (t)+n(t) (2.3) k=1 = As(t)+n(t), (2.4) A = [a(θ 1 ),...,a(θ K )], (2.5) s(t) = [s 1 (t),...,s K (t)] T, (2.6) where [ ] T is the transpose operator, and n(t) is an additive white Gaussian noise (AWGN) with zero mean and variance σ 2. The steering vector a(θ k )isacomplexvectorincludinga

36 23 phase shift of a source signal at the lth element, (1 l L) awayfromthefirstantenna (reference point), and is represented as a(θ k ) = [1,e j 2π λ d sin θ k,...,e j 2π λ (L 1)d sin θ k ] T, (2.7) where λ is the wavelength of the signal source. The key idea of the array sensor is to use the changes in the propagation environment by an event. To analyze the changes of interest, we use the data correlation matrix R xx of the received signal vector x(t). The data correlation matrix R xx can be estimated as follows: R xx = E[x(t)x(t) H ] (2.8) where E[ ] and[ ] H denote the ensemble average and the conjugate transpose of vector [ ], respectively. In general, the AWGN is uncorrelated with the source signal s(t). Therefore, the data correlation matrix R xx can be simplified as follows: R xx = E[As(t)s(t) H A H ]+E[n(t)n(t) H ] + E[As(t)n(t) H ]+E[n(t)s(t) H A H ] }{{} 0 = ASA H + σ 2 I (2.9) where S = E[s(t)s(t) H ]andiisthe identity matrix. Assuming the noise is independent in each element, E[n(t)n(t) H ]=σ 2 I. Because of the ergodic hypothesis (i.e., the ensemble average is equal to the time average), the ensemble average of the data correlation matrix R xx can be replaced with the time average. The estimated data correlation matrix ˆR xx is then written for time t = t 1,t 2,...,t Ns, ˆR xx = 1 N s N s i=1 x(t i )x(t i ) H (2.10)

37 24 where N s is the number of snapshots. As N s,betterestimationaccuracyof ˆR xx can be obtained [35]. In this dissertation, we treat the estimated data correlation matrix ˆR xx as the data correlation matrix R xx Subspace-Based Method The subspace-based method [59] decomposes R xx into the orthogonal signal and noise subspaces via the eigenvalue decomposition (EVD). The L-dimension data correlation matrix R xx can be computed with the EVD as follows: L R xx = λ l v l vl H = VΛV H, (2.11) l=1 V = [v 1, v 2,, v L ], (2.12) Λ = diag{λ 1,λ 2,,λ L }, (2.13) where diag{ } is a diagonal matrix, λ l and v l are the lth eigenvalue and eigenvector, respectively. When the L-element array antenna receives K signals, the dimension of the signal eigenvectors V S is K and that of the noise eigenvectors V N is L K as follows: V S = [v 1,...,v K ], (2.14) V N = [v K+1,...,v L ], (2.15) where λ 1 λ K λ K+1 λ L σ 2. (2.16) Thus, the eigenvalue matrix Λ is decomposed into signal and noise eigenvalues. The signal subspace spanned by the signal eigenvectors V S represents the propagation environment of interest. The array sensor discards the noise eigenvectors and use the signal eigenvectors and eigenvalues, to detect the change of propagation environment of interest.

38 Cost Functions of Signal Subspace To detect an event, the array sensor uses cost functions of the signal eigenvectors and eigenvalues. The cost function based on the signal eigenvectors [34, 35] is defined as: P i (t) = vi H (t no )v i (t) (0 P i (t) 1), (2.17) where v i (t no )istheith signal eigenvector obtained in advance, also referred to as the reference vector, and v i (t) istheith signal eigenvector obtained at the observation time t. Thus,the larger the value of P i (t) isthesmallerthechangeoftheenvironment,whereasthesmaller the value of P i (t) isthelargerthechangeoftheenvironment.theeigenvectorisstationary even in noise and fading environment, because it does not include RSS information. The cost function based on the signal eigenvalues [46] is defined as Q i (t) =1 λ i(t) λ i (t no ) λ i (t no ) (Q i (t) 1), (2.18) where λ i (t no )istheith signal eigenvalue obtained in advance, also referred to as the reference value, and λ i (t) istheith eigenvalue obtained at the observation time t. LikeP i (t), the larger Q i (t) is,thesmallerthechangeoftheenvironmentis,andthesmallerq i (t) is,thelarger the change of the environment is. The eigenvalue is less stationary than the eigenvector, however, using Q i (t), we can detect even when small changes happen. The goal of [34, 35] is to detect human presence that can be used for security applications such as intrusion detection. To do this, the authors use a binary classifier using a decision threshold P th,whichisdefinedas: 1 P (t no ) >P th >P(t event ), (2.19) where, P (t no )andp (t event )arethecostfunctionsofthesignaleigenvectorsmeasuredata time when no event and some events occur, respectively. In the no event case at time

39 26 t, thevalueofcostfunctionp (t) mustbeverycloseto1. Thisisbecausethepropagation environment at time t ob is unchanged compared to the propagation environment at time t no. On the contrary, in the some events case at time t, thevalueofcostfunctionp (t) must be smaller than 1. This is because the propagation environment at time t ob is changed compared to the propagation environment at time t no. 2.4 Support Vector Machine (SVM) In the past decade, SVMs have been widely used for classification in many applications such as image processing, natural language processing, and various antenna processing [65, 66, 67]. This is because SVMs outperform the other machine learning algorithms, particularly compared to neural networks, owing to their excellent generalization capabilities. Unlike neural networks, SVMs allow to train a model with a smaller amount of training datasets with large dimensions to achieve global optimality. SVMs were originally binary classifiers, and can be applied not only for linearly separable cases but also for non-separable cases [49, 50]. In general, the larger the number of training samples becomes, the more difficult the linear separability becomes; the larger the dimension of the feature space becomes, the easier the classification process does. However, mapping into high dimensional feature space causes high complexity. To deal with this issue, SVMs use the kernel-based technique to find a maximum-margin hyperplane between the two classes as shown in Fig The kernel maps the feature vector into high dimensional feature spaces without computing features in the mapped space. Although the dimension of the feature space, transformed with non-linear mapping function, becomes very large, the complexity of the SVM does not increase, because the objective function in the SVM depends on the inner product of input patterns only. We first explain the simplest case, for linear machines trained on separable data. Then, we consider the non-separable case where the training set is non-linearly separable.

40 27 Margin x j w f ( x) > 0 Class : +1 1 w 1 w Hyperplane ξi f ( x) < 0 Class : 1 x i Figure 2.3: An illustration of support vectors (blue circles and yellow squares) and optimal hyperplane for the case of two classes. x i and x j are ith and jth features of x, respectively. w is a weight vector, which defines a direction perpendicular to the hyperplane. w is the Euclidean norm of w. ξ i,i =1,,l is the margin slack variable and it allows some misclassification samples.

41 Separable Case The binary classifier is executed by using a real-valued function f : X R n R. The training input data are {x i,y i },i = 1,,l, where l is the number of data. We refer to x i as training instances and y i as the label of the training instance. Suppose that we have a separating hyperplane between a positive class and a negative class. The separating hyperplane can be written as w x + b = l w i x i + b =0 (2.20) i=1 where w is a weight vector, which defines a direction perpendicular to the hyperplane. b is abiasthatmovesthehyperplaneparalleltoitself. Thedecisionfunctionof(2.20)forthe optimal hyperplane is defined as f(x) =sgn w x + b =0 (2.21) where sgn represents the class assigned to x. The length of the perpendicular line from the hyperplane to the origin can be expressed as b,where w is the Euclidean norm of w w. Theinputdatasampleslyingontheshortestdistancefromtheoptimalhyperplaneare named the support vector. Therefore, the simplest case of maximum margin between the positive and negative classes can be 2 w value of w to find the maximum margin. subject to as shown in Fig.??. We can use the minimum min w 2 2, (2.22) x i w + b +1, for y i =+1 (2.23) x i w + b 1, for y i = 1. (2.24)

42 Non-Separable Case The simplest case of SVM mentioned above works only for data sets that are linearly separable in the feature space, and therefore cannot be used in many real-world situations. We need to consider the non-separable case to handle the training set. To solve the non-separable case, we can define the margin slack variable ξ i,i =1,,l,andaddconstraintsto(2.22), (2.23), and (2.24), so that we can rewrite them as: min w C l ξ i (2.25) i=1 subject to x i w + b +1 ξ i, for y i =+1 (2.26) x i w + b 1+ξ i, for y i = 1 (2.27) where ξ i 0impliesincorrectclassificationof{x i,y i }. This means that some misclassification can be available by the slack variable ξ i. Thus, l i=1 ξ i is an upper bound on the number of training errors. C is a penalty parameter that can be chosen by the user. By using the dual Lagrange multiplier optimization problem, (2.25), (2.26), and (2.27), can be formulated as the objective function max L D (α) = l α i 1 l α i α j y i y j x i x j (2.28) 2 i=1 i,j=1 subject to 0 a i C, l α i y i =0 (2.29) i=1 where α i is the Lagrange coefficient, and it is upper bounded by the parameter C. The complexity of finding an optimal hyperplane for non-linear classification can be reduced by using the non-separable case mentioned above and the kernel trick.

43 Kernel-based Technique Kernel-based technique called kernel trick consists of using a kernel function, x i Φ(x i ), that maps the input data to a higher dimensional space, which is called the feature space. Therefore, the decision function for the optimal hyperplane of the non-linear classification problem can be expressed as follows: Ξ Ξ f(x) = α i y i Φ(s i )Φ(x)+b = α i y i K(s i, x)+b (2.30) i=1 i=1 where Ξ is the number of support vectors, and s i are the support vectors. We can use the kernel function K(s i, x) insteadofφ(s i )Φ(x) [49,50]. There are three kernel functions mainly used for SVM; linear, polynomial, and Gaussian radial basis function (RBF) kernel. In general, Gaussian RBF kernel function shows better performance than the others. However, the others also outperform Gaussian RBF kernel in some applications. For instance, for text classification, the linear kernel has better performance than Gaussian RBF kernel. Unfortunately, we can not select the optimal kernel before comparing the kernel functions using experiments and/or simulation results, since each system has a different data construction and features have all different characteristic. Therefore, we will compare these three kernel functions are defined as: Linear kernel K(x i, x j )=x T i x j (2.31) Polynomial kernel K(x i, x j )=(x T i x j ) d,d=3 (2.32) Gaussian RBF kernel K(s i, x) =exp( γ s i x 2 ),γ>0 (2.33)

44 31 where, x and s i are the input vector and the support vector of ith, respectively. The optimal kernel parameter γ is searched by k-fold cross validation in the training procedure [76]. First, the training data are separated into n subsets of equal size. One of those subsets is considered as the validation set and the rest are used for the training procedure. By using the cross validation procedure, the error of training data is calculated n times with different combinations of training and validation data for the given total data. Among these three kernel functions, the Gaussian RBF kernel has been popular for practical use, since its key advantage over the other kernels is having less numerical difficulties [50]. A more detailed description of these kernel functions and examples of their comparison may be found in [68] Multiclass Support Vector Machines SVM can be used for not only two-class classification but also multi-class classification, named multiclass SVMs. In general, there are two major methods to solve multi-class classification problems: one-against-all and one-against-one approaches. The one-againstall approach constructs µ SVM models where µ is the number of classes. The ith SVM is trained using all of the data in the ith class with positive labels, and all other data with negative labels. The one-against-one approach trains a two-class SVM model for any two classes from the training set, which for a µ-class problem results in µ(µ 1)/2 SVM models. In the classification phase, a voting procedure assigns the class of the classification pattern to the class with the maximum number of votes [50]. In this dissertation, we use one-againstone approach for multiclass SVMs, because its performance including time cost is better than that of one-against-all approach [50]. To simplify, SVM is used in place of multiclass SVM in the following chapters in this dissertation. We use LibSVM to implement multiclass SVMs, which is a library for SVM implemented by Chang and Lin [68].

45 Conclusion In this chapter, we have discussed the background material of different approaches used to estimate locations and activities in device-free sensing technologies. Traditional device-free sensing technologies such as computer vision, PIR, and ultrasound are presented. Next, we have reviewed different RF-based device-free sensing technologies used to estimate locations and activities of people. Most of RF-based device-free localization technologies use the RSS measurements to estimate locations with either a model-based method or a fingerprint-based method. In activity recognition technologies, micro-doppler-based and RSS-based systems are also discussed. We have then reviewed the array sensor and the SVM which used to extract the signal subspace features and to estimate the locations and activities of a person. Table 2.1 summarizes the differences between the proposed methods in this dissertation and state-of-the-art RF-based localization systems. Table 2.2 summarizes the differences between the proposed method in this dissertation and state-of-the-art RF-based activity recognition systems.

46 33 Table 2.1: Comparison of different RF-based localization systems RTI [24] Nuzzer [32] Array Sensor (Chapter 3) Array Sensor (Chapter 4) Metric RSS RSS Signal eigenvectors and eigenvalues Signal eigenvectors and RSS Accuracy High Moderate High High Temporal stability Low Low High High Localization in No Yes Yes Yes NLOS environments Training (database) No Yes Yes Yes required Number of devices Many Few Few Couple (Tx and Rx) Multiple Tx (or Rx) Yes Yes Yes No required Deployment restriction Yes Yes Yes No

47 34 Table 2.2: Comparison of different RF-based activity recognition systems Doppler Radar [19] FM-based [11] Array Sensor (Chapter 5) Metric micro-doppler RSS Signal eigenvectors and eigenvalues Accuracy High Moderate High Temporal stability High Low High Activity recognition in No Yes Yes NLOS environments Training (database) Yes Yes Yes required Number of devices One One Couple (Tx and Rx) (Monostatic) (Rx-only) Multiple Tx (or Rx) No No No required Deployment restriction Yes Yes No

48 35 Chapter 3 Device-free Localization with Multiple Channels and Multiple Subarrays using Array Sensor 3.1 Introduction In section 2.3, we have briefly reviewed the array sensor approach based on the signal subspace features (i.e., the cost functions of signal eigenvectors and eigenvalues). The signal eigenvectors and eigenvalues representing the propagation environment of interest, are estimated using measurements from the array sensor. Although the method in [34, 35] can detect human presence using a binary classifier (i.e., a decision threshold), it cannot estimate the location of a person, due to the lack of signal subspace features. In [34, 35], a single channel transmitter is assumed in the system. Thus, only the largest eigenvector and eigenvalue can be measured as signal subspace features. In addition, the binary classifier cannot be directly used as a location estimator, since the indoor propagation environments are too complex. In this chapter, we propose a device-free localization method, incorporating the signal

49 36 subspace features from multiple channels and multiple subarrays with an SVM. We use multiple channel transmitters to obtain multiple signal subspace features. Moreover, a multiple subarray-based feature extraction method can also increase the signal features without additional antennas, using SSP on the subarrays. Based on the combination of two feature sets, one from the multiple channels and the other from the multiple subarrays, we apply an SVM to estimate location of a person. In addition, this work investigates the impact of the deployment between a transmitter and a receiver, particularly the height of elements of the array sensor. The experimental results show that the proposed method can estimate the location of a person with a RMSE of 1.47 m, when the number of signal features is 20. The rest of this chapter is organized as follows: Section 3.2 presents the proposed localization method. We show the experimental results in Section 3.3. Finally, we conclude this chapter in Section Proposed Localization Method We depict the different scenarios of the array sensor in an indoor environment shown in Fig Multiple Channels If there are I uncorrelated sources and K i multipath signals from the ith transmitter in the indoor environment, the received signal vector is represented as x(t) = = I K i α ik a(θ ik )s i (t β ik )+n(t) (3.1) i=1 k=1 Ã s(t)+n(t), (3.2) Ã = [a(θ 11 ),...,a(θ 1KI ), a(θ 21 ),...,a(θ IKI )] (3.3) where α ik is the complex attenuation coefficient and β ik is the reference delay for the

50 37 #"!./0(1/'! "! #!./0(1/'! $%&'()*+(,-! (a)./0(1/'! $!!"! %!./0(1/'! #"%! #"$! #"&! "! #! (b) $! %!!"! Figure 3.1: Examples of the different scenarios of the array sensor in an indoor environment. (a) Single channel, (b) Multiple channels. kth signal path from ith transmitter, with α ik 0,α i1 =1,andβ ik 0. Intheindoor environment, the difference of propagation delay among multipath waves is negligibly small, that is β ik β. Therefore,therankof S = E[ s(t) s(t) H ]isi and the dimension of the signal subspace D S equals to I.

51 38 d #0 #1 #M 1 #M #L 1 subarray #1! subarray #2! subarray #N! Figure 3.2: L-element array divided into M-element subarray R xx R f 1 R f 2 R f SSP L! R f N M! M! M! L! M! Figure 3.3: L L correlation matrix and M M sub-correlation matrix

52 Spatial Smoothing Processing (SSP) Spatial smoothing processing (SSP) is the method that separates coherent signals [51, 52]. It does not need to increase transmitters and receivers to use uncorrelated signals. The fundamental theory of SSP is that the phase relationships among coherent signals are different from one element to another, and the cross-correlation value becomes small, owing to the averaging effect by the parallel shift of the position of reception. Thus, we do not need to increase the number of array elements. If the L-element linear array is divided into M-element subarrays, we get N subarrays, where N = L M + 1, as shown in Fig The received signal vector in the nth subarray is obtained by x f n(t) = [x n (t),...,x n+m 1 (t)] T (3.4) n = 1,...,N. (3.5) The new M-dimension correlation matrix is obtained by the spatial average of the N sub-matrices as shown in Fig R f SSP = 1 N R f N n, (3.6) R f n = E[x f n(t)x f n(t) H ] (3.7) The forward averaged correlation matrix can be improved by adding the backward averaging process, called the forward-backward SSP (FB-SSP) [52]. The correlation matrix obtained by FB-SSP is represented as follows: n=1 R FB SSP = Rf SSP + Rb SSP 2 (3.8) where R b SSP is the backward correlation matrix obtained by xb n(t) =[x n+l 1 (t),...,x 1 (t)] T.

53 40 However, in real multipath environments, there are many incoming signals and it is difficult for a few elements to separate all coherent signals. The FB-SSP can separate 2N signals per group of coherent signals and the dimension of the whole subspace is M corresponding to the rank of R FB SSP.Then,thedimensionofthesignalsubspaceD S is extended as D S =min{2ni,m}. (3.9) The proposed method uses the FB-SSP to separate coherent signals and extend the dimension of signal subspace. From here, we describe FB-SSP simply as SSP Proposed Localization Algorithm The proposed localization algorithm is shown in Fig Training Phase If the dimension of the signal subspace is extended, we can use many cost functions as features for the SVM. Assume that we classify N p positions. In the training phase, we get the received signals x p (t) (p =1,...,N P )whenapersonstandsatpositionp for T N observation times. From the signals, we compute the cost functions without SSP, P i (t),q i (t), and those with SSP, P SSP j (t),q SSP (t) foreachdata,wheret =1,...,T N. That is, we have N P T N training j samples. Next, all the cost functions are combined in one feature vector as z p = [P 1 (t),...,p I (t),q 1 (t),...,q I (t),p SSP 1 (t), The dimension of z p (the number of features) is...,pd SSP S (t),q SSP 1 (t),...,q SSP D S (t)] T. (3.10)

54 41!!!" Received Signals Correlation Matrix EVD Cost Functions SSP Correlation Matrix EVD Cost Functions Feature Extraction" SVM Training Location" Figure 3.4: Proposed localization algorithm

55 42 F = 2I, w/o SSP 2D S, with SSP 2(I + D S ). w/o SSP with SSP. (3.11) The features are mapped into a high dimensional space by an RBF kernel and then the training model is obtained Localization Phase In the localization phase, although we get the cost functions and the feature vector in the same way as in the training phase, we do not know what position this feature vector is classified to. However, once the SVM has been trained, then all future unknown samples can be classified in real time. We localize the unknown position of a standing person based on the algorithm. 3.3 Experimental Results In this section, we show three experimental results obtained in different environments under various scenarios. Before introducing experimental results for localization, we first introduce an experimental result for the person intruding, stopping, or moving. All experiments are conducted in a non-line-of-sight (NLOS) condition. In NLOS there is no direct-path signal that is dominant over the signal subspace spanned by the eigenvector and thus the signal subspace spanned by the eigenvector enhances the impact of multipath signals that capture the change of environment.

56 43 Figure 3.5: The room used for experiment Experiment 1: Detection of Person s Activities, Intruding, Walking, and Stopping We show one of our experimental results obtained in the room shown in Fig We use a transmitter and its transmission frequency is GHz. The size of the array antenna is cm 3 and the number of array elements is 8. Fig. 3.6 shows an experimental result for the person intruding, stopping, or moving. In this experiment, a person opens the door, enters the room, and passes through points A, B, C, then goes through the door as shown in Fig The person stops for 20 seconds at each point, A, B, C. The cost function P (t) changessignificantlywhenthedooropens. P (t) also

57 44 no event! C! A! walking! B! leaving! walking! entering! Figure 3.6: An example of the changes of the signal eigenvector in the room due to the movement of a person, such as entering, walking, stopping, and leaving. fluctuates significantly when the person moves and fluctuates moderately when the person stops. This happens because the change of environment, such as the door opening, the existence of the person, and the person s motion, changes the propagation of the radio waves and thus the first eigenvector as well. Therefore, the cost function, that is the correlation between the first eigenvector in the reference and in the observation, also changes. From this result, we can easily see the person s movement; intruding, stopping or moving Experiment 2: Localization with SSP We show the experimental results using multiple coherent signals to improve localization accuracy. Fig. 3.7 shows this experimental environment. We use three transmitters (Tx1, Tx2, and Tx3) in this experiment. Experimental parameters are listed in Table 3.1. Each transmitter transmits a signal with a different frequency as described in Table 3.2. Three

58 45 Table 3.1: Experimental parameters Transmission power 10 dbm Modulation method No modulation Transmitter Dipole antenna Receiver 8-element linear array Sampling rate 60 Msps Number of snapshots 1024 Figure 3.7: The room used for the experiment 2

59 46 Table 3.2: Transmission frequencies Transmitter Tx1 Tx2 Tx3 Frequency GHz GHz GHz transmitters were set up on the windows side, and a receiver (Rx) was set up on the wall s side. There are also some obstacles between the transmitters and the receiver to make a NLOS situation. In total, 16 points are selected as training and testing points shown in Fig In the training phase, we obtained 100 observation data (approximately 15 seconds) when a person stands at each position. Five persons participated in the experiment. In total 8000 (= ) samples are collected, then trained by the SVM. We use z p = [P 1 (t),...,p Ds (t),q 1 (t),...,q Ds (t)] as the feature vector of the conventional method and eq. (3.10) as that of the proposed method. In the testing phase, a person enters the room from the door 1, walks from point 1 to 16 in the route indicated by solid arrows, stands at each position for 10 seconds, then walks from position 16 to the door 2 in the route indicated by dot arrows, and exits from the room. The testing data can be obtained in real time and we localize the person in a continuous way with the SVM Cost Function in the Testing Phase Fig. 3.8 shows the change of the cost function P i (t) (i =1, 2, 3) without SSP. The reference eigenvectors are obtained in advance when there is no person in the room. The dimension of the signal subspace is three and we can use those cost functions, that is i =1, 2, 3, because there are three uncorrelated transmitters in the room. From this figure, we can see that the

60 47 Figure 3.8: The change of cost function P i (t) (i =1, 2, 3) without SSP Figure 3.9: The change of cost function P i (t) (i =4, 5, 6) without SSP

61 48 Figure 3.10: The change of cost function P SSP i (t) (i =4, 5, 6) with SSP (N =3,M =6) proposed method can observe a whole room same as the conventional method. Fig. 3.9 shows the change of cost function P i (t) (i =4, 5, 6) without SSP. These cost functions cannot detect any events, because the 4th, 5th, and 6th eigenvectors are the basis vectors of the noise subspace. Thus, we can see that the cost function P i (t) (i =4, 5, 6) includes noise information. Therefore, the dimension of the signal subspace is three, we can use only P i (t) (i =1, 2, 3) as features. Fig shows the change of Pi SSP (t) (i =4, 5, 6) with SSP (N =3,M =6). The dimension of the signal subspace is extended to six as in eq. (3.9). These cost functions can also be used as features of SVM, because the SSP can separate coherent signals into multiple uncorrelated signals.

62 49 Table 3.3: Comparison of localization accuracy and RMSE. D s =Dimensionofthesignalsubspace,N = Number of subarrays, F = Number of features. Method D S N F Accuracy (%) RMSE (m) (a) w/o SSP (b) w/o SSP (c) w/o SSP (d) w/ SSP (e) w/ SSP (f) w/ SSP (g) w/ SSP (h) w/ SSP (i) w/o SSP w/ SSP (j) w/o SSP w/ SSP (k) w/o SSP w/ SSP (l) w/o SSP w/ SSP (m) w/o SSP w/ SSP

63 Comparison of Localization Accuracy and RMSE Table 3.3 shows the localization accuracy and root mean square error (RMSE) of the conventional method (w/o SSP) and proposed method (w/o SSP w/ SSP). We define the accuracy as the correct probability that the estimated position is p when the person stands at position p, andrmseasthedistanceerrorbetweentruepositionandestimatedposition. D S is the dimension of the signal subspace used for cost function, N is the number of subarrays used for SSP, and F is the number of features used for SVM. From these results in each method, we can see that the large F shows the higher localization accuracy and lower RMSE. This happens because the SVM learning ability is improved by increasing the number of features. We can also see that in the same condition (F =6), the w/o SSP achieves higher accuracy than w/ SSP. This happens because SSP reduces the effective aperture of the array antenna. Compared to w/ SSP and w/o SSP w/ SSP, (g), (h) and (i), (j), w/o SSP w/ SSP shows higher accuracy than w/ SSP. This happens because we can observe multiple paths Probability Map Fig shows examples of the localization probability map for the method without SSP and the method with SSP. In the figures, each small map is divided into 16 blocks related to 16 positions in Fig Fig (a) shows the result of localization using only P 1 (t)andq 1 (t). From this figure, the method without SSP can hardly localize the position of the standing person. We can see high localization accuracies at only three points (P =4,P =10,P =13). Fig (b) shows the result of localization using twenty cost functions P i (t), Q i (t) (1 i 3) and P SSP j (t), Q SSP (t) (1 j 7). This method uses two subarrays of SSP j (N =2). Theaccuraciesofallpointsexceptatthep =1arehigherthan80%. Thus,we can see that the proposed method that uses SSP, shows better localization performance than the other one.

64 51 (a) (b) Figure 3.11: Examples of the localization probability maps. Note that P is the human position number shown in Fig (a) Localization accuracy without SSP (F = 2),(b) Localization accuracy with SSP (F =20).

65 52 Figure 3.12: The room used for the experiment 3. The numbers in circle show the points to localize human s position Experiment 3: Impact of the Array Antenna Placement on Localization Performance We investigate the impact of the array antenna placement on the localization performance. If the receiver and/or transmitter is placed on a higher position than the target object, the change of propagation by the target is small, thus, it may affect the localization performance of array sensor. Therefore, this experiment attempts to figure out the impact of the array antenna placement on the localization performance. Fig shows the experimental en-

66 53 Table 3.4: RMSE results RMSE (m) Method F (A) Rx: 2.3 m (B) Rx: 0.7 m w/o SSP w/o SSP w/o SSP w/o SSP w/ SSP w/o SSP w/ SSP w/o SSP w/ SSP vironment of the experiment 3. Experimental parameters are listed in Table 3.1 which are same as those in the experiment 2. The transmitter (Tx) is placed on a chair of 0.4 m height from the floor. To evaluate localization performance of the receiver position, we conducted two types of experiments; the receiver (Rx) is placed on a desk of 2.6 m (A) and 0.7 m (B) height. (A) means that the elements are higher than a target object s height, and (B) means that the elements are lower than the target. We collected training data for three persons. In the training phase, we obtained the data for each position when a person stands at each position for 15 seconds. The training data were labeled 25 classes. In the testing phase, a person moves from point 1 to 25, standing at each position for 10 seconds. We compare the RMSE of the two types of Rx heights, (A) and (B). The localization RMSE results are summarized in Table 3.4. From these results, we can see that (B) s results show higher localization performance than (A) s ones. This happens because the target object can impact the eigenvector and eigenvalue spanning the signal subspace, when the antenna height is lower than the target object.

67 Conclusion In this chapter, we have proposed a localization method that uses the combination of the signal subspace features from multiple channels and multiple subarrays with an SVM. We showed that the number of the signal subspace features could be increased without adding the array elements at the receiver. The extended signal subspace features have been applied to estimate locations of a person, using the SVM-based localization. In addition, this work investigates the impact of the deployment between the transmitter and the receiver, particularly the height of the elements of the array sensor. The experimental results show that the proposed method can estimate the location of a person with a root mean square error (RMSE) of 1.47 m, when the number of signal features is 20. We also found that when the array elements are placed lower than the target object, the localization accuracy could be improved.

68 55 Chapter 4 Device-Free Localization using Multiple SVMs 4.1 Introduction In this chapter, we discuss the proposed device-free localization method using multiple SVMs to enhance localization accuracy. As mentioned in section 3.2, the method requires many signal features obtained from multiple channels to assure an acceptable localization accuracy. In many cases, however, multiple channels may not be able to exist in the area of interest, in which case the number of features is decreased and the localization accuracy is degraded. Moreover, the method presents another problem which decreases localization accuracy. That is, the method can estimate the trained locations only. In case the distribution of trained locations is very sparse, the localization accuracy is also degraded. In general, creating a training data set is very time-costly, because human subjects have to conduct experiments for each location. To overcome the aforementioned problems, we present a new device-free localization method called passive localization with array sensor (PLAS), which uses multiple SVMs to select optimal features among small number of features. We use a fingerprint-based technique

69 56 with multiclass support vector machines (SVMs) based on a combination of array signal features with spatial and temporal averaging. We evaluate the localization performance of the proposed system in different propagation environments: LOS and NLOS. In addition, we analyze two types of receive antenna placement: centralized and distributed antennas. The experimental results show that the localization accuracy can be improved by the proposed system, particularly in the centralized antenna case. Moreover, they show that the proposed system can improve localization accuracy compared to the conventional RSS-only based system. The main contributions of this study are summarized as follows: We propose the PLAS method that uses not only RSS measurements, but also signal subspace features, i.e., the change of signal eigenvector. We introduce a combination of SVMs for fingerprint-based device-free localizations. In addition, we design a probabilistic-based outlier mitigation scheme to filter out the potential outliers far away from the mean of the estimated ones, which further improves the quality of the classifier. We also investigate and evaluate the impact of antenna deployments: centralized and distributed antennas. Finally, through extensive experimental results, we evaluate the impact of system parameters and show that the proposed method improves accuracy by 26 % compared to the conventional distributed RSS-based device-free localization system, under the median distance error (MDE) metric for the centralized antenna deployment case. The rest of this chapter is organized as follows. Section 4.2 introduces the proposed localization method, PLAS, includingthefingerprintingalgorithmandtheerrormitigation. We describe the experimental results in Section 4.3. Finally, we conclude this chapter in Section 4.4.

70 The PLAS System Model In this section, we first give an overview of the PLAS system, then describe the proposed algorithm Overview In this chapter, we focus on fingerprint-based device-free localization. As general fingerprintbased localization systems, the PLAS system also has two phases: training and localization. In the training phase, we construct a radio map that has measured signal features and its corresponding locations. Then, we train the radio map to generate localization models using multiclass SVMs. In the localization phase, we collect radio waves with the array sensor in real time. The measured radio waves are extracted and used as signal features. With the SVM models, we classify unknown locations based on the signal features, where the target person is located. Fig. 4.1 shows the detailed algorithm of the system. In the following subsection, we will describe feature extraction. We then show the proposed localization method with feature selection scheme, then how an estimated location error can be mitigated to improve localization accuracy Feature Extraction As mentioned in subsection 2.3.4, we can extract signal features from the signal eigenvectors of the data correlation matrix of the received signal data. We use the cost function of the signal eigenvectors P i (t) whichisexpressedineq.(2.17),wherei and t are the index of signal eigenvectors and observation time, respectively. We consider, without loss of generality, a single channel transmitter and an array antenna at the receiver side as shown in Fig. 3.1(a). In such scenario, we can only obtain the first signal eigenvector of the data correlation matrix [34, 35]. We use the value of the cost function P 1 (t) asasignalfeature. Tosimplify,the

71 Figure 4.1: Algorithm of proposed localization system 58

72 59 notation used throughout the rest of this chapter, P (t) isusedinplaceofp 1 (t). RSS is easily affected by the spatial and temporal variance. Yet, RSS has the ability for localization particularly in LOS environments. When a person is located along the LOS path, the dominant LOS signal is significantly decreased in RSS (db) [24]. Therefore, we also use RSS values from each element of the array as a feature. RSS value of each element i of array at time t is expressed as follows. N s RSS i (t) =20log 10 ν + V i exp(jφ i ) i=1 (4.1) where ν, V i,φ i are the AWGN noise, the magnitude, and the phase of the ith multipath wave impinging on the receiver antenna, respectively Localization with Error Mitigation Scheme We consider feature selection based on the aforementioned features. All the features are grouped into the δ-combination ( δ ),where andδ are the size of the feature set and the size of feature subset, respectively. The subset size δ is an important parameter, and can be tunable and predetermined. We will discuss the impact of the parameter in Section 4.3. Each subset is trained using independent multi-class SVMs during the training phase. After training the SVM models, the models are used to estimate the person s locations during the localization phase. The outputs of each SVM may be different, because the features of SVM are all different. We count the frequencies of the outputs given by the SVM classifiers. Thus, a reliable output shows the higher frequency and the others are lower than that. We select the k most reliable output locations from the all SVM outputs. The value of the system parameter k ranges from 1 to 5 and is selected using a greedy search algorithm with other system parameters. Note that when the number of SVMs is 1, k is also 1, since the output of the SVM is 1. In this study, we use k =4. Experimentinsection4.3hasproven that this value gives the optimal results Using the parameter k, theestimatedlocationȳ is

73 60 calculated as follows. ȳ = 1 k Θ(i) i=1 k Θ(i)ȳ (4.2) i=1 where Θ(i) andȳ are the probability of a location and the vector of estimated locations, respectively. The spatial averaging process is related to the conventional method in [32]. In the time averaging phase, we use a time averaging technique to avoid being affected by outliers of outputs. The time averaging is also similar to that in [32], however, we modify the process to make localization performance robust to outliers. If the length of the time averaging window w is larger than the observation time t, wemeasuretheeuclidean distance D = {d 1,d 2,,d w } between each location ȳ i and the mean of locations ŷ within w. TheelementsofdistancevectorD are ordered in descending order. Since outliers are the elements most likely to have the highest distance from the center, they would be located at the beginning of the sorted array D. Thismakeseasytofindoutliers,byiteratingD until the first element below outlier threshold κ is found, where the optimum threshold value is determined empirically. Next the outliers are replaced with the estimated location closest to ŷ. Wecanexpresstheestimatedlocationclosesttoŷ as argmin ŷ ȳ i α, α {1, 2} (4.3) ȳ i w where. α is the L α distance. In this chapter, we use α =2. Finally, we estimate the averaged location as follows. ỹ = 1 t ȳ(i) (4.4) w i=t w+1 The outlier threshold κ may affect the localization accuracy. We will discuss the impact of the parameter in Section 4.3.

74 61 Table 4.1: Description of Experiments No. Propagation condition Antenna placement 1 LOS Centralized antenna 2 LOS Distributed antenna 3 NLOS Centralized antenna 4 NLOS Distributed antenna 4.3 Experimental Results In this section, we present our experimental results to show the localization performance of the proposed PLAS system compared with the state-of-the-art device-free localization system, Nuzzer, as a conventional system. First, we describe the experimental environment, setup, and system parameters. Second, we show the measurements comparison between RSS and P (t). Third, we evaluate the impact of the system parameters on localization performance. Finally, we compare the localization performance of the proposed method and the conventional one. As a baseline for both localization methods, we also use a random estimation which returns uniformly distributed random locations without any signal feature information. For comparison between Nuzzer and PLAS, we use the RMSE, the median distance error (MDE), and the cumulative distribution function (CDF) versus distance error Experimental Environment We conducted four different experiments to evaluate the effect of propagation condition and receive antenna deployment summarized in Table 4.1. These evaluations are important in determining whether the system has generality, flexibility, and simplicity, and it can be used for real applications. Note that all experiments were conducted at different times for

75 Figure 4.2: Experimental environment. 62

76 63 Table 4.2: Experimental Parameters Transmit antenna Dipole antenna Modulation method No modulation Transmit frequency GHz Number of transmit antennas 1 Number of receive antennas 4 Sampling frequency 60 Msps Number of snapshots 1024 Grid distance 1.5 m Number of training samples 100 samples/location Number of localization samples 500 samples/location training and testing. We performed the experiments in a typical classroom as shown in Fig The experimental environment was a complex area bounded by glass windows on one side and concrete walls on the other. We set a single band of 2.4 GHz (IEEE802.11g), which was continuously transmitting from the transmitter (Tx). We used a four-element antenna array at the receiver (Rx). To analyze the effect of antenna placement, we set up the antenna array at the green and purple triangles for centralized and distributed antennas, respectively. Thus, we had four received signals for calculating signal features. Before generating fingerprinting data of the target, we first collected the array data in an empty room, as reference value. Note that it takes only a few seconds, e.g., in less than 10 seconds. Three human subjects conducted the experiment for training and testing. We consider a localization scenario in which a single person is in a single room environment, to prove the feasibility of the proposed algorithms and to show how to obtain the optimal values of the system s parameters. In future work, we will extend the proposed algorithms

77 64 for multiple people localization in multi-room environments. We measured data while each subject stood at the particular position (see Fig. 4.2). The reference vector v no in eq. (9) was collected in advance at the same room with no people. The distance between two training locations was 1.5 m. In total, 16 training locations for training were sampled. Also, an independent test set at 12 locations was collected at different times. Consequently, we summarize the experimental parameters in Table 4.2. To analyze the effect of propagation condition, we conducted experiments in both conditions: LOS and NLOS. In LOS condition, there is a direct-path signal between Tx and Rx. However, in NLOS condition, there is no direct-path signal that is dominant over the signal subspace spanned by eigenvectors, thus the signal subspace spanned by the eigenvectors enhances the impact of multipath signals that capture the change of the environment. We also considered two different types of antenna placement, distributed and centralized. We will show the results and analysis in the next section Comparison between RSS and Signal Eigenvector Figs. 4.3 and 4.4 show examples of RSS and signal eigenvector P (t) valueswhennoevent occurs and a person stands at the training locations. As we can see from Fig. 4.3(a) and Fig. 4.4(a), RSS values fluctuate over observation numbers despite that no event occurs. In contrast, P (t) valuesareverystableinbothevents,noeventoccursandapersonstandsas shown in Fig. 4.3(b) and Fig. 4.4(b). More interestingly, compared with Fig. 4.4(a) and Fig. 4.4(b), RSS values while a person stands overlap with the no event case. However, P (t) values while a person is standing still, are different from no event case and more temporally stable. Note that the vertical interval of P (t) (Fig. 4.3(b) and Fig. 4.4(b)) is 0.02 which is ten times smaller than that of RSS (Fig. 4.3(a) and Fig. 4.4(a)).

78 RSS (db) Observation Number (a) RSS No event Standing P(t) Observation Number (b) P (t) No event Standing Figure 4.3: Measurements comparison between (a) RSS and (b) P (t) for no event and a person standing. A person stands at location 8 (see Fig. 4.2).

79 RSS (db) Observation Number (a) RSS No event Standing P(t) Observation Number (b) P (t) No event Standing Figure 4.4: Measurements comparison between (a) RSS and (b) P (t) for no event and a person standing. A person stands at location 9 (see Fig. 4.2).

80 Centralized antenna in LOS Centralized antenna in NLOS 2.4 RMSE (m) Subset Size (δ) (a) Centralized antenna in LOS Centralized antenna in NLOS 2.4 RMSE (m) Outlier Threshold (κ) (b) Figure 4.5: Impact of system parameters on RMSE during centralized antenna in LOS and NLOS cases. (a) Subset size δ varies from 1 to 5 while k =4,w=100,κ=1.0. (b) Outlier threshold κ varies from 0.2 to 2 while k =4,w=100,δ=3.

81 Impact of Parameters on Localization Accuracy Impact of Subset Size (δ) As explained in Section 4.2.3, the subset size δ of all the features in the PLAS is an important parameter. If the size of δ is too small, only a few signal features will affect the system s localization accuracy. In contrast, if the size of δ is too large, the combination of the feature set and its corresponding classifiers would be too few, then the localization accuracy would be degraded. Fig. 4.5(a) shows the impact of δ on RMSE in meters. The RMSEs go up drastically as δ increases beyond 3 in both LOS and NLOS cases. Since the number of features in these experiments is 5, the maximum combination of features is 10, i.e., ( 5 3),whichcontributesto the improvement of localization accuracy. Therefore, we use δ =3intheseexperiments Impact of Outlier Threshold (κ) In general, a person can move in a limited area within a short time period. The purpose of outlier threshold κ (in meters) is to mitigate the distance error by outliers, and the optimum threshold value is determined empirically. As explained in Section 4.2.3, if the κ is set too low, well-estimated locations would be misclassified as outliers. If the κ is set too high, many outliers would be remained and so will not contribute the error mitigation. Fig. 4.5(b) shows the impact of the threshold κ. Themostaccuratelocalizationisachievedwhenκ values are 0.8 and 1 in LOS and NLOS, respectively. We set κ =1intheseexperiments Comparison with Other Algorithms Figure 4.6 shows the CDFs of the distance error in meters in the random estimation, Nuzzer, and PLAS. From Fig. 4.6(a), we can see that PLAS outperforms the other techniques in the centralized antenna case. The RMSE and MDE of the PLAS are 2.06 m and 1.68 m, respectively, and achieve 16 % (RMSE) and 26 % (MDE) improvement compared to

82 CDF Random 0.1 Nuzzer PLAS Distance Error (m) (a) CDF Random 0.1 Nuzzer PLAS Distance Error (m) (b) Figure 4.6: CDFs of the random estimation in LOS, Nuzzer, and PLAS: k =4,w =100,δ = 3,κ=1.0. (a) Centralized antenna in LOS, (b) Distributed antenna in LOS.

83 CDF Random 0.1 Nuzzer PLAS Distance Error (m) (a) CDF Random 0.1 Nuzzer PLAS Distance Error (m) (b) Figure 4.7: CDFs of the random estimation in NLOS, Nuzzer, and PLAS: k = 4,w = 100,δ = 3,κ= 1.0. (a) Centralized antenna in NLOS, (b) Distributed antenna in NLOS.

84 71 Table 4.3: Comparison of All Experiments No. RMSE MDE Nuzzer PLAS Nuzzer PLAS 1(LOS&Centralized) 2.45 m 2.06 m 2.27 m 1.68 m 2(LOS&Distributed) 1.95 m 2.08 m 1.64 m 1.82 m 3 (NLOS & Centralized) 2.36 m 1.99 m 2.23 m 1.74 m 4 (NLOS & Distributed) 1.77 m 2.13 m 1.49 m 1.85 m Average 2.13 m 2.06 m 1.91 m 1.77 m Nuzzer, as shown in Fig. 4.6(a). On the other hand, during the distributed antenna case in LOS, the accuracy of PLAS is similar to or slightly lower than that of the conventional one, as shown in Fig. 4.6(b). These results happen in NLOS cases as well shown in Fig. 4.6(b) and Fig. 4.7(b). This is because a distance between antenna elements d that exceeds the half wavelength λ/2, i.e., the distributed antenna case, produces ambiguity errors [63]. Consequently, we can say that the localization accuracy of the PLAS is dependent on the receive antenna placement rather than propagation condition. We summarize the localization accuracy of all experiments by comparing the PLAS and the Nuzzer in Table 4.3. In summary, our proposed system can improve the average localization accuracy to 2.06 m (RMSE) and 1.77 m (MDE), which significantly outperforms the state-of-the-art device-free localization system Nuzzer. 4.4 Conclusion In this chapter, we have presented a multiple SVMs based method with an outlier mitigation scheme for device-free localization systems. The proposed method outperforms considerably

85 72 the conventional method, particularly in the case where the antennas are close to each other. We also evaluated the performance of the proposed method under different propagation conditions: LOS and NLOS. In addition, we conducted the two types of receive antenna placement: centralized and distributed antennas. The experimental results showed that the localization accuracy can be enhanced by the proposed method.

86 73 Chapter 5 State Classification Method for Human Activity Recognition 5.1 Introduction In this chapter, we study the method of implementation for device-free activity recognition in the array sensor. The proposed methods in Chapters 3 and 4, focus on the device-free localization using the array sensor. Unlike the localization methods mentioned above, here we take into account temporal changes of activities using a time window to select signal features. For example, when a person walks around the room, the temporal change of propagation environment is different from that when standing, sitting, falling and others. To find a best fit time window, we validate the length of the time windows experimentally. An SVM is applied to recognize human activity, and is experimentally validated. Results of extensive experiments including real life situations, (e.g., standing in a bathroom), show that the proposed method can recognize the target states with high accuracy. In addition, the proposed method confirms that the signal eigenvector as a feature has higher classification accuracy than the RSS in activity recognition cases. Moreover, we also investigate the impact of kernel functions of SVM upon classification accuracy.

87 74 The rest of this chapter is organized as follows. In Section 5.2, we introduce the proposed state classification method for activity recognition using an SVM. We show the experimental evaluation in Section 5.3 and conclude this chapter in Section Proposed State Classification Method In this section, we describe our proposed state classification algorithm for the array sensor. As described in section 2.3, the conventional threshold-based method uses a threshold to classify two states which are linearly separable case: one is very close to 1, and the other one is smaller than 1. However, in non-linearly separable case, it is difficult to set an optimum threshold. This means that the optimum threshold should be flexible and it can classify nonlinearly separable data. To solve the problem, we use one of the well-known and powerful machine learning algorithm, SVM. Our proposed approach follows a three-phase process: feature extraction, training, and state classification. A flowchart of our proposed algorithm is shown in Fig As shown in this figure, the feature extraction phase is a common process for both training and state classification phases. The main process is a general machine learning algorithm which used in localization algorithms as mentioned in the previous chapters. The most difference between this study for activity recognition and the other studies for localization, is the feature extraction process. As aforementioned above, human activity is time varying. We consider the temporal changes of activities of a person using a time window to select signal features, i.e., signal eigenvector. Note that the feature extraction phase is a common process for both training and state classification phases Feature Extraction In the feature extraction phase, array signal vector are measured at the array sensor. The measured array signal vector are analyzed via array signal processing to obtain the data

88 75!!!" Received Signals Feature Extraction" Correlation Matrix EVD Cost Functions Feature Vector SVM Training State Classification Figure 5.1: Proposed state classification algorithm.

89 76 Label τ x y 1 y 2 L P (1) P (2 ) P(τ ) L P ( τ +1) P ( τ + 2) P( 2τ ) M M M O M n y n P(( n 1 ) τ + 1 ) P(( n 1 ) τ + 2 ) L P( n τ ) Figure 5.2: An illustration of feature vector with its label. correlation matrix R. As shown in Section 2.3, from the data correlation matrix, we can obtain eigenvectors and eigenvalues that span signal subspace, and eigenvectors and eigenvalues that span noise subspace. From the eigenvectors and eigenvalues spanning the signal subspace, we compute cost functions P (t) andq(t). We use the cost functions based on eigenvector and eigenvalue as features for state classification. We also consider the values of the cost functions in a specific length τ (called frame length) as a feature vector, as shown in Fig This is because human or object movement is time dependent. Measured values of cost functions within τ, areusedforonefeaturevectorwithlabely i Training From the feature extraction phase, we can obtain feature vector with the frame length τ. In the training phase, we first assign a y i label for each feature vector x. Thelabelsrepresent state classes. Then, we select optimal kernel parameter and penalty parameter given the

90 77 training data. To find appropriate kernel parameter and penalty parameter, we validate the training data including feature vectors with its label. We use the k-fold cross validation method and the grid search method. The training data are separated into n subsets of equal size. One of those subsets is considered as the validation set and the rest are used for the training procedure. In the cross validation procedure, the error of training data is calculated n times with different combinations of training and validation data. During the cross validation procedure, kernel parameter and penalty parameter are changed by the grid search method. We use κ variables to find the optimal kernel and penalty parameters, γ 1,,γ κ and C 1,,C κ, respectively. After finishing the cross validation procedure with the grid search, we can find an optimal kernel and penalty parameters. Then, the training data with the optimal parameters are mapped into high dimensional feature space by kernel trick and the SVM model is obtained State Classification The goal of the state classification phase is to find unknown class labels corresponding to states. In the state classification phase, we first make feature vector in the same way as in the training phase. Then, using the SVM model from the above training phase, we can classify all unknown samples in real time. We classify the state of human or object based on above algorithm. 5.3 Experimental Evaluation To evaluate the performance of our proposed method, we show our experimental results obtained in different environments: a bathroom and an office room. Transmission frequencies for a bathroom environment (experiment 1) and an office room (experiment 2) are GHz

91 78 Table 5.1: Experimental parameters Experiment 1 Experiment 2 Transmission frequency GHz GHz Transmission power 10 dbm Modulation method No modulation Transmitter Dipole antenna Receiver 8-element patch antennas Sampling rate 20 Msps Number of snapshots and GHz, respectively. The receiver consists of 8-element patch antennas. There are some obstacles between the transmitter and the receiver to make non-line-of-sight (NLOS) situation. In addition, the transmitter and the receiver are configured independently such that time synchronization is not required. The number of snapshots for the experiment 1 and the experiment 2 are 1000 and 8192, respectively. Experimental parameters are summarized in Table 5.1. We evaluate whether a classification is accurate, not only for the simple case (e.g., walking) but also for various cases (e.g., standing, sitting, and falling). In this chapter, the classification accuracy is evaluated as follows: Experiment 1 The number of correctly classified data The number of total data 100 %. (5.1) We evaluate the performance of our proposed method under real-life scenarios in a bathroom as shown in Fig We define seven states for classifying as described in Table 5.2. Also, we design six scenarios combining the seven states, which are described in Table 5.3. In

92 79 Figure 5.3: Experimental environment 1 Table 5.2: Descriptions of seven states used in Experiment 1 State Description (1) No event Nothing happens (2) Walking ApersoniswalkingfromAtoB,openingdoorsAand Bintheprocess (3) Entering into a bathtub ApersonisenteringintoabathtubatC (4) Standing while showering ApersonisstandingwhileshoweringatB (5) Sitting while showering ApersonissittingwhileshoweringatB (6) Falling down Apersonisstanding,thenfallsatB (7) Passing out ApersonispassingoutinthebathtubatC

93 80 Table 5.3: Descriptions of six scenarios used in Experiment 1 Scenario 1 (S1) State (1), then (2), then (3) Scenario 2 (S2) State (1), then (2), then (7) Scenario 3 (S3) State (1), then (2), then (6) Scenario 4 (S4) State (1), then (2), then (3), then (6) Scenario 5 (S5) State (1), then (2), then (4), then (6) Scenario 6 (S6) State (1), then (2), then (5), then (6) the training and classification phases, we obtained the data while a human subject performs each action detailed in Table 5.2. Training data in scenario 1, 2, 3, 4, 5, and 6 are measured during 90 s, 90 s, 60 s, 120 s, 120 s, and 120 s, respectively. In practice, samples of array output data are measured during 1 s. From the samples, we only use samples to reduce computation time. Then we use 1000 samples of the samples as snapshots to calculate the correlation matrix R. Thus,we can obtain 10 features (i.e., cost function) per second. Actual measured time is dependent on the sampling frequency and the number of snapshots of the system. When the sampling rate and the number of snapshots are f s and N s,respectively,theactualmeasuredtimefor one feature is 1 f s N s. The measurement time for classification data are also same as that for the training data. We repeat the experiments twice for each scenario: the first measurement data of the experiment are used for training, and the other data are used for classification. Thus, the number of training and classification data is total = 1800 (S1) (S2) (S3) (S4) (S5) (S6).

94 Accuaracy (%) Frame length (τ) Figure 5.4: Average classification accuracy with different frame length τ values. Classification Result (Label) Classified Label Actual Label Time (s) Figure 5.5: An example of classification results. Label 1: No event. Label 2: Walking. Label 4: Standing while showering. Label 6: Falling down.

95 Classification Results of Proposed Method We evaluate the effect of the frame length τ on our proposed classification method for six scenarios. Fig. 5.4 shows the average classification accuracy results with different τ values, i.e., {τ =1, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50}. Fromtheresults,wecanseethatτ =30 gives more accurate classification than the others. The average accuracy of this experiment with τ =30andGaussianRBFkernelis96.5 %.Ifτ is set very small, a very small amount of data is used for the feature, and the classification accuracy is degraded. On the other hand, if τ is set very large, a target state within the τ will not affect significantly to the classification. We present an example of multi state classification results by our proposed method with τ = 30 as explained below. Fig. 5.5 shows an example of classification results: the actual class label corresponding to each state for comparison with the result of our proposed method. Classification results show that the classified labels are same as the actual ones except 2 labels. Although there are two classification errors in this result, it is not important because the errors can be easily removed by observing that some rapid transitions of labels are not possible for persons. The result is similar to other scenarios in the experiment State Classification based on RSS To show the effectiveness of our proposed method, we compare the conventional state classification based on RSS. We use the mean of the 1000 RSSs samples. Table 5.4 shows the accuracy comparison results with different τ values, i.e., {τ =1, 10, 30}, betweentheconventional method based on RSS and the proposed method. From Table 5.4, we can see that the proposed method is much better than the conventional method. This is because that RSS variations over time are large owing to fading and noise.

96 83 Table 5.4: Accuracy comparison between RSS (conventional) and signal subspace feature using the proposed method Average Classification Accuracy (%) Conventional Proposed τ =1 τ =10 τ =30 τ =1 τ =10 τ =30 S S S S S S Avg Figure 5.6: Comparison of kernel functions

97 84 Table 5.5: Descriptions of five states used in Experiment 2 State Description (1) No event Nothing happens (2) Walking Apersoniswalkingfromthedoor1tothedoor2viaBandA (3) Standing ApersonisstandingstillateitherAorB (4) Sitting ApersonissittingateitherAorB (5) Falling Apersonisstanding,thenfallsateitherAorB Comparison of Kernel Functions As noted in subsection 2.4.3, the kernel function of SVM is one of main factors to affect its classification accuracy. However, it is difficult to choose the best kernel function before testing system, because each system has different data construction. In this experiment, the kernel functions (i.e., linear, polynomial, and Gaussian RBF), are used for choosing the best suited for our data. The average accuracies with a linear kernel and a polynomial kernel are % and %, respectively, as shown in Fig We can see that Gaussian RBF kernel has better classification accuracy than the other kernel functions. These results are similar to [69], which uses SVM for infrared gait recognition. In [69], it is also shown that SVM with Gaussian RBF kernel achieves better performance than the other kernel functions. Based on these results, we choose the Gaussian RBF kernel as best kernel for our system Experiment 2 In this experiment, we evaluate our proposed method based on SVM with Gaussian RBF kernel, in an office room with concrete and glass walls as shown in Fig The transmitter and receiver (array antenna) are fixed in NLOS condition. In the NLOS case, there is no direct-path signal that is dominant over the signal subspace spanned by eigenvector.

98 85 Figure 5.7: The setup for the office room of the array sensor. Note that the transmitter Tx is placed under the desk to make NLOS environment.