Extended Touch Mobile User Interfaces Through Sensor Fusion

Transcription

1 Extended Touch Mobile User Interfaces Through Sensor Fusion Tusi Chowdhury, Parham Aarabi, Weijian Zhou, Yuan Zhonglin and Kai Zou Electrical and Computer Engineering University of Toronto, Toronto, Canada Abstract This article explores an efficient sensor fusion algorithm for detecting and classifying user taps on any neighboring surface even in the presence of various background acoustics. The fusion algorithm employs a tier classifier combining microphone and accelerometer detection of user taps on ios platform resulting into 100%success rate for the datasets studied in this paper. Fusion of these two sensors eliminates the need for any added filtering, knowledge of precise sensor positioning or use of any specialized piezoelectric sensors, as has been done in past research, as well as gives a robust classification with high successrate even as the signal to noise ratio significantly degrades. Keywords extended touch surface, tactile acoustic interface, TAI, tap localization, tap detection, tap inference I. INTRODUCTION Extended touch technology enables the capability to transform an ordinary surface into an input medium for digital devices, an inexpensive and convenient alternatives to modernday touch interfaces. A solution that is portable, results in high-accuracy detection, and is robust to various ambient noise, could be used to develop software solutions that can turn an every day surface into a keyboard, tap-input, gaming controller, gaming platform, musical instrument simulator, or any types of smart surfaces. This technology could also be applied to infer key-strokes from a physical keyboard placed on the surface, thus introducing a new topic in mobile security exploration. The same technology could possibly be extended to many other forms of human computer interaction (HCI) products such as one could imagine developing a dance movement detection application using the extended touch technology to develop a virtual dance and workout instructor. In this paper, we investigate a sensor fusion algorithm that extends the Extended Touch technology presented in [1] and makes it robust to various background acoustics and random noise on ios platform. Our proposed solution uses the vibration and acoustic impulse response of a tap combined with machine learning algorithm to learn surface characteristics, which is then exploited to derive the location of the tap. Fusion of accelerometer and microphone helps build a reliable classifier that results in high success rate even as the signal to noise ratio significantly degrades. These sensors are ubiquitous in most modern day smart phones, have low power foot-print and high dynamic range, making them an excellent choice for feature development on portable devices. II. PROBLEM STATEMENT AND PRIOR WORK The problem we are aiming to solve in this paper is demonstrated in Figure 1, where our objective is to employ a Fig. 1: Inference of user-tapped location on the surface robust learning algorithm on the ios platform that can reliably detect user tapped locations even in challenging environment with background sound and degrading SNR. The vibration generated from the tap reaches the device and is read using the accelerometer and microphone. Taps from different locations generate different impulse responses due to variable distance travelled, and various degrees of reflections, scattering and dispersion in the medium. The uniqueness of the impulse response pertaining to different tap locations is utilized by the Extended Touch technology [1] to differentiate and localize user taps on a neighboring surface. However, if there is ongoing conversation, music playing or any types of background acoustic present during the tap classification, it can become increasingly difficult to tell apart background acoustics from the impulse response corresponding to a tap event using just microphone signal alone as initially proposed in [1]. In this paper, a sensor fusion algorithm is explored that eliminates the confusion of when a true tap event has occurred resulting into a highly successful classifier that is efficient and robust to background noise. In the literature, there exists several signal analysis and machine learning techniques that solve related problems [2] [5]. Most techniques apply time delay of arrival that uses multiple spatially separated sensors whose precise positioning is known to solve for tap localization. Some use more sensitive piezoelectric sensor, sometimes combined with amplification, with cross-correlation as the feature vector and a weighted averaging of training locations to output tap locations. Neural network was also applied to some of the solutions, although not to much success. The detailed background is discussed in the prelude of this paper in [1] in the prior work section. Past approaches while providing valuable background to our research, have different limitations that make them not directly transferable to the problem solved in this paper.

2 There has also been several security related research that studied inference of keystrokes from nearby keyboard using accelerometer and acoustic sensors. This was achieved by using accelerometer recordings corresponding to key press events from an Apple wireless keyboard placed 2 cm away from the phone in [5]. They trained a Neural Network using features such as mean, max, min, FFT, MFCC from accelerometer. However, due to the limited operational frequency of iphone accelerometer, which is 100Hz, the neural net only achieved a detection rate of 25.89%. Their second approach used the same feature vector along with relative position of subsequent pairwise key-presses, trained with 150 key strokes per letter at random order and timing - constituting of 3900 distinct events, combined with word matching. This resulted in slightly better performance but varied between 91% to 65% detection rate depending on the left/right or far/near positions and the size of the data set. The same problem was solved in [6] by using microphone data to train a neural network, where a PC microphone to listen to key-stroke events sampled at 44.1kHz. The FFT of the acoustic sample was used together with pairwise key pressing event to train the neural net with 100 key press samples per key. They managed to achieve a detection rate of 79% with the acoustic feature vector with the same amount of training. These investigations while being reasonably successful, require a large training phase to train the neural net combined with precise knowledge of relative key positioning of various keyboards. Moreover, the performance degrades with distance and with duration of detection. A solution that requires significant training phase can not be applied to our problem statement as the software needs to calibrate quickly to a surface to be truly portable. In this paper, a sensor fusion algorithm is presented that augments the K Nearest Neighbour algorithm discussed in [1], that can use only 1 or 2 training samples per class to give a high detection rate even in the presence of challenging background acoustic level. While the solution presented is robust to noise, it is also portable due to the short training phase that doesn t depend on any prior knowledge of sensor positioning or surface parameters. III. WAVE PROPAGATION AND ANALYSIS TECHNIQUE The waves generated from a tap can be modeled as Rayleigh waves and Love Waves [7], [8]. The wave propagation characteristics depends on various surface parameters such as thickness of the medium, elasticity, and boundary conditions giving rise to variable wave-propagation characteristics due to reflection, absorption and dispersion in the medium. If the surface boundary dimensions and elasticity is known, one could solve for the source of acoustic signal using the Image model presented in [9] that solves for reverberation period by modeling reflections with 3-dimensional mirror images of the source. A second acoustic analysis technique called Time Delay of Arrival (TDOA) described in [2], [10] could also be used to solve for the acoustic source given the use multiple spatially separated sensors whose relative positioning is precisely known. However, on ios platform, the sensors positioning vary from device to device, and our investigation of gyroscope signal did not show any significant detection for a tap events. Moreover, as the solution needs to work on any surface of arbitrary size whose dimensions would be unknown to the tap-inference software, image model also can not be applied. Therefore, our algorithm takes advantage of a different property of wave propagation where a vibration source in a closed system gives rise to a unique impulse response depending on the source-location. A. Cross-correlation and K Nearest Neighbour Given any surface, taps at different locations generate unique impulse response corresponding to the location of the source. The Extended Touch technology presented in [1] employs a k Nearest Neighbour algorithm that applies this uniqueness property to solve for tap inference. The knn uses cross-correlation on the audio feature vector to determine the closest match for a sample tap combined with a voting scheme to output a predicted tap location, details are provided in [1]. This is achieved by first storing a map of known locations to corresponding impulse response samples, referred as the training data. A newly recorded tap sample is classified using this training database. This is a statistical pattern matching approach which does not require prior knowledge of surface properties, or an extensive training phase as needed to train a neural network. Moreover, in this paper, we illustrate how accelerometer can be used to augment the detection to only run the classifier if a tap is detected, resulting into very low misclassification rate even in the presence of noise. IV. SENSORS In this section a brief discussion of the two sensors - accelerometer and microphone, used by the sensor fusion algorithm is presented. Their combined usage results in an efficient classifier that requires minimal training and is robust to various ambient noise. The gyroscope on the iphone was also investigated, however did not show any significant detection corresponding to a tap event. A. Accelerometer Accelerometer on ios devices and other consumer products has been used for a wide variety of applications - user motion and activity detection, shake, tap, vibration or sudden impulse detection, aid GPS in navigation and so on. Various quantities such as mean, max, and min amplitude, mean wave period, interval average acceleration as function of time, interval root mean square acceleration as function of time, trim-mean acceleration vs time, power spectrum density vs frequency, and various signal analysis techniques could be used to extract information from an accelerometer signal. However, one of the constraining factor of accelerometer is max-allowable frequency and dynamic range available on a particular platform. Apple uses STMicroelectronics LIS302DL piccole accelerometer, which measures acceleration in units of g, and gives a maximum frequency of 100 Hz, resolution of 0.018g and dynamic range of +/-2.3g1 [11]. Our investigation found that while accelerometer picks up distinct impulse response corresponding to a tap event, the sensor resolution is too low to contain enough wave-propagation characteristics to be useful in a cross-correlation analysis. Moreover, the accelerometer is sensitive to ambient vibration and can be quite noisy.

3 B. Microphone Microphone on ios devices vary in model, quality and number between generations and particular models. On iphone 4 and ipas, there are two microphones - and are known to have a sharp cutoff in frequency response at 250Hz [12]. On iphone 5, there are three microphones that support HD Voice - one in front and back near the camera, and one at the bottom - has double the frequency and spectrum width as of the previous models [13]. The extra microphones are used for noise cancellation, and are not accessible as separate audio units. High quality low latency audio input can be obtained by sampling at the recommended rate of 44.1kHz. Our investigation of microphone data corresponding to taps on a surface show a high correlation between signals generated by tapping in the same location, and low cross-correlation across tap impulse response from different locations. Moreover, in this paper we present how the cross-correlation can reliably indicate similarity between two signals from the same tap location even in the presence of background acoustic and various levels of gaussian noise. V. EXPERIMENTAL RESULTS - K NEAREST NEIGHBOUR CLASSIFIER A simple iphone application was developed to prototype a binary classifier using cross-correlation to infer tap locations. The algorithm stores one template waveform per tap location for each of the two sensors - microphone and accelerometer. The application then measures average accelerometer over a moving window, and once it detects a significant peak in accelerometer reading, it records a 0.23s microphone signal and does a maximum cross-correlation match with the two templates and outputs the best match as the predicted tap location. This was tested with 300 test samples over three different surfaces - hollow wood, glass, and metal, and resulted in a success rate of 92.66% [1]. A k Nearest Neighbour (knn) classifier was then simulated offline using iphone sensor recording from 17 different tap locations. The knn used the same maximum cross-correlation values between two signals as the distance measure to sort and rank predictions for output classes. The simulation included only 1 training sample per output class, and a k = 2 for the nearest neighbour calculation, and resulted in 92% successrate. The details of the results are discussed in [1]. VI. SENSOR FUSION FOR A ROBUST CLASSIFIER The k Nearest Neighbour algorithm to detect and localize taps using maximum cross correlation as the feature vector results in a very successful classifier. However, this task can become increasingly difficult in a noisy environment with various types of background sound. In this section, the effect of background sound on the performance of the classifier is studied, and a method is presented which will significantly improve the performance of the tap inference with background noise. A. Tap detection in the presence of background sound To study the effect of background sound on tap inference, we start by analyzing microphone and accelerometer data of knock taps while people are talking in the background. Fig. 2: Tap detection with talking in the background at (0, 20) Fig. 3: Tap detection at (-10, 0), talking in the background Each data sample consists of a time-series sensor recording of 2.5s length with continuous background sound. The actual tap occurs sometime within the time window. An example data sample consisting of microphone and accelerometer signal corresponding to a tap is shown in figures 2 and 3. The accelerometer shows no detection until the actual tap occurs, while microphone s recording of various background noise makes it difficult to narrow down the region where the actual tap has occurred. Therefore a peak in accelerometer reading could be used as a strong indication to detect a tap event and subsequently run the knn classifier on the microphone signal. As the first step of the analysis, we take a closer look at the tap classification error that would occur if accelerometer was not used to decide when to perform the cross-correlation to run

4 knn algorithm. To simulate tap detection algorithm running on a device every 10th of a second to output a detected tap and it s location, a moving window of 0.1s was run over the test data set consisting of about 4 test samples per location, each containing 2.5s of sensor recording per sample, with talking in the background and a tap event sometime within the test sample, which gives 100 detection windows per tap location. The microphone signal from this detection window (treated as test samples in this case) was then used to compute maximum cross-correlation feature vector relative to each of the training taps, and classify using the knn classifier with an added logic where no tap is detected if the goodness of fit (maximum crosscorrelation) is below a pre-set threshold. The classification results is discussed for different example threshold values and 0.05, which corresponds to the maximum cross-correlation that must be achieved for a test sample to be considered a valid tap sample and classification. The misclassification rate is then calculated as follows: error rate = n wt hresh where N tott hresh (1) n wt hresh = n wrong goodness of fit > threshold N tott hresh = N total goodness of fit > threshold TABLE I: Confusion matrix, talking dataset 1, threshold 0.1, 325 samples: 3 taps locations, 0.1s sliding window Predicted locations (0,20) (10, 0) (-10, 0) (0,20) (10,0) (-10,0) A second experiment was run with talking in the background that included more tap locations, same test setup, and taps that are as close as 5cm apart. The confusion matrix for data set with threshold 0.05 is shown in tables II and III. The results from both experiments demonstrate that in the presence of background acoustics, even by applying a threshold approach, taps can be falsely detected even when there are no taps, and misclassified leading to a high confusion rate and classification error. TABLE II: Confusion matrix, talking dataset 2, 7 tap locations, threshold 0.05, 0.1s sliding window Predicted locations (+10, 0) (+10,+10) (+15, 0) (+20, 0) (10,0) (10,10) (15,0) (20,0) (20, 20) (30, 30) (5, 0) TABLE III: Confusion matrix, talking dataset 2, 7 tap locations, threshold 0.05, 0.1s sliding window (continued) Predicted locations (+20,+20) (+30,+30) (+5, 0) (10, 0) (10,10) (15, 0) (20, 0) (20, 20) (30, 30) (5, 0) Fig. 4: Error rate, threshold = 0.1, 0.1s moving window across all test samples, a total of 325 detection windows The above simulation was run over two data sets from two different experiments with the same setup but containing different tap locations. For both of these data sets, 2 training samples per tap class, and k = 2 was used for the knn classifier. The training samples were clean tap signals with no background noise, where as the test samples contained talking in the background. The confusion matrix from dataset 1 computed over the sliding window consisting of essentially 325 test samples is shown in table I for threshold 0.1. The misclassification rate over the sliding window of detection algorithm is shown in figure 4 which shows that depending on whether the window coincided with a true tap event or not, the false detection rate could be high. The confusion matrix for the tap classes shows that there are substantial confusion between the tap classes in the presence of background sound even though a goodness of fit threshold is used. Now, we look at the slice of the microphone signal where the accelerometer does show a peak and classify the test samples using the same algorithm. The algorithm now starts the microphone cross-correlation over a 0.1s window only once the accelerometer reading peaks 20% past the minimum of the maximum accelerometer reading amplitude measured during the training phase. The algorithm uses the same max crosscorrelation feature vector and threshold value as before. The results show that the miss-classification rate drops to 0% for the same data set as presented in table I and for the data set from experiment 2 whose confusion matrix was shown in table II and III. The classification success rate for experiment 1 and 2 for all the tap locations was 100%. Therefore, the proposed algorithm has two tier logic - detection and classification. First tier detects a tap event by measuring the z-axis accelerometer amplitude within a 0.1s moving window. If the accelerometer peaks above the pre-decided value, measured during training phase by taking the minimum of the accelerometer z-axis max amplitudes, the next 0.1s window of microphone reading is passed onto the 2nd tier which classifies the sample by

5 employing the knn classifier with a specified goodness of fit threshold. Application of this tier classifier logic that checks the state of accelerometer to decide whether to perform knn classification, shows a significant performance improvement and reliable detection over the previous method that only relied on audio feature vector. TABLE IV: Confusion matrix for dataset 1, tier classifier using sensor fusion, 3 tap locations, 0.1s detection window Predicted locations (0,20) (10, 0) (-10, 0) (0,20) (10,0) (-10,0) B. Misclassification rate with degrading SNR In this section, a more in depth analysis is presented to gaze the performance gain obtained by using accelerometer to aid microphone detection of taps on the surface. To quantify the advantage, we study the rise in misclassification rate as the Signal to Noise ratio (SNR) degrades in the presence of background noise. The decrease in SNR is simulated by adding Gaussian white noise to the test samples by applying the Matlab function: awgn(< test sample >, < desired snr >, measured ) The same datasets discussed in the previous section containing 2.5s sensor recording were injected with the gaussian noise, to obtain varying target SNR levels to simulate various noise levels. Moreover, the trend of misclassification rate as function of degrading SNR is studied for different thresholds that could be used as a measure of goodness of fit by the classifier. We again use a moving window of 0.1s over each of the test samples and output a predicted tap location using the knn classifier based on maximum cross-correlation match. The figures 5 and 6 show the misclassification error rate as function of background noise with varying SNR for the 1st dataset for different threshold values for tap locations (0,20), (10,0) and (-10,0). The curve with circles show the misclassification rate when no accelerometer signal is used to detect a true tap event. The curve with stars show the error rate when classification is performed using the tier sensor fusion algorithm which detects a tap with accelerometer in the first tier and subsequently applies the knn classifier on the audio signal if the conditions in first tier is met. The error rate for the circle plot with threshold 0.1 corresponding to classification without using sensor fusion in Figure 5 abruptly stops at SNR 0 because none the samples below 0 SNR get classified as they fail to meet the goodness of fit threshold, leading to 0 for both denominator and numerator in equation (1) for error rate calculation. Figure 6 shows the error rate with and without sensor fusing for threshold 0.05 as the SNR degrades. The figures clearly show that in the presence of background noise, as the SNR significantly degrades, the tier classifier using sensor fusion gives a significant performance improvement up to a vary low SNR. Figure 7 show the misclassification error rate as function of background noise with varying SNR for the second experiment containing tap locations (+10, 0), (+10,+10), (+15, 0), (+20, Fig. 5: Misclassification rate as function of degrading SNR with talking in the background, threshold = 0.1, tap locations (0,20), (10,0) and (-10,0) Fig. 6: Misclassification rate as function of degrading SNR with talking in the background, threshold = 0.05, tap locations (0,20), (10,0) and (-10,0) 0), (+20,+20), (+30,+30), (+5, 0). The results from this data set also demonstrates that the error rate is very large when only microphone is used for tap inference, as opposed to sensor fusion algorithm that is robust to noise even in the presence of increasing background noise.

6 into high detection rate even in the presence of background acoustics, it also becomes robust to degrading SNR of up to 10dB or even lower. We demonstrated that by using simple machine learning algorithm with a training phase, a portable solution is achieved that works on any surface where the device is placed. Fig. 7: Misclassification rate as function of degrading SNR with talking in the background, threshold = 0.05, locations (+10, 0), (+10,+10), (+15, 0), (+20, 0), (+20,+20), (+30,+30), (+5, 0) Therefore, our proposed sensor fused solution for a robust tap localization solution is a tier classifier. The first tier identifies a tap event by running accelerometer detection over a moving window of 0.1s. This is done by first choosing a threshold for accelerometer z-axis based on the training data set, where it is set to be equal to the 20% of the minimum of the maximum amplitudes in accelerometer z-axis measured from all the template signals in the training phase. This threshold is then used to detect a true tap event during the detection phase. Once the first tier detects a tap using this criterion, a 0.1s window of microphone signal is captured which is passed onto the second tier. The second tier then applies the knn classifier which uses cross-correlation analysis on the audio feature vector combined with a maximum voting scheme and and the minimum goodness of fit requirement to output a predicted tap location for the tap sample. This corresponds to a tap detection response time of about 1/10 th of a second and results into an efficient and robust tap inference solution that can quickly train, and detect a tap with high accuracy even in the presence of various types of background acoustics and degrading SNR. VII. CONCLUSION AND FUTURE RESEARCH In this paper, a tier sensor fusion algorithm is presented that uses accelerometer and microphone on ios platform to detect and localize taps with 100% detection rate (for the data sets studied in this paper) even in the presence of people conversing in the background. Moreover, it is demonstrated that by fusing these two sensors the algorithm not only results Extended touch technology could be used to define new ways of interacting with our smartphones that allow for a natural interface for various use cases. One such example is developing a gaming interface or virtual musical instruments that use taps at different locations on the surface as input, which can easily be extended to multi-player platform. One could also utilize the detection capability to map various user defined actions such as sending an , flipping through pages or snoozing the alarm, to different tap locations. The technology can also be extended to detect other types of taps such as jumping on the floor which can be detected to develop a work-out or dance detection system. The robustness of the algorithm to degrading noise level and background sound makes such extended touch applications more usable and feasible user interface. Live videos demonstrating some of the applications of extended touch technology discussed above could be viewed at [14]. Future research step for this problem would be to recognize contiguous tap locations where not all tap locations have pre-existing templates in the training database. REFERENCES [1] T. Chowdhury, P. Aarabi, A. Heidari, W. Zhou, Y. Zhonglin, K. Zou, and B. Liu, Extended touch surface, IEEE International Conference on Multimedia and Expo, 2013, to appear. [2] S. Pollen and N. Radtke, Experiments in single-sensor acoustic localization, Research Project, Milwaukee School of Engineering, Tech. Rep., [3] N. Checka, A system for tracking and characterizing acoustic impacts on large interactive surfaces, Master s thesis, Electrical and Computer Engineering, MIT, [4] A. Crevoisier and C. Bornand. ( ) Tai-chi: Tangible acoustic interface. Research on Tangible acoustic interface. [Online]. Available: [5] P. Marquardt, A. Verma, H. Carter, and P. Traynor, Decoding vibrations from nearby keyboards using mobile phone accelerometers, In Proceedings of the 18th ACM conference on Computer and communications security, [6] D. Asonov and R. Agrawal, Keyboard acoustic emanations, In Proceedings of the IEEE Symposium on Security and Privacy, [7] A. Crevoisier and P. Polotti, A new musical interface using acoustic tap tracking, Universitat Pompeu Fabra (UPF), Barcelona, Spain, Tech. Rep., [Online]. Available: [8] T. H. Heaton, Surface waves, chapter 5, lecture notes, California Institute of Technology. [Online]. Available: heatont/ [9] J. Allen and D. Berkley, Image method for efficiently simulating smallroom acoustics, Journal of the Acoustical Society of America, [10] M. Fink and C. Prada, Acoustic time reversal mirror, Inverse Problems, [11] L. Gregor and J. Sadler. Gyroscope and accelerometer. [Online]. Available: emendelo/classes/spring12/csc360 [12] Audio frequency response of iphones. [Online]. Available: [13] iphone 5 microphone specifications. [Online]. Available:

7 [14] Extended touch: New apl technology makes any surface touchsensitive by just placing a mobile device on it. [Online]. Available: