A BAG-OF-FEATURES APPROACH TO ACOUSTIC EVENT DETECTION. Department of Computer Science, TU Dortmund University, Dortmund, Germany

Transcription

1 A BAG-OF-FEATURES APPROACH TO ACOUSTIC EVENT DETECTION Axe Pinge, René Grzeszick, and Gernot A. Fink Department of Computer Science, TU Dortmund University, Dortmund, Germany ABSTRACT The cassification of acoustic events in indoor environments is an important task for many practica appications in smart environments. In this paper a nove approach for cassifying acoustic events that is based on a Bag-of-Features approach is proposed. Me and gammatone frequency cepstra coefficients that originate from psychoacoustic modes are used as input features for the Bag-of representation. Rather than using a prior cassification or segmentation step to eiminate sience and background noise, Bag-of-Features representations are earned for a background cass. Supervised earning of codebooks and tempora coding are shown to improve the recognition rates. Three different databases are used for the experiments: the CLEAR sound event dataset, the D-CASE event dataset and a new set of smart room recordings. Index Terms Event detection, sound cassification, Bag-of-Features 1. INTRODUCTION The cassification of sounds in indoor environments is important for many practica appications. The detection and cassification of acoustic events can be used for meeting and onine ecture anaysis and annotation [1]. For speech enhancement and speaker tracking [2] detecting non-speech events can improve the robustness in rea word appications. The task is difficut because of the diversity of the acoustic events. Human speech is comprised of sounds of different phone casses, e.g. vowes, posives and fricatives that have individua spectrum and time characteristics. Other sound types are aso compex because they are comprised of a variety of individua sounds, e.g. chair movement can produce knocking and rubbing sounds, handing paper can incude rusting and knocking on the tabe and so on. Sounds ike footsteps are individuay different depending on the person and kind of shoes. It is desirabe for a sound cassification method to be abe to hande the diverse composition and generaize in a way to cover different, possiby unheard reaizations of the sound types. Over the ast decades, a number of approaches for acoustic event detection have been proposed [3 5]. State-of-the- This work was supported by the German Research Foundation (DFG) under contract number Fi 799/5-1. Siding Window MFCC BoSF Cassification Fig. 1. Overview of the Bag-of-Features based method that is used for acoustic event detection. art approaches in speaker cassification are based on a singe Gaussian mixture mode (GMM) [6]. Others use a set of GMMs that are individuay trained for each cass, where the GMMs estimates are summed over a frames and the cass with the highest ikeihood is chosen. Since the summation discards any tempora information, the method is sometimes termed Bag-of-Frames [5,7]. Since considerabe progress has been made by appying insights from human perception in the fied of computer or machine vision, simiar approaches have been advocated for acoustics [8]. The Bag-of-Features approach originated in text retrieva [9]. It has successfuy been used in various pattern recognition appications in recent years [ 12]. For exampe, in image cassification the Bag-of-Features is known to generaize we over very diverse casses, producing state-ofthe-art resuts [11]. Recenty, a basic version was appied to acoustic event cassification [13]. In this paper a nove Bag-of-Features approach based on soft quatization with GMMs is introduced. Experiments show that is abe to distinguish very diverse sound event casses. 2. METHOD For the acoustic event detection and cassification, a singe microphone or beamformed signa is processed in short time windows of 0.6 s every 0.05 s. For a given sampe n, a set of feature vectors Y n = (y 1... y K ) is cacuated for a frames in this window. These features are then softy quantized by a GMM and cassified by an mutinomia maximum ikeihood cassifier. Rather than using a prior cassification step to eiminate sience and background noise, as done in severa systems (cf. [3]), the rejection cass Ω 0 is trained with recordings where no event occurred Features For sound and especiay speech processing, the me frequency cepstra coefficients (MFCCs) are one of the most c 1 pubished in: Int. Conf. on Acoustics, Speech and Signa Processing (ICASSP) 2014

2 widey used features. The input signa is fitered by a me frequency fiter bank, from the ogarithm of its magnitude the discrete cosine transform (DCT) is computed and its second to 13th coefficient is used. The ong history of psychoacoustic research has been compemented by computationa modeing of the human hearing process [14] where ERB-spaced gammatone fiterbanks are used. From that the gammatone frequency cepstra coefficients (s) were derived [15]. In our impementation, we repaced the fiterbank of the MFCCs by inear phase gammatone fiters. The fiters are defined in the spectra domain using a gammatone approximation [16] with center frequency f b and bandwidth w b G (b) (f) = (1 + j(f f b )/w b ) 4, (1) where j is the imaginary unit Bag-of-Super-Features A Bag-of-Features approach (cf. [17]) is used for buiding a codebook of acoustic words from the training set. Most Bagof-Features approaches use custering agorithms, e.g. the Loyd agorithm, on the compete training set to derive a codebook and ater assign each feature to a centroid by hard quantization. However, disregarding the abes in the custering step can ead to mitigation of significant differences (cf. [18]). A remedy for this effect is to buid codebooks of size I for a C casses Ω c separatey and then concatenating into a arge super-codebook. Here, the expectation-maximization (EM) agorithm is appied to a feature vectors y k for each cass Ω c in order to estimate I means and deviations µ i,c, σ i,c for a C casses. We concatenate a means and deviations into a super-codebook v with L = I C eements v =(I c+i) = (µ i,c, σ i,c ) (2) where the index computed form the cass index c and the Gaussian index i as = I c + i. Using this codebook, a soft quantization of a feature vector y k can be computed as q k, (y k, v ) = N (y k µ, σ ). (3) Then, a histogram b can be computed over a K frames of the input window by b (Y n, v ) = 1 q k, (y k, v ). (4) K We refer to this method as Bag-of-Super-Features in anaogy to the super-vector construct used in speaker identification [6] Tempora Pyramid Since a Bag-of-Features is an orderess representation a tempora information within the frame Y n is ost. However, k this information may be important for distinguishing different acoustic events. In the ast years severa approaches have been pubished in order to address this probem. For exampe, spatia features [12] or pyramids [19]. The pyramid scheme is directy appied to the Bag-of- Super-Features approach by subdividing the window in a tempora manner. For a feature vector of the n th window two sub-histograms b (1) (Y n, v ) = 2 K/2 q k, (y k, v ) K b (2) (Y n, v ) = 2 K k=1 K k=k/2+1 and q k, (y k, v ) (5) are defined for the first and the second tempora haf. In addition, a max pooing step is used for computing the histogram for the whoe window by { } b (3) (Y n, v ) = max b (1) (Y n, v ), b (2) (Y n, v ). (6) A three histograms are then concatenated into a singe feature vector ( ) b(y n, v) = b (1) (Y n, v), b (2) (Y n, v), b (3) (Y n, v) (7) that represents the compete window Cassification The probabiity of an acoustic word for a given cass P (v Ω c ) is estimated using a set of training sampes Y n Ω c for each cass c by Lapacian smoothing: P (v Ω c ) = 1 + Y n Ω c b (Y n, v ) L + L m=1 Y n Ω c b m (Y n, v m ) Since a casses are assumed to be equay ikey and have the same prior, maximum ikeihood cassification is used. The posterior is estimated using the reative frequency of a acoustic words P (Y n Ω c ) = P (v Ω c ) b (Y n,v ). (9) v v 3. EVALUATION In order to derive a working system for the smart room at TU Dortmund University, severa recordings were made. Different features were evauated using the proposed cassification method. The proposed method and reated ones were compared in cassification performance. The event detection capabiity was tested with a scripted recording in the smart room and severa others from existing corpora. (8) 2 pubished in: Int. Conf. on Acoustics, Speech and Signa Processing (ICASSP) 2014

3 3.1. Event Cassification & Mode Parameters In order to investigate the performance of different methods, recordings of various typica sound events were made in a smart conference room at TU Dortmund University. The microphones were embedded in a tabe as shown in figure 2 and recorded at 48 khz. Each recording featured a certain sound type and asted over 60 s. To evauate the cassification performance on unknown data, a second test set of recordings was made on a different day with a different person producing the sounds. In the recordings time stretches with occurrences of the events were abeed. A methods were evauated using cross-vaidation on the training and test set. Using the Bag-of-Super-Features-Pyramid () approach, different feature types were evauated. Figure 3 shows the resuts. Aong with the MFCCs the s, inear prediction coefficients (LPC) and a non-negative matrix factorization (NMF) [20] of the me frequnecy magnitudes were evauated. The MFCCs and GFFCs have the owest error on the test set. Their combination achieves the highest score. Both LPC and NMF show a significanty higher error on the test set and seem to be unabe to generaize successfuy. Figure 4 shows the cassification errors for different methods using a combination of MFCC and features. The Bag-of-Frames mode (BoFr) using MFCCs ony that is described in [5] is appied to the sound cassification probem and used as a baseine. The Bag-of-features (BoF) mode performs worse than the baseine if the codebook is computed in an unsupervised manner. However, there is a significant improvement using the Bag-of-Super-Features (BoSF). This strengthens the view that the use of a supervised codebook estimation aows for a better modeing of the diverse acoustic event casses. Incorporating tempora information by the pyramid scheme further improves the resuts. For comparison, a Nearest Neighbor cassifier and an SVM were aso appied to the pyramid mode. They both perform significanty worse than the mutinomia maximum ikeihood cassifier. In order to determine the infuence of the codebook size training set test set MFCC MFCC & LPC NMF Fig. 3. Cassification error for different features of smart room recordings. A features were evauation with the BoFS-P method using a mutinomia maximum ikeihood cassifier training set test set BoFr [5] (ML) BoSF (ML) BoF (ML) (SVM) (KNN) Fig. 4. Cassification error for different methods for the smart room recordings. The Bag-of-Feature methods were evauated using the combination of MFCC and features. the approach has been evauated for different sizes of L. The resuts in Figure 5 show that aready a comparaby sma codebook size of L = 121 yieds good resuts, which equas 11 centroids per cass. Therefore, in the foowing experiments a codebook size of 11 centroids per cass was chosen. Compared to other Bag-of-Features cassification approaches where codebooks of severa thousand centroids are used this size is remarkaby sma (see [,11]). The advantage of this is two-fod: First, the quantization adds an additiona abstraction to the data such that it generaizes better. Large codebooks approximate the data better but are not abe to generaize we over the very diverse acoustic events. Second, sma feature representations are fast to compute and cassify which faciitates the use of the method in rea time acoustic event detection. The proposed method can be computed in just 5% of the rea time using python on a standard PC. 15 training test Fig. 2. Smart room with microphones embedded in tabe Fig. 5. Cassification error for the approach for the smart room recordings using MFCC and features with different codebook sizes L. L 3 pubished in: Int. Conf. on Acoustics, Speech and Signa Processing (ICASSP) 2014

4 ground truth fitered t[s] sience door steps chairs cups pouring paper roing keyboard aptop speech other Fig. 6. Resuts for event detection in the smart environment, acoustic events are shown in distinct coors Event Detection When denoting as g, e, and t the number of ground truth, estimated and correct events, precision P and reca R can be defined aong with the F-measure F as in [5] P = t e, R = t g, F = 2P R P + R. () For the event detection performance, the non-event cass Ω c is excuded in the counts. The metrics are evauated framebased and cass-based, for the atter a casses are evauated individuay and the average is computed Smart Room Recording In order to estabish the systems performance for event detection in ive scenarios, sequences with various events were recorded in the smart room. The cassifier was chosen by the evauations above and trained again with the event recordings. Tabe 1 ists the overa detection metrics. The nonevent cass had 83% precision and 68% reca. This can be attributed to the fact that the training data for other casses contained portions of sience. The speech cass was detected with 97% precision and 87% reca. In figure 6, the detection resuts for the sequence are visuaized in coor. Smoothing may be desirabe for practica appications. Basic post fitering can be done by seecting the most frequent detection in the ast 1 s and discarding cases where its occurrence covers ess than 0.3 s CLEAR Within the CHIL project, the CLEAR campaign investigated the detection of acoustic events. The proposed method was tested on the ITC data, which contains three different training sets and a test set for three separate days [3]. For the non-event cass, the non-abeed portions from the training data were used. In this manner, 88% precision and 84% reca were achieved. Tabe 1 shows the performance over a experiments in the deveopment set. The phone vibration cass had 0% reca, for a other casses an F vaue of over 75% was achieved. dataset method metric F P R Smart frames 71.9% 74.3% 69.6% Room casses 77.3% 82.7% 72.5% frames 75.8% 79.3% 72.6% CLEAR casses 75.5% 79.2% 72.2% frames 52.3% 51.7% 53.2% casses 59.5% 64.8% 57.7% D-CASE NMF [5] frames 20.6% 29.1% 16.0% baseine casses 13.5% 11.6% 21.7% Tabe 1. Resuts for the acoustic event detection on the three datasets: smart room recordings, CLEAR and DCASE D-CASE The proposed method was evauated on the recent IEEE AASP Chaenge Detection and Cassification of Acoustic Scenes and Events Event Detection deveopment set. Since the training data consists of event recordings ony, nonabeed portions of the scripts not used in the test were used for training in order to have training data for the non-event cass. The performance averaged over a experiments in the deveopment set are presented in tabe 1. The non-event detection had 88% precision and 90% reca. The baseine event detection proposed in [5] is outperformed and the resuts of our method are aso highy competitive with respect to the resuts of the chaenge CONCLUSION In this paper an event detection approach using supervised trained GMM codebooks of MFCCs an s for Bag-of- Features histograms with tempora coding was presented. Highy competitive resuts on different difficut datasets for acoustic event cassification and detection were achieved. The use of a singe sience cass for non-events coud be shown to be highy successfu. The good speech detection quaity is important for many appications. The method can be easiy impemented and computed fast enough for rea-time appication. 1 Resuts are to be pubished ( 4 pubished in: Int. Conf. on Acoustics, Speech and Signa Processing (ICASSP) 2014

5 5. REFERENCES [1] Iain McCowan, Danie Gatica-Perez, Samy Bengio, Guiaume Lathoud, Mark Barnard, and Dong Zhang, Automatic anaysis of mutimoda group actions in meetings., IEEE transactions on pattern anaysis and machine inteigence, vo. 27, no. 3, pp , Mar [2] Axe Pinge, Danie Hauschidt, Marius H Hennecke, and Gernot A Fink, Mutipe Speaker Tracking using a Microphone Array by Combining Auditory Processing and a Gaussian Mixture Cardinaized Probabiity Hypothesis Density Fiter, in IEEE Int. Conf. on Acoustics, Speech, and Signa Processing, Prague, Czech Repubic, 2011, pp [3] Andrey Temko, Robert Makin, Christian Zieger, Dušan Macho, Ciment Nadeu, and Maurizio Omoogo, CLEAR Evauation of Acoustic Event Detection and Cassification Systems, in Mutimoda Technoogies for Perception of Humans, Rainer Stiefehagen and John Garofoo, Eds., vo of Lecture Notes in Computer Science, pp Springer Berin Heideberg, [4] Annamaria Mesaros, Toni Heittoa, Antti Eronen, and Tuomas Virtanen, Acoustic Event Detection in Reaife Recordings, in European Signa Processing Conference, Aaborg, Denmark, 20, pp [5] Dimitrios Giannouis, Dan Stowe, Emmanoui Benetos, Mathias Rossigno, and Mathieu Lagrange, A Database and Chaenge for Acoustic Scene Cassification and Event Detection, in European Signa Processing Conference, Marrakech, Morocco, [6] Hao Tang, Stephen M Chu, Mark Hasegawa-Johnson, and Thomas S Huang, Partiay supervised speaker custering, Pattern Anaysis and Machine Inteigence, IEEE Transactions on, vo. 34, no. 5, pp , [7] Jean-Juien Aucouturier, Boris Defrevie, and Francois Pachet, The bag-of-frames approach to audio pattern recognition: A sufficient mode for urban soundscapes but not for poyphonic music, The Journa of the Acoustica Society of America, vo. 122, no. 2, pp , [8] Richard F. Lyon, Machine Hearing An Emerging Fied, IEEE Signa Processing Magazine, Sept. 20. [] Leonard Rothacker, Marca Rusino, and Gernot A. Fink, Bag-of-Features HMMs for Segmentation-Free Word Spotting in Handwritten Documents, in Proc. Int. Conf. on Document Anaysis and Recognition, Washington DC, USA, [11] Ken Chatfied, Victor Lempitsky, Andrea Vedadi, and Andrew Zisserman, The devi is in the detais: an evauation of recent feature encoding methods, in British Machine Vision Conference, [12] René Grzeszick, Leonard Rothacker, and Gernot A. Fink, Bag-of-features representations using spatia visua vocabuaries for object cassification, in IEEE Int. Conf. on Image Processing, Mebourne, Austraia, [13] Stephanie Pancoast and Murat Akbacak, Bag-of- Audio-Words Approach for Mutimedia Event Cassification, in Interspeech, Portand, OR, USA, [14] DeLiang Wang and Guy J. Brown, Eds., Computationa Auditory Scene Anaysis: Principes, Agorithms, and Appications, IEEE Press, [15] Yang Shao, Soundararajan Srinivasan, and DeLiang Wang, Incorporating auditory feature uncertainties in robust speaker identification, in IEEE Internationa Conference on Acoustics, Speech, and Signa Processing, 2007, pp [16] Masashi Unoki and Masato Akagi, A Method of Signa Extraction from Noisy Signa based on Auditory Scene Anaysis, Speech Communication, vo. 27, no. 3, pp , [17] Stephen O Hara and Bruce A Draper, Introduction to the bag of features paradigm for image cassification and retrieva, arxiv preprint arxiv: , [18] Svetana Lazebnik and Maxim Raginsky, Supervised earning of quantizer codebooks by information oss minimization, Pattern Anaysis and Machine Inteigence, IEEE Transactions on, vo. 31, no. 7, pp , [19] Svetana Lazebnik, Cordeia Schmid, and Jean Ponce, Beyond bags of features: Spatia pyramid matching for recognizing natura scene categories, in Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on. IEEE, 2006, vo. 2, pp [20] Danie D. Lee and H. Sebastian Seung, Learning the Parts of Objects by Non-negative Matrix Factorization., Nature, vo. 401, no. 6755, pp , Oct [9] Ricardo Baeza-Yates and Berthier Ribeiro-Neto, Modern Information Retrieva, ACM Press, pubished in: Int. Conf. on Acoustics, Speech and Signa Processing (ICASSP) 2014