ABNORMAL SOUND EVENT DETECTION USING TEMPORAL TRAJECTORIES MIXTURES

Transcription

1 ABNORMAL SOUND EVENT DETECTION USING TEMPORAL TRAJECTORIES MIXTURES Debalya Chakrabarty Mounya Elhilali Departent of Electrical and Coputer Engineering Johns Hopkins University, Baltiore, MD, USA. ABSTRACT Detection of anoalous sound events in audio surveillance is a challenging task when applied to realistic settings. Part of the difficulty stes fro properly defining the noral behavior of a crowd or an environent (e.g. airport, train station, sport field). By successfully capturing the heterogeneous nature of sound events in an acoustic environent, we can use it as a reference against which anoalous behavior can be detected in continuous audio recordings. The current study proposes a ethodology for representing sound classes using a hierarchical network of convolutional features and ixture of teporal trajectories (MTT). The fraework couples unsupervised and supervised learning and provides a robust schee for detection of abnoral sound events in a subway station. The results reveal the strength of the proposed representation in capturing non-trivial coonalities within a single sound class and variabilities across different sound classes as well as high degree of robustness in noise. Index Ters Anoalous sound events, Hierarchical network, Convolutional feature representation, Mixture of teporal trajectory odels 1. INTRODUCTION Defining abnoral behavior in an audio recording is a challenging task. First of all, there is no universal definition of what abnorality eans. Second, even what is noral cannot be easily defined given the coplex nature of sound sources in realistic scenarios. To date, ost research efforts in anoaly detection have ainly focused on detection of isolated events in continuous recordings such as shouts [1], screas [2], laughs [3], gunshots and explosions [5] etc. However, for setting up a surveillance syste in an environent like a train or subway station, detecting abnoralities based on exaining isolated events becoes highly inefficient since collections of such isolated events can overlay noral behavior. Instead, we consider the proble of obtaining a good odel representation of noral behavior in the environent. We are particularly interested in odels that can capture non-trivial coonalities across various sound events as well as their interactions in the context of a coplex scene. Modeling acoustic scene behavior ultiately reduces to a choice of feature representation and learning odel that can best characterize the yriad events that can be encountered in acoustic scenes. Mel-Frequency Cepstral Coefficients (MFCC) are the ost widely used representation in acoustic event detection tasks. They provide a copact and efficient apping of the spectral characteristics of siple scenes [6, 7, 8]. Unfortunately, their perforance does not generalize to real world environents which are inherently dynaic and often corrupted by noise. In order to accurately report the intricacies of such realistic scenarios, it is iperative that any odeling of acoustic characteristics captures both spectral and teporal nuances of the signal over ultiple resolutions and tie-constants [9, 10]. Work in this direction has often eployed two-diensional tie-frequency filter-banks using Gabor filters, localized Fourier bases or even bioietic spectro-teporal receptive fields [11]. In [12], Lee et al. reported a localized and rich tiling of the spectro-teporal space of sound classes derived fro unsupervised learning of unlabeled data in the context of Restricted Boltzann Machines (RBM) [13]. In the current work, we build on this rich basis set; and extend applicability of unsupervised learning using RBMs to the proble of anoaly detection in audio recordings. Operating on this feature analysis often coes a robust backend classifier whose role is to capture variability across different instances of the sound class. Unsupervised classifiers like Support vector achines (SVM) and Gaussian ixture odels (GMM) have proved to be very efficient in odeling the ean statistics of analytical audio features in tasks of screa, laughter and gunshot detection [14, 15]. These odels do provide well defined average representation of isolated events but fail to capture the inforation contained in the teporal dynaics of these events. In contrast, HMM based odels are capable of capturing such teporal trajectories [15]. However, because of their arkovian constraint, they becoe inefficient in odeling the long ter teporal dependencies across events essential to obtain a global context of an acoustic scene. Recent work started using ore representationally powerful generative odels based on dis-

2 Fig. 1. Block diagra of MTT based abnoral sound event detection tributed hidden states, such as Conditional RBMs [16] to learn representation of teporal dynaics fro data rather than explicitly odeling the under hard wired assuptions. In the current study, we develop a hybrid RBM-CRBM schee for odeling noral acoustic behavior in a subway station. An event such as noral conversation aong riders is typically coprised of ultiple sub-events like speech, laugh, cheerful banter etc., each having its own set of spectral and teporal dynaics. In order to capture these different odes of teporal dynaics as well as their interactions and transitions across each other, we propose a ixture of dynaic trajectories that can decopose the global teporal space of a noral event into ultiple trajectories, each of which belongs to a seantically different sub-event. We develop an integrated fraework of learning the localized spectro-teporal attributes in an unsupervised fashion as well as capturing their different odes of teporal trajectories by using a set of ixtures of teporal trajectories (MTTs). The fraework flags as abnoral events that don t fall within the span of learned trajectories. The organization of this paper is as follows: Section 2 provides a detailed description of the proposed ethodology using a hybrid RBM-MTT fraework. Section 3 outlies the experiental setup and event detection results, while section 4 provides conclusion and discussion of the results. 2. METHOD Our proposed fraework for abnoral sound event detection coprises 3 ain processing blocks; acoustic odeling using RBM, dynaic odeling using MTT and finally using these odels for abnoral sound event detection as shown in Figure 1. The proposed syste operates on tie frequency representation of acoustic signal. A tie-frequency auditory spectrogra y(t, f) is extracted fro each audio file based on a odel of peripheral processing in the aalian auditory syste [17]. The spectrogra representation y(t, f) is sapled with frae size of 10 s. 10 consecutive fraes are then grouped together to for a one diensional vector x in a process of shingling [18]. A dataset of n sapled patches given by X = x 1,x 2,...,x n is fored, where x (i) R N and N = 1280 in our case Acoustic odeling using RBM We use Sparse restricted Boltzann achine (RBM) as the unsupervised learning algorith to discover features fro the unlabeled dataset X. Sparse RBMs are undirected graphical odels with K binary hidden variables [19]. We train the first layer RBM representations coprised of 400 hidden units using the contrastive divergence (CD) approxiation with sae type of hyper-paraeters and sparsity penalty as used in [20]. The training produces the weights W k for k =1, 2,...,400 which are a representation of localized spectro-teporal attributes. In order to get a representation siilar to localized 2D filters, we transfor these one diensional weights W k into h k (t, f) where t =10andf = 128. We apply these 2D filters over the tie-frequency patch y(t, f) extracted fro the labeled dataset of noral conversations to obtain filter responses r k (t) given by: r k (t) = y l (τ,f)h(t τ,f)dτ (1) f Filter responses r k (t) are used as our feature representation for the next processing block Dynaic odeling using MTT Next, a ixture of CRBMs (CRBM) [21] is proposed as a dynaical ixture odel to decopose the global teporal space of a noral event into ultiple trajectories, where each such trajectory belongs to a particular sub-event. A dynaical ixture odel can be created by introducing a ixture coponent variable, q, with M possible states [21]. The dynaical odel is defined by a joint distribution: p(γ t, z t, q t γ N )=exp( E(γ t, z t, q t γ N ))/Z(γ N ) (2) where γ t is real valued representation of current filter response, z t is a collection of binary hidden units such that z (0, 1), and γ N contains the history of past N filter responses to provide a way for capturing the long ter teporal

3 dependencies across the responses. The energy function E is given by: E(γ t, z t, q t γ N )= 1 (γ it ĉ it ) 2 z jt ˆdjt 2 i j (3) q t W ij γ it z jt where W captures the interactions between the filter responses and hidden variables and the dynaical ters ĉ it and ˆd jt are linear functions of previous N filter responses γ N, given by: ĉ it = ( q t C i + ) A il γ ln l ˆd jt = ( q t D j + ) (4) B il γ ln l i,j library [23] to train our first layer RBM bases. BBC sound effects library contain classes like Abience, Anials, Office, Transportation and Musical etc. Because of such heterogeneity across the scenes, RBM weights are not biased towards one particular kind of scene. The dataset used for abnoral sound events detection contains recordings of audio events in a etro station [24]; the duration of each file ranging fro 1 inute to about 6 inutes. We resaple each recording in the dataset to 8 KHz and preprocess the through a preephasis filter with coefficients [1 0.97] in order to boost the high frequencies. The recordings contain events like noral speech, laughter, cheerful banter etc. annotated as noral conversation. The fraes belonging to noral events are split randoly into 80 % for training the MTT odels and rest 20 % as developent and test set. The recordings also contain events like train passing by, shout, screa, fights, aggressive behavior etc. which we consider as abnoral in our analysis and include the in the test set for detection. where C and D are static biases and A and B are autoregressive odel paraeters. The paraeter set θ = ( W, A, B, C, D) of CRBM are learned using contrastive divergence (CD) approxiation. We refer the reader to [21] for details of learning CRBM by CD. This learned paraeter set θ becoes our representation of ixture of teporal trajectories (MTT) odels. We use M=10 assuing a ixture of 10 coponents can span the entire teporal trajectory space of a single event and use 200 hidden units in our CRBM architecture Abnoral Sound Event Detection In the detection stage, we use the easure of log-likelihood score of a given test frae under our learned MTT odel to decide whether the frae under consideration belongs to an abnoral event or noral conversation [21]. A test audio signal is processed through the learned RBM weights to obtain feature representation r k (t) as per equation 1. On applying the paraeter set θ over r k (t), we obtain a log likelihood score L given by: L = log(p(r t r N ; θ)) = log( z t,q t p(r t, z t, q t r N )) (5) We copare this likelihood score L with a threshold value obtained fro developent set and we label the frae as noral if L>threshold or abnoral if L<threshold Data 3. EXPERIMENTAL SETUP AND RESULTS We prepare an unlabeled training dataset by randoly ixing the recordings fro both TIMIT [22] and BBC sound effects 3.2. Syste variants The perforance of an abnoral sound event detection syste depends on how good our odel representation is. The key aspect of our odel representation is based on a set of ixtures of teporal trajectories capturing the interactions and transitions across ultiple events in a coplex acoustic scene. In order to quantify its iportance and effect on syste perforance, we contrast our proposed syste against 3 syste variants based on siilar generative fraework and backbone architecture but with variabilities in ixture coponents and trajectory representations. In one case, we train our MTT odel using M =1to see how the perforance of the detection syste changes when a single ixture coponent is used to odel different odes of teporal trajectories existing within a single event. Secondly, in order to quantify the iportance of teporal trajectories based representation, we build a detection syste by replacing CRBM block with a regular RBM that odels only the localized spectroteporal odulations without any inforation of long ter teporal dependencies. Our final syste is based on learning first layer RBM bases only fro noral conversation and use these learned bases for detecting the abnoral events Results and Analysis Figure 2 shows the ROC for each of the detection systes by including/excluding the MTT stage as well as varying the nuber of ixtures capturing the teporal trajectories. The figure shows that our proposed syste using MTT odel with M = 10 perfors the best in ters of true positive rate. When MTT is replaced by a RBM layer in the fraework, we see that the detection perforance of the syste degrades because of incapability of RBM based representation in capturing the long ter teporal dependencies. Single layer RBMs

4 True positive rate MTT with M = 10 MTT with M = 1 2 layer RBM Single layer RBM False positive rate Fig. 2. ROC curves for 4 systes regarding to detection of abnoral sound events trained only on noral conversations gives the worst perforance in ters of true positives; the ain reason being the first layer RBM bases trained on a sall set of data are not able to capture a good representation of localized spectroteporal attributes. As a result of this poor characterization, we get a lot ore false negatives for this syste copared to other systes. For MTT odel with M =1, the observation gets interesting. We see that its detection perforance is better than RBM based systes, thus illustrating the iportance of long ter teporal dependencies over short ter teporal structure for better characterization of sound events. However, the true positive rate for this syste decreases when copared to MTT odel with M =10. This observation is ainly accounted for by the fact that due to presence of different odes of teporal trajectories within an event of noral conversation, MTT with M =1fails to span the entire teporal trajectory space of such a broader class. As a result, when an event like laughter occurs in a continuous audio, the syste detects it as an abnoral event even though it is labeled as noral conversation. To provide ore insight into the idea of MTTs capturing different odes of teporal trajectories, we apply our MTT odel to a saple recording of noral conversation aong riders in a subway station. At several points during the conversation, other than noral speech, there are instances of laugh, exciteent etc. which are non stationary events having their own set of dynaics. In our experiental analysis, we find that fraes belonging to the instances of laugh and exciteent are assigned to coponents 1, 4 and 7 with an average probability of ; while coponent 9 captures the teporal trajectories of noral speech in the conversation with an average probability of This probabilistic assignent of fraes to different coponents of MTT confirs our intuition that MTT with desired nuber of coponents is able to segent an event with different odes of teporal trajectories into statistically salient sub-events. We further test the robustness of the proposed syste by adding noise fro NOISEX-92 database [25] to the test set SNR (db) F-easure (%) for 4 detection systes MTT 2 layer (M=1) RBM MTT (M=10) Single layer RBM Clean db db dB db db Table 1. Abnoral sound events detection results for 4 systes at different SNR levels at different SNR levels of 20, 10, 0, -5 and -10 db. The perforance of the systes are easured in ters of percentage F-easure. We see fro Table 1 that MTT (M=10) based detection syste not only outperfors the other three syste variants in clean scenario but exhibits robustness in presence of noise as well. When noise level increases, the detection perforance of our MTT based syste degrades at a uch lower rate copared to the other three syste variants. We also observe that for upto 10 db SNR, our proposed syste gives a very satisfactory perforance in detecting the abnoral sound events. Another interesting point to note fro Table 1 is that even MTT odels with M =1perfors better than RBM based odels for all noise cases. This clearly shows the iportance of incorporating the inforation of teporal trajectories along with localized spectro-teporal attributes in the odel representation of sound events for a robust characterization.. 4. CONCLUSION In this work, we develop a hybrid RBM-MTT fraework for abnoral sound event detection in subway station by using a joint representation of localized spectro-teporal attributes with ixtures of teporal trajectories. Such a joint representation is very effective in capturing the intricate details and coonalities across a broader sound class spanned by ultiple events. We show that MTT as a dynaical ixture odel spans the coplete teporal trajectory space of a coplex acoustic scene by decoposing it into ultiple trajectories, each of which belongs to a particular sub-event. In abnoral sound event detection task, the detection accuracy iproves by an absolute 7 % over RBM class of odels when inforation of different odes of teporal dynaics is incorporated in odel representation of sound objects via our proposed MTT. We also find that our MTT based representation augents the detection syste with high degree of noise robustness at low SNR levels, thus illustrating the fact that the joint representation provides a uch robust characterization of broader sound classes.

5 5. REFERENCES [1] V.K. Mittal and B. Yegnanarayana, Production features for detection of shouted speech, in Consuer Counications and Networking Conference (CCNC), 2013 IEEE, Jan 2013, pp [2] M.K. Nandwana, A. Ziaei, and J.H.L. Hansen, Robust unsupervised detection of huan screas in noisy acoustic environents, in Proceedings of ICASSP 15, April 2015, pp [3] M. Knox and Nikki Mirghafori, Autoatic laughter detection using neural networks, in Interspeech, Antwerp, Belgiu, August [4] S. Ntalapiras, I. Potaitis, and N. Fakotakis, On acoustic surveillance of hazardous situations, in Proceedings of ICASSP 09, April 2009, pp [5] X.Zhuang, X.Zhou, T.Huang, and M.Hasegawa- Johnson, Feature analysis and selection for acoustic event detection, in Proceedings of ICASSP 08, [6] O. Kalini, S. Sundara, and S. Narayanan, Saliency driven unstructured acoustic scene classification using latent perceptual indexing, in Proceedings of MMSP 09, Rio de Janiero, Brazil, October 5-7, [7] W. Nogueira, G. Roa, and P. Herrera, Sound scene identification based on MFCC, binaural features and a support vector achine (SVM) classifier, in IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), [8] S. Neala and M. Elhilali, Relevant spectro-teporal odulations for robust speech and nonspeech classification, [9] C.V. Cotton and D.P.W. Ellis, Spectral vs. spectroteporal features for acoustic event detection, in Applications of Signal Processing to Audio and Acoustics (WASPAA), 2011 IEEE Workshop on, Oct 2011, pp [10] M. Kleinschidt, Localized spectro-teporal features for autoatic speech recognition, in Proceedings of Eurospeech, 2003, pp [11] Honglak Lee, Peter Pha, Yan Largan, and Andrew Y. Ng, Unsupervised feature learning for audio classification using convolutional deep belief networks, in Advances in Neural Inforation Processing Systes 22, pp Curran Associates, Inc., [12] Geoffrey E. Hinton, Learning ultiple layers of representation, Trends in Cognitive Sciences, vol. 11, no. 10, pp , 2015/09/18. [13] Weiin Huang, Tuan Kiang Chiew, Haizhou Li, Tian Shiang Kok, and J. Biswas, Screa detection for hoe applications, in Proceedings of ICIEA 10, June 2010, pp [14] G. Valenzise, L. Gerosa, M. Tagliasacchi, F. Antonacci, and A. Sarti, Screa and gunshot detection and localization for audio-surveillance systes, in Proceedings of AVSS 07, Washington, DC, USA, 2007, pp [15] Graha W Taylor, Geoffrey E Hinton, and Sa T Roweis, Modeling huan otion using binary latent variables, in Advances in neural inforation processing systes, 2006, pp [16] T. Chi, P. Ru, and S. A. Shaa, Multiresolution spectroteporal analysis of coplex sounds, The Journal of the Acoustical Society of Aerica, vol. 118, no. 2, pp , [17] J. Salaon and J.P. Bello, Unsupervised feature learning for urban sound classification, in Proceedings of ICASSP 15, April 2015, pp [18] Marc aurelio Ranzato, Christopher S. Poultney, Suit Chopra, and Yann Lecun, Efficient Learning of Sparse Representations with an Energy-Based Model, in Neural Inforation Processing Systes, 2006, pp [19] K Cho, T Raiko, and A Ilin, Enhanced gradient for training restricted boltzann achines, Neural Coputation, vol. 25, no. 3, pp , March [20] G.W. Taylor, L. Sigal, D.J. Fleet, and G.E. Hinton, Dynaical binary latent variable odels for 3d huan pose tracking, in Coputer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on, June 2010, pp [21] Willia M. Fisher, George R. Doddington, and Kathleen M. Goudie-Marshall, The DARPA Speech Recognition Research Database: Specifications and Status, in Proceedings of DARPA Workshop on Speech Recognition, 1986, pp [22] The bbc sound effects library original series, May [23] W. Zajdel, J.D. Krijnders, T. Andringa, and D.M. Gavrila, Cassandra: audio-video sensor fusion for aggression detection, in Proceedings of AVSS 07, Sept 2007, pp [24] A. P. Varga, H. J. M. Steeneken, M. Tolinson, and D. Jones, The noisex-92 study on the effect of additive noise on autoatic speech recognition, Tech.Rep., Speech Research Unit, Defense Research Agency, Malvern, U.K., 1992.