Audio/Video Supervised Independent Vector Analysis Through Multimodal Pilot Dependent Components

Transcription

1 th European Signal Processing Conference (EUSIPCO) Auio/Vieo Supervise Inepenent Vector Analysis Through Multimoal Pilot Depenent Components Francesco Nesta, Saee Mosayyebpour, Zbyněk Kolovský, Karel Paleček Conexant Systems, 9 Main Street, Irvine, CA (USA). {francesco.nesta, saee.mosayyebpour}@conexant.com Faculty of Mechatronics, Informatics, an Interisciplinary Stuies, Technical University of Liberec, Stuentská, Liberec, Czech Republic. {zbynek.kolovsky, karel.palecek}@tul.cz Abstract Inepenent Vector Analysis is a powerful tool for estimating the broaban acoustic transfer function between multiple sources an the microphones in the frequency omain. In this work, we consier an extene IVA moel which aopts the concept of pilot epenent signals. Without imposing any constraint on the e-mixing system, pilot signals epening on the target source are injecte into the moel enforcing the permutation of outputs to be consistent over time. A neural network traine on acoustic ata an a lip motion etection are jointly use to prouce a multimoal pilot signal epenent on the target source. It is shown through experimental results that this structure allows the enhancement of a preefine target source in very ifficult an ambiguous scenarios. Inex Terms inepenent vector analysis; source separation; inepenent component analysis; speech enhancement; multimoal processing; I. INTRODUCTION Inepenent Vector Analysis (IVA) is a popular tool for unsupervise multichannel source separation []. Its virtues are relate to the ability to avoi the permutation problem of traitional narrow-ban frequency-omain methos for source separation [], []. Differently from Inepenent Component Analysis (ICA), IVA uses a multivariate source moel in orer to jointly estimate the separate components in each frequency. The multivariate moel allows to bypass the nee for aitional permutation solver algorithms, which often rely on prior assumptions on the geometrical interpretation of the mixing system [], []. On-line implementations [] an several other extensions have been propose in []. Nevertheless, espite its potential, IVA is still not wiely use in commercial applications such as VoIP an ASR preprocessing. Inee, the effectiveness of IVA is intrinsically limite by the core paraigm of unsupervise source separation: The nature of the sources of interest is not explicitly efine. Therefore, although the internal permutation problem is solve by the multivariate moel, the external orer of the recovere sources cannot be guarantee. The same output can contain portions of ifferent source signals at ifferent time instants, especially when the mixing conitions are not static. To overcome this issue, geometrical constraints have been employe by imposing that a given output signal is associate The work of Zbyněk Kolovský was supporte by The Czech Science Founation through Project No. -9S to a source having known angular position []. However, these constraints cannot work well if the source an the noise are locate at similar angles or when the target source position cannot be uniquely efine. Furthermore, these constraints make IVA similar to aaptive beamforming [9] or to geometrically constraine ICA [], thus limiting the potential of multivariate moeling []. To mitigate the mentione IVA ambiguities but without imposing any geometrical constraint, in [] we propose to moify the multivariate moel by injecting pilot signals that are mutually epenent with the sources of interest. The pilot signals were efine to be proportional to the posterior probabilities to observe each source, given an observe wieban spatial or spectral feature. Inspire by [], in this work we further exten the moel by consiering posteriors erive from multimoal signals: A neural network is traine using extensive prior acoustic ata such that it prouces a pilot signal that contains posteriors of the target source ominance in the observe mixture. Another pilot signal is erive base on the lip etection in a vieo recore together with auio. This allows to isambiguate the separation in cases where the target as well as the interference are speech signals. Experimental evaluations are carrie out to confirm the effectiveness of the supervise structure in separating speech from noise sources, in ifficult ambiguous scenarios, e.g. when the target an the noise are both speech sources an are locate at a similar angular irection. II. SUPERVISED IVA N source signals are assume to be recore by an array of M microphones. Let S k n an X k m be the STFT coefficients obtaine for the kth frequency bin, the nth source an the mth mixture signal, respectively. Let S k = [S k S k N ]T an X k = [X k X k M ]T. The mixing moel is X k = H k S k +N k, () where N k = [N k,,n k M ]T is the vector of backgroun noise an interference signals, an H k inicates the mixing matrix for the kth frequency bin. Assuming N = M, the objective of IVA is to estimate a set of e-mixing matrices ISBN EURASIP 9

2 th European Signal Processing Conference (EUSIPCO) W k = {W k nm}, k =,...,K, where K is the number of the frequency bins. The e-mixing matrices jointly recover inepenent multiimensional sources Y n = [Y n,,y K n ], n =,...,N, where Y k n = M WnmX k m, k () m= up to a scaling ambiguity, which can be subsequently resolve by applying the Minimal Distortion Principle [] to each matrix W k. A typical way to moel the sources is with a multivariate spherical super-gaussian istribution efine as [] a n = [a n,...,a K n ] T, f(a n ) = α exp K a k n. () k= In the supervise IVA (S-IVA) [], the multivariate moel () is extene by injecting aitional pilot epenent components. In this work we consier Q pilots for each source, P n,...,p Q n, which will be relate to ifferent moalities: ã n = [a n,...,a K n, Pn,..., q Pn Q ], f(ã n ) = αexp K Q a k n + γq P n q, () k= q= where γ q is an hyper-parameter controlling the influence of each pilot. To obtain the upate rule, the Maximum Likelihoo (ML) [] approach is use by consiering the cost function K N L = log etw k + E[logf(Ỹn)], () k= n= where Ỹn = [Yn,...,Y n K, Pn,..., Q Pn Q ] enotes the extene observation output vector. The expectation E[ ] is approximate with the time average over the frames. Then, by taking the erivatives of () with respect to Wnm k an applying the natural graient moification to maximize (), we obtain the upate rule W k nm = (I nm E[φ k (Y n )(Y k m) ])W k nm, () W k new = W k ol +η W k, where η is the aaptation rate, I nm inicates the nmth element of the ientity matrix, an the nonlinearities φ k ( ), k =,...,K are the score functions relate to the ensity (), namely, φ k (Ỹn) = Y k n K j= Y j n + Q q= γ q P q n. () As the pilot components o not epen on W k nm, the secon sum in () remains constant uring the optimization. This way, any IVA algorithm can be moifie to its supervise version. This simplifie multivariate moel is obtaine by assuming that all the source components are zero mean an uncorrelate []. III. DEFINITION OF PILOT SIGNALS The propose metho can be relate to a previous work in [] where a user-guie source activity was use to supervise the IVA aaptation. However, the formulation of S-IVA is far more general as it can naturally inclue many supervising moalities, through the efinition of multiple pilot signals. In this work, the pilot signals are erive from auio an vieo information. As we use the spherical Laplacian moel in (), the pilots are assume complex-value zero-mean signals, uncorrelate to the frequency bins but with a epenent time-varying variance. Therefore, only the variance of the pilots has to be efine. By inicating with a l n an b l n the posteriors of source activity erive from the auio an vieo moalities (with a l n, b l n [,]), the pilot signal variance is efine as [ ] K c(l) = E X k N n(l), () n k= P n(l) = (a l n) c(l), P n(l) = (b l n) c(l), where l is the time frame inex, E[ ] inicates the expectation, which is approximate as a smooth time-average an the term c(l) rescales the pilot to a ynamic range proportional to the sum of the frequency components. A. Derivation of a l n through an Acoustic Neural Network A neural network (NN), traine to solve a regression problem, is use to preict the source activity posteriors a l n. Namely, the network is traine to estimate the power ratio between the true target speech an the noisy mixture. Any machine learning metho for regression can be use, such as recurrent neural networks (see, e.g., []) but we foun that a naive multilayer fee forwar NN, often name eep NN (DNN), is sufficiently accurate to prouce a useful preiction. Let Sn kl be the klth time-frequency representation of the th signal corresponing to the nth source, inclue in the training set D n. Example mixtures for the training are obtaine as X kl = n S kl n. (9) The DCT an the logarithm is applie to X kl to efine the transforme features X kl = DCT[ln( Xkl )], where k is the inex of the DCT coefficient. For the frame l, the input layer is efine as v l = [ X l, X l l+, X ], () where X l l Kl = [ X,, X ], K < K (i.e. only the first K DCT coefficients are use). Two hien layers of neurons are use with the hyperbolic tangent as the activation function. The softmax function is use in the output layer, which has imension N. Each output represents a ominancerelate feature for the nth source. For the th mixture at the Here we assume that a proper care is aopte to scale the emixing matrices in orer to keep their norm within a limite range. ISBN EURASIP 9

3 th European Signal Processing Conference (EUSIPCO) - -. Clean Speech Noisy speech DNN posteriors samples ( ) Fig.. Block iagram for the multimoal supervise S-IVA Fig.. Examples of DNN posteriors preiction lth frame, the training output labels are efine as g l = [g, l,gn], l gn l = k Skl n ki Skl i. () In this work we focus on the scenario where the source of interest is speech while any other non-speech acoustic event is consiere as noise (i.e. the noise can be consiere as compose by multiple sources). For the training of the DNN, a large set of k mixtures was generate by ranomly combining noise examples with speech sentences in the TIMIT atabase. Noises were collecte from ifferent sources an the ataset was esigne to balance the amount of noises belonging to ifferent categories. Noise signals selecte i not contain any speech, as the scope of the network is only to iscriminate between speech an noise. Two atasets of k mixtures were generate for both cross-valiation an testing. After training, the output preiction for the nth source at the lth frame, inicate as a l n, is obtaine through the feeforwar propagation of the input vectors v l compute on the test recorings. Figure shows an example of the DNN output for a given test recoring use in the experimental evaluation. We want to highlight that, although in this work our target is a speech source, S-IVA can also be applie to separate other type of acoustic sources, e.g. musical sources, as long as the DNN can iscriminate them from their time-frequency representation. B. Derivation of b l n from lip-motion etection In scenarios where the target source an the noise are acoustically similar, aitional moalities can be use to isambiguate the efinition of the target source. Here, we use the vieo signal synce with the auio recoring to extract the lip motion of a main target speaker. In orer to track the movement of the speakers lips, a set of facial lanmarks is extracte from each frame of the vieo using the Ensemble of Regression Trees algorithm []. A subset of lanmarks escribing the inner lip region then efines a polygon whose area r i approximately correspons to the mouth opening in the ith frame. Then, the mean m i an variance v i of consecutive values r i,...,r i+ are compute an normalize to the range,. The posteriors b l n are prouce by a logistic regression classification of the (m i,v i ) feature pair for each vieo frame. To this en, all frames of the vieo sequence were manually labele as either speech or non-speech an then split into train an valiation sets in a : ratio. Since the auio an vieo stream were capture at ifferent rates, resampling was applie in orer to prouce a signal consistent with the time-frequency representation use by IVA. IV. EXPERIMENTAL EVALUATION We conuct experiments with M =. An on-line S-IVA implementation is realize through upating the e-mixing matrices at each frame l accoring to Y k (l) = W k (l)x k (l), φ k Y k [Ỹn(l)] = n(l) K j= Y n(l) j +, () Q q= γ q Pn(l) q W k nm(l) = (I nm φ k [Ỹn(l)]Y k m(l) )W k nm(l), W k (l+) = W k (l)+η W k (l). () The scaling normalization is applie to each bin to stabilize the convergence as in []. The signal mixtures are transforme in their corresponing time-frequency representation through Short-time Fourier Transform with the Hanning winow of 9 points with % overlap. After separations, the images of the target source at each microphone are recovere through MDP, an signals are transforme back to the time-omain using overlap-a. Two ifferent experimental evaluations were carrie out: Test: separation with pilots base on acoustic features only; Test: separation base on auio/vieo combine acoustic features. The block iagram of the supervise IVA is epicte in Fig.. A. Test: Separation of speech from noise In this experiment, the vieo information was not available. Thus, P n(l) in () was set to, for each n an l. Recorings were mae with two microphones with mutual istance of. m. Signals were recore at f s = khz in a room of size. m with T = ms. Partially iffuse noise was simulate accoring to the Quest stanar ISBN EURASIP 9

4 th European Signal Processing Conference (EUSIPCO) by playing multichannel signals through louspeakers consisting of ifferent types of real-worl noise such as cafeteria, roa noise, train station, etc. The target speaker was recore at the istance of m from the center of the microphones at ifferent angles. Note, this can be consiere an uneretermine scenario, as the noise was generate by playing partially uncorrelate signals through multiple louspeakers. In orer to valiate the robustness of the propose approach, a ataset of mixtures was generate by combining speech signals (speakers at ranom angles) with ranomly selecte noise examples (not inclue in the training set of the DNN moel). Performance were evaluate by computing both the Noise-to-Speech ratio improvement (NSRi) at the noise output, an the Signal-to-Distortion ratio improvement (SDRi) at the speech output. Inee, it shoul be note that the scenario is highly uneretermine an a complete goo speech extraction system shoul make use of both speech an noise estimates [][9]. Fig. shows the performance average over the test recorings, comparing stanar IVA with S-IVA with tune for the best SDRi ( = ). It is seen that S- IVA consistently improves the average performance compare to stanar IVA, i.e. when =, as the source orer for IVA is not guarantee to be consistent over all test samples. In a secon experiment we evaluate the robustness of S-IVA to an inaccuracy in the VAD. To simulate errors in the DNN preiction, artificial noise was ae as ã l n = ( β)a l n +β ran() () Fig. shows the performance with varying β, emonstrating the robustness of S-IVA to noisy pilot signals. B. Test: Separation of target speech from noise speech In this experiment, we consier S-IVA enowe by a multimoal auio/vieo pilot signal. A target speaker was recore live in a front of a commercial laptop while simulating a NSRi SDRi Fig.. Performance of IVA (i.e. = ) an S-IVA (tune with = ) Fig β NSRi SDRi Robustness of S-IVA versus noise in the pilot signal. Fig.. speech. Fig.. speech. γ = = = SNRi performance for the multimoal S-IVA when the noise is γ = = = SDRi performance for the multimoal S-IVA when the noise is VoIP conversation. Noise was generate by recoring a TV, locate in the back of the laptop, at a istance of about meters. Although the speaker position is known in avance, it is worth noting that applying spatial constraints as in [] woul not be effective in these conitions. In fact, the angular positions of the target an of the noise sources are very close to each other. For a more etaile analysis of this aspect, see the experimental evaluation in []. A multimoal auio/vieo pilot signal is generate as in () an the performance were evaluate by varying both the parameter an γ. In a first experiment, we consier a recoring where the TV noise contains only spoken news. This scenario is very ifficult for the acoustic DNN as it cannot iscriminate between the target an noise speech. Figures an show the SNRi an SDRi performance average over the target an noise source. It is straightforwar that the pilot base on the acoustic DNN preiction is not reliable as by increasing the performance egraes. On the other han, the vieo information unoubtely provies a robust supervision as both SDRi an SNRi increase with γ. In a secon experiment, we consier a recoring where the TV noise containe a mix of speech an music. From Figures an it can be seen that the acoustic DNN preiction is more effective in these noise conitions as the presence of nonspeech relate events, helps S-IVA to converge to the correct source orer. Interestingly, this experiment shows that the best performance is obtaine when combining both auio an vieo information together. Inee, while the lip-etection accuracy ISBN EURASIP 9

5 th European Signal Processing Conference (EUSIPCO) 9 γ = = = Fig.. SNRi performance for the multimoal S-IVA when the noise is a mix of speech an music. γ = = = Fig.. SDRi performance for the multimoal S-IVA when the noise is a mix of speech an music. shoul not be sensitive to the presence of acoustic noise, other isturbances coul make it less reliable. For example, false etections are prouce by movements of the lips that happens also when no speech is prouce. This might also suggest that more effective multimoal formulations can be efine in alternative to (), in orer to better reflect the statistical correlation of the errors prouce by each moality. V. CONCLUSIONS In this work we have presente a supervise extension of Inepenent Vector Analysis. A pilot signal is injecte in the multivariate moel to steer the estimation towar the extraction of a specific wante source. A multimoal pilot signal was efine combining both auio an vieo information. A eep neural network was use to prouce time-varying posteriors of source ominance in orer to iscriminate speech from acoustic noise events. A lip motion etection was use to istinguish between the activity of the esire speaker from that of interfering speech. It is shown that, without explicit constraints to the emixing system, it is possible to have a consistent enhancement of a specific target source in ifficult scenarios, such as in far-fiel, in uneretermine conitions an when sources propagate from a similar irection. It was shown that when S-IVA is supervise by the DNNbase pilot signal, goo performance can be obtaine if the noise oes not contain any speech. On the other han, when the noise is a speech source, the performance obtaine with a vieo-base pilot signal clearly outperforms the acoustic supervision. Nevertheless, it was also observe that in mixe noise conitions the best performance was obtaine by combining auio an vieo moalities together. This result suggests that further work is require to esign multimoal formulations more effective than a naive weighte combination of each single moality. Furthermore, future work might also explore the use of EEG-base pilot signals, to realize effective biofeeback source enhancement methos []. REFERENCES [] I. Lee, T. Kim, an T.-W. Lee, Inepenent vector analysis for convolutive blin speech separation, in Blin Speech Separation. Springer, Sep.. [] H. Sawaa, R. Mukai, S. Araki, an S. Makino, A robust an precise metho for solving the permutation problem of frequency-omain blin source separation, IEEE Transactions on Speech an Auio Processing, vol., pp., Sep.. [] Y. Mori, H. Saruwatari, T. Takatani, S. Ukai, K. Shikano, T. Hiekata, an T. Morita, Real-time implementation of two-stage blin source separation combining simo-ica an binary masking, in IWAENC,, pp. 9. [] H. Saruwatari, S. Kurita, K. Takea, F. Itakura, T. Nishikawa, an K. Shikano, Blin source separation combining inepenent component analysis an beamforming, EURASIP J. Appl. Signal Process., vol., no., pp.,. [] H. Sawaa, R. Mukai, an S. Makino, Direction of arrival estimation for multiple source signals using inepenent component analysis, in Proceeings of ISSPA, vol., Jul., pp.. [] T. Kim, Real-time inepenent vector analysis for convolutive blin source separation, IEEE Transactions on Circuits an Systems I: Regular Papers, vol., no., pp.,. [] J. Harris, B. Rivet, S. M. Naqvi, J. A. Chambers, an C. Jutten, Real-time inepenent vector analysis with stuent s t source prior for convolutive speech mixtures, in IEEE International Conference on Acoustics, Speech an Signal Processing (ICASSP), April, pp.. [] A. H. Khan, M. Taseska, an E. A. P. Habets, A Geometrically Constraine Inepenent Vector Analysis Algorithm for Online Source Extraction. Cham: Springer International Publishing,, pp. 9. [9] M. Branstein an D. War, Microphone Arrays. Springer Verlag,. [] L. Parra an C. Alvino, Geometric source separation: Merging convolutive source separation with geometric beamforming, IEEE Transaction on Speech an Auio Processing, vol., no., pp., Sep.. [] D. Lahat, T. Aali, an C. Jutten, Multimoal ata fusion: an overview of methos, challenges, an prospects, Proceeings of the IEEE, vol., no. 9, pp. 9,. [] F. Nesta an Z. Kolovský, Supervise inepenent vector analysis through pilot epenent components, in IEEE International Conference on Acoustics, Speech an Signal Processing (ICASSP), March. [] K. Matsuoka an S. Nakashima, Minimal istortion principle for blin source separation, in Proceeings of International Symposium on ICA an Blin Signal Separation, San Diego, CA, USA, Dec.. [] T. Ono, N. Ono, an S. Sagayama, User-guie inepenent vector analysis with source activity tuning, in IEEE International Conference on Acoustics, Speech an Signal Processing (ICASSP), March, pp.. [] F. Weninger, J. R. Hershey, J. L. Roux, an B. Schuller, Discriminatively traine recurrent neural networks for single-channel speech separation, in Signal an Information Processing (GlobalSIP), IEEE Global Conference on, Dec, pp.. [] V. Kazemi an J. Sullivan, One millisecon face alignment with an ensemble of regression trees, in IEEE Conference on Computer Vision an Pattern Recognition, CVPR, Columbus, OH, USA, June -,,, pp.. [] S. Douglas an M. Gupta, Scale natural graient algorithms for instantaneous an convolutive blin source separation, in Proceeings of ICASSP, vol. II, Apr., pp.. [] F. Nesta an M. Matassoni, Blin source extraction for robust speech recognition in multisource noisy environments, Comput. Speech Lang., vol., no., pp., May. [9] F. Nesta, T. Thormunsson, an Z. Kolovský, On-line multichannel estimation of source spectral ominance, in International Conference on Latent Variable Analysis an Signal Separation. Springer,, pp.. [] N. Das, S. Van Eynhoven, T. Francart, an A. Bertran, Aaptive attention-riven speech enhancement for eeg-informe hearing prostheses, in Engineering in Meicine an Biology Society (EMBC), IEEE th Annual International Conference of the. IEEE,, pp.. ISBN EURASIP 9