TWO-STAGE SPEECH/MUSIC CLASSIFIER WITH DECISION SMOOTHING AND SHARPENING IN THE EVS CODEC

Transcription

1 TWO-STAGE SPEECH/MUSIC CLASSIFIER WITH DECISION OOTHING AND SHARPENING IN THE EVS CODEC Vladimir Malenovsky *, Tommy Vaillancourt *, Wang Zhe, Kihyun Choo, Venkatraman Atti *VoiceAge Cor., Huawei Technologies, Samsung Electronics Co., Ltd., Qualcomm Inc. ABSTRACT In most internationally recognized standardized multi-mode codecs, signal classification is erformed in a single ste by either linear discrimination or SNR-based metrics. The seech/music classifier of the EVS codec achieves greater discrimination than these single-ste models by combining Gaussian mixture modelling (GMM) with a series of context-based imrovement layers. Additionally, unlike traditional GMM classifiers the EVS model adots a short hangover eriod, allowing it to track transitions between music and seech. Misclassifications are mitigated by alying a novel decision smoothing and sharening technique. The results in relatively static environments demonstrate that the new two-stage aroach with selective hangover leads to classification accuracies comarable to seech/music classifiers with longer hangovers. They also show that the new aroach leads to faster and more accurate switching of coding modes than conventional classifiers for more comlex audio environments such as advertisements, jingles and seech suerimosed on music. Index Terms EVS, GMM, seech/music classification, smoothing, sharening. INTRODUCTION The Enhanced Voice Services (EVS) codec comrises numerous coding modes, each of which is tailored for a secific class of inut signals over a given range of bitrates []. For examle, the ACELP mode is most efficient when alied on seech signals at low and mid bitrates. The MDCT mode is suitable for ure music or mixed content at low, mid and high bitrates. The Generic signal coder (GSC) technology rovides good quality on generic audio content at low bitrates []. In order to aly the most aroriate coding mode for any tye of inut signal at any moment, the EVS codec uses a novel robust technique of seech/music classification. Song et al. have roosed an imrovement to the seech/music classifier of the 3GPP2 V codec [2] based on the GMM. In their roosed method, GMM features are calculated as running averages of arameters, including the following: LSF, signal energy, reflection coefficients and a eriodicity counter. The initial decision of the EVS seech/music classifier is also based on GMM [3] but its features are calculated either instantaneously (in the current frame) or as a moving average between those in the current and the revious frames. Thus, the effective memory of the EVS GMM is just one or two frames. This shorter memory results in a quicker resonse time by the classifier to abrut transitions from music to seech; a situation that haens frequently in many scenarios including rofessionally recorded radio transmissions, or informally when user talks in the resence of background music at discotheques or in shoing malls, cafés and ubs. Song et al. [2] reort an almost 60% imrovement in the detection of music by their GMM method when comared to the baseline V codec. This imrovement is achieved on any signal containing some trace of music, incl. seech suerimosed on music. Whilst this may be aroriate for the V codec, for the EVS codec this would lead to the selection of the MDCT coding mode resulting in distortion of seech utterances. In addition, there are frequent misclassifications of seech onsets resulting from long hysteresis. The EVS seech/music classifier oerates on ms frames reviously declared as active by the VAD []. The internal samle rate of the classifier is The classifier has been trained and otimized on a signal database samled at 6000 Hz and down-samled to 2800 Hz. The algorithm has low comutational comlexity and relatively modest memory footrint. The block diagram of the classifier is shown in Fig.. 2. FEATURE SELECTION The comlexity of the classifier is minimized by reusing arameters that have been calculated in earlier stages of the codec re-rocessing, i.e. during LP analysis, sectral analysis and oenloo itch analysis. The initial selection of the feature set for the GMM was erformed by analyzing the correlation matrix of the comlete set of features whilst running the codec on a training database. This technique was reorted by Karnebäck in [4] where the behavior of the 4Hz modulation feature was analyzed using critical bands. We have extended Karnebäck s method by incororating feature averages, their derivatives and logarithms comuted between the current and the revious frames. In this way a set of 68 candidate features, with minimal mutual correlation, was derived and analyzed. The list was then runed by examining histograms of individual features and calculating their discrimination otential according to the following metric, U m ( j) m ( j), () M ( s) ( mus ) 2 j0 where m (s) and m (mus) are the histograms of the feature calculated on the seech and the music training database, resectively, and M=256 is the total number of bins covering the values of the feature, normalized in the range [0;]. Note that each histogram was normalized with the total number of frames in the database. The discrimination otential U ranges from 0 (no discrimination) to.0 (maximum discrimination). This is illustrated in Fig. 2. This rocedure leads to the following set of 2 features (with their resective discrimination otentials) finally

2 feature extraction and normalization log-robability of seech and music sarse sectrum imrovement - seech - music histogram GMM itch n. correlation LSF2 LSF3 LSF4 LSF5 LSF6 tonality sharening and smoothing state machine, hangover vocal music imrovement onset/attack correction nonstationarity LP error energy sectral diff. s. stationarity st stage (GMM-based) 2 nd stage (context-based) Fig. 2: Normalized histograms of features and marginal densities of the GMM (x-axis scaled to [0;]) Fig. : Schematic diagram of the EVS seech/music classifier being selected from the initial 68 candidates; oen-loo itch (0.36), normalized correlation (0.37), 5 LSF arameters ( ), tonality (0.56), non-stationarity (0.54), residual LP error energy (0.36), sectral difference (0.38) and sectral stationarity (0.35). A detailed descrition of these features and the exact method of their calculation are rovided in []. 3. THE STATISTICAL MODEL The GMM uses 6 comonents (mixtures). It has been trained by the Exectation Maximization (EM) algorithm [5] on a large database containing 2 hours of clean seech, 2 hours of noisy seech and 3 hours of music (classical, modern, rock and o, jazz, etc.). The clean seech database contained both male and female talkers in 7 different languages. The noise database contained car, street, office and babble noise at various SNR levels. The music database was selected from a collection of various genres of classical and modern music, mainly instrumental. The level of all inut signals was normalized to -26 dbov rior to training. The GMM is a weighted sum of K-comonent Gaussian densities (K=6) given by the equation K ( x ) wk ( x μk, Σ k ), (2) k where x is a N-dimensional feature vector (N=2) reresenting the current frame, wk, k=,...,k are the comonent weights and ( x μk, Σ k) are the comonent Gaussian densities. The feature vector is normalized rior to the robablity calculation. Fig. 2 (dotted lines) illustrates the marginal Gaussian densities of the GMM after training. The GMM yields two raw robabilities, s and m, for the seech and the music models, resectively. By comaring the values of s and m in each frame it is ossible to obtain a raw discrimination measure between seech and music (-RAW). This is achieved by calculating the difference of the log-robability as f log( ) log( ). (3) m s The results of the classification based on the -RAW are shown later in this aer in Table III. The ercentage of correct decisions aears relatively low due to the fact that only -2 frames are taken into account during the feature extraction hase. 4. DECISION OOTHING AND SHARPENING It was observed that the dynamic range of -RAW is relatively low and fluctuates around zero, esecially in the resence of mixed signals. This is illustrated in Fig. 3. On the other hand, it is clear that -RAW reacts very quickly to transitions from music to seech and vice versa. To fully exloit the discrimination otential of -RAW, f is smoothed and sharened by the following adative auto-regressive (AR) filter f f ( ) f, (4) [ ] c c where γc is a filtering factor in the range [0;] and the suerscrit [ ] denotes a value from the revious frame. The filtered decision, f, is denoted -SS. The filtering factor is a combination of smoothing and sharening effects. It is calculated in each frame according to the following formula r, 0.0, (5) c E c where γe is the scaled relative frame energy and r is the sloe of -RAW. The relative frame energy Er is comuted as the ratio between the current frame energy and the estimated background noise energy []. It is scaled in the seech/music classifier as E 5, 0.0. (6) E r E The scaled relative frame energy has values close to in energetically significant segments and values close to 0.0 in the background noise. Therefore, if the signal energy is high more emhasis is ut on f to follow the raw decision more closely. On the other hand, if the SNR is low, it is naturally more difficult for the classifier to make a correct short-term decision and more reliance is ut on ast data; the decision is therefore smoothed. The gradient of -RAW is also used to sharen the decision during transitions from music to seech (otential seech). This situation otentially occurs when f < 0 and f f. In this case r [ ] [ ] f f r r, 0. r, (7) where is initialized (reset) to the value of f each time when f 0 f. Thus, the gradient of -RAW is ositive only during frames when f is falling below zero and it reresents a quantitative measure of the decrease. This creates a sharening effect which is illustrated in Fig. 3 (see last but one trace for f).

3 The box is held by a bright red snaer To make ure ice, you freeze water VAD 0 8 frames state I E U S U S U I f - γ E 0.0 r 0. γ c 0.0 f - D 0 I INACTIVE, E - ENTRY, U - UNSTABLE, S - STABLE Fig. 3: Decision smoothing and sharening on a seech samle with background music 5. SIGNAL PARTITIONING AND HANGOVER The efficiency of the seech/music classiffier is further imroved by combining the raw decisions from 0-7 revious frames. This is referred to as the hangover addition or decision smoothing. Alying hangover in every frame would lead to misclassifications for seech onsets where ast information has little relevance. In turn, it would lead to the selection of an inaroriate coding mode. To avoid this, the signal is first artitioned and a taylored variablelength hangover is individually alied to each section. The inut signal is artitioned using a simle state machine as shown in Fig. 4. The INACTIVE state is selected as the initial state. It is switched to ENTRY state when the VAD goes to unity. The ENTRY state marks the first onset after a longer eriod of silence. After 8 frames in the ENTRY state the classifier enters into the ACTIVE state which marks a stable signal with sufficient energy. If the energy suddenly dros closer to the level of the background noise the classifier s state is changed to UNSTABLE where it may stay for u to2 frames. After this eriod it reverts back to the INACTIVE state. If the energy suddenly increases while the classifier is in the UNSTABLE state, the classifier enters the ACTIVE state, byassing the ENTRY state. This ensures continuity of classification during short auses. With the signal artitioned by the state machine hangover is alied according to the conditions defined in Table II. The final decision of the seech/music classifier (-HO) is binary. With the state machine in the INACTIVE state, the classifier outut is always zero. In the UNSTABLE state, the classifier decision of the revious frame is maintained. In the ENTRY state, the classifier outut is based on a weighted sum of kent revious values of -RAW where kent is the frame counter of the ENTRY state. The weighting factors are given in Table III. k 6. CONTEXT-BASED IMPROVEMENT LAYERS The final decision of the seech-music classifier (-HO) is reviewed and otentially corrected in the second stage of the INACTIVE (bckgnd. noise) VAD 0 2 frames energy ENTRY (onset or attack after silence) UNSTABLE (energy close to bckgnd. noise) energy or VAD 0 Fig. 4: State machine for signal artitioning TABLE I: Final decision with hangover ACTIVE (stable signal) state condition -HO (D) INACTIVE 0 UNSTABLE ENTRY STABLE [,.., 3] 0, D energy or VAD 0 kent [ k ] 0 f k f k 2 classifier. The concet of decision correction in the second stage has been reorted by Chou et al. in [8]. Here, the corrections are alied only during secific signal contexts and are usually based on long-term statistics. As an examle, unaccomanied background vocal music may contain strong tonal characteristics. Tyically, the first stage of the classifier declares these frames as seech and suggests using the ACELP aradigm. In reality, the MDCT or the GSC mode are better suited to encode such content. Deending on the signal context, a three-way classification is erformed at certain bitrates to select among ACELP, GSC or MDCT for imroved efficiency while avoiding any serious artifacts. To achieve this, two further indeendent state machines are deloyed gathering statistics over eight -HO decisions. In addition, tonal features and some other features from the first stage are re-analyzed to determine otential errors in the classification. In the case that the tonal feature analysis indicates the resence of strong vocal music and if the revious decisions were mainly indicating seech, then the decision is adjusted to music []. The context-based switching mechanism also emloys the following two new features; sectral sarseness and LP efficiency. Sectral sarseness is calculated from the log-energy sectrum obtained from the sectral analysis and sorted in descending order f D f 0 0, D 0 otherwise D TABLE II: Weighting factors for -HO in ENTRY state kent α0 α α2 α3 α4 α5 α6 α

4 of magnitude. It reresents the minimum sectral bandwidth that covers 75% of the total signal energy, i.e. f ss S( f ) 0.75Et, (8) f 0 where fss is the bandwidth reresenting sectral sarseness, Et is the total signal energy and S(f) is the sorted er-bin log-energy sectrum []. The LP efficiency is based on a ratio of residual energies of the LP analysis in the logarithmic domain, i.e. Eerr (3) log Eerr (), (9) where Eerr(3) and Eerr() are the residual energies of 3 th order and st order LP analysis, resectively. To reduce frequent switching the LP efficiency is smoothed by summing ε over a eriod of eight consecutive frames, i.e 0 [ i] i7. (0) It has been found exerimentally that signals with high sectral sarseness are better coded with the MDCT mode. At the same time, signals with high rediction gain are more naturally encoded with the GSC technology. The MDCT coding mode is selected by default. The GSC coding mode is selected only for signals with high LP efficiency and non-sarse sectrum. Some hysteresis is alied to the mode selection if the sectral sarseness is reasonably stable. A detailed descrition of all of the contextbased imrovement technologies is rovided in []. 7. EVALUATION AND TESTING The EVS seech/music classifier has been evaluated using several test signals. The clean seech samles (~3-5s) were taken from a seech corus of Recommendation ITU-T P.800-comliant sentence airs [9]. The music samles (~5-s) were obtained from a rorietary database of music extracts. Clean and noisy seech, instrumental and vocal music and mixed signals from real radio recordings have been evaluated. The samles in each test were concatenated by inserting aroximately 3s of silence. Background signal (noise or music) was added over the clean signal at aroriate SNR levels. The manual labeling of the real radio recordings was achieved by subjectively listening to the synthesized signal obtained by forcing the codec to either ACELP or MDCT coding mode and icking the otimum quality decision. Recommendation ITU-T G.9 library software tools were used for signal maniulation and level adjustment [6]. Comarison of the classifier described here with the GMM method of Song et al. [2] and with the RMS and zero crossings method of Panagiotakis et al. [7] is summarized in Table III. The method by Song et al. is denoted V and the method by Panagiotakis et al. is denoted RMS0X. The table shows the ercentage of correct seech and music decisions on the following testing signals. The clean signal contains only clean seech and silence. The 5 db car signal contains noisy seech signal with 5 db car noise in the background. In case of db music, the seech signal is combined with classical background music. The radio mix contains real recordings of some Canadian radio stations in which seech is overlaing with multi-genre music at various SNR levels (ranging from 30 db u to 0 db). The rock and o and classical testing signals contain only instrumental, single-genre music whereas oera contains only vocal music. occurence [%] music seech reaction time [frames] From the results it can be seen that the EVS seech/music classifier clearly outerforms the method by Panagiotakis et al. on all tested signals. The method roosed by Song et al. works well for certain music signals but it comletely fails to detect seech in the radio mix assages. The raw decision of the EVS seech/music classifier (-RAW) is useful in clean and noisy seech and in ure music, but for mixed content, the reliability falls to 60-70%. The smoothing and sharening technique (-SS) rovides the best results on all tested signals. The accuracy is imroved by u to 3% without sacrificing the erformance in raidly changing content (radio mix). The detection of music in radio mix signal is only 0.5% worse in -SS comared to -RAW. Finally, the addition of hangover to the decision (-HO) slightly imroves the detection of seech at the exense of some loss in the detection of music. This is exected as the objective of -HO is to reduce accidental switching of -SS during stable sections while maintaining the fast detection of seech onsets. Fig. 5 shows the measured reaction time of the classifier for transitions from music to seech and vice-versa. The classifier can be seen to react to sudden transitions from music to seech within ms ( frame). The reaction time in the oosite direction is between ms (5 8 frames). By comarison, the reaction time of the classifier without the smoothing and sharening method is aroximately ms in both directions, i.e. more than eight times slower. 8. CONCLUSION seech music Fig. 5: Reaction time to content switching (-HO) TABLE III: Comarison of seech/music classification accuracy seech RMS0X V -RAW -SS -HO clean db car db street db music radio mix classical oera radio mix music rock and o In this aer we have resented the EVS seech/music classifier. Unlike traditional GMM-based techniques, this classifier has a very short hangover eriod. Frequent misclassification is revented by means of a novel technique based on decision smoothing and sharening. The method described here uses adative hangover logic which achieves very short reaction times for seech onsets. The final decision is also adjusted by a series of context-based imrovement techniques. The overall accuracy on ure seech and music signals is more than 92% which is comarable to state-ofthe-art classifiers and results show that the EVS seech/music classifier outerforms the methods of Panagiotakis et al. and Song et al on all tested signals. The algorithm forms art of the 3GPP EVS codec ublished as TS [].

5 9. REFERENCES [] 3GPP Sec., Codec for Enhanced Voice Services (EVS); Detailed Algorithmic Descrition, TS , v.2.0.0, Se. 4. [2] Song, Ji-Hyun, et al. "Analysis and imrovement of seech/music classification for 3GPP2 V based on GMM." IEEE Signal Processing Letters, 08, [3] J. Pinquier, C. Senac, R. Andre-Obrecht, Seech and music classification in audio documents, in Proc. of Int. Conf. on Acoustics, Seech and Signal Processing (ICASSP), 02, Vol. 4, [4] S. Karnebäck. Discrimination between seech and music based on a low frequency modulation feature. Euroean Conf. on Seech Comm. and Technology, ages , 0. [5] A. P. Demster, N. M. Laird and D.B. Rubin "Maximum likelihood from incomlete data via the EM algorithm", J. Royal Statiscal Soc., vol. 39, no.,. 38, 977 [6] ITU-T G.9, Software tools for seech and audio coding standardization. International Telecommunication Union (ITU), Series G., 0 [7] C. Panagiotakis and G. Tziritas, A seech/music discriminator based on RMS and zero-crossings, IEEE Transactions on Multimedia, Vol. 7, No., Feb. 05. [8] W. Chou and L. Gu, Robust singing detection in seech/music discriminator design, in Proc. of Int. Conf. on Acoustics, Seech and Signal Processing (ICASSP), May 0, Vol. 2, [9] ITU-T P.800, Methods for Subjective Determination of Transmission Quality. International Telecommunication Union (ITU), Series P., August 996.