Speech Endpoint Detection Based on Sub-band Energy and Harmonic Structure of Voice Yanmeng Guo, Qiang Fu, and Yonghong Yan ThinkIT Speech Lab, Institute of Acoustics, Chinese Academy of Sciences Beijing 100080 {yguo,qfu,yyan}@hccl.ioa.ac.cn Abstract. This paper presents an algorithm of speech endpoint detection in noisy environments, especially those with non-stationary noise. The input signal is firstly decomposed into several sub-bands. In each sub-band, an energy sequence is tracked and analyzed separately to decide whether a temporal segment is stationary or not. An algorithm of voiced speech detection based on the harmonic structure of voice is brought forward, and it is applied in the non-stationary segment to check whether it contain speech or not. The endpoints of speech are finally determined according to the combination of energy detection and voice detection. Experiments in real noise environments show that the proposed approach is more reliable compared with some standard methods. 1 Introduction Speech endpoint detection (EPD) is to detect the beginning and ending boundaries of speech in the input signal, which is important in many areas of speech processing. An accurate speech endpoint detection is crucial for the recognition performance in improving the recognition accuracy and reducing the computing complexity. Endpoint detection discriminates speech from noise by some features of the signal, such as energy[1][2], entropy[3][4], LSPE[5], statistic properties [6][7][8], etc.. Some methods treat speech and noise as separate classes and detect speech by models of speech and noise. These methods perform well in specific environments, but degrade rapidly when the models mismatch the environment. However, if the discrimination is based on some heuristically derived rules relating to the signal features, its performance relies on the properties directly, and it is easier to adapt to the unknown environments. For the practical speech recognition, it is critical to detect speech reliably under diverse circumstances. This paper develops a robust endpoint detection method that combines the advantages of several features by rules. Short-time energy is the most widely used parameter in endpoint detection[1][2][9][10]. But it is not sufficient if the noise level is high. Fortunately, the This work is (partly) supported by Chinese 973 program (2004CB318106), National Natural Science Foundation of China (10574140, 60535030), and Beijing Municipal Science and Technology Commission (Z0005189040391)
spectral energy distributions of speech and noise are often different, so speech is not corrupted equally in different frequency. This fact is exploited in this paper by analyzing and tracking the energy in 4 sub-bands, and taking more importance to the sub-bands with drastic energy variation. Another shortcoming of energy parameter is to misclassify high level noise as speech when the noise is time-varying. In this paper, this problem is solved by involving voice detection. If the non-stationary segment contains voiced speech, speech is detected, otherwise, it is classified as noise. Detecting voiced speech is also an important strategy to distinguish between speech and noise. Generally, voiced speech can be detected by tracking pitch[11], measuring periodicity[5][12][13], etc., but those methods are often disturbed by low-frequency noise or the abrupt changes of noise. However, voice has obvious harmonic structure in frequency domain even in very noisy case, and his paper proposes a robust algorithm that detects voice by adaptively checking such structure in frequency domain. This paper is organized as follows. The theory of the proposed algorithm is described in section 2. Section 3 evaluates and analyzes the performance of the algorithm. The conclusion is given in Section 4. 2 Algorithm Assuming that the speech and additional noise are independent, the short-time energy of input signal is given by E x = E s + E n, where E s and E n represent the energy of speech and noise, respectively. Thus the position of speech signal can be determined by searching the segments where E x > E n. However, noise may be non-stationary, and its energy E n can hardly be estimated precisely. To solve this problem, we classify the input signal into two categories: the stationary, which is assumed to exist all the time, and the non-stationary, which may contain speech, noise, or both. The stationary component can be tracked using specified model mentioned in Sect.2.2. Then we apply voice detection method proposed in Sect.2.3 in the non-stationary segments to detect speech. The structure of the algorithm is shown in Fig.1. 2.1 Preprocessing The 8KHz sampled noisy speech is divided into L frames with each frame 20ms long and overlapped 50%. After being applied by a window function and analyzed by short-time Fourier transform (STFT) of N (N 256) points, the energy of kth frequency bin in the ith frame can be derived from the spectrum, and represented as P i (k), where 0 k < N/2. Setting borders at {0,500,1000,2000,4000}Hz, the signal is divided into 4 non-overlapped sub-bands. Thus the energy of subband m in frame i can be obtained by summing up the energy of its frequency components, and denoted as E x, m (i), where m = 0, 1, 2, 3, and i = 1, 2,...L.
Input Signal Stationary Noise N Detect Voice Update Noise Model Y N Contain Voice Y Search Endpoints Output Endpoints Fig. 1. Flowchart of the proposed algorithm 2.2 Energy Detection Based on the character of energy sequence, additional noise can be classified to 5 classes here: stable noise, slow-varying noise, impulse noise, fluctuant noise and step noise. All the kinds of noise are independent and additional, and their sum is the input noise. Stable noise, such as thermal noise or the noise of running machine, basically has stable energy distribution, and its energy sequence follows ergodic Gaussian distribution. Slow-varying noise denotes the noise whose energy distribution changes slowly, and the noise of wind blowing or coming car can be classified into this kind. In a short interval, it can be looked approximately as stable noise. Impulse noise involves those whose energy rise and fall rapidly, and its energy only keeps nonzero in short period. The typical examples are smack and click. Fluctuant noise has varying energy all the time, and it includes the babble noise, continual bump in car and the noise of several passing vehicles. Step noise is the noise whose energy distribution changes abruptly like steps, and it includes noise from turning on a machine as well as the noise from abrupt changes in telecommunication channel. It can be classified separately to stable, slow-change or fluctuant noise before and after the step. Accordingly, the noise energy of sub-band m in frame i is expressed as E n, m (i) = E p, m (i)+e q, m (i) in the duration of 100 300ms, which is about the length of a syllable. E p, m (i) is an ergodic stationary Gaussian random sequence composed of stable noise, slow varying noise and the stationary section of step noise, and E q, m (i) is a non-stationary sequence made from other noise. Hence, in the total energy of {E x, m (i) = E s, m (i)+e p, m (i)+e q, m (i)}, {E s, m (i)} and {E q, m (i)} are both non-stationary sequences that are difficult to be discriminated only by energy, and that s the reason to apply the voice detection. Stationary noise modeling An adaptive model is set up to track the stationary noise for each sub-band. For clarity, we omit the argument m hereafter in description of the model initialization and update. Let {E p (i)} denote the energy sequence of stationary noise in sub-band m, and its probability distribution function is f(e p ) = (1/ 2πσ)exp( (E p
µ) 2 /2σ 2 ) in a short period, where µ and σ are mean and variance respectively. Define the normalized variance λ = σ/µ, then f(e p ) = (1/ 2πλµ)exp( (E p /µ 1) 2 /2λ 2 ), where λ represents the relatively dynamic range. {E p (i)}is the only stationary component in{e x (i)}, so its distribution can be estimated in the segments where {E x (i)} is stationary. However, {E p (i)} only keeps stationary and ergodic in short period, and it dominates the signal in even less time. Therefore, its distribution is assumed to be stable in 200 300ms ( l frames), and µ and λ can be estimated by the beginning 80 120ms signal ( r frames) of it. jth model (j-1)th model Update model jth model Detect by energy r frames l-r frames jth Analysis Window ( j+1)th model Update model Detect by energy Delete 1 frame Shift 1 frame Input 1 frame (j+1)th Analysis Window Fig. 2. Strategy of update the analysis window Accordingly, set the analysis window of l frames and calculate model parameters by its beginning r frames. Then set energy threshold as θ = µ + µ λ/α, and apply it to test the latter l r frames, where α is the sensitivity coefficient and 0 < α < 1. When a new frame is inputted, the analysis window shifts one frame, and the model is updated to calculate new θ, as shown in Fig.2. Model initialization and update The model for the sub-band is initialized in the first analysis window by its energy of beginning r frames. Set µ to their mean ε 1 = 1 r r i=1 E x(i), and set λ to the normal variance ξ 1 = [ r i=1 (E x(i) ε 1 ) 2 ] 1/2 /(ε 1 r). Initial signal may compose non-stationary components, and the distribution of {E p (i)} is also time-varying, so the model is adjusted in all the following analysis windows to track the distribution of {E p (i)}. Take the jth analysis window for example, as shown in Fig.2, get the mean and normal variance of the beginning r frame, denoted as ε j and ξ j, then update λ and µ in the following 5 cases. 1. The input signal occasionally contains short silence or just constant component because of hardware errors. Hence, if ε j < µ sil, set µ = µ sil where µ sil is the experimental minimum of µ. 2. For the same reason, if ξ j < λ sil, and ε j < ε j 1, set λ = λ sil where λ sil is the experimental minimum of λ.
3. If ε j < µ c and ξ j < λ c, then set µ = ε j and λ = ξ j, where c is a constant and 1 < c < 1.5. This is to track the decreasing or slow varying noise; 4. If ε j < µ c and ξ j < ξ j 1 < ξ j 2, the noise is getting stationary and its level is lower, so set µ = ε j and λ = ξ j. 5. If ε j (1 + ξ j ) < µ (1 + λ j ), the noise is decreasing too, so set µ = ε j and λ = ξ j as well. The above cases are checked one by one, and once a condition is met, update the parameters by it and neglect all the following cases. If none condition is met, keep the current λ and µ. Band selection and threshold setting The presence of speech improves the energy level in every sub-band, and in most cases, this is obvious in the sub-bands that are dominated by speech. Hence, the non-stationary signal are detected in the latter l r frames of the current analysis window. If the mean energy of r continuous frames in a sub-band is higher than θ, then the sub-band detects non-stationary signal. And for the consecutive r frames, if 2 of the 4 sub-bands detects non-stationary signal, and the mean energy in the other 2 subands is higher than µ, the non-stationary signal is detected. 2.3 Voice Detection Based on Harmonic Structure The voice detection is carried out in the current analysis window after detecting non-stationary signal. In general, voice is modeled as the production of vocal tract excited by periodic glottal flow, so the short-term spectrum voice has energy peaks on pitch and harmonic frequencies. It is reflected in narrow band spectrograms as parallel bright lines, because pitch varies slowly. Most noise don t have such character, so checking harmonics is an effective method for voice detection. The harmonic components dominant the energy of voice, so the harmonic character remains outstanding even with background noise. However, the spectral energy envelop of speech varies with pitch and formants, and the energy distribution of noise is also time-varying. Hence, the speech spectrum is not corrupted equally in different frequency, and the bands with clear harmonic character are also time-varying. In this paper, the voice is detected by an adaptive method of searching clear harmonic character in a wide band, and the information of neighbor frames is considered as well. This strategy keeps robust against distortions, low-frequency noise and pitch tracking error. Peak picking in a frame The spectral energy of voiced speech usually has peaks at harmonics, which are multiples of the fundamental frequency. However, some peaks will be submerged in corruption of noise, while a lot of spurious peaks are brought up. Fortunately, under most circumstances, at least 3 4 consecutive harmonics will keep clear, that is to say, a frame of corrupted voice has 3 4 spectral energy peaks with a spacing of fundamental frequency(60 450Hz) between adjacent ones. To detect the harmonics in frame i, peaks are picked from P i (k) as follows.
1. Extract all the local peaks in the spectrum. 2. Eliminate the peaks that are lower than an experimental threshold to delete some peaks caused by noise. 3. Merge the trivial (low and narrow) peaks into the dominant ones nearby. 4. Eliminate the remaining peaks with relatively small height or width. Matching peaks with harmonics If frame i contains voice, there will be spectral energy peaks at the multiples of fundamental frequency. We take various F 0 to see if {Q i (n)} match its multiples, in which F 0 is incremented in a step length of F = 1.5Hz within the range of [60Hz, 450Hz]. If {Q i (n)} contains peaks in the position of at least 4 consecutive harmonics, or 3 peaks matching the 1,2,3 multiples of F 0, then record the peaks as potential harmonics for F 0. It is assumed that every frame contains voice of one speaker at most. To eliminate the spurious harmonics, the continuity of F 0 and harmonics are checked. For the consecutive frames numbered from i b to i e, if F 0 fluctuates within a limited extent and its harmonics, n F 0 (n + 3) F 0, are all matched in those frames, then frame i b to i e is detected as voice. The case of n = 0 means that the harmonics of F 0, 2F 0 and 3F 0 are concerned. 2.4 Speech Endpoint Determination Start point determination As is shown in Fig.1, if there exists voice in the analysis window, the non-stationary signal can be confirmed as speech, and its start point is searched in every sub-band based on energy. Take sub-band m as example (and number m omitted), after finding voice in the analysis window, the start point of speech is searched forward by ε i and ξ i from the first voiced frame. If a frame satisfies ε i > ε i+1 and ε i > θ, then it is detected as the start point for this sub-band. The earliest start point of all the sub-bands are detected as the temporary start frame b s. The onset of speech usually increases the signal energy abruptly, so the noise model keeps stable in the beginning of speech, and the energy threshold keeps low. Moreover, voice usually locates after unvoice, and it has much higher energy than unvoice, so the temporary start point probably locates in the unvoice section or before the unvoice. To get the refined start point, the second step is to find the local maximum of ξ i near frame b s, and set it as the beginning point. If there isn t any ξ i maximum for a sub-band in a predefined boundary, it keeps b s. At last, the earliest start frame of the sub-bands is detected as the start of speech. End point determination The end point is searched after finding the start point. End threshold θ is initialized as θ = µ + µ λ. The parameters µ and λ are updated by ε i and ξ i based only on the criteria 1, 2 and 5 whenever a new frame shifts into the analysis window. Thus the noise can be updated by the weaker or more stationary noise in speech pauses. For every sub-band, if none of the successive l frames in the analysis window meets the rule that every frame has energy higher than θ, where l is an experimental threshold with range of
8 < l < 20, then the end frame is detected as the first frame of that analysis window. And if 3 sub-bands detect end point, and there exists no voice in the analysis frame, the endpoint is determined. 3 Evaluation and Analysis The accuracy of endpoint detection has a strong influence on the performance of speech recognition by triggering on and off the recognizer on speech boundaries. In this paper, the performance of the proposed algorithm is tested using a grammar-based speaker-independent speech recognition system, and the reference endpoint detection approaches are ETSI AFE[14] and VAD in G.729B[9]. Due to the wide range of speech recognition equipments and circumstances, the test data are recorded in several environments by PDA, telephone, and mobile phones, and each test file contains a segment of noisy speech with 2 6 syllables of Chinese words. Table 1. Recognition Performance Results Correct Rate(%) Error Rate(%) Rejection Rate(%) Data total ETSI G729 Proposed ETSI G729 Proposed ETSI G729 Proposed 1 6787 88.5 83.7 86.7 11.3 13.6 12.4 0.19 2.68 0.80 2 4498 62.3 62.0 67.3 37.3 35.0 30.2 0.39 2.96 2.38 3 1591 78.8 73.0 81.5 21.1 21.4 18.7 0 5.53 0.13 4 1592 62.3 61.9 63.4 37.5 37.8 35.5 0.13 0.25 1.07 5 796 39.2 40.9 42.6 60.8 59.0 56.5 0 0 0.88 6 1992 46.3 46.6 46.8 50.3 51.0 49.4 3.26 2.36 3.77 7 795 63.3 63.3 65.7 36.5 36.6 33.6 0.12 0 0.75 8 499 75.9 77.7 82.2 13.8 15.2 9.42 10.2 7.01 8.42 9 48 72.9 68.8 75 27.0 31.2 25 0 0 0 10 500 74.6 72.6 84.4 19.2 21.4 11.2 6.2 6 4.4 11 498 77.9 69.5 78.7 14.6 19.0 14.5 7.43 12.0 6.83 Table 1 shows the comparative results for different EPDs. Data 1 and data 2 were both recorded in quiet office by telephone, but data 2 was in the hands-free mode, so there are much more noise of clicks, electric fan and so on. Data 3 was recorded by PDA in office with window opened, so there exists much more impulse noise than data 1 and 2. Data 4 was recorded by PDA in supermarket when there were not too many clients. Hence, the most important noise was the stepping and clicking of the clients, and occasionally with some voice of them. Data 5 was recorded in an airport lounge by PDA, and the broadcast was not playing. The main interference was from people talking, stepping and baggage moving nearby. Data 6 was recorded near a noisy roadside, and there were people talking, moving and vehicles running. Data 7 was recorded by PDA mobile outside the gate of a park. There people talking and vehicles moving around. Data 8 was recorded by GSM mobile on the side of a high way, so the
noise from wind and vehicle was really serious. Data 9 was also recorded by GSM mobile near the a high way, but there was music played and the mobile telephone was in mode of adaptive noise-canceling, so whenever the speech begins, the volume of noise is suppressed automatically. Data 10 was recorded by CDMA mobile in office with window opened, and the speech volume was very low because of the telecom channel. Data 11 was recorded in office of opened window, but the equipment was PAS mobile, so the volume was a little higher than data 10. As can be seen in Table 1, the proposed algorithm has performance comparative to G.729B and ETSIAFE in quiet environment, and outperforms them in most noisy environments, especially in the time-varying noise. For the database with noise of vehicles, steps, clicks and other environment noise, such as data 2, 3 and 8, the proposed algorithm is much superior than the standards. Even if the energy of non-stationary noise is high, it is still rejected by voice detection and can not enter the recognizer, because there is not harmonic structure in its spectrum. However, the voice detection cannot be so effective for the noise from other human s voice, as can be seen in data 4, 5, 6 and 7. Such noise has the character of harmonic as well, so some of them could be detected as voice. Fortunately, the energy detection serves as the first step for speech detection, and the noise with low energy is rejected first. Then, the voice detection also discriminates some interfering voice from the user s voice, because the harmonics in far-away speech are usually not as continuous and clear as the user s. For the interfering talkers nearby, some more effective approaches are still needed. In the case of quite environment, as is in data 1, the proposed algorithm has better performance than G.729B, whereas not as good as ETSIAFE. This is because of the mis-hit in voice detection when the user s speech is short or hoarse. After all, it is still acceptable for most practical purposes. The advantage of adaptive energy tracking is clear in the case of data 9, 10 and 11, in which the volume is low or time-varying. By tracking the stationary noise in 4 sub-bands, the non-stationary signal is detected to start the voice detection, so the final detection is not affected by the level or the variance of signal. 4 Conclusion This paper puts forward a speech endpoint detection algorithm for real noisy environment. It performs reliably in most noisy environments, especially in those with abrupt changes of noise energy, which is typical in mobile and portable circumstances. The following research will focus on the environments with interfering speech and music. 5 Acknowledgement The authors would like to thank Heng Zhang for his helpful suggestions.
References 1. J.F.Lynch, J.G.Josenhans, R.E.Crochiere: Speech/silence segmentation for realtime coding via rule based adaptive endpoint detection. Proc. ICASSP, Dallas (1987) 1348 1351 2. Li, Q.: Robust endpoint detection and energy normalization for real-time speech and speaker recognition. IEEE Transactions on Speech and Audio Processing 10(3) (2002) 146 157 3. Huang, L.S., Yung, C.H.: A novel approach to robust speech endpoint detection in car environments. Proc. ICASSP (2000) 1751 1754 4. Weaver, K., Waheed, K., Salem, F.M.: An entropy based robust speech boundary detection algorithm for realistic noisy environments. IEEE Conference on Neural Networks (2003) 680 685 5. R.Tucker: Voice activity detection using a periodicity measure. IEE Proceedings 139(4) (1992) 377 380 6. H.Othman, T.Aboulnasr: A semi-continuous state transition probability hmmbased voice activity detection. Proc. ICASSP V (2004) 821 824 7. S.Gazor, W.Zhang: A soft voice activity detector based on a laplacian-gaussian model. IEEE Transactions on Speech and Audio Processing 11(5) (2003) 498 505 8. Li, K., Swamy, M.N.S., Ahmad, M.O.: Statistical voice activity detection using low-variance spectrum estimation and an adaptive threshold. IEEE Transactions on Speech and Audio Processing 13(5) (2005) 965 974 9. Benyassine, A., Shlomot, E., Su, H.Y.: Itu recommendation g.729 annex b: A silence compression scheme for use with g.729 optimized for v.70 digital simultaneous voice and data applications. IEEE Communication Magazine (1997) 64 73 10. Marzinzik, M., Kollmeier, B.: Speech pause detection for noise spectrum estimation by tracking power envelope dynamics. IEEE Transactions on Speech and Audio Processing 10(2) (2002) 109 118 11. Zhang, T., Kuo, C.C.J.: Audio content analysis for online audiovisual data segmentation and classification. IEEE Transactions on Speech and audio Processing 9(4) (2001) 441 457 12. Tanyer, S.G., Ozer, H.: Voice activity detection in nonstationary noise. IEEE Transactions on Speech and Audio Processing 8(4) (2000) 478 482 13. Seneff, S.: Real-time harmonic pitch detector. IEEE Transactions on Acoustics, Speech, and Signal Processing ASSP-26(4) (1978) 360 365 14. ETSI: Es 201 108 recommendation: Speech processing, transmission and quality aspects (stq); distributed speech recognition; advanced front-end feature extraction algorithm; compression algorithms. (2002)