Multipitch analysis of polyphonic music and speech signals using an auditory model

Transcription

1 1 Multipith analysis of polyphoni musi and speeh signals using an auditory model Anssi Klapuri, Member, IEEE Institute of Signal Proessing, Tampere University of Tehonology, Korkeakoulunkatu 1, FIN-3372 Tampere, Finland tel , fax , anssi.klapuri@tut.fi Abstrat A method is desribed for estimating the fundamental frequenies of several onurrent sounds in polyphoni musi and multiple-speaker speeh signals. The method onsists of a omputational model of the human auditory periphery, followed by a periodiity analysis mehanism where fundamental frequenies are iteratively deteted and anelled from the mixture signal. The auditory model needs to be omputed only one, and a omputationally effiient strategy is proposed for implementing it. Simulation experiments were made using mixtures of musial sounds and mixed speeh utteranes. The proposed method outperformed two referene methods in the evaluations and showed a high level of robustness in proessing signals where important parts of the audible spetrum were deleted to simulate bandlimited interferene. Different system onfigurations were studied to identify the onditions where pith analysis using an auditory model is advantageous over onventional time or frequeny domain approahes. Index Terms Fundamental frequeny estimation, pith pereption, musi information retrieval, aousti signal analysis. I. INTRODUCTION Pith analysis of polyphoni musi and multiple-speaker speeh signals is useful for many purposes. Appliations inlude automati musi transription, speeh separation, strutured audio oding, and musi information retrieval. The task of estimating the fundamental frequenies (Fs) of several onurrent sounds multiple-f estimation is losely related to sound separation and auditory sene analysis, sine an algorithm performing this task goes a long way towards organizing a omplex signal into its onstitutent sound soures [1]. This paper proposes a method for doing this in singlehannel audio signals. A number of different approahes have been proposed for multiple F estimation (see [2], [3] for a review). The first algorithms were developed to transribe polyphoni musi and were more or less heuristi in nature [4], [5], [6], [7]. Methods based on modeling the auditory sene analysis (ASA) funtion in humans were later proposed by Mellinger [8], Kashino and Tanaka [9], Ellis [1], Godsmark and Brown [11], and Sterian [12], presumably inspired by Bregman s work on human ASA [13]. Signal model based Bayesian inferene methods were investigated by Goto [14], Davy et al. [15], Cemgil [16], and Kameoka et al. [17]. Most reently, unsupervised learning methods, suh as independent omponent analysis, sparse oding, and non-negative matrix fatorization, have been proposed A. Klapuri is with Institute of Signal Proessing, Tampere University of Tehnology, Tampere, Finland ( anssi.klapuri@tut.fi). by Casey and Westner [18], Lepain [19], Smaragdis and Brown [2], Abdallah and Plumbley [21], and Virtanen [22]. Auditory model based methods represent an important thread of work in this area sine the early 199s. By these we mean methods whih employ a peripheral hearing model to alulate an intermediate data representation that is then used in further signal analysis. The rationale in doing this is that humans are very good at resolving sound mixtures and therefore it seems natural to employ the same data representation that is available to the human brain. Auditory model based methods have been proposed at least by Meddis and Hewitt [23], de Cheveigné [24], and Wu, Wang, and Brown [25] for speeh signals, and by Martin [26], Tolonen and Karjalainen [27], and Marolt [28] for musi signals. The mentioned methods are oriented towards pratial pith extration in speeh and musi, as opposed to the work on pith pereption models themselves, whih aim at reproduing psyhophysial data and phenomena in an aurate manner. Exellent reviews on pith pereption models an be found in [29], [3]. This paper proposes a multiple F estimator whih onsists of a omputational model of the human auditory periphery, followed by a periodiity analysis method where Fs are iteratively deteted and anelled from the mixture signal. For both parts, omputationally effiient tehniques are presented whih make the overall method more than twie faster than real time on a PC with a 2.8 GHz Pentium 4 proessor. In partiular, a mehanism is desribed whih speeds up the analysis at the subbands of an auditory model. In the periodiity analysis part, we replae the onventional autoorrelative analysis with a transform whih is more robust in polyphoni signals and an be used for a wide pith range between 4 Hz and 2.1 khz. Some parts of this work have been previously published in two onferenes papers [31] and [32]. One goal of this paper is to identify the onditions where an auditory model based pith analysis has signifiant advantage over more onventional time or frequeny domain approahes. It will be shown that these onditions inlude espeially the proessing of bandlimited signals or signals were parts of the spetrum are not usable due to bandlimited interferene. The method was evaluated using mixtures of musial sounds and mixed speeh utteranes. The results are ompared with two referene methods [27] and [33]. Also, we ompare alternative onfigurations of the proposed method where either the auditory model is disabled or the iterative estimation and anellation mehanism is replaed with a joint estimator.

2 2 x(n) Auditory model Intermediate data representation U(k) Period detetion Canel deteted sound and iterate Fig. 1. An overview of the proposed method. An auditory model is followed by the iterative detetion and anellation of the most prominent period. x(n) Auditory filterbank x (n) Compress, y (n) Y (k) p retify, DFT( ) p lowpass U(k) ˆτ... Fig. 2. Struture of the peripheral hearing model. An input signal is proessed with a bandpass filterbank, after whih the subband signals are ompressed, retified, and lowpass filtered. Short-time magnitude spetra are alulated within the bands, raised to power p, and then summed aross bands. + ˆτ Magnitude (db) Amplitude Frequeny (Hz) 5 x Time (ms) Amplitude Magnitude (db) Frequeny (Hz) Time (ms) Fig. 3. The upper panels show the magnitude responses of two gammatone filters (solid line) and those of the proposed approximation (dashed line). The lower panels show the impulse responses of the two gammatone filters (solid line) and the differene between the impulse reponses of the gammatone filter and the proposed approximation (dotted line). The left and right panels orrespond to enter frequenies 1 Hz and 1 khz, respetively..1 II. PROPOSED METHOD Figure 1 shows an overview of the proposed method, where the two parts, the auditory model and the iterative F detetion part, are learly seen. The auditory model is detailed in Fig. 2. Exept for the omputational effiieny onsiderations whih are desribed later, the auditory model follows the struture of modern pith pereption models. In these, a signal is generally proessed as follows: 1) An input signal x(n) is passed through a bank of linear bandpass filters whih models the frequeny seletivity of the inner ear [34], [35]. 2) The signal x (n) at band (aka hannel) is subjeted to non-linear proessing to obtain a signal y (n) whih models the level of neural ativity in the auditory nerve fibers representing hannel [36], [37]. 3) Periodiity analysis of some form takes plae for the signals y (n) within the hannels [38], [39]. 4) Periodiity information is ombined aross the bands. As a onrete example of Steps 3 4, Meddis and Hewitt [38] omputed short-time autoorrelation funtion (ACF) estimates r,t (τ) within the hannels at suessive times t, and then summed these to obtain a summary ACF, r t (τ) = r,t(τ), where prominent peaks were used to predit the pereived pith. Different parts of the system shown in Figs. 1 2 are now desribed in more detail. A. Auditory filterbank The most important parameter of the auditory filters is their bandwidth. The equivalent retangular bandwidths (ERB) 1 of 1 The ERB of a filter is defined as the bandwidth of a perfetly retangular filter whih has an integral over its power response whih is the same as for the speified filter. the filters we use are b =.18f Hz, (1) where f is the filter s enter frequeny, b is the bandwidth, and =,1,...,C 1 is the subband index. These bandwidths have been reported for humans in [4]. In order that the power responses of the auditory filters would sum approximately to a flat response, the enter frequenies are distributed uniformly on a ritial-band sale, f = 229[1 (ξ1+ξ)/21.4 1], (2) where ξ is the ritial-band-number of the lowest band and < ξ 1 < 1 determines the band density. We use a total of 7 filters having enter frequenies between 65 Hz and 5.2 khz, orresponding to ξ = 2.3 and ξ 1 =.39. The power and impulse responses of the auditory filters have been studied in humans and other mammals and are quite aurately known [34], [35]. The gammatone filter provides an exellent fit to the experimental data, and is therefore widely used [41]. Figure 3 illustrates the frequeny response and the impulse response of the gammatone filter. Slaney has proposed a omputationally effiient implementation of the gammatone filter by using a asade of four seond-order IIR filters [42]. We propose a different implementation for two reasons. Firstly, we wanted to attenuate the tails of the power response further away from the filter s enter frequeny, sine the spetral variation of musial sounds is very large and we wanted to ensure that a ertain filter is not dominated by frequeny omponents too far from its enter frequeny. Seondly, the omputational effiieny is improved by using filter setions whih have only oeffiient values ±1 in the numerator of their z-transform, thus reduing the number of multipliation operations needed.

3 3 The proposed filter struture uses two types of IIR resonators as building bloks, referred to as Resonator 1 and 2 in the following. An individual auditory filter onsists of a asade of these. The z-transform of Resonator 1 is of the form (1 z 1 )(1 + z 1 ) H 1 (z) = ρ 1 (1 Ae iθ1 z 1 )(1 Ae iθ1 z 1 ) 1 z 2 = ρ 1 1 Aos(θ 1 )z 1 + A 2, (3) z 2 where the parameters ρ 1, θ 1, and A are derived in Appendix A. Resonator 2 is of the same form but without the zeros, having a z-transform of the form H 2 (z) = ρ 2 1 (1 Ae iθ2 z 1 )(1 Ae iθ2 z 1 ). (4) Appendix A desribes the alulation of the parameters A, θ 1,2, and ρ 1,2, and hoosing the optimal onfiguration of seond-order setions. It was found that a asade of four resonators, two of eah type, leads to the most aurate result. In a floating-point implementation, the fators ρ 1,2 an be ombined into a single salar to speed up the omputation. The upper panels of Figure 3 ompare the frequeny response of the gammatone filter with the proposed approximation at two different enter frequenies, 1 Hz and 1 khz. The biggest inauraies our near zero frequeny, where the proposed filter has a deeper noth than the gammatone filter. In pratial appliations, omplete suppression of the d omponent is merely a desirable feature. The lower panels illustrate the impulse responses of the two gammatone filters, with a dotted line showing the differene between the gammatone filter and the approximation. B. Neural transdution The signal x (n) at eah band is proessed to model the transform harateristis of the inner hair ells (IHC) whih produe firing ativity in the auditory nerve. Several omputational models of the IHCs have been proposed in the literature [36]. A problem with these is that a realisti IHC model depends ritially on the absolute level of its input and has a dynami range of about 25 db only [43], [37]. As a onsequene, most pratial systems have replaed an aurate IHC model by a asade of signal proessing operations that model the main harateristis of the IHCs expliitly: (i) dynami level ompression, (ii) half-wave retifiation, and (iii) lowpass filtering [1], [44], [25], [31]. This is also the approah followed here. Compression was implemented with an automati gain ontrol, saling the signal x (n) within analysis frame t with the fator γ,t = σ ν 1,t, (5) where σ,t is the standard deviation of the signal x (n) within the frame t. From the viewpoint of an individual analysis frame, the ompression flattens ( whitens ) the spetral energy distribution, sine the saling fators γ,t normalize the auditory hannel varianes towards unity when < ν < 1. Here the value ν =.33 is applied. For omparison, Ellis Amplitude Magnitude Magnitude Time (ms) Frequeny (Hz) Frequeny (Hz) Fig. 4. The upper panel shows the subband signal x (n) at a band with enter frequeny 2.7 khz. The example signal is a trumpet sound with F 185 Hz. The middle panels shows the magnitude spetrum of the subband signal and the lower panel shows the spetrum after half-wave retifiation. [1] normalized the varianes of the subband signals to unity, orresponding to ν =. Tolonen and Karjalainen, in turn, applied inverse warped-linear-predition filtering on the input wide-band signal whih leads to a very similar result [27]. The ompressed subband signals are subjeted to half-wave retifiation (HWR), defined as HWR(x) = max(x,) = 1 (x + x ). (6) 2 Figure 4 illustrates the HWR for a subband signal x (n) of a trumpet sound at a band with enter frequeny 2.7 khz. The upper two panels show the subband signal in time and frequeny domains. The lowest panel shows the spetrum of the subband signal after retifiation, that is, the spetrum of HWR(x (n)). As an be seen, the retifiation generates spetral omponents at the baseband and on twie the hannel enter frequeny. The former represent the spetrum of the amplitude envelope of x (n). It onsists of beating omponents whih orrespond to the frequeny intervals between the input partials. In the ase of a harmoni sound, the interval orresponding to the F usually dominates. Figure 5 illustrates the bandwise magnitude spetra of a trumpet sound after the within-band ompression and retifiation, DFT{ 1 2 γ (x (n) + x (n) )}. Note that here a logarithmi frequeny sale is used. The retified signal of Fig. 4 appears at the band with enter frequeny 2.7 khz. As an be seen, the retifiation maps the ontribution of higher-order partials to the position of the F and its few multiples in the spetra. Moreover, the degree to whih an individual overtone partial m is mapped to the position of the fundamental inreases along with m. This is beause the auditory filters beome wider at the higher enter frequenies and the partials therefore have more neighbours with whih to generate the differene frequenies (beating) in the amplitude envelope. This is nie, sine organizing the higher partials to their fundamental is very diffiult in polyphoni musi. The retifiation does this automatially, without the need to

4 4 Center frequeny (Hz) Magnitude Frequeny (Hz) Center frequeny (Hz) Magnitude Frequeny (Hz) Fig. 5. The upper panel shows ompressed and retified spetra at a few auditory hannels for a trumpet sound (F 185 Hz). The lower panel shows a summary spetrum whih was obtained by summing over the subbands. Fig. 6. The upper panel shows ompressed and retified spetra at a few auditory hannels for an amplitude-modulated noise signal (modulation frequeny 185 Hz). The lower panel shows a summary spetrum obtained by summing over subbands. resolve individual higher-order partials. An auditory model allows simulating the pith pereption for a wide range of signals. For example, let us onsider amplitude-modulated white noise. It is known from psyhoaoustis that suh a signal will ause a pith perept orresponding to the modulation frequeny. Figure 6 shows the bandwise magnitude spetra of a white noise signal whih was amplitude-modulated with the funtion (1 + os(ωn)), where ω orresponded to 185 Hz. As an be seen, the spetrum is noisy at lower subbands, but at higher bands, the spetrum of the amplitude envelope (as generated by the HWR) shows a lear peak at 185 Hz, whih is also visible in the summary spetrum in the lower panel. Although this partiular ase is trivial to reprodue with other methods too, auditory models by definition simulate hearing for a large variety of signals [38], [45]. The spetral omponents generated around 2f were not found useful, sine these are not guaranteed to math the harmoni series of the sound, due to non-ideal harmoniity. On the ontrary, lowpass filtering the retified signal so as to rejet the harmoni distortion around twie the enter frequeny (here alled distortion spetrum ) was found to improve the F analysis. A diffiulty in doing this, however, is that the passband of the auditory filter overlaps the distortion spetrum at the lowest hannels. The problem an be solved by noting that HWR an be written as HWR(x) = 1 2 (x + x ). In order to ahieve a lean suppression of the distortion spetrum also at the lowest hannels, the signal x (n) is first full-wave retified as y (n) = x (n), the resulting signal is lowpass filtered using a utoff frequeny f, summed with the original signal x (n), and finally saled down by two. In addition to improving the F analysis, this allows the overall system to be implemented very effiiently as will be explained in the next setion. The signal at hannel after the ompression, retifiation, and lowpass filtering is denoted by y (n). C. Effiient omputation of frequeny-domain representation The signals y (n) are bloked into frames whih are then Fourier transformed. In more detail, eah frame is Hamming windowed, zero-padded to twie its length, and then the shorttime Fourier transform is applied. The resulting transform at hannel and time frame t is denoted by Y,t (k). The bandwise spetra are raised to power p and then summed to obtain a summary spetrum U t (k) = Y,t (k) p. (7) This intermediate data representation is used in all subsequent proessing. To understand why a frequeny-domain representation is omputed, let us onsider again as an example the summary ACF of Meddis s and Hewitt s model [38] whih was mentioned in the beginning of Set. II. The short-time ACF estimates within the subbands an be effiiently omputed as r,t (τ) = IDFT( Y,t (k) 2 ), where IDFT denotes the inverse Fourier transform and Y,t (k) is the short-time Fourier transform of y (n) in time frame t, zero-padded to twie its length before the transform. The summary ACF, in turn, an be omputed as r t (τ) = IDFT( R t (k)), where R t (k) = Y,t(k) 2. Note that the spetra Y,t (k) 2 an be summed before the IDFT beause the IDFT and summing are linear operations and their order an therefore be reversed. Based on the above disussion, we an see that the summary ACF representation of Meddis and Hewitt ould be alulated simply using p = 2 in (7) and by replaing the period detetion module in Fig. 1 with the inverse Fourier transform. 2 This is not what we will do, however, sine the intermediate representation U t (k) allows a lot of flexibility in designing the periodiity analysis mehanism, and we will utilize that to make the estimator more robust in polyphoni signals. 2 It should be noted, however, that the IHC model and some other details in [38] were different from those employed here.

5 5 It is lear that omputation of the Fourier transforms Y,t (k) at 7 subbands inurs a high omputational load. In the following, we desribe a tehnique whih redues this load roughly by fator 1. Let us denote one windowed and zero-padded time frame of x (n) by vetor x,t and the orresponding ompression saling fator by γ,t (see (5)). The ompressed, retified, and lowpass filtered signal y (n) an then be written as y,t = 1 2 γ,t(x,t + e x,t ), (8) where e is the impulse response of the distortion-suppression lowpass filter at band, and denotes onvolution. Using (8), the summary spetrum U t (k) an be written as U t (k) = [ ] 1 p DFT 2 γ,t(x,t + e x,t ) = 1 2 p γ p,t DFT(x,t ) + DFT(e x,t ) p. (9) In pratie, the spetra of x,t and e x,t are nonoverlapping in all exept few lowest bands and (9) an be approximated by U t (k) 1 2 p γ,t DFT (x,t ) p p γ,t DFT (e x,t ) p. (1) The benefit of this form is that the first term on the righthand side of (1) an be written as 1 2 p γ,t DFT (x,t ) p ζ Γ t (k)x t (k) p, (11) where X t (k) is the spetrum of the wideband input signal x(n) in frame t, ζ is a normalizing onstant, and Γ t (k) is a frequeny response obtained by linearly interpolating between the values γ,t defined at the enter frequenies f. This approximation is valid between the lowest and the highest subband enter frequeny, provided that the enter frequenies and bandwidths obey (1) (2). It follows that bandwise Fourier transforms need to be omputed only for the signals e x,t whih represent the bandwise amplitude envelopes. As the bandwidth of these signals is narrow, the signals e x,t are deimated down to the sampling rate Hz before omputing the DFTs. This means signifiant omputational savings sine an analysis frame of 248 samples at 441 Hz sampling rate, for example, shrinks to 256 samples at Hz sampling rate. After the deimation and the DFT, the alulated bandwise spetra are substituted to the seond term on the right-hand side of (1). The first term is obtained from (11). This tehnique signifiantly improves the omputational effiiently of the auditory model. D. Periodiity analysis As mentioned in the previous setion, U t (k) an be used to ompute the summary ACF by using p = 2 in (7) and simply inverse Fourier transforming U t (k) in eah frame: r t (τ) = IDFT(U t (k)). (12) Instead of (12), we will use a periodiity analysis method whih improves the robustness in polyphoni signals and is able to handle the wide range of pith values enountered in musi. In the proposed method, the saliene, or strength, of a period andidate is alulated as a weighted sum of the amplitudes of the harmoni partials of the orresponding F. More exatly, the saliene s t (τ) of a fundamental period andidate τ in frame t is alulated as s t (τ) = M m=1 w(τ,m) max k κ τ,m U t (k), (13) where m is the partial index and the funtion w(τ,m) determines the weight of partial m of period τ in the sum (the weights will be explained later). The set κ τ,m onsists of a range of frequeny bins in the viinity of the m:th overtone partial of F andidate f s /τ, where f s denotes the sampling rate. More exatly, κ τ,m = [ mk/(τ + τ/2),..., mk/(τ τ/2) ], (14) where denotes rounding to the nearest integer and τ denotes spaing between suessive period andidates τ. In the onventional ACF, τ = 1, that is, the spaing between fundamental period andidates τ equals the sampling interval. Later in this setion we will desribe an algorithm whih allows a very dense sampling of τ (small τ). This has the onsequene that all the sets κ τ,m in (13) ontain exatly one frequeny bin, in whih ase the non-linear maximization operation vanishes and s(τ) beomes a linear funtion of U t (k), making it analytially more tratable. The basi idea of (13) is intuitively appealing sine the Fourier theorem states that a periodi signal an be represented with spetral omponents at integer multiples of the inverse of the period. Indeed, formulas and priniples resembling (13) have been used for F estimation by a number of authors, under different names and in different variants although these have used the DFT spetrum instead of an auditorily motivated representation. Already in 196s and 7s, Shroeder introdued the frequeny histogram and Noll the harmoni sum spetrum (see [46, p.414]). Parsons [47] and de Cheveigné [24] disuss harmoni seletion methods, and more reently, Walmsley [48] uses the name harmoni transform for a similar tehnique. In the time domain, these tehniques an be implemented using a bank of omb filters, where eah filter has its harateristi feedbak delay τ and the energy at the output of the filter defines the saliene. In the auditory modeling literature, Cariani [49] proposed to use omb filters to separate onurrent vowels with different Fs. Also, the strobed temporal integration mehanism of Patterson [5, p.186] is losely related. Equation (13) and other omb-filter like solutions have two advantages ompared to the ACF. First, it is lear that (13) omputes the saliene of the period τ using only spetral omponents that are related to the period in question. This improves the robustness in polyphoni signals, sine the spetral omponents between the partials have no effet on s t (τ), whih improves the signal-to-noise ratio (SNR) of the

6 6 estimation. Seondly, it is very diffiult to ahieve a wide pith range using the ACF. This is beause any signal ontaining signifiant low frequeny omponents shows high orrelation for short lags (high frequenies). In polyphoni signals, the ACF is not robust above about 6 Hz: it is not able to handle the so-alled spetral pith. 3 The proposed saliene funtion (13) behaves robustly for a pith range of at least 4 Hz 2.1 khz and has no theoretial upper limit. The weights w(τ,m) determine the mapping from U t (k) to s t (τ). These have been studied in [32], where the following parametri form was found: w(τ,m) = f s/τ + ǫ 1 mf s /τ + ǫ 2. (15) Note that f s /τ is the F value orresponding to τ and that (15) redues to 1/m if the moderation terms ǫ 1 and ǫ 2 are omitted. The terms ǫ 1 2 Hz and ǫ 2 32 Hz are important for low-frequeny partials and for low Fs. The sum in (13) an be limited to M = 2 terms, sine weights beyond that are relatively small. As explained in Fig. 5, the higher partials are mapped to the position of the fundamental and its few multiples due to the retifiation at subbands, and as a onsequene, the entire harmoni series of a sound ontributes to the saliene. The form of (15) allows fast omputation of the salienes s t (τ) as follows. First, U t (k) is filtered using only the denominator of (15), replaing the numerator with unity. This an be done sine the denominator depends only on the frequeny of the partial and not on the period. Then s t(τ) is omputed using (13), but omitting the weights w(τ, m). Finally, eah period s t(τ) is weighted by the numerator of (15). It remains to hoose the value of p in (7). We tested two values, p = 1 (magnitude spetrum) and p = 2 (power spetrum), optimizing the parameters ǫ 1 and ǫ 2 in (15) in both ases and monitoring the resulting saliene funtions. The value p = 1 led onsistently to more reliable analysis results and was therefore hosen. Varying p is losely related to the generalized ACF [52], defined as r gen (τ) = IDFT( DFT(x) p ), (16) where x denotes the signal under analysis. The onventional ACF is obtained with p = 2. As disussed by Tolonen and Karjalainen in [27], hoosing a proper value for p improves the reliability and noise robustness of the F analysis. They suggest using the value.67. E. Iterative estimation and anellation The global maximum of the funtion s t (τ) in frame t is a robust indiator of one of the orret Fs in polyphoni signals. However, the seond or third-highest peak is often due to the same sound and loated at τ that is half or twie the position of the highest peak. Therefore we employ an iterative tehnique where eah deteted sound is anelled from the mixture before deiding the next F. A similar idea has been previously utilized for example in [24], [33], and [15]. 3 To the author s knowledge, the best solution so far for normalizing out this problem is the YIN algorithm by de Cheveigné and Kawahara [51] Algorithm 1: Fast searh of the maximum of s(τ) Q 1 τ low (1) τ min τ up (1) τ max q best 1 while τ up (q best ) τ low (q best ) > τ pre do # Split the best blok and ompute new limits Q Q + 1 τ low (Q) (τ low (q best ) + τ up (q best ))/2 τ up (Q) τ up (q best ) τ up (q best ) τ low (Q) # Compute new salienes for the two blok-halves for q {q best,q} do Calulate s max (q) using Equations (13)-(14) with w(τ,m) = fs/τ low(q)+ǫ 1 mf s/τ up(q)+ǫ 2 τ = (τ low (q) + τ up (q))/2 τ = τ up (q) τ low (q) end # Searh again the best blok q best arg max q [1,Q] s max (q) end Return ˆτ = (τ low (q best ) + τ up (q best ))/2 s(ˆτ) = s max (q best ) Let us first look at an effiient way of finding the maximum of s(τ). Here we omit time indies for simpliity. Somewhat surprisingly, the global maximum of s(τ) and the orresponding value of τ an be found with a fast algorithm that does not require evaluating s(τ) for all τ. This is another motivation for the iterative estimation and anellation approah where only the maximum of s(τ) is needed at eah iteration. Let us denote the minimum and maximum fundamental period of interest by τ min and τ max, respetively, and the required preision of sampling τ by τ pre. A fast searh of the maximum of s(τ) an be implemented by repeatedly splitting the range [τ min,τ max ] into smaller bloks, omputing an upper bound for the saliene within eah blok q, s max (q), and ontinuing by splitting the blok with the highest s max (q). Let us denote the number of bloks by Q and the upper and lower limits of blok q by τ low (q) and τ up (q), respetively. Index of the highest-saliene blok is denoted by q best. The algorithm starts with only one blok with upper and lower limits at τ min and τ max, and then repeatedly splits the best blok into two halves, as detailed in Algorithm 1. 4 As a result, it gives the maximum of s(τ) and the orresponding value of τ. On lines of the algorithm, in order to obtain an upper bound for the saliene s(τ) within range [τ low (q),τ up (q)], Eq. (13) is evaluated using the given values for w(τ,m), τ, and τ. Splitting a blok later on an only derease the value of s max (q) when omputed for the new blok-halves. Note that the best blok has to be re-sought after eah splitting in order to quarantee onvergene to the global maximum. In addition to being fast to ompute, Algorithm 1 allows 4 In pratie, it is even more effiient to start with [(τ max τ min )/τ pre] 1/2 bloks beause this narrows the ranges κ τ,m in (14).

7 7 TABLE I SUMMARY OF THE PARAMETERS OF THE PROPOSED METHOD Auditory filterbank 7 subbands between 65Hz and 52Hz Level ompression in (5) ν =.33 Moderation terms in (15) ǫ 1 = 5Hz/2Hz (46ms/93ms frame), ǫ 2 = 32Hz Spetrum power in (7) p = 1 Canellation weight d = 1 Polyphony estim. in (17) γ =.66 searhing the maximum of s(τ) with a very high auray, that is, with a high preision of the found period ˆτ. The iterative estimation and anellation goes as follows: 1) A residual spetrum U R (k) is initialized to equal U t (k), and a spetrum of deteted sounds U D (k) to zero. 2) A fundamental period ˆτ is estimated using U R (k) and Algorithm 1. The maximum of s(τ) determines ˆτ. 3) Harmoni partials of ˆτ are loated in U R (k) at bins mk/τ. The magnitude spetrum of the Hamming window is translated to these frequenies, weighted by w(ˆτ,m)u R ( mk/τ ), and added to U D (k). 4) The residual spetrum is realulated as U R (k) max(,u(k) du D (k)), where d 1 ontrols the amount of the subtration. 5) If there are sounds remaining in U R (k), return to Step 2. Note that the purpose of the anellation is ultimately to suppress harmonis and subharmonis of ˆτ in s(τ). This should be done in suh a way that the residual is not orrupted too muh to detet the remaining sounds at the oming iterations. These onfliting requirements are effetively met by weighting the partials of a deteted sound by w(τ,m) in Step 3 before adding them to U D (k). In pratie this means that the higher partials are not entirely anelled from the mixture sine w(τ,m) 1/m. When the number of sounds in the mixture is not given, it has to be estimated. This task, polyphony estimation, is aomplished by stopping the iteration when a newly-deteted sound ˆτ j at iteration j no longer inreases the quantity j i=1 S(j) = s(ˆτ i) j γ, (17) where γ =.66 was found empirially. Note that S(j) would be monotonially dereasing for γ = 1 (average of s(ˆτ i ):s) and monotonially inreasing for γ = (sum). The value of j maximizing (17) is taken as the estimated polyphony ˆP. Table I summarizes the parameters of the proposed method. III. RESULTS Simulation experiments were arried out to evaluate the auray of the proposed method in analyzing polyphoni musi and multiple-speaker speeh signals. The results are ompared with two referene methods [27] and [33], whih have been shown to be quite aurate and for whih reliable implementations were available. Also, we disuss alternative onfigurations of the proposed system where either (a) the auditory model is replaed with a DFT-based analysis frontend or (b) the iterative estimation and anellation mehanism is replaed with a joint estimator. A. Referene methods The first referene method, denoted by TK, has been proposed by Tolonen and Karjalainen in [27]. The authors used it to analyze mixtures of musi and speeh sounds. The method is motivated by an auditory model but divides an input signal into two hannels only, below and above 1 khz. An implementation was arefully prepared based on the referene, and the original ode by the authors was used in the warped linear predition part of the algorithm. The seond referene method, denoted by AK, was proposed by the present author in [33] and is based on spetral tehniques. The method was originally designed for polyphoni musi transription. Two alternative onfigurations of the proposed method are used in the evaluations in order to investigate the importane and possible drawbaks of the desribed tehniques. The first onfiguration, denoted by alt-dft, allows us to study the role of the auditory model. It is otherwise idential to the proposed method but does not apply half-wave retifiation at the subbands (see Set. II-B). As a result, the auditory filterbank does not need to be alulated at all, but U t (k) is obtained from (11), where the ompression oeffiients γ,t were omputed from the Fourier spetrum. All parameters of the system were separately optimized for this onfiguration. Another onfiguration, denoted by alt-joint, replaes the iterative estimation and anellation with an algorithm where all Fs are estimated jointly. This allows us to investigate how the iterative searh strategy affets the results. The joint estimator has been desribed in [32] and is not detailed here. B. Results for musi signals Test ases for musial signal analysis were obtained by mixing reorded samples from musial instruments. The aousti material onsisted of samples from the MGill University Master Samples olletion, the University of Iowa website, IRCAM Studio Online, and of independent reordings for the aousti guitar. There were altogether 2842 samples from 32 musial instruments, omprising brass and reed instruments, strings, flutes, the piano, the guitar, and mallet perussions. Semirandom sound mixtures were generated by first allotting an instrument and then a random note from its playing range, restriting the pith between 4 Hz and 2.1 khz when 93 ms analysis frame was used and between 65 Hz and 2.1 khz when 46 ms frame was used. This was repeated to get the desired number of sounds whih were mixed with equal meansquare levels. Varying the relative levels would make the task even harder, but this was not tested. One thousand test ases were generated for mixtures of one, two, four, and six sounds. One analysis frame immediately after the onset of the sounds was given to the multiple-f estimators. The onset of a sound was defined to be at the time where the waveform reahed 1/3 of its maximum value over the beginning 2ms. For the referene method TK, the test samples were limited below 53 Hz in pith (2.1 khz for the other methods), beause the auray of the method degrades rapidly beyong that. This seems to be due to the limitations of ACF for high Fs as disussed in Set. II-D.

8 Multiple F estim. 46 ms frame a a b de Conurrent sounds Predominant F estim. 46 ms frame b de Conurrent sounds Multiple F estim. 93 ms frame ab de a Conurrent sounds Predominant F estim. 93 ms frame b de Conurrent sounds F (otaves from passband edge) F (otaves from passband edge) F (otaves from passband edge) F (otaves from passband edge) Fig. 7. Multiple-F estimation (top) and predominant-f estimation (bottom) results in 46 ms and 93 ms analysis frames. The number of onurrent sounds varied from 1 to 6. Reading left to right, eah stak of six thin bars orresponds to the error rates of (a) proposed method, (b) referene TK, () referene AK, (d) onfiguration alt-dft, (e) onfiguration alt-joint. Fig. 8. Error rates for highpass filtered (top) and bandpass filtered signals (bottom). The left panels show results for isolated sounds and right panels for two-sound ombinations. The errors are shown as a funtion of the F, expressed in relation to the passband s lower edge. The solid line shows results for the proposed method and dashed line for the onfiguration alt-dft. Figure 7 shows F estimation results of the proposed and the referene methods in 46-ms and 93-ms analysis frames. Here the number of Fs to extrat, the polyphony, was given as a side-information to the estimators: we will evaluate the polyphony estimation separately. Two different error rates are shown. Multiple-F estimation rates (blak bars) were omputed as the perentage of all Fs that were not orretly deteted in the input signals. In predominant-f estimation (white bars), only one F in the mixture was being estimated and it was defined to be orret if it mathed the F of any of the omponent sounds. A orret F estimate was defined to deviate less than 3% from the referene F, making it round to a orret musial note. As an be seen, the proposed method outperforms the referene methods TK and AK learly in all polyphonies. Interestingly, the onfiguration alt-dft performs almost equally well in these lean, wideband signals. This would indiate that musi signals that ontain no drums an be proessed quite well without resorting to the use of an auditory model. This is beause most of the energy of the musial sounds is at their low harmonis, for whih the bandwise non-linearity (retifiation) is less important from the F analysis viewpoint. Conerning the iterative searh proedure, in turn, the onfiguration alt-joint does not perform better in multiple- F estimation despite of being onsiderably more omplex omputationally (see [32]). In predominant-f estimation, the joint estimator is better in hoosing the most reliable among the estimates that it has. Figure 8 ompares the robustness of the proposed method and the onfiguration alt-dft, when only a part of the entire spetrum an be used for F estimation. This an be the situation for example when a noise soure (suh as drums) oupies the other bands. The upper panels show error rates for a highpass-filtered signals. Four utoff frequenies, 25 Hz, 5 Hz, 1 khz, and 2 khz, were applied, and the results are averaged over these. The error rates are shown as a funtion of Fs, whih vary from 5.5 otaves below the utoff to 2.5 otaves above the utoff frequeny. The upper left panel shows results for isolated sounds and the upper right panel for two-sound ombinations. The proposed method is signifiantly more robust than the alt-dft onfiguration: for monophoni sounds, F estimation an be performed in about 9% of ases even when only partials 4 otaves above the fundamental are present. In brief, the auditory model based method is learly better in utilizing the higher-order overtones of a harmoni sound. This is due to the retifiation applied at subbands as explained around Fig. 5. In two sound ombinations, the robustness differene between the methods is still lear, although often the estimation is onfused by the other sound, espeially if it has many strong partials at the passband. The lower panels of Fig. 8 show F estimation results when only one-otave band of the signal is used. The lower boundary of the band was loated at the above-mentioned four positions, and the results are averaged over these. F values at the x-axis are expressed in relation to the lower edge of the band. As expeted, when the fundamental partial of the sound is within the passband (F between otaves and 1 in the figure), errors are seldom made. On the other hand, F estimation beyond the band is hopeless sine all the partials are filtered out. Again, the auditory model based method is signifiantly more robust than the alt-dft onfiguration. Figure 9 shows F estimation results in varying levels of wideband (5Hz 1kHz) pink noise. As disussed in [33], this noise type is the most disturbing for F estimation, as

9 ms analysis frame SNR 1 db 5 db db Conurrent sounds ms analysis frame SNR 1 db 5 db db Conurrent sounds Fig. 9. F estimation results in varying levels of wideband pink noise. The left and right panels shows error rates in 46ms and 93ms analysis frames, respetively. Count * Polyphony Count * Polyphony Count * Polyphony Count * Polyphony Fig. 1. The bars show histograms of polyphony estimates for the proposed method and a 93 ms analysis frame. The asterisks indiate the true polyphony (1, 2, 4, and 6, from left to right). ompared with same levels of white noise or drum sounds. The signal-to-noise ratio (SNR) is here defined as the ratio between the noise and the sum of the musial sounds in the analysis frame. Thus, the SNR from the viewpoint of an individual sound is muh worse in higher polyphonies. Figure 1 illustrates the results of estimating the number of onurrent sounds in a 93 ms analysis frame. The asterisk indiates true polyphony in eah panel, and the bars show a histogram of the estimates. The estimation an be done only approximately, and it seems that more than one analysis frame would be needed to do it more aurately. C. Results for speeh signals Speeh signals were obtained from the CMU ARCTIC database of Carnegie Mellon University [53]. We used a total of reorded utteranes from two male and two female US English speakers. Multiple-speaker speeh signals were simulated by mixing signals from the database. The mixed signals were allotted independently from the database, however ensuring that the same speaker did not our twie in a mixture. The root mean square levels of the signals were normalized over the entire utterane before mixing, and the mixture signals were trunated aording to the shortest utterane. Two hundred independently randomized test ases were generated for one, two, and four-speaker mixtures. Referene F urves were obtained by analyzing eah utterane in isolation using the Praat program [54]. The CMU ARCTIC database inludes pith-marks extrated from the eletroglottogram (EGG) signals using the CMU Sphinx program, but no hand orretions had been made on these, and espeially the voiing information was found very unreliable. TABLE II RESULTS FOR SINGLE-SPEAKER SIGNALS IN 32MS AND 64MS FRAMES Gross errors (%) Fine errors (ent) Method 32ms 64ms 32ms 64ms Proposed method Referene TK [27] alt-dft onfiguration Therefore the Praat output is used as the ground truth. To ensure that the Praat estimates were reasonably reliable, we ompared them with the pith-marks provided in the database, onsidering only segments where both soures laimed the signal to be voied. As a result, gross disrepanies (>2% differene in F) were found in 1.9% of the frames, and the standard deviation of the remaining fine errors was 27 ents (there are 12 ents in an otave). In all the results to be presented, the Fs were estimated independently in eah analysis frame, without attempting to trak a ontinuous pith urve over the utteranes. Table II shows results for single-speaker signals (isolated utteranes) using the proposed method, referene TK, and the alt-dft onfiguration. The referene method AK performs poorly in short analysis frames and is therefore not used here. Gross error rates were omputed as the perentage of time frames where the estimate differed more than 2% from the Praat referene. Fine errors were omputed for the remaining frames as the standard deviation of the differene between the estimate and the Praat referenes, measured in ents. Only voied frames were proessed: voiing detetion was not implemented. Both the auditory model based and the alt- DFT onfiguration perform well, within the limits of Praat s reliability. Figure 11 shows the gross error rates for multiple-speaker speeh signals. Estimating the number of speakers was not attempted, but the estimators were informed about the number of voied speakers in eah frame, and only this amount of Fs were extrated. 5 Here the referene method TK performs muh better than for musial sounds, although still being inferior to the proposed method. The auditory model based method and the alt-dft onfiguration perform approximately equally. Figure 12 shows results for highpass-filtered speeh signals, simulating the ase that the lower portions of the spetrum are missing (defetive audio reprodution) or orrupted by noise. The left panel shows results for individual utteranes and the right panel for two-speaker mixtures. The proposed auditory model based method degrades graefully as a funtion of the utoff frequeny, whereas the alt-dft onfiguration gets onfused (presumably by formants) as soon as the lowest and strongest partials are dropped. D. Disussion For pratial reasons, only two referene methods (TK and AK) ould be used above. Diret omparison with other methods is diffiult sine the experimental onditions vary 5 In priniple, the saliene values ould be used for voiing estimation (f. (17)), but further optimization would be required for speeh signals.

10 1 Gross errors (%) ms analysis frame Proposed Ref. TK alt DFT Number of speakers Gross errors (%) ms analysis frame Proposed Ref. TK alt DFT Number of speakers Fig. 11. Gross error rates for one, two, and four-speaker signals using the proposed method, the referene TK, and the alt-dft onfiguration. The left and right panels orrespond 32-ms and 64-ms analysis frames, respetively. Gross error rate (%) Number of speakers: Highpass filter utoff Gross error rate (%) Number of speakers: Highpass filter utoff Fig. 12. Error rates for highpass-filtered speeh signals. The solid line represents the proposed auditory model based method and the dashed line the alt-dft onfiguration. greatly. However, the method AK has been ompared to human performane in [33], and was found to perform very similarly with trained human musiians in musial interval and hord identifiation tests (using 19 ms frame for the method and 1 s for humans). The proposed method ahieves similar or better auray in twie shorter analysis frames. In multiple-speaker pith estimation, Wu, Wang, and Brown used the method TK as a referene in their reent work [25]. They too report lear improvement over the method TK in simulations. The experimental setup was quite different from here and prevents diret omparison of error rates. Singlespeaker pith estimation was reently studied by Cheveigné and Kawahara [51]. They report approximately 1% error rate for the best method. Here Praat was used to produe the ground truth pith traks, whih does not allow aurate error measurement in the single-speaker ase. However, the inauraies in the ground truth are quite harmless in multiplespeaker pith estimation where the error rates are still relatively large. IV. CONCLUSIONS An auditory model based F estimator was proposed for polyphoni musi and speeh signals. A series of tehniques was desribed to make the auditory model and the subsequent periodiity analysis omputationally effiient and therefore pratially appealing. In the simulations, the method outperformed learly the two referene methods TK and AK. Espeially, both theoretial and experimental evidene was presented whih shows that an auditory model based F estimator is good at utilizing the higher-order overtones of a harmoni sound. This is due to the nonlinearity applied at the subbands of a peripheral hearing model, whih is diffiult to reprodue in pure time or frequeny domain methods. The improved proessing of higher harmonis is partiularly important in situations where the entire wideband signal is not available due to bad audio reprodution, or noise soures oupying parts of the audible spetrum. When analyzing lean, wideband signals, the auditory model did not have a lear advantage over the alt-dft onfiguration, although it still performed very well. This seems to be due to the fat that most of the energy of musi and speeh sound is at the lowest partials, for whih the bandwise nonlinearity is less important. Robustness of the proposed pith estimator for narrowband signals an be utilized for example in proessing noiseontaminated speeh signals. Time-frequeny regions representing the lean speeh an be loated by proessing the signal within subbands and by using the bandwise pith values and their salienes as ues for speeh separation. The properties of the proposed estimator were not fully exploited in the present work, whih foused on proessing individual time frames only. For example, the SNRs of different bands ould be estimated and the subbands weighted aordingly in (7). In the future work, more effort is put on using the longerterm ontext in assoiating ertain time-frequeny regions to ertain instruments/speakers or noise soures. The proposed method has also been used for feature extration in transribing realisti musial reordings. The work has been reported in [55] and audio examples an be found at V. APPENDIX A This appendix desribes the design of the two resonators (3) and (4) that are used to approximate the gammatone filter. The gammatone filter is defined by its impulse response as g(t) = at η 1 e 2πBt os(2πf t + ϕ), (18) where f is the enter frequeny of the filter, a = (2πB) η /Γ(η) ensures unity response at the enter frequeny, and Γ( ) denotes the gamma funtion. Choosing η = 4 leads to a shape of the power response that mathes best with that found in humans. The parameter B = 1.19b ontrols the ERB bandwidth b of the filter [56, p.256]. The phase parameter ϕ has no importane and a zero value an be used. We did not try to simulate any partiular value. In the following, we desribe the alulation of parameters A, θ 1,2, and ρ 1,2 in (3) (4) assuming that the number J of resonator setions to be applied in a asase is given. Choosing J and the optimal ombination of the two resonator types an be done by trial and error sine the number of alternatives is small. It was found that a asade of four resonators, two of eah type, leads to the best result. Let us first onsider the enter frequeny of Resonator 1. Power response of (3), after some straightforward algebra, an be written as H 1 (e iω ) 2 ρ 2 = 1(1 os 2 (ω)) a 1 + a 2 os(ω) + A 2 os 2 (ω), (19)