RETREIVING PITCH OF THE SINGING VOICE IN POLYPHONIC AUDIO. Saurabh Kumar Shandilya and Preeti Rao

Transcription

1 RETREIVING PITCH OF THE SINGING VOICE IN POLYPHONIC AUDIO Saurabh Kumar Shandilya and Preeti Rao Department of Eletrial Engineering, Indian Institute of Tehnology, Bombay Powai, Mumbai ABSTRACT This paper explores the extration of melodi pith ontour from the polyphoni soundtrak of a song. The motivation for this work lies in developing automati tools for the melody-based indexing of the database in a musi retrieval system. The melody is assumed to be arried by the singer s voie aompanied mainly by perussive instruments. This senario is typial of a large lass of Indian movie songs. The hallenges raised by this appliation are presented. A pith detetion method based on a pereptual model is shown to be a promising approah to the traking of voie pith in the presene of strong perussive bakground. 1. INTRODUCTION The advent of Internet has made it feasible for people to share not only thoughts but also emotions and musi. The soul of musi is melody and that of melody is pith. So the importane of pith detetion in musi related software is signifiant. In this paper we address the problem of pith detetion assoiated with the singing voie in the ontext of the polyphoni reordings of Indian movie songs. In the field of speeh proessing, pith detetion algorithms have always evoked muh researh interest. Every new appliation demands a new approah. Our problem an be viewed as that of estimating the time-varying pith of a singing voie in the presene of orhestral bakground. This work is motivated by the requirements of query-by-humming based musi retrieval systems. One suh system under development at IITB [1] targets a database of Indian film songs. In the interest of building an indexed database, as well as enabling its ontinuous augmentation with new songs, it would be very valuable to have available a tool that automatially extrats the needed pith information from original polyphoni sound traks. The problem is espeially important for our target database sine, unlike Western musi orpora (e.g. [2]), aurate MIDI versions of Indian musi are very sant or non-existent. The real hallenge to pith detetion is due to polyphoni nature of the songs. Various instruments keep oming and disappearing at intervals along with the main singing voie. Computer sientists view this problem as that of CASA (Computational Audio Sene Analysis) [3,12,13] where they are primarily interested in identifying the various objets (instrument voies) that ontribute to the audio sene at any time. This step may be useful to solving the problem of estimating features of interest from speifi voies but is not essential [3,4,5]. Early works in the field were too speifi but later on the impetus shifted towards a general approah, whih an give results independent of type and number of instruments being played. While most of the work has been done under the name of multiple-pith-detetion, where the soures are assumed to be harmoni, our ase is not exatly the same, as we shall see soon. In this paper we test the appliation of a pereption model based pith detetion algorithm and investigate its advantages against onventional pith detetion algorithms. But first we present some interesting fats about the genre of Hindi

2 movie songs, whih help us in defining our problem better. 2. SOURCE BASED CLASSIFICATION At a oarse level, audio segments in a soundtrak an be lassified into three lasses: silene, orhestral bakground only, and bakground plus singing voie. The melody of the song is typially arried in the time-varying pith of the singing voie. We therefore fous on this lass, whih further an be haraterized by the partiular type of bakground instrument: perussive or nonperussive. Perussive sounds may have strong partials but are not stritly harmoni. From a study of songs in the database of our interest, we have found that segments ontaining singing voie almost always ontain purely perussive instruments in the bakground. These instruments ontribute to the rhythm but not to the melody of the musi. Therefore, the problem boils down to finding aurate pithontour of the voie in the presene of perussive sounds. Further, in the framework of Indian musi, the perussions belong to one of the two ategories [6]: membranous perussions oming from instruments with struk membranes e.g. drum, tabla, pakhawaj, dhol, nall, et., and non-membranous perussions oming from instruments having solid resonators e.g. jal-tarang, manjira, ghatam, himpta, ghungaru et. Perussive sounds belonging to the lass of membranous perussions are haraterized by sharp inrease in instantaneous energy followed by an exponential deay. These sounds are extremely dynami in nature and they disturb the harmoni struture of the singing voie when they our. In the frequeny domain these onsets are deteted by a smearing aross frequeny. Perussions of these kinds don t pose a big problem beause the effet dies down very quikly, and pith estimate may go wrong for only a small fration of the duration of the note. These deviations an be retified by median smoothing the pith values over note duration (assuming the notes are separated by some method). But the perussions of nonmembranous variety leave a trail of strong partials in some regions that deay only slowly. This an ause the pith estimate to go haywire over a signifiant portion of the note, espeially when the energy of the voie signal is not high relative to the bakground. 3. PITCH DETECTION ALGORITHMS A pith detetion algorithm works in three stages: (i) preproessing, whih may be linear or non-linear filtering; (ii) estimation of andidate pith-values from the pre-proessed signal; (iii) post-proessing to rejet / modify spurious pith values. Candidate pith values are generated by either pure time-domain methods or spetral domain methods. Autoorrelation funtion (ACF) based method is the most widely used time-domain method. A related algorithm in this is the Rabiner-Gold parallel proessing algorithm, whih extrats pith by running deision logi on several derived time-domain parameters [7]. Spetral domain methods look for harmoniity in the frequeny domain to find out pith; i.e. they look for partials in speifi spetral loations, or for the intervals between partials. The above methods do not expliitly take into aount any model of pith pereption by humans. Sine we an easily identify a tune even in the presene of bakground perussions, it is reasonable to expet that a study of the pith pereption mehanism an offer lues that would help in extrating the melody information that is buried in a perussive bakground. In 1991, Meddis and Hewitt proposed a theory for the subjetive pereption of pith [8]. This model is essentially based on deteting the dominating spetral interval by observing the beats between frequeny partials. The theory inorporates several distint funtional bloks of the hearing mehanism. The aousti signal entering the ear undergoes a filtering equivalent to outer/middle ear transformation. In the inner ear, the basilar membrane ats as a bank of bandpass filters with bandwidth inreasing with enter frequeny. The output of eah filter represents the motion of basilar membrane and the next stage involves onversion of this motion to neural signals. The model hypothesizes that the probability of spike ourrene is proportional to the permeability of the hair ell membrane.

3 Permeability in turn is related to the instantaneous amplitude of the filtered signal. At the next stage, for eah filtered hannel the histogram of time intervals among spikes (not just suessive spikes) is generated. A running ACF, whih is a linear autoorrelation alulated over a moving window entered at eah sample, is one way to simulate this. Finally these autoorrelation funtions along different hannels are summed up to give summary autoorrelation funtion, whih is examined for peaks to detet the pith period. So the important onlusion here is that the proess of pith pereption involves proessing in individual bands. These bands are also represented best using psyhoaoustially derived filters. 4. IMPLEMENTATION OF THE M-H MODEL ALGORITHM Borrowing the important keys from the above-desribed theory we have implemented an algorithm, whih imitates the behavior of human ear in a simplisti way. The input audio signal in 16 bit PCM format, sampled at the rate of 44.1 khz is passed through an auditory filter-bank as obtained from VOICEBOX [9]. We have used 27 standard gammatone filters with enter frequenies ranging from 123 Hz to khz or 4 to 30 on the equivalent retangular bandwidth sale (a slight modifiation of the ritial band sale) [10]. In eah frequeny band the amplitude envelope of the filtered signal is obtained. This an be ahieved by using standard demodulation tehnique. First the signal in eah hannel is half-wave retified. Then the half-wave retified signal is low pass filtered to allow frequenies up to 700 Hz, assuming that this is the maximum expeted pith frequeny. Envelope alulation by halfwave retifiation is onsistent with the Meddis-Hewitt ear model, aording to whih probability of spike generation is proportional to instantaneous amplitude, and low-pass filtering is justified by the arguments relating to refratory periods. In eah band an ACF is alulated. Keeping in view the range of Hz in whih we are expeting the pith to lie in, we have used a 20 ms frame and frame-shift of 50%. We again had a hoie here between linear ACF and irular ACF and we have used both the approahes. An unbiased ACF does not assume the signal to be zero outside the window and thus steady and sharp hanges in amplitude beause of onset of notes would ause farther peaks to inrease in magnitude leading to too low error. A biased estimate of ACF (referred as linear ACF now onwards) is relatively free from this hazard under varying amplitude onditions, whih are ommon in singing. A linear ACF assumes the signal to be zero outside the frame while irular ACF assumes the pattern to be repeating outside the window of size N. N τ 1 i= 0 af ( k, τ ) = y( k + i) * y( k + i + τ ) (1) af ir ( k, τ) = N 1 i= 0 y ( k + i)* y ( k + mod( i + τ, N)) (2) Where k and τ are position of window and orrelation lag respetively and y is the input signal. The final task is more analytial in nature as we have 27 ACF arrays eah of them ontaining information about frequeny ontent in eah band. Numerous onlusions an be drawn regarding pith-values, harmoniity, note onset et. Constraining ourselves to pith-detetion we build a summary autoorrelation funtion whih is a summation of various ACFs (aross the hannels) raised to power p. We are urrently using a power of 2. ACFsq = ( ACF ) summary p (3) This auses the underlying pith to reinfore peaks at orresponding loations. 5. EVALUATION We have ompared the auray of the pith-estimates obtained by this method with that obtained by the traditional autoorrelation based method. Traditional method involved alulation of ACF using Eqn.1, with window size of 20ms and shift of 50%. The position of the most prominent seondary peak is taken as the estimate of pith period.

4 Evaluation of a pith detetor an be done by omparing with manual estimates (observe waveform, spetrum) or by synthesizing a sound and listening. The typial pith errors in the ontext of this paper are gross (not fine) and therefore an be easily deteted by listening. A omparison to the original song was performed by listening to the resynthesized melody using estimated pith values and loal energy estimates (after smoothing out the narrow bursts of energy due to perussive strikes). The resynthesis program generates tones of fundamental frequeny equal to the speified pith values (or its integral multiple, whih is equivalent to playing same note on higher otaves). To test the algorithm we have used a number of audio segments, whih had perussive instruments ranging from drum (membranous perussion) to himpta (nonmembranous perussion). In order to provide insight into the problem and the different approahes, we disuss in detail one example taken from the film song jai jai shiv Shankar kaata lage naa kankar from an old Hindi movie whih had himpta and manjira in the bakground. Fig. 1 shows a spetrogram of the seond phrase with manually marked syllablenotes. Fig. 2 shows the energy ontour and pith ontours as obtained from the two different pith detetion methods. The raw autoorrelation based method failed to detet the orret pith at ertain instants e.g. during note /kan/. Figure 3 shows how the wrong peak was piked up. This has happened at a plae where the syllable (the nasal kan ) was weak and perussion was strong. Low energy of the sung syllable is evident from the low-levels of energy ontour near ms (See Fig. 2(a)). So the interferene from bakground affets the low-energy voie regions strongly. The gist of this is that the relative strength of voie and perussion energy plays a ruial role. The M-H method appears to be more robust, suessfully retrieving the pereptually orret pith. Also in portions where either a note is starting/ending or a perussion instrument has just been struk, the M-H method fares muh better than the traditional ACF pith detetor. Audio lips and test results are available at [11]. The failure of ACF based approah an be explained by the fat that presene of strong partials from the perussion perturbs the voie pith peak pattern due to additional temporal modulation. Figure 1: Spetral view of audio fragment kata lage na kankar with himpta and manjira in bakground. Arrows indiate the points where the traditional method fails. Figure 2: (a) Short time energy variation for song of Fig. 1; (b) pith ontour obtained by raw ACF based method () pith ontour obtained by M-H method. On the other hand, the M-H method does independent proessing in separate bands, so orruption of harmoni ontent by a few strong partials is redued. In any given frequeny band, the presene of two or more harmonis leads to beats of frequeny equal to the differene of two harmonis, whih is the fundamental frequeny itself. Envelope detetion followed by band-pass filtering is a step towards obtaining this beat frequeny. A good harmoni struture as is found in singing voie strengthens the peak in the ACFs at a lag equal to pith period, when individual ACFs

5 are ombined. Speial are has to be taken when the fundamental frequeny is really high as in the ase of female singers. It may happen that one band may omprise only one partial and no beats are generated, hene envelope detetion followed by low pass filtering may be misleading. In suh ases i.e. when fundamental frequeny turns out to be higher than 500 Hz, the value should be ross heked with the summary ACF for the filtered signals on whih envelope detetion has not been performed and the lower of the two values should be hosen. Figure 3: (a) Raw irular ACF of a frame of note /kan/ over non-redundant portions only (Cirles indiate peak used for omputation of pith) (b) Summary ACF for filtered-only signals () Summary ACF for envelope of filtered signals. 6. CONCLUSIONS We have investigated the appliation of a pereption based pith-detetion algorithm to the problem of melody retrieval from the polyphoni audio of Indian film songs. It was shown that the traditional ACF based method ould fail, esp. in the presene of nonmembranous perussive sounds. The Meddis- Hewitt pith model based method seems to be promising for pith detetion espeially when the energy related with the singing voie is low. Its higher robustness may be attributed to the independent bandwise proessing for ACF peaks. Work is in progress on refining this algorithm and testing its appliability to a variety of audio samples. The inorporation of prior knowledge also needs to be onsidered due to its potential in improving performane further by enabling the tuning of algorithm parameters. If the nature of perussion (e.g. spetral properties) is known, alulation of summary ACF an be restrited to unorrupted bands only. This would suggest a better way of ombining the ACFs. Development of deision logi for presene of singing voie will take us yet another step ahead in the diretion of obtaining note boundaries and hene noteontours. REFERENCES 1) M. A. Raju, B. Sundaram and P. Rao, TANSEN: A query-by-humming based musi retrieval system, Pro. NCC ) Tuneserver: 3) M. Goto and S. Hayamizu, A Real-time Musi Desription System: Deteting Melody and Bass Lines in Audio Signals, Workshop on CASA, pp , Aug-99 4) A. Klapuri, Wide-band pith estimation for natural sound soures with inharmoniities, 106th Audio Engineering Soiety Convention, Munhen, Germany, ) A. Klapuri, T. Virtanen, J. M. Holm, Robust multipith estimation for the analysis and manipulation of polyphoni musi signals, Pro. COST-G6 Conferene on Digital Audio Effets, Italy, ) Indian instruments by David Courtney musi/instruments.html 7) B. Gold and L. Rabiner, Parallel proessing tehniques for estimating [pith periods of speeh in time domain, JASA, Vol. 46(2), ) R. Meddis and M. J. Hewitt, Virtual pith and phase sensitivity of a omputer model of the auditory periphery. I: Pith identifiation, JASA, Vol. 89, June ) Voiebox home page, Deptt. Of Eletrial and Eletronis Engg., Imperial College, London. iebox.html, August ) Global Index, J.O.Smith, CCRMA, Stanford University, 11) 12) A. Bregman, Auditory Sene Analysis, MIT Press, ) D.F. Rosenthal and H.G. Okunu (Eds.), Computational Auditory Sene Analysis, Lawrene Erlbaum Asso., 1998.