Singing Voice Detection in North Indian Classical Music

Transcription

1 Singing Voice Detection in Noth Indian Classical Music Vishweshwaa Rao, S. Ramakishnan and Peeti Rao Depatment of Electical Engineeing Indian Institute of Technology Bombay, Mumbai 476, India {vishu, amakishnan, Abstact Singing voice detection is essential fo content-based applications such as those involving melody extaction and singe identification. This aticle is concened with the accuate detection of singing voice phases in noth Indian classical vocal music. The component sound souces in such music fit into a typical famewok (voice, hythm and done). We have used this a-pioi knowledge to enhance the voice in the pesence of accompaniment. A Gaussian Mixtue Model (GMM) classifie is evaluated using fame-level featue vectos extacted fom a epesentative data set. A theshold based method, applied to suitable audio featues, is then used to automatically divide the audio signals into vaiable-length homogenous segments i.e. vocal o non-vocal. Segment-level classification decisions ae made by gouping fame-level decisions ove individual segments. The pefomance of the classifie is evaluated in tems of the classification accuacies of the vocal and non-vocal fames. I. INTRODUCTION The melody of a musical piece is a vey impotant attibute fom a Music Infomation Retieval (MIR) pespective. The melody may be defined as the tempoal tajectoy of the pitch of the pedominant melodic instument which, in note-based music, coesponds to the sequence of note pitches and duations. Melody extaction theefoe involves detecting the pesence of the melodic instument in the audio ecod, and tacking its pitch. In most Indian music, the pedominant melodic instument is the human singing voice. In classical vocal music, the voice is suppoted by instuments poviding the done (tanpua) and hythmic accompaniment (tabla). Pitch detection in the pesence of simultaneously playing instuments is a difficult poblem. Methods that tack the dominant pitch by applying smoothness constaints ove continuously sung phases achieve easonable accuacies [1]. The eliable detection of singing voice segments is necessay fo the effectiveness of such methods. It is also essential in systems that attempt to identify the singe/atist [] [3]. Singing voice detection can be viewed as an audio classification poblem whee featues that distinguish the vocal egions fom pue instumental egions ae fed to a classifie peviously tained on labeled data. In [4] a Multi-Laye Pecepton (MLP) is used to segment songs into vocal and non-vocal egions based on Peceptual Linea Pediction (PLP) featues. In [] featues such as enegy, aveage Zeo Cossing Rate (ZCR), hamonic coefficients and spectal flux ae computed at egula intevals and a theshold-based measue is used to detect the onset of singing. In [3] a combination of band-pass filteing and hamonicity detection ae used to extact vocal segments with limited success. [5] uses a Suppot Vecto Machine (SVM) classifie to detect the pesence of the singing voice using featues such as Linea Pediction Coefficients (LPC), LPC-deived Cepstum (LPCC), MFCC spectal powe, shot-tem enegy and ZCR. [6] uses MFCCs along with a GMM classifie. Additionally, fame-level likelihoods fo individual classes ae pooled ove automatically detected segments to aive at a segment label. In [7] tained multi-model HMMs and a-pioi musical knowledge ae used to sepaate vocal and non-vocal egions. In the pesent wok we investigate audio featues suitable fo use in a simple ule-based classifie to detect singing segments in Indian classical vocal music whee the singe is accompanied by the tanpua and tabla. The accompanying instuments exhibit cetain chaacteistics simila to the singing voice necessitating a econsideation of audio featues fo vocal/non-vocal classification fo Indian classical music. In the inteest of obtaining obust classification acoss a vaiety of audio signals unde this categoy, we attempt to find featues based on the study of known and obseved acoustic signal chaacteistics. In the next section, we discuss the signal chaacteistics of the singing voice and the accompaniment. A method to suppess the accompanying instumental backgound elative to the singing voice is then poposed. Fo classification we follow the famewok developed in [6]. We descibe the featues we have used as input into the classification module and povide fame-level classification esults. Futhe, audio featues that exhibit vaiation in value acoss vocalinstumental boundaies o vice vesa ae descibed and evaluated fo segment boundaies detection. The segments between the boundaies so obtained ae assumed to be homogenous in tems of vocal/non-vocal categoization. Two methods of deciding the final segment labels ae then descibed. Table 1. Statistics of the vocal and non-vocal segments fo the audio data set Avg. Total Segments Numbe Longest Shotest duation duation Vocal s.53 s 5.44 s s Non-vocal s.11 s 1.73 s s

2 A. Singing Voice II. SIGNAL CHARACTERISTICS That the ange of fundamental fequency (F) fo singing is geate than that fo speech is well known. Additionally the singing voice, in noth Indian classical singing, is eplete with lage pitch modulations, which seve an impotant aesthetic function within the melodic contou. These modulations ae futhe magnified in the hamonic content of the voice located in the highe egions of the spectum. Howeve, the locations of significant voice hamonics in the spectum ae usually esticted to below 5 khz. With espect to aticulation in singing, the vowel duations ae much longe than in speech. In Indian classical singing especially, the atio of vowel to consonant duation within a single sung phase is vey high, with the dominant vowel being /a/. The duation of sung phases also shows lage vaiation, fom as shot as 1 second to as long as 15 seconds. B. Done (Tanpua) The tanpua is an ovetone-ich stinged instument, usually having fou stings that ae epeatedly plucked in succession thoughout the music pefomance. Two of the stings ae tuned to the singe s tonic, one to an octave above this tonic and the last one is usually tuned to eithe the fouth o the fifth. Due to the slow decay of the plucked sound, a nea-unifom, continuous done is clealy audible in the backgound poviding the vocalist (and listenes) with a efeence point in the tonal famewok. The tanpua signal enegy is obseved to be spead ove a vey lage numbe of egulaly spaced patials thoughout the spectum up to about 1 khz. This leads to cetain fequency bands dominated entiely by tanpua patials, thus enhancing its loudness elative to the voice. C. Pecussion (Tabla) The tabla consists of a pai of dums, one lage base dum, the bayan, and a smalle teble dum, the dayan. Tabla pecussion consists of a vaiety of stokes, often played in apid succession, each labeled with a mnemonic. Two boad classes of stokes, in tems of acoustic chaacteistics, ae: 1. tonal stokes that decay slowly and have a nea-hamonic spectal stuctue (thus eliciting a pitch pecept) and. impulsive stokes that decay apidly and have a noisy spectal stuctue. The pitch pecept elicited by tonal tabla stokes falls within the pitch ange of the human singing voice. The acoustic chaacteistics of vaious tabla stokes wee studied fom Paag Chodia s database available at It was found that while all the impulsive stokes had simila acoustic chaacteistics, thee was a lage vaiability in those of the diffeent tonal stokes. Howeve, the significant hamonic content in the tonal stokes was only found below 1.5 khz. The local signal-to-intefeence atios (SIRs) aound the onsets of individual tabla stokes can be as low as -1 db. III. AUDIO DATA FOR EXPERIMENTS A typical noth Indian classical music pefomance stats at a low tempo whee only the voice and tanpua ae pesent. The tabla stokes, if pesent, duing this time ae spaced widely apat in time. As the pefomance pogesses the tabla stokes ae moe closely intespesed and the vocalist sings at a medium tempo. Towads the end of the pefomance the tabla playing becomes vey apid as does the speed of singing. With this in mind, the data selected fo the subsequent studies on pepocessing, classification and segmentation ae excepts fom the stat, middle and end of ecodings of seven diffeent noth Indian classical vocal pefomances spanning 3 minutes in total. Thee of the ecodings ae of female atists and fou ae of male atists. The ecodings ae compised of sounds geneated by a single vocalist, a pai of tablas and a tanpua and, in some cases, a seconday melodic instument (SMI). The statistics of the vocal and instumental segments acoss the entie data set is shown in Table 1. As we can see, the vocal segments compise nealy 75% of the data. IV. PRE-PROCESSING We see fom the pevious section that the accompanying instuments shae seveal tempoal and spectal popeties with the singing voice. In the vocal egions, the instumental backgound has a clea pesence and can dominate the voice duing softly sung egions. It is of inteest theefoe to look fo methods to enhance the singing voice when pesent to facilitate the eliable detection of vocal egions. The tanpua spectum is stongly hamonic and ovelaps in ange with the voice spectum, paticulaly below 5 khz, uling out the use of linea filteing. Most noth Indian classical music ecodings howeve have initial segments, of duation 4 seconds o moe, whee only the tanpua signal is pesent. This, along with the elative stationaity of the tanpua specta, diects us towads investigating a spectal subtaction-based tanpua suppession scheme. Spectal subtaction [8] is a well known technique fo noise suppession and has been used extensively in speech pocessing. As its name implies, it involves the subtaction of an aveage noise powe spectum, estimated duing non-speech egions, fom the powe spectum of the noisy signal. The enhanced signal is econstucted fom the modified magnitude spectum and oiginal phase spectum of the noisy signal. The assumptions made ae that the noise is additive and stationay to the degee that its spectum in the non-speech egions is simila to that duing speech. In the pesent context, the aveage noise (tanpua) magnitude spectum is estimated ove the initial 4 seconds of the audio tack. Such a long segment is used to aveage out the effects of plucking stings tuned to diffeent fundamental fequencies (Fs). This spectum is then subtacted fom all subsequent specta followed by half-wave ectification and finally signal econstuction using the ovelap-add (OLA) method. Fig. 1 shows the effect of spectal subtaction based tanpua suppession fo a typical noth Indian classical vocal segment.

3 The figue on the left shows the time domain wavefom and naow-band spectogam of a segment of music whee thee is only tanpua pesent fom to 5 sec and both tanpua and the singe s voice fom 5 to 14 sec. The figue on the ight shows the time domain wavefom and naowband spectogam of the econstucted signal. Spectal subtaction educes the noise floo, as is evident fom the attenuated wavefom in the econstucted signal (fom to 5 seconds in the figue on the ight). The tanpua spectum is now educed to a low level of esidual noise due to the patial subtaction of its hamonics. We see that the voice hamonics ae elatively unaffected. The tabla stokes due to thei highly non-stationay chaacteistics suvive the spectal subtaction. Since the majo fequency content of the tabla lies below about 15 Hz, the featues used fo classification ae chosen such that they emphasize the diffeence in the enegies of the voice and the tabla above 15 Hz. V. FRAME- LEVEL CLASSIFICATION Fo classification we make use of the Gaussian Mixtue Model (GMM) classifie. In this section we discuss the need fo subdividing vocal and non-vocal classes futhe, featues extacted and the evaluation of the classifie pefomance on the given data. A. Classes The objective of the classification stage is to label individual audio fames as vocal (V) o non-vocal (NV). Howeve, within each of these classes the vaiability in audio chaacteistics due to the vaying natue of the sound poduction mechanism (fo singing) and the diffeent sound souces (fo instuments) is consideable. Fo example, a stop and a vowel, both fall in the vocal class but thei audio chaacteistics ae vey diffeent. In ode to account fo this vaiability we futhe subdivide the V and NV classes into fou sub-classes each. Fig. 1. Time domain wavefom (above) and naowband spectogam (below) of a segment of a typical noth Indian classical ecoding (left) and the tanpua suppessed vesion (ight). The initial segment, fom to 5 sec, contains only tanpua The V class is divided into (1) vowels, () semivowels and nasals, (3) stops and (4) ficatives. The NV class is divided into (5) tanpua only, (6) tanpua and SMI, (7) tanpua and tabla, and (8) tanpua, tabla and SMI. The numbe of tokens (featue vectos) available pe class afte manual annotation of the dataset based on the above 8 classes is 4578, 387, 19, 57, 531, 447, 14 and 6996 espectively. B. Featues Based on knowledge of the acoustic chaacteistics of the undelying classes, seven featues ae extacted pe fame ( ms) of audio using 4 ms analysis window. Spectal oll-off is the fequency below which X% of the signal enegy is concentated [9]. 7% was found to povide good sepaation between tonal tabla stokes, tanpua and the voice. This featue is defined as k F k Fn X [ k] =.7 X [ k] (1) k = k k = k whee F is the oll-off fequency. Fn is the Nyquist fequency. kf, is the spectal bin numbe whose cente is neaest to F Hz. X[k] is the magnitude spectal value of the kth spectal bin fo the th fame. Lowe values of this featue ae expected fo the fames dominated by the tabla and highe values fo the fames dominated by tanpua. This was especially because the tabla has high enegy content in the lowe fequencies. Hamonic enegy (HE) is sum of the stength of individual hamonics, till 5 khz, given the pedominant pitch. The pedominant pitch is computed using the algoithm descibed in [1]. It is given by knf k = kf [ ] HE = X k () whee F is the fundamental fequency and NF is the lagest multiple of F below 5 khz. The spectal bin numbes (k) coesponding to the closest local maxima, within a 3% neighbouhood, of the expected hamonic location ae used in the computation. This featue is expected to have high value duing the voiced sounds and lowe values in the instumental egions. Sub-band enegy atio (ER) is the atio of the enegy of a two fequency bands in spectum. We have used of these featues, one giving the atio of the enegy of a band anging fom 5 khz to 8 khz to the enegy of a band anging fom khz to 1.5 khz and the othe giving the atio of the enegy of a band anging fom.5 khz to 5 khz to the enegy of a band anging fom khz to 1.5 khz. These ae given by k 8 k 5 X ( k ) X ( k ) k = k 5 k = k 5 E R 1 = ; E R = k1 5 k1 5 X ( k ) X ( k ) k = k k = k Lowe values of this featue ae expected duing fames whee the voice is pesent since the concentation of hamonic enegy fo the voice is below 5 khz. (3)

4 Sub-band flux is the spectal flux, as defined in [9], computed ove a fequency band anging fom 1.5 khz to 5 khz. It is defined as Flux = k5 X k X 1 k k = k15 ( ( ) ( ) ) (4) It is expected that this featue will geneally have a high value duing the sung segments, whose specta show significant inte-fame vaiation as compaed to the moe stationay puely instumental segments. Audio spectal flatness (ASF) [9] ASF of a spectal band b is defined as the atio of the geometic and the aithmetic means of the powe spectum coefficients within that band. It is given by A S F ( b ) = ( ih ( b ) il ( b ) + 1) ih ( b ) k = il ( b ) ih ( b ) X [ k ] ih ( b ) il ( b ) 1 k = il ( b ) ( + ) X [ k ] whee ih(b) and il(b) ae the highest and lowest fequencies of the spectal band b. It is a measue of deviation of the spectal fom fom that of a flat spectum. Flat specta coespond to noise o impulse-like signals. Thus high flatness indicates noisiness. Low flatness values geneally indicate the pesence of hamonic components. The band chosen fo classification was 1.5 khz 3 khz. Voice has a hamonic stuctue until 5 khz and hence gives a high value fo this featue. As mentioned befoe, the dominant hamonics of the tonal tabla stokes ae pesent below 1.5 khz. Hence this featue was computed ove the band anging fom khz. Aveage sub-band enegy (SE) is the aveage enegy of the spectal sub-band anging fom 1.5 khz to 5 khz pe bin. The Shannon Mutual Infomation values [1] fo each of the above featues ae estimated as.747 (oll-off),.616 (HE),.437 (ER1),.4 (ER),.3939 (Flux),.468 (ASF) and.4484 (SE) espectively. Also, the coelation between featues was estimated to be low. C. Evaluation We pefom 1-fold coss validation to evaluate the oveall pefomance of the classifie. A vaiable numbe of mixtues pe sub-class ae used to tain the classifie. The numbe of mixtues pe sub-class finally used was aived at by tials ove a epesentative set of mixtues whose uppe limit was govened by the numbe of tokens pe sub-class available. Hee we used 1, 6,, 1, 8, 7, and 1 mixtues fo subclasses (1) to (8) espectively. Table shows the oveall confusion matix fo the esulting vocal and non-vocal classes is computed by gouping the esults fo individual sub-classes ove all 1 folds. Table. Oveall confusion matix fo fame-level V/NVclassification V (Classified) NV (Classified) V (Gound tuth) 73.7 % 6.3 % NV (Gound tuth) 7.47 % 7.54 % (5) VI. AUDIO SEGMENTATION AND SEGMENT CLASSIFICATION Analysis windows, used in shot-tem audio analysis, should be as shot as possible so that the signal within the window is almost stationay. Howeve, shot-tem classification, as descibed in the pevious section, is not entiely eliable since the infomation within a fame is limited. The sensation of a sound textue o long-tem natue of sound occus due to the combined effect of tempoal and spectal chaacteistics ove a longe duation. In ode to captue this, we take the appoach of segmenting the audio ecod into homogenous egions and then using the fame-level classifie output ove these egions to classify the segment as vocal o non-vocal. The segment bounday detection is achieved by locating abupt changes in suitably chosen shot-tem featues. A. Implementation The two featues used fo segmentation ae sub-band flux and sub-band enegy atio (ER) extacted using a fixed window length of 3 ms and a fame ate of 1 fames/sec. These ae defined as in Section V.B. The bands fo the enegy atio computation now ange fom 5 khz to 9 khz and fom 1.5 khz to 5 khz. Fo bounday making using SE, we use an uppe and lowe fixed theshold. Empiical values of these thesholds that wee found to give good esults ae.3 and.. Boundaies ae maked at fames whee the ER cosses a value of.3 with positive slope o cosses. with a negative slope. Fo bounday making using Flux, again we use two fixed thesholds. Empiical values of these thesholds that wee found to give good esults ae 4 (fo uppe) and (fo lowe) db. Boundaies ae maked at fames whee the subband flux cosses a value of 4 db with positive slope o cosses db with a negative slope. The final set of boundaies is the union of the boundaies detected using each of the above featues individually. Futhe, this set is puned by examining the boundaies in a left-to-ight manne and applying the following ule. If two boundaies ae found to occu within 1 ms of each othe then the late bounday is discaded. This ule is based on the obsevation that the duation of a sung phase o a puely instumental inte-phase beak almost always exceeds ms. The making of these boundaies within shot intevals of each othe was caused by eithe successive stong tabla stoke onsets o pitch vaiations in the singing voice. B. Evaluation of Segmentation Method Hee, the accuate making of sung phase onsets and offsets is of moe impotance than the eduction of boundaies maked within a homogenous segment (false alams). We have obseved that the duation of a sung phase aely falls below 1 sec. A phase onset o offset is said to be coectly detemined when the neaest maked bounday lies within a neighbouhood of 15 ms. Using this citeion we found that 78.9 % of actual phase boundaies have been coectly identified.

5 Offset (sec) Fig.. Histogam of bounday location offsets Fig. shows a histogam of the offsets fom actual boundaies, obtained fom annotated data, occuing duing bounday making. We can see that the majoity of the boundaies ae maked within ms of the manually maked boundaies. Thee ae also few vey high offsets (aound 1.1 sec). These occu duing audio segments whee the Signal-to- Intefeence Ratio (SIR) is vey low i.e. whee the voice is vey soft. This happens sometimes duing the end of sung phases whee the voice gadually tails off. C. Segment Classification Results The fist method of gouping fame-level classification esults ove segments used hee is majoity voting (MV) i.e. if the majoity of fames of a potion ae classified as vocal the potion is classified as vocal, and vice vesa. The second method used involves gouping the fame-level log likelihoods (LLH) fo each sub-class ove the segment. A segment is classified as vocal if the maximum oveall likelihood of all sub-classes occus fo a vocal sub-class. Fo evaluation of segment classification, the classifie was tained using the entie data set. Then all single audio excepts wee individually input as testing data to the classifie. Oveall confusion matices wee computed befoe gouping, afte gouping using majoity vote and using log likelihoods. These ae shown in Table 3. The bounday make used detects seveal exta boundaies in addition to the sung phase boundaies. To get a pespective on how much impovement in segment classification is possible by impoving the pefomance of the bounday make, Table 4 shows the confusion matices fo gouping using majoity vote and using log likelihoods using the bounday makes extacted fom the manual annotation of individual excepts (ideal boundaies). Table 3. Oveall confusion matices fo segment classification using automatically detected boundaies MV LLH Befoe gouping Afte gouping V NV V NV V 79.5 %.75 % % 15.5 % NV 5. % 74.8 % 1.31 % 78.7 % V 79.5 %.75 % 8.87 % % NV 5. % 74.8 %.44 % % Table 4. Oveall confusion matices fo segment classification using manually detected boundaies MV LLH Afte gouping V NV V 9.4 % 7.96 % NV 4.4 % % V 91.1 % 8.88 % NV.98 % 77. % Fom Table 3 it can be seen that the accuacy of the system impoves when fame-level decisions ae gouped ove segments. It can also be seen that both methods of gouping studied povide simila esults. Fom Table 4, we note that an impovement in bounday making will esult in still highe classification accuacies. VII. SUMMARY The poblem of accuate detection of the pesence of the singing voice is studied in the context of noth Indian classical vocal pefomances. Based on the signal chaacteistics of the voice and the tanpua, a pe-pocessing method fo tanpua suppession was poposed. A GMM classifie using appopiately selected featues was evaluated ove a epesentative dataset. Futhe impovement in classification accuacies was achieved by gouping fame-level decisions ove segments, which wee automatically detected using the poposed bounday making algoithm. We ae cuently addessing the impovement of the accuacy of bounday detection by investigating new featues elated to the time tajectoy of the tonal components. REFERENCES [1] A. Bapat, V. Rao, and P. Rao, Melodic contou extaction fo Indian classical vocal music, in Poc. Intenational Wokshop on Atificial Intelligence and Music (IJCAI-7), Hydeabad, India, 7, pp [] T. Zhang, System and method fo automatic singe identification, in Poc. IEEE Intenational Confeence on Multimedia and Expo (ICME), pp , 3. [3] Y. Kim, and B. Whitman, Singe identification in popula music using voice coding featues, in Poc. 5th Intenational Conf. on Music Infomation Retieval, Bacelona, Oct. 1-14, 4. [4] A. Beenzweig, D. Ellis, and S. Lawence, Using voice segments to impove atist classification of music, in Poc. AES nd Intenational Confeence,. [5] N. Maddage, C. Xu, and Y. Wang, An svm-based classification appoach to musical audio, in Poc. 4th Intenational Conf. on Music Infomation Retieval, Washington D.C.., Oct. 6-3, 3. [6] Y. Li and D. Wang, Sepaation of singing voice fom music accompaniment fo monoaual ecodings, IEEE Tans. on Audio, Speech and Language Pocessing, vol. 15, no. 4, pp [7] T. New, A. Shenoy, and Y. Wang, Singing voice detection in popula music, Poc. 1th annual ACM Intenational Confeence, New Yok, Oct. 1-16, 4. [8] S. Boll, Suppession of acoustic noise in speech using spectal subtaction, IEEE Tans. on Audio, Speech and Signal Pocessing, vol. 7, no., pp , [9] G. Peetes, A lage set of audio featues fo sound desciption (similaity and classification) in the CUIDADO poject, CUIDADO I.S.T. Poject Repot 4. [1] R. Battiti, Using mutual infomation fo selecting featues in a supevised neual net leaning, IEEE Tans. on Neual Netwoks, vol. 5, no. 4, pp , 1994.