GENERALISED PRIOR SUBSPACE ANALYSIS FOR POLYPHONIC PITCH TRANSCRIPTION. Derry FitzGerald, Matt Cranitch

Transcription

1 Poc. of the 8th Int. Confeence on Digital Audio Effects (DAFX-5), Madid, Spain, Septembe 2-22, 25 GENERALISED PRIOR SUBSPACE ANALYSIS FOR POLYPHONIC PICH RANSCRIPION Dey FitzGeald, Matt Canitch Cok Institute of echnology Rossa Avenue, Bishopstown, Cok, Ieland Eugene Coyle Dublin Institute of echnology Kevin St., Dublin, Ieland ABSRAC A efomulation of Pio Subspace Analysis (PSA) is pesented, which estates the poblem as that of fitting an undecomplete signal dictionay to a spectogam. Futhe, a genealization of PSA is deived which allows the tansciption of polyphonic pitched instuments. his involves the tanslation of a single fequency pio subspace of a note to appoximate othe notes, ovecoming the poblem of needing a sepaate basis function fo each note played by an instument. Examples ae then demonstated which show the utility of the genealised PSA algoithm fo the puposes of polyphonic pitch tansciption.. INRODUCION Pio Subspace Analysis (PSA) was fist poposed as a technique fo tansciption and sound souce sepaation of dum sounds [], and was found to be successful at tackling the tansciption of cetain types of dum sounds. Howeve, the method was not suitable fo the tansciption of pitched instuments, as it equied an individual pio subspace fo each note of a pitched instument. he emainde of this pape descibes a efomulation and extension of the oiginal PSA algoithm to allow the tansciption of polyphonic music. Section 2 descibes the efomulation of PSA in tems of fitting an undecomplete signal dictionay to a timefequency epesentation of a signal, and Section 3 descibes an extension to this model to allow polyphonic pitched instument tansciption. Section 4 shows peliminay esults obtained using this genealised PSA algoithm. 2. PRIOR SUBSPACE ANALYSIS - A REFORMULAION Given an input signal, PSA assumes that a magnitude spectogam of the signal Y esults fom the supeposition of l unknown spectogams Y j. Futhe, it is assumed that each of these spectogams can be epesented as the oute poduct of an invaiant fequency basis function, and an invaiant amplitude basis function, in the manne of Independent Subspace Analysis [2]. his yields: l l Y = Yj = a js () j It is then assumed that thee ae known fequency basis functions o pio subspaces a p that ae good appoximations to the actual subspaces. Substituting fo the a j with these pio subspaces yields: l Y a s (2) p j In matix notation this becomes: Y Aps (3) As oiginally fomulated, PSA obtained estimates of s by multiplying the oveall spectogam by the pseudo-invese of the fequency basis functions to obtain an intial estimate of s. Independent Component Analysis (ICA) [3] then pefomed to yield an impoved estimate of s. In ecent yeas, it has been poposed that sound souce sepaation can be achieved by means of spase decomposition in a signal dictionay [4]. he signal dictionay used in this eseach consisted of a wavelet packet dictionay. Moe ecently, it has been poposed that sound souce sepaation in single channel signals can be caied out by fitting an ovecomplete signal dictionay to the signal, in conjunction with knowledge of spectal cues such as the head elated tansfe function [5]. he above eseach suggests a diffeent view of the pio subspaces used in PSA, namely that the pio subspaces ae a signal dictionay, albeit a vey undecomplete signal dictionay. he PSA poblem can then be stated as follows: given a signal dictionay, A p, and a spectogam Y, find an estimate of s given some suitable citeia. In this case, a suitable citeia would be to assume that the data is spase in natue. Using the pseudo-invese is not suitable as it assumes the data is gaussian in natue. While following the use of the pseudo-invese with ICA goes some way to solving this poblem, the PSA poblem is close in fomulation to Non-negative matix factoisation (NMF) [6] and Non-negative Spase Coding (NNSC) [7]. Both NMF and NNSC attempt to appoximate a nonnegative matix x of size n x m, such as a spectogam, by decomposing it into a mixing matix A of size n x, and a set of featue vectos s, of size x m: x As = A s (4) a= whee is the numbe of basis functions chosen to epesent the oiginal data. Both methods assume that the input data is spase in natue [7], and have been used fo souce sepaation and tansciption of polyphonic audio [8,9,]. Howeve, both suffe fom the poblem of choosing a suitable to give the best intepetation of the data, though this is less of a poblem fo NMF. Futhe, both suffe fom pemutation ambiguities. Both algoithms stat by andomly initialising A and s, ensuing that the initialisations ae non-negative. Both methods use a multiplicative DAFX-

2 Poc. of the 8th Int. Confeence on Digital Audio Effects (DAFX-5), Madid, Spain, Septembe 2-22, 25 update ule fo s, while NNSC updates A via gadient descent, and NMF via a multiplicative update. Both A and s ae then updated sequentially until convegence. In the case of PSA, initial estimates of A ae available, and so it is only necessay to update fo s. his esults in a new fomulation fo PSA, descibed below in pseudo-code:. Obtain a magnitude spectogam Y of the input signal. 2. Randomly initialise s, whee s contains the amplitude envelopes of the souces, ensuing that the data is nonnegative. 3. Update s using eithe of these update ules s = s. ( Ap Y)./( Ap * Ap s + λ) (NNSC) ( A ( Y./ ( A s ). A O s = s. p p / p (NMF) whee A p ae the pio subspaces, denotes matix tanspose,.* denotes elementwise multiplication, and./ denotes elementwise division. λ is a non-negative scala, and O is an all-ones matix the same size as Y. 4. Iteate step 3. to convegence. In tests the NMF-based update ule was found to give bette pefomance than the NNSC-based ule, suggesting that the NMF-based ule povides a bette fit to the undelying data. Also, in most cases 5 iteations was found to give sufficient convegence. Anothe useful update ule fo s is that poposed by Abdallah []. It should be noted that the use of that update ule equies the use of a powe spectogam as opposed to a magnitude spectogam, and that the pio subspaces have to be modified in accodance with this. It is impotant to point out at this stage that the new PSA algoithm offes significant impovements ove the oiginal PSA algoithm. Fistly, the use of non-negativity means that the esults obtained will be moe consistent with eal wold situations, whee negative amplitudes of souces cannot occu. his was a poblem with the oiginal PSA algoithm in that the amplitude envelopes obtained wee sometimes physically implausible. Secondly, because we ae only updating s, and the pio subspaces ae held constant, thee is no longe any pemutation and scaling ambiguities in the algoithm. his means that the ecoveed souces ae diectly associated with the pio subspaces, and thee is no longe any need to identify the souces afte pocessing. In the case of using the new PSA algoithm fo dum tansciption of snae, kick dum and hi-hats, this means that the assumptions used to identify the souces in the oiginal PSA tansciption algoithm ae no longe equied. hese assumptions wee that the kick dum had a lowe spectal centoid than the snae, and that the hi-hats occued moe fequently than the snae dum. he elimination of these assumptions allows the new algoithm to function in a wide ange of cicumstances. When tested on the same data set as the oiginal PSA algoithm, the pefomance impoved to a 94.7% success ate, as opposed to the 92.5% success ate achieved with the oiginal algoithm. able, below, shows the esults obtained using the oiginal PSA algoithm, while able 2 shows the esults obtained using the efomulated PSA algoithm. It can be seen that the pefomance in tansciption of snaes and hi-hats has impoved, while thee has been a small degadation in the ecovey of the kick dum. Nevetheless, it can be seen that the new algoithm has outpefomed the oiginal PSA algoithm. It should be noted that a simila efomulation of the PSA algoithm descibed above was aived at independently by Paulus et al [2]. Howeve, fo the puposes of pitched instument tansciption, both efomulations suffe fom the need fo an individual pio subspace fo each note pesent. Methods to ovecome this poblem ae pesented in Section 3. ype otal Missing Incoect % Snae Kick 33 Hats Oveall able : Dum ansciption Results using the oiginal PSA algoithm ype otal Missing Incoect % Snae 2 Kick Hats Oveall able 2: Dum ansciption Results using the efomulated PSA algoithm. 3. GENERALISED PRIOR SUBSPACE ANALYSIS It can be seen fom the above that an extended model is needed to eflect the situation whee vaious notes fom the same instument occu ove the couse of a spectogam. Pevious wok attempting to deal with this includes the non-linea Independent Subspace Analysis model poposed by Vincent et al [3]. In this model, chod specta ae epesented as sums of note powe specta, and note specta ae epesented as sums of instument dependant logpowe specta. Note duations ae then modeled using Hidden Makov Models. ime-fequency analysis was caied out using a log-fequency scale and successful tansciption was obtained fo two duo ecodings. A potential way of ovecoming the poblem of dealing with multiple notes belonging to a single souce is to assume that the notes belonging to a single souce consist of tanslated vesions of a single fequency basis function. his single fequency basis function is then taken to epesent the typical fequency spectum of any note played on the instument in question. his is a simplified appoximation of the eal situation, whee the fequency spectum of the note does vay with pitch. Despite this, the assumption does epesent a valid appoximation ove a limited pitch ange. A vesion of this assumption is used in commecial music samples and synthesises, whee a ecoded note of a given pitch is used to geneate othe notes in poximity to the oiginal note. It should be noted that the use of this assumption also places a futhe estiction on the type of spectogam being analysed, namely that the fequency esolution of the spectogam must be logaithmic in scale. Figue, below, shows the fequency specta of two diffeent notes played on a Fench hon. It can be clealy seen that the specta of the two notes ae vey simila, and so the spectum of eithe note can be appoximated by a tanslation of the othe note. It is also assumed that no significant infomation is contained in the extemes of the tanslated fequency basis function. It can be seen that this assumption holds fo the specta shown in Figue. his assumption also sets limits on how fa a given basis function can be tanslated. Fo example, tanslating the fist of the DAFX-2

3 Poc. of the 8th Int. Confeence on Digital Audio Effects (DAFX-5), Madid, Spain, Septembe 2-22, 25 two specta shown by moe than 2 bins to the left will esult in pat of the fist patial to be moved to the end of the fequency spectum, whee it is clealy not supposed to occu Fequency bin Figue. Specta of two notes of a Fench hon o tanslate a given n x vecto, an n x n tanslation matix can be used. Such a tanslation matix can be geneated by eaanging the columns of the identity matix. Fo example, to achieve a shift up of one, the tanslation matix would be obtained fom I(:,[n, :n-]) whee I denotes the identity matix, and whee the odeing of the columns is contained in the squae backets. A simple example of a shift up of one, which is obtained by a shift to the left of the ones in the identity matix is given below fo a 5 x matix = Fo the tansciption of a single instument playing multiple notes, the signal model now becomes: Y = j As j (6) whee Y is a log-fequency spectogam of size n x m, A is an n x vecto containing a typical hamonic pofile and s j contains the amplitude basis function, of size x m associated with tanslation matix j of size n x n. An algoithm which attempts to lean both A and s j fom an input spectogam is descibed in [4]. he utility of this signal model can be seen in that a single basis function can now be used to model a pitched instument and can be seen as a means of genealising the PSA model to deal with pitched instuments, as a single pio subspace of an instument note can be used to geneate othe notes fom the instument. Suitable pio subspaces fo a given instument can then be obtained via a numbe of methods. NMF can be pefomed on a single note of an instument to obtain a hamonic pofile, o a single fame of a log-fequency spectogam with a well established hamonic pofile can be chosen as a basis function. Fo a pedefined set of tanslations and a given instument pio subspace, the signal model can be ewitten as: (5) Y = A s j j = A s (7) whee A j = j A, and A is a matix of size n x, and s is a matix of size x m. In othe wods, the signal model can be collapsed to the standad PSA model. heefoe, the same optimisation techniques can be used to obtain s as in the efomulated PSA model. he esulting genealised PSA algoithm fo the tansciption of a pitched instument can then be summaised as follows:. Obtain a spectogam with log-fequency esolution of the input signal. 2. Detemine tansfomation matices j fo a given ange of tanslations. 3. Obtain tanslated vesions A j of pio subspace A fom A j = j A. 4. Randomly initalise s, ensuing non-negativity. 5. Update s using the update ule: s = s. ( A ( Y./ ( A s ). / A O whee denotes matix tanspose,.* denotes elementwise multiplication, and./ denotes elementwise division, and O is an all-ones matix the same size as Y 6. Iteate step 5 to convegence. he NNSC-based update ule was found to be unsuited fo the puposes of pitched instument tansciption and so is not included in the algoithm. Peliminay esults obtained using the algoithm ae detailed in Section RANSCRIPION USING GENERALISED PRIOR SUBSPACE ANALYSIS o test the effectiveness of the genealised PSA algoithm, a numbe of simple tests wee caied out. Fistly, a ecoding was made of a sampled piano playing a C majo scale fom note C5 to note C6. his sampled piano made use of 4 sepaate piano note samples pe octave, A pio subspace was obtained fo piano note G5 fom a completely diffeent sampled piano. he esults obtained fo s ae shown in Figue 2 below, in which a shift of one coesponds to a pitch change of a semitone. he ange of tanslations was set to +/-, though a geate numbe could have been used without affecting the esult. It can be seen that the algoithm has successfully captued the notes in the input wavefom. As the pio subspace was a G5 piano note, this note has a shift of in the above plot, and it can be seen that the notes played do indeed follow the patten of a majo scale. As the tanslated pios ae fixed, the algoithm does not suffe fom the souce odeing poblem inheent in blind souce sepaation algoithms, and so the ecoveed amplitude basis functions will be pesented in the coect ode. his means that the basis functions can be plotted in a manne simila to a piano oll, as shown below in Figue 2. he second test caied out on the algoithm involved the tansciption of a seies of thee-note piano chods. he same sampled piano was used as in the pevious example, and the same pio, a piano-note of pitch G5 was used. A pianooll plot of the midifile used to geneate the audio signal is shown in Figue 3. his was geneated using the Miditoolbox [5]. Figue 4 then shows the output of the genealised PSA algoithm. Remembeing that a tanslation of coesponds to note G5, it can be seen that DAFX-3

4 Poc. of the 8th Int. Confeence on Digital Audio Effects (DAFX-5), Madid, Spain, Septembe 2-22, 25 the algoithm has successfully tanscibed the piano chods, with the notes clealy distinguishable fom any noise in the basis functions. his shows that the algoithm is capable of tanscibing polyphonic music. libay, with 4 samples pe octave. he tanslation ange was set to +/- 25, and the pitch ange of the actual signal was fom D5 to E4. he piano pio fom the pevious examples was used in an attempt to see if a single hamonic pio was capable of tanscibing moe than a single instument. he midi-file used to ceate the audio signal is shown in Figue 5. In this figue the tumpet, is epesented by black, and the fench hon, on channel 2, is epesented by gey. Figue 2. Outputof algoithm fom a piano scale Figue 5. Pianooll of tumpet and Fench hon. Figue 3. Pianooll of 3 note piano chods Figue 4. Output of algoithm fomaudio signal of piano he thid test pefomed on the algoithm was an audio signal which contained a tumpet and a fench hon playing sepaate melody lines. Both wee ceated fom an ochestal samples Figue 6. Output of algoithm fom tumpet and fench hon example Figue 6 then shows the output of the genealised PSA algoithm. It can be seen that the notes have been successfully ecoveed by the algoithm. his suggests that a genealised hamonic pofile has the potential to be able to successfully tanscibe a wide ange of instuments. Fo tansciption puposes, this can be consideed a stength, in that it would not be necessay to have a diffeent pio fo each instument in ode fo successful tansciption to occu. Howeve, fo the puposes of sound souce sepaation, this would not be useful. Finally, the algoithm was applied to a ecoding of a gand piano, ecoded in a small theate in evebeant conditions to see how the algoithm would deal with eal-wold audio. Futhe, to see if a synthetic hamonic pio could be used fo tansciption, a hamonic pio was geneated fom a sum of 7 sinu- DAFX-4

5 Poc. of the 8th Int. Confeence on Digital Audio Effects (DAFX-5), Madid, Spain, Septembe 2-22, 25 soids, with each patial having half the amplitude of the pevious patial. he pio used coesponded to the pitch of middle C (26.63 Hz). he output of the genealised PSA algoithm is shown in Figue 7, below. When compaed by hand to the ecoding, the algoithm was found to successfully ecove the vast majoity (93%) of the notes played, with the exception of a small numbe of low amplitude notes. his was afte thesholding and elimination of shot duation activations of s. It is possible that some fom of peceptual weighting as descibed in [6] on the input spectogam may be of use in ecoveing these notes. he success of this test shows that the algoithm can function on eal-wold signals, and that a synthetic pio can be used to attempt tansciption of a eal instument. It also demonstates that a single pio can function ove a wide pitch ange, in this case dealing successfully with a ange of two octaves. Figue 7. Output of algoithm fom gand piano ecoding 5. CONCLUSIONS A efomulation of PSA as a signal dictionay fitting poblem has been pesented. Following on fom this, a genealisation of the PSA algoithm was deived which uses tanslations of a single feuquency basis functions to epesent diffeent notes. he effectiveness of the genealised PSA algoithm as a method fo polyphonic music tansciption was then demonstated using both midigeneated and eal-wold signals. It also demonstates that a well chosen undecomplete signal dictionay, in this case a single dictionay element and tanslations theeof, can be used to extact much meaningful infomation fom audio signals. Futue wok will concentate on identifying accuately the pefomance of the algoithm fo tansciption of polyphonic music in a wide ange of situations, and on attempting to impove pefomance though the use of peceptual weighting. It is also intendend to attempt to tanscibe both pitched instuments and pecussive instuments simultaneously by appending a set of pecussion pios to the set of tanslated pitch pios. 6. ACKNOWLEDGEMENS 7. REFERENCES [] D. FitzGeald, Automatic Dum ansciption and Souce Sepaation, PhD. hesis, Dublin Institute of echnology, 24 [2] M.A. Casey and A. Westne, Sepaation of Mixed Audio Souces By Independent Subspace Analysis in Poc. Of ICMC 2, pp. 54-6, Belin, Gemany. [3] P. Comon, Independent component analysis - a new concept?, Signal Pocessing, 36(3): , Apil 994 [4] M. Zibulevsky and B. A. Pealmutte, Blind Souce Sepaation by Spase Decomposition in a Signal Dictionay, Neual Computation, 3(4): [5] B. A. Pealmutte and A. M. Zado. Monaual Souce Sepaation Using Spectal Cues. In Poceedings of ICA24, Septembe 22-24, 24, Ganada, Spain, pages [6] D. Lee, and H. Seung, Algoithms fo non-negative matix factoization. Adv. Neual Info. Poc. Syst. 3, (2). [7] P.O. Hoye, Non-negative spase coding Neual Netwoks fo Signal Pocessing XII (Poc. IEEE Wokshop on Neual Netwoks fo Signal Pocessing), pp , Matigny, Switzeland, 22. [8] H. Asai,. Non-negative Matix Factoization: A possible way to lean sound dictionaies, [9]. Vitanen, Sound Souce Sepaation Using Spase Coding with empoal Continuity Objective, Poc. of Intenational Compute Music Confeence (ICMC23), Singapoe, 23. [] P. Smaagdis, J.C. Bown, "Non-negative Matix Factoization fo Polyphonic Music ansciption", IEEE Wokshop on Applications of Signal Pocessing to Audio and Acoustics (WASPAA), pp. 77-8, Octobe 23 [] S. A. Abdallah and M. D. Plumbley, Polyphonic tansciption by non-negative spase coding of powe specta., Poceedings of the 5th Intenational Confeence on Music Infomation Retieval (ISMIR 24), Bacelona, Spain, Octobe -4, 24. [2] J. Paulus and. Vitanen, Dum tansciption with nonnegative spectogam factoization, submitted to the submitted to Euopean Signal Pocessing Confeence 25. [3] E. Vincent and X. Rodet, Music tansciption with ISA and HMM. In Poceedings of ICA, 24. Septembe 22-24, 24, Ganada, Spain, pages [4] D. FitzGeald, M. Canitch, E. Coyle, Shifted Non-negative Matix Factoisation fo Sound Souce Sepaation, Statistics in Signal Pocessing confeence, Bodeaux, Fance, July 25. [5]. Eeola, and P. oiviainen, MIR in Matlab: he Midi oolbox. In Poceedings of 5th Intenational Confeence on Music Infomation Retieval (ISMIR 24), pp , Bacelona, 24. [6]. Vitanen, Sepaation of sound souces by convolutive spase coding, in Poc. of ISCA utoial and Reseach Wokshop on Statistical and Peceptual Audio Pocessing, 24. his eseach was suppoted by funding fom the Iish Reseach Council fo Science, Engineeing and echnology. DAFX-5