Note separation of polyphonic music by energy split

Transcription

1 Universiy of Cambridge, UK, ebruary 0-, 008 Noe separaion of polyphonic music by energy spli KRISTÓ ACZÉL 1, ISTVÁN VAJK Deparmen of Auomaion and Applied Informaics udapes Universiy of Technology and Economics 3-9. Műegyeem rkp., H-1111, udapes HUNGARY Absrac: - A polyphonic musical recording is he superposiion of many separae racks which are downmixed o fewer channels. Decomposiion of such a signal o separae insrumen racks or noes has always been a challenge. I is heoreically impossible o exrac he componen racks wihou he informaion ha was los a he superposiion. In his paper a new way of sound separaion of mono-aural digial recordings is proposed where los informaion is recovered by using a model of real insrumens in order o make he separaion of individual musical noes possible. Key-ords: - sound separaion, insrumen prin, polyphonic music, energy spli 1 Inroducion A musical recording ypically is a polyphonic maerial ha is he composiion of many individual noes originaing from a number of insrumens. The recording is made using many microphones. However heir oupu signals are hen downmixed o fewer, ypically one or wo channels. Once he recording is downmixed, no correcion can be made o he individual noes in i. Our long erm ineres in sound separaion is moivaed by he problem of correcing hese recordings if hey have incorrec noes (eiher in frequency, volume, inonaion, lengh ec). e allow a reasonable amoun of user inpu and processing ime in favor of separaion qualiy. User inpu involves enering he musical score (noe saring/ending imes, frequencies, used insrumens). Alhough due o he naure of real-life music his inpu will never be 100% accurae, i can be precise enough for geing a firs esimae on he noe parameers in he recording, insead of relying on musical ranscripion ([1], [], [3], [4], [5]) and insrumen recogniion ([6], [7], [8], [9]) algorihms which are inferior o he performance of a human lisener. The complexiy of separaion can be aribued o he fac ha he informaion o be rerieved is acually missing from he signal. This issue made researchers approach he problem in many differen ways. [10] is a sound source separaion algorihm ha requires no prior knowledge on he insrumen noes in he recording, and performs he ask of separaion d purely on azimuh discriminaion wihin he sereo field. The resuls are impressive. However, separaing individual noes is no in he focus, only insrumen groups. [1], [13], [14] describe a mehod which separaes harmonic sounds by applying linear models for he overone series of he sound. The mehod is d on a wo-sage approach: afer applying a mulipich esimaor o find he iniial sound parameers, more accurae sinusoidal parameers are esimaed in an ieraive procedure. Separaing he specra of concurren musical sounds is d on he specral smoohness principle [11]. eamforming echniques [15] along wih he Independen Componen Analysis framework offer a differen way of separaion. A large array of microphones is employed for recording an even. The ravel ime of he ime signal and he difference of he recorded signals are hen used in calculaions ha increase he receiver sensiiviy in he direcion of waned signals and decrease he sensiiviy in oher direcions. However, hese mehods rely on cerain preliminary condiions and sudio seup o achieve good resuls. The following secions will guide he reader hrough he separaion process, each secion covering one imporan block of he algorihm. Secion shows an overview of hese blocks and he signal flow in he sysem. The whole separaion process is carried ou in frequency domain, hus he firs sep is convering he signal. This sep is covered in secion 3 along wih he inverse ransformaion used a he end of he separaion process. Secion 4 describes he insrumen model ha is used in he sysem, while secion 5 covers he deails of he acual separaion process, he ISSN: ISN:

2 Universiy of Cambridge, UK, ebruary 0-, 008 Simplified Energy Spli (SES) mehod. Missing informaion on noe inonaion ( playmode ) and volume levels mus be calculaed prior o he acual separaion. Secion 6 deals wih he mos obvious soluion for he problem. Overview of he separaion process This secion shows an overview of he separaion process. Shor descripions of he building blocks are given which are discussed in deail in laer secions. irs, all ime-domain signals are convered o frequency domain. indowed T is employed for his purpose along wih requency Esimaion ([1]) and Phase Memory ([18]) mehods. The separaion is aided by sored insrumen samples. These samples are sored one by one in bandogram forma which is basically a specrogram spli o subbands, in which he energy is summed. andograms originaing from insrumen noes are sored in he sample sore. One insrumen will have a number of samples differing by heir frequency and playmode. This makes possible o selec he sample ha bes maches he noe in he recording o help he separaion process. The collecion of bandogram samples from he same insrumens will be called an insrumen prin. The playmode and volume deecor receives he original recording, he musical score and he insrumen prins. Is role is o selec one sample from each insrumen prin ha bes fis he original insrumen in he recording. The Simplified Energy Splier ges he specrogram of he recording and he bandogram of he seleced samples as inpu. I splis he energy in he specrogram of he original recording o componens ha resemble he inpu bandograms. inally, he frequency-domain signals are convered back o ime domain. igure 1 shows he block-diagram of he sound separaion process. The following noaion is used for differen represenaions of signals: Simple waveform () Simple T (S): specrogram soring c ampliude and ϕ phase for each bin. requency esimaed specrogram (): c ampliudes and ϕ phases remain he same as hose of simple T, bu an f frequency value is sored in addiion for each bin. andogram (): A specrogram spli o subbands, in which he energy is summed. Only hese sums are sored, no deailed informaion on bin ampliudes and no phase informaion eiher. The following chapers will guide he reader sep by sep hrough he separaion process. Each secion will cover one block in deail. Due o size limiaions, however, some of he blocks will no be exhausively discussed in his paper. 3 Transformaion o frequency domain This secion proposes an easy, ye powerful algorihm ha is able o generae a specrogram of he recording ha is much more precise for musical analysis han he convenional T specrogram. Earlier lieraure [16], [17] covered differen ransformaion mehods in order o deermine he bes possible means for analysis of audio signals. Curren research has examined he analysis of polyphonic musical signals in paricular [18], [19]. In [1] a frequency esimaion mehod is shown, ha calculaes frequencies presen in he original signal from subsequen phase values. or a frame saring a ime he T coefficiens and phases are c and φ, respecively. In his documen he ime index will be omied in some of he equaions for undersandabiliy. Two subsequen frames are needed by he algorihm for he calculaion. Assuming ha wo subsequen frames sar a 1 and, a frequency f can be compued for each bin. The nominal frequency of he k h bin is samplerae fk k. framesize The frequency of each bin will iae from his value, and can be expressed as: wih f ϕ fk +, π ( ) exp 1 exp ϕ ϕ + ( 1 1 ) π fk, ϕ ϕ ϕ + l π where exp exp ϕ is he phase of bin k in ime ; ϕ is he expeced phase; ϕ is he iance beween he expeced and measured phase; f is he esimaed frequency of bin k in ime and l Z : π < ϕ + π. The greaer he ime ISSN: ISN:

3 Universiy of Cambridge, UK, ebruary 0-, 008 Insrumens of User inpu Insrumen noes for sampling he recording T T + T S S req Es andogram calcul S req Es andogram calcul. req Es Insr1 Sample sore 1..s1 Insr 1..s Playmode and volume deecor Simpified Energy Splier noes T -1 T -1 T -1 Remainder igure 1: Signal flow and block diagram of he separaion process Separaed noes difference beween he sar of he frames he more precise he esimaed value of f. On he oher hand, big ime differences limi he maximum deecable disance beween f and f k. Someimes, mainly for lower frequencies or complex signals wih many componens, f will flucuae around he real value ha is presen in he original signal. The frequencies for periodic waves can be found more precisely by aking he weighed average of he las m, curren and nex m phase iaions. This exension will be referred o as Phase Memory (PM) and he new PM esimaed frequency can be calculaed as where fˆ ˆ ϕ fk +, π ( ) 1 ˆ ϕ x ϕ c x x x x ϑ( x), c ϑ( x) x where ϑ ( x) denoes he weighing funcion. inding opimal ϑ ( x) is ou of he scope now. I is imporan o menion ha, alhough he proposed PM mehod is a very effecive ool for signal analysis, i is no used in he ransformaion back o ime domain. In a usual case, where he arge is only he isolaion of noes, no even f is needed, a simple inverse T using c and ϕ values will accomplish he ask. However, in cases like pich shifing, where he separaed noes (and hus f frequencies) are alered, i may be necessary o recalculae he ϕ phases from he new f values before he inverse T. ISSN: ISN:

4 Universiy of Cambridge, UK, ebruary 0-, Insrumen prins The main complexiy of sound separaion lies in he paradox ha we need o regain informaion from a signal ha does no fully conain i. A some poin we will definiely have o inpu addiional informaion ino he separaion sysem o complee he missing daa. Human liseners, who are known o be able o do he separaion in heir mind, use memories of insrumens and memories of he noes in he musical piece being performed. This is heir source of addiional informaion. Copying naure has been proven o be he righ approach many imes. This secion shows a way of implemening a memory of known insrumens, rying o mimic he way he human brain works. The represenaion used o sore insrumen feaures will be called an insrumen prin. [] presens experimens ha examine he dynamic aribues of imbre evaluaing he role of onses in similariy judgmens. I also gives an overview of researches pursuing he idenificaion of he mos imporan properies of insrumen sounds ha make a human lisener able o disinguish beween hem. The insrumen prins in his paper are in par d on hese researches, in he sense ha hey conain he feaures ha were found imporan in he experimens menioned. However, separaion purposes require more informaion on insrumens han pure idenificaion does. An insrumen prin conains samples from an insrumen on differen frequencies and wih differen inonaions, playmodes. The erm playmode refers o he way he insrumen was played, e.g. he hardness of a piano key hi, he blowing srengh of he flue or he inonaion of a saxophone noe. One prin can have more han one playmode dimensions, depending on he way an insrumen can be played. These canno always be defined by mahemaical definiions, very ofen hey can only be expressed by subjecive erms (e.g. sharpness, warmh ec.). The insrumen prin is a collecion of samples on differen frequencies f and also wih differen values in he playmode space M [ m1, m,... m p ]. I can be regarded as a funcion A( M, f, f, ) k showing how ampliudes hrough he frequency range change over ime for a specific noe a f frequency, played wih a playmode M, wih he + condiions, m, f R, < m < m, 0 < x and 0< f 0000Hz. 0 x x,max In realiy, a sample will no sore a coninuous specogram, only a bandogram which represens he sum energy characerisics in cerain frequency subbands ha are aligned on a logarihmical frequency scale. One sample is calculaed from a sound signal conaining one noe as where A M, f, b, c, b 0,5 b+ 0,5 R ˆ f, R < fk < f b log R ˆ f idenifies he specific subband, while R is an experimenal value defining he resoluion in frequency range, ha is, he number of subbands per ocave. Experimens showed ha R 1 provides good enough resoluion in log frequency. The number of sored insrumen samples is finie boh in frequency and playmode spaces. Missing samples will be inerpolaed from he exising ones when needed. 5 The Simplified Energy Splier This secion describes he hear of he separaion process, he Simplified Energy Splier (SES). irs, he main issue of separaion will be briefly presened. Since he original decomposiion problem canno be solved due o he lack of informaion, furher on a cerain simplificaion will be proposed ha, alhough lowers he qualiy, makes i possible o carry ou he separaion even under hese circumsances. y applying his change he separaion problem will be simplified o an energy spli problem. inally, he reader will be guided hrough he implemenaion of he spli process iself. The equaion sysem of he original separaion problem is i f c s, orig i, r, n where c [ c e γ τ ] is he mixed signal which is, n he inpu of he separaion algorihm; orig σ i, r, n s ˆ [ s e τ ] are he original noes. Time is i, i,, n now represened as, where r sands for he curren frame and τ is he ime difference beween subsequen frames. The above undeermined sysem of equaions canno be solved unambiguously wihou oher consrains o add. ISSN: ISN:

5 Universiy of Cambridge, UK, ebruary 0-, 008 Unforunaely our knowledge on he original noes is raher limied. No precise informaion is available on he frequency f ( ) _ of he original noes, heir saring/ending imes and heir playmode which also changes over ime. Each original noe will resemble one insrumen sample in our daa o some exen, bu here are no perfec maches ever. I is obvious ha under hese circumsances we will no be able o decompose he recording c o an array of signals ha are perfec orig replicas of he original s i ones. The arge is o decompose i o signals ha resemble he original ones, or lacking he original noes a leas he samples ha are used in he separaion. The erm resemblance is of course an expression aken from real life raher han an exac mahemaical measure. In he case of an auomaed algorihm, however, i mus be defined in an exac manner in order o be able o inerpre and validae he oucome of he separaion algorihm. Due o space consideraions he definiion of resemblance is no discussed in his paper. As he original separaion problem canno be solved, simplificaions have o be made. The mos obvious change is eliminaing he unknown ) σ i,0 τ, k phases from he equaion sysem: ) σ γ i,, k, k This rephrases he original problem o c sˆ + c ˆ i, i where s ˆi is separaed noe i and ĉ is he remainder of he recording. This percepually moivaed modificaion explois he fac ha he human ear does no differeniae by he phase of he heard sinusoids, we only hear magniude differences. Of course any modificaions will have a smaller or greaer impac on he qualiy of he separaion, causing arifacs in he oupu. The modified equaion does no handle periodic signals wih closely locaed frequencies well. If wo or more signals cancel each oher, his effec will also appear in he separaed noes. Experimens showed ha his radeoff is accepable in mos cases. In he energy spli sep bandograms of he righ samples will be used o recreae specrograms of he noes o be separaed from he remaining par of he recording. Semi-linear decomposiion will be used. i Assuming ha we know he exac frequency, volume and playmode of a cerain noe ha we wan o separae, he following ieraive algorihm can be proposed o divide he energy beween he arge noes. e sar ou wih he original requency Esimaed T image of he recording. In each sep a fracion of he energy of he seleced samples is ransferred from he T of he recording o he T of he separaed noes. This ensures a fair division of he energy of he recording. Le cˆ [0],0, c [0],0, c denoe he iniial energy residing in he recording. s ˆ[ d ], i, will denoe he curren energy in he separaed noes, being s ˆ i 0 iniially. Each sep [d] conains i subseps, [0],, in which an α fracion of he energy in he reference sample A i is ransferred from he curren remaining energy c ˆ[ d ], i, o he separaed noe s ˆ[ d+ 1], i, if sill possible, as in wih c [ d ], i, α b 0,5 b+ 0,5 R ˆ R f < fk, < f : α δ D oherwise : cˆ [ d ], b, i, [ d ], b, i, cˆ [ d ], i 1, (1 ) Ai ( Mi, f _ i, b, Ti, sar ) cˆ [0],0, k, b 0,5 b+ 0,5 R ˆ f, R < fk < f [ d ], i, The curren energy in noe i (which is being isolaed) can be calculaed in sep [d] as sˆ sˆ + ( cˆ c ˆ ) [ d+ 1], i, [ d ], i, [ d ], i 1, [ d ], i, which is he saring value of he nex [d+1] sep for separaed noes, while cˆ c ˆ [ d+ 1],0, [ d ], I, is he saring value of he nex sep for he remaining energy in he recording, where I is he number of insrumens in he ime frame. ih D denoing he number of seps, c [ D],0, is he remaining par of he recording afer he separaion s will represen he coefficiens of and ˆ[ D ], i, insrumen i afer he separaion. ISSN: ISN:

6 Universiy of Cambridge, UK, ebruary 0-, Playmode deecion In he previous secion all noe parameers were assumed o be known. However, his is no he usual scenario. hile he user can inpu he locaion of he insrumen noes in frequency and ime, he/she may no be capable of enering eiher he playmode marix M or he volume. urher on he auomaic deecion of he playmode and volume will be covered. To carry ou he energy spli sep an opimal playmode marix M mus be found. or he sake of convenience he volume will also be incorporaed in M from now on. M is by definiion perfec if he separaed noes are he perfec replicas of he paren insrumen samples ha were used in he energy spli, and he remaining par is zero. In general, an M marix is considered good provided he energy spli sep ha uses M generaes noes ha resemble heir paren sample while leing c [D] ge as close o zero as possible. All combinaions of M marices will be esed for separaion error, and he one causing he leas error will be seleced. Depending on he size and possible values of M he number of seps needed for finding he bes combinaion of he insrumen samples may require huge compuaional power. If we consider he playmode space o be coninuous, i is no even possible o ierae hrough all he combinaions. inding an algorihm faser han brue force ieraion, however, is ou of he scope of his aricle. 7 Implemenaion and es scenario This secion shows an example scenario of a difficul problem in he area of sound separaion. The resuls illusrae he qualiy of he mehod proposed for isolaing noes in pracice. In he es case wo insrumens a piano and a saxophone played he une in igure. As shown, he wo insrumens played on he same frequency, which is one of he hardes cases of sound separaion. The insrumens were recorded separaely, one afer he oher. Their signals were hen mixed ogeher and fed o he separaion algorihm. igure : musical score of he analysed fragmen Insrumen prins were also buil from hese wo insrumens by sampling all halfones in he frequency range in quesion a hree differen playmodes ( sof, neural and hard ). igure 3: waveform of he original piano rack igure 4: waveform of he original sax. rack igure 5: waveform of he mixed signal igure 6: waveform of he firs separaed piano noe igure 7: waveform of he firs separaed sax. noe igure 3 o igure 7 show he wo inpu insrumens, he mixed signal, and wo separaed noes. Human liseners confirmed ha he separaed noes did sound like real insrumens, even if somewha disored. 8 Conclusion The paper has shown a mehod for separaing single insrumen noes from a recording using prerecorded insrumen prins and he Simplified Energy Splier algorihm. The resuls are quie promising. The example waveforms in Secion 7 along wih a few oher separaion samples can be downloaded from hp://avalon.au.bme.hu/~aczelkri/separaion. or recordings ha only conain harmonically unrelaed noes he algorihm provides very clear resuls. In real life, however, consonan noes wih overlapping overones are usually favored over dissonan ones. Our es resuls show ha even in cases where some noes are locaed on each oher s or overone frequencies he separaion provides reasonably good resuls. ISSN: ISN:

7 Universiy of Cambridge, UK, ebruary 0-, 008 Acknowledgemens: This work has been suppored by he fund of he Hungarian Research und (gran number T68370) References: [1] G. E. Poliner, D. P.. Ellis, A.. Ehmann, E. Gomez, S. Sreich;. Ong, Melody Transcripion rom Music Audio: Approaches and Evaluaion, IEEE Transacions on Audio, Speech and Language Processing, Volume 15, Issue 4, May 007, pp [] H. Thornburg, R. J. Leisikow, J. erger, Melody Exracion and Musical Onse Deecion via Probabilisic Models of ramewise STT Peak Daa, IEEE Transacions on Audio, Speech and Language Processing, Volume 15, Issue 4, May 007, pp [3] J. P. ello, L. Daude; M.. Sandler, Auomaic Piano Transcripion Using requency and Time-Domain Informaion, IEEE Transacions on Audio, Speech and Language Processing, Volume 14, Issue 6, Nov. 006, pp [4] A. T. Cemgil, H. J. Kappen; D. arber, A generaive model for music ranscripion, IEEE Transacions on Audio, Speech and Language Processing, Volume 14, Issue, March 006, pp [5] S. A. Abdallah, M. D. Plumbley, Unsupervised analysis of polyphonic music by sparse coding, IEEE Transacions on Neural Neworks, Volume 17, Issue 1, Jan. 006 pp [6] A. A. ieczorkowska, A. Czyżewski, Rough Se ased Auomaic Classificaion of Musical Insrumen Sounds, Elecronic Noes in Theoreical Compuer Science, Volume 8, Issue 4, March 003, pp [7] J. C. rown, Compuer idenificaion of musical insrumens using paern recogniion wih cepsral coeffciens as feaures, J. Acous. Soc. of Am., Vol. 105 (1999), pp [8] P. Herrera, X. Amariain, E. alle, X. Serra, Towards insrumen segmenaion for music conen descripion: a criical review of insrumen classificaion echniques, Proc. of Inernaional Symposium on Music Informaion Rerieval (ISMIR 000), Plymouh, MA, 000. [9] P. Herrera-oyer, G. Peeers, S. Dubnov, 003. Auomaic classificaion of musical insrumen sounds, J. New Music Res. Volume 3, 3 1. [10] D. arry, R. Lawlor, E. Coyle, Sound Source Separaion: Azimuh Discriminaion and Resynhesis, Proc. of 7h Inernaional Conference on Digial Audio Effecs, DAX 04, Naples, Ialy, 004. [11] A. Klapuri, Mulipich esimaion and sound source separaion by he specral smoohness principle, Proc. of IEEE Inernaional Conference on Acousics, Speech and Signal Processing, Sal Lake Ciy, USA, 001. [1] T. Viranen, A. Klapuri, eparaion of Harmonic Sounds Using Mulipich Analysis and Ieraive Parameer Esimaion, Proc. of IEEE orkshop on Applicaions of Signal Processing o Audio and Acousics, New Yor USA, 001. [13] T. Viranen, A. Klapuri, Separaion of Harmonic Sound Sources Using Sinusoidal Modeling, Proc of. IEEE Inernaional Conference on Acousics, Speech and Signal Processing, Isanbul, Turkey, 000. [14] T. Viranen, A. Klapuri, Separaion of harmonic sounds using linear models for he overone series, Proc. of IEEE Inernaional Conference on Acousics, Speech and Signal Processing, Orlando, la, USA, May 00. [15] N. Miianoudis, M. E. Davies, Using eamforming in he audio source separaion problem, 7h In Symp on Signal Processing and is Applicaions, Paris, July 003 [16] R. Pinelon, J. Schoukens, Sysem Idenificaion, A frequency domain approach, ISN , iley-ieee Press, May 001, pp [17] S. Gade, H. Herlufsen, Use Of eighing uncions in DT/T Analysis (Par I), rüel & Kjær Technical Review, No. 3., 1987 [18] K. Aczél, Sz. Iváncsy, Musical source analysis wih DT, Proc. of MicroCAD 006 Inernaional Scienific Conference, Miskolc, Hungary, March 006. [19] K. Aczél, Sz. Iváncsy, Musical source analysis: specrogram versus cochleagram, in Press, MicroCAD 007 Inernaional Scienific Conference, Universiy of Miskolc (Miskolc, Hungary), March 007. [0] K. Aczél, Sz. Iváncsy, Sound separaion of polyphonic music using insrumen prins, Proc of EUSIPCO 007, Poznan, Poland, Sep [1] S. M. ernsee, Pich Shifing Using he ourier Transform hp:// ml/pshifsf.hml ( ) [] P. Iverson; C. L. Krumhansl, Isolaing he dynamic aribues of musical imbre, The J. Acous. Soc. of America, Volume 94, Issue 5, November 1993, pp ISSN: ISN: