Mid-level representations for audio content analysis *Slides for this lecture were created by Anssi Klapuri

Transcription

1 Mid-level represetatios for audio cotet aalysis *Slides for this lecture were created by Assi Klapuri Represetatios 1 Sources: Ellis, Rosethal, Mid-level represetatios for Computatioal Auditory Scee Aalysis, IJCAI, Schörkhuber et al. A Matlab Toolbox for Efficiet Perfect Recostructio Time-Frequecy Trasforms with Log-Frequecy Resolutio, AES Virtae, Audio Sigal Modelig with Siusoids plus Noise, MSc thesis, TUT, Cotets Itroductio Desirable properties of mid-level represetatios STFT spectrogram Costat-Q trasform Siusoids plus oise Perceptually-motivated represetatios 1 Itroductio The cocept of mid-level data represetatios is useful i characterizig sigal aalysis systems Represetatios 2 The aalysis process ca be viewed as a sequece of represetatios from a acoustic sigal ( low level ) towards the aalysis result ( high ) Usually itermediate abstractio levels are eeded betwee the two sice high-level iformatio is ot readily visible i the raw acoustic sigal A appropriate mid-level represetatio fuctios as a iterface for further aalysis ad facilitates the desig of efficiet algorithms Desirable properties of mid-level represetatios Represetatios 3 It is atural to ask if a certai mid-level represetatio is better tha others i a give task. Ellis ad Rosethal list several desirable qualities for a midlevel represetatio: Compoet reductio: the umber of objects i the represetatio is smaller ad the meaigfuless of each is higher compared to the idividual samples of the iput sigal The soud should be decomposed ito sufficietly fie-graied elemets so as to support soud source separatio by groupig the elemets to their soud sources. Ivertibility: the origial soud ca be resythesized from its represetatio i a perceptually accurate way, Psychoacoustic plausibility of the represetatio. Categorizatio of some mid-level represetatios Represetatios 4 Ellis ad Rosethal classify represetatios accordig to three coceptual axes choice betwee fixed ad variable badwidth of the iitial frequecy aalysis discreteess: is represetatio structured as meaigful chuks? dimesioality of the trasform (some possess extra dimesios) Figure: [Ellis-Rosethal]

2 2 Complex-valued STFT spectrogram Represetatios 5 STFT spectrogram Represetatios 6 STFT = short-time Fourier trasform Time-domai sigal x() is trasformed ito timefrequecy domai by employig discrete Fourier trasform (DFT) i successive time frames complex spectra: all iformatio is preserved amout of data remais the same To some extet fulfills the criteria of supportig soud source separatio (sources overlap less i timefrequecy tha i time domai) ivertibility: the origial soud ca be perfectly recostructed psychoacoustic plausibility: frequecy aalysis (albeit i a differet form) happes i the auditory system too Example sigal (music) top: time-domai (zoomed i) middle: time-domai bottom: STFT spectrogram (magitudes) Spectrum estimatio Represetatios 7 Widowig Represetatios 8 Spectrum of audio sigals is typically estimated i short cosecutive segmets, frames Why? the Fourier trasform models the sigal with statioary siusoids (costat spectrum) real audio sigals are ot statioary but vary through time framewise processig assumes the sigal is time-ivariat i short eough frames For audio sigals, the frame legth typically varies betwee 10ms 100ms, depedig o the applicatio for speech sigals ofte 25ms Trasiet-like souds are difficult to represet ad process i the frequecy domai time blurrig (but let s see costat-q trasform later...) Widowig is essetial i frame-wise processig weight the sigal with a widow fuctio w(k) prior to trasform as a rule of thumb, widowig is always eeded: oe caot just take a short part of a sigal without widowig sigal i frame m: widowed sigal: short-time spectrum: N 1 x m ( ), 0,..., N 1 x m ( ) w( ) k X ( k) x ( ) w( ) W m 0 m

3 Widowig Represetatios 9 Widowig i framewise processig Represetatios 10 Example: spectrum of a siusoid with/without widowig 1. No widowig (=rectagular widow), siusoid at a spectral bi 2. No widowig, radom off-bi frequecy spectral blurrig! 3. Haig widow, siusoid at a spectral bi 4. Haig widow, radom off-bi frequecy ok There are differet types of widows, but most importat is ot to forget widowig altogether Figure: Haig widows (Hammig works too) adjacet widows sum to uity whe frames overlap 50% all parts of the sigal get a equal weight framewise processig time (ms) I each frame, the sigal is weighted with the widow fuctio ad short-time discrete Fourier trasform is calculated This yields a spectrogram complex spectrum i each frame over time frequecy time Widowig i aalysis-sythesis systems Represetatios 11 Sie widow is useful i aalysis-sythesis systems (see Figure) Widowig is doe agai i resythesis to avoid artefacts at frame boudaries i the case that the sigal is maipulated i the f-domai Figure below: 50% frame overlap leads to perfect recostructio if othig is doe at subbads sigal i oe frame widowig DFT... processig at subbads (freq. domai)... iverse DFT widowig output (overlap-add frames) Recostructig the time domai sigal: overlap-add techique Represetatios 12 Recostructig a sigal from its spectrogram: 1. iverse Fourier trasform the spectrum of each frame back to time domai 2. apply widowig i each frame (e.g. sie or Haig widow) 3. successive frames are positioed to overlap 50% or more, ad summed sample-by-sample

4 3 Costat-Q trasform (CQT) Represetatios 13 Costat-Q trasform (CQT) Represetatios 14 Time-frequecy represetatio where the frequecy bis are uiformly distributed i log-frequecy ad their Q-factors (ratios of ceter frequecies to badwidths) are all equal I effect, that meas that the frequecy resolutio is better for low frequecies ad the time resolutio is better for high frequecies Musically ad perceptually motivated frequecy resolutio of the ier ear is approx. costat Q above 500 Hz i music (equal temperamet), ote frequecies obey F k = 440Hz 2 k/12 (piao otes) Mathematical defiitio: -1 (, ) = ( ) ( - )e - 2 / =0 where N is legth of iput sigal, g k (m) is zero-cetered widow fuctio that picks oe time frame of the sigal at poit ad f s sample rate Compare CQT with short-time Fourier trasform (STFT): -1 (, )= ( ) ( - ) - 2 / =0 where ow the widow fuctio h(m) is the same for all frequecy bis I CQT, to achieve costat Q-factors, the support of the widow (time legth of sigificat o-zero values) is iversely proportioal to f k I CQT, the ceter frequecies are geometrically spaced: = 0 2 / where B determies the umber of bis per octave ad f 0 is lowest bi I DFT, the ceter frequecies are liearly spaced: = Costat-Q trasform Represetatios 15 Toolbox for computig CQT Represetatios 16 Time-domai widow fuctio Frequecy resolutio CQT is essetially a wavelet trasform, but with rather high frequecy resolutio (typically bis/octave) covetioal wavelet trasform techiques caot be used Matlab Toolbox for CQT ad ICQT [Schörkhuber et al. 2013]: Efficiet computatio achieved by 1. FFT of the etire iput sigal 2. apply oe CQT frequecy-bi wide badpass o the (huge) spectrum 3. move subbad aroud zero 4. iverse-fft trasform the arrowbad spectrum to get CQT coefficiets over time for that bi

5 Toolbox for computig CQT Represetatios 17 Represetatios 18 Due to the way that CQT is computed by the toolbox, the frequecydomai respose of a idividual frequecy bi ca be cotrolled perfectly (o sidelobes), but the effective time-domai widow has sidelobes (that exted over the etire sigal) Figure: the respose of oe time-frequecy elemet as a fuctio of frequecy (left) ad as a fuctio of time (right) STFT spectrogram for the same sigal (either high or low frequecies blur) Represetatios 19 STFT spectrogram for the same sigal (either high or low frequecies blur) Represetatios 20

6 Costat-Q trasform Represetatios 21 Drawbacks of CQT Represetatios 22 A Reasos why CQT has ot more widely replaced FFT i audio sigal processig: 1. CQT is computatioally more itesive tha DFT spectrogram 2. CQT produces a data structure that is more difficult to hadle tha the time-frequecy matrix (spectrogram) Example applicatio: Pitch shiftig Represetatios 23 4 Siusoids plus oise model (Recap from SGN-14006) Represetatios 24 The toolbox icludes PITCHSHIFT.m to implemet that Pitch shiftig is a atural operatio i CQT domai: 1. CQT 2. traslate CQT coefficiets or i frequecy 3. retai phase coherece 4. iverse CQT Examples: origial - 6 semitoes +6 semitoes Trasiets at high freqs. are retaied due to short frame Time stretchig ca be doe by pitch shiftig + resamplig Sigal model x( t) N 1 a ( t)cos 2 f ( t) t ( t) r( t) sigal x(t) is represeted with N siusoids (freq, amplitude, phase) ad oise residual r(t) Additive sythesis accordig to Fourier theorem, ay sigal ca be represeted as a sum of siusoids makes sese oly for periodic sigals, for which the amout of siusoids eeded is small o-determiistic part would require a large umber of siusoids use stochastic modelig

7 Represetatios 25 Represetatios 26 Represetatios 27 Represetatios 28

8 Represetatios 29 Represetatios 30 Represetatios 31 Siusoids+oise model Aalysis Represetatios 32 Block diagram [Virtae 2001] 1. detect siusoids i framewise spectra 2. estimate siusoid parameters ad resythesize 3. subtract siusoids from origial sigal 4. model the oise residual We get siusoid parameters oise level at differet subbads

9 Siusoids+oise model Detectig ad estimatig siusoids Represetatios 33 Siusoids+oise model Trackig the peaks Represetatios 34 Block diagram: [Virtae01] Spectral peaks are iterpreted as siusoids 1. peak : local maximum i magitude spectrum 2. peak frequecy, amplitude, ad phase ca be picked from the complex spectrum Trackig the peaks detected i successive frames gives parameters of a time-varyig siusoid siusoidal trajectory time If eeded, spectral peaks i successive frames ca be associated ad joied ito time-vayrig siusoids frequecy, amplitude, ad phases joied ito curves Figure: peak trackig algorithm [Virtae2001] based e.g. o the track derivatives; try to form a smooth track kill: if o cotiuatio foud, ed the siusoid birth: if spectral peak is ot a cotiuatio for a existig siusoid, create a ew oe Siusoids+oise model Sythesis of siusoids Represetatios 35 Siusoids+oise model Sythesis, subtractio from origial Represetatios 36 Additive sythesis s( t) N 1 a ( t)cos 2 f ( t) t Ofte trackig the peaks is ot ecessary, but sythesize siusoids i each frame separately, keep the parameters fixed i oe frame widow the obtaied sigal with Ha widow overlap-add ( t) Sythesized siusoids vs. the origial sigal (upper pael) Residual obtaied from subtractio (lower pael)

10 Siusoids+oise model Modelig the oise residual Represetatios 37 Siusoids+oise model Noise sythesis from parameters Represetatios 38 Residual is obtaied by subtactig sythesized siusoids from the origial sigal i the time-domai Residual sigal is aalyzed frame-by-frame calculate spectrum R t (f) i frame t subdivide the spectrum ito 25 perceptual subbads (Bark scale) calculate short-time eergy at each bad b,b=1,2,...,25 E ( b) t f b R t f 2 Noise residual is represeted parametrically i each frame, store oly the short-time eergies withi Bark bads, E t (b) this modelig ca be doe, because the auditory system is ot sesitive to eergy chages withi oe Bark bad i the case of oise R ( f ) t E t Sythesis 1. geerate magitude spectrum, where the eergy withi each Bark bad is shared uiformly withi the bad 2. geerate radom phases 3. iverse Fourier trasform to time-domai 4. widowig with Ha widow 5. overlap-add Siusoids+oise model Properties Represetatios 39 Itelliget compoet groupig Represetatios 40 Audio examples: Siusoids+oise model has several ice properties satisfies the compoet reductio property (see slide 3) ivertibility: sythesized sigal has reasoable quality (trasiet souds are problematic) the model is geeric: ay soud ca be processed straightforward to compute (especially if peak trackig is skipped) maipulatio such as time stretchig ad pitch shiftig is easy The represetatio also supports souds source separatio to some extet, see ext slides Auditory orgaizatio i humas has bee foud to deped o certai acoustic cues Two compoets may be grouped, i.e., associated to a commo soud source by 1. Spectral proximity (time, frequecy) 2. Harmoic cocordace frequecies compoets i itegral ratios these compoets are deduced to be produced by a commo source 3. Sychroous chages of the compoets commo oset / offset commo AM / FM modulatio equidirectioal movemet i the spectrum 4. Spatial proximity (agle of arrival) Cues may compete ad coflict

11 Example: groupig siusoids Represetatios 41 Soud separatio Example: groupig implemeted Represetatios 42 Siusoidal model reveals the cues better tha the timedomai sigal Estimate perceptual distace betwee each two spectral compoets siusoids are classified ito groups origial sies Soud separatio Example: groupig implemeted Represetatios 43 5 Perceptually-motivated represetatios Represetatios 44 Classified sets of siusoids ca be sythesized separately Peripheral hearig 1. Frequecy selectivity of the ier ear Bak of liear badpass filters auditory chaels 2. Mechaical-to-eural trasductio Compressio, rectificatio, lowpass filterig Detailed models exist, too I brai, for pitch processig: 3. Periodicity aalysis withi chaels 4. Combiatio across chaels Betwee-chael phase differeces do ot affect Sigal i the auditory erve ot directly observable (yet) Brai

12 Perceptually-motivated represetatios Represetatios 45 The sigal travelig i the auditory erve fibers from the auditory periphery to the brai ca be viewed as a midlevel represetatio The idea of usig the same data represetatio as the huma auditory system is very appealig Auditory periphery is quite accurately kow Auditory filterbak Represetatios 46 Bad-wise processig is a iheret part of hearig Figure: Frequecy resposes (top) ad impulse respose (bottom) of a few auditory filters bc fc 24.7Hz Badwidths proportioal to ceter frequecy: Gammatoe filters [Slaey93] Mechaical-to-eural trasductio Simplified model: a. Compressio (ad level adaptatio) b. Half-wave rectificatio c. Lowpass filterig Represetatios 47 Mechaical-to-eural trasductio Half-wave rectificatio Represetatios 48 Half-wave rectificatio withi subbads iput partials + beatig partials (freq. itervals btw the iput partials) Compressio: Memoryless: scale the sigal with a factor a c = ( c ) 1 where c is the std of the sigal withi chael c Spectral flatteig (whiteig) whe 0< <1

13 Mechaical-to-eural trasductio Half-wave rectificatio Represetatios 49 Autocorrelatio withi chaels Represetatios 50 Rectificatio maps the cotributio of higher-order partials to the positio of the F0 ad its few multiples i the spectrum The extet to which harmoic h is mapped to the positio of the fudametal icreases as a fuctio of h Harmoic soud Amplitude-modulated oise Autocorrelatio: Summary autocorrelatio: combiig across chaels Correlogram Represetatios 51 Performig periodicity aalysis withi critical bads produces a threedimesioal volume r c (, for chael c at time ad lag Figure below illustrates the correlogram Iput sigal was a trumpet soud with F0 260 Hz (period 3.8 ms) left: the 3D correlogram volume. middle: zero-lag face of the correlogram (= power spectrogram) right: oe time slice of the volume, from which summary ACF ca be obtaied by summig over frequecy.