Rhythm Analysis in Music EECS 352: Machine Percep;on of Music & Audio Zafar Rafii, Winter 24
Some Defini;ons Rhythm movement marked by the regulated succession of strong and weak elements, or of opposite or different condi;ons. [OED] Zafar Rafii, Winter 24 2
Some Defini;ons Beat Basic unit of ;me in music Zafar Rafii, Winter 24 3
Some Defini;ons Tempo Speed or pace of a given piece, typically measured in beats per minute (BPM) Zafar Rafii, Winter 24 4
Some Defini;ons Bar (or measure) Segment of ;me defined by a given number of beats A 4- beat measure drum payern. [hyp://en.wikipedia.org/wiki/metre_(music)] Zafar Rafii, Winter 24 5
Some Defini;ons Meter (or metre) Organiza;on of music into regularly recurring measures of stressed and unstressed beats Hypermeter: 4- beat measure and 4- measure hypermeasure. Hyperbeats in red. [hyp://en.wikipedia.org/wiki/metre_(music)] Zafar Rafii, Winter 24 6
Some Applica;ons Onset detec;on Tempo es;ma;on Beat tracking Higher- level structures Zafar Rafii, Winter 24 7
Prac;cal Interest Iden;fy/classify/retrieve by rhythmic similarity Music segmenta;on/summariza;on Audio/video synchroniza;on And source separa;on! Zafar Rafii, Winter 24 8
Intellectual Interest Music understanding [Dannenberg, 987] Music percep;on Music cogni;on And Fun! Zafar Rafii, Winter 24 9
Onset Detec;on (what?) Iden;fy the star;ng ;mes of musical elements E.g., notes, drum sounds, or any sudden change See novelty curve [Foote, 2] -.5.5 2 2.5 Beginning of Another one bites the dust by Queen. Zafar Rafii, Winter 24
Onset Detec;on (how?) Analyze amplitude (drums have high energy!) Analyze other cues (e.g., spectrum, pitch, phase) Analyze self- similarity (see similarity matrix) All the note onsets -.5.5 2 2.5 Beginning of Another one bites the dust by Queen. Zafar Rafii, Winter 24
Tempo Es;ma;on (what?) Iden;fy periodic or quasi- periodic payerns Iden;fy some period of repe;;on See beat spectrum [Foote et al., 2] -.5.5 2 2.5 Beginning of Another one bites the dust by Queen. Zafar Rafii, Winter 24 2
Tempo Es;ma;on (how?) Analyze periodici;es using the autocorrela6on Compare the onsets with a bank of comb filters Use the Short- Time Fourier Transform (STFT) Tempo at the kick- snare level -.5.5 2 2.5 Beginning of Another one bites the dust by Queen. Zafar Rafii, Winter 24 3
Beat Tracking (what?) Iden;fy the beat ;mes Iden;fy the ;mes to which we tap our feet See (also) beat spectrum [Foote et al., 2] -.5.5 2 2.5 Beginning of Another one bites the dust by Queen. Zafar Rafii, Winter 24 4
Beat Tracking (how?) Find op;mal beat ;mes given onsets and tempo Use Dynamic Programming [Ellis, 27] Use Mul;- Agent System [Goto, 2] Beats at the kick- snare level -.5.5 2 2.5 Beginning of Another one bites the dust by Queen. Zafar Rafii, Winter 24 5
Higher- level Structures (what?) Rhythm, meter, etc. Music understanding See (again) beat spectrum and similarity matrix -.5.5 2 2.5 Beginning of Another one bites the dust by Queen. Zafar Rafii, Winter 24 6
Higher- level Structures (how?) Extract onsets, tempo, beat Use/assume addi;onal knowledge E.g., how many beats per measure? Etc. Hi- hat level Beat level Kick level -.5.5 2 2.5 Beginning of Another one bites the dust by Queen. Zafar Rafii, Winter 24 7
State- of- the- Art Some interes;ng links Dannenberg s ar;cles on beat tracking: hyp://www.cs.cmu.edu/~rbd/bib- beayrack.html Goto s work on beat tracking: hyp://staff.aist.go.jp/m.goto/proj/bts.html Ellis Matlab codes for tempo es;ma;on and beat tracking: hyp://labrosa.ee.columbia.edu/projects/beayrack/ MIREX s annual evalua;on campaign for Music Informa;on Retrieval (MIR) algorithms, including tasks such as onset detec;on, tempo extrac;on, and beat tracking: hyp://www.music- ir.org/mirex/wiki/mirex_home Zafar Rafii, Winter 24 8
The Autocorrela;on Func;on Defini;on Cross- correla;on of a signal with itself = measure of self- similarity as a func;on of the ;me lag Beginning of Another one bites the dust by Queen. - 2 4 6 8 2 4 6 8.5 2 4 6 8 2 4 6 8 lag (s) Autocorrela;on plot. Zafar Rafii, Winter 24 9
The Autocorrela;on Func;on Applica;on Iden;fy repea;ng payerns Iden;fy periodici;es Beginning of Another one bites the dust by Queen. Periodicity of about 4 s - 2 4 6 8 2 4 6 8.5 2 4 6 8 2 4 6 8 lag (s) Autocorrela;on plot. Zafar Rafii, Winter 24 2
The Autocorrela;on Func;on Applica;on Iden;fy repea;ng payerns Iden;fy periodici;es Periodic signal + random signal..5 2 3 4 5 6 7 8 9 sample.5 2 3 4 5 6 7 8 9 lag (sample) Autocorrela;on plot. Zafar Rafii, Winter 24 2
The Autocorrela;on Func;on Calcula;on a(j) = /n j i= n j x(i)x(i+j) 6 Signal x 4 2 3.5.5 2 2.5 3 3.5 4 4.5 sample 3 6 4 2 5 3 Autocorrelation a 5 3 -.5.5.5 2 2.5 3 3.5 lag (sample) Zafar Rafii, Winter 24 22
The Autocorrela;on Func;on Calcula;on a(j) = /n j i= n j x(i)x(i+j) x(i)= samples 2 3 4 3 3 x(i+)= 3 3 a(j)= 2 3 lags Zafar Rafii, Winter 24 23
The Autocorrela;on Func;on Calcula;on a(j) = /n j i= n j x(i)x(i+j) x(i)= 2 3 4 3 3 x(i+)= 3 3 a(j=)= +9++9/4 =5 samples a(j)= 5 2 3 lags Zafar Rafii, Winter 24 24
The Autocorrela;on Func;on Calcula;on a(j) = /n j i= n j x(i)x(i+j) x(i)= samples 2 3 4 3 3 x(i+)= 3 3 a(j)= 5 2 3 lags Zafar Rafii, Winter 24 25
The Autocorrela;on Func;on Calcula;on a(j) = /n j i= n j x(i)x(i+j) x(i)= 2 3 4 3 3 x(i+)= 3 3 a(j=)= 3+3+3/3 =3 samples a(j)= 5 3 2 3 lags Zafar Rafii, Winter 24 26
The Autocorrela;on Func;on Calcula;on a(j) = /n j i= n j x(i)x(i+j) x(i)= x(i+2)= 3 3 samples 2 3 4 3 3 a(j)= 5 3 2 3 lags Zafar Rafii, Winter 24 27
The Autocorrela;on Func;on Calcula;on a(j) = /n j i= n j x(i)x(i+j) x(i)= x(i+2)= 3 3 a(j=2)= +9/2 =5 samples 2 3 4 3 3 a(j)= 5 3 5 2 3 lags Zafar Rafii, Winter 24 28
The Autocorrela;on Func;on Calcula;on a(j) = /n j i= n j x(i)x(i+j) x(i)= x(i+3)= 3 3 samples 2 3 4 3 3 a(j)= 5 3 5 2 3 lags Zafar Rafii, Winter 24 29
The Autocorrela;on Func;on Calcula;on a(j) = /n j i= n j x(i)x(i+j) x(i)= x(i+3)= 3 3 a(j=3)= 3/ =3 samples 2 3 4 3 3 a(j)= 5 3 5 3 2 3 lags Zafar Rafii, Winter 24 3
The Autocorrela;on Func;on Calcula;on a(j) = /n j i= n j x(i)x(i+j) x(i)= samples 2 3 4 3 3 a(j)= 5 3 5 3 2 3 lags Zafar Rafii, Winter 24 3
The Autocorrela;on Func;on Calcula;on a(j) = /n j i= n j x(i)x(i+j) Periodic sequence of 2 samples 6 4 2 3 Signal x.5.5 2 2.5 3 3.5 4 4.5 sample 3 6 4 2 5 3 Autocorrelation a 5 3 Lag = similarity with itself -.5.5.5 2 2.5 3 3.5 lag (sample) Period of 2 samples Zafar Rafii, Winter 24 32
The Autocorrela;on Func;on Notes The autocorrela;on generally starts at lag = similarity of the signal with itself Wiener- Khinchin Theorem: Power Spectral Density = Fourier Transform of autocorrela;on.5 2 4 6 8 2 4 6 8 lag (s) Autocorrela;on plot. Zafar Rafii, Winter 24 33
Foote s Beat Spectrum Defini;on Using the autocorrela;on func;on, we can derive the beat spectrum [Foote et al., 2] Beginning of Another one bites the dust by Queen. - 2 4 6 8 2 4 6 8.5 2 4 6 8 2 4 6 lag (s) Beat Spectrum. Zafar Rafii, Winter 24 34
Foote s Beat Spectrum Applica;on The beat spectrum reveals the hierarchically periodically repea;ng structure Beginning of Another one bites the dust by Queen. Periodicity at the measure level Periodicity at the kick level - 2 4 6 8 2 4 6 8 Periodicity at the beat level.5 2 4 6 8 2 4 6 lag (s) Beat Spectrum. Zafar Rafii, Winter 24 35
Foote s Beat Spectrum Calcula;on - Compute the power spectrogram from the audio using the STFT (square of magnitude spectrogram) Audio 2 4 6 8 2 4 6 8 frequency (khz) 2.5.5 x 4 Power spectrogram 2 4 6 8 2 4 6 8 Zafar Rafii, Winter 24 36
Foote s Beat Spectrum Calcula;on 2 x 4 Compute the autocorrela;on of the rows (i.e., the frequency channels) of the spectrogram Power spectrogram 2 x 4 Autocorrelation plots frequency (khz).5.5 frequency (khz).5.5 2 2 4 6 8 2 4 6 8 Spectrogram at khz 2 4 6 8 2 4 6 lag (s) Autocorrelation at khz.5 2 4 6 8 2 4 6 8 2 4 6 8 2 4 6 lag (s) Zafar Rafii, Winter 24 37
Foote s Beat Spectrum Calcula;on 2 x 4 Compute the mean of the autocorrela;ons (of the rows) Power spectrogram 2 x 4 Autocorrelation plots frequency (khz).5.5 frequency (khz).5.5 2 4 6 8 2 4 6 8 2 4 6 8 2 4 6 lag (s) Beat spectrum.5 2 4 6 8 2 4 6 lag (s) Zafar Rafii, Winter 24 38
Foote s Beat Spectrum Notes The first highest peak in the beat spectrum does not always correspond to the repea;ng period! The beat spectrum does not indicate where the beats are or when a measure starts! This is how you find the period.5 2 4 6 8 2 4 6 lag (s) This is not the period Beat Spectrum. Zafar Rafii, Winter 24 39
Foote s Beat Spectrum Notes The beat spectrum can also be calculated using the similarity matrix [Foote et al., 2] A beat spectrogram can also be calculated using successive beat spectra [Foote et al., 2] Beat Spectrogram. 4 lag (s) 3 2 2 4 6 8 2 4 6 8 Zafar Rafii, Winter 24 4
Foote s Beat Spectrum Ques;on Can we use the beat spectrum for source separa;on?... To be con;nued Zafar Rafii, Winter 24 4
References R. B. Dannenberg, Music Understanding by Computer, 987/988 Computer Science Research Review, Carnegie Mellon School of Computer Science, pp. 9-28, 987. J. Foote, Visualizing Music and Audio using Self- Similarity, in 7 th ACM Interna6onal Conference on Mul6media (Part ), Orlando, FL, USA, pp. 77-8, October 3- November 5, 999. J. Foote, Automa;c Audio Segmenta;on using a Measure of Audio Novelty, in IEEE Interna6onal Conference on Mul6media and Expo, New York, NY, USA, vol., pp. 452-455, July 3- August 2, 2. J. Foote and S. Uchihashi, The Beat Spectrum: A New Approach to Rhythm Analysis, in IEEE Interna6onal Conference on Mul6media and Expo, Tokyo, Japan, pp. 88-884, August 22-25, 2. M. Goto, An Audio- based Real- ;me Beat Tracking System for Music With or Without Drum- sounds, Journal of New Music Research, vol. 3, no. 2, pp. 59-7, 2. D. P. W. Ellis, Beat Tracking by Dynamic Programming, Journal of New Music Research, vol. 36, no., pp. 5-6, 27. M. Müller, D. P. W. Ellis, A. Klapuri, and G. Richard, Signal Processing for Music Analysis, IEEE Journal of Selected Topics in Signal Processing, vol. 5, no. 6, pp. 88-, October 2. Wikipedia, Rhythm, hyp://en.wikipedia.org/wiki/rhythm, 22. Wikipedia, Meter, hyp://en.wikipedia.org/wiki/metre_(music), 22. Zafar Rafii, Winter 24 42
The Similarity Matrix Defini;on Matrix where each point measures the similarity between any two elements of a given sequence 8 Similarity matrix. 6.9 4 2 8 6 4 2.8.7.6.5.4.3.2. 2 4 6 8 2 4 6 8 Zafar Rafii, Winter 24 43
The Similarity Matrix Applica;on Visualize ;me structure [Foote, 999] Iden;fy repea;ng/similar payerns Similarity between the ;mes at 2 s and s. Region of high self- similarity around 3 s. Region around 3 s repea;ng around 8 s, 2 s, and 7 s. 8 6 4 2 8 6 4 2 Similarity matrix. 2 4 6 8 2 4 6 8 Zafar Rafii, Winter 24 44.9.8.7.6.5.4.3.2. Very similar Very dissimilar
The Similarity Matrix Calcula;on The similarity matrix S of X is basically the matrix mul;plica;on between transposed X and X, ayer (generally) normaliza;on of the columns of X S( j, j 2 ) = k= n X(k, j )X(k, j 2 ) / k= n X(k, j ) 2 k Zafar Rafii, Winter 24 45
The Similarity Matrix Calcula;on - Compute the magnitude spectrogram from the audio using the STFT Audio 2 4 6 8 2 4 6 8 frequency (khz) 2.5.5 x 4 Magnitude spectrogram 2 4 6 8 2 4 6 8 Zafar Rafii, Winter 24 46
The Similarity Matrix Calcula;on 2 x 4 Normalize the columns of the spectrogram by dividing them by their Euclidean norm Magnitude spectrogram 2 x 4 Normalized spectrogram frequency (khz).5.5 X frequency (khz).5.5 X 2 4 6 8 2 4 6 8 X j 2 4 6 8 2 4 6 8 X j X j (i)= X j (i)/ k= n X j (k) 2 Zafar Rafii, Winter 24 47
The Similarity Matrix frequency (khz) Calcula;on 2.5.5 x 4 Compute the dot product between any two pairs of columns and save them in the similarity matrix Normalized spectrogram 2 4 6 8 2 4 6 8 X j X X j 2 j 8 6 4 2 8 6 4 Similarity matrix S 2 S( j, j 2 ) = k= n X j (k) X j 2 (k) 2 4 6 8 2 4 6 8 j 2 Zafar Rafii, Winter 24 48
The Similarity Matrix coefficients 2 Notes 8 6 4 2 The similarity matrix can also be built from other features (e.g., MFCCs, chromagram, pitch contour) The similarity matrix can also be built using other MFCC-based similarity matrix 8 measures (e.g., Euclidean distance) Mel-Frequency Cepstrum Coefficients (MFCC) 2 4 6 8 2 4 6 8 6 4 2 8 6 4 2 Zafar Rafii, Winter 24 49 2 4 6 8 2 4 6 8