Rhythm Analysis in Music

Size: px

Start display at page:

Download "Rhythm Analysis in Music"

Joel Tyler
6 years ago
Views:

1 Rhythm Analysis in Music EECS 352: Machine Percep;on of Music & Audio Zafar Rafii, Winter 24

2 Some Defini;ons Rhythm movement marked by the regulated succession of strong and weak elements, or of opposite or different condi;ons. [OED] Zafar Rafii, Winter 24 2

3 Some Defini;ons Beat Basic unit of ;me in music Zafar Rafii, Winter 24 3

4 Some Defini;ons Tempo Speed or pace of a given piece, typically measured in beats per minute (BPM) Zafar Rafii, Winter 24 4

5 Some Defini;ons Bar (or measure) Segment of ;me defined by a given number of beats A 4- beat measure drum payern. [hyp://en.wikipedia.org/wiki/metre_(music)] Zafar Rafii, Winter 24 5

6 Some Defini;ons Meter (or metre) Organiza;on of music into regularly recurring measures of stressed and unstressed beats Hypermeter: 4- beat measure and 4- measure hypermeasure. Hyperbeats in red. [hyp://en.wikipedia.org/wiki/metre_(music)] Zafar Rafii, Winter 24 6

7 Some Applica;ons Onset detec;on Tempo es;ma;on Beat tracking Higher- level structures Zafar Rafii, Winter 24 7

8 Prac;cal Interest Iden;fy/classify/retrieve by rhythmic similarity Music segmenta;on/summariza;on Audio/video synchroniza;on And source separa;on! Zafar Rafii, Winter 24 8

9 Intellectual Interest Music understanding [Dannenberg, 987] Music percep;on Music cogni;on And Fun! Zafar Rafii, Winter 24 9

10 Onset Detec;on (what?) Iden;fy the star;ng ;mes of musical elements E.g., notes, drum sounds, or any sudden change See novelty curve [Foote, 2] Beginning of Another one bites the dust by Queen. Zafar Rafii, Winter 24

11 Onset Detec;on (how?) Analyze amplitude (drums have high energy!) Analyze other cues (e.g., spectrum, pitch, phase) Analyze self- similarity (see similarity matrix) All the note onsets Beginning of Another one bites the dust by Queen. Zafar Rafii, Winter 24

12 Tempo Es;ma;on (what?) Iden;fy periodic or quasi- periodic payerns Iden;fy some period of repe;;on See beat spectrum [Foote et al., 2] Beginning of Another one bites the dust by Queen. Zafar Rafii, Winter 24 2

13 Tempo Es;ma;on (how?) Analyze periodici;es using the autocorrela6on Compare the onsets with a bank of comb filters Use the Short- Time Fourier Transform (STFT) Tempo at the kick- snare level Beginning of Another one bites the dust by Queen. Zafar Rafii, Winter 24 3

14 Beat Tracking (what?) Iden;fy the beat ;mes Iden;fy the ;mes to which we tap our feet See (also) beat spectrum [Foote et al., 2] Beginning of Another one bites the dust by Queen. Zafar Rafii, Winter 24 4

15 Beat Tracking (how?) Find op;mal beat ;mes given onsets and tempo Use Dynamic Programming [Ellis, 27] Use Mul;- Agent System [Goto, 2] Beats at the kick- snare level Beginning of Another one bites the dust by Queen. Zafar Rafii, Winter 24 5

16 Higher- level Structures (what?) Rhythm, meter, etc. Music understanding See (again) beat spectrum and similarity matrix Beginning of Another one bites the dust by Queen. Zafar Rafii, Winter 24 6

17 Higher- level Structures (how?) Extract onsets, tempo, beat Use/assume addi;onal knowledge E.g., how many beats per measure? Etc. Hi- hat level Beat level Kick level Beginning of Another one bites the dust by Queen. Zafar Rafii, Winter 24 7

18 State- of- the- Art Some interes;ng links Dannenberg s ar;cles on beat tracking: hyp:// beayrack.html Goto s work on beat tracking: hyp://staff.aist.go.jp/m.goto/proj/bts.html Ellis Matlab codes for tempo es;ma;on and beat tracking: hyp://labrosa.ee.columbia.edu/projects/beayrack/ MIREX s annual evalua;on campaign for Music Informa;on Retrieval (MIR) algorithms, including tasks such as onset detec;on, tempo extrac;on, and beat tracking: hyp:// ir.org/mirex/wiki/mirex_home Zafar Rafii, Winter 24 8

19 The Autocorrela;on Func;on Defini;on Cross- correla;on of a signal with itself = measure of self- similarity as a func;on of the ;me lag Beginning of Another one bites the dust by Queen lag (s) Autocorrela;on plot. Zafar Rafii, Winter 24 9

20 The Autocorrela;on Func;on Applica;on Iden;fy repea;ng payerns Iden;fy periodici;es Beginning of Another one bites the dust by Queen. Periodicity of about 4 s lag (s) Autocorrela;on plot. Zafar Rafii, Winter 24 2

21 The Autocorrela;on Func;on Applica;on Iden;fy repea;ng payerns Iden;fy periodici;es Periodic signal + random signal sample lag (sample) Autocorrela;on plot. Zafar Rafii, Winter 24 2

22 The Autocorrela;on Func;on Calcula;on a(j) = /n j i= n j x(i)x(i+j) 6 Signal x sample Autocorrelation a lag (sample) Zafar Rafii, Winter 24 22

23 The Autocorrela;on Func;on Calcula;on a(j) = /n j i= n j x(i)x(i+j) x(i)= samples x(i+)= 3 3 a(j)= 2 3 lags Zafar Rafii, Winter 24 23

24 The Autocorrela;on Func;on Calcula;on a(j) = /n j i= n j x(i)x(i+j) x(i)= x(i+)= 3 3 a(j=)= +9++9/4 =5 samples a(j)= lags Zafar Rafii, Winter 24 24

25 The Autocorrela;on Func;on Calcula;on a(j) = /n j i= n j x(i)x(i+j) x(i)= samples x(i+)= 3 3 a(j)= lags Zafar Rafii, Winter 24 25

26 The Autocorrela;on Func;on Calcula;on a(j) = /n j i= n j x(i)x(i+j) x(i)= x(i+)= 3 3 a(j=)= 3+3+3/3 =3 samples a(j)= lags Zafar Rafii, Winter 24 26

27 The Autocorrela;on Func;on Calcula;on a(j) = /n j i= n j x(i)x(i+j) x(i)= x(i+2)= 3 3 samples a(j)= lags Zafar Rafii, Winter 24 27

28 The Autocorrela;on Func;on Calcula;on a(j) = /n j i= n j x(i)x(i+j) x(i)= x(i+2)= 3 3 a(j=2)= +9/2 =5 samples a(j)= lags Zafar Rafii, Winter 24 28

29 The Autocorrela;on Func;on Calcula;on a(j) = /n j i= n j x(i)x(i+j) x(i)= x(i+3)= 3 3 samples a(j)= lags Zafar Rafii, Winter 24 29

30 The Autocorrela;on Func;on Calcula;on a(j) = /n j i= n j x(i)x(i+j) x(i)= x(i+3)= 3 3 a(j=3)= 3/ =3 samples a(j)= lags Zafar Rafii, Winter 24 3

31 The Autocorrela;on Func;on Calcula;on a(j) = /n j i= n j x(i)x(i+j) x(i)= samples a(j)= lags Zafar Rafii, Winter 24 3

32 The Autocorrela;on Func;on Calcula;on a(j) = /n j i= n j x(i)x(i+j) Periodic sequence of 2 samples Signal x sample Autocorrelation a 5 3 Lag = similarity with itself lag (sample) Period of 2 samples Zafar Rafii, Winter 24 32

33 The Autocorrela;on Func;on Notes The autocorrela;on generally starts at lag = similarity of the signal with itself Wiener- Khinchin Theorem: Power Spectral Density = Fourier Transform of autocorrela;on lag (s) Autocorrela;on plot. Zafar Rafii, Winter 24 33

34 Foote s Beat Spectrum Defini;on Using the autocorrela;on func;on, we can derive the beat spectrum [Foote et al., 2] Beginning of Another one bites the dust by Queen lag (s) Beat Spectrum. Zafar Rafii, Winter 24 34

35 Foote s Beat Spectrum Applica;on The beat spectrum reveals the hierarchically periodically repea;ng structure Beginning of Another one bites the dust by Queen. Periodicity at the measure level Periodicity at the kick level Periodicity at the beat level lag (s) Beat Spectrum. Zafar Rafii, Winter 24 35

36 Foote s Beat Spectrum Calcula;on - Compute the power spectrogram from the audio using the STFT (square of magnitude spectrogram) Audio frequency (khz) x 4 Power spectrogram Zafar Rafii, Winter 24 36

37 Foote s Beat Spectrum Calcula;on 2 x 4 Compute the autocorrela;on of the rows (i.e., the frequency channels) of the spectrogram Power spectrogram 2 x 4 Autocorrelation plots frequency (khz).5.5 frequency (khz) Spectrogram at khz lag (s) Autocorrelation at khz lag (s) Zafar Rafii, Winter 24 37

38 Foote s Beat Spectrum Calcula;on 2 x 4 Compute the mean of the autocorrela;ons (of the rows) Power spectrogram 2 x 4 Autocorrelation plots frequency (khz).5.5 frequency (khz) lag (s) Beat spectrum lag (s) Zafar Rafii, Winter 24 38

39 Foote s Beat Spectrum Notes The first highest peak in the beat spectrum does not always correspond to the repea;ng period! The beat spectrum does not indicate where the beats are or when a measure starts! This is how you find the period lag (s) This is not the period Beat Spectrum. Zafar Rafii, Winter 24 39

40 Foote s Beat Spectrum Notes The beat spectrum can also be calculated using the similarity matrix [Foote et al., 2] A beat spectrogram can also be calculated using successive beat spectra [Foote et al., 2] Beat Spectrogram. 4 lag (s) Zafar Rafii, Winter 24 4

41 Foote s Beat Spectrum Ques;on Can we use the beat spectrum for source separa;on?... To be con;nued Zafar Rafii, Winter 24 4

42 References R. B. Dannenberg, Music Understanding by Computer, 987/988 Computer Science Research Review, Carnegie Mellon School of Computer Science, pp. 9-28, 987. J. Foote, Visualizing Music and Audio using Self- Similarity, in 7 th ACM Interna6onal Conference on Mul6media (Part ), Orlando, FL, USA, pp. 77-8, October 3- November 5, 999. J. Foote, Automa;c Audio Segmenta;on using a Measure of Audio Novelty, in IEEE Interna6onal Conference on Mul6media and Expo, New York, NY, USA, vol., pp , July 3- August 2, 2. J. Foote and S. Uchihashi, The Beat Spectrum: A New Approach to Rhythm Analysis, in IEEE Interna6onal Conference on Mul6media and Expo, Tokyo, Japan, pp , August 22-25, 2. M. Goto, An Audio- based Real- ;me Beat Tracking System for Music With or Without Drum- sounds, Journal of New Music Research, vol. 3, no. 2, pp. 59-7, 2. D. P. W. Ellis, Beat Tracking by Dynamic Programming, Journal of New Music Research, vol. 36, no., pp. 5-6, 27. M. Müller, D. P. W. Ellis, A. Klapuri, and G. Richard, Signal Processing for Music Analysis, IEEE Journal of Selected Topics in Signal Processing, vol. 5, no. 6, pp. 88-, October 2. Wikipedia, Rhythm, hyp://en.wikipedia.org/wiki/rhythm, 22. Wikipedia, Meter, hyp://en.wikipedia.org/wiki/metre_(music), 22. Zafar Rafii, Winter 24 42

43 The Similarity Matrix Defini;on Matrix where each point measures the similarity between any two elements of a given sequence 8 Similarity matrix Zafar Rafii, Winter 24 43

44 The Similarity Matrix Applica;on Visualize ;me structure [Foote, 999] Iden;fy repea;ng/similar payerns Similarity between the ;mes at 2 s and s. Region of high self- similarity around 3 s. Region around 3 s repea;ng around 8 s, 2 s, and 7 s Similarity matrix Zafar Rafii, Winter Very similar Very dissimilar

45 The Similarity Matrix Calcula;on The similarity matrix S of X is basically the matrix mul;plica;on between transposed X and X, ayer (generally) normaliza;on of the columns of X S( j, j 2 ) = k= n X(k, j )X(k, j 2 ) / k= n X(k, j ) 2 k Zafar Rafii, Winter 24 45

46 The Similarity Matrix Calcula;on - Compute the magnitude spectrogram from the audio using the STFT Audio frequency (khz) x 4 Magnitude spectrogram Zafar Rafii, Winter 24 46

47 The Similarity Matrix Calcula;on 2 x 4 Normalize the columns of the spectrogram by dividing them by their Euclidean norm Magnitude spectrogram 2 x 4 Normalized spectrogram frequency (khz).5.5 X frequency (khz).5.5 X X j X j X j (i)= X j (i)/ k= n X j (k) 2 Zafar Rafii, Winter 24 47

The Similarity Matrix frequency (khz) Calcula;on 2.5.

48 The Similarity Matrix frequency (khz) Calcula;on x 4 Compute the dot product between any two pairs of columns and save them in the similarity matrix Normalized spectrogram X j X X j 2 j Similarity matrix S 2 S( j, j 2 ) = k= n X j (k) X j 2 (k) j 2 Zafar Rafii, Winter 24 48

The Similarity Matrix coefficients 2 Notes 8 6 4 2 The similarity matrix can also be built from

, MFCCs, chromagram, pitch contour) The similarity matrix can also be built using other

49 The Similarity Matrix coefficients 2 Notes The similarity matrix can also be built from other features (e.g., MFCCs, chromagram, pitch contour) The similarity matrix can also be built using other MFCC-based similarity matrix 8 measures (e.g., Euclidean distance) Mel-Frequency Cepstrum Coefficients (MFCC) Zafar Rafii, Winter

Rhythm Analysis in Music

Rhythm Analysis in Music EECS 352: Machine Perception of Music & Audio Zafar Rafii, Winter 24 Some Definitions Rhythm movement marked by the regulated succession of strong and weak elements, or of opposite