Preeti Rao 2 nd CompMusicWorkshop, Istanbul 2012
o Music signal characteristics o Perceptual attributes and acoustic properties o Signal representations for pitch detection o STFT o Sinusoidal model o Pitch detection algorithms o Polyphonic context and predominant pitch tracking o Applications in MIR 2
Digital audio format: PCM Sampling rate: 44.1 khz, 22.05 khz Amplitude resolution: 16 bits/sample *The Physics Classroom:http://www.glenbrook.k12.il.us/gbssci/ phys/class/sound/u11l2a.html WiSSAP 2007
Interesting sounds are typically coded in the form of a temporal sequence of atomic sound events. E.g. speech -> a sequence of phones music -> an evolving pattern of notes An atomic sound event, or a single gestalt, can be a complex acoustical signal described by a set of temporal and spectral properties => an evoked sensation. Department of Electrical Engineering, IIT Bombay
A sound of given frequency components and sound pressure levels leads to perceived sensations that can be distinguished in terms of: o loudness <-- intensity o pitch <-- fundamental frequency o timbre ( quality or colour ) <--ther spectro-temporal properties Department of Electrical Engineering, IIT Bombay
low pitch tone Air pressure variation T0 = 10 msec high pitch tone Frequency = 100 Hz 1 Hertz = 1 vibration/sec Frequency = 300 Hz T0 = 3.3 msec Department of Electrical Engineering, IIT Bombay
Musical pitch scale low pitch high pitch semitone = 2 1/12 Department of Electrical Engineering, IIT Bombay
o The construction of a musical scale is based on two assumptions about the human hearing process: o The ear is sensitive to ratios of fundamental frequencies (pitches), not so much to absolute pitch. o The preferred musical intervals, i.e. those perceived to be most consonant, are the ratios of small whole numbers. o A musical sound is typically comprised of severalfrequencies. The frequencies are evident if we observe the spectrum of the sound Department of Electrical Engineering, IIT Bombay
300 Hz 600 Hz 900 Hz 300 Hz + 600Hz 300 Hz + 600Hz + 900Hz Department of Electrical Engineering, IIT Bombay
Sound atoms : Single tone signal x 1 ( t) X 1 ( f ) 0.7 0.8 0 t(ms) 50 f (Hz) -0.6 500
Non-tonal Signal x 2 ( t) X 2 ( f ) 0.7 0.2 0 t(ms) 50 f (Hz) -0.5 500
Complex tone signal x 3 ( t) 0.5 X 3 ( f ) 0.2 0 t(ms) 50 f (Hz) -0.4 500 1000
Bandpass noise signal x 4 ( t) X 4 ( f ) 0.3 1 0-0.3 t(ms) 50 250 800 f (Hz)
A flute note x 1 ( t) X 1 ( f )db 0.5-20 0-0.5 t(ms) 50-70 5 f (khz)
o We see that the distinctive signal characteristics are more evident in the frequency domain. o The ear is a frequency analyzer. It represents a unique combination of analysis and synthesis => we do not perceive spectral components but rather the composite sounds. o We observe that a single note is perceived as one entity of well-defined subjective sensations. This is due to the spatial pattern recognition process achieved by the central auditory system. 15
Major dimensions of music for retrieval are melody, rhythm, harmony and timbre. o Melody, harmony -> based on pitch content o Rhythm -> based on timinginformation o Timbre -> relates to instrumentation, texture A representationof these high-level attributes can be obtained from pitch, timing and spectro-temporal information extracted by audio signal analysis. Representations are then compared via a similarity measure to achieve retrieval. 16
o The temporal pattern of frame-level features can offer important cues to signal identity Texture windows Audio signal <= duration: 0.5 1.0 s Analysis windows <= duration: 50 100 ms Frame-level features Feature vector Feature Extraction Feature summary M. F. Martin and J. Breebaart, "Features for Audio and Music Classification," in Proc.ISMIR, 2003. 17
Melody: pitch related feature Melody is the temporal sequence of notes usually played by a single instrument (fixed timbre). The discrete notes (pitches) are typically selected from a musical scale. frequency/note time
o Typical implementation: o Pitch detection is carried out on the audio signal at uniformly spaced intervals o The pitch sequence is segmented into notes (regions of relatively steady pitch) o Notes are labeled o Note patterns are matched to determine melodic similarity o Challenges: o Note segmentation can be a difficult task o Pitch detection in polyphonic music is tough 19
Monophonic Signal: cues to perceived pitch Spectrum Waveform A. de Cheveigne. Multiple F0 estimation. In D.-L. Wang and G.J. Brown, editors, Computational Auditory Scene Analysis : Principles, Algorithms and Applications, IEEE Press / Wiley, 2006. Schroeder histogram PDA Department of Electrical Engineering, IIT Bombay
o Time (Lag) domain: maximise autocorrelation value o Frequency domain: minimise error between estimated and predicted harmonic structures o Other 21
22
Music and speech signals are typically time-varying in nature => a time-frequency representation is required to visualize signal characteristics. The short-time Fourier transform (STFT) affords such a representation based on an assumption of signal quasistationarity. The window shape dictates the time and frequency resolution trade-off. X S m= ( ω, n) = x( m) w( n m) e jωm Department of Electrical Engineering, IIT Bombay
w(n-m) x(m) x(m)w(n-m) X( n, ω) DFT 0 ω π
ai[ t] t Φ[ ] i I[ t] I [ t] ˆ[ x t]= a[ t]cos Φ [ t] + et [ ] i = 1 i -amplitude variation of i th sinusoidal component ( partial ) - total phase (represents both frequency and phase variation) -Number of partials, can vary with time i Φ [ t] = ω[ t] t + ϕ[ t] i i i Model parameters to be estimated: { a i, ω i, ϕ i } l
Audio signal x DFT Peak detection Peak tracking Sinusoid parameters { a i, ω i, ϕ i } l Window Additive synthesis _ Tonal component + Σ Residual For the smooth evolution of the signal, sine components are detected in each frameand linked to tracks from the previous frame based on frequency proximity.
50 40 30 Spectral magnitude Fixed threshold (MaxPeak - 40 db) Final peaks picked 20 Magnitude (db) 10 0-10 -20-30 -40-50 0 500 1000 1500 2000 2500 3000 Frequency (Hz) 50 40 30 20 Spectral magnitude Envelope - 20 db Envelope - 25 db Envelope - 30 db Magnitude (db) 10 0-10 -20-30 -40-50 0 500 1000 1500 2000 2500 3000 Frequency (Hz)
Match spectrum around peak with that of ideal sinusoid. Apply threshold to the error. Department of Electrical Engineering, IIT Bombay
Peak tracking sine peak D Frequency track born track dies C B A 0 1 2 3 4 Time
Singer (main melody) Tanpura(drone) Frequency (Hz) 2000 1500 1000 500 Tabla(percussion) Ghe Na Tun Harmonium (secondary melody) 0 0 5 10 15 20 Time (sec)
o Input : magnitudes+ locations of sinusoids o For a range of trial fundamentals, generate predicted harmonics o MinimiseTWM errorw.r.t. trial fundamentals Predicted Components a 800 700 600 500 400 300 Measured Components b 800 700 420 375 Err total Err Err = + ρ N K p m m p 200 200 100 100 Nearest Neighbour Matching Department of Electrical Engineering, IIT Bombay
Department of Electrical Engineering, IIT Bombay
p E(p,j) W(p,p') E(p',j+1) p Pitch candidates, j Frame (time instant) E Measurement cost (local), W Smoothness cost j Minimize the Global transition cost over the singing spurt Department of Electrical Engineering, IIT Bombay
Department of Electrical Engineering, IIT Bombay
Polyphonic audio signal Signal representation Multi-F0 analysis Voice F0 contour Singing voice detection Predominant-F0 trajectory extraction
37
Pitch class profile opitch histogram osimilarity measure involves match between histograms 38
Positive phrases Negative phrase
Detects phrases melodically similar to Guru Bina pitch contour Swaras: S SN R Positive phrases Emphatic beat sam Negative phrase
43
Polyphonic audio signal Signal representation Multi-F0 analysis Voice F0 contour Singing voice detection Predominant-F0 trajectory extraction
o Input : magnitudes+ locations of sinusoids o For a range of trial fundamentals, generate predicted harmonics o MinimiseTWM errorw.r.t. trial fundamentals Predicted Components a 800 700 600 500 400 300 Measured Components b 800 700 420 375 Err total Err Err = + ρ N K p m m p 200 200 100 100 Nearest Neighbour Matching Department of Electrical Engineering, IIT Bombay
Predicted to measured error N p n p Err p m = f n (f n) + ( ) [q f n (f n) r] n= 1 Amax Significant term : Δf / (f) p o Δf = frequency mismatch error of = partial frequency a Measured to predicted error K p k p Err m p = f k (f k) + ( ) [q f k (f k) r] n= 1 Amax a Department of Electrical Engineering, IIT Bombay
Melody detection system [1]
o F0 search range (male/female) o p, q, r o ρ (male/female) o Window length (pitch range and rate of variation) o Smoothness cost parameter (rate of pitch variation) o Voicing threshold Department of Electrical Engineering, IIT Bombay
o Window lengthis an analysis parameter that influences the accuracy of sinusoidal modeling of the signal o Closely-spaced components in the polyphony => need for higher frequency resolution = longer windows o Pitch variation with time can be rapid in ornamented regions => need for better time resolution = shorter windows Department of Electrical Engineering, IIT Bombay
o Easily computable measures for adapting window length o Signal sparsity: a sparse spectrum is more concentrated => better represented sinusoidal components o Window length selection (20, 30, 40 ms) based on maximizing signal sparsity
1. V. Raoand P. Rao, Vocal melody extraction in the presence of pitched accompaniment in polyphonic music, IEEE Transactions on Audio, Speech and Language Processing, vol. 18, no. 8, pp. 2145 2154, Nov. 2010. 2. V. Rao, P. Gaddipatiand P. Rao, Signal-driven window adaptation for sinusoid identification in polyphonic music, IEEE Transactions on Audio, Speech, and Language Processing, Jan. 2012. 51