Ruohua Zhou, Josh D Reiss ABSTRACT KEYWORDS INTRODUCTION

Transcription

1 Subitted for; Algoriths and Systes, Edited by W. Wang, Published by IGI Global, ISBN-13: , July, Music Onset Detection Ruohua Zhou, Josh D Reiss Center for Digital Music, Electronic Engineering Departent Queen Mary University of London, UK Eail: zhou.ruohua@gail.co,reiss.josh@elec.qul.ac.uk ABSTRACT Music onset detection plays an essential role in usic signal processing and has a wide range of applications. This chapter provides a step by step introduction to the design of usic onset detection algoriths. The general schee and coonly-used tie-frequency analysis for onset detection are introduced. Many ethods are reviewed, and soe typical energy-based, phase-based, pitch-based and supervised learning ethods are described in detail. The coonly used perforance easures, onset annotation software, public database and evaluation ethods are introduced. The perforance difference between energy-based and pitch-based ethod is discussed. The future research directions for usic onset detection are also described. KEYWORDS Music onset detection, soft onsets, energy-based, phase-based, pitch-based, RTFI, STFT, ulti-band processing, supervised learning, steady tie span INTRODUCTION The audio signal is often considered to be a succession of the discrete acoustic events. The ter usic onset detection refers to detection of the instant when a discrete event begins in a usic signal. Music onset detection plays an essential role in usic signal processing and has a wide range of applications such as autoatic usic transcription, beat-tracking, tepo identification and usic inforation retrieval. Different sound sources (instruents) have different types of onsets that are often classified as soft or hard. The huan perception of the onset is usually related to the salient change in the sound s pitch, energy or tibre. Hard onsets are characterized by sudden increases in energy, whereas soft onsets show ore gradual change. Hard onsets can be well detected by energy-based approaches, but the detection of soft onsets reains a challenging proble. Let us suppose that a note consists of a transient, followed by a steady-state part, and the onset of the note is at the beginning of the transient. For hard onsets, energy changes are usually significantly larger in the transients than in the steady-state parts. Conversely, when considering the case of soft onsets, energy changes in the transients and the steady-state parts are coparable, and they do not constitute reliable cues for onset detection any ore. Consequently, energybased approaches fail to correctly detect soft onsets. Stable pitch cues enable to segent a note into a transient and a steady-state part, because the pitch of the steady-state part often reains stable. This fact can be used to develop appropriate pitch-based ethods that yield better perforances, for the detection of soft onsets, than energy-based ethods. However, although any approaches use energy inforation, only a few pitch-based ethods have been proposed in the literature. We discuss general schee for onset detection and how to develop an onset detection algorith step by step. Many existing ethods are described, and a few typical ethods are to be described in detail. Perforance evaluation and future research directions will also be discussed. The organization of this chapter is listed as follows. The ALGORITHMS section explains the general schee for usic onset

2 Subitted for; Algoriths and Systes, Edited by W. Wang, Published by IGI Global, ISBN-13: , July, detection and typical algoriths for energy-based, phase-based, pitch-based and supervised learning approaches. In the section on PERFORMANCE EVALUATION, the evaluation of onset detection approaches is discussed, and several established evaluation results are presented. Finally, the section on FURTHER RESEARCH DIRECTIONS discusses possible research directions, inspired by huan perception, which could be applied to the field of usic onset detection. ALGORITHMS General Schee Many different onset detection systes have been described in the literature. As shown in Fig. 1, they typically consist of three stages; tie-frequency processing, detection function generation, and peakpicking (Bello et al., 2005). At first, a usic signal is transfored into different frequency bands by using a filter-bank or a spectrogra. For exaple, the Short Tie Fourier Transfor (STFT) and the Resonator Tie Frequency Iage (RTFI) are two useful tie-frequency analysis tools for onset detection. Then, the output of the first stage is further processed to generate a detection function at a lower sapling rate. Finally, a peak-picking operation is used to find onset ties within the detection function, which is often derived by inspecting the changes in energy, phase, or pitch. Figure 1 Three stages of usic onset detection: tie-frequency processing of the audio signal, producing an onset detection function, and using peak-picking to identify onsets Tie-frequency processing Music signals are tie-varying, and ost of the analysis tasks require a joint tie-frequency analysis. One coonly-used tie-frequency analysis tool is Short Tie Fourier Transfor (STFT). The Fourier Transfor and its inverse can transfor signals between the tie and frequency doains. It can ake it possible to view the signal characteristics either in tie or frequency doain, but not to cobine both doains. In order to obtain a joint tie-frequency analysis for non-stationary signals, the STFT cuts the tie signal into different fraes and then perfor a Fourier Transfor in each frae. The STFT can be defined as follows, STFT t j (, ) s( ) w( t) e d (1) The STFT at tie t is the Fourier Transfor of a local signal, which is obtained by ultiplication of a signal s (t) and a short window function w( t) centered at tie t. When oving the window along the signal tie axis, we can calculate the STFT at different tie instants and obtain a joint tie-frequency analysis. The discrete STFT of a signal s (n) can be defined as follows, X ( n) N / 21 2 jl s( nh l) w( l) e (2) l N / 2 where w(l) is a N-point window, h is the hop size and denotes the different frequency bins.

3 Subitted for; Algoriths and Systes, Edited by W. Wang, Published by IGI Global, ISBN-13: , July, Another useful tie-frequency representation for usic signal analysis is the Resonator Tie Frequency Iage (RTFI). To better explain the RTFI, first a frequency-dependent tie-frequency (FDTF) analysis is defined as follows: FDTF t s w t e j ( t (, ) ( ) (, ) ) d (3) Unlike the STFT, the window function w of the FDTF ay depend on the analytical frequency. This eans that tie and frequency resolutions can be changed according to the analytical frequency. At the sae tie, Eq. (3) can also be expressed as convolution: FDTF ( t, ) s( t) I ( t, ) (4) where j t I ( t, ) w ( t, ) e (5) Equation (3) is ore suitable for expressing a transfor-based ipleentation, whereas Eq.(4) leads to a straightforward ipleentation of a filter bank with ipulse response functions expressed in Eq. (5). On the one hand, we can also group the ters in the STFT definition differently to obtain the filter bank interpretation for STFT. However, there are two ain differences between the band-pass filter ipleentations of STFT and FDTF. As illustrated in Fig. 2, one difference is that, in the ipleentation of the STFT, the bandwidth of the band-pass filter is kept fixed and independent of its centre frequency ω. Instead, for the FDTF, the bandwidth of the band-pass filter can be changed according to centre frequency ω. Another difference is that, in the ipleentation of the STFT, the output of every band-pass filter centered at ω is then deodulated to zero frequency. Such a deodulation process does not exist in the ipleentation of the FDTF. Figure 2 Filter bank ipleentation of STFT and FDTF Coputational efficiency and siplicity are the two essential criteria used to select an appropriate filter bank for ipleenting the FDTF. The order of the filter bank needs to be as sall as possible to reduce coputational cost. The basic idea behind the filter-bank-based ipleentation of FDTF is to realize

4 Subitted for; Algoriths and Systes, Edited by W. Wang, Published by IGI Global, ISBN-13: , July, frequency-dependent frequency resolution by possibly varying the filters bandwidths with their center frequencies. Therefore, the ipleenting filters ust be siple so that their bandwidths can be easily controlled according to their center frequencies. The RTFI selects a first-order coplex resonator filter bank to ipleent a frequency-dependent tie-frequency analysis. The RTFI can be expressed as follows: RTFI ( t, ) s( t) I ( t, ) (6) R t r ( )( t ) j ( t ) ( ) s ( ) e e d 0 r where ( r ( ) j ) t I R ( t, ) r( ) e, t 0 (7) In these equations, I R denotes the ipulse response of the first-order coplex resonator filter with oscillation frequency ω. The factor r(ω) before the integral in Eq. (6) is used to noralize the gain of the frequency response when the resonator filter s input frequency is the oscillation frequency. The decay factor r is dependent on the frequency ω and deterines the exponent window length and the tie resolution. At the sae tie it also deterines the bandwidth (i.e., the frequency resolution). The frequency resolution of tie-frequency analysis ipleented by the filter bank is defined as the equivalent rectangular bandwidth (ERB) of the ipleenting filter, according to the following equation: 2 0 B ERB H ( ) d (8) where H(ω) is the frequency response of a band pass filter and the axiu value of H(ω) is noralized at 1 (Hartann, 1997). The ERB value of the digital filter can be expressed according to angular frequency as follows: B ERB ( ) r( )(0.5 arctan( )) (9) r( ) In ost practical cases, the resonator filter exponent factor is nearly zero, so arctan(ω/r(ω)) can be approxiated to 0.5π, and Eq. (9) is approxiated as follows: B ERB ( ) r( ) (10) The resolution B ERB can be set through a ap function between the frequency and the exponential decay factor r. For exaple, a frequency-dependent frequency resolution and corresponding r value can be paraeterized as follows: B ERB ( ) d c, d c 0, c 0, d 0 (11) ERB r( ) B ( )/ ( d c) / (12) The coonly used frequency resolutions for usic analysis are special cases of the paraeterized resolutions in Eq.(11). When d=0, the resolution is constant-q; when c=0, the resolution is unifor; when d , c=0.1079, the resolution corresponds to the widely-accepted resolution of an auditory filter bank (Moore & Glasberg, 1996). As the RTFI has a coplex spectru, it can be expressed as follows: where A(t,ω) and φ(t, ω ) are real functions. j ( t, ) RTFI( t, ) A( t, ) e (13) RTFI Energy ( t, ) A( t, ) (14) It is proposed to use a coplex resonator digital filter bank for ipleenting a discrete RTFI. To reduce the eory usage of storing the RTFI values, the RTFI is separated into different tie fraes, and the average RTFI value is calculated in each tie frae. The average RTFI energy spectru can be 2

5 Subitted for; Algoriths and Systes, Edited by W. Wang, Published by IGI Global, ISBN-13: , July, expressed as follows: nm 2 1 A( n, ) 20 log 10 ( RTFI ( l, ) ) (15) M l( n1) M 1 where n is the index of a frae, M is an integer, and the ratio of M to sapling rate is the duration tie of each frae in the average process. RTFI(l, ω ) represents the value of the discrete RTFI at sapling point l and frequency ω. A detailed description of the discrete RTFI can be found in the references (Zhou, 2006; Zhou & Mattavelli, 2007). Energy-based detection In early ethods, the aplitude envelope of a usic signal was used to derive the detection function. The aplitude envelope can be constructed by rectifying and soothing the signal: N / 2 1 s k N / 2 C ( n) ( n k) w( k) (16) where w(k) is N-point window. A variation on this is to derive the detection function fro local energy, instead of aplitude. N / s k N / 2 E ( n) ( n k) w( k) (17) In the siplest case, differences in a signal s aplitude or energy are used to detect note onsets. However, such an approach has been shown to give poor results. Researchers have found it useful to separate the analyzed signal into a nuber of bands and then detect onsets across the different frequency bands. This constitutes the key eleent of ulti-band processing. For exaple, Goto utilizes the sudden energy changes to detect onsets in seven different frequency ranges and uses these onsets to track the usic beats by a ultiagent architecture (Goto, 2001). Klapuri divides the signal into 21 frequency bands by the nearly critical-band filter bank (Klapuri, 1999). Then, he uses aplitude envelopes to find onsets across these frequency bands. Duxbury et al. introduce a hybrid ultiband processing approach for onset detection (Duxbury, Sandler and Davies, 2002). In the approach, an energy-based detector is used to detect hard onsets in the upper bands, whereas a frequency based distance easure is utilized in the lower bands to iprove the detection of soft onsets. Wang et al. proposes a novel approach based on usic spectra agnitude (Wang et al., 2008). They first decopose the usic spectra agnitude into linear teporal bases by nonnegative atrix factorization, and then use the linear teporal bases to construct detection functions. The spectru is used to easure the energy change in the tie-frequency doain. The first-order difference of energy has been utilized to evaluate the energy change and derive a detection function. However, the first-order difference is usually not able to precisely ark onset ties. According to psychoacoustic principles, a perceived increase in the signal aplitude is relative to its level. The sae aount of increase can be perceived ore clearly in a quiet signal (Klapuri, 1999). Consequently, as a refineent, the relative difference can be used to better locate onset ties. For exaple, when the STFT is selected as the tie-frequency processing tool, the spectru D can be defined as follows, 2 2 D ( n) 20 log 10 ( X ( n) ) 20 log 10 ( X ( n 1) ) (18) where X (n) is the discrete STFT of the input signal.

6 Subitted for; Algoriths and Systes, Edited by W. Wang, Published by IGI Global, ISBN-13: , July, The coonly-used energy-based detection ethods can be generalized as follows, N 1 M ( n) H ( D ( n)) (19) N 1 where H( x) ( x x ) / 2 is the half-wave rectifier function, N is the total nuber of frequency bins in the spectru D, and M is the detection function. The detection function is further soothed by a oving average filter and a siple peak-picking operation is used to find the note onsets. In the peak-picking operation, only those peaks having values greater than a threshold θ are considered as the onset candidates. In this paragraph, it is explained that why tie-frequency decoposition can greatly iprove the energybased detection ethods, and why the positive flux is necessary. In any cases, during the note transition tie, the current note ay decay and decrease energy, while the new note ay begin and increase in energy, and the change in total energy is not noticeable. If the two successive notes have different pitch, then the energy-increasing and energy-decreasing will occur in the different frequency channels after an appropriate tie-frequency decoposition. As only the energy-increasing is the useful cue for onset tie, energy-decreasing during the decay of the forer note should be ruled out. When suing energy change across all frequency channels to derive a detection function, the type of flux is liited to the positive flux. Accordingly, only the energy increase during the onset tie of the latter note is considered. Phase-based detection As opposed to the standard energy-based detection, phase-based detection akes use of the spectral phase inforation as its source of inforation. The STFT can also be considered as coplex band-bass filter banks with equal bandwidth, and the STFT coefficient X (n) denotes the output of the th filter. In cases where there is only one sinusoid coponent passing the th band-pass filter and at the sae tie this sinusoid coponent is stable, the output of the th filter ust have a nearly constant frequency. Therefore, the difference between two consecutive unwrapped phase values of X (n) ust reain nearly constant: ( n) ( n 1) ( n 1) ( n 2) (20) where φ (n) is defined as the 2π-upwrapped phase of the STFT coefficient X (n). The phase deviation φ (n) can also be defined as: ( n) ( n) 2 ( n 1) ( n 2) 0 (21) During the steady-state portion of the signal, φ (n) is nearly equal to zero. During the transient portion, the frequency of X (n) is not constant, and φ (n) tends to be large. The analysis can be extended to the distribution of phase deviations of all frequency bins of the STFT (Bello & Sandler, 2003). During the steady-state part of the signal, the distribution is pointed. In the transient part, the corresponding distribution is flat and wide. These observations can be quantified by calculating the inter quartile range and the kurtosis coefficient of the distribution. Phase-based onset detection has deonstrated better perforance in the detection of the soft onset than standard energy-based ethods. However, it is susceptible to phase distortion and to phase noise introduced by the phases of low energy coponents. Pitch-based detection The approaches that use only energy and/or phase inforation are not satisfactory for the detection of soft onsets. Pitch-based detection appears as a proising solution for the proble. Pitch-based approaches can

7 Subitted for; Algoriths and Systes, Edited by W. Wang, Published by IGI Global, ISBN-13: , July, use stable pitch cues to segent the analyzed signal into transients and steady-state parts, and then locate onsets only in the transients. A pitch-based onset detection syste is described in the reference (Collins, 1999). In the syste, an independent constant-q pitch detector provides pitch tracks that are used to find likely transitions between notes. For the detection of soft onsets, such a syste perfors better than other state-of-the-art approaches. However, it is designed only for onset detection in onophonic usic. As polyphonic pitch estiation reains a largely unsolved proble, this akes it difficult to construct a pitch-based onset detection for polyphonic usic. This subsection describes a pitch-based approach that detects onsets in real polyphonic usic (Zhou, Mattavelli and Zoia, 2008). The onaural usic signal is used as the input signal at a sapling rate of 44.1 khz. The syste applies the RTFI as the tie-frequency analysis. The center frequencies of the discrete RTFI are set according to a logarithic scale. The resolution paraeters in Eq. (11) are set as d=0 and c= The frequency resolution is constant-q and equal to 0.1 seitones. Ten filters are used to cover the frequency band of one seitone. A total of 960 filters are necessary to cover the analyzed frequency range that extends fro 26 Hz to 6.6 khz. The RTFI energy spectru is averaged to produce the RTFI average energy spectru in units of 10s. It is well known that the huan auditory syste reacts with different sensitivities in the different frequency bands. This fact is often described by tracing equal-loudness contours. Jensen suggests a detection function called the perceptual spectral flux (Jensen & Andersen, 2004), in which he weighs the difference frequency bands by the equal-loudness contours. Collins uses the equal-loudness contours to weight the different ERB scale bands and derive another detection function (Collins, 2005a). Considering these works, in the ethod described here, the average RTFI energy spectru is transfored following the Robinson and Dadson equal-loudness contours, which have been standardized in the international standard ISO-226. To siplify the transforation, only an equal-loudness contour corresponding to 70 db is used to adjust the average RTFI energy spectru. The standard provides equal-loudness contours liited to 29 frequency bins. Then, this contour is used to get the equal-loudness contours of 960 frequency bins by cubic spline interpolation in the logarithic frequency scale. Let us identify this equalloudness contour as Eq(ω ) in db. Then, the spectru Y can be calculated as follows: Y ( k, ) A( k, ) Eq( ) (22) where ω represents the angular frequency of the th frequency bin. A(k, ω ) denotes the average RTFI energy spectru that is defined in Eq. (15). The usic signal is structured according to notes. It is interesting to observe that an energy spectru is organized according to note pitches rather than to a single frequency coponent. Then, the spectru Y is further recobined to yield the spectru R according to a siple haronic grouping principle: 5 1 R( k, ) Y ( k, i ) (23) 5 i 1 In practical cases, instead of using Eq. (23), the spectru R can be easily calculated on the logarithic scale by the following approxiation: 5 1 R( k, ) Y ( k, A i ) (24) 5 In Eq. (24) and (25), i 1 A [ 5] [0,120,190,240,279] (25) / 120, is fro 1 to 680 and the corresponding pitch range is 26Hz to 1.32kHz. To reduce noise, a 5 5 ean filter is used for the low-pass filtering of the spectru R according to the expression:

8 Subitted for; Algoriths and Systes, Edited by W. Wang, Published by IGI Global, ISBN-13: , July, S( k, ) R( k i, ) (26) j 25 i2 j2 To show energy changes ore clearly, the spectru D is calculated by the n th -order difference of spectru S. D( k, ) S( k, ) S( k n, ) (27) where the difference order n is set as 3 in a heuristic way. F( k, ) S( k, ) ax(( S( k, )) 1: N ) (28) where N is the total nuber of frequency bins, and spectru F is the relative easure of the axiu of S. Finally the spectra D and F together are considered as the input for the second stage of the onset detection algoriths. The energy-based detection algorith does not perfor well for detecting soft onsets. Consequently, a pitch-based algorith has been developed to iprove detection accuracy of soft onsets. A usic signal can be separated into transients and steady-state parts. The basic idea behind the algorith is to find the steady-state parts by using stable pitch cues and then look backward to locate onset ties in the transients by inspecting energy changes. In ost cases, a note has a spectral structure where doinant frequency coponents are approxiately equally spaced. The energy of a note is ainly distributed in the first several haronic coponents. Let us suppose that all the energy of a note is distributed in the first 10 haronic coponents. For a onophonic note with fundaental frequency ω, usually its spectru Y (Eq.22) can have peaks P(ω, A 1), P(2ω, A 2 ) P(10ω, A 10 ) at the haronic frequencies. P(ω, A) denotes the spectral peak that has value A at frequency ω. In ost cases, the corresponding spectru R (Eq. 23) presents the strongest spectral peak P(ω, (A 1+ A 2+ A 3+ A 4+ A 5)/5) at the fundaental frequency of the note. Accordingly, the fundaental frequency of a onophonic note can be estiated by searching for the axiu peak in the note s spectru R. For a polyphonic note, the predoinant pitches can be estiated by searching the spectral peaks that have values approaching or equal to the axiu in spectru R. These peaks are near the fundaental frequencies of the note s predoinant pitches; hence, the peaks are naed predoinant peaks. The spectru F (Eq. 28) is the relative easure of the axiu of R. Consequently, in F, the predoinant peaks have values approxiately equal to 0 db. To know how a pitch changes in a usic signal, F can be calculated in each short tie frae in units of 10s to get a two-diensional tiefrequency spectru. Given the tie-frequency spectru F of a signal, if there is always a predoinant peak around a frequency ω 1 in every tie frae of a tie span, this eans that there is a stable pitch in the tie span, and it can be assued that the tie span corresponds to a steady-state part. The tie span can be called steady tie span. The iages of the tie-frequency spectru are very useful to validate algorith developent by visual inspection. Several different usic signals and their spectra have been analyzed during the experiental work. It can be coonly observed that, during the steady-state part of a note, there are always one or ore steady tie spans, which are located just after the note s onset. Consequently, the steady-state parts of a signal can be found by searching steady tie spans in the signal s spectru F. The pitch based algorith described here consists of two steps: 1) Searching possible note onsets in every frequency channel. 2) Cobining the detected onset candidates across all the frequency channels. In the first step, the algorith searches for possible pitch onsets in every frequency channel. When searching in a certain frequency channel with frequency ω 1, the detection algorith tries to find only the onset where the newly occurring pitch rightly has an approxiate fundaental frequency ω 1. In each

9 Subitted for; Algoriths and Systes, Edited by W. Wang, Published by IGI Global, ISBN-13: , July, frequency channel with frequency ω 1, the algorith searches the steady tie spans, each of which corresponds to the steady-state part of a note having a predoinant pitch with fundaental frequency ω 1. Given a tie-frequency spectru F(k, ω ), a tie span T [k1, k2] (in units of 10s) is considered to be steady if it eets the following three conditions: ( F ( k, )) (29) 1, k k1: k 2 1 )) 1, k k1: k 2) ax(( F ( k, (30) where α 1 and α 2 are two thresholds. The third condition is that Su(ω ) has a spectral peak at the frequency ω 1, k 2 k k1 2 Su ( ) F ( k, ) (31) The boundary (k1 and k2) of a tie span can be easily deterined as follows. F t (k) is the tie-frequency spectru F in the frequency channel with frequency ω 1. Ft ( k) ( F ( k, )) (32) 1 Then, a two-value function P(k) is defined as: 1, Ft ( k) 1 P( k) (33) 0, Ft ( k) 1 G ( k) P( k) P( k 1) (34) where G(k) is the first-order difference of P(k). The beginning of a tie span corresponds to the tie at which G(k) assues the value 1 and the end of the tie span is the first instant, when G(k) assues the value -1. After all the steady tie spans have been deterined, the algorith looks backward to locate onsets fro the beginning of each steady tie span using the spectru D (Eq. 27). For a steady tie span T [k1, k2], the detection algorith locates the onset tie by searching for ost noticeable energy-change peak larger than the threshold α 3 in spectru (D(k, ω )) =1, k=(k1-30):k1. The search is done backward fro the beginning of a steady tie span, and the searching range is liited inside the 0.3-s window before the steady tie span. The tie position of this energy-change peak of the spectru D is considered as a candidate pitch onset. After all frequency channels have been searched, the pitch onset candidates are found and can be expressed as follows: Onset_ C( k, ) 0, =1, 2, 3, N, (35) where k is the index of tie frae and N is the total nuber of the frequency channels. If Onset_C(k, ω )=0, no onset exists in the k th tie frae of the th frequency channel. If Onset_C(k, ω )>0, there is an onset candidate in the k th tie frae of the th frequency channel, and the value of Onset_C(k, ω ) is set to the value of D(k, ω ). In the second step, the detection algorith cobines the pitch onset candidates across all the frequency channels to generate the detection function as follows, N 1 DF( k) Onset _ C( k, ) (36) N 1 The detection function is low-pass filtered by a oving average filter. Then, a peak-picking operation is used to find the onset ties. If two onset candidates are neighbors in a 0.05-s tie window, then only the onset candidate with the larger value is kept. A bow violin excerpt is provided to exeplify the specific usage and advantage of the pitch-based

10 Subitted for; Algoriths and Systes, Edited by W. Wang, Published by IGI Global, ISBN-13: , July, algorith. The exaple is a slow-attacking violin sound. Very strong vibrations can be observed fro its spectru Y reported in Fig. 3. Because of the vibrations, noticeable energy changes also exist in the steady-state parts of the signal. Therefore, the energy changes are not reliable for onset detection in this case. In the energy-based detection function, it is seen that there are any spurious peaks that are, in fact, not related to the true note onsets. Consequently, the energy-based detection algorith shows very poor perforance in this exaple. Frequency (Hz) Adjusted Energy Spectru for Bow Violin Exaple Tie (second) Figure 3 Bow Violin exaple: adjusted energy spectru (spectru Y).

11 Subitted for; Algoriths and Systes, Edited by W. Wang, Published by IGI Global, ISBN-13: , July, Figure 4 Bow Violin exaple: noral pitch energy spectru (spectru F). The vertical lines in the iage denote the positions of the true onsets. The Figure is adapted fro (Zhou, Mattavelli and Zoia, 2008) 0 Spectru F in the Frequency Channel 294 Hz 10 Noral Pitch Energy (db) Steady Tie Span T [2.44 second, 3.50 second] Tie (second) Figure 5 Bow Violin exaple: search of steady tie spans in one frequency channel. The Figure is adapted fro (Zhou, Mattavelli and Zoia, 2008)

12 Subitted for; Algoriths and Systes, Edited by W. Wang, Published by IGI Global, ISBN-13: , July, 10 Searching range for locating onset Spectru D in Frequency Channel 294Hz Steady State Tie Span T[2.44 second, 3.50second] Energy Change (db) 5 0 located onset by axiu peak in searching range Tie (second) Figure 6 Bow Violin exaple: location of the onset position backward fro steady tie span. The Figure is adapted fro (Zhou, Mattavelli and Zoia, 2008)

13 Subitted for; Algoriths and Systes, Edited by W. Wang, Published by IGI Global, ISBN-13: , July, Figure 7 Bow Violin exaple: onset candidates in all the frequency channels. The dots denote the detected onset candidates, the vertical lines are true onsets. Fig. 4 illustrates the spectru F of the exaple, and the vertical lines in the iage denote the positions of the true onsets. It can be clearly observed that there is always at least one steady tie span (white spectral line) just behind an onset position. The algorith searches every frequency channel to find steady tie spans, each of which is assued to correspond to a steady-state part. For exaple, steady tie spans are searched in frequency channel 294 Hz. As shown in Fig. 5, in the spectru F of this frequency channel, there is a tie span T [244, 320] (in units of 10 s). T has values larger than the threshold α 2=-10dB, and presents its axiu up to 0 db. There is also a peak rightly at a frequency of 294Hz in the Su T (ω ), which is obtained by the following expression: Su T 320 ( ) Fv ( k, ) (37) k 244 Fv(k, ω ) is the tie-frequency spectru F of the bow violin exaple. T is considered to be a steady tie span because it eets the three conditions, which were introduced earlier and used to judge if the tie span is steady. Then, the detection algorith locates the onset position by searching for a noticeable energy change peak larger than the threshold α 3 (in this exaple, α 3 =2) in the spectru D of the frequency channel. The searching window is liited inside the 0.3-s window before the steady tie span T. As shown in Fig. 6, in the spectru D of the frequency channel 294Hz, a peak with a value larger than the threshold α 3 is positioned near 2.42 s. The tie position is considered as a candidate onset tie. Here the pitch-based algorith uses stable pitch cues to separate the signal into the transients and the steady-state parts, and searches the onset candidates by energy changes only in the transients. So the energy changes caused by the vibrations in steady-steady parts are not considered as detection cues. The

14 Subitted for; Algoriths and Systes, Edited by W. Wang, Published by IGI Global, ISBN-13: , July, dots in Fig. 7 denote the detected onset candidates in the different frequency channels by the pitch-based detection algorith. It can be observed that the onset candidates are nearly around the true onset positions. Finally the detection algorith cobines the pitch onset candidates across all the frequency channels to get the final result. Supervised learning Soe approaches to onset detection are not copatible with the typical procedure described earlier, and they are based on achine learning (Marolt, et al., 1999; Chuan & Elaine, 2008; Davy & Godsill, 2002). Generally speaking, these ethods first perfor a tie-frequency analysis, such as STFT or constant-q analysis for the usic signal. Then, corresponding energy and phase inforation is used to classify every frae as being onset or non-onset by supervised learning algoriths. A typical supervised learning ethod is described as follows (Lacoste & Eck, 2007). Figure 8 A usic onset detection ethod based on supervised learning As shown in Fig. 8, the ethod first perfors a STFT analysis to produce a spectrogra, which is then used as the input variables to a feed-forward neural network (FNN). The FNN is trained to evaluate how uch every spectrogra frae can be classified as being onset. The outputs of the network are then treated with a siple peak-picking algorith based on a oving average. The FNN has two hidden layers and a single neuron in the output layer. The tanh activation function is used for hidden layers, and the logistic sigoid activation function is selected for the output layer. The conjugate gradient descent is eployed as learning function. Although every spectrogra frae could also be classified as a siple 0 or 1 (onset/nononset), it is ore reasonable to use sooth changes to odel the transition between onset and nononset. Accordingly, the target trace for every frae can be a cobination of the peaked Gaussians, which is centred on the labelled onset tie. 2 ( t i ) T ( t ) exp( ) (38) 2 i Where T(t) represents the target trace value of the spectrogra frae at the tie position of t, τ i is the i th labelled onset tie and σ is the width of the peak, chosen to be 10 illiseconds. In the training phase, the network produces a predicted value for every tie step. As shown in the following equation, the error function is the su of the squared error over all input patterns. where O(t j ) represents the output of the neural network for pattern j. 2 E ( T ( t j ) O ( t j )) (39) j The goal of the neural network is to predict an onset trace at the tie of every tie step. The input variables to the neural network are selected fro the spectrogra of the input signal. The nuber of input variables should be as sall as possible, because the large size of input variables akes it very difficult to train the neural network. Accordingly, when predicting an onset trace for a certain frae, it is reasonable that only the spectrogra points near the frae are selected as the input variables. In the ethod, the

15 Subitted for; Algoriths and Systes, Edited by W. Wang, Published by IGI Global, ISBN-13: , July, input variables are soe spectrogra points within a tie-frequency window centred on the predicted frae. To reduce the nuber of input variables, the spectru points in the input window are randoly sapled both on the tie and frequency axis. Unifor sapling is used along frequency axis, and the values along the tie axis are sapled according to a noral distribution. More than 200 input variables are sufficient to yield a good result. The input window has a width of 200 illiseconds. The window height is 90% of the height of the spectrogra. Thus, when oving the input window across the frequency axis, there are ultiple siilar input windows for the sae tie step and this will yield a ore robust odel. Accordingly, the network has ultiple predicted values for a single frae, and the ultiple values are erged to generate a detection function Finally, a siple peak picking algorith is used to find onset ties fro the detection function. The peak picking algorith can be expressed as follows. p ( t ) d ( t ) u ( t ) (40) where u ( t ) g * d ( t ) (41) where g is the Gaussian filter, d denotes the detection function, and p is the peak trace. In the peak picking algorith, a high-pass filter is used to isolate the interesting fast change inforation fro the slow change, which is considered to be not related to onsets. The high pass filter is ipleented by subtracting the Gaussian-filtered signal fro the original signal. In the final step, in the peak trace p, each peak with a value larger than threshold K is considered to represent an onset tie. The position of the onset is calculated as the centre of ass of all points inside the peak. The optial threshold K can be selected fro training saples. PERFORMANCE EVALUATION Perforance Measures To evaluate the detection ethod, the detected onset ties ust be copared with the reference ones. For a given reference onset at tie t, if there is a detection within a tolerance tie-window [t-50s, t+50s], it is considered to be a correct detection (CD). If not, there is a false negative (FN). The detections outside all the tolerance windows are counted as false positives (FP). The F-easure, Recall and Precision easures are used to suarize the results. The Precision and Recall can be expressed as: NCD P (42) NCD NFP NCD R (43) N N CD where N CD is the nuber of correct detections, N FP is the nuber of false positives and N FN is the nuber of false negatives. These two easures can be suarized by the F-easure defined as: PR F 2 (44) P R Onset Annotation It is iportant to construct a reference database with reliable annotated onset ties. Onset detection is a subjective task, usually there is no perfect reference for onset ties. In ost cases, the onset ties need to be hand labeled. For percussive onsets, the reference ties can be precisely labeled by visualizing the FN

16 Subitted for; Algoriths and Systes, Edited by W. Wang, Published by IGI Global, ISBN-13: , July, wavefor in audio display software. In the spectrogra of the analyzed signal, the concurrent fast energy-increasing of different haronic coponents of a usic note can be often visualized at the onset tie. Accordingly, the spectrogra is also a very useful tool to help annotate the onset ties, although it is not very precise. The ost precise and flexible ethod for onset annotation is to listen to signal slice with the support of visualizations. The Sound Onset Labelizer (SOL) is a free onset annotation software for research purpose (Leveau, Daudet and Richard, 2004). This software provides a user-friendly interface to annotators. Both the wavefor and the spectrogra of the analyzed signal can be seen in the screen of the software s GUI. The spectrogra and wavefor parts have the sae tie axis. Using the tool, onset labels can be first annotated in the spectrogra by visual inspection, and then they can be ore precisely adjusted by aural feedbacks. More reliable reference onset ties can be achieved by cross-validation aong the different expert annotators. As onset annotation is a very tie consuing task, it is ore efficient to share soe public database aong different researchers. Leveau et al. provide a reliable public distributed database for onset detection, which contains 17 short usic sequences in different usic instruents and genres (Leveau, Daudet and Richard, 2004). The validated onset labels for ore than 700 corresponding onsets are also freely distributed. Perforance Coparison and MIREX Evaluation It is quite difficult to ake perforance coparison aong the different ethods for onset detection, because there is a shortage of public database available, and different researchers use different test databases. There is a direct coparison between an energy-based ethod and a pitch-based one in the literature (Zhou, Mattavelli and Zoia, 2008). Both ethods are based on the RTFI tie-frequency analysis. The pitch-based ethod has been described in the above section. The sae test dataset was used for the evaluation. The test database contains 30 usic sequences of different genres and instruents. In total there are 2543 onsets and ore than 15-inutes of tie duration. The usic files are classified into the following classes: piano, plucked string, sustained string, brass, winds, coplex ixes. Here the piano is considered as a single class because ost of the piano usic contains any hard onsets. The total test results on the test dataset are suarized in Table I. The energy-based algorith perfors better than does the pitch-based algorith on the piano and coplex usic, which contains several hard onsets. The energy-based detection gains 5.0% for piano usic and 8.4 % for the coplex usic. Conversely, the pitch-based detection algorith perfors better in the brass, winds and sustained string, in which note onsets are considered to be softer. For the sustained string, the pitch-based algorith gains Table 1 Coparison between the energy-based and pitch-based ethods Average F- Measure (Pitchbased) Average F- Measure (Energybased) Piano Coplex Mixes Plucked String (Guitar, Violin, Cello) Brass (Trupet, Horn) Winds (Clarinet, Flute, Oboe) Sustained String (Quartet, Violin, Viola) 92.7% 82.6% 87.6% 93.2% 88.4% 87.0% 97.7% 91.0% 83.6% 87.8% 80.8% 44.1%

17 Subitted for; Algoriths and Systes, Edited by W. Wang, Published by IGI Global, ISBN-13: , July, Figure 9 Precision coparison of energy-based and pitch-based onset detections. 42.9% and greatly iproves the perforance fro 44.1% to 87.0%. In addition, the pitch-based algorith gains 5.4%, 7.6% for brass and winds, respectively. A coparison between the precisions of the pitch-based and energy-based ethods is shown in Fig. 9. The coparison clearly suggests that the pitch-based ethod has a uch better precision than the energybased ethod. The pitch-based ethod overperfors the energy-based algorith for the detection of soft onsets. The reason for iproved perforance can be explained as follows. Energy-based approaches are based on the assuption that there are relatively ore salient energy changes at the onset ties than in the steady-state parts. In case of soft onsets, the assuption can not stand. The significant energy changes in the steadystate parts can islead energy-based approaches and cause any false positives. Conversely, the proposed pitch-based algorith can first utilize stable pitch cues to separate the usic signal into the transients and the steady-state parts, and then find note onsets only in the transients. The pitch-based algorith reduces the false positives that are caused by the salient energy changes in the steady-state parts, and greatly iproves the onset detection perforance of the usic signal with any soft onsets. Because of the reduction of false positives, it also gets a better precision. Siilar to pitch-based onset detection, phase-based ethod has a good perforance for the detection of soft onsets. But phase-based detection is very susceptible to phase distortion and to phase noise introduced by the phases of low energy coponents. Copared to pitch-based detection, phase-based detection shows very low perforance for the onset detection in real polyphonic usic signal.

18 Subitted for; Algoriths and Systes, Edited by W. Wang, Published by IGI Global, ISBN-13: , July, Music Inforation Retrieval Evaluation Exchange (MIREX) provides a foral evaluation fraework (MIREX, 2005). Fro MIREX 2005 to MIREX 2007, several onset detection ethods have been evaluated on the sae dataset in the audio onset detection task. The test dataset contains 85 usic sequences of different genres and instruents. In total there are 14-inutes of tie duration. The dataset contains four different classes such as solo dru, solo polyphonic pitched instruents, solo onophonic pitched instruents and coplex ixtures. In addition, the onophonic pitched instruents class is subdivided into 5 different subclasses: brass, wind, sustained strings, plucked strings, bars and bells, singing voice. In the MIREX 2005~2007 onset detection tasks, any ethods have been evaluated. Alost all the ethods fail to detect onsets in singing voice. This result suggests that it is quite difficult to develop a general onset detection ethod, which can work well for both singing voice and the other usic. Most of the subitted ethods are energy-based and show a poor perforance on the detection of the classes: solo brass, solo wind, solo sustained string. These classes usually contain a large nuber of soft onsets. The significant energy changes in the steady-state parts can islead energy-based approaches and cause any false positives. As explained before, the pitch-based detection can clearly outperfor the energybased detection for the detection of soft onsets. In addition, the phase inforation is also useful to iprove the detection for soft onsets. The Zhou and Reiss ethod cobines the energy-based and pitchbased detection so that the ethod has uch better perforance on the solo brass class and solo wind class than other ethods (Zhou and Reiss, 2007). The Lee ethod cobines energy and phase inforation and achieves the best perforance on the sustained string class (Lee, Shiu and Kuo, 2007). According to the average F-easure, the overall perforance difference between the first four best ethods [Zhou and Reiss, 81%; Lee 80%; Lacoste and Eck, 80%; Robel, 80%] is inor (Zhou and Reiss, 2007; Lee, Shiu and Kuo, 2007; Lacoste & Eck, 2005; Robel, 2007). FUTURE RESEARCH DIRECTIONS Since the nature of usic signals is quite varied, no single ethod is suitable for all usic signals. different detection ethods could be used for different types of sound events to achieve better perforances (Collins, 2005b; Ricards, 2005). Inspired by the huan perceptual syste, further iproveents ay arise by cobining ultiple sipler ethods using a voting echanis. These iproveents could be achieved by developing ore efficient classification algoriths capable of assisting usic onset detection. The classification algoriths would autoatically estiate the doinant onset type for the usic signal being analyzed. Then, the adaptive cobination of different ethods is expected to iprove the overall perforance. In addition, as the huan ear perfors uch better for onset detection than an autoatic detection ethod, coputational auditory odels such as a loudness odel could also play an iportant role in the further research. REFERENCES Bello, J.P. & Sandler, M. (2003). Phase-based note onset detection for usic signals. Proc. IEEE Int. Conf. Acoustics, Speech, and Signal Processing (ICASSP-03), Hong Kong, pp Bello, J.P., Daudet, L., Abadia, S., Duxbury, C., Davies, M. & Sandler, M.B. (2005). A tutorial on onset detection in usic signals, IEEE Trans. Speech and Audio Signal Processing, vol. 13, pp

19 Subitted for; Algoriths and Systes, Edited by W. Wang, Published by IGI Global, ISBN-13: , July, Chuan, C. & Elaine C. (2008). Audio Onset Detection Using Machine Learning Techniques: The Effect and Applicability of Key and Tepo Inforation. University of Southern California Coputer Science Departent Technical Report No Collins, N.(1999). Using a pitch detector as an onset detector. Proc. International Conf. On Music Inforation Retrieval. Collins, N. (2005a). A coparison of sound onset detection algoriths with ephasis on psychoacoustically otivated detection functions. AES Convention 118, Barcelona. Collins, N. (2005b). A change discriination onset detector with peak scoring peak picker and tie doain correction.. MIREX 2005 audio onset detection contest: fro Davy, M. & Godsill S. (2002). Detection of abrupt spectral changes using support vector achines an application to audio signal segentation," in Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP'02), vol. 2, pp , Orlando, Fla, USA. Duxbury, C., Sandler, M., & Davies, M. (2002). A hybrid approach to usical note onset detection. Proc. 5th International Conf. Digital Audio Effects (DAFX-02), Haburg, Gerany. Goto, M. (2001). An audio-based real-tie beat tracking syste for usic with or without dru-sounds. Journal of New Music Research, vol. 30, No. 2, pp Hartann, W.M. (1997). Signals Sound and Sensation, Aerican Institute of Physics Press. Jensen, K. & Andersen, T.H. (2004). Causal rhyth grouping. Proc. 2 nd International Syposiu on Coputer Music Modeling and Retrieval, Esbjerg, Denark. Klapuri, A. (1999). Sound onset detection by applying psychoacoustic knowledge. Proc. IEEE International Conf. Acoustics, Speech, and Signal Processing (ICASSP-99), pp Lacoste, A. & Eck, D. (2005) Onset detection with artificial neural network for MIREX MIREX 2005 audio onset detection contest: fro Lacoste, A. & Eck, D. (2007). A supervised classification algorith for Note Onset Detection. EURASIP Journal on Advances in Signal Processing, vol. 2007, Article ID 43745, 13 pages. Lee, W., Shiu, Y. and Kuo, C. (2007) Musical onset detection with linear predication and joint features. MIREX 2007 audio onset detection contest: fro Leveau, P., Daudet, L. & Richard, G. (2004). Methodology and Tools for the evaluation of autoatic onset detection algoriths in usic. Proc. 5 th International Conf. On Music Inforation Retrieval, Barcelona.

20 Subitted for; Algoriths and Systes, Edited by W. Wang, Published by IGI Global, ISBN-13: , July, Marolt, M., Kavcic, A. & Privosnik, M. (1999). Neural networks for note onset detection in piano usic. Proc. International Conf. On Coputer Music. MIREX (2005), First Annual Music Inforation Retrieval Evaluation exchange (MIREX), fro Moore, B.C.J. & Glasberg, B.R. (1996). A revision of Zwicker s loudness odel. ACTA Acustica, vol. 82, pp Ricard, J. (2005). An ipleentation of ulti-band onset detection. MIREX 2005 audio onset detection contest: fro Robel, A. (2007) Onset detection in polyphonic signals by eans of transient peak classification. MIREX 2007 audio onset detection contest: fro Wang W., Luo Y., Chabers J.A. & Sanei S.(2008). Note Onset Detection via Nonnegative Factorization of Magnitude Spectru. EURASIP Journal on Advances in Signal Processing, vol. 2008, Article ID , 15 pages, doi: /2008/ Zhou, R. (2006). Feature Extraction of Musical Content for Autoatic Music Transcription. Ph.D. dissertation, Swiss Federal Institute of Technology, Lausanne, fro Zhou, R. & Mattavelli (2007). A new tie-frequency representation for usic signal analysis. Signal Processing and Its Applications, ISSPA th International Syposiu on, pp Zhou, R. & Reiss, J.D. (2007) Music onset detection cobining energy-based and pitch-based approaches. MIREX 2007 audio onset detection contest: fro Zhou, R., Mattavelli, M. & Zoia, G. (2008). Music onset detection based on Resonator Tie-frequency Iage. IEEE Trans. Audio, Speech and Language Processing, vol. 16, pp ADDITIONAL READING SECTION Bello, J.P. & Sandler, M. (2003). Phase-based note onset detection for usic signals. Proc. IEEE Int. Conf. Acoustics, Speech, and Signal Processing (ICASSP-03), Hong Kong, pp Bello, J.P., Duxbury, C., Davies, M., & Sandler, M.(2004). On the use of phase and energy for usical onset detection in the coplex doain. IEEE Signal Processing Letter, vol. 11, no. 6, pp Bello, J.P., Daudet, L., Abadia, S., Duxbury, C., Davies, M. & Sandler, M.B. (2005). A tutorial on onset detection in usic signals, IEEE Trans. Speech and Audio Signal Processing, vol. 13, pp Chuan, C., & Elaine C. (2008). Audio Onset Detection Using Machine Learning Techniques: The Effect and Applicability of Key and Tepo Inforation. University of Southern California Coputer Science Departent Technical Report No