Frequency Domain Artificial Reverberation using Spectral Magnitude Decay

Transcription

1 sing Spectral Magnitde Decay Earl Vickers 1, Jian-Lng (Larry) W 2, Praveen Gobichettipalayam Krishnan 3, and Ravirala Narayana Karthik Sadanandam 4 1 The Sond Gy, Inc., Seaside, CA 93955, USA sfx@sfxmachine.com 2 Stanford Center for Compter Research in Msic and Acostics, Stanford, CA 94305, USA larryw@ccrma.stanford.ed 3 University of Missori, Rolla, MO 65409, USA pgk5pd@mr.ed 4 University of Missori, Rolla, MO 65409, USA rns3r8@mr.ed ABSTRACT A novel method of prodcing artificial reverberation in the freqency domain, sing spectral magnitde decay, is presented. The method involves accmlating the magnitdes of the short-time Forier transform, based on the desired decay time as a fnction of freqency. Compared to time domain methods sch as feedback delay networks, the crrent method reqires less memory and provides independent control of the reverb energy and decay time in each freqency bin. Compared to convoltion reverbs, the crrent approach offers flexible parametric control over the decay spectra and a comptational cost that is independent of decay time. 1. INTRODUCTION This paper presents a method for prodcing artificial late reverberation in the freqency domain sing spectral magnitde decay. This method offers a different set of tradeoffs compared to previos reverberation methods sch as time domain feedback loops and convoltion reverbs. Late reverberation refers to the diffse portion of the room response (typically starting arond 100 ms.), characterized by a very large nmber of echoes and an intensity that is relatively independent of the position within the room [1]. Late reverberation lends itself to a statistical description.

2 Time domain reverb algorithms, sch as feedback delay networks [2], simlate late reverberation at a low comptational cost while providing independent control over a nmber of perceptally relevant parameters. The reverb decay envelope can be made freqency dependent; however, de to the expense of placing mlti-band eqalizers in each feedback path, sally only two or three bands of decay time control are provided. spectral content, or basic tool will be the phase vocoder [5], which allows a signal to be analyzed into a time-freqency grid, optionally modified in varios ways, and resynthesized, as shown in Figre 1. Convoltion reverbs, typically implemented in the freqency domain, convolve a sorce signal with a desired implse response. Convoltion prodces an excellent simlation of the acostics of a particlar physical space, bt it lacks some of the flexibility of the time domain algorithms; also, the comptational cost is greater, especially for long decay times. The crrent approach is inspired partly by the observation that freqency domain time-scaling based on the phase vocoder often sffers from an nwanted phasiness, reverberation, or loss of presence [3]. Given that we are trying to prodce reverberation, this side effect of the phase vocoder might prove to be an advantage, or at least not a liability. Or approach is also inspired by Moorer s observation that the responses in the finest concert halls sonded remarkably similar to white noise with an exponential amplitde envelope [4]. Or algorithm, performed in the short-time Forier transform (STFT) domain, attenates and accmlates the spectral magnitdes, which are combined with a compted phase signal. This method yields an implse response with a smooth, exponentially decaying envelope and independent control over room energy and decay time at each freqency. The crrent method reqires less memory than feedback delay networks, and, nlike convoltion reverbs, it can prodce extremely long reverb decay times at no additional comptational cost. We will discss reverberation in the context of a nmber of related topics: time-scaling, time-freezing, magnitde accmlation and phase coherence. We will also discss possible directions for ftre research. 2. THE STFT AND THE PHASE VOCODER Given that we wish to extend the temporal evoltion of a signal while retaining independent control over its Figre 1: Phase Vocoder. The phase vocoder can be efficiently implemented sing the STFT [6]. An overview of two complementary views of the phase vocoder, the filter bank interpretation and the Forier transform interpretation, are given in [7]. Some practical implementation details are provided in [8] and [9]. A common application of the phase vocoder is to implement time-scaling while preserving the pitch and spectra. To review, this is done as follows [10]: 1. Compte the STFT of the inpt signal. ( ) = h n X t a,! k # n="# x is the inpt signal, ( )x( t a + n)e " j! k n $, where X is the inpt s STFT representation, h n ( ) is the analysis window, Page 2 of 15

3 ! k = 2!k N channel (or bin), N is the size of the discrete Forier transform, R a is the analysis hop size in samples, is a set of sccessive integers starting at 0, and t a = R a is the time of the th analysis frame. is the center freqency of the kth vocoder 2. For each channel k and analysis time-instant t a, (a) Obtain the phase! k (t a ) =!X(t a," k ), (b) Calclate the heterodyned phase increment!" k =! k (t a )#! k (t a #1 )# R a $ k (c) Take its principal determination (between ± π ), denoted by, which can be pφ k regarded as the amont of freqency deviation from Ω, k (d) Estimate the instantaneos freqency, 1 ˆω k ( ta ) = Ωk + pφk, R a (e) Set the phase of the time-stretched STFT at synthesis time ts = Rs, where R s is the synthesis hop size, according to the phasepropagation formla, Y ( t s, Ω k ) = Y ( t, Ω ) + R ωˆ ( t 1 s k s k a (f) Set the magnitde Y t, Ω ) = X ( t, Ω ), ( s k a k (g) Obtain the otpt signal by overlap-adding the IDFTs of the synthesis frames Y, Ω ), as follows: y(n) = " ( ) ( t s k # w n!t s y n!t s =!" N!1 y (n) = 1 " Y ( t s,# k )e j# k n N k=0 ) ( ), where, 3. TIME-FREEZE, TIME-SCALING, LOOPING AND REVERBERATION In Babylonian mythology there are hints of a specially constrcted room in one of the ziggrats where whispers stayed forever. [11] Or langage sggests that there is something abot reverberation that echoes, resonates and reverberates deep within the hman psyche. One motivation for the crrent research is to find ways to prodce long or infinite reverberant decays [12] that sond extremely smooth, as if the original sond were frozen in midair. To begin with, we will consider modifying techniqes sed for time-scaling, time-freezing, and the creation of perceptally smooth loops for extending the sstain portion of msical notes, as sed in sampling keyboards and software programs [13] Time-Freeze If we temporarily ignore certain time domain effects of reverberation, sch as the diffsion of individal echoes, the spectral effects of reverberation are similar to what might be achieved by freezing each STFT inpt frame and imposing a freqency-dependent decay on the spectral magnitdes. Time-freezing, or infinite time-scaling, involves extending the magnitde spectrm of an STFT frame indefinitely while propagating the phases based on instantaneos freqency estimates. A nmber of papers have described phase-vocoder-based time-scaling [7, 14] and varios improvements intended to redce the aforementioned phasiness artifacts [10, 15]. Some techniqes for freezing time are mentioned in [15] and [16], inclding the idea of alternating between inpt hops of +1 and -1 samples in consective frames. A simple time-freeze algorithm is shown in Figre 2. Page 3 of 15

4 Y ( t,! k ) = g (! k ) Y ( t "1,! k ), increment the phases:!y ( t," k ) =!Y ( t #1," k )+ $! k ( t f ), and convert the magnitde-phase representation back into a rectanglar representation: Y ( t,! k ) = Y ( t,! k ) e i"y ( t,! k ) Then we perform inverse Forier transforms and overlap-add as before. The reslting time-frozen signal has a freqency-dependent decay time (which can be infinite if the feedback gains are initialized to nity). The otpt sonds smooth and frozen. However, the reslt is perceived as artificial and mechanical, largely becase the phase increment is exactly the same from one frame to the next, withot the freqency modlations typical of msical sonds, and with none of the random phase flctations expected from tre reverberation. The otpt from nvoiced or other noiselike signals can sond especially nnatral becase coherent periodicities are imposed pon the entire dration of the time-freeze, reslting in tonal noise Ongoing Parallel Time-Freeze Figre 2: Time-Freeze algorithm. Immediately prior to freezing the sond, we captre the phase deltas for each freqency channel k,!! k t f ( ) =! k t f ( )"! k t f "1 ( ), where t f denotes the time of the frame immediately preceding the time freeze. The above process simlates the application of a reverblike decay to a single adio frame. To apply this effect to an ongoing adio signal, one might imagine the following procedre: begin a new time-freeze process in parallel for each sccessive inpt frame (applying a freqency-dependent decay to the spectral magnitdes), and sm the time-aligned otpts. (See Figre 3.) The reslting strctre is somewhat analogos in form to a mlti-tapped delay line, i.e., the canonical reverb strctre. Then, at each time t, we apply a freqency-dependent gain, g Ω k ( ), to the magnitdes: Page 4 of 15

5 First, we can consider discarding frozen frames once their magnitde has decayed to a sfficiently low level; e.g., -60 db. This wold greatly improve the efficiency when implementing short reverb decay times. The nmber of frozen frames we need to compte is given by m = T r f s R, where m is the nmber of frozen frames to retain, T r is the -60 db reverberation time (in seconds), f s is the sample rate (in Hz), and R is the hop size (in samples). Figre 3: Reverb-like effect sing parallel time-freeze. Clearly, sch an implementation wold reslt in a data storage and comptational explosion, bt a simlation of the process yields a very pleasant, reverb-like Vega response [17]. The implse response, however, consists of a decaying series of implses repeated every N samples (see Figre 4), de to the lack of phase randomization. This isse will be addressed in sections 5 and 6. For example, a 1 second decay time with a 2048 sample hop size at 44,100 Hz wold reqire s to keep track of abot 22 frozen frames at any given time. At each hop time, for each frozen frame, we need to scale the magnitdes of each freqency channel, increment the corresponding phases, convert the magnitdes and phases to a rectanglar representation and accmlate the reslts. Even with the above optimization, this cold still be comptationally brdensome for longer reverb decay times, sch as infinity. For a given freqency bin k, the magnitde of each frozen frame is attenated by the same factor g( Ω k ) at each hop, reslting in a smooth exponential decay, as depicted in the top part of Figre 5 for for frozen frames (overlapping dashed lines). However, the sm of the for magnitdes (solid crve) does not have a smooth exponential decay, de to phase incoherence becase each frozen frame has a slightly different instantaneos freqency. Figre 4: Implse response of the parallel time-freeze Optimizing the Parallel Time-Freeze Assming we can find ways to address the nwanted periodicity of the implse response, we can imagine a nmber of ways to optimize the parallel time-freeze. Likewise, in the bottom half of Figre 5, even thogh each of the for frozen frames has a constant phase increment (instantaneos freqency), as shown by the horizontal dashed lines, the instantaneos freqency of the sm of the for frames (solid crve) is non-linear. Unfortnately, there does not appear to be a simple mathematical identity that will let s increment the phase of the sm of a nmber of frozen frames, withot having to keep track of each one separately. Page 5 of 15

6 accmlation of the spectral magnitdes of the incoming frames. The desired strctre can be viewed as a freqency domain analoge of Moorer s improved comb filter (Figre 6), which had a freqency-dependent gain in the feedback loop [4] and was a precrsor of the feedback delay network (FDN) reverberator [2]. Figre 5: Magnitde (top) and instantaneos freqency (bottom) of freqency bin 13 for for adjacent frozen frames (dashed lines) and the sm of the for frozen frames (crved solid lines). Note that the instantaneos freqency vales tend to become extreme when the corresponding magnitdes reach minima. However, the idea of applying magnitde attenation and phase increments to each frozen frame independently is an artificial constrct to begin with, so it may be no more arbitrary to apply them to the sm of the frozen frames, if we can find a way to calclate a sitable phase increment (possibly based on a magnitde-weighted average of the instantaneos freqencies of the individal frozen frames). Alternatively, we may be able to consolidate grops of frames having similar instantaneos freqencies. Figre 6: Comb filter with freqency-dependent feedback gain (after Moorer, 1979). More work needs to be done in this area. In the meantime, we may be able to approximate or rnning time-freeze (or, perhaps, jst the evoltion of the older frames) sing a recrsive implementation. 4. RECURSIVE SPECTRAL MAGNITUDE DECAY Jst as time domain late reverberators typically se infinite implse response (IIR) feedback loops instead of brte force finite implse response (FIR) convoltion, we wold like to find a more efficient, recrsive method of spectral decay. Instead of starting a separate time-freeze process beginning with each new frame of inpt data, it wold be more efficient, thogh perhaps not identical in reslt, to perform a single leaky Figre 7: Spectral Magnitde Decay with freqency-dependent feedback gain. Page 6 of 15

7 Or recrsive Spectral Magnitde Decay algorithm is shown in Figre 7. (Note the similarity to the strctre of the comb filter.) At each hop time (or frame), a Forier transform is calclated over the windowed inpt signal, reslting in an STFT representation [10, 18], # $. X( t,! k ) = h( n)x( t + n)e " j! k n n="# Hanning windows, given sfficient overlap and applying an additional gain factor of g to each sccessive window, yields a close (thogh not perfect) approximation of an exponential decay, as shown in Figre 8. (Dividing the reslting decay by a tre decaying exponential reveals a slight ripple, which appears to be insignificant for an overlap of R = N / 4.) The Forier transform is added to a feedback Forier transform; the sm is delayed by one frame and its magnitde is attenated as a fnction of the desired freqency-dependent decay time. The attenated magnitde is combined with an artificial phase signal to prodce the feedback Forier transform. The delayed sm STFT, Y ( t,! k ), is converted back to the time domain, windowed and overlap-added to prodce the otpt, as shown previosly. At each frame, the accmlated magnitde in each freqency bin is given by the eqation: Y ( t,! k ) = X( t "1,! k ) + Y ( t "1,! k ) # g (! k ) To determine the g (! k ) attenation vales, we begin by specifying T r (Ω k ), the reverberation time (in seconds) reqired for the sond pressre to decay 60 db at each freqency! k. Assming the decay rate is linear in db, the rate at which the sond pressre decays dring one STFT hop shold eqal the rate implied by T r (Ω k ), as follows: 20log 10 ( g(! k )) = "60 R / f s ( ). T r! k Therefore, the attenation for the k th bin is given by [1, 2]. g (! k ) = 10 "3R T r (! k ) f s Figre 8: Smooth qasi-exponential decay (top crve) prodced by overlap-adding scaled Hanning windows (lower crves). The fade-in time at the beginning of the implse response might be a desirable featre, to enable a smooth cross-fading between early reflections and late reverb, bt the fade-in time and fixed window length cold be problems when attempting to prodce very short reverb decay times. In sch cases, we may want to se shorter windows, FFT lengths and hop sizes. An Energy Decay Crve (EDC) can be obtained by integrating the energy remaining in the implse response after time t:! EDC(t) = " h 2 (!) d!, where h t ( ) is the room s implse response [19, 1]. Figre 9 illstrate the algorithm s implse response (sing a random phase calclation) and the corresponding EDC. t If we want the reverb s implse response to have an exponential decay, the individal windowed STFT frames shold overlap-add to prodce a smoothly decaying exponential. Smming a series of overlapping Page 7 of 15

8 Implse response resembles decaying noise In the time domain, the reverb s implse response shold resemble noise with an exponential decay envelope. The implse response shold not be overly sparse or obviosly repetitive; for example, it shold not consist of a single implse repeated every N samples. If possible, there shold be parametric control over the echo density Sb-channel freqency resoltion Figre 9: Spectral Magnitde Decay implse response, above; and Energy Decay Crve (in db), below. A 3D Energy Decay Relief (EDR) of the Spectral Magnitde Decay s implse response wold reveal different decay rates at different freqencies, de to the freqency-dependent feedback gains. 5. THE PHASE PROBLEM The Spectral Magnitde Decay method of Section 4 reqires the generation of an artificial phase signal to be combined with the accmlated magnitde response. The phase generation is somewhat more complicated than for the traditional phase vocoder, as sed in timestretching or pitch-shifting, becase the reverb s otpt phase is not a simple fnction of the inpt phase, de to the feedback loop. For example, we can t simply mimic the phase of the inpt signal, becase the inpt may have gone silent while the reverberant decay contines. We will characterize the desired phase signal and explore some phase generation methods Desired Phase Signal Characteristics The desired phase signal has a nmber of, possibly illdefined, properties: Ideally, the phase increments from one frame to the next shold provide additional freqency resoltion beyond that given by the center freqencies of the spectral bins. For example, we wold prefer the reverb from a piano note to be at the same pitch as the original note (even thogh, owing to modal pecliarities, certain real rooms are said to exhibit the contrary behavior). If or phase algorithm fails this test, we may be forced to se excessively long FFT sizes in an attempt to provide sfficient freqency resoltion Freqency weighting according to signal history We wold like the instantaneos freqency information, as spplied by the phase deltas, to be based on the inpt signal s entire recent past, not jst the last cople of inpt frames. This cold be problematic considering that, given a long reverb decay time, each freqency bin may be accmlating data from many seconds worth of fndamental freqencies, harmonics, freqency sweeps, vibrato, splatter from adjacent bins, notes of qestionable pitch, transients, noise, and who knows what. The phase vocoder s assmption of one partial per freqency bin may not be overly restrictive when applied to a simple inpt signal, bt it cold prove troblesome in the case of a reverb, where each STFT bin is soaking p the past like a sponge. (On the other hand, the traditional phase vocoder does not fall apart and refse to process adio to which reverb has already been applied; it sets an example by making the best of a difficlt sitation.) If the decaying remains of several different partials are all competing for representation within a single STFT bin, greater representation shold be afforded to those partials having greater magnitde and longer dration, Page 8 of 15

9 with more recent contribtions being weighted more heavily, in accordance with that bin s decay rate. Ideally this weighting shold happen as if by magic, with no gly ad hoc procedres No nwanted periodicities The response to sstained inpt signals sch as msical notes shold not have obvios STFT-related periodicities, either at the hop period or at the FFT size N Control of phase coherence and roghness At times we may desire an artificially smooth effect like of the time-freeze. However, if we want a natral sonding reverb, the phase evoltion shold avoid being too rigid, mechanical, or nnatrally smooth. Therefore, it may be sefl to have parametric control over the amont of phase coherence vs. phase randomization. Also, the otpt shold not have an npleasant amont of roghness or excessive beating, as discssed in sections 5.2 and 5.3. In short, the phases shold be sch that the reverb sonds good for a wide variety of inpt signals. Many phase generation methods fail to meet one or more of the above criteria Phase Coherence We will examine two types of phase incoherence as they apply to the crrent algorithm: horizontal and vertical incoherence. Horizontal (or interframe) incoherence is cased by nwanted phase changes within a single bin from one STFT frame to the next. This can reslt if a phase vocoder fails to propagate the phase correctly when performing time scaling. Vertical (or intraframe) incoherence is a loss of the original phase relationship between adjacent STFT channels. The reslting amplitde modlation (or beating) is said to be a case of the phase vocoder s characteristic reverberant sond. As mentioned previosly, it is not clear to what extent phase incoherence may be a problem in the case of a reverb Roghness, Beating and Related Unpleasantries The aditory sensation of roghness is familiar from the sond of an ot-of-tne piano, in which rapid beating reslts from nearby strings whose vibrations go in and ot of phase with each other. Many perceptal experiments have been done regarding the aditory sensation of roghness. Roghness is characterized as a rapid series of brief aditory events, where the time interval is short enogh (i.e., less than abot 30 ms.) that the events are not perceived as individal events. Aditory roghness is most prononced when the sond incldes spectrally coherent flctations, in which case the roghness can be minimized by randomizing the Forier magnitdes and phases. This type of randomization happens atomatically in reverberant environments. Indeed, a redction in roghness may be one of the perceptal benefits of adding reverberation [20, 21]. High-qality reverb algorithms, and even actal room responses, can still exhibit some fltter, beating, roghness or nevenness, despite high echo and modal densities. Even thogh reverberation blrs and smoothes transients and complex signals, it makes sine-like signals less smooth, becase the lack of phase coherence cases amplitde flctations (see Figre 11 in the next section). However, these flctations may be too slow to fit the above definition of roghness Reverse-Engineering Reverb In the wild, reverberation is created by a smmation of delayed echoes, a process that can be viewed as a convoltion or tapped FIR delay, as shown in Figre 10. Figre 10: Tapped delay line. It is typical to examine a reverb in terms of its time domain implse response, bt since we are working in the freqency domain, it might be instrctive to Page 9 of 15

10 consider how sch as system responds to a qasisinsoidal inpt. The steady-state response of an FIR filter to a sine wave is a sine of the same freqency, with amplitde and phase determined by the gains and delays of the taps. In the real world, however, a sine-like sond wold have finite attack and decay times. We wold like to nderstand how these transitions affect the amplitde, instantaneos freqency and phase coherence of the otpt signal and corresponding STFT. Figre 11 shows the response of a simlated reverb to a time-limited sine wave. The reverb was simlated by windowing white noise with an exponential decay; the reslt was convolved with a windowed sinsoid having instantaneos attack and decay. Instead of a smooth response, we see a great deal of amplitde flctation (Figre 11, bottom) reslting from phase differences at the convoltion otpt taps. (The reslt sonds a bit like Morse code.) The amont of flctation appears to increase with longer decay times. Figre 12: (Top) Envelope of the sm of two delay taps in response to a windowed sinsoid. (Middle) Magnitde of the sinsoid s center STFT channel (Bin 80) and two adjacent channels. (Bottom) Phase increment (freqency) of the same three STFT channels. The top portion of Figre 12 shows the overall otpt amplitde. When the sine wave s attack reaches a new delay tap (arond the midpoint of the plot), the amplitde of the overall otpt signal sddenly changes (in this case, drops), reflecting the extent to which the new tap s otpt is in phase with the otpt of the preceding tap(s). Figre 11: (Top) Envelope of windowed sinsoid. (Middle) Envelope of white noise windowed with an exponential decay. (Bottom) Envelope of the windowed sinsoid convolved with the exponential decay. In Figre 12, we take a closer look at this phenomenon. Here, we see the otpt of the sm of two delay taps in response to a sine wave with instantaneos onset. In the middle part of Figre 12, we see the magnitde response of the three STFT channels closest to the freqency of the inpt sinsoid (all of whom are within the region of inflence of the spectral peak) [10]. The sdden changes in magnitde cased by the emergence of the signal from the new delay tap are smoothed by or se of a Hanning window. In the bottom portion of Figre 12, we see the instantaneos freqencies of the same three STFT channels, as measred by the phase difference between adjacent frames. Here, we observe that the instantaneos freqencies, which had settled into a steady state, are sddenly disrpted by the emergence of the new delayed sinsoid. As with the magnitdes, the sdden changes in phase are filtered by the window, Page 10 of 15

11 casing short, smooth variations in the instantaneos freqencies of the nearby channels. Once the disrption has passed, all the adjacent channels converge in phase and freqency to match the inpt sinsoid, ths resming vertical phase coherence. The flctations in instantaneos freqency are generally too small and brief to be perceived as pitch changes. The main perceptible effect of the tapped delay s sine response is the flctation of the magnitdes. It is nclear to what extent the phase flctations are perceived independently from their impact on the otpt amplitde, bt one way or another they appear to be related to the phasiness phenomenon. Each time the attack or decay of a windowed sinsoid reaches a new delay tap, there is a sdden loss of horizontal phase coherence, as well as a brief distrbance of the vertical phase coherence. The time it takes the phases to reach a new eqilibrim depends on the attack or decay time as well as the FFT window sed. It has been estimated that a high-qality reverb shold have an echo density of as many as 10,000 echoes per second [2]. This implies that the aforementioned phase distrbances shold occr at a very high rate relative to typical STFT window and hop sizes. However, looking at the sine response of or simlated reverb in Figre 11, we notice that, de to convoltion s inherent smoothing property, the large amplitde flctations happen on a mch slower time-scale, on the order of typical STFT frame rates. Any one particlar echo will have minimal impact on the overall signal becase its ability to affect the otpt phase is limited by its magnitde in relation to that of the overall signal Propagation of Instantaneos Freqency Figre 13: Phase comptation based on contination of the instantaneos freqency. If we compte and propagate the instantaneos freqencies of the sm of the inpt and feedback STFTs (from Figre 7), the reslting phase algorithm (Figre 13) resembles that of or time-freeze algorithm in Section 3. The response of the reslting Spectral Magnitde Decay algorithm to a windowed sinsoidal inpt is a perfect, thogh artificially smooth, enveloped sine wave, as shown in Figre 14. Note that the amplitde of the otpt grows as more energy is fed into the system and begins to decay as the inpt goes silent. The sine response does not sffer from any of the phase incoherence exhibited in Figre 11; however, this changes when we introdce more complex inpt signals. The above observations sggest that we might consider disrpting the phase from time to time, casing a brief loss of phase coherence. 6. PHASE GENERATION METHODS There are many possible ways of generating the phase signal. We will begin with a deterministic phase propagation method. Figre 14: Time-limited sinsoidal inpt (top); smooth Spectral Magnitde Decay response sing freqency propagation (bottom). Page 11 of 15

12 If we compte the instantaneos freqencies by taking phase differences between two frames that are one sample apart, as recommended in [16] and [22], the implse response is the same as that of or time-freeze method; i.e., a decaying, N-periodic series of implses, as shown in Figre 4. However, if we derive!! k by sbtracting the phases of frames R samples apart (instead of one sample apart), the implse response is somewhat less reglar, perhaps becase the instantaneos freqency is only known modlo 2! [22]. Note that or basic Spectral Magnitde Decay algorithm, shown in Figre 7, atomatically weights the inflence of the phases of the inpt and feedback signals according to their respective magnitdes. As mentioned in [14] (in the context of phase locking), In a sm of complex nmbers, the smmand with the greatest modls natrally has the strongest effect on the phase of the sm. Ths, this algorithm shold satisfy the desired phase signal characteristics and However, while the time-freeze reverb (Figre 3) has a pleasant qality, the Vega response [17] of or Spectral Magnitde Decay algorithm sing instantaneos freqency propagation (Figres 7 and 13) can be somewhat jittery, in violation of characteristic This may be de to corrption of the instantaneos freqency information, becase we are deriving a single freqency from the sm of the inpt and feedback signals in violation of the phase vocoder s assmption of one partial per freqency channel. Frthermore, as mentioned, the implse response is extremely sparse and N-periodic, in violation of characteristic Phase Randomization The resemblance of room responses to decaying white noise and the natral occrrence of phase randomization in reverberant environments sggest that we consider randomizing the phases. The phases of each freqency channel k can be modified at each frame by adding a random offset [23]: s! k ( t ) =! k t ( )+V k, " k,, where! k ( t ) is the instantaneos phase of the k th freqency channel at time t,! k s ( ) is the instantaneos synthesis (otpt) phase of t the k th freqency channel at time t,! k, is a niform random variable over [-π,π], and [ ]. V k,! 0,1 Ths, when V k, = 0 for all freqency channels k and all times t, no phase randomization is applied. If V k, = 1, the phase offsets will be completely random [23]. When the inpt to or Spectral Magnitde Decay algorithm is an implse, nvoiced speech or other noiselike signals, phase randomization (or phase dithering ) prodces a high-qality response; many msical inpts also yield an acceptable otpt qality. Unfortnately, with short STFT windows and pitched or sine-like inpt signals, phase randomization can prodce a whisperization effect, becase short Forier transforms have a small nmber of channels with poor freqency resoltion, and the randomized phases can t help define the instantaneos freqencies [8]. On the other hand, long windows prodce long latency times, which can be nacceptable for real-time applications. In addition, fll phase randomization does not allow freqency resoltion finer than that of the FFT channel spacing, in violation of characteristic As a reslt, we consider combining the instantaneos freqency propagation with partial phase randomization Partial Phase Randomization By controlling V k,, we have a great deal of freedom regarding which channels ndergo phase randomization, how freqently, and to what degree. As V k, increases from 0 to 1, the reslting random modlation disrpts the long-term periodicities, increasing the effective bandwidth of each sinsoidal component from a spectral line to a narrowband noise resembling the Forier transform of the synthesis window [23]. (Tre reverberation may have a similar broadening effect on inpt spectral lines.) Page 12 of 15

13 Ideally, we wold like to preserve and reverberate the pitch information while adding a controlled amont of phase randomization. However, there may be an inherent conflict between reqirements and To the extent we randomize the phases to diffse the implse response, the reslting freqency modlations may tend to blr the sb-bin freqency resoltion. Since this presmably occrs in tre reverberation as well, this may not be a problem so mch as a control opportnity. We can apply phase randomization either inside or after the feedback loop. Applying randomization inside the loop, as shown in Figre 15, may diffse the implse response more efficiently, sing a smaller amont of randomization, than if we dither the phases otside the loop. However, adding randomization within the loop may permanently corrpt the sb-bin freqency information. The optimal placement and depth of phase randomization, as well as which freqency bins it shold be applied to and how often, remain sbjects for frther experimentation. For example, we may want to apply different amonts of phase randomization depending on whether a freqency channel is identified as being in the vicinity of a spectral peak. Also, we may want to apply more phase randomization to higher freqency channels, on the assmption that they don t serve as coherent harmonics Intermittent Phase Disrption Along the lines of what we fond in Section 5.4 regarding the phase disrption observed when a note onset or offset reached a new delay tap, we might consider temporarily disrpting the phases from time to time, and then allowing phase coherence to resme. One way to do this wold be to advance or retard the phases at irreglar intervals, by simply scaling all the phases by an integer mltiple as is done in integer time-scaling [8, 10]. The reslting temporal hickp wold momentarily disrpt the phase coherence. Figre 15: Spectral Magnitde Decay algorithm, inclding phase randomization, roomlevel(! ) and drymix. 7. RESULTS 7.1. Implse Response Given sfficient phase randomization, the implse response of the Spectral Magnitde Decay algorithm is qite good and resembles decaying noise, with the desired freqency-dependent decay times and no nwanted coloration or metallic effects. The perceived echo density can be qite high bt (especially if the phase randomization is applied at the otpt) does not necessarily increase over time, nlike in real rooms and other reverb algorithms. Perceptally, the lack of increasing echo density does not appear to be a problem, since late reverb is generally defined to begin at the point where individal reflections become indistingishable. Page 13 of 15

14 7.2. Msic Response Given FFT sizes in the neighborhood of 8192 samples and moderate amonts of phase randomization, the Spectral Magnitde Decay reverb has a reasonably good qality otpt, thogh not good as that of convoltion or the best time domain FDN reverbs. There can be a slight echoey qality to the otpt, and if the FFT sizes are redced, the algorithm begins to sffer from whisperization. The parallel time-freeze method prodces a higher qality otpt and, given limited phase randomization, is more tolerant of smaller FFT sizes. However, as mentioned previosly, there are efficiency isses for longer decay times. 8. REMAINING PROBLEMS AND AREAS FOR FUTURE RESEARCH We wold like to frther explore the following areas: 1. When, where, and how often shold we apply what amonts of phase randomization? 2. Shold we randomize the spectral magnitdes instead of (or in addition to) the phases? 3. How can we control the modal density (average nmber of resonances per Hz) [2] to simlate the coloration of small, highly reflective rooms (e.g., bathrooms)? 4. What is the case of the echoey or jittery qality sometimes exhibited by the Spectral Magnitde Decay reverb, and what is the best way to address this problem? 5. Can the Spectral Magnitde Decay reverb be modified to tolerate smaller FFT sizes, to redce the latency? 6. How can the efficiency of the parallel time-freeze method be improved? 9. CONCLUSIONS The Spectral Magnitde Decay method enables analysis-based synthesis of late reverberation, with easily controllable decay times as a fnction of freqency. If desired, mltiple otpt channels cold be generated sing mltiple inverse Forier transforms with different phase randomization. The Spectral Magnitde Decay method reqires less memory than time domain Feedback Delay Networks, and it gives independent control over the reverb energy and decay time in each freqency channel. Unlike convoltion reverbs, the crrent approach provides simple parametric control over the decay spectra, with a comptational cost independent of decay time. This method cold be especially promising for systems in which the adio has already been transformed into the freqency domain for other types of processing. Errata or additional information on this topic may be provided at Some of the methods described in this paper may be the sbject of a pending patent application. 10. REFERENCES [1] W. Gardner, Reverberation Algorithms, in Applications of Digital Signal Processing to Adio and Acostics, Klwer Academic Pblishers, Norwell, MA, [2] J.-M. Jot and A. Chaigne, Digital delay networks for designing artificial reverberators, Proc. 90th Conv. Adio Eng. Soc. (preprint no. 3030), [3] J. Laroche and M. Dolson, Phase-Vocoder: Abot this phasiness bsiness, Proc. IEEE Workshop on Applications of Signal Processing to Adio and Acostics, New Paltz, NY, [4] J. Moorer, Abot this reverberation bsiness, Compter Msic Jornal, 3(2):13-28, [5] J. Flanagan and R. Golden, "Phase vocoder," Bell Syst. Tech. J., vol. 45, pp , 1966 November. Page 14 of 15

15 [6] D. Griffin and J. Lim, "Signal estimation from modified short-time forier transform," IEEE Trans. Acost., Speech, Signal Processing, vol. ASSP-32, no. 2, pp , 1984 April. [7] M. Dolson, The phase vocoder: A ttorial, Compter Msic Jornal, vol. 10, pp , [8] D. Arfib, F. Keiler, U. Zölzer, Time-freqency Processing, DAFX Digital Adio Effects, John Wiley & Sons, Ltd., England, [9] A. De Götzen, N. Bernardini, D. Arfib, Traditional (?) Implementations of a Phase-Vocoder: The Tricks of the Trade, Proceedings of the COST G-6 Conference on Digital Adio Effects (DAFX-00), Verona, Italy, 2000 December 7-9. [10] J. Laroche and M. Dolson, Improved Phase Vocoder Time-Scale Modification of Adio, IEEE Transactions on Speech and Adio Processing, Vol. 7, No. 3, 1999 May. [11] R. Schafer, The Tning of the World: The Sondscape, Alfred A. Knopf, New York, [12] Silophone web site, [13] Infinity DSP Sample Looping Tools manal, Antares Adio Technologies, [14] J. Laroche, Time and pitch scale modification of adio signals, in Applications of Digital Signal Processing to Adio and Acostics, Klwer Academic Pblishers, Norwell, MA, [17] S. Vega, Tom s Diner, Solitde Standing, A&M Records, [18] D. Dorran, E. Coyle, R. Lawlor, "An Efficient Phasiness Redction Techniqe for Moderate Adio Time-Scale Modification, Proceedings of the 7 th International Conference on Digital Adio Effects (DAFx 04), Naples, Italy, 2004 October 5-8. [19] M. Schroeder, New method of measring reverberation time, J. Acost. Soc. Am., 37: , [20] E. Terhardt, On the perception of periodic sond flctations (roghness), Acostica 30, , [21] E. Terhardt, Aditory roghness, [22] J. Brown and M. Pckette, A high resoltion fndamental freqency determination based on phase changes of the Forier transform, J. Acost. Soc. Am., 94(2): , [23] M. Macon and M. Clements, Sinsoidal Modeling and Modification of Unvoiced Speech, IEEE Transactions on Speech and Adio Processing, pp , 1997 November. [15] M. Pckette, Phase-locked Vocoder, IEEE ASSP Conference on Applications of Signal Processing to Adio and Acostics, Mohonk [16] R. Sssman and J. Laroche, Application of the Phase Vocoder to Pitch-Preserving Synchronization of an Adio Stream to an External Clock, Proc IEEE Workshop on Applications of Signal Processing to Adio and Acostics, New Paltz, NY, 1999 October Page 15 of 15