MUS421/EE367B Applications Lecture 9C: Time Scale Modification (TSM) and Frequency Scaling/Shifting

Transcription

1 MUS421/EE367B Applications Lecture 9C: Time Scale Modification (TSM) and Frequency Scaling/Shifting Julius O. Smith III Center for Computer Research in Music and Acoustics (CCRMA) Department of Music, Stanford University Stanford, California 9435 March 24, 214 Outline Time Scale Modification (TSM) Vocoder methods SOLA (Synchronous OverLap Add) method TSM Duality with Frequency Scaling Frequency Shifting by Complex Modulation and Single-Sideband Filtering 1

2 Time-Scale Modification Using a WOLA Phase Vocoder 1. Perform a short-time Fourier transform (STFT) using hop size R. 2. To slow down by the factor α > 1, resynthesize using hop size R α, creating the new spectral frames by interpolating the original STFT. Spectral magnitude may be interpolated in a straightforward manner (e.g., using linear interpolation of nearest adjacent spectral frames) Spectral phase is tricky, since there s no exact way to do it Reference: Time-Scale Modification with Inconsistent Constraints by Tom F. Quatieri and T. E. Hanna 2

3 Phase Continuation in a Time-Scaling Vocoder There are two conflicting desiderata when deciding how to continue the phase from one frame to the next: (1) Sinusoids should pick up where they left off in the previous frame (2) The relative phase from bin to bin should be preserved in each FFT To satisfy condition (1), it is necessary to replace the original phase of each frame by the phase corresponding to smooth continuation from the previous frame (which is generally an interpolated frame) Altering the phase of a spectral frame changes its amplitude envelope in the time domain thus, it no longer looks like a windowed signal segment Using the WOLA framework helps because the post-window guarantees a smooth cross-fade from frame to frame Random amplitude-modulation distortion is generally heard as reverberation (a.k.a. phasiness ) 3

4 When condition (2) is violated, the signal frame suffers dispersion in the time domain For steady-state signals (filtered noise and/or steady tones), temporal dispersion should not be audible, while frames containing distinct pulses will generally become more smeared out in time It is not possible in general to satisfy both conditions (1) and (2) simultaneously, but either can be satisfied at the expense of the other Generally speaking, transient frames should emphasize condition (2) (preserve relative phase across the spectrum), allowing the overlap-add cross-fade to take care of the phase discontinuity at the frame boundaries For stationary frames, satisfying condition (1), (phase continuation from frame to frame) is more valuable 4

5 More Recent Phase Continuation Methods for the Time-Scaling Phase Vocoder See references in the text for pointers to these and other articles on time-scale modification using the phase vocoder: Miller Puckette: Phase-Locked Vocoder Jean Laroche and Mark Dolson: Phase-Vocoder Processed in Separate Frequency Bands This work indicates it is largely sufficient to preserve relative phase across FFT bins (i.e., satisfy condition (2)) only at spectral peaks and their immediate vicinity. 5

6 Time-Scaling Phase Vocoder in Matlab Software resources available via the Internet: Dan Ellis has posted some clean and simple Matlab software 1 implementing time-scale modification using the phase vocoder. There are also some sound examples 2 comparing it to the SOLA-FS (Synchronous OverLap Add, Fixed Synthesis) algorithm by Hejna and Musicus (a time-domain method based on an overlap-add decomposition of the time waveform, with input windows shifted and output windows regularly spaced for a fixed output synthesis window-rate said to be more computationally efficient than SOLA) James Salsman distributes Matlab for analysis 3 and synthesis 4 (with time scaling) based on a variant of the phase-locked vocoder ftp://ftp.bovik.org/matlab/af.m 4 ftp://ftp.bovik.org/matlab/fs.m 6

7 Example Vocoder Waveforms Test Case Two constant-amplitude sinusoids Frequencies separated by 1 Hz (2 beats/sec) Vocoder Parameters 2x time expansion Vocoder frame size = 45ms 22 Hz (frame size half a beat pulse) Vocoder hop size = 1/4 frame (Hann or Hamming window) Vocoder Types Phase-Continued Vocoder (Dan Ellis s Matlab Code) Salsman s Phase-Locked Vocoder (Puckette, Laroche, Dolson) SOLA-FS by Hejna and Musicus (Dan Ellis s Matlab Code) 7

8 Phase-Continued Vocoder Waveforms at 2X Expansion 1 pvoc samples (1:11264).5 Amplitude Time (samples) 1 Output samples (1:22528).5 Amplitude Time (samples) x 1 4 Phase is continuous across frames Relative phase not preserved across FFT bins 8

9 Phase-Continued Vocoder Spectra at 2X Expansion 6 pvoc samples (1:11264) Magnitude (db) Frequency (khz) 8 Output Spectrum samples (1:22528) 6 Magnitude (db) Frequency (khz) 9

10 Phase-Locked Vocoder Waveforms at 2X Time Expansion 1 salsman samples (1:11264).5 Amplitude Time (samples) 1 Output samples (1:22528).5 Amplitude Time (samples) x 1 4 Amplitude envelope better preserved 1

11 Phase-Locked Vocoder Spectra at 2X Time Expansion 6 salsman samples (1:11264) Magnitude (db) Frequency (khz) 8 Output Spectrum samples (1:22528) 6 Magnitude (db) Frequency (khz) 11

12 SOLA-FS Waveforms at 2X Expansion 1 solafs samples (1:11264).5 Amplitude Time (samples) 1 Output samples (1:22528).5 Amplitude Time (samples) x 1 4 Time domain method Phase preserved within each frame 12

13 Phase not continuous across frames, but frames cross-fade at best locations in time, based on maximum cross-correlation SOLA-FS Spectra at 2X Expansion 6 solafs samples (1:11264) Magnitude (db) Frequency (khz) 8 Output Spectrum samples (1:22528) 6 Magnitude (db) Frequency (khz) 13

14 TSM by Synchronous OverLap Add (SOLA) SOLA method SOLA-FS method SOLA works well for single voices Vocoder works well for legato choruses 14

15 Frequency Scaling/Shifting Frequency Scaling Frequency scaling is the Fourier dual of Time-Scale Modification (TSM) Thus, one can perform TSM and resample to implement frequency scaling Frequency scaling preserves harmonicity Frequency Shifting Frequency shifting does not preserve harmonicity: X(ω) X(ω ) Can modulate by a complex sinusoid (as in the STFT filter-bank interpretation) preceded or followed by filtering out negative-frequency components A single-sideband filter is easily designed as a half-band lowpass filter h(n) modulated by j n to rotate its frequency response by π/2 (see FIR Hilbert-Transform Design in the lecture on the Window Method for FIR Filter Design) 15