Physics-Based Sound Synthesis

Transcription

1 1 Physics-Based Sound Synthesis ELEC-E Audio Signal Processing, Lecture #8 Vesa Välimäki Sound check Course Schedule in General issues (Vesa & Fabian) History and future of audio DSP (Vesa) Digital filters in audio (Vesa) Audio filter design (Vesa) Analysis of audio signals (Vesa) No lecture (Evaluation week for Period III) Audio effects processing (Fabian) Synthesis of audio signals (Fabian) D sound (Prof. Ville Pulkki) Physics-based sound synthesis (Vesa) Sampling rate conversion (Vesa) Audio coding (Vesa) Vesa Välimäki 2

2 2 Outline Physical modeling of musical instruments (Soitinmallinnus in Finnish) Classification of methods and some examples Digital waveguide modeling Modeling of stringed musical instruments The acoustic guitar, the harpsichord, the piano, the Clavinet, the electric guitar, virtual air guitars, Two demos: Virtual MS-20 Filter & Flutter Echo Modeling Vesa Välimäki 3 DEMO MS-20 Filter Joonas & Mikko Vesa Välimäki

3 3 Taxonomy of Sound Synthesis Methods (J. O. Smith, 1991) (Source: Vesa Välimäki 5 Karplus-Strong Algorithm Invented by Karplus and Strong (late 1970s; published 1983) A self-modifying wavetable Wavetable is interpreted as a delay line, where the waveform propagates Modifier = average of 2 contiguous samples Dullness is reduced considerably Amazing results with simple processing Control fundamental freq. Noise Wavetable (Delay line) Out Modifier Vesa Välimäki 6

4 4 Karplus-Strong: The Movie Averaging of zero-mean random numbers Initial state (random) Two-point average of previous state Vesa Välimäki 7 Karplus-Strong Sound Examples Four octaves of C-major scale (65 Hz Hz) played with the Karplus-Strong algorithm 1) Sampling rate 8000 Hz 2) Sampling rate Hz 3) Sampling rate Hz 4) Sampling rate Hz High notes are out of tune! f = f s / (L + 0.5), where the half-sample delay is caused by averaging Wavetable length L is rounded to the nearest integer Timbre and decay time depend on sampling rate Vesa Välimäki 8

5 5 Tuning Problem Desired vs. actual fundamental frequency of KS algorithm for a 50-kHz sampling rate (left) Corresponding approximation error in semitones (right) Quantized Frequency (khz) Desired Frequency (khz) Pitch Error (Semitones) Frequency (khz) Vesa Välimäki 9 Sound Synthesis by Physical Modeling Simulate the sound production mechanism In speech synthesis and coding, this is an old tradition High-quality synthesis of string and wind instruments More sophisticated signal processing than earlier Compared to subtractive synthesis, sampling & wavetable synthesis, FM synthesis,... Combines linear & nonlinear methods, filtering, sampling,... Considered computationally expensive Vesa Välimäki 10

6 6 Physical Modeling Techniques 1) Source-Filter Modeling Physically informed subtractive synthesis 2) Finite Difference Modeling Discretized wave equation (difference equations) 3) Mass-Spring Networks CORDIS-ANIMA by ACROE (Grenoble, France) 4) Modal Synthesis Modalys software by IRCAM (Paris, France); formerly Mosaic The functional transform method by Trautmann and Rabenstein 5) Wave Digital Filters Method to convert models of continuous-time systems into digital filters 6) Digital Waveguide Synthesis Physical modeling using digital filters and delays; Main topic of this lecture Vesa Välimäki 11 Source-Filter Model Source signal can be recorded or synthetic Filter can model a resonating structure For example, the vocal tract, the soundbox of an instrument, Glottal waveform extracted from speech (Prof. Paavo Alku) With formant filters: Vesa Välimäki 12

7 7 Finite-Difference Models Discretization of the partial differential equation Discretized approximation and its recursive implementation Vesa Välimäki 13 Finite-Difference Models The next state is computed based on 2 previous states For the basic wave equation Time Space Vesa Välimäki 14

8 8 Mass-Spring Networks A distributed mechanical system can be approximated by a finite number of small mass elements (Figure from: T. D. Rossing: The Science of Sound. Second Edition. Addison-Wesley, 1990.) Vesa Välimäki 15 Mass-Spring Networks (2) Mass elements are connected with spring elements Micro-dampers are used to simulate losses A mass-spring network can be simulated with a digital filter structure (Cadoz et al., 1983; Välimäki et al., 2006) Basic elements are discretized separately Apply Hooke s law (F = -kx) and Newton s second law (F = ma) Vesa Välimäki 16

9 9 Mass-Spring Networks The Movie For example, 3 masses 3 transversal modes Movie link: Vesa Välimäki 17 Modal Synthesis Vibrating modes of a structure (instrument) are modeled separately Filter bank of 2 nd -order digital resonators R k (z) Type and location of excitation affects the input of each resonator R 1 ( z) y(n) Input Matrix R 2 ( z)... R K (z) Vesa Välimäki 18

10 10 Modalys Modal synthesis software developed at IRCAM (Paris, France) (Adrien, 1991; Morrison & Adrien, 1991) Contains many excitation and resonator models Analyzed parameters for instruments bodies, bells, strings, bars, Allows arbitrary connections Blow a string, bow a tube, pluck a drum etc. Sound examples 1.Trumpet 2.Clarinet (very bad player) 3.Bowing a metal tube 4.Gong with time-varying properties Source: Vesa Välimäki 19 Wave Digital Filters Developed by Fettweis (1970-) to simulate analog filters Similarly, a lumped acoustic model can be converted into a digital one using the bilinear transform Mass Inductor Damper Resistor Spring Capacitor Useful for the modeling of nonlinear excitation mechanisms Hammer of the piano, reed in wind instruments Also linear resonator models E.g. woodwind bore (tube) Currently used for modeling musical electronic circuits Vesa Välimäki 20

11 11 History of Musical Instrument Modeling Pythagoras studied string vibrations (about 500 B.C.) Basics of string vibration were understood around 1770 Sound synthesis by mathematical modeling (Hiller & Ruiz, 1971) Mass-spring networks for physical modeling of musical instruments in Grenoble, France (1970 s) Karplus and Strong invented a modified wavetable synthesis method in 1979 (published in 1983) Jaffe and Smith extended the Karplus-Strong algorithm (1983) Julius Smith developed the theory of digital waveguides (1985-) Products since 1994 (Yamaha, Korg, Creative, Roland ) Vesa Välimäki 21 Taxonomy of Sound Synthesis Methods (J. O. Smith, 1991) (Source: Vesa Välimäki 22

12 12 Plucked-String Synthesis Model Originally developed at CCRMA, Stanford University (Jaffe & Smith, 1983) Further developments at TKK (Välimäki et al., 1996; Tolonen 1998, Karjalainen et al., 1998) In Out FD Delay line H l (z) Fractional delay filter Loop filter Fundamental frequency Decay rate Vesa Välimäki 23 Tuning Filter Fractional delay filter F(z) fine-tunes the delay-line length so that the pitch is right Two alternatives 1) Allpass filter (Jaffe & Smith, 1983) First-order allpass filter usually Transient problems in time-varying cases (e.g., glissando) 2) Lagrange interpolation (Karjalainen & Laine, 1991; Välimäki et al., 1996) Linear interpolation (1st-order Lagrange) not good enough, because high frequencies will be damped Order 3 or 4 sufficient at 44.1 khz Vesa Välimäki 24

13 13 Waveguide Synth Tuning with Allpass filter Phase delay of the first-order allpass filter at low frequencies (dc): D = (1 a 1 ) / (1 + a 1 ) Phase delay: ( )/ Phase response divided by angular frequency Note the minus sign Solve a 1 for desired delay D: a D D x(n) Vesa Välimäki 25 - a 1 z -1 a 1 y(n) Tuning with Lagrange Interpolation Use, for example, a 3rd-order Lagrange FD filter (N = 3), which is a four-tap FIR filter Coefficient formulas for 3rd-order Lagrange FD filter (1 < D < 2): Vesa Välimäki 26

14 14 Loop Filter Loop filter H l (z) controls the decay rate of harmonics One choice is a one-pole filter ( leaky integrator ) (Välimäki et al., 1996) 1 a Hl ( z) g 1 az Parameter g determines overall decay rate (0 < g < 1) Parameter a 1 determines frequency-dependent decay ( 1 < a 1 < 0) Vesa Välimäki 27 Structure of Basic String Model Implementation using a one-pole loop filter and a 3rd-order FIR interpolation filter (Laurson et al., 2001) x(n) y(n) y L d (n) g( n)[1 a( n)] h(3) h (2) h (1) h(0) 1 z 1 z 1 z FIR fractional delay filter a(n) y 1 ( n) M 1 z Delay line z Loop filter Vesa Välimäki

15 15 Commuted String Instrument Model Instrument body is expensive to model A high-order digital filter is needed Computational savings are obtained by commuting the pluck, string, and the body (Smith, 1993; Karjalainen et al., 1993) a) (n) Excitation String Body e(n) E(z) S(z) B(z) y(n) b) b(n) E(z) x(n) S(z) y(n) Commuted piano synthesis (Van Duyne and Smith, 1995) Vesa Välimäki 29 Parameter Estimation Anechoic recordings of guitar playing Signal analysis using short-time Fourier transform (Välimäki et al. 1996; Tolonen 1998) Parameter estimation using iterative methods (Erkut et al., 2000) Excitation signals obtained by either by 1) Inverse filtering (Välimäki et al. 1996) OR 2) by subtracting the harmonics from recorded tones & equalizing (Tolonen 1998; Välimäki & Tolonen, 1998) Vesa Välimäki 30

16 16 Short-Time Fourier Analysis Signal analysis using short-time Fourier transform (Välimäki et al. 1996; Tolonen 1998) Vesa Välimäki 31 Harmonics = Spectral Peaks Peak picking from the magnitude spectrum of each frame (Välimäki et al. 1996; Tolonen 1998) Vesa Välimäki 32

17 17 Envelopes of harmonics are created by connecting spectral peaks in time Decay rate estimate by linear regression (Välimäki et al. 1996; Tolonen 1998) Decay Rate of Harmonics Vesa Välimäki 33 One-pole loop-filter design (Välimäki et al. 1996) Target response is obtained from the decay rates Loop Filter Design Vesa Välimäki 34

18 18 Excitation signals obtained by either by 1) inverse filtering (Välimäki et al. 1996) Extracting Input Signal OR 2) subtracting harmonics from recorded tones & equalizing (Tolonen 1998; Välimäki & Tolonen, 1998) Vesa Välimäki 35 Structure of Guitar Synthesizer Commuted waveguide synthesis (sampling + modeling) Two string models in parallel (horizontal and vertical polarization) Excitations & special effects stored in a database (Laurson et al., 2001) Database of excitation signals To sympathetic coupling matrix Special effects Timbre control Pluckingpoint filter S h ( z) From sympathetic coupling matrix S v ( z ) Out Vesa Välimäki 36

19 19 Synthesis Examples Classical acoustic guitar 1. Guitar music from the Baroque era (J. S. Bach) 2. Modern guitar music (J. A. Muro) Sound examples available on the web: Vesa Välimäki 37 How about other instruments? Yes, we can do it! ( if they are stringed instruments) Vesa Välimäki 38

20 20 TKK Clavichord Synthesizer Modification of the TKK guitar synthesizer (Välimäki et al., 2003) Commuted waveguide synthesis with more samples Beginning of soundbox response in excitation signal, the rest is sampled! Trigger at release time Trigger at attack time Soundbox responses g sb End knocks g release Excitation signals Timbre control g i1 S 1 g c g o1 Sound g i2 g o2 S Vesa Välimäki 39 TKK Clavichord Synthesizer Clavichord synthesizer played by the ENP control software (produced by Mikael Laurson and Jonte Knif, Sibelius Academy) J. J. Froberger, Gigue from Suite no. 30 in A minor V. Välimäki et al., Computer Music J., Issue #1, 2003 (Sound examples on CMJ DVD with issue #4, 2003) Vesa Välimäki 40

21 21 Ancient keyboard instrument Since the year 1400 Based on plucked strings Like the guitar More complicated than guitar or clavichord Wide soundboard 1-2 keyboards 2-4 registers Harpsichord Vesa Välimäki 41 Anechoic Recordings Vesa Välimäki 42

22 22 Plucking Machinery (Picture from: T.D. Rossing, The Science of Sound, 2nd Ed. Addison-Wesley, 1990) Vesa Välimäki 43 TKK Harpsichord Synthesis Model Modification of the clavichord synthesis model (Välimäki et al. 2004) Excitation and key release sound are samples Vibrating string sound is synthesized algorithmically: S(z) ja R(z) Tail of soundboard response is modeled with a reverb algorithm Vesa Välimäki 44

23 23 Slight Change in String Model S(z) We inserted a feedforward comb filter Can increase the difference in decay rates of neighboring harmonics x (n) y(n) F(z) - L z 1 1 z R z + r Ripple filter a b - One-pole filter Vesa Välimäki 45 Before One-pole filter, delay line, FD filter in a feedback loop since 1991 x (n) y(n) F(z) L z z 1 a b - One-pole filter Vesa Välimäki 46

24 24 After One-pole filter, delay line, FD filter, and a feedforward comb filter in a feedback loop x (n) y(n) F(z) - L z 1 1 z R z + r Ripple filter a b - One-pole filter Vesa Välimäki 47 Some Variation We ve Always Wanted Measured data Approximations: No variations (1-pole filter only) Variations (one-pole filter and comb filter) Vesa Välimäki 48

25 25 But Does it Sound Better? If not, it doesn t pay to do it Informal listening test 1. Original recorded sound 2. Synthesis with old method 3. Synthesis with new method 4. Original recorded sound 5. Synthesis with new method Conclusion: Slight improvement in quality and slight increase in cost It pays Vesa Välimäki 49 Soundbox Modeling Soundbox amplifies and colors the string sound Coloration is imitated with a digital filter Soundboard resonances Short string parts behind the bridge Highest octave of 4-foot register (without dampers) Higher modes of string (not properly damped) Vesa Välimäki 50

26 26 Soundboard Impulse Response Vesa Välimäki 51 Digital Model of the Soundboard Use the source-filter principle Modeling with a reverb algorithm (Välimäki et al. 2004) Modification of the Feedback Delay Network Also 10th-order equalizer Delay line Lowpass Allpass DL 1 H 1 (z) A 1 (z) x(n). DL N H N (z) A N (z) y(n) g fb Vesa Välimäki 52

27 27 Measured Soundboard Impulse Response Vesa Välimäki 53 Modeled Soundboard Impulse Response Vesa Välimäki 54

28 28 Digital Waveguide Models Well suited to synthesis of string and wind instruments String or tube is modeled with a delay-line loop Excitation is either a sample or a nonlinear system Bowed string and wind instrument models use a model of the excitation Nonlinear mapping of input pressure or velocity Similarities to waveshaping synthesis Commuted synthesis of bowed strings also possible (Smith, 1995) Vesa Välimäki 55 Definition: Digital Waveguide Digital waveguide = bi-directional delay line (Smith, 1985) A model for 1-D wave propagation, such as a string or a tube Every sampling instant contains a sample of a propagating wave Interpretation of samples can be chosen Displacement for strings; sound pressure or volume velocity for tubes z -1 y + (n,k) z -1 z -1 z -1 y(n,k) z -1 z -1 z -1 z -1 y - (n,k) Vesa Välimäki 56

29 29 Output Signal from Digital Waveguide Principle of superposition: Sum of signals propagating in opposite directions at the same point z -1 y + (n,k) z -1 z -1 z -1 y(n,k) z -1 z -1 z -1 z -1 y - (n,k) Vesa Välimäki 57 Scattering in a Digital Waveguide If acoustic impedance Z changes, scattering occurs For example, tube radius changes, as in the vocal tract (Kelly & Lochbaum, 1962) Part of the wave is reflected while part gets through, when cross-sectional area A changes Vesa Välimäki 58

30 30 Kelly Lochbaum Tube Model (1962) Vesa Välimäki 59 Lossless String Model A lossless, rigidly terminated string Reflection coefficient 1 at both ends For theoretical use only (Figure from: Vesa Välimäki 60

31 31 Lossy Waveguide Model Lowpass filters at the ends of the digital waveguide Vesa Välimäki 61 Single-Delay-Loop Model If there s only 1 input and 1 output, all loss filters can be combined! (Smith, 1992; Karjalainen et al., 1998) A single-delay-loop model results Looks a lot like the Karplus Strong algorithm, but the filter is different Excitation to be stored in the delay line must imitate a pluck Vesa Välimäki 62

32 32 Waveguide Wind Instrument Models A woodwind model consists of 2 parts 1) Linear tube and open end (digital waveguide and filters) 2) Nonlinear coupling of excitation For example, clarinet model (Smith, 1986), flute model (Välimäki et al., 1992), brass instrument models (Cook, 1992) Vesa Välimäki 63 Products Using Waveguide Synthesis Physical modeling synthesizers Yamaha VL-1 & VL-7, 1994 Korg Prophecy, 1995 Sound cards Creative Sound Blaster AWE64 Creative Sound Blaster Live! Technology patented by Stanford University and Yamaha Yamaha VL-1 VL-1 sound examples Trumpet Tenor sax Shakuhachi (Sound examples from: Vesa Välimäki 64

33 33 Waveguide Modeling of the Clavinet Vesa Välimäki 65 Modeling and Synthesis of the Clavinet A waveguide model with novel features (Gabrielli et al., 2013) Excitation pulse model (velocity-dependent tangent hit) Damping (yarn) model with different string length Nonlinear pickup model Equalization filters simulating the tone switches Collaboration between Aalto University, Università Politecnica delle Marche, Ancona, Italy & Univ. of Edinburgh, UK Leonardo Gabrielli, Dr. Stefan Bilbao Vesa Välimäki 66

34 34 Waveguide Clavinet Synthesizer Synthesis with good yarning Played from MIDI keyboard Synthesis with bad yarning Vesa Välimäki 67 Electric Guitar Modeling and Synthesis A waveguide model with novel ideas (Lindroos et al., 2011) A new plucking model (pulse + filtered noise) Time-varying loop gain (to account for two-stage decay) Inharmonic comb filter model for the pickup position Distortion helps much Vesa Välimäki 68

35 35 TKK Virtual Air Guitars Digital model of an electric guitar with hand movement recognition Three alternative implementations: 1) Data gloves, 2) Colorful gloves and a web camera, 3) Control sticks (acceleration sensors & ultra-sound radar) Vesa Välimäki 69 Virtual Slide Guitar An air guitar played with a bottle neck! (Pakarinen, Puputti &Välimäki, 2008) Check our video: Vesa Välimäki 70

36 36 Conclusions Several methods available for physics-based sound synthesis Source-filter, modal, mass-spring, FDTD, wave digital, and digital waveguide models Digital waveguide synthesis used to be the most popular method With faster computers, modal synthesis and FDTD methods are becoming more attractive Vesa Välimäki 71 References (1) J. M. Adrien, The missing link: modal synthesis, in: G. De Poli, A. Piccialli, and C. Roads, eds. Representations of Musical Signals. MIT Press, Cambridge, Massachusetts, B. Bank, Physics-based sound synthesis of the piano, Report no. 54, TKK Lab. of Acoustics and Audio Signal Processing, June Report and sound examples available online at: C. Cadoz, A. Luciani, and J.-L. Florens, Responsive input devices and sound synthesis by simulation of instrumental mechanisms: the CORDIS system, Computer Music J., vol. 8 pp , L. Gabrielli, V. Välimäki, H. Penttinen, S. Squartini & S. Bilbao, A digital waveguide based approach for Clavinet modeling and synthesis, EURASIP J. Applied Signal Processing, Audio examples at: L. Hiller and P. Ruiz, Synthesizing musical sounds by solving the wave equation for vibrating objects: parts I and II, J. Audio Eng. Soc., vol. 19, no. 6, pp and no. 7, pp , 1971 D. Jaffe and J. O. Smith, Extensions of the Karplus-Strong plucked string algorithm, Computer Music Journal, vol. 7, no. 2, pp , Vesa Välimäki 72

37 37 References (2) M. Karjalainen, V. Välimäki, and T. Tolonen, From the Karplus-Strong algorithm to digital waveguide and beyond, Computer Music J., vol. 22, no. 3, pp , Available online at: K. Karplus and A. Strong, Digital synthesis of plucked string and drum timbres, Computer Music Journal, vol. 7, no. 2, pp ,1983. M. Laurson, C. Erkut, V. Välimäki, and M. Kuuskankare, Methods for Modeling Realistic Playing in Acoustic Guitar Synthesis, Computer Music J., vol. 25, no. 3, pp , N. Lindroos, H. Penttinen, and V. Välimäki, Parametric electric guitar synthesis, Computer Music Journal, vol. 35, no. 3, pp , Fall J. Morrison and J. M. Adrien, Control mechanisms in the MOSAIC synthesis program, in Proc. Int. Computer Music Conf., pp , Montreal, J. Pakarinen, T. Puputti, and V. Välimäki, Virtual slide guitar, Computer Music Journal, vol. 32, no. 3, pp , Fall Pakarinen, J., Tikander, M., and Karjalainen, M., ''Wave Digital Modeling of the Output Chain of a Vacuum-Tube Amplifier,'' in Proc. Int. Conf. Digital Audio Effects (DAFx'09), Como, Italy, Sept., H. Penttinen, J. Pakarinen, V. Välimäki, M. Laurson, H. Li, and M. Leman, Model-based sound synthesis of the guqin, Journal of the Acoustical Society of America, vol. 120, no. 6, pp , Dec Vesa Välimäki 73 References (3) J. O. Smith, Viewpoints on the history of digital synthesis, in Proc. Int. Computer Music Conf. (ICMC 91), pp. 1 10, Montreal, Canada, Oct A revised version is available at: J. O. Smith, Physical modeling using digital waveguides, Computer Music Journal, vol. 16, no. 4, pp ,1992. Available on-line at T. Tolonen, V. Välimäki, and M. Karjalainen, Evaluation of Modern Sound Synthesis Methods. Report no. 48, Lab. of Acoustics and Audio Signal Processing. Helsinki Univ. of Tech., Espoo, V. Välimäki, J. Huopaniemi, M. Karjalainen, and Z. Jánosy, Physical modeling of plucked string instruments with application to real-time sound synthesis, J. Audio Eng. Soc., vol. 44, no. 5, pp , May V. Välimäki, M. Laurson, and C. Erkut, Commuted waveguide synthesis of the clavichord, Computer Music Journal, vol. 27, no. 1, pp , Spring V. Välimäki, J. Parker, and J. S. Abel, Parametric spring reverberation effect, Journal of the Audio Engineering Society, vol. 58, no. 7/8, pp , July/August V. Välimäki, H. Penttinen, J. Knif, M. Laurson, and C. Erkut, Sound synthesis of the harpsichord using a computationally efficient physical model, EURASIP Journal on Applied Signal Processing, vol. 2004, no. 7, pp , June V. Välimäki, J. Pakarinen, C. Erkut, and M. Karjalainen, Discrete-time modelling of musical instruments, Reports on Progress in Physics, vol. 69, no. 1, pp. 1-78, Jan Vesa Välimäki 74