Music Technology Group, Universitat Pompeu Fabra, Barcelona, Spain {jordi.bonada,

Transcription

1 GENERATION OF GROWL-TYPE VOICE QUALITIES BY SPECTRAL MORPHING Jordi Bonada Merlijn Blaauw Music Technology Group, Universitat Pompeu Fabra, Barcelona, Spain {jordi.bonada, ABSTRACT In this paper we introduce a morph-based approach or generating voice source aperiodicities requently associated with strong vocal expressions, especially in singing. In our approach the excitation characteristics o one signal are combined the undamental requency and spectral envelope characteristics o another signal. An exemplar sustained sample o the target voice quality is looped and resampled in the time domain in order to generate a continuous signal matching the input voice s undamental requency. This operation preserves most o the sample s voice quality characteristics at the cost o scaling time and requency dimensions. While we ound the temporal scaling to be acceptable in many contexts, the requency scaling has to be inverted in order to generate appropriate spectral content or the source excitation s entire bandwidth. This can be accomplished by the phase-locked vocoder method o modulating harmonic bands in requency. Finally, the input signal s harmonic amplitudes and phases are applied to the transormed morph sample, allowing or a simple one-dimensional control o morph amount by linear interpolation with the input signal. The proposed system is evaluated and the results are discussed. Index Terms speech synthesis, singing synthesis, human voice, harmonic analysis, voice quality. INTRODUCTION Over the past years many researchers have identiied the importance o voice qualities outside o the modal range in transmitting emotion and mood []. Varying voice quality is also considered important or obtaining a natural-sounding results in a wide range o speech processing applications such as TTS, voice conversion and voice transormation. The ocus o this paper is on a class o voice qualities that are oten described as harsh, rough, hoarse or, which we will reer to as type in this paper. In singing, these -type voice qualities are requently used as an expressive resource or may be part o certain types o singing voices (e.g. a Louis Armstrong-type singing voice) [2]. The spectrum o modal singing voice can be approximated by a series o relatively stable harmonics and low energy noise. In contrast, what we consider the deining property o type voice qualities is the presence o rapid changes o timbre, timing and strength o source excitation events. These changes result in the appearance o sub-harmonics in the spectrum, and modulation patterns in the time-domain waveorm (jitter and shimmer). I those modulations are periodic, the undamental period and the period o modulation (macro period) can be clearly visible. Within the scope o this paper we assume that or -type voice qualities the perceived eect is mainly a result o the signal s source excitation rather than its spectral envelope (aside rom pulse-to-pulse variations). Our goal is to be able to generate -type voice qualities in a given, principally modal, singing voice signal, whether it is recorded or synthetic. In other words, we want our output signal to contain a source excitation appropriate or a -type voice quality, while at the same time containing pitch and spectral envelope obtained rom the input voice. We can approach this problem rom two directions. The more common approach is to model the characteristics o the source excitation associated with -type voice qualities and then apply these to the modal input signal. A second approach, the one that is pursued in this paper, is to take a recording that already contains the desired voice quality and then modiy its undamental requency and spectral envelope to match that o the input voice. There has been a considerable amount o work on the irst type o approach, the parametric modeling o -type source excitation. Schoentgen [] or instance gives an extensive overview o dierent methods o generating jitter and shimmer patterns, which is typically derived by statistical analysis o real recordings, or using models inspired by the physiology o the voice source. The modulation patterns are then typically applied by identiying and transorming individual voice source pulses in the input signal using (e.g. using TD-PSOLA). Examples o applications using this method include voice transormation [] and voice conversion [5]. An alternative approach proposed by Loscos [6] models the amplitude and phase behavior o subharmonics in the time-requency domain. The main problem with these approaches is that accurate estimation o parameters o these models can be diicult and costly, and ultimately the results obtained will be limited by the accuracy o the underlying model. There can be a high variability in properties o -type voice qualities between dierent singers or even between dierent utterances o the same singer, which makes coming up with a single model that covers all cases diicult. Finally, a problem that is oten mentioned with these methods

2 is that the results are very dependent on the type o input voice used. The second type o approach has been relatively unexplored. However, it could be considered a case o voice morphing, a concept that, in many variations, has been applied to a wide range o speech processing applications. Typically these use parallel recordings, apply a time-alignment between source and target, separate source excitation and spectral envelope or both, and do some orm o interpolation or replacement o either or both o the components. Cano [7] applies the time-aligned spectral envelope o one singer with the excitation (pitch mainly) o another to allow real-time singer impersonation or Karaoke. Another application is to smoothly interpolate between two speakers by independently interpolating source excitation and spectral envelope [8]. A more restrictive case o morphing has been applied to interpolate between spectral envelopes associated with certain voice qualities or emotions o a single speaker or singer [9, 0]. The main problem with applying any o these existing studies to our problem is that in our application, and many others, parallel recordings are not available. Another basic diiculty with using a morph-based approach is that processing voice signals with -type voice qualities can be diicult. Most vocoders used to transorm voice signals are designed or principally harmonic signals and are not particularly suited or handling sub-harmonics or detailed jitter/shimmer patterns. Using these methods directly will typically result in low quality outputs or a loss o voice quality. For instance, the phase-locked vocoder [] where harmonic requency bands are shited in order to pitch-shit signals works well when there is only one partial present per harmonic band. However, i a harmonic band contains sub-harmonics, all partials should be requency-shited by dierent amounts and their phases propagated dierently. Even the more advanced approaches that explicitly include a model o the glottal source (e.g. [2, ]) are typically limited to the lax/modal/tense range o voice qualities. Kawahara [] recently proposed some vocoder methods designed to speciically deal with the inegrained undamental requency variations oten observed in -type voice qualities. While these approaches seem to work well or resynthesis without modiication, whether they can provide satisactory generation or transormation o type voice qualities has not yet been ully explored. 2. PROPOSED SYSTEM Our aim is to modiy the voice quality characteristics o a given voiced utterance, without having to explicitly model the inner structure o the spectrum beyond its harmonic structure. For that purpose, we propose to morph the input signal with a sample that already posses the desired voice quality. The idea is to combine the excitation characteristics o this sample with the timbre characteristics o the input voice utterance. The sample is not constrained to be phonetically equivalent to the input voice. On the contrary, the idea is to use a reduced database o voice quality samples or any target utterance. In addition, we intend to integrate the morph eect into an existing singing voice synthesizer, and to be able to control the degree o morph. Figure shows the steps o the proposed morphing system. MORPH SAMPLE Random looping Harmonic analysis Resampling Spectral analysis (windowing, FFT) Harmonic mapping Filter (resampled morph sample spectrum) Spectral mixer Spectral synthesis (IFFT, OLA) OUTPUT VOICE INPUT VOICE Harmonic analysis (input spectrum) Amount Fig.. Block diagram o the proposed system. First a sample o the target voice quality must be prepared. A sustained vowel with rather stable undamental and consistent voice quality is preerred. The undamental requency can be precomputed and, i needed, manually corrected. To produce a signal o any duration, two loop points are manually set. Looping is done alternating directions and applying small random osets to the loop points to reduce perceived repetition. In order to match the undamental requency o the input voice, the looped sample is then resampled in the time-domain using a windowed sinc. Resampling is especially well-suited or our purposes because it does not require identiying the signal s spectral structure (harmonics, sub-harmonics, noise) while it preserves the excitation characteristics with high quality. The drawback is the inherent scaling o time and requency dimensions. For a given rame, we deine the resampling actor as, r = p v /p s, where p v and p s are the undamental requencies o the input and sample voices respectively. A value greater than means that the morph sample is played back aster than it was recorded, and thereore the duration gets shorter and requencies higher. In other words, s t = /r and s = r, where s t is the temporal scaling and s the requency scaling. Temporal scaling can be acceptable or certain voice qualities, or in general i the resampling actor is small enough. This can be minimized using several samples at dierent undamental requencies or each target voice quality. One appropriate scenario is an arpeggio o vowels, where notes are looped and interpolated. In addition, the arpeggio can also be good or dealing with pitch-dependent eects within a given voice quality. Frequency scaling aects both the requency o the harmon-

3 ics and the voice timbre, deined here as the harmonic spectral envelope. Since harmonics are shited in requency, the noise present in their requency bands is also shited, producing unnatural synthesis results. A solution is to invert the requency scaling by shiting in requency the harmonic bands. Depending on the resampling actor, this means dropping or repeating some harmonics requency bands. The mapping unction is deined as, i m i = r i = 0,,..., N () where i is the harmonic index and N the number o harmonics. Note that the mapped index is constrained to be a positive integer. The phase-locked vocoder [] is an appropriate technique here. One argument is that the resampled signal already matches the target undamental requency, and that thereore the requency and phase relation between harmonics and their surrounding sub-harmonics is correct. By contrast, i phase-locked vocoder was used or transposition, in order to maintain the correct requency relation between harmonics and sub-harmonics, they would have to be shited by dierent amounts. This implies having to estimate their parameters. This would be especially diicult when the macro period is not stable since then the spectrum o a single rame would contain dierent sub-harmonic structures. The synthesis spectrum Y or the bins k corresponding to the i th harmonic band becomes, Y [k] = Ŝ[k + d i]g i e jθi (2) d i = p v (m i i) L s () where Ŝ is the resampled morph sample spectrum, k is the spectral bin index, d i the requency shit (in bins) or the i th harmonic band, L the rame length, s the sampling rate, g i is a gain, and θ i a phase correction. The gain actor compensates or the timbre dierences between the input and the resampled sample voices. It is computed as, g i = A v [i]/a s [m i ], where A v and A s are the harmonic amplitudes or input and morph sample respectively. Harmonic amplitudes are computed using parabolic interpolation o the bins surrounding the main peak o the magnitude spectrum within each harmonic band. The phase correction θ i compensates or the dierences between harmonic phases o the morph sample and the input voice utterance. It is computed as, θ i = ϕ v [i]/ϕ s [m i ], where ϕ v and ϕ s the harmonic phases o input and morph sample respectively. Note that when the sample is played backwards in time, the sign o its harmonic phases has to be inverted. A potential problem arises when the morph sample is transposed to a higher undamental requency, i.e. r >. In this case, some o the high requency content o the spectrum may all above the Nyquist requency, thereore introducing some aliasing. I we apply a low-pass ilter to correct this, than we eliminate the high requency harmonics, and we cannot use them at synthesis (harmonic mapping). This can be quite noticeable, especially or high transposition actors. A better approach, although increasing computational cost, is to oversample the signal to a sampling rate high enough as to raise the Nyquist requency above the last harmonic requency band ater resampling. This implies a longer analysis window length, so more computations involved in computing the FFT. However, the remaining steps o the system are not aected at all. In our experiments, we initially computed harmonic parameters rom the resampled morph sample. We ound that oten the morph eect was reduced or low target undamental requencies. This is related to the change o the ratio between window length and undamental period. In our experiments the synthesis window length is constant (208 points at. khz). This is motivated by the intended integration o the morph eect into an existing singing synthesizer. Thereore the change o undamental requency during resampling modiies the relative window length respect to the undamental period. When the analysis window is longer than the macro period, sub-harmonics appear in the spectrum. However, when the morph sample is transposed down by resampling, the relative window length gets shorter and sub-harmonics disappear. In other words, we could say that instead o quasi-stable sinusoids plus sub-harmonics, the spectrum exhibits non-stationary (or modulated) sinusoids, and harmonic parameters vary signiicantly in consecutive rames. In such case, the harmonic gains g i actually remove these interrame amplitude variations, and at synthesis we obtain quasistable sinusoids. Nevertheless, the situation can be signiicantly and easily improved i we estimate harmonic parameters rom the original morph sample, beore resampling, ensuring we apply a window long enough so that sub-harmonics appear in the spectrum. This is illustrated in Figure 2. Note that harmonic parameters no longer match the actual spectrum o the resampled signal, but are the ones used to compute the gain and phase corrections, and to transorm the spectrum. Using this method the synthesized voice mostly preserves the modulations o the morph sample, and the degree o -type voice quality remains similar regardless o resampling actor. Once harmonic requencies, phases and amplitudes o the transormed morph sample and the input signal match, they can be reely mixed. We can vary between modal and voice qualities, or instance, by simply controlling a linear interpolation actor. While other mixing strategies, such as requencydependent mixing, are possible, this approach does have its limitation. For instance, we cannot directly control aspects such as jitter/shimmer amount or macro period duration.. EVALUATION In order to get an idea o the perormance o our system we conducted two MOS-type listening tests. In total 7 subjects participated, most having a background in music and signal processing. The irst test consisted o singing voice recordings morphed

4 RESAMPLING HARMONIC MAPPING & FILTERING H H H2 H H 2 2 spectrum o the morph sample H2 H H spectrum o the resampled morph sample input spectral envelope Fig. 2. An illustration o the process o resampling and harmonic mapping. Note that in the solid line o the middle plot sub-harmonics are no longer visible (window size L), while in the dashed line they are (window size L = L/r). with sustain samples. We combined recordings containing with 5 excerpts where was added synthetically by morph (in random order). We used input recordings rom dierent singers; 2 male and 2 emale, with pitches spanning about one octave. In all cases the morph sample and input voice were provided by dierent singers. We asked subjects to rate the overall quality and naturalness rom to 5, and whether they thought the excerpt was synthetic or a recording. The second test consisted o a single song generated using a singing voice synthesizer based on concatenation o diphone samples with modal voice quality. To increase naturalness o the synthesis, pitch and phonetic timing inormation was extracted rom a recording. We processed the output o the synthesizer to artiicially add voice quality in certain places. The morph amount (interpolation actor) control was manually set so that the usage o s was similar to the recording. Subjects would rate a version with, a version without and the target recording (with ), not only or quality, but also or expressiveness (simultaneously). The results o the listening tests are shown in igure. The recorded s were rated as slightly below good, while the s generated by morph were rated as above average. It is interesting to see the relatively high rate o conusion between real and synthetic s, shown in Table. Using voice quality morph to add expression to synthesized song resulted in an increase o perceived expressiveness, while perceived quality and naturalness decreases by a similar amount (but still well above average ). A number o sound examples that we have used in the listening tests can be ound online at: Overall quality & naturalness ( 5) 5 2 Transormed recording Real Overall quality & naturalness ( 5) 5 2 No etized song Rec No Fig.. Results o listening tests showing mean opinion scores (thick line), standard deviations above and under mean (box) and minimums and maximums (whiskers). Actual Rec Answer Real etic Not sure Real 66.7% 9.2%.7% etic.2% 5.2%.76% Table. Conusion matrix between recorded excerpts and excerpts with synthetic generated by morph.. CONCLUSIONS In this paper we have introduced a new method o generating -type voice quality in a given input signal based on spectral morphing. Unlike most existing methods, our method does not rely on a parametric model and does not require any parameter estimation o the -type signal beyond its harmonic parameters. While similar morph approaches oten require parallel recordings, our method instead uses a looped exemplar sample o the target voice quality. This sample is not restricted to exactly match the input phonetically or in pitch. Pitch-shiting is achieved by method based on resampling which is suitable or signals containing sub-harmonics, which is not typically the case with existing vocoders. While the scope o this paper was limited to -type voice qualities, another big advantage this morph-based method is that it is relatively widely applicable. The method shows promising results or other voice qualities such as tense and breathy. However, we have ound that or some voice quality transormations a corresponding change in spectral envelope is very signiicant. While our method only changes source excitation, or the class o -type voice qualities we have ound this to be suicient in order to produce convincing results in many cases. The listening tests seem to validate these indings. 5. ACKNOWLEDGMENTS This research has been unded by Yamaha Corporation. Expressiveness ( 5)

5 6. REFERENCES [] C. Gobl and A. Ní Chasaide, The role o voice quality in communicating emotion, mood and attitude, Speech Commun., vol. 0, no. -2, pp , 200. [2] K. Sakakibara, L. Fuks, H. Imagawa, and N. Tayama, Growl voice in ethnic and pop styles, in Proc. International Symposium on Musical Acoustics (ISMA), 200. [] H. Kawahara and M. Morise, Analysis and synthesis o strong vocal expressions: Extension and application o audio texture eatures to singing voice, in Proc. Acoustics, Speech, and Signal Processing (ICASSP), 202, pp [] J. Schoentgen, Stochastic models o jitter, J. Acoust. Soc. Am., vol. 09, pp , 200. [] D. Ruinskiy and Y. Lavner, Stochastic models o pitch jitter and amplitude shimmer or voice modiication, in Proc. IEEE 25th Convention o Electrical and Electronics Engineers in Israel (IEEEI), 2008, pp [5] A. Verma and A. Kumar, Introducing roughness in individuality transormation through jitter modeling and modiication, in Proc. Acoustics, Speech, and Signal Processing (ICASSP), [6] A. Loscos and J. Bonada, Emulating rough and voice in spectral domain, in Proc. 7th Int. Conerence on Digital Audio Eects (DAFX), 200. [7] P. Cano, A. Loscos, J. Bonada, M. de Boer, and X. Serra, Voice morphing system or impersonating in karaoke applications, in Proc International Computer Music Conerence (ICMC), [8] H. Banno, K. Takeda, K. Shikano, and F. Itakura, Speech morphing by independent interpolation o a spectral envelope and source excitation, Electronics and Communications in Japan (Part III: Fundamental Electronic Science), vol. 82, no., pp. 22 0, 999. [9] H. Kawahara and H. Matsui, Auditory morphing based on an elastic perceptual distance metric in an intererenceree time-requency representation, in Proc. Acoustics, Speech, and Signal Processing (ICASSP), 200. [0] T. Yonezawa, N. Suzuki, K. Mase, and K. Kogure, Gradually changing expression o singing voice based on morphing, in Proc. Interspeech, 2005, pp [] J. Laroche, Frequency-domain techniques or highquality voice modiication, in Proc. 6th Int. Conerence on Digital Audio Eects (DAFX), 200. [2] A. Roebel, S. Huber, X. Rodet, and G. Degottex, Analysis and modiication o excitation source characteristics or singing voice synthesis, in Proc. Acoustics, Speech, and Signal Processing (ICASSP), 202, pp [] H.-L. Lu and J. O. Smith, Glottal source modeling or singing voice synthesis, in Proc International Computer Music Conerence (ICMC), 2000, pp