Statistical Singing Voice Conversion based on Direct Waveform Modification with Global Variance

Transcription

1 INTERSPEECH 15 Statistical Singing Voice Conversion based on Direct Wavefor Modification with Global Variance Kazuhiro Kobayashi, Tooki Toda, Graha Neubig, Sakriani Sakti, Satoshi Nakaura Graduate School of Inforation Science, Nara Institute of Science and Technology NAIST, Japan {kazuhiro-k, tooki, neubig, ssakti, Abstract This paper presents techniques to iprove the quality of voices generated through statistical singing voice conversion with direct wavefor odification based on spectru differential DIFFSVC. The DIFFSVC ethod akes it possible to convert singing voice characteristics of a source singer into those of a target singer without using vocoder-based wavefor generation. However, quality of the converted singing voice still degrades copared to that of a natural singing voice due to various factors, such as the over-soothing of the converted spectral paraeter trajectory. To alleviate this over-soothing, we propose a technique to restore the global variance of the converted spectral paraeter trajectory within the fraework of the DIFFSVC ethod. We also propose another technique to specifically avoid over-soothing at unvoiced fraes. Results of subjective and objective evaluations deonstrate that the proposed techniques significantly iprove speech quality of the converted singing voice while preserving the conversion accuracy of singer identity copared to the conventional DIFFSVC. Index Ters: statistical singing voice conversion, direct wavefor odification, spectral differential, global variance, Gaussian ixture odel 1. Introduction A singing voice is one of the ost expressive coponents in usic. In addition to pitch, dynaics, and rhyth, the linguistic inforation of the lyrics can be used by singers to express ore varieties of expression than other usic instruents. Although singers can also expressively control their voice tibre to soe degree, they usually have a difficulty in changing it widely e.g. changing their own voice tibre into that of another specific singer owing to physical constraints in speech production. If singers could freely control their voice tibre beyond their physical constraints, it would open up entirely new ways for singers to express ore varieties of expression. Singing synthesis 1,, ] has been a growing interest in coputer-based usic technology. Entering notes and lyrics to the singing synthesis engine, users e.g., coposers and singers can easily produce a synthesized singing voice which has a specific singer s voice characteristics, different fro those of the users. Previous work has proposed techniques to flexibly control the synthesized singing voice as the users want by autoatically adjusting paraeters of the singing synthesis engine so that the variation of power and pitch in the synthesized singing voice is siilar to that of the given users natural singing voice 4, 5]. Although these technologies using singing synthesis engines are effective to produce the singing voices desired by the users, it is essentially difficult to produce synthesized singing voices by controlling all singing voice coponents including lyrics on the fly. Singing voice conversion SVC, on the other hand, converts a source singer s singing voice into another target singer s singing voice 6, 7]. This akes it possible to produce the desired singing voices on the fly, enabling singers to sing songs with their desired voice tibre, not liited by physical constraints. One of the typical ethods is based on statistical voice conversion VC techniques 8, 9]. A conversion odel is trained in advance using acoustic features, which are extracted fro a parallel data set of song pairs sung by the source and target singers. The trained conversion odel akes it possible to convert the acoustic features of the source singer s singing voice into those of the target singer s singing voice in any song while keeping the linguistic inforation of the lyrics unchanged. Recently eigenvoice conversion EVC techniques 1, 11] have also been successfully applied to SVC 1] to develop ore flexible SVC systes capable of achieving conversion between arbitrary source and target singers, even if a parallel data set is not available. However, speech quality of the singing voice converted by SVC is usually degraded copared to that of the natural singing voice due to various errors caused by not only the acoustic feature conversion process, but also the vocoding process for wavefor generation. To iprove speech quality of the converted singing voice, we have proposed an SVC ethod with direct wavefor odification based on spectru differential DIFFSVC 1]. DIFFSVC can avoid using the vocoder fraework in generation of the excitation signal by directly filtering an input singing voice wavefor with a tie sequence of spectral feature differentials estiated by a differential Gaussian ixture odel GMM derived fro the conventional GMM used in the standard SVC ethod. Although DIFFSVC is applicable to only situations in which pitch conversion is not necessary such as in intra gender conversion, voice tibre of the input singing voice can be successfully converted into that of the target singer while achieving speech quality significantly higher than the standard SVC ethod. The direct wavefor filtering tends to keep odulation coponents of the converted spectral paraeter trajectory larger copared to those in the standard SVC with the vocoder-based wavefor generation. However, they are still significantly saller than those of natural spectral paraeter trajectories as the converted spectral paraeter trajectory tends to be excessively soothed. This over-soothing effect is well known as a factor causing quality degradation in the synthesized sining voice. In this paper, to alleviate the over-soothing effect, we propose a paraeter generation algorith considering global variance GV for DIFFSVC. GV is a well-known feature to easure the over-soothing effect 9]. To restore the GV of the converted spectral paraeter trajectory, we odify the objective Copyright 15 ISCA 754 Septeber 6-1, 15, Dresden, Gerany

2 function to deterine a tie sequence of the spectral differential. Additionally, we ipleent a process soothing the spectral differential at unvoiced fraes to avoid the over-soothing effect at unvoiced sounds. We conduct subjective and objective evaluations, deonstrating that the proposed DIFFSVC ethod significantly iproves speech quality of the converted singing voice copared to the conventional DIFFSVC ethod.. Statistical singing voice conversion with direct wavefor odification DIFFSVC DIFFSVC consists of a training process and a conversion process. In the training process, a joint probability density function of spectral features of a source singer and the differential between the source and target singers is odeled with a differential GMM, which is directly derived fro a traditional GMM. As the spectral features of the source and target singers, we eploy D-diensional joint static and dynaic feature vectors X t =x t, x t ] of the source and Y t =y t, y t ] of the target consisting of D-diensional static feature vectors x t and y t and their dynaic feature vectors x t and y t at frae t, respectively, where denotes the transposition of the vector. As shown in 7], their joint probability density odeled by the GMM is given by P X t, Y t λ M = α N =1 Xt Y t ] ; µ X µ Y ], Σ XX Σ XY Σ YX Σ YY where N ; µ, Σ denotes a Gaussian distribution with a ean vector µ and a covariance atrix Σ. The ixture coponent index is. The total nuber of ixture coponents is M. λ is a GMM paraeter set consisting of the ixturecoponent weight α, the ean vector µ, and the covariance atrix Σ of the -th ixture coponent. The GMM is trained using joint vectors of X t and Y t in the parallel data set, which are autoatically aligned to each other by dynaic tie warping. Then, the differential GMM is analytically derived fro the trained GMM by transforing the paraeters. Let D t = ] d t, d t denote the static and dynaic differential feature vector, where d t = y t x t. The joint probability density function of the source and differential spectral features is shown as follows: P X t, D t λ M Xt ] µ X = α N ; D t =1 µ D = µ Y Σ XD Σ DD = Σ DX = Σ XX µ D ], Σ XX Σ DX Σ XD Σ DD ] ] 1 µ X = Σ XY Σ XX 4 + Σ YY Σ XY Σ YX. 5 In the conversion process, the converted spectral feature differential is estiated fro the source singer s spectral features based on the differential GMM in the sae anner as axiu likelihood estiation of speech paraeter trajectory with the GMM 9]. The voice tibre of the source singer is converted into that of the target singer by directly filtering the speech wavefor of the input natural singing voice with the converted spectral feature differential. Tie sequence vectors of the source features and the spectru feature differential are denoted as X =X 1,, X T ] and D =D 1,, D T ] where T is the nuber of fraes included in the tie sequence of the given source feature vectors. A tie sequence vector of the converted static features ˆd =ˆd 1,, ˆd T ] is deterined as follows: ˆd = argax P D X, λ s.t. D = Wd 6 d T M P D X, λ = P X t, λp D t, X t, λ 7 t=1 =1 where W is a transforation atrix to expand the static feature vector sequence into the joint static and dynaic feature vector sequence 14] and the probability density function at frae t is given by P D t, X t, λ = N E D,t V D = µ D = Σ DD + Σ DX Σ DX D t ; E D,t, V D Σ XX X t µ X 8 9 Σ XX Σ XD. 1. DIFFSVC considering global variance In order to iprove the speech quality of the converted singing voice in DIFFSVC, we propose two techniques: 1 restoration of the GV of the converted spectral feature trajectory and soothing of the converted spectral feature differential at unvoiced fraes. The GV of the target static feature vector over the tie sequence is written as vy =v1, v,,vd,,vd] 11 vd = 1 T y td ȳd T 1 ȳd = 1 T t=1 T y τ d 1 τ=1 where y td shows the d-th coponent of the target feature vector at frae t. The probability density function of the GV is odeled as follows: P vy λ v = N vy; µ v, Σ vv 14 where λ v is a paraeter set of a Gaussian distribution for which the ean vector and covariance atrix are µ v and Σ vv, respectively. The converted feature differential trajectory is deterined by axiizing a new objective function as follows: ˆd = argax P D X, λ ω P vy λ v s.t. D = Wd 15 d where y =x + d] and the constant ω denotes a paraeter for controlling the balance between the two likelihoods. The converted feature differential trajectory is iteratively updated by using the steepest descent ethod as follows: ˆd i+1 th = ˆd i th + α ˆd i th

3 where α is a step size paraeter. The gradient vector ˆd i th is given by d i th = L 17 d d=d i th L d = ω W V D Wd+ W V D E D ] + v 1, v,, v t,, v T 18 ] E D = E D 1,1,, ED t,t,, E D T,T 19 V D = diag V ] D 1,, V D t,, V D T v t = v t1, v t,,v td,,v td ] 1 v td = T pv d vy µ v y t d ȳ d where p v d indicates the d-th colun vector of the inverse atrix of Σ vv. An initial feature differential trajectory for the iterative update is deterined by filtering in the conventional DIFFSVC as follows: ˆd td = µ vd ŷtd ŷd + ŷd xtd vd where ŷ td indicates the converted feature at frae t deterined by the conventional DIFFSVC and ŷd indicates its average over a tie sequence. It has been reported that unvoiced consonants e.g. /s/, /sh/ are less affected by speaker individuality copared to voiced sounds e.g. /ae/, /n/ in noral speech 15]. Based on this finding, in order to alleviate the over-soothing effect as uch as possible, we iniize the aount of conversion at unvoiced fraes by soothing the converted feature differential at those fraes. We ipleent this process on top of the previously described DIFFSVC with GV by odifying E D,t and V D at unvoiced fraes as follows: E D,t = { for static & delta 4 { V D = V D for static for delta 5 where V D shows delta coponents of the inverse atrix of the covariance atrix in Eq. 8. These paraeter odifications ake the converted spectral feature differential soothly vary at unvoiced fraes. Note that we avoid updating the converted spectral feature differential at the unvoiced fraes in Eq Experiental evaluation 4.1. Experiental conditions We evaluated speech quality and singer identity of the converted singing voices to copare the conventional and proposed DIFFSVC ethods. We used singing voices of 1 Japanese traditional songs, which were divided into 15 phrases, where the Preference score %] a Conventional DIFFSVC Proposed DIFFSVC 95% confidence interval Figure 1: Results of preference test. a speech quality of converted singing voice, b conversion accuracy of singer individuality. duration of each phrase was approxiately 8 seconds. ales and feales sang these phrases. The sapling frequency was set to 16 khz. STRAIGHT 16] was used to extract spectral envelopes, which were paraeterized to the 1-4th el-cepstral coefficients as the spectral features. The frae shift was 5 s. The el log spectru approxiation MLSA filter 17] was used as the synthesis filter. We used 8 randoly selected phrases for the GMM training and the reaining 7 phrases were used for evaluation. The speaker-dependent GMMs were separately trained for individual singer pairs deterined in a round-robin fashion within intra-gender singers. The nuber of ixture coponents was 18. Two preference tests were conducted. The first test evaluated speech quality of the converted singing voices. The converted singing voice saples of the conventional and proposed DIFFSVC ethods for the sae phrase were presented to listeners in rando order. The listeners selected which saple had better sound quality. The second preference test evaluated the singer identity conversion accuracy. A natural singing voice saple of the target singer was presented to the listeners first as a reference. Then, the converted singing voice saples of the conventional and proposed DIFFSVC ethods for the sae phrase were presented in rando order. The listeners selected which saple was ore siilar to the reference natural singing voice in ters of singer identity. The nuber of listeners was 6 and each listener evaluated 54 saple pairs. They were allowed to replay each saple pair as any ties as necessary. 4.. Subjective evaluation Figure 1 a indicates the result of the preference test for the speech quality. The proposed DIFFSVC ethod generates the converted speech with better speech quality than the conventional DIFFSVC ethod. Figure 1 b indicates the result of the preference test for the singer identity. The conversion accuracy of the singer identity of the proposed DIFFSVC ethod is not significantly different fro that of the conventional DIFFSVC ethod. Although the proposed DIFFSVC ethod avoids accurately converting spectral features at unvoiced fraes, it still yields conversion accuracy of singer individuality alost equal to that of the conventional DIFFSVC ethod. These results deonstrate that the proposed DIFFSVC ethod is capable of converting voice tibre with higher speech quality while causing no degradation in the conversion accuracy of singer identity copared to the conventional DIFFSVC ethod. b 756

4 Log F Mel-cepstru coefficient Source 4 Target 1 w/o GV estiated 1 4 w/ GV 5estiated st diension th diension th diension st diension Frae index Frae index Figure : Exaple of trajectories of spectral feature sequences. Note that the duration of Target trajectories is different fro the other trajectories. 4.. Analysis of converted feature trajectories To ore deeply analyze what yields naturalness iproveents in the proposed DIFFSVC ethod, we exaine in detail the spectral feature trajectories of singing voices, which are given by Source el-cepstral coefficients extracted fro the source singer s natural singing voice Target el-cepstral coefficients extracted fro the target singer s natural singing voice w/ GV estiated el-cepstral coefficient differentials estiated with the proposed DIFFSVC ethod w/ GV filtered el-cepstral coefficients extracted fro the singing voice converted in the proposed DIFFSVC ethod w/o GV estiated el-cepstral coefficient differentials estiated with the conventional DIFFSVC w/o GV filtered el-cepstral coefficients extracted fro the singing voice converted in the conventional DIFFSVC ethod Figure shows the individual trajectories and the logarithic F trajectory. It can be observed fro Source and Target that higher-order el-cepstral coefficients tend to have rapidly varying fluctuations. It has been reported in 18] that these fluctuations are well odeled by the odulation spectru and strongly affect speech quality of the converted speech. In the proposed ethod w/ GV estiated, the converted feature differential trajectory is soothly connected fro the end of voiced segents to the start of voiced fraes thanks to the proposed soothing process at unvoiced fraes. This yields a converted feature trajectory w/ GV filtered aintaining natural fluctuations at unvoiced fraes. On the other hand, these fluctuations are obviously reduced in the conventional ethod w/o GV filtered. We can also see that the GV of the converted feature trajectory at higher-order el-cepstral coefficients is restored ore effectively by the proposed ethod w/ GV filtered copared to the conventional ethod w/o GV filtered. These results iply that the proposed ethod effectively approxiates the target spectral fluctuations by using those of the source spectral trajectory and the GV of the target spectral trajectory. Figure shows the GVs calculated fro several trajectories of el-cepstral coefficients. The GV in the conventional w/o GV filtered w/ GV filtered th diension th diension Global variance Target w/o GV filtered w/o GV estiated w/ GV filtered w/ GV estiated Order of el-cepstru Figure : GVs of several el-cepstral sequences. ethod w/o GV filtered significantly decreases copared to that of Target. On the other hand, the GV in the proposed ethod w/ GV filtered is close to that of Target. This GV restoration yields significant iproveents in speech quality of the converted singing voice. Note that the GV of the feature differential trajectories in the proposed ethod w/ GV estiated are still siilar to those of the conventional ethod w/o GV estiated. This shows the effectiveness of the proposed ethod odeling not the GV of the differential trajectory but the GV of the converted trajectory. 5. Conclusions In order to iprove quality of singing voice conversion based on direct wavefor odification DIFFSVC, we have proposed DIFFSVC considering global variance and soothing of the conversion function at unvoiced fraes. The experiental results have deonstrated that the proposed DIFFSVC ethod akes it possible to convert voice tibre of a source singer into that of a target singer with higher speech quality while not causing any adverse effects on the conversion accuracy of speaker identity copared to the conventional DIFFSVC ethod. In future work, we plan to apply the DIFFSVC fraework to crossgender conversion. 6. Acknowledgeents Part of this work was supported by JSPS KAKENHI Grant Nubers: 686 and 15H76, and by the JST OngaCREST project. 757

5 7. References 1] H. Kenochi and H. Ohshita, VOCALOID Coericial singing synthesizer based on saple concatenation, Proc. IN- TERSPEECH, pp , Aug. 7. ] K. Saino, M. Tachibana, and H. Kenochi, A singing style odeling syste for singing voice synthesizers. Proc. INTER- SPEECH, pp , Sept. 1. ] K. Oura, A. Mase, T. Yaada, S. Muto, Y. Nankaku, and K. Tokuda, Recent developent of the HMM-based singing voice synthesis syste - Sinsy, SSW7, pp , Sept. 1. 4] T. Nakano and M. Goto, VocaListener: A singing-to-singing synthesis syste based on iterative paraeter estiation, Proc. SMC 9, pp. 4 48, July 9. 5] T. Nakano and M. Goto, Vocalistener: A singing synthesis syste able to iic a user s singing in ters of voice tibre changes as well as pitch and dynaics, Proc. ICASSP, pp , May 11. 6] F. Villavicencio and J. Bonada, Applying voice conversion to concatenative singing-voice synthesis, Proc. INTERSPEECH, pp , Sept. 1. 7] Y. Kawakai, H. Banno, and F. Itakura, GMM voice conversion of singing voice using vocal tract area function, IEICE technical report. Speech Japanese edition, vol. 11, no. 97, pp , Nov. 1. 8] Y. Stylianou, O. Cappé, and E. Moulines, Continuous probabilistic transfor for voice conversion, IEEE Trans. SAP, vol. 6, no., pp , Mar ] T. Toda, A. W. Black, and K. Tokuda, Voice conversion based on axiu likelihood estiation of spectral paraeter trajectory, IEEE Trans. ASLP, vol. 15, no. 8, pp. 5, Nov. 7. 1] T. Toda, Y. Ohtani, and K. Shikano, One-to-any and any-toone voice conversion based on eigenvoices, Proc. ICASSP, pp , Apr ] Y. Ohtani, T. Toda, H. Saruwatari, and K. Shikano, Many-toany eigenvoice conversion with reference voice, Proc. INTER- SPEECH, pp , Sept. 9. 1] H. Doi, T. Toda, T. Nakano, M. Goto, and S. Nakaura, Singing voice conversion ethod based on any-to-any eigenvoice conversion and training data generation using a singing-to-singing synthesis syste, Proc. APSIPA ASC, Nov. 1. 1] K. Kobayashi, T. Toda, G. Neubig, S. Sakti, and S. Nakaura, Statistical singing voice conversion with direct wavefor odification based on the spectru differential, Proc. INTERSPEECH, pp , Sept ] K. Tokuda, T. Yoshiura, T. Masuko, T. Kobayashi, and T. Kitaura, Speech paraeter generation algoriths for HMM-based speech synthesis, Proc. ICASSP, pp , June. 15] M. Sabur, Selection of acoustic features for speaker identification, IEEE Trans. ASSP, vol., no., pp , ] H. Kawahara, I. Masuda-Katsuse, and A. Cheveigné, Restructuring speech representations using a pitch-adaptive tie-frequency soothing and an instantaneous-frequency-based f extraction: Possible role of a repetitive structure in sounds, Speech Counication, vol. 7, no. -4, pp , Apr ] S. Iai, K. Suita, and C. Furuichi, Mel log spectru approxiation lsa filter for speech synthesis, Electronics and Counications in Japan Part I: Counications, vol. 66, no., pp. 1 18, ] S. Takaichi, T. Toda, A. Black, and S. Nakaura, Modulation spectru-based post-filter for g-based voice conversion, AP- SIPA ASC, Dec