A Digital Signal Processor Implementation of Silent/Electrolaryngeal Speech Enhancement based on Real-Time Statistical Voice Conversion

Transcription

1 INTERSPEECH 03 A Digital Signal Processor Ipleentation of Silent/Electrolaryngeal Speech Enhanceent based on Real-Tie Statistical Voice Conversion Takuto Moriguchi, Tooki Toda, Motoaki Sano, Hiroshi Sato, Graha Neubig, Sakriani Sakti, Satoshi Nakaura Graduate School of Inforation Science, Nara Institute of Science and Technology, Japan Foster Electronic Copany, Liited, Japan {takuto-, tooki, neubig, ssakti, s-nakaura}@is.naist.jp, { sano, hrssato}@foster.co.jp Abstract In this paper, we present a digital signal processor (DSP) ipleentation of real-tie statistical voice conversion (VC) for silent speech enhanceent and electrolaryngeal speech enhanceent. As a silent speech interface, we focus on nonaudible urur (NAM), which can be used in situations where audible speech is not acceptable. Electrolaryngeal speech is one of the typical types of alaryngeal speech produced by an alternative speaking ethod for laryngectoees. However, the sound quality of NAM and electrolaryngeal speech suffers fro lack of naturalness. VC has proven to be one of the proising approaches to address this proble, and it has been successfully ipleented on devices with sufficient coputational resources. An ipleentation on devices that are highly portable but have liited coputational resources would greatly contribute to its practical use. In this paper we further ipleent real-tie VC on a DSP. To ipleent the two speech enhanceent systes based on real-tie VC, one fro NAM to a whispered voice and the other fro electrolaryngeal speech to a natural voice, we propose several ethods for reducing coputational cost while preserving conversion accuracy. We conduct experiental evaluations and show that real-tie VC is capable of running on a DSP with little degradation. Index Ters: statistical voice conversion, real-tie processing, reduction of coputational cost, DSP, non-audible urur, electrolaryngeal speech. Introduction Speech counication is one of the ost widely used ethods for huan counication and there is no question that it is a part of our everyday life. However, any barriers still exist in speech counication; e.g., we would have trouble speaking in quiet environents such as in a library as the sound would annoy others; and we ay lose the ability to produce a natural voice after undergoing surgery to reove speech organs. In order to break down these barriers, new technologies have been developed, such as silent speech interfaces for allowing people to speak while keeping silent,, 3, ] and speakingaid systes for enhancing unnatural and less intelligible speech produced by vocally handicapped people, 6, 7]. Non-Audible Murur (NAM) 8] has been proposed as one for of the silent speech interface. NAM is a very soft whispered voice, which is acoustically defined as articulated respiratory sounds without vocal-fold vibration conducted through the soft tissues of the head. It is directly detected fro the skin surface by attaching a NAM icrophone, which is one of the body-conductive icrophones, behind the ear. Although NAM can be produced while keeping silent, its sound quality and intelligibility are very low because of the very soft voice and body-conductive recording 9, 0]. To ake it possible to use NAM in speech counication, it is essential to ake it sound ore natural and intelligible. Electrolaryngeal (EL) speech is produced by an alternative speaking ethod for laryngectoees whose larynx has been reoved by laryngectoy, which is surgery to treat laryngeal cancer. To produce EL speech, the laryngectoee uses an external device called an electrolarynx to echanically generate excitation sounds. EL speech is quite intelligible but its sound quality is very unnatural owing to the echanical excitation sounds. Lack of naturalness in EL speech prevents the laryngectoee fro soothly counicating with others. Therefore, it is strongly desired to develop techniques to iprove quality of EL speech. To address these issues, speech enhanceent ethods based on statistical voice conversion (VC) techniques, ] have been proposed, e.g., silent speech enhanceent based on NAM-to-Whisper, which converts NAM into a whispered voice 0], and electrolaryngeal (EL) speech enhanceent based on EL-to-Speech, which converts EL speech into noral speech 3]. It has been reported that the trajectory-wise conversion processing ] is effective for iproving naturalness of NAM and EL speech. Moreover, towards the use of these enhanceent techniques in huan-to-huan counication, a low-delay conversion ethod approxiating the trajectorywise conversion processing with the frae-wise conversion processing has been proposed ]. Furtherore, a real-tie ipleentation of these enhanceent techniques has been proposed and successfully ipleented on devices with sufficient coputational resources ]. Towards the practical use of these enhanceent techniques, it would be useful to further ipleent the on devices that are highly portable (e.g., even with no network access) but have liited coputational resources. In this paper, we ipleent real-tie enhanceent systes based on VC, such as NAM-to-Whisper and EL-to-Speech, on a digital signal processor (DSP), a highly portable and copact device. Because the coputational resources of the DSP are very liited, we propose several ethods for reducing the coputational cost while preserving conversion accuracy. We experientally show that the proposed real-tie enhanceent systes have been successfully ipleented on the DSP with little degradation in conversion accuracy.. Speech enhanceent techniques based on statistical voice conversion.. NAM-to-Whisper and EL-to-Speech Figure shows the conversion process of NAM-to-Whisper and EL-to-Speech. In NAM-to-Whisper 0], the el-cepstral seg- Copyright 03 ISCA August 03, Lyon, France

2 ent features of NAM are converted into the el-cepstru of a whispered voice. Next, the converted whispered voice is synthesized by filtering white noise excitation signals with the converted el-cepstru. As a conversion odel for estiating the converted el-cepstru of a whispered voice fro the elcepstral segent of NAM, a Gaussian ixture odel (GMM) is used. A parallel data set consisting of NAM and a whispered voice uttered by the sae speaker is used to train the GMM. On the other hand, in EL-to-speech 3], the el-cepstral segent features of EL speech are converted into not only the el-cepstru of noral speech but also F 0 and aperiodic coponents 6] separately. Next, the converted noral speech is synthesized by filtering ixed excitation signals, which are generated by the converted F 0 and aperiodic coponents 7], with the converted el-cepstru. Therefore, three GMMs are used for estiating the three speech paraeters fro the el-cepstral segent of EL speech. To train these GMMs, a parallel data set consisting of EL speech uttered by a laryngectoee and noral speech uttered by a target non-disabled speaker is used... Training To allow conversion to function in real-tie, coputationally efficient spectral analysis based on the Fast Fourier Transfor (FFT) is used to extract the el-cepstru of the source speech ]. Given the el-cepstral feature vector x t at frae t, as the source features, a el-cepstral segent feature vector X t at frae t is extracted fro a joint vector created by concatenating several el-cepstral feature vectors fro frae t C to frae t + C as follows: X t = E x t C,, x t,, xt+c] + f, () where denotes transposition of the vector. The transforation atrix E and the bias vector f are deterined by principal coponent analysis. On the other hand, to extract target speech paraeters, such as el-cepstru, log-scaled F 0, and aperiodic coponents, high-quality speech analysis ethods, such as STRAIGHT 8] or el-generalized cepstral analysis 9], are used because quality of the target speech paraeters directly affects quality of the converted speech. Let us assue a feature vector of each target speech paraeter y t at frae t. As the target features, a joint static and dynaic feature vector Y t = ] y t, Δy t is created at each frae, where the dynaic feature vector Δy t is calculated as y t y t. The joint source and target feature vector ] X t, Y t is created at each frae by perforing tie alignent to the parallel data. The joint probability density of the source and target feature vectors is odeled with a GMM as follows: P (X t, Y t λ M = = (X,Y )) ( ] ) ; α N X t, Y t μ (X,Y ), Σ (X,Y ), () where N ( ; μ, Σ ) is a Gaussian distribution with ean vector μ and covariance atrix Σ. The paraeter set of the GMM λ (X,Y ) whose total nuber of ixture coponents is M is coposed of the ixture coponent weight α, the ean vector, and the covariance atrix Σ (X,Y ) of each ixture coponent. At the th ixture coponent, the ean vector and the covariance atrix Σ (X,Y ) are written as = μ (X) μ (Y ) ], Σ (X,Y ) = Σ (XX) Σ (XY ) Σ (YX) Σ (YY) ], (3) Figure : Conversion process of NAM-to-Whisper (upper figure) and EL-to-Speech (lower figure). where μ (X) and μ (Y ) features, respectively. Σ (XX) denote ean vectors of source and target Σ (YY) Σ (XY ), and Σ (YX) denote covariance or cross covariance atrices of the source and target features, respectively..3. Conversion Tie sequence vectors of the source and target features are denoted as X = ] X,, X T and Y = ] Y,, Y T, respectively. The tie sequence vector of the converted static features ŷ = ] ŷ,, ŷ T is calculated as ( ŷ = argax P Y X, λ (X,Y )) subject toy = Wy, () y where W is a transforation atrix that converts a tie sequence vector of the static features y into a tie sequence vector of the joint static and dynaic features Y 0]. To alleviate uffled speech caused by the over-soothing effect, global variance (GV) ] is further considered in conversion into elcepstru of the target speech. The conversion process given by Eq. () is not suitable for real-tie processing as it is a batch process using all fraes over an utterance. To achieve a real-tie conversion process, Eq. () is approxiated with a low-delay conversion process ]. First, the suboptial ixture coponent t at each frae is deterined as follows: ) ˆ t = argax P ( X t, λ (X,Y ) ( ) () = argax N X t ; μ (X), Σ (XX). Next, we deterine the converted static feature vector ŷ t L at frae t L by updating the segent vector of the converted static features ŷ t L,, ŷ t ] frae by frae using Kalan filtering without considering the GV. Finally, the converted static feature vector is enhanced frae by frae with a GV-based postfilter ]. 3073

3 3. Reduction of coputational cost 3.. Diagonalization of source covariance atrices As full covariance atrices are used in NAM-to-Whisper and EL-to-Speech, the coputational cost for the ixture coponent selection given by Eq. () is high. Although it still works in real-tie on devices with sufficient coputational resources, we have found that it does not work in real-tie on the DSP. Therefore, we ipleent diagonalization of the source covariance atrices proposed in ] to reduce the coputational cost. The source covariance atrices Σ (XX) are factorized into ixture-dependent diagonal covariance atrices Σ (XX),diag and a global full transforation atrix A as follows: Σ (XX) A Σ (XX),diag A. (6) This covariance structure akes the coputational cost equal to that necessary when using the diagonal covariance atrices because the global transforation atrix can be applied in advance to the transfors for the feature extraction shown in Eq. (). As reported in ], this approxiation ethod tends to cause only a sall degradation in conversion accuracy. 3.. Progra optiization To achieve real-tie processing on the DSP, we perfor several progra optiizations, such as pre-calculation of the twiddle factor of the FFT and the use of DSP-specific functions. These optiizations cause no adverse effect on conversion accuracy. To further reduce the coputational cost, soe operations are approxiated; e.g., exponential and logarithic functions are approxiated with piecewise linear functions; and high-order cepstral coefficients are approxiated with zero values to reduce the coputational cost for transforing cepstral coefficients to el-cepstral coefficients using a first-order all-pass filter. These approxiations ay cause adverse effects on conversion accuracy Increase of frae shift In real-tie conversion processing, all procedures at each frae, such as feature extraction, conversion, and synthesis, should be finished within the frae shift. Although the frae shift is set to s in the traditional conversion systes 0, 3, ], we change it to 0 s. Because feature extraction and conversion are perfored only once at each frae, coputational costs for these procedures are not changed according to the frae shift. Consequently, the real-tie factor calculated as a rate of the processing tie divided by the frae shift is reduced by half for these procedures. On the other hand, the coputational cost for synthesis increases as 0 s of the wavefor signal needs to be generated at each frae. Moreover, we change a few paraeters, such as the nuber of fraes to extract the el-cepstral segent (related to C in Eq. ()) and the nuber of delay fraes L in the low-delay conversion described in Section.3, to keep latency in conversion alost the sae as in s frae shift. Table shows the latency difference caused by changing frae shift fro s to 0 s. The frae shift change ay cause adverse effects on conversion accuracy due to the decrease of tie resolution. 3.. Siplification of the ixed excitation odel The STRAIGHT ixed excitation odel 6] is used in traditional EL-to-Speech systes 3]. It is effective for iproving Table : Latency difference caused by increasing frae shift. Frae shift C L C + L Latency s] s s 0 Figure : Two-band ixed excitation odel. quality of converted speech but it needs to estiate aperiodic coponents and calculate a weighted su of a pulse train and white noise excitation with frequency-dependent weighting values deterined fro the estiated aperiodic coponents 7]. To reduce the coputational cost for generating the excitation signals, a two-band ixed excitation odel used in Haronic plus Noise Model (HNM) ] is ipleented as a sipler ixed excitation odel. As shown in Fig., this odel generates the excitation signal using a pulse train in low-frequency bands and white noise in high-frequency bands. The axiu voiced frequency is used to define the boundary between these two frequency bands. Although this value norally changes frae by frae, we approxiate it with a fixed value ( khz in this paper) to further reduce the coputational cost. Tieinvariant low/high-pass filters designed by a low-order Butterwoth filter are used to efficiently generate the ixed excitation signals.. Experiental evaluations.. Experiental conditions We conducted experiental evaluations of our proposed ipleentation in NAM-to-Whisper and EL-to-Speech. We used TMS30C678 (37 MHz) as a floating point DSP. NAM-to-Whisper: Each of two ale speakers and one feale speaker recorded 0 newspaper sentences in NAM with NAM icrophone and in a whispered voice with an airconductive icrophone. The sapling frequency was 6 khz. For each speaker, 70 newspaper sentences were used for training and the other 70 newspaper sentences were used for test. The 0 th through th el-cepstral coefficients were used as spectral features. The speaker-dependent GMMs for elcepstral estiation were trained for the individual speakers. The nuber of ixture coponents of each GMM was set to 3. The real-tie factor and spectral conversion accuracy with elcepstral distortion in the following systes were evaluated: Offline based on a batch-type process 0], Baseline based on the conventional real-tie ipleentation without the diagonalization ], Diag ipleenting the diagonalization for Baseline, Fast ipleenting progra optiizations for Baseline, 0s changing frae shift fro s to 0 s in Baseline, Diag+Fast cobining Diag and Fast, Diag+Fast+0s cobining Diag, Fast, and 0s. EL-to-Speech: One laryngectoee recorded 0 phonetically balanced sentences in EL speech and another non-disabled speaker recorded the sae sentences in noral voices. The 307

4 Real-tie factor Real-tie factor Synthesis Conversion Feature extraction Baseline Diag Diag+Fast Diag+Fast +0s Conversion systes for NAM-to-Whisper Synthesis Conversion Feature extraction Baseline Diag Diag+Fast Diag+Fast +0s Conversion systes for EL-to-Speech Figure 3: Real-tie factor in each syste of NAM-to-Whisper (upper figure) and EL-to-Speech (lower figure). sapling frequency was 6 khz. Forty sentences were used for training and the other 0 sentences were used for test. The 0 th through th el-cepstral coefficients were used as spectral features and log-scaled F 0 and aperiodic coponents were used as excitation features. The nubers of ixture coponents of the GMMs were set to 3 for the spectral estiation, 6 for the F 0 estiation, and 6 for the aperiodic estiation. The realtie factor and spectral conversion accuracy with el-cepstral distortion in Offline, Baseline, Diag, Fast, 0s, Diag+Fast, and Diag+Fast+0s were evaluated. In Fast, the siplification of the ixed excitation odel as well as the progra optiizations was also ipleented for Baseline. We also conducted an opinion test on naturalness using a -scaled opinion score (: very bad to : excellent) as a subjective evaluation. Twelve listeners evaluated naturalness of original EL speech (EL) and three types of converted speech by Offline, Baseline, and DSP that is an actual DSP conversion syste based on Diag+Fast+0s... Experiental results Figure 3 shows the real-tie factor of feature extraction, conversion, and synthesis processing in each syste for NAM-to- Whisper and EL-to-Speech. The diagonalization (Diag) significantly reduces the real-tie factor in conversion. The progra optiization (Diag+Fast) significantly reduces the realtie factor in feature extraction. In EL-to-Speech, it also significantly reduces the real-tie factor in synthesis thanks to the siplified ixed excitation odel. The real-tie factor in total processing can be successfully reduced below by further changing the frae shift fro s to 0 s (Diag+Fast+0s). Therefore, Diag+Fast+0s is capable of running on the DSP in real-tie. Figure shows spectral conversion accuracy using the elcepstral distortion (MCD) in each syste for NAM-to-Whisper Mel-cepstral distortion db] Mel-cepstral distortion db] Offline Baseline Fast Diag 0s Diag+Fast +0s Conversion systes for NAM-to-Whisper Offline Baseline Fast Diag 0s Diag+Fast +0s Conversion systes for EL-to-Speech Figure : Mel-cepstral distortion in each syste of NAM-to- Whisper (upper figure) and EL-to-Speech (lower figure). Mean Opinion Score (MOS) 3 9% confidence interval EL Offline Baseline DSP Conversion systes for EL-to-Speech Figure : Result of opinion test on naturalness in EL-to-Speech. and EL-to-Speech. MCD before conversion is 9.8 db in NAM and 9. db in EL speech. In NAM-to-Whisper, the online conversion syste (Baseline) does not cause any significant degradation copared with the offline conversion syste (Offline). Although Fast causes no degradation, Diag, 0s, and Diag+Fast+0s cause around a db increase in MCD. On the other hand, in EL-to-Speech, Baseline causes nearly a 0. db increase in MCD copared with Offline. Diag and Diag+Fast+0s further cause around a db increase in MCD. We can see fro these results that the real-tie conversion systes (Diag+Fast+0s) can be ipleented on the DSP while keeping the increase of MCD less than around 0. db and conversion accuracy of the ipleented DSP syste is still sufficient. Figure shows a result of the opinion test on naturalness in EL-to-Speech. Although DSP causes slight degradation in naturalness copared with Offline and Baseline, it is sall enough to be insignificant in practice. DSP still yields significant iproveents in naturalness of EL speech (EL). We also confired that DSP does not cause any significant degradation in naturalness copared with Offline in NAM-to-Whisper as well although we don t show any results due to space liitation.. Conclusions In this paper, we have proposed several ethods for reducing the coputational cost of speech enhanceent processing based on real-tie statistical voice conversion (VC) and have successfully ipleented real-tie VC systes for enhancing non-audible urur (NAM) and electrolaryngeal (EL) speech on a Digital Signal Processor (DSP). The experiental results deonstrate that the DSP systes have the capability to significantly enhance NAM and EL speech and run in real-tie. Acknowledeent: Part of this work was supported by JSPS KAKENHI Grant Nuber

5 6. References ] B. Denby, T. Schultz, K. Honda, T. Hueber, J.M. Gilbert, and J.S. Bruberg, Silent speech interfaces, Speech Counication, Vol., No., pp , 00. ] S.-C. Jou, T. Schultz, and A. Waibel, Adaptation for soft whisper recognition using a throat icrophone, Proc. INTERSPEECH, pp , Jeju Island, Korea, Sep ] T. Schultz and M. Wand, Modeling coarticulation in EMG-based continuous speech recognition, Speech Counication, Vol., No., pp. 3 33, 00. ] T. Hueber, E.-L. Benaroya, G. Chollet, B. Denby, G. Dreyfus, and M. Stone, Developent of a silent speech interface driven by ultrasound and optical iages of the tongue and lips, Speech Counication, Vol., No., pp , 00. ] H.R. Sharifzadeh, I.V. McLoughlin, and F. Ahadi, Reconstruction of noral sounding speech for laryngectoy patients through a odified CELP codec, IEEE Trans. Bioedical Engineering, Vol. 7, No. 0, pp. 8 8, 00. 6] H. Liu, Q. Zhao, M.-X. Wan, and S.-P. Wang, Enhanceent of electrolarynx speech based on auditory asking, IEEE Trans. Bioedical Enginering, Vol. 3, No., pp , ] A.Hisada and H.Sawada, Real-tie clarification of esophageal speech using a cob filter, Proc. ICDVRAT, pp. 39 6, Sep ] Y. Nakajia, H. Kashioka, N. Capbell, K. Shikano, Non- Audible Murur (NAM) recognition, IEICE Trans. Inforation and Systes, Vol. J87-D-II, No. 9, pp , 00. 9] T. Hirahara, M. Otani, S. Shiizu, T. Toda, K. Nakaura, Y. Nakajia, and K. Shikano, Silent-speech enhanceent using body-conducted vocal-tract resonance signals, Speech Counication, Vol., No., pp , 00. 0] T. Toda, M. Nakagiri, K. Shikano, Statistical voice conversion techniques for body-conducted unvoiced speech enhanceent, IEEE Trans. Audio, Speech, and Language Processing, Vol. 0, No. 9, pp. 0 7, 0. ] Y. Stylianou, O. Cappe, and E. Moulines, Continuous probabilistic transfor for voice conversion, IEEE Trans. Speech and Audio Processing, Vol. 6, No., pp. 3, 998. ] T. Toda, A. W. Black, and K. Tokuda, Voice conversion based on axiu likelihood estiation of spectral paraeter trajectory, IEEE Trans. Audio, Speech, and Language Processing, Vol., No. 8, pp. 3, ] K. Nakaura, T. Toda, H. Saruwatari, and K. Shikano, Speakingaid systes using GMM-based voice conversion for electrolaryngeal speech, Speech Counication, Vol., No., pp. 3 6, 0. ] T. Muraatsu, Y. Ohtani, T. Toda, H. Saruwatari, and K. Shikano, Low-delay voice conversion based on axiu likelihood estiation of spectral paraeter trajectory, Proc. INTERSPEECH, pp , Brisbane, Australia, Sep ] T. Toda, T. Muraatsu, and H. Banno, Ipleentation of coputationally efficient real-tie voice conversion, Proc. INTER- SPEECH, Portland, USA, Sep. 0. 6] H. Kawahara, J. Estill, and O. Fujiura, Aperiodicity extraction and control using ixed ode excitation and group delay anipulation for a high quality speech analysis, odification and synthesis syste STRAIGHT, Proc. MAVEBA, Firenze, Italy, Sep ] Y. Ohtani, T. Toda, H. Saruwatari, and K. Shikano, Maxiu likelihood voice conversion based on GMM with STRAIGHT ixed excitation, Proc. INTERSPEECH, pp , Pittsburgh, USA, Sep ] H. Kawahara, I. Masuda-Katsue, and A. de Cheveigne, Restructuring speech representations using a pitch-adaptive tiefrequency soothing and an instantaneous-frequency-based F 0 extraction: possible role of a receptive structure sounds, Speech Counication, Vol. 7, No. 3, pp , ] K. Tokuda, T. Kobayashi, T. Masuko, and S. Iai, Melgeneralized cepstral analysis - a unified approach to speech spectral estiation, Proc. ICSLP, pp. 03 0, Yokohaa, Japan, Sep ] K. Tokuda, T. Yoshiura, T. Masuko, T. Kobayashi, and T. Kitaura, Speech paraeter generation algoriths for HMM-based speech synthesis, Proc. ICASSP, pp.3 38, Jun ] Y. Stylianou, Applying the Haronic Plus Noise Model in Concatenative Speech Synthesis, IEEE Trans. Speech and Audio Processing, Vol. 9, No., pp 9, Jan