Mandarin Singing-voice Synthesis Using an HNM Based Scheme *

Transcription

1 JOURNAL OF INFORMATION SCIENCE AND ENGINEERING 27, (2011) Madari Sigig-voice Sythesis Usig a HNM Based Scheme * Departmet of Computer Sciece ad Iformatio Egieerig Natioal Taiwa Uiversity of Sciece ad Techology Taipei, 106 Taiwa I this paper, HNM (harmoic plus oise model) is ehaced ad used to desig a scheme for sythesizig a Madari Chiese sigig voice. Ehacemets made iclude a Lagrage-iterpolatio based estimatio of spectral evelope, piecewise liear mappig of time axes, fixed-pace placemet of cotrol poits, ad other modificatios for aalyzig HNM parameters ad efficiet executio. I terms of the ehacemets ad the sigalsythesis equatios rewritte here, a Madari sigig-voice sythesis system is built. I the system, each Madari syllable is recorded just oce for aalyzig HNM parameters. The, the HNM parameters of a source syllable are used to sythesize sigig syllables of diverse pitches ad duratios. This system ca parse a sog score file ad sythesize its lyric syllables sigals i real-time. Also, the skill of portameto (pitch glidig) sigig is implemeted. Accordig to the perceptio tests, our system ca ideed sythesize sigals of sigig voice that are cosistet i timbre, of o reverberatio, ad much clearer tha a PSOLA (pitch sychroous overlap add) based scheme. Keywords: sigig-voice sythesis, harmoic-plus-oise model, spectral evelope, timbre cosistecy, reverberatio 1. INTRODUCTION Curretly, a specially desiged robot ca play a musical istrumet (e.g., piao or trumpet) as well as a huma ca play. It would be attractive if a humaoid robot (of facial expressio) could sig a sog as expressive as oe sug by a real siger. Toward this goal, we bega to study the sythesis of a sigig voice. Recetly, our sigig-voice sythesis system was itegrated ito a two-wheel robot [8]. Nevertheless, our sythesis system was just i its early phases because we had expeded more effort o sigal quality tha o expressiveess whe developig this system. Used here, sigal quality actually meas sigal clarity. That is, a sigal that is less oisy ad less reverberat is better i quality. We thik that the sythesis of a high sigal-quality sigig voice is a essetial basic step. Our reaso is that performace rules [17] ca be further used ad sigig expressios such as vibrato ad huskiess [12, 17, 22] ca be further added to the clea sythetic sigal. Several techiques have bee proposed to sythesize istrumetal music sigals, icludig additive sythesis, subtractive sythesis, FM (Frequecy Modulatio) sythesis, alog with others [3, 13]. As to the sythesis of a sigig voice, the techiques developed iclude phase vocoder [3, 13], format sythesis [3, 13], liear-predictio based source- Received Jauary 5, 2009; revised April 21 & Jue 22, 2009; accepted August 13, Commuicated by Chi-Teg Li. * This research was partially supported by Natioal Sciece Coucil of Taiwa uder Grat No. NSC E

2 304 filter model [3, 7, 13], siusoidal model [10], EpR (Excitatio plus Resoaces) model [1, 2], PSOLA sythesis [18], corpus-based sythesis [2, 9, 11], etc. I this paper, however, we choose to ehace HNM (harmoic plus oise model) i the hope of obtaiig higher sigal quality ad fluecy levels. Based o the ehaced HNM, we desig a sigal-sythesis scheme to implemet a Madari Chiese sigig-voice sythesis system. HNM is proposed by Y. Styliaou [20, 21] ad is thought to belog to the class of additive sythesis. It splits the spectrum of a sigal frame ito two halves of uequal widths i order to better model the spectrum. The lower frequecy half is modeled as cosistig of harmoic partials while the higher frequecy half is modeled as cosistig of oise sigal compoets. Although the Madari Chiese laguage is adopted here for studyig sigig-voice sythesis, the techique developed should be applicable to other laguages, especially to syllable-promiet laguages. Madari is a syllable-promiet laguage, ad has oly 408 differet syllables if the superimposed toes are ot distiguished. Each syllable of Madari is of the structure, C x VC. The iitial, C x, may be ull, a voiced cosoat, or a uvoiced cosoat while the fial, C, ca just be ull or a asal // or /g/. As to the ucleus, V, it may be a vowel, diphthog, or triphthog. Therefore, we take syllable as the uit for sythesis processig. I this paper, the HNM based scheme is depicted i Fig. 1. First, a ote s data is iput ad parsed. The, the comprisig phoemes of the ote s lyric syllable are plaed for their duratios. I terms of these duratios, a time-axis mappig fuctio ca be costructed. More detailed explaatio of this is give i sectio 3.1. Next, a pitch cotour for the ote is computed. If the lyric syllable is to be sug i portameto (pitch glidig), the computatio of the pitch cotour becomes more complicated. This will be explaied i sectio 3.6. I additio, to keep the timbres of sythetic syllables cosistet, the HNM parameters of a lyric syllable must be adjusted i a appropriate way. This will be explaied i sectios 3.2 ad 3.3. I the last block of Fig. 1, the sigals for the uvoiced ad voiced segmets of a lyric syllable are sythesized with a ehaced HNM. Start Iput ad parse a ote. ed of file? Y N Pla phoemes duratios. Costruct time mappig fuctio. Compute pitch-cotour with or without portameto. Determie pitch origial ad tued HNM parameters. score file Stop syllables HNM param. Sythesize uvoiced ad voiced sigals with HNM. sigals of sigig voice Fig. 1. Mai flow of the HNM based sythesis scheme.

3 MANDARIN SINGING-VOICE SYNTHESIS USING HNM 305 If the C x segmet of a syllable is a short uvoiced cosoat, e.g. /b/, /d/, its sythetic sigal will be copied directly from the correspodig segmet i the recorded syllable. Here, whe we write /b/, /d/, we refer to the uvoiced ad o-aspirated Madari versios of these phoemes. If the C x segmet is a log uvoiced cosoat, e.g. /s/, /p/, its sythetic sigal will be geerated as oise sigal with HNM. Otherwise, the C x is a voiced cosoat (e.g. /m/, /r/) ad will be cosidered together with the remaiig phoemes. As the remaiig phoemes of a syllable are all voiced, their sythetic sigal will be geerated as harmoic partials plus oise sigals with HNM. The methods for sythesizig harmoic ad oise sigals are explaied i sectios 3.4 ad 3.5, respectively. 2. PARSING OF SCORE FILE For our sigig voice sythesis system, a special file format is used to store a sog s score. I detail, each lie of the score file except the first lie cotais a musical ote s iformatio, i.e., pitch symbol, umber of beats, ad lyric syllable. The iformatio i the first lie cosists of sog ame, tempo (e.g., 120 meas 120 beats per miute), ad duty ratio (e.g., 85 meas 85% of a ote s duratio is used for sigig). Parsig is used to slice out the 3 fields i a lie ad to iterpret the meaigs of these fields. The pitch symbol of a ote is of the format XYZ (e.g. G3#). X deotes the toe ame, Y deotes the toe rage, ad Z is # (sharp) or b (flat). Through iterpretatio, the pitch symbol is coverted to a umeric value of pitch frequecy. After all otes pitch frequecies are determied, automatic key shiftig is performed. Key shiftig must be doe i order to traslate the pitch rage of the score file to match the pitch rage of the perso who utters the Madari syllables used as the sythesis uits. Thus, the key of the sythetic sog is ot the origial key defied i the score file. Here, automatic key shiftig is doe i the followig steps: (a) fid the maximum ad miimum values from the otes pitch frequecies; (b) take the average of the maximum ad miimum values foud; (c) compute the ratio of the perso s mea pitch frequecy to the average value computed i the previous step; (d) multiply each ote s pitch frequecy with the ratio computed. After the umber of beats for a ote is parsed, the tempo value i the first lie ca be used to compute the time legth of a ote. A ote, however, is usually ot sug i its full duratio because some small ratio of this duratio is reserved for breathig or trasitig to its followig ote. Next, the lyric syllable of a ote is parsed. Usually, each ote has a uique lyric syllable assiged to it. Sometimes, though, two or three cosecutive otes may be assiged the same lyric syllable. This meas that the syllable should be sug i portameto. This situatio is idicated i the score file with a covetio. Whe a ote is to be assiged the same lyric syllable as its precedig ote, the third field for this ote will cotai a special character such as. 3. SYNTHESIS OF SIGNAL WAVEFORM Note that each Madari Chiese syllable has oly oe recorded utterace here. Therefore, it becomes importat that the timbres of sythetic syllables are kept cosistet. The solutio for keepig timbre cosistet ca be foud i the literature [3, 19]. Accurate estimatio of spectral amplitude evelope is the key poit. Nevertheless, the amplitude

4 306 evelope estimatio method adopted i the origial HNM [20, 21] is a complicated global approximatio method that may ot be accurate eough. If the spectral evelope is estimated with sufficiet accuracy, the values of a syllable s HNM parameters ca the be adequately adjusted to obtai cosistet timbre whe the pitch frequecy of the syllable is chaged. I additio, ote that a syllable s duratio eeds to be legtheed or shorteed accordig the umber of beats assiged to its correspodig ote, ad liear time warpig usually results i lower perceived fluecy. Therefore, how to warp the time axis of a sythetic syllable i order that a more fluet syllable sigal ca be sythesized is also a problem. A solutio method to this problem, however, is ot foud i the origial HNM. Furthermore, direct copyig of HNM parameters from a aalysis frame to a cotrol poit [3, 13] is adopted i origial HNM. Here, i cotrast, we determie the HNM parameter values for a cotrol poit by iterpolatio. 3.1 Plaig of Phoeme Duratio Whe a syllable begis with a short uvoiced phoeme, e.g. /bau/, the time legth of the sythetic short-uvoiced phoeme is plaed as its source legth i the recorded syllable. I cotrast, whe started with a log uvoiced phoeme, the legth of the sythetic log-uvoiced phoeme is plaed by multiplyig its source legth with a factor Fu. Fu is computed as the sythetic syllable s legth divided by the recorded syllable s legth. Its value, however, is cofied to withi the rage from 0.6 to 1.2. The values, 0.6 ad 1.2, are determied empirically. They are used to reserve a miimum duratio to have the phoeme perceived, ad limit the maximum duratio to mimic a real siger s duratio plaig for a log uvoiced phoeme. For the plaig of voiced phoemes duratios, cosider the syllable, /ma/, for example. Suppose i the recorded sigal of /ma/, the three phoemes, /m/, /a/, ad //, occupy Rm, Ra, ad R ms, respectively, ad Rv = Rm + Ra + R. Also, suppose that Dm, Da, ad D represet the time legths of the three phoemes withi the sythetic syllable, ad Dv = Dm + Da + D. First, we let the iitial values of Dm ad D be 0.85 Dv (Rm/Rv) ad 0.85 Dv (R/Rv), respectively. Next, we pla the values of Dm, Da, ad D with a procedure [6] that iteratively decreases the values of Dm ad D, ad icreases the value of Da till the ratio, Da/Dv, is greater tha a defied value (e.g. 0.5). This procedure is desiged accordig to the observatio that the cosoat-to-vowel duratio ratio will become smaller whe the syllable is sug withi a sog tha uttered i isolatio. Note that i recordig the sigals of Madari syllables, each syllable is uttered i isolatio. Aother pheomeo created by recordig syllables i isolatio is that the duratios, Rm ad R, of a recorded syllable may differ largely but the sythetic duratios, Dm ad D, should be plaed to have about equal legths. After the values of Dm, Da, ad D are determied, a mappig fuctio from the phoemes i the sythetic syllable to the correspodig phoemes i the recorded syllable ca be established. This mappig fuctio is show by the solid lie depicted i Fig. 2, i.e. a piecewise liear time warpig fuctio. Accordig to the costructed mappig fuctio, a cotrol poit placed o the time-axis of a sythetic syllable ca the be mapped to locate its correspodig aalysis frames i the recorded syllable. For example, the dashed lies i Fig. 2 represet the mappigs from sythetic-phoeme boudaries to their correspodig source-phoeme boudaries.

5 MANDARIN SINGING-VOICE SYNTHESIS USING HNM 307 t recorded R Ra Rm Dm Da D sythetic t Fig. 2. A piecewise liear time mappig fuctio. I this paper, each source syllables is recorded at a samplig rate of 22,050Hz. I aalyzig HNM parameters, frame size is set to 512 sample poits (23.2ms) ad frame shift is set to 256 sample poits. I sythesizig a sigig voice, however, the cocept of cotrol poit [3, 13] is adopted. The term cotrol poit is used istead of frame because the HNM parameters for a cotrol poit withi the sythetic voiced segmet are obtaied by iterpolatig the parameters from its two correspodig aalysis frames, i.e. ot directly copyig parameters from a aalysis frame ito a cotrol poit (ote that origial HNM uses direct copyig). The details of the iterpolatio method are described i sectio 3.2. Here, i the sythetic voiced segmet, adjacet cotrol poits are always placed 100 sample poits (4.5ms) apart. A fixed pace, 100 sample poits, is adopted because more accurate cotrol of spectrum progressio is iteded. For example, cosider the situatio where a syllable of a diphthog kerel is to be sythesized with pitch glidig ad uder piecewise liear time mappig. As to the umber 100, it is selected by tradig off accurate spectral evelope with computatio burde. I cotrast, whe sythesizig the log-uvoiced segmet, the HNM parameters of a cotrol poit are obtaied by just copyig the HNM parameters from oe correspodig aalysis frame withi the recorded uvoiced segmet. 3.2 Pitch-origial HNM Parameters To determie the HNM parameter values for a cotrol poit withi the sythetic voiced segmet, the first step is to do time-positio mappig accordig to the costructed mappig fuctio as show i Fig. 2. Suppose a cotrol poit s time positio, ts, o the sythetic time axis is mapped to t r frames o the time axis of the recorded syllable. The, we use the HNM parameters aalyzed from the two frames umbered t r ad t r + 1 to iterpolate HNM parameters for the cotrol poit. Curretly, we do the iterpolatio i a liear way: + 1 i i i r r A = (1 w) A + w A, = t, w = t, (1) 1 Fi = (1 w) Fi + w F + i, (2) ˆ+ 1 θ = w ( θ θ ) + θ, (3) i i i i where A i, F i, ad θ i deote the amplitude, frequecy, ad istataeous phase of the ith

6 308 harmoic partial i the th aalysis frame, ad Ai, F i, ad θ i deote the amplitude, frequecy, ad istataeous phase of the ith harmoic partial for the cotrol poit. Note that, 1 i Eq. (3), θ ˆ + i represets the uwrapped phase of θ +1 i versus θ 1 i, i.e. θ ˆ + i = puw(θ +1 i, θ i, F +1 i, F i ). The phase θ +1 i is uwrapped i order that the phase differece is cotrolled to withi the rage from π to π. Here, the phase uwrappig is doe as: Fi + Fi 256 i i i i i π puw( θ, θ, F, F ) = θ 2 ( ) M 2 π, (4) 2 22, Fi + F 256 i M = θi 2 π ( ) θi + θc, 2π 2 22, Fi + Fi 256 π, if θi 2 π( ) θi θc = 2 22,050, π, otherwise where 256 is the frame shift i sample poits ad 22,050 is the samplig frequecy. Sice θ i ad θ i +1 are aalyzed from adjacet sigal frames, the extra accumulated phase must also be cosidered, which is estimated here as 256/22,050 2π(F i + F i +1 )/2. I origial HNM, the oise sigal compoets are represeted with 10 cepstrum coefficiets [15, 20]. Therefore, for each cotrol poit, 10 cepstrum coefficiets should be derived. Here, the cepstrum coefficiets from the two mapped aalysis frames are liearly iterpolated to derive the cepstrum coefficiets for the cotrol poit. 3.3 Pitch-tued HNM Parameters After the parameters Ai, F i, ad θ i for pitch-origial harmoic partials are computed, the parameters A k, F k ad θ k for pitch-tued harmoic partials should be computed o each cotrol poit withi the voiced segmet. Note that the pitch-height defied by F i, i = 1, 2,, is the origial pitch predetermied i recordig time. Thus, the pitch-height of a cotrol poit must be tued i order to follow the pitch height defied by the correspodig music ote. For example, let the pitch defied by the harmoic frequecies, F i, be 100Hz, ad we eed a pitch-height of 150Hz. Apparetly, a simple tuig method is to set the values of A k, F k ad θ k as F k = Fk 150/100, A k = Ak, ad θ k = θ k. This is illustrated i Fig. 3. From this figure, it ca be see that the pitch ca ideed be tued from 100Hz to 150Hz. The format frequecies, however, are also scaled up. For example, the first format is shifted from 240Hz o the solid-lie spectral curve to 360Hz o the dashed-lie spectral curve i Fig. 3. The shiftig of format frequecies will cause the timbre be distictly chaged. As a result, the timbre of a sythetic syllable will ot be cosistet ad will vary with the scalig factors (e.g. 150/100) set for differet cotrol poits. To have cosistet timbre, oe priciple is to keep the spectral evelope uchaged [3]. This implies that the amplitude A k of the pitch-tued harmoic partial located at frequecy F k (i.e., k times of the pitch frequecy to be tued) must be computed accordig to a estimated spectral evelope. Therefore, it is importat to accurately estimate the spectral-evelope curve. I the past, a few solutio methods for this problem have bee

7 MANDARIN SINGING-VOICE SYNTHESIS USING HNM 309 Amp 240Hz 360Hz Freq Fig. 3. Pitch tuig with spectral evelope scaled simultaeously. proposed [16, 19]. Here, cosiderig the two factors of efficiet computig ad sufficiet accuracy, we decide to estimate the spectral evelope by Lagrage iterpolatig the sequece of pairs, ( Fi, A i) i.e. a local approximatio method. I detail, for the kth harmoic frequecy F k, we first fid a pitch-origial harmoic frequecy F j, from F 1, F 2, F 3,, that is earest to ad less tha F k. The, the four pitch-origial harmoic partials of the frequecies, Fj 1, Fj, Fj+ 1, ad F j 2, + are used to perform order-three Lagrage iterpolatio [4] to compute the value of A k : j 2 j 2 Fk Fh A + + k = A m. m= j 1 h= j 1 Fm F (5) h h m As to the order of iterpolatio, the soud sythesized by order-two iterpolatio is perceived to be slightly less delicate. Therefore, order-three iterpolatio is chose. A illustratio of this method of pitch tuig without chagig spectral evelope is show i Fig. 4. I this figure, the pitch is scaled up by a factor of 1.25 but the timbre is preserved. Similarly, the phase θ k of the pitch-tued harmoic partial located at frequecy F k ca also be iterpolated with the four pitch-origial partials of frequecies, Fj 1, F j, F j 1, + ad F j+ 2. The phases of the four partials, θj 1, θj, θj+ 1, ad θ j 2, + however, must be uwrapped beforehad to prevet phase discotiuities. That is, the uwrapped phases, Amp Freq Fig. 4. Spectral evelope is kept while tuig pitch.

8 310 ˆ θ ˆ ˆ ˆ ˆ j 1 = θj 1, θj = puw( θj, θj 1, 0, 0), θj+ 1 = puw( θj+ 1, θj, 0, 0) ad ˆ θ ˆ j+ 2 = puw( θj+ 2, θj+ 1, 0, 0), are used istead i the iterpolatio processig. 3.4 Sythesis of Harmoic Sigal For the harmoic sigal, H(t), betwee the th ad ( + 1)th cotrol poits, its sample values are computed with these equatios (rewritte by us): L Ht () = ak() t cos( φk ()), t t= 0,1,,99, (6) k = 0 t + 1 ak() t = A k + ( A k A k ), 100 (7) φ () ( 1) 2 ()/22,050, (0) ˆ k t = φk t + π fk t φk = θk, (8) t + 1 fk () t = F k + ( F k F k ), 100 (9) where L is the umber of harmoic partials, 100 is the umber of samples betwee the th ad ( + 1)th cotrol poits, 22,050 is the samplig rate, a k (t) is the time-varyig amplitude of the kth partial at time t from the start of the th cotrol poit, φ k (t) is the cumulated phase for the kth partial, f k (t) is the time-varyig frequecy for the kth partial, ad θ = ˆ 1 puw( θk, θ k,0,0), i.e. uwrapped phase of ˆ 1 θ k versus ˆ θ k. I Eqs. (7) ad (9), liear iterpolatio is used, which seems sufficiet accordig to perceptio testig. Accordig to Eq. (9), the istataeous harmoic frequecy is iterpolated i a liear maer. Therefore, the cumulated phase at the boudary time poit, φ k (100), would ot be cotiuous to the iitial phase of the ext cotrol poit. These kids of discotiuities, i.e. φ k (100) φ +1 k (0), will iduce amplitude discotiuities to sigal waveform, ad cause clicks to be heard. To avoid these kids of discotiuities, a basic method has bee adopted i the literature [20, 21]. That is, the amout of mismatched phase, ξ k, at the boudary poit, t = 100, is computed beforehad. The, this amout is divided ad shared amog the 100 sample poits betwee two adjacet cotrol poits. Accordigly, the phases of the sigal samples (especially those aroud the boudary poit) will advace smoothly. Here, we compute the amout of mismatched phase as ξ k = puw(φ k (100), φ k +1 (0), 0, 0) φ k +1 (0) (10) where the phase uwrappig fuctio, puw(x, y, 0, 0), is as defied i Eq. (4). Accordig to our derivatio φ k (100) ca be directly computed as (100) π (0) ( + φ k = φk + F k + F k ). (11) 22, The formula of Eq. (11) is obtaied by recursively evaluatig Eqs. (8) ad (9). The, by dividig ad sharig ξ k with the samples betwee two adjacet cotrol poits, Eq. (6) is modified to: ˆ k

9 MANDARIN SINGING-VOICE SYNTHESIS USING HNM 311 L t H () t = ak() t cos φk () t ξk, t = 0,1,, (12) k = 0 Let L be the umber of harmoic partials o the th cotrol poit. The value of L is computed as dividig the MVF (maximum voiced frequecy) by the pitch frequecy, i.e. L = MVF( )/ F 1. I geeral, L may ot be equal to L +1. Hece, we set the value of L, i.e. the umber of partials, i Eqs. (6) ad (12) to the greater of L ad L +1. Suppose here that L is less tha L +1. The, the parameter values for the exteded partials o the th cotrol poit must be defied. Here, cosiderig the cotiuity of sigal-waveform, we 1 1 simply let A + + = 0, F = F, θ = θ, for k = 1 + L, 2 + L,, L +1. k k k k k 3.5 Sythesis of Noise Sigal For the oise sigal, we sythesize it as a summatio of siusoidal compoets accordig to the sigal model of HNM [20]. Let G k be the frequecy of the kth siusoid. As G k does ot chage with time, we defie G k = 100 k (Hz) accordig to the thesis of Styliaou [20]. For the th cotrol poit, however, the idex k of G k is ot started from 1 ad its startig value, K s, is determied accordig to the MVF of this cotrol poit, i.e. K s = MVF()/100. I cotrast, the ed value of the idex k is always a fixed value, K e = 11,025/100. The MVF value of a aalysis frame is determied durig HNM parameter aalysis [20]. Let B k be the oise amplitude for the kth siusoid o the th cotrol poit. To determie its value, the 10 cepstrum coefficiets, o the th cotrol poit, represetig the oise spectral evelope are first appeded with zero values ad iversely trasformed (iverse discrete Fourier trasform) to the spectral domai [15, 20]. The, expoetiatio is take to obtai the correspodig spectral magitude coefficiets, X j, j = 0, 1,, 2,047. Accordig to the magitudes X j, the value of B k ca be obtaied by liearly iterpolatig the two adjacet magitudes, X i ad X i+1, whose frequecies surroud the frequecy of G k. Whe the values of K s ad B k for the th cotrol poit are kow, the oise-sigal samples betwee the th ad ( + 1)th cotrol poits ca be computed with the equatios (rewritte here): K e Nt ( ) = bk ( t) cos( γk + t 2 π Gk/22,050 ), t= 0,1,, 99, (13) k= K s t + 1 bk () t = Bk + ( Bk Bk ), 100 (14) γ k = γ -1 k π G k /22,050, (15) where K s is set to the lesser of K s ad K s +1 ad where γ k is the iitial phase for the kth siusoid o the th cotrol poit. I Eq. (14), the time-varyig amplitude, b k (t), is liearly iterpolated. For the sythesis of the log uvoiced segmet i Fig. 1, Eqs. (13), (14) ad (15) ca still be used to geerate sigal samples. Nevertheless, the lower boud of the summatio idex, k, i Eq. (13) will ow be fixed to 1. This is equivalet to settig all the MVF values to the costat, 0Hz, for all the cotrol poits withi the uvoiced segmet.

10 Sythesis of Portameto Sigig Usually a lyric syllable is assiged oe musical ote. A syllable, however, may occasioally be assiged two (or three) otes. Whe a syllable is assiged more tha oe ote, it should be sug i portameto. That is, the pitch-cotour of the syllable should trasit smoothly from the former ote s pitch to the latter ote s pitch i the middle portio. A example pitch-cotour is show i Fig. 5. The duratio of the voiced segmet of a syllable is divided ito three time itervals of equal legths. The left ad right itervals are plaed to sig stable pitches of the two otes i order that they ca be explicitly perceived. The middle iterval is used to trasit the pitch smoothly. Freq (Hz) 290 C Time Fig. 5. Example pitch-cotour for a syllable sug i portameto. G3 I this paper, the pitch-cotour of a lyric syllable is plaed before its pitch-tued HNM parameters are calculated. Suppose that the two otes to be sug i portameto are of the pitch frequecies P a ad P b. We first divide the cotrol poits withi the voiced segmet of the syllable ito three groups. The, the cotrol poits withi the first ad third itervals are directly assiged the pitches of P a ad P b respectively. For the th cotrol poit i the secod iterval, however, its pitch, P, is defied with a cosie based fuctio: ( Pa + Pb) ( Pa Pb) P = + cos( ) 2 2 M π (16) where M is the umber of cotrol poits i the secod iterval. This cosie based trasitio fuctio is desiged accordig to our heuristic. It has a good property that the slopes at the left ad right eds are both zero. Hece, the trasitio part of the pitch cotour coects smoothly to the left ad right stable parts. 4.1 System Implemetatio 4. SYSTEM IMPLEMENTATION AND TESTING Madari Chiese has oly 408 differet syllables if the superimposed toes are ot distiguished [23]. Hece, we just record ad save each of these syllables oce for aalyzig their HNM parameters. Each of these syllables is uttered i isolatio ad i a level

11 MANDARIN SINGING-VOICE SYNTHESIS USING HNM 313 toe by a female i a soudproof room. The, a HNM aalysis program is developed to aalyze these syllables. The aalysis method is based o the oe proposed by Styliaou [20], but some modificatios are made. For example, the frequecy values of harmoic peaks i a spectrum are more precisely estimated with parabolic iterpolatio, ad the frequecy values of harmoic peaks are all saved for latter use i Eq. (2). I additio, a aalysis frame s MVF is more strictly defied as its followig four harmoic cadidates must all be checked to be ot harmoic peaks. If ay oe of its followig four harmoic cadidates is checked to be a harmoic peak, the curret MVF will ot be admitted. This MVF checkig rule is useful ad ca prevet detectig lower ad icorrect MVF values for such syllables with vowel /i/ as their ucleus. I developig the program for sythesizig a Madari sigig voice, the methods described i sectio 2 are used to parse a iput score file ad the methods described i sectio 3 are used to sythesize the sigal waveforms for the lyric syllables. Sice the umber of computatios is cosiderable, the sythesis program is difficult to ru i realtime o a ordiary persoal computer (e.g., a 2.6MHz Petium CPU based). Nevertheless, we ited to sythesize sigig voices ad play the sigal waveforms i real-time. This is because our sythesis program will be itegrated ito a humaoid robot to show the skill of sigig. Therefore, we tried to fid possible bottleecks. As a result, a major bottleeck is foud to be the FFT (fast Fourier trasform) [13, 15] operatio for trasformig cepstrum coefficiets back to the spectrum domai for determiig the oise sigals amplitudes. Whe the FFT legth is chaged from 4,096 to 1,024 poits, the sythesis speed is largely improved ad achieves 3 times real-time speed. That is, a sythetic sigig voice of 3 secods i legth eeds oly 1 secod of CPU time to sythesize. Here, the frequecy spacig of 21.53Hz (22,050/1,024) betwee two adjacet bis is thought to be sufficiet because the frequecies of adjacet siusoidal compoets are 100Hz apart. 4.2 System Testig Sigal Timbre ad Clarity To show the ability of the HNM-based sythesis scheme, spectrograms for the sigals of the syllable /wa/ are aalyzed with the package, WaveSurfer, ad show i the lower part of Fig. 6. The spectrogram at the left side of Fig. 6 is for the recorded syllable /wa/ while the spectrogram at the right side is for a sythetic syllable /wa/ sug i portameto. Whe the two spectrograms at the two sides are compared, it ca be foud that the format traces have same curve shape ad same frequecy height. This explais why they will have same timbre. Also, as see i the upper part of Fig. 6, the pitch height ad shape of the sythetic syllable are very differet from those of the recorded syllable. Nevertheless, the clarity ad aturaless of the sythetic sigig sigal are still kept i a high level. For demostratio, we have set up a web page [5]. From this web page, the sigal waveforms for the two syllables i Fig. 6 ca be heard ad compared. 4.3 System Testig Perceptio of Reverberatio ad Fluecy Usually, the udesired effect of reverberatio may be heard from sythetic audio sigals. Here, two Madari sog files, deoted as SA ad SB, were sythesized for testig the reverberatio effect. SA was sythesized with the HNM-based scheme studied here, ad SB was sythesized with our PSOLA-based scheme studied previously. The details

12 314 Fig. 6. Pitch-cotours ad spectrograms for the recorded ad sythetic syllables of /wa/. of PSOLA are referred to relevat literature [14, 18]. The two files, SA ad SB, ca also be dowloaded from a web page [5]. We ivited 12 persos to participate the perceptio tests. Each perso was allowed to listeig to SA ad SB a urestricted umber of times. The, he or she was asked to give a score about the reverberatio level of SB whe compared to SA. The score, 2 ( 2), is defied as SB (SA) is apparetly more reverberat tha SA (SB) while the score, 1 ( 1), is defied as SB (SA) is slightly more reverberat tha SA (SB). Otherwise, the score, 0, should be give if they caot be distiguished. After the perceptio tests, the averaged score was computed to be 1.25 ad its stadard deviatio was Therefore, the HNM-based scheme is better tha the PSOLA-based scheme i reducig the effect of reverberatio. About the reverberatio effect foud i PSOLA, we thik there are two possible causes. The first is that the pitch markers (or peaks) labeled for a syllable may ot be sychroized well with their correspodig glottal epochs ad may stagger aroud correct sychroizatio poits. We fid that the speech waveforms of some recorded syllables with /a/ kerel are especially difficult to label i regards to their pitch markers. The other cause is that whe the pitch of the sythetic syllable is tued to be cosiderably higher tha the pitch of the recorded syllable, three or more adjacet pitch periods of the recorded syllable will be overlapped ad added. This will also result i reverberatio. Furthermore, the PSOLA based scheme requires cosiderable labor to maually check ad correct the automatically labeled pitch markers. I cotrast, the HNM based scheme requires o such kid of labor. As to the issue of real-time executio, both schemes ca be executed i real-time but the PSOLA based scheme is much faster tha the HNM based scheme. I this study, however, sigal quality is the major cocer. Note that a piecewise-liear time mappig fuctio is proposed i sectio 3.1 to promote the fluecy level of a sythetic sog. Here, fluecy is defied as the lyric syllables of a music setece are heard as fluetly coected ad ot just cocateatio of idepedet syllables. To show the effectiveess of the proposed mappig fuctio, two more sog files, deoted as SC ad SD, were sythesized usig aother female s recorded syllables to aalyze HNM parameters. Here, SC was sythesized with a liear time mappig, ad SD

13 MANDARIN SINGING-VOICE SYNTHESIS USING HNM 315 was sythesized with a piecewise liear time mappig. The, the same 12 persos as metioed were ivited to compare the fluecy level of SD with SC. Each perso was allowed listeig to SC ad SD multiple times. The, he or she was asked to give a score cocerig the fluecy level of SD whe compared to SC. The score, 2 ( 2), is defied as SD (SC) is apparetly more fluet tha SC (SD) while the score, 1 ( 1), is defied as SD (SC) is slightly more fluet tha SC (SD). Otherwise, the score, 0, should be give. As a result, the averaged score was computed to be 1.08 ad its stadard deviatio was Therefore, the piecewise liear time mappig ca ideed sigificatly promote the fluecy level. 5. CONCLUDING REMARKS I this paper, a piecewise liear fuctio is proposed to map a cotrol poit o the sythetic time axis to two adjacet aalysis frames of a recorded syllable. Although such a time-axis mappig method may ot be the best, it ca promote the fluecy level of a sythetic sigig syllable sigificatly, accordig to the results of perceptio tests. Next, cotrol poits are placed i a fixed pace for sythesizig voiced segmet. This ca provide more accurate cotrol of spectrum progressio. As to the estimatio of the spectral amplitude evelope for a sigal frame, a order three Lagrage iterpolatio based method is proposed. This is because we must cosider computig efficiecy ad evelope accuracy simultaeously i order to sythesize sigig voice i real-time. Although Lagrage iterpolatio is simple ad may seem iaccurate, the timbres of the sythetic syllables are very close to their correspodig recorded syllables accordig to perceptio testig. I additio, we have added a MVF checkig rule to prevet detectig a erroeous MVF value from a sigal frame. I terms of the ehacemets ad the sigal-sythesis equatios rewritte here, we have built a Madari Chiese sigig voice sythesis system. I this system, each Madari syllable eeds oly oe recorded utterace, ad its aalyzed HNM parameters are used to sythesize sigig syllables of diverse duratios ad pitches. Also, by elimiatig the computatioal bottleeck i trasformig cepstrum coefficiets back to oise spectrum, the system ca ow ru smoothly i real-time. Furthermore, perceptio tests have bee coducted to compare the sigig-voice sigals sythesized respectively by the HNM ad PSOLA based schemes. The results show that the HNM based scheme proposed here ca ideed be used to sythesize a Madari sigig voice of cosistet timbre ad much higher sigal quality (much clear ad without reverberatio) tha our PSOLA based scheme studied previously. As to the sigig expressio factors, vibrato ad huskiess, we will study them i a further study ad itegrate them ito the sythesis scheme preseted here. REFERENCES 1. J. Boada ad A. Loscos, Sample-based sigig voice sythesizer by spectral cocateatio, i Proceedigs of the Stockholm Music Acoustics Coferece, 2003, pp J. Boada ad X. Serra, Sythesis of the sigig voice by performace samplig ad spectral models, IEEE Sigal Processig Magazie, Vol. 24, 2007, pp

14 C. Dodge ad T. A. Jerse, Computer Music: Sythesis, Compositio, ad Performace, Schirmer Books, New York, J. D. Faires ad R. Burde, Numerical Methods, Books/Cole Publishig Compay, Pacific Grove, CA, H. Y. Gu ad H. L. Liau, 6. H. Y. Gu ad Y. Z. Zhou, A HNM based scheme for sythesizig Madari syllable sigal, Iteratioal Joural of Computatioal Liguistics ad Chiese Laguage Processig, Vol. 13, 2008, pp Y. E. Kim, Sigig voice aalysis/sythesis, Ph.D. Thesis, Massachusetts Istitute of Techology, W. C. Lee, H. Y. Gu, K. L. Chug, et al., The realizatio of a music readig ad sigig two-wheeled robot, i Proceedigs of IEEE Workshop o Advaced Robotics ad its Social Impacts, 2007, pp C. Y. Li, T. Y. Li, ad J. S. R. Jag, A corpus-based sigig voice sythesis system for madari Chiese, i Proceedigs of the 13th ACM iteratioal Coferece o Multimedia, 2005, pp M. W. Maco, L. Jese-Lik, J. Oliverio, M. A. Clemets, ad E. B. George, A sigig voice sythesis system based o siusoidal modelig, i Proceedigs of Iteratioal Coferece o Acoustics, Speech, ad Sigal Processig, 1997, pp Y. Mero, High quality sigig sythesis usig the selectio-based sythesis scheme, Ph.D. Thesis, Departmet of Iformatio ad Commuicatio Egieerig, Uiversity of Tokyo, Y. Mero ad K. Hirose, Sythesis of vibrato sigig, i Proceedigs of IEEE Iteratioal Coferece o Acoustics, Speech, ad Sigal Processig, 2000, pp F. R. Moore, Elemets of Computer Music, Pretice-Hall, Eglewood Cliffs, NJ, E. Moulies ad F. Charpetier, Pitch-sychroous waveform processig techiques for text-to-speech sythesis usig diphoes, Speech Commuicatio, Vol. 9, 1990, pp D. O Shaughessy, Speech Commuicatios: Huma ad Machie, IEEE Press, Piscataway, NJ, A. Robel ad X. Rodet, Efficiet spectral evelope estimatio ad its applicatio to pitch shiftig ad evelope preservatio, i Proceedigs of Iteratioal Coferece o Digital Audio Effects, 2005, pp X. Rodet, Sythesis ad processig of the sigig voice, i Proceedigs of the 1st IEEE Beelux Workshop o Model Based Processig ad Codig of Audio, 2002, pp N. Schell, G. Peeters, S. Lemouto, P. Maoury, ad X. Rodet, Sythesizig a choir i real-time usig pitch sychroous overlap add, i Proceedigs of Iteratioal Computer Music Coferece, 2000, pp D. Schwarz ad X. Rodet, Spectral evelope estimatio ad represetatio for soud aalysis-sythesis, i Proceedigs of Iteratioal Computer Music Coferece, 1999, pp Y. Styliaou, Harmoic plus oise models for speech, combied with statistical methods, for speech ad speaker modificatio, Ph.D. Thesis, Ecole Natioale Supèrieure

15 MANDARIN SINGING-VOICE SYNTHESIS USING HNM 317 des Télécommuicatios, Paris, Frace, Y. Styliaou, Modelig speech based o harmoic plus oise models, Noliear Speech Modelig ad Applicatios, G. Chollet, et al., eds., 2005, pp F. Thibuult ad P. Depalle, Adaptive prpcessig of sigig voice timbre, i Proceedigs of Caadia Coferece o Electrical ad Computer Egieerig, 2004, pp Madari Romaizatio Table, Bureau of Cosular Affairs, Miistry of Foreig Affairs, R.O.C., Taiwa, Hug-Ya Gu ( ) received the B.S. ad M.S. degrees i Computer Egieerig from Natioal Chiao Tug Uiversity i 1983 ad 1985, respectively, ad the Ph.D. degree i Computer Sciece ad Iformatio Egieerig from Natioal Taiwa Uiversity i Curretly, he is a Associate Professor i the Departmet of Computer Sciece ad Iformatio Egieerig, Natioal Taiwa Uiversity of Sciece ad Techology, Taipei. Also, he is oe of the Sectio Editors of Iteratioal Joural of Computatioal Liguistics ad Chiese Laguage Processig. His research iterests iclude speech sigal processig, computer music sythesis, ad iformatio hidig. Huag-Liag Liao ( ) was bor i He received the B.S. degree i Iformatio Egieerig from Tatug Uiversity, Taipei, i 2003, ad the M.S. degree i Computer Sciece ad Iformatio Egieerig from Natioal Taiwa Uiversity of Sciece ad Techology, Taipei, i 2006.