Voice Conversion Application (VOCAL)

Transcription

1 0 Intenational Confeence on Uncetainty Reasoning and Knowledge Engineeing Voice Convesion Application (VOCAL) Liliana, Resmana Lim, Elizabeth Kwan Infomatics Depatment, Industial Technology Faculty Peta Chistian Univesity Suabaya, Indonesia Abstact Recently, a lot of woks has been done in speech technology. Text-to-Speech and Automatic Speech Recognition have been the pioities in eseach effots to impove the human-machine inteaction. The ways to impove natualness in human-machine inteaction is becoming an inpotant matte of concen. Voice convesion can be seved as a useful tools to povide new insights elated to pesonification of speech enabled systems. In this eseach, thee ae two main paametes ae consideed vocal tact stuctue and pitch. Fo convesion pocess speech is esolved in two components, excitation component and filteed component using Linea Pedictive Coding (LPC). Ptich is detemined by autocoelation. Afte obtained the acoustic components fom souce speake and taget speake, then the acoustic components will be mapped one-to-one to eplaced the the acoustic featue fom souce speake to taget speake. At least, signal is modified by esynthesis so the esulted speech would peceive as if spoken by taget speake. Keywods-Speech Pocessing, Audio Signal Pocessing, Voice Convesion, Linea Pedictive Coding, Autocoelation, PSOLA. I. INTRODUCTION The last few yeas thee has been a apid development in speech technology. The study was conducted to impove the quality of human-compute inteaction. These studies have poven binging benefits to the wide community, such as technology, Text-To-Speech (TTS), which continues to be efined, speech-to-text, fo the speech ecognition. Using TTS fo example, will help the blind o illiteate to undestand the meaning of a text on the system. Cuently, speech ecognition and TTS is top pioity eseach in the field of speech technology that aimed at impoving human-compute inteaction. Impoving the natual communication in human-compute inteaction has become an impotant issue to be addessed. Voice convesion technology is expected to answe these poblems, due to the ability associated with the pesonification of the speech system povided by voice convesion. Voice convesion is a method that aims to modify someone's speech (souce speake) so as if spoken by anothe peson (taget speake). Geneal basic famewok of voice convesion can be seen in figue. Figue. Geneal Famewok fo Voice Convesion Souce: Tuk (003, p. ) The system gets input fom the taget speake and the souce speake. Fom the input, the system can take some voice chaacteistics such as pitch, vocal tact paametes, and speed, which then though a pocess of tansfomation will be conveted into outputs (tansfomed speech). The ability to change chaacteistics of a speake's voice can be applied in many fields, fo example, in the field of voiceove (dubbing). Moeove, it can also be used to poduce egeneated voice of the actess / acto who had died o who had lost his beautiful o young voice because of age. The othe application, it can be used to poduce sound actess / acto in cetain othe languages which ae not spoken by actess / acto. In addition to the field of voiceove, voice convesion technology can also be used to develop speech pocessing technologies, such as Text-To- Speech (TTS), eades, Inteactive Voice Response (IVR), and othes. Geneally, a TTS system can only geneate the sound of seveal speakes. This is due to the high costs equied to enich the database of the voice on the TTS system and the time needed to ecod and pocess the sound fom each speake. Voice convesion can be used to geneate a new voice fo TTS system without the need to conduct a long pocess. Seveal studies have been developed to ovecome the poblems of using voice convesion in TTS, some of the eseache ae Bob Dunn (003), A. Cain and M. Macon (998a), A. Cain and M. Macon (998b), as well as C.M. Ribeio and I.M. Tancoso (997). The development of voice convesion technology is also expected to assist the development of secuity and foensic applications that use speech signal fo the identification and authentication pocedues. In this study, the appoach with DSP (Digital Signal Pocessing) is possible because each peson has a unique vocal quality due

2 to the stuctue of vocal tact and pitch ange owned. Vocal tact and pitch is called static speech paametes, the paametes in which the speake has limited contol (Mecwan, et al., 009). II. VOICE CONVERSION Thee ae many methods that can be used in voice convesion. This aises the difficulty to obtain the optimal method fo all possible combinations of chaacteistics and speakes (Tuk, 003). Diffeent voice convesion system can use diffeent methods, but at least most of the system consists of seveal components such as methods to epesent the specific chaacteistics of the speakes, method to map the chaacteistics of the souce and taget, methods to tansfom the souce chaacteistics of speech. A. Digitization of Sound Sound is wave include the macoscopic and involve ai molecules ae pessed and expanded unde the action of some physical equipment. Fo example, in an audio speakes vibating system, wave vibates back and foth and poduce longitudinal pessue which we accept as sound (Li & Dew, 004). Sound has a continuous value, in contast to digital sound which has a limited ange. So, wanting to use digital vesions of sound waves, it takes the fom of a digital epesentation of audio infomation. Resampling is a pocess to change the sampling ate of the signal which aims to impove the efficiency of vaious signal pocessing opeations. Down-sampling is intended to educe the sampling ate, while up-sampling is intended to incease the sampling ate (Toivonen, 009). The algoithm used to impove the pecision of the esampling method is sine intepolation (Boesma, 00). This algoithm assumes that the signal is the sum of sine functions. Because of the limitation of the depth intepolation, the sum will be multiplied by Hanning window. Hanning window can be calculated using equation until 6 s l floo( s); () s s + () l ϕ s (3) l s l ϕ ϕ l (4) y( s) y ( ( ))( ( ( ) ( ))) = i N i sin c π ϕl i / / cos π ϕl i / ϕl N (5)... + yl i sin c( π ( ϕ + i ) )( / + / cos( π ( ϕ + i ) /( ϕ + N ))) i=... N + whee sine function is defined as : sin c 0 ;sin c x sin x / x fo x (6) ( ) ( ) 0 B. Deconvolution Speech is a convolution between the souce x(t) with input to the filte esponse h(t). To analyze the signals so that obtained chaacteistics of the output y(t), it equies a pocess to sepaate the signals so that they can be analyzed individually. Sepaation pocess is called deconvolution (see Figue ). In this study, we used Linea Pedictive Coding (LPC) to do the deconvolution pocess. Figue. Analog Signal Souce : Li & Dew (004, p. 7) Figue shows the natue of one-dimensional sound. Change the value fom time to time in the amplitude, the pessue inceases o deceases ove time. Fully digitize the signal samples needed in each dimension in time and in amplitude, sampling and quantization espectively. Sampling means measuing the quantity, usually at a unifom distance. Sampling is done on the time basis. Sampling ate means the numbe of sample taken at a distinct time. Fo audio, the sampling ates ange fom 8 khz (8000 samples pe second) to 48 khz. The sample taking based on the amplitude o voltage dimension efeed to quantization. Quantization ates geneally ange between 8-bit to 6-bit, 8-bit quantization divides the vetical axis into 56 levels, and 6-bit divide it into 65,536 levels. Figue II. Deconvolution System to Sepaate a Complex Signal Souce : Uppeman (008, p.5) LPC method is chosen based on computational efficiency (Mecwan, et al., 009). LPC is a method used to pedict samples fom the speech signal though seveal pevious samples. LPC coefficients can be used to divide the speech signal into two pats, tansfe function and excitation. N-th sample can be pedicted fom the sequence of samples which epesented the weighted sum of p pevious samples. : p sˆ = a s[ n k] (7) k= k A set of samples (p) efes to a sequence of LPC. Typically, p is between 0 and 0, theeby to poduce an accuate epesentation with limited computing. Weight of the pevious sample (ak) is used to minimize squaed eo between the actual value with the pedicted value. This is

3 due to the need fo eo signal e[n], also called LPC esiduals, see equation 8 e[ n] = s[ n] sˆ[ n] = s[ n] a s[ n k] (8) p k= The eo signal e[n] can be pocced using z-tansfom, see equation 9. p p = k = k E( S( ak S( z S( ak z = S( A( (9) k= k= Thus, we get a epesentation of eo signal E( as the poduct of oiginal signal S( with tansfe function A(. The spectum of eo signal E( will have a diffeent stuctue depended on whethe the sound souces including voiced o unvoiced. When LPC coefficients ae computed successfully, these coefficients can be used to calculate the eo signal e(n). Implementation whee s(n) as input and e(n) is called the Speech Analysis esults Filte (Pak, Sung-won, 007), as shown in figue 3. Whee A( is defined as: Figue 3. Speech Analysis Filte Souce: Pak (007, p. 7) M A ( z i ) = a z (0) i i= While the implementation which uses eo signal e(n) as input and s(n) as output is called Speech Synthesis Filte, as shown in figue 4. Figue 5 will illustate how the LPC applied on the voice convesion system Figue 4. Speech Synthesis Filte Souce: Pak (007, p.8) Figue 5. Voice Convesion Scheme Souce: Uppeman (008, p.6) k A( is the tansfe function between the oiginal signal S(n) and the excitation component e(n). Tansfe function of the speech signal is the pat that elates to sound quality, which distinguishes the sound of one peson with anothe peson. Meanwhile, the excitation component of speech signal is the pat that elates to cetain sounds and wods poduced. As shown in Figue 5, we can put into the oiginal signal filtes to obtain excitation component. Leaving the excitation component into the invese filte ( / A () allows to get the signal oigin. C. Pitch Peiod Computation Accoding to Dunn (003), pitch detection has always been a complex issue in speech pocessing. Seveal algoithms fo detecting pitch poposed ae context-specific algoithm, can only wok well fo cetain cases. This equies a combination of appoaches to detect the pitch of the voice. This method basically uses Autocoelation of Voiced o unvoiced detection. Mecwan (009) stated that the pitch needed to detect the signal with a window fo at least two times the length of the peiod that may aise. At fist, do check whethe the window is included Voiced o not, this is done by compaing the aveage enegy of the window with a theshold. Accoding to Boesma(00), thee two main pocesses that need to be obtained to calculate the pitch peiod. These pocesses ae pitch analysis and glottal pulses calculation. This pitch analysis algoithm is based on autocoelation method (Boesma, 993). The algoithm is descibe as following: Step. Is a pepocessing step to eliminate the sidelobe of the Fouie tansfom using a Hanning window on the signal coming close to the Nyquist fequency. Step. Pefom global initialization fo absolute peak value of the signal (detail can be seen in Step 3.3). Step3. Since the method used is shot-tem analysis method, the analysis caied out on small sections (fames) taken fom the signal with the default timestep 0.0 seconds. Fo each fame, the candidates sought the maximum numbe of candidates. Unvoiced candidate will always be thee. The following steps will be executed fo each fame: Step3.. The length of the fame (window length) is detemined fom the minimum pitch (fundamental fequency) that wanted to detect. Window length should be long enough to cove thee peiods (eg, the minimum pitch to be detected 75Hz, then the window length is 40ms). Step3.. Deductions fo local aveage. Step3.3. The fist candidate is the unvoiced candidate, which will always exist. Stength of these candidates is calculated fom the two thesholds, the voicing theshold (theshold which detemines whethe including Voiced) and silence theshold (theshold which detemines whethe including silence). Fo example, voicing theshold is 0.4 and the silence theshold of 0.05, the fame will include voiceless if no Autocoelation peak above 0.4 o the local 3

4 absolute peak of less than 0.05 times the absolute global peak. Step3.4. Multiplied by the window function, see equation a() t = x tmid T + t μ x w() t () Step3.5. add 0 (zeo) as many as the window length. This pocess is done because the intepolation needs autocoelation value at least half of the window length. Step3.6. add 0 (zeo) until the numbe of the sample eaches a powe of two multiplication. Step3.7. do Fast Fouie Tansfom using equation a~ iωt ω = a t e dt () ( ) ( ) Step3.8. Squae the sample on fequency domain. Step3.9. do Invese Fast Fouie Tansfom using equation 3 ω ( τ ) = ~ ωτ ( ω) d i a a e (3) π Step3.0. then divide by autocoelation fom computed window fom step 3.5 until 3.9 as shown in equation 4 a ( ) ( τ ) x τ (4) τ w ( ) Step3.. find the position and the continue maxima value fom x ( τ ) using algoithm bent (Pess 99). The maxima position is laid between minimum and maximum pitch. While the maximum pitch must be laid between the minimum pitch and Nyquist fequency. Local stength fo unvoiced candidate can be witten as equation 5 ( local absolute peak )/( global absolute peak ) ( ) (5) VoicingTheshold + max 0. SilenceTheshold / + VoicingTheshold R And voiced candidate with highest local stength, can be calculated using equation 6. R ( τ max ) OctaveCost log( MinimimPitch τ max ) (6) OctaveCost paametes intended to poduce a bette fundamental fequency. Fom all the peiodic signals which have peak with the same height, will be selected the lowest lag. Value fo OctaveCost is default by 0.0 fo the citeion of 0% (squae of 0%). Afte epeating the pocess above, will be taken seveal pais of fequency and stength (F ni, R ni ), whee n is the index fo the fame (fom to numbe of fames) and i is the index fo candidate numbe (fom to the numbe of candidates). The best candidate fom each fame (locally best candidate) is the candidate with the highest R value. But thee is possibility whee thee ae equally stong candidates in some fames. Theefoe, it equied a step to select the global path finde (descibed in step 4). This is intended to minimize the amount of incidental Voicedunvoiced decision and a big jump fequency. Step 4. Fo each fame n, p n is a numbe fom to the numbe of fame candidates. Whee { p n n numbe of fames} defines a path though the candidates: {(F npn, R npn ) n numbe of fames}. With all the possible paths, cost can be defined using equation 7. numbeoffames ({ pn} ) = tansitioncost( Fn, p, F ) n npn cost n= n= numbeoffames R npn (7) Whee tansition cost function is defined as equation 8. 0 if F = 0 and F = 0 (8) tansitioncost( F, F ) = VoicedUnvoicedCost if F = 0 xo F = 0 F log 0 0 OctaveJumCost if F and F F Thee is a possibility VoicedUnvoicedCost and OctaveJumpCost have same value, 0. The globally best path is the path with least cost. The second pocess of pitch peiod computation is glottal pulses calculation. If a human wod is epesented in time domain, then the peodic patten of voiced sound, such as voiced obstuents ([b], [d], [g]), sonoant consonants ([n], [m], [l]) and all vowels will be found, as shown in figue 6 (Eulenbeg, John, 00). Figue 6. The epeating patten of human s wod Souce: Eulenbe (00) Evey patten identified epeatedly is called cycle. The duation of each cycle called the glottal pulse o the pitch peiod length (τ in Figue 6). The pupose of this step is to take advantage of the pitch contou infomation which has been poduced befoe, to put the glottal pulses. Numbe of points (o glottal pulses) is detemined by the following algoithm: Step. The fist point t is ablsolute extemum value of the amplitude, whose value is between t mid -T 0 / and t mid +T 0 /, whee t mid is the midpoint of the inteval, and T 0 is the peiod in t mid, which can be obtained fom the pitch contou intepolation. Step. Fom the fist point, seach the points t i until each left end of the inteval. Points must be between t i -. T 0 (t i- ) dan t i- 0.8 T 0 (t i- ). Step3. do the same as Step fo the ight side of t i, the seach is done until it eaches the ight end of the inteval. Step4. Although Voiced and unvoiced decision is detemined in the pevious section, the points will still be 4

5 eliminated if the coelation value was less than 0.3. Howeve, points can be added at the end of the inteval if its coelation value geate than 0.7. D. Tansfomation Afte going though the pocesses mentioned above (analysis phase), it will get the paametes of the speake souce. This paamete is mapped so that the combination can poduce sound fom the speake's pupose. Some impotant paametes (speech paametes), among othes ae Voiced / unvoiced flag, filte coefficient (taget), the mean value of gain (taget), souce pitch, the taget pitch. Speech paametes wee used to econstuct the output so that sounds like the taget speech. Pitch-Synchonous Ovelap and Add (PSOLA) is a method used to manipulate the pitch of a speech signal so that it matches the pitch of the taget speakes. Basic algoithm of PSOLA consists of 3 stages (Uppeman, Gina, et al., 008). Fist stage, the speech signal is divided into seveal smalle signals and ovelap each othe. This is obtained by windowing aound the "pitch mak" o peak amplitude (peak ampitude) fom the fist signal (oiginal signal). Windowed segments (the pat that has been though a pocess of windowing) usually consist of two to fou pitch peiods. The second phase, small signals ae epeated o modified in a way emoved fom the speech segments, depending on the pitch of the taget speakes, whethe highe o lowe than the pitch-souce speakes. This esulted in a modified duation of the signals that lead to fundamental changes in fequency. The last stage, the emaining segments combined by ovelapping and e-adding. The esult is a signal with the same spectum with the oiginal but with a diffeent fundamental fequency. Thus, the pitch of his voice changed, but the quality emains the same. making this application, especially at the voice segmentation pocess. In addition thee is also the speake vaiability (divesity of speakes). Evey peson has a unique voice. But the sound poduced by a single peson can also vay. Theefoe, vaiations in sound can be classified based on the actual ponunciation (the same wod can be ponounced in diffeent ways; is neve exactly the same when viewed fom the acoustic wave), speaking style, gende, vocal tact anatomy, speaking ate, dialect, etc. The design of the application of voice convesion system outline is shown in Figue 7 below: III. PROBLEM ANALYSIS The complexity of human language is one of the difficult poblems faced in making this voice convesion application. When a man speaks, the ai exhaled fom the lungs and then though the mouth and nasal cavity. Ai flows out though the mouth, esticted and manipulated by the tongue and lips. This manipulation pocess poduce contaction and expansion against the aiflow eleased, the acoustic waves called sound. Howeve, a set of a voice seies alone can not fom wods and sentences. In a speech, it has useful infomation, such as hythm, intonation and emphasis that indicate gammatical stuctue, o also an emphasis on cetain wods to indicate the level of inteest, and othes. The infomation can be diffeent fom one peson to anothe. Vaiations in speech style ae not limited to the inceasing complexity of Figue 7.Flowchat of Vocie Convesion System IV. EXPERIMENT A. Expeiment on Voice Segmentation Based on Syllable Voice segmentation is necessay in dividing the signal into smalle pats based on the syllable. Thus, the expected chaacteistics of the speake can still be maintained. To segment the voice in this application, we use end-point detection with a combination of theshold volume and theshold zeo-cossing ate. These following expeiments will be descibed in tems of wavefom, whee the pink line 5

6 maks the beginning of the segment, and light geen line maks the end of the segment. Results fo the ponunciation of moe complex wods, "Sistem Cedas" povides diffeent esults fo diffeent speakes, the success ate of 50% fo the expeiment above. This is elated to the divesity of speakes o so-called speake vaiability. Divesity is what causes the speake system is difficult to segment accuately. B. Expeiment on Pitch Similaity Between Simulated Speake and Taget Speake Hee is a test to find out how close the pitch of the sound poduced by the application with the sound of the souce speake and taget speake. Table. Compaison of Pitch Calculation Souce Taget No wod Feq Feq Speake Speake (H (H Conveted Feq (H Good Kath Liz Hai Kath Zefan Saya Liz Will Hallo Zefan 6.03 Liz A Will Zefan Figue 8. Segmentation esult fo wod hai fom fou speakes. Fom above: (a) Kath (b) Liz (c) Will (d) Zefan. The expeimental esult fo wod with one syllable of fou speakes, shows the succesfulness of thi application. Then, we also test moe complex wods, Sistem Cedas. The segmentation esult can be seen in figue 9. Figue 9. Segmentation esult fo wod Sistem Cedas fom fou speakes, fom above: (a) Kath (b) Liz (c) Zefan (d) Will Table. The Pecentage of Pitch Calculation No wod Taget Conveted Feq (H Feq (H Good 66,949 63,579 Hai 76,590 7,438 3 Saya 70,389 7,438 4 Hallo 00,87 00, A 9,575 88,679 Pitch of the sound poduced by this application is appoximate the pitch of the taget speakes, although not exactly the same. Fom the compaison of the ates can be seen that the pitch modification is quite successful (the aveage pecentage of success of 98.67%). V. CONCLUTION Based on the expeiments esult, we can conclude some of the following: The segmentation esult is good enough to detect syllable in cetain wods, depend on the input which can be vay widely. Segmentation successfulness is depended on the length of utteance. The longe the utteance, the lowe the success ate will be. The esult of segmentation will affect the whole pocess of voice convesion. A good segmentation will poduce good esult. Modify the pitch gives good esults. This is poven by the aveage pecentage, 98.67% as shown on table. The convesion fom woman voice to man voice has the best esult among othe convesion schemes. It is because women have moe geneal and smooth voice chaacteistics than men. 6

7 REFERENCES Boesma, Paul. (993). Accuate Shot-Tem Analysis of Fundamental Fequency and the Hamonic-to-Noise Ratio of Sampled Sound. Poceedings of the Institue of Phonetics Sciences 7: Univesity of Amstedam. Boesma, Paul. (00). Paat, a system fo doing phonetics by compute. Glot Intenational. Dunn, Bob. (003). Speech Signal Pocessing and Speech Recognition. IEEE Signal Pocessing Society. Eulenbeg, John. (00). Fundamental Fequency and the Glottal Pulse. Retieved May, 00, fom l_pulse_peiod.html Jyh-Shing Roge Jang. (009). Audio signal pocessing and ecognition. Retieved Febuay, 00, fom Jyh-Shing Roge Jang. Speech and audio pocessing toolbox. Retieved Febuay, 00, fom Kain, A. dan Macon M. (998a). Spectal Voice Convesion fo Text-To- Speech Synthesis. IEEE ICASSP 998. Kain, A. dan Macon M. (998b). Text-To-Speech Voice Adaptation fom Spase Taining Data. ICSLP 998. McClellan, James, dkk. (998). DSP Fist: A Multimedia Appoach. Pentice-Hall, Inc., New Jesey. Mecwan, Akash, dkk. (009). Voice Convesion Algoithm. Intenational Confeence on Advances in Computing, Communication and Contol 009. Milosevic, Banislava. Multimedia Taining Kit: Intoduction to Digital Audio Handout. Retieved Januay 3, 00, fom Pak, Sung-won. (007). Chapte 7 Linea Pedictive Speech Pocessing. Retieved Januay 6, 00, fom Patton, Joshua. (007). Pitch Synchonous Ovelap and Add (with Fomant Pesevation). Retieved May, 00, fom l Pelton, Godon E. (993) Voice pocessing. McGaw-Hill, Inc. Pess, William H., dkk. (99). Numeical Recipes in C. Cambidge Univesity Pess. Rabine, Lawence dan Biing-hwang Juang (993). Fundamentals of Speech Recognition. Pentice Hall, Inc., New Jesey. Ratana, Dhany Suya. (009). Kompaasi Kompesi Audio AAC dengan MP3. ITT TELKOM. Retieved Januay, 00, fom 8:multimedia&id=49:audio-extension-wav-mp3- aac&option=com_content&itemid=5. Ribeio, C.M. dan I. M. Tancoso. (997). Phonetic Vocoding with Speake Adaptation. Euospeech 997, Rhodes, Geece. Tuk, Oytun. (003). New Methods fo Voice Convesion. M.S Thesis. Toivonen, Hannu T., (009). Chapte. Multiate Digital Signal Pocessing. Retieved May 5, 00, fom Uppeman, Gina, dkk. (008). Methods fo Voice Convesion. Retieved May 0, 009, fom Wikipedia the Fee Encyclopedia. (009). Mean Opinion Scoe. Retieved Octobe 7, 009, fo Wilson, Scott. (003). WAVE PCM File Fomat. Retieved Januay, 00, fom Ze-Nian Li dan Mak S. Dew. (004). Fundamentals of multimedia. Uppe Saddle Rive, NJ: Peason Education, Inc. 7