Complex-valued restricted Boltzmann machine for direct learning of frequency spectra

Transcription

1 INTERSPEECH 17 August, 17, Stockolm, Sweden Complex-valued restricted Boltzmann macine for direct learning of frequency spectra Toru Nakasika 1, Sinji Takaki, Junici Yamagisi,3 1 University of Electro-Communications, Tokyo, Japan National Institute of Informatics, Tokyo, Japan 3 University of Edinburg, Edinburg, UK nakasika@uec.ac.jp,takaki@nii.ac.jp,jyamagis@nii.ac.jp Abstract In tis paper, we propose a new energy-based probabilistic model were a restricted Boltzmann macine (RBM) is extended to deal wit complex-valued visible units. Te RBM tat automatically learns te relationsips between visible units and idden units (but witout connections in te visible or te idden units) as been widely used as a feature extractor, a generator, a classifier, pre-training of deep neural networks, etc. However, all te conventional RBMs ave assumed te visible units to be eiter binary-valued or real-valued, and terefore complex-valued data cannot be fed to te RBM. In various applications, owever, complex-valued data is frequently used suc examples include complex spectra of speec, fmri images, wireless signals, and acoustic intensity. For te direct learning of suc te complex-valued data, we define te new model called complex-valued RBM (CRBM) were te conditional probability of te complex-valued visible units given te idden units forms a complex-gaussian distribution. Anoter important caracteristic of te CRBM is to ave connections between real and imaginary parts of eac of te visible units unlike te conventional real-valued RBM. Our experiments demonstrated tat te proposed CRBM can directly encode complex spectra of speec signals witout decoupling imaginary number or pase from te complex-value data. Index Terms: restricted Boltzmann macine, deep learning, complex-valued data, speec encoding, speec syntesis 1. Introduction Deep learning is one of te recent ottest topics in wide researc fields suc as artificial intelligence, macine learning, and signal processing including image classification, speec recognition, etc[1]. Many models ave been proposed so far as a tool of deep learning; one of te most widely-used and famous models is a deep belief-net (DBN) [] tat stacks multiple restricted Boltzmann macines (RBMs) layer-by-layer. Te RBM, wic is a probabilistic model tat consists of visible and idden units, as often been used alone as a feature extractor, a generator, and a classifier as well as a pre-training sceme of deep neural networks, and many extensions of te RBM ave been also proposed [3,, 5, ]. Altoug te RBM as been used in so many tasks, te RBM traditionally assumed visible units to be eiter binary-valued or real-valued [, 7, ]. Concerning speec signal processing, representations based on amplitude spectra of speec suc as MFCC and mel-cepstra are normally used as input features of speec recognition or output features of speec syntesis, because it is known tat te amplitude spectra are more effective and relevant to our auditory field tan pase spectra in suc tasks. However, tese features teoretically lack pase information. More specifically, te use of te amplitude-based features only cannot represent te original complex values correctly. In oter signal processing as well, tere are many cases were we ave to deal wit complex-valued actual data suc as fmri images, wireless signals, acoustic intensity, etc. Oter macine learning models, tat is neural networks, Boltzmann macines, and non-negative matrix factorization (NMF) [9], ave teir extensions proposed to represent complex-valued data [1, 11, 1]. In tis paper, we newly propose an extension of te RBM tat deals wit complex-valued data, and evaluate its effectiveness troug experiments using artificial data and speec spectra. Te proposed model called complex-valued RBM (CRBM) consists of complex-normal visible units and Bernoulli idden units. Te CRBM as important caracteristics of aving no connections across dimensions in te same layers but aving connections between real and imaginary parts of eac of te visible units unlike te conventional RBM. Terefore, it is easy to estimate te parameters using Gibbs sampling or contrastive divergence [], and it is expected to aggregate te information propagated from te complex-valued visible units into te idden units. Suc caracteristics can not be seen in an extension of Boltzmann macine (directional-unit Boltzmann macine (DUBM) [11]) tat feeds complex-valued data aving connections across dimensions, wic makes te parameter estimation difficult. Anoter difference between te proposed CRBM and te DUBM is te form of complex-valued visible units; te visible units in te CRBM are in rectangular form aving real and imaginary components, wile tose in te DUBM are in polar form aving pase components wit amplitude components. Since te conditional probability of visible units given idden units in te CRBM form a complex-normal distribution, wic makes te real and imaginary components Gaussian-distributed, respectively, we can generate samples from te distribution straigtfowardly. Tis paper is organized as follows: In Section we overview te conventional real-valued RBM. In Section 3 we define te proposed Complex-valued RBM and sow its parameter estimation algoritm. In Section we sow our experimental results and conclude our findings in Section 5.. Preliminary A restricted Boltzmann macine (RBM [, 13]), one of te most-widely used energy-based models, is convenient for representing latent features tat are cannot be observed but surely exist in te background. An RBM was originally introduced as an undirected grapical model tat defines te distribution Copyrigt 17 ISCA 1 ttp://dx.doi.org/1.137/interspeec.17-5

2 z z q b c W b W Figure 1: Grapical representation of a complex-valued RBM. of binary visible variables wit binary idden (latent) variables, and was later extended to deal wit real valued-data known as a Gaussian-Bernoulli RBM (GB-RBM) []. It as been, owever, reported tat te original GB-RBM ad some difficulties because te training of te parameters was unstable. Later, an improved learning metod for a GB-RBM as been proposed by Co et al. [13] to overcome te difficulties 1. In te modeling using an RBM, te joint probability p(v, ) of real-valued visible units v R I and binary-valued idden units {, 1} J (I and J indicate te numbers of dimensions in te visible and idden units, respectively) is defined as follows: p(v; θ) = p(v, ; θ) (1) p(v, ; θ) = 1 U(θ) e E(v,;θ) () E(v, ; θ) = 1 v Σ 1 v b Σ 1 v c v Σ 1 W (3) U(θ) = e E(v,;θ) dv () were θ = {b, c, W, σ} indicates a set of parameters, wic contains bias parameters of te visible units b R I, bias parameters of te idden units c R J, te connection weigt parameters between visible-idden units W R I J, and te standard deviation parameters associated wit te dimension independent Gaussian visible units σ R I tat defines Σ (σ ) (te function ( ) returns a diagonal matrix wose diagonal vector is te argument). From te above definition, te conditional probabilities p(v ) and p( v) form simple distributions as: p(v ) = N (v; b + W, Σ) (5) p( v) = B(; f(c + W Σ 1 v)) () were N ( ; µ, Σ), B( ; π), and f( ) indicate te multivariate Gaussian distribution wit te mean µ and te covariance matrix Σ, te multi-dimensional Bernoulli distribution wit te success probabilities π, and an element-wise sigmoid function, respectively. 1 In te remaining of tis paper, we refer to te improved GB-RBM just as an RBM. c 3.1. Definition 3. Complex-valued RBM In tis section, we will define an extension of te RBM (complex-valued RBM; CRBM) tat can feed complex-valued data but as a real-valued probability distribution (cost function used in parameter estimation) like te conventional RBM. In te CRBM, we will give a restriction were tere are no connections between visible units or idden units, wic enables easy estimation of parameters just as an RBM does; owever, in order to capture te relationsips between te real and imaginary parts of eac complex-valued visible unit, we will allow te model to ave connections between te real and imaginary parts. Based on te above extension, we formulate te CRBM tat as I-dimensional complex-valued visible units z C I and J- dimensional binary-valued idden units {, 1} J as follows in tis paper: p(z; θ) = p(z, ; θ) (7) p(z, ; θ) = 1 U(θ) e E(z,;θ) () E(z, ; θ) = 1 [ ] H [ ] Φ z z 1 z z [ ] H [ ] Φ b b 1 c z z (9) [ ] H [ ] Φ z z 1 W W U(θ) = e E(z,;θ) dz, (1) were and H denote complex-conjugate and Hermitiantranspose, respectively. b C I, c R J, and W C I J are bias parameters of te visible units and te idden units, and te biased connection weigts between visible and idden units, respectively. In order to make te restrictions, te extended covariance matrix Φ consists of a covariance matrix Γ and a pseudo-covariance matrix C, bot of wic are diagonal matrices, as [ ] Γ C Φ (11) and C H Γ H Γ (γ), γ R+ I C (δ), δ C I (1) were γ and δ are variance and pseudo-variance parameters of te complex-valued visible units, respectively. To summarize, te set of parameters of te CRBM is θ = {b, c, W, γ, δ}. Introducing two vectors p and q defined as γ p γ δ RI (13) δ q γ δ CI (1) were te fraction bar denotes element-wise division, we can

3 rewrite te energy function in Eq. (9) as follows: E(z, ; θ) = z H (p)z + R(z H (q) z) R(z H (p)b) R(z H (q) b) c R(z H (p)w) R(z H (q) W), (15) wic confirms tat 1) te above energy function E and te probability distribution are real-valued, and tat ) tere are connections between te complex-valued visible units and teir conjugates for eac dimension but no connections between different dimensions. Furtermore, wen we use unbiased parameters: te energy function E becomes b (p)b + (q) b (1) W (p)w + (q) W, (17) E(z, ; θ) = 1 zh (p)z + 1 zh (p) z + z H (q) z + z H ( q)z z H b z H b c z H W z H W, (1) wic indicates tat z and z are symmetric to eac oter as sown in Figure 1. From te above definition, te conditional probabilities p(z ) and p( z) can be derived as follows: p(z ) = CN (z; b + W, Γ, C) (19) p( z) = B(; f(c + R(W H z))) () were CN ( ; µ, Γ, C) is a multivariate complex normal distribution a mean vector µ, a covariance matrix Γ, and a pseudocovariance matrix C: 1 p(z) = π D det(γ)det(q) { exp 1 [ ] H [ z µ Γ C z µ C H Γ H ] 1 [ z µ z µ ] } (1) Q = Γ C H Γ 1 C. () 3.. Parameter estimation In tis paper, we estimate te parameters of te CRBM θ using complex-valued gradient ascend so as to maximize te loglikeliood of te complex-valued training data z: L(θ) = log p(z; θ) (3) = log = log p(z, ; θ) () e E(z,;θ) log e E( z, ;θ) d z. (5) Te complex-valued gradient ascend iteratively updates eac parameter wit a learning rate α > as: θ (new) θ (old) + α, () θ were te partial gradients in Eq. () are te Wirtinger derivative: θ = 1 ( R(θ) i θ = 1 ( R(θ) + i ) I(θ) I(θ) (7) ). () Te partial gradients of eac parameter include te expectations of te partial gradients to te energy function along te training data and te model, wic can be approximated and easily calculated wit contrastive divergence (CD) [] in a similar way to te conventional RBM because of te restriction of te CRBM. Te partial gradients of eac parameter to te energy function can be derived as: b = (p) z + ( q)z (9) c = (3) W = ( (p) z + ( q)z) (31) γ = (p + q ) + R(p q p q ) (3) δ = p q + q + p q q p, (33) were and denote element-wise product and absolute, respectively, and p = 1 z R(z ( b + W)) (3) q = 1 z z ( b + W). (35) Te gradients of variance and pseudo variance tend to be larger tan tose of te oter parameters. For stable training, we replace te parameters as γ e r and δ e s, and update using te gradients of r and s, respectively.. Experiments.1. Evaluation using artificial data In order to evaluate te effectiveness of te proposed CRBM, we first conducted an experiment using one-dimensional complex-valued artificial data (N = ). Te artificially created data is illustrated in Figure as black dots, wic as correlations between te real and imaginary parts. In tis experiment, we compared te CRBM wit a GB-RBM aving two visible units; one is for te real part, anoter is for te imaginary part. We trained bot models wit two idden units using stocastic gradient descent (SGD) wit a learning rate of.1, a momentum of.1, a batc size of, and a number of epocs as. After te training, we randomly generated samples from te models; te samples from te CRBM and te GB-RBM are sown as red dots on te above and on te below of Figure, respectively. As sown in Figure, we can see tat te proposed CRBM could represent te distribution of te complex-valued artificial data more accurately tan te GB-RBM. Tis is because te CRBM can capture te relationsips between te real and imaginary parts wile GB-RBM does not capture te correlations between tem. 3

4 1515 Im Re Figure 3: Amplitude spectra of te original speec (above) and of te reconstructed one from te CRBM (below) CRBM GB-RBM 1 55 MSE Im Re 1 Epoc Figure : Artificially created 1D complex-valued data (black dots) and random samples (red dots) generated from te trained models: te proposed CRBM (above) and te conventional GBRBM (below). Figure : MSE during te training of te CRBM (red line) and te GB-RBM (blue line). MSE of 51.9 from te CRBM, wic outperformed te MSE of 5.1 from te GB-RBM... Evaluation using speec data 5. Conclusion Second, we conducted an encoding-and-decoding experiment using speec data from te Repeated Harvard Sentence Prompts (REHASP) corpus. From te corpus, we randomly selected 3 repeats of 3 sentences, processed te sort-time Fourier transform (STFT) wit a windows lengt of 5 wit overlapping samples, and trained te CRBM wit idden units (i.e., I = 19 and J = ). For evaluation, we first estimated idden units from te test data tat was different from te training data (encoding), and ten reconstructed te visible units from te idden units (decoding). Te performance of te encoding-and-decoding was evaluated using an objective criteria of mean-squared error (MSE) between te original data and te reconstructed one. In tis experiment, we trained te model wit a learning rate of.1, a momentum of.1, a batc size of 1, and a number of epocs as 5. For comparison, we also trained a GB-RBM wit doubled visible units, tat is a super vector aving te real and imaginary parts of te complexvalued spectra, wit te same configuration. Figure 3 sows an example of te reconstruction using te CRBM. As sown in Figure 3, te reconstructed spectra was fairly closed to te original spectra. Figure sows MSE calculated during te training, comparing te proposed CRBM wit te GB-RBM. We notice from Figure tat te CRBM converged more quickly tan te GB-RBM, and te MSE in convergence (at around 1 epocs) of te CRBM is muc smaller tan tat of te GB-RBM. For te test data, we also obtained te In tis paper, we proposed an extension of restricted Boltzmann macine (RBM) tat can feed complex-valued visible units, called complex-valued RBM (CRBM). We formulated te model and sowed tat te conditional probability of visible units given idden units formed a complex-valued normal distribution. To evaluated te performance of te proposed CRBM, we conducted experiments using artificial complex-valued data and complex spectra of speec. Troug te experiments, we conclude tat te model would be effective to represent complex-valued data especially wen te real and imaginary parts are correlated wit eac oter. In te future, we would like to fully investigate te performance of te CRBM wen using speec data in terms of subjective criteria, and to apply te model to represent oter complex-valued actual data in te real world. Future work also includes extensions of te CRBM; e.g., te model stacking multiple idden layers layer-by-layer like deep Boltzmann macine (DBM) [3], and te model aving complex-normal idden units instead of Bernoulli idden units so as to extract complex-valued latent features. Acknowledgements: Tis work was partially supported by JST ACT-I Grant Number (JPMJPR1UG), by MEXT KAKENHI Grant Numbers (, 15H1, 1K19, 1H3), and by Te Telecommunications Advancement Foundation Grants. Te autors would like to tank Dr. H. Kameoka from NTT Communication Science Laboratories for is valuable advice and comments. ttp://datasare.is.ed.ac.uk/andle/13/51

5 . References [1] Y. LeCun, Y. Bengio, and G. Hinton, Deep learning, Nature, vol. 51, no. 7553, pp. 3, 15. [] G. E. Hinton, S. Osindero, and Y. W. Te, A fast learning algoritm for deep belief nets, Neural computation, vol. 1, no. 7, pp ,. [3] R. Salakutdinov and G. E. Hinton, Deep Boltzmann macines, AISTATS, pp. 55, 9. [] A. Krizevsky and G. E. Hinton, Factored 3-way restricted boltzmann macines for modeling natural images, Journal of Macine Learning Researc, 1. [5] K. Son, G. Zou, C. Lee, and H. Lee, Learning and Selecting Features Jointly wit Point-wise Gated Boltzmann Macines, ICML (), 13. [] T. Nakasika, T. Takiguci, and Y. Minami, Non-Parallel Training in Voice Conversion Using an Adaptive Restricted Boltzmann Macine, IEEE/ACM Transactions on Audio, Speec, and Language Processing, vol., no. 11, pp. 3 5, 1. [7] Y. Freund and D. Haussler, Unsupervised learning of distributions of binary vectors using two layer networks, Computer Researc Laboratory, pp , 199. [] H. Lee, C. Ekanadam, and A. Y. Ng, Sparse deep belief net model for visual area V, in Advances in neural information processing systems,, pp. 73. [9] D. D. Lee and H. S. Seung, Learning te parts of objects by non-negative matrix factorization, Nature, vol. 1, no. 755, pp , [1] I. Nemoto and T. Kono, Complex neural networks, Systems and computers in Japan, vol. 3, no., pp. 75, 199. [11] R. S. Zemel, C. Williams, and M. C. Mozer, Lending direction to neural networks, Neural Networks, vol., no., pp , [1] H. Kameoka, N. Ono, and K. Kasino, Complex NMF: A new sparse representation for acoustic signals, ICASSP, pp , 9. [13] K. Co, A. Ilin, and T. Raiko, Improved learning of Gaussian- Bernoulli restricted Boltzmann macines, in Proc. ICANN. Springer, 11, pp