ADAPTIVE INITIALIZED JACOBI OPTIMIZATION IN INDEPENDENT COMPONENT ANALYSIS

Transcription

1 ADAPTIVE INITIALIZED JACOBI OPTIMIZATION IN INDEPENDENT COMPONENT ANALYSIS y Juan J. Murillo-Fuenes and y Rafael Boloix-Torosa z Francisco J. González-Serrano, y ATSC. ESI. Universidad de Sevilla. Paseo de los Descubrimienos sn, 9 Sevilla, Spain. z DT. EPS, Universidad Carlos III de Madrid. Buarque 5, 89 Leganés, Madrid, Spain. murillo@esi.us.es ABSTRACT In his paper, we focus on he fourh order cumulan based adapive mehods for independen componen analysis. We propose a novel mehod based on he Jacobi Opimizaion, available for a wide se of minimum enropy (ME) based conrass. In his algorihm we adapively compue a momen marix, an esimae of some fourh order momens of he whiened inpus. Saring from his marix, he soluion o he n-dimensional ME ICA problem may be solved a any ime by means of he Jacobi Opimizaion approach. In he experimens included, we compare his mehod o previous ones such as he ademl or he EASI, obaining a beer performance.. INTRODUCTION Independen Componen Analysis (ICA) [] involves he ask of compuing he marix projecion of a se of componens ono anoher se of so called independen componen. Here, he objecive is o minimize he saisical independence of he oupus. If we know he inpus o he ICA o be linear insananeous mixure of a se of sources. The ICA process provides an esimaes of he original sources. Here, and in he conex of his paper, neiher he original sources nor he mixure marix iself are known. This is he Blind Separaion of Sources (BSS) [] where he aim is o obain a non-observable se of signals, he so-called sources, from anoher se of observable signals regarded as mixures. In his paper we focus on BSS/ICA approaches based on he minimizaion of he enropy (ME) [], [3]. Mos of hem are based on he minimizaion of a cos funcion, he conras, in he wo dimensional case. In he n-dimensional problem he Jacobi Opimizaion (JO) [] is used, i.e, we operae pairwise minimizing he associaed -dimensional conras for every whiened-signal pair in urn over several sweeps unil convergence. This process may be carried ou adapively, as he ademl mehod in []. The ademl algorihm is based on he adapive learning of a differen se of parameers for every pair and sweep. These parameers are funcions of he fourh order momens of he oupus for ha pair and sweep. Thus, he algorihm is, a he same ime, learning he sysem and he soluion. In his paper we propose a new mehod where he learning of he sysem and he compuaion of he soluion are decoupled. On he one hand we adapively learn he momens of he whiened mixures. On he oher hand, we may compue he soluion o he ICA/BSS problem a any momen from hese momens by using he JO applied o any -dimensional conras. The paper is organized as follows. We firs inroduce he marix model of he BSS problem. In Secion, we propose a general conras o be used in he JO. Nex, we exend he ademl o be available for hese family of funcions. We devoe Secion 3 o he new algorihm proposed in his paper, he Adapive Iniilized Jacobi Opimizaion (AIJO). Some experimenal resuls are included in Secion. We end wih main conclusions... Marix model in he BSS/ICA In he BSS/ICA insananeous model, he enries of he n mixure vecor x a ime are insananeous linear combinaions of he n saisically independen sources (componens) s, i.e., x = As. We assume x o be a saionary ergodic random sequence and ha he mixing marix A is non-singular. Under hese assumpions, i is possible o esimae a separaion marix B o obain he sources as y = Bx. This separaing marix B can be decomposed ino he produc of a whiening W and a roaion V marix. Hence, y = Bx = VWAs = Vz =; ;::: () The fourh order approximaion o he ME conras [] yields ffi ME (y) ß 8 ffime (y) = 8 X i (C y iiii ) () where, for zero-mean signals, C y iiii = E[y i ] 3E[y i ] are he marginal cumulans or auocumulans of he ih oupu and

2 E[ ] denoes mahemaical expecaion. Noice ha conras ffi ME assumes he oupus are decorrelaed. Thus, he problem reduces o he compuaion of marix V. Several approximaions o he conras in () have been proposed.. GENERALIZED ADAPTIVE METHOD In his secion we firs include a general expression for mos of he fourh order momens ME based conras and hen rewrie he ademl mehod for hem... A General Conras The pair (p; q) of whiened inpus may be wrien in polar coordinaes as [z p () z q ()] T =[r() cos(fi()) r() sin(fi())] T. As marix V performs a roaion of so ha ρ() = +fi() is he angle of vecor y he produc y = Vzyields» r()cos(ρ()) r()sin(ρ()) = V ( )» r()cos(fi()) r() sin(fi()) If we denoe by (r();ff()) he zero-mean uni-variance pair of sources μs = [μs p () μs q ()] T, a correc esimae should mee ρ = + fi() =ff() +kß=, k =; ; 3;:::. The conrass and esimaes of he roaion angles may be wrien in polar form by means of he following complexvalued linear combinaions (cenroids) [5] of he saisics of he oupus (3) ο = E[r ()e jfi() ] () = E[r ()e jfi() ] (5) fl = E[r ()] 8 (6) where j = p. A wide se of esimaes accep a general expression, he so called weighed esimaors (WE) [6], [7]. In order o exend he WE esimaor o he ML [8] or MK [9] case we propose he generalized weighed esimaor (GWE) as GW E (! fl ;! ο )= (! ο! fl ο +(! ο ) ) (7) <! ο < ;! fl = ±;fl where ( ) supplies he principal value of is argumen. As described in [6], paricular cases for his conras funcion has been proposed as approximaions o he ME conras funcion: EML = GW E (fl; ) [5], AML = GW E (fl; =3) [7] and MaSSFOC = GW E (fl; =) []. Wih he GWE in (7) we may rewrie he esimaors in [], MK [9], [], SKSE or ML [8] as GW E (±; ). I also may be proved [3] ha he minimizaion of ffi ME ( ) yields he angle SICA = GW E (fl; 3=7) (8).. Exended ademl We firs sudy he JO o exend he previous GWE conras o he n-dimensional case. In his sense he ademl [] is and adapive algorihm o solve he EML conras based on he JO. We rewrie his algorihm o solve he GWE. Noice ha in he JO we operae pairwise compuing he wo dimensional esimae GW E for every signal pair in urn over several sweeps. As his process is carried ou a each ieraion, we may adapively learn he following saisics a sweep (c) for he pair (p; q)as ο (c;pq) (c;pq) fl (c;pq) =( ν)ο (c;pq) + νe[r c;pq ()ejρ c;pq () ] (9) =( ν) (c;pq) + νe[r c;pq ()ejρ c;pq () ] () =( ν)fl (c;pq) + ν(e[rc;pq ()] 8) () Besides, as we esimae he roaion marix V under he whiening consrain, we mus compue z = Wx. Thus, we firs should updae he whiening marix W. In he following, we will use he relaive gradien based [] whiening algorihm W + = W + The adapive algorihm yields I (W x)(w x) T + j(w x) T (W x)j W () Algorihm Adapaive JO for he GWE esimaes (AJO- GWE). A each sample insan,. Whiening. Updae he whiening marix W as in () and se c =, y = z = W x and V = I.. One sweep (c). For all g = n(n )= pairs (y p ;y q ), i.e., for» p<q» n,do (a) Compue ο (c;pq), (c;pq) (b) Compue he Givens angle ο (c;pq), (c;pq), fl (c;pq)., fl (c;pq) in (9)-(). GW E in (7) by using (c) If GW E > min, do updae he roaion marix V and roae he pair (y p ;y q ) wih roaion angle GW E. 3. End? If he number of sweep c saisfies p c =+ n or no angle GW E has been updaed, compue he separaion marix as B = VW and sop. Oherwise go o sep for anoher sweep wih c ψ c ADAPTIVE INITILIZED JACOBI OPTIMIZATION In he previous secion he Jacobi opimizaion was inroduced o exend he problem o n dimensions. In he

3 sep.a of Algorihm, he Givens angle pq is compued by using equaions (9)-(). Simple calculus and rigonomerics show ha hese saisics ο (c;pq), (c;pq) and fl (c;pq) may be wrien as a funcion of he momens of he oupus E[yp y q ]; E[y p ]; E[y q ]; E[y pyq 3 ]; and E[y3 p y q] a sweep c. Bearing his in mind, we will face nex he compuaion of he whole se of momens jus one ime a an iniial sage and hen roae hem a each sep of he algorihm. Proposiion Given he model y = Vz in (), here exis a symmeric l l, l = n(n + )=, marix M (a(k; l);b(i; j)) = μ z ijkl ; (3) a diagonal consan marix S and vecors v pp; vpq and vqq such ha he fourh order momens of he oupus, y p and y q, yields E[yp y q ]=v ppsmsv T qq () E[yp ]=v ppsmsv T pp (5) E[yq ]=v qqsmsv T qq (6) E[y p yq 3 ]=v pqsmsv T qq (7) E[yp 3 y q]=vppsmsv T pq ; (8) Where M is a l l, l = n(n +)=, symmeric marix whose enries are he fourh-order momens μ z ijkl :» i» j» n;» k» l» n. The momen μ z ijkl is sored in he enry M (a; b), where he column a and row b yield b = a = h=n i+ h=n k+ h +(j i) :» i» j» n (9) h +(l k) :» k» l» n () Noice ha μ ijkl = μ jikl = μ kjil = μ ikjl. Hence, we only esimae he subse of differen momens μ z ijkl» i» j» k» l» n. The compuaion of ()-(8) may be rewrien by inroducing a pair of roaion vecors o lef-righ muliply marix M. The enries a of hese vecors wrien as a funcion of he enries of he uniary marix V in () yields vpp(a) = V (p; k)v (p; l) vpq(a) = V (p; k)v (q; l) +V (p; k)v (q; l) vqq(a) = V (q; k)v (q; l) () where he indexes k; l and a are relaed hrough (). Finally, S is a diagonal marix whose enries S(a; a), a as in (), yield S(a; a) = l 6= () S(a; a) = =; l = (3) The formulaion inroduced above allows an easy compuaion of he oupu saisics for a given roaion marix, as he enries V (p; q) involved are easily arranged in a pair of roaions vecors. The marix momen may be easily updaed wih each new sample as M =( ν)m + νm z () where M z is marix M in (3) compued wih jus he sample of he whiened inpus, z. The adapive algorihm yields Algorihm Adapaive Iniialized JO for he GWE esimaes (AIJO-GWE). Se V = I. A each sample insan,. Whiening. Updae he whiening marix W as in () and z = W x.. Marix Momen Iniializaion. Adapively compue marix M in () wih z. Each N samples,. Se sweep number c =.. One sweep (c). For all g = n(n )= pairs (y p ;y q ), i.e., for» p<q» n,do (a) Compue momens in ()-(8) by using M. (b) Compue he Givens angle GW E in (7) by using ο,, fl in ()-(6) (wih [z p () z q ()] T = [y p () y q ()] T ). (c) If GW E > min, do updae he roaion marix V wih roaion angle GW E. 3. End? If he number of sweep c saisfies p c =+ n or no angle GW E has been updaed, compue he separaion marix as B = V W and sop. Oherwise go o sep for anoher sweep wih c ψ c +. In Algorihm he learning of he sysem and he compuaion of he soluion are decoupled. In fac, he AIJO algorihm is divided in wo parallel subrouines. On he one hand we updae he momens of he oupus wih he las sample. On he oher hand, we compue he soluion B = V W each N samples. The main advanage of his design is ha we improve he performance. Noice ha in he AJO we updae he saisics ο (c), (c) and fl (c) wih samples of he las esimaed oupus y, and hese ones depend on he previous esimaions ο (c), (c ) and oupus. Thus, convergence in he las sweeps and pairs is condiioned o he behaviour of he firs ones. Hence, for large numbers of sources we need o increase he oal number of sweeps and he convergence deerioraes.

4 . COMPUTATIONAL COMPLEXITY We now measure he compuaional burden of he adapive algorihm presened and compare i wih oher mehods. We will consider a floaing poin operaion (flop) as a real muliplicaion. A each sample insan algorihm mus:. Whiening: he whiening algorihm in () akes n 3 + n flops.. Marix Momen calculaion: as described in [5] he number of muliplicaions and accumulaions necessary o compue M z (n+3)! are approximaely. (n )!! Since here are some duplicaed muliplicaions in he calculaion of he momens, his number could be reduced o (n+3)! (n )!! + (n+)n 3. Marix Momen updaing: adapively compuing marix M in () akes ( n(n+) ) flops. Each N samples, for each signal pair:. Compuaion of Momens: as described in [5] he number of muliplicaions and accumulaions necessary o compue he necessary momens are approximaely Kg(l 3 + l), where g = n(n ), as defined in Algorihm, is he number of signal pairs, l = n(n+) is he dimension of he momens marix and K» + p n is he number of sweeps. 5. Compuaion of GW E : using equaion (8) i would ake abou f =6flops. 6. Roaion: four flops. This makes (n+3)! (n )!! + (n+)n per ieraion plus less han p ( + n)( n(n ) n(n+) +( n(n+) ) + n 3 + n flops )[( n(n+) ) ]flops each N ieraions. This figures can be compared wih hose of oher adapive mehods, such as ademl, AROT [6] and EASI []. In [] auhors esimae he number of flops per ieraion for hese hree mehods obaining he following compuaional complexiies: C ademl = (8 + f )( + p n)g, C AROT = ( + f )( + p n)g, where f = 6 when using equaion (8), and C EASI = n 3 +3n + ln, where each nonlineariy elemen akes l flops (e.g., for cubic nonlineariies l =). An exra number of flops would have o be added in he normalized version. C ademl and C AROT do no include he number of flops in he whiening sage, so n 3 + n mus be added o hose complexiies figures. Hence, he compuaional burden of Algorihm is always higher han for he ademl, he AROT and he EASI mehods when N =. However, as N increases and for a reduced number of sources we can force he complexiy of Algorihm o be below he complexiy of he ademl and he AROT, and of he order of he EASI mehod. This is illusraed in Fig., where we display he number of flops per ieraion versus he number of sources needed for hese four adapive mehods for values of N =5and N =. We can see how for a number of independen sources equal or below n =7when N =5, or a number of independen sources equal or below n = when N =, he complexiy of Algorihm is lower han for hose of he ademl and he AROT mehods. I can also be observed in Fig. ha, when N = or higher and n» 8, he compuaional burden of Algorihm is of he order of ha for he EASI, leading however o a beer soluion, as described in he nex secion. Flops.5 x Number of sources Fig.. Compuaional complexiy as a funcion of he number of sources for ademl(ffi), AROT(), EASI(Π) and algorihm wih N=5 (solid) and N= (dashed). 5. EXPERIMENTAL RESULTS In he presen secion he performance of he iniialized JO for he GWE esimaes (AIJO-GWE) is o be illusraed in a variey of simulaions. We will use he esimaes in (8), herefore in he following he AIJO-GWE for his esimae will be referred as AISICA. In he experimens we compare AISICA mehod wih oher adapive procedures: he ademl [] and he EASI []. The adapaion coefficien was seleced for he whiening sage and he EASI mehod as = :5. The learning rae in his paper was se o ν =:. The soluion of he AISICA mehod was calculaed a each sample, i.e. N =. The performance index Q = i= j= jp ij j max k jp ik j + j= i= jp ij j n (5) max k jp kj j

5 where P =(p ij )= BA, is used as a measure of separaion and for he sake of comparing he performance of each mehod. As firs experimen we face he mixure of hree independen sources: a binary sequence, a uniformly disribued process and a sinusoid wih random frequency and phase. A random regular 3 3 marix whose enries are uniformly disribued in [ ; ] is chosen on each realizaion. We display in Fig. he rajecories of he global sysem enries obained by (a) AISICA, (b) ademl and (c) EASI, respecively, averaged over independen realizaions. Modulus of global marix elemens (a) (b) (c) Ieraion number Fig.. Modulus of global sysem coefficiens averaged over mixure realizaions for (a) AISICA mehod, (b) ademl mehod and (c) EASI mehod. Number of sources: n=3 (uniform, binary and sinusoid). In Fig. 5(a) we display he performance index of he hree mehods. We can see ha he performance index for he AISICA mehod is always lower han hose for ademl and EASI, and also he saionary sae is reached faser han in he oher wo cases. As second experimen, o see he he performance of he hree algorihms as he number of sources increases, we face he mixure of eigh independen sources. All of hem bu wo were uniformly disribued process, he oher wo sources were a binary sequence and a sinusoid wih random frequency and phase. The values of he parameers in his case are he same ha in he firs experimen. We can see a Fig. 5(b) he evoluion of he performance index for all hree mehods and see again ha he one for he AISICA is always lower han hose for ademl and EASI cases, also he saionary sae is reached faser han in he oher wo mehods. The resuls displayed in Fig. 5(b) also illusrae he slow convergence speed of ademl algorihm. Perform. Index (db) Perform. Index (db) Perform. Index (db) Ieraion number (a) Ieraion number (b) Ieraion number (c) Fig. 3. Performance Index for he AISICA(), ademl(ffi) and EASI(Π) mehods for (a) n=3 (uniform, binary and sinusoid), (b) n=8 (6 uniform, binary and sinusoid) and (c) n=8 (5 uniform, Laplacian, binary and sinusoid). We have shown in he previous secion ha he compuaional complexiy of he AISICA algorihm is higher han ha of he EASI mehod. However, i is imporan o poin ou ha AISICA has no ha limiaion concerning

6 which pdf he source signal have, ha is found in he EASI case. To consider ha, we replaced in he previous simulaion one of he uniformly disribued sources and inroduce a Laplacian disribued one. In Fig.5(c) we display he performance index of he hree mehods under he described circumsances. The index for he EASI mehod reaches a higher value in he saionary sae in his case han in Fig.5(b), since EASI is no able o separae correcly all he sources, while he index for he AISICA reaches almos he same values as in Fig.5(b). 6. CONCLUSIONS Based on he ademl mehod, we have propose a novel algorihm o adapively compue he soluion o e ICA problem. The mehod is based upon he Jacobi Opimizaion algorihm and is available for a wide se of fourh order ME conras. The main advanage of he mehod is ha he learning of he sysem and he compuaion of he ICA soluion have been decoupled ino wo differen rouines. A each sample insan, we updae a marix wih momens of he inpus. On he oher hand, he ICA soluion may be compued by using his marix a any ime. Wih his new scheme we provide a beer performance han he ademl or he EASI algorihms as shown in he experimens. Regarding he compuaional burden, he complexiy of his novel mehod can be reduced o he order of ha of he EASI. Besides, he mehod is available for virually any source probabiliy densiy funcion. 7. REFERENCES [] P. Comon, Independen componen analysis, a new concep?, Signal Processing, vol. 36, no. 3, pp. 87 3, Apr. 99. [] J. F. Cardoso, Blind signal separaion: Saisical principles, Proceedings of he IEEE, vol. 86, no., pp. 9 5, Oc 998. [3] J. F. Cardoso, High-order conrass for independen componen analysis, Neural Compuaion, vol., no., pp. 57 9, Jan 999. [] Vicene Zarzoso and Asoke K. Nandi, Adapive blind source separaion for virually any source probabiliy densiy funcion, IEEE Transacions on Signal Processing, vol. 8, no., pp , February. [5] Vicene Zarzoso and Asoke K. Nandi, Blind separaion of independen sources for virually any source probabiliy densiy funcion, IEEE Transacions on Signal Processing, vol. 7, no. 9, pp. 9 3, Sepember 999. [6] Vicene Zarzoso, Frank Herrmann, and Asoke K. Nandi, Weighed closed-form esimaors for blind source separaion, in h Inernaional Workshop on Saisical Signal Processing, Singapore, Augus. [7] M. Ghogho, A. Swami, and T. Durrani, Approximae maximum likelihood blind source separaion wih arbirary source pdfs, in IEEE Workshop on Saisical Signal and Array Processing (SSAP ), Pocono Manor Inn, Pennsylvania, Aug. [8] F. Harroy and J.-L Lacoume, Maximum likelihood esimaors and cramer-rao bounds in source separaion, Signal Processing, vol. 55, no., pp , 996. [9] E. Moreau and O. Macchi, Higher order conras for self-adapive source separaion, Inernaional Journal of Adapive Conrol and Signal Processing, vol., no., pp. 9 6, Jan 996. [] F. Herrmann and A.K. Nandi, Blind separaion of linear insananeous mixure using close forms esimaors, Signal Processing, vol. 8, pp ,. [] P. Comon and E. Moreau, Improved conras dedicaed o blind separaion in communicaions, in IEEE Inernaional Conference on Acousics, Speach and Signal Processing, Munich, Germany, 997, vol. V, pp [] J. F. Cardoso and A. Souloumiac, Blind beamforming for non gaussian signals, Proceedings IEE F, vol., no. 6, pp , Dec 993. [3] J.J. Murillo-Fuenes and F. González-Serrano, Independen componen analysis wih sinusoidal fouhorder conras, in Inernaional Conference on Audio, Speech and Signal Processing, Sal Lake Ciy, USA, May, vol. V, pp [] J. F. Cardoso and B. H. Laheld, Equivarian adapive source separaion, IEEE Transacions on Signal Processing, vol., no., pp , Dec 996. [5] Rafael Boloix-Torosa Juan J. Murillo-Fuenes and Francisco J. González-Serrano, Iniialized jacobi opimizaion in independen componen analysis, in ICA3, Submied, Nara, Japan, April 3. [6] P. Comon, Separaion of sochasic processes, in Workshop Higher Order Specral Anal., Va, CO, June 989.