Comparison of Gradient descent method, Kalman Filtering and decoupled Kalman in training Neural Networks used for fingerprint-based positioning

Transcription

1 Comparson of Gradent descent method, Kalman lterng and decoupled Kalman n tranng Neural Networs used for fngerprnt-based postonng Claude Mbusa Taenga, Koteswara Rao Anne, K Kyamaya, Jean Chamberlan Chedou Insttute of Communcatons Engneerng, Unversty of annover Appelstr 9A D-367, annover Germany {taenga, raoanne, yandogh, chedou}@antun-hannoverde Abstract The success of neural networ archtectures depends heavly on the avalablty of effectve learnng algorthms Radal bass functon (RB) neural networs provde attractve possbltes for solvng sgnal processng and pattern classfcaton problems Gradent descent tranng (GD) of RB networs has proven to be much more effectve than more conventonal methods owever, gradent descent tranng can be computatonally expensve and ts learnng speed s very slow Ths paper compares (GD) to the method based on ether Kalman flterng (K) or decoupled Kalman flter (DEK) These new methods prove to be qucer than gradent descent tranng whle stll provdng good performance at the same level of effectveness as they are used n fngerprnt-based postonng Keywords Neural Networ tranng, Kalman lter, Gradent Descent, Decoupled Kalman lter, Postonng I INTRODUCTION The problem of provdng a relable and accurate poston of a moble staton (MS) n wreless communcaton systems has attracted a lot of attenton n recent years Postonng systems can be roughly classfed nto three maor categores: systems usng sgnal strength measurements, systems usng ether tme of arrval and angle of arrval of a rado sgnal, and systems usng dead reconng technques The frst two categores can be called rado-locaton methods as they rely on the propagaton propertes of rado sgnals Ths paper addresses a novel approach whle applyng sgnal strength measurements for the postonng of a GSMbased moble staton It uses rado sgnal strengths from the servng and neghborng base statons, whch are contnuously measured n the moble staton These are appled to a prevously traned artfcal neural networ for postonng ngerprnt based Postonng usng neural networs can be made usng ether classfcaton or functon approxmaton When usng classfcaton, the area of nterest s dvded nto small sectons and the dentfcaton of the secton n whch the moble staton can be found s the tas performed by the classfcaton or the second case, a functon approxmates the relatonshp between the receved sgnal strengths and the dstance between the moble staton and the antenna Trlateraton can subsequently be made when at least three dstances to nown base staton locatons are nown Ths s an ndrect postonng urthermore, the receved sgnal strength nput can also be used to drectly model the two-dmensonal coordnates of the moble staton Ths method s referred to as drect postonng [] Par and Sanberg [] have proven that Radal Bass uncton (RB) neural networs wth one layer of RB functons are capable of unversal approxmaton or ths reason ths paper uses a RB archtecture wth one hdden layer of neurons n order to compare three ey tranng approaches: Gradent Descent (GD), Extented Kalman lter (EK), and Decoupled Extended Kalman lter or fngerprnt based postonng, a method that s used n ths paper, the poston of the moble system s automatcally found nowng the set of sgnal strength for that pont GPS postons are used as references durng the tranng phase [3] or ths purpose, one need an element or a system capable to relate the receved GSM power levels of the surroundng cells n a pont and the poston of ths pont gven by GPS Ths element has to acqure enough ntellgence and abstracton of such a not obvous relaton, and use t to mae future poston predctons related to gven sgnal strenghs measurements A neural networ (NN) s used for that purpose ΙΙ DESCRIPTION O TE TREE TRAINING METODS USED: DG, EK, DK There are several algorthms avalable for tranng the weghts of a neural networ [4] [5] [6] Most of them are based on computaton of the gradent of an output error measured wth respect to networ weghts Recently, several authors [7][8][9][] have noted that the Kalman lters (eg EK) can also be used for tranng networs to perform the desred nput-output mappngs In ths paper we use the RB neural networ archtecture as t provdes attractve possbltes for solvng sgnal processng and pattern classfcaton problems [] An archtecture of one hdden-layer of RBs s used to compare the three methods (GD,EK and DEK) 446

2 The RB NN can be descrbed as follows The nput data s (are) represented by x n g, beng passed drectly to hdden layer Suppose there are c neurons n the hdden layer Each of the c neurons n the hdden layer apples an actvaton functon, whch s a functon of the Eucldean dstance (e the square of the Eucldean norm of the two vectors) between the nput and the prototype vectors v, as shown n g There are many choces for g(), functon n the hdden layer of RB NN The most common choce s a Gaussan functon of the form ( v / β ) g( ν ) e () where g (ν ) s the Gaussan functon, β s a real constant, [] Another choce s the nverse multquadratc functon / g ( v) ( v + β ) () where β s a real constant [] A further choce of the g() functon s /( p) g( v) [ g ( v) ] (3) where g ( v) av + b (4) g s called generator functon, p s a real number greater than, a> and b, [6] If a and p3, the hdden layer functon s reduced to the nverse multquadratc functon One output of the RB n g can be wrtten as follows: w w w c w w wc x v ) y (5) wn wn wnc x vc ) or all the outputs x v ) xm v ) y ym W (6) x vc ) xm vc ) Thus, let s wrte : Y W [] (7) In the followng the three dfferent tranng polces are descrbed n detal A Gradent Descent Tranng Method To use the gradent descent to mnmze the tranng error, one does defne an error functon, Y Y (8) E where Y s the matrx of desred values for the RB output, and s the square of the roebnus norm of a matrx, whch s equal to the sum of the squares of the elements of the matrx It has been shown; see Ref [], whch s gven n ths case by: w v where M ( y y ) h ( n), M n ' g ( v )( v) y (9) ( y y ) w ( c) () s the element n the th row and th column of the Y matrx of Eq7 and y s the correspondng element n the Y matrx The RB can be optmzed by performng the followng updates of weghts ( w ) and prototypes ( v ): w w η ( n), () w v η ( c) () v v where η s the step sze of the gradent descent method Ths optmzaton stops when w and v reach local mnma [] B Extended Kalman lter (EK) Tranng Method The Kalman lter s also used to tran a general multnput, mut-output RB networs or lnear dynamc systems wth whte nose process and whte measurement nose, the Kalman flter s nown to be an optmal estmator [] or nonlnear systems wth colored nose, the Kalman flter can be extended by lnearzng the system around the current parameter estmates [6, ] There are several algorthms avalable for tranng the weghts of the NN Most of them are based on computaton of the gradent of an output error measured wth respect to the networ weghts Recently, several authors [4, 9] have noted that the extended Kalman flter can also be used for the purpose of tranng networs to perform desred nput-output mappngs Assume the state space model below, []: x + + f (, ) w (3) y h(, x ) + v (4) 447

3 where x s the state of the system, y s the measurement model, w and v are ndependent, zeromean, Gaussan nose processes of the covarance matrces Q and R respectvely The frst step of EK s computng the lnearzed state matrces : f (, x) + x x (5) h (, x ) (6) x x Once matrces +, and are evaluated, they are then used n frst order Taylor approxmaton of nonlnear functons (, ) ( x, ) + +, ( x, x ), (7) (, ) ( x, ) + +, ( x, ), (8) The tranng problem usng Kalman flter theory can now be descrbed as fndng the mnmum mean-squared error estmate of the state x usng all observed data When assumng that w+ w + ω s the state of the neural networ, and y h ( w, u, v ) + v the observaton or measurement equaton whch represents the networ s desred response vector y as a nonlnear functon of nput vector u, the weght parameter vector w, and for the RB the prototype vector parameter v The soluton to the tranng problem s gven by the followng recurson [] : T A [ R + P K ], (9) K P A, () w w + + K ξ, () T P + P + K K P + Q () Ths Kalman recurson process can be explaned wth followng words An nput tranng pattern u s propagated through the networ to produce an output vector y The dervatve matrx s obtaned, then te Kalman gan matrx s computed accordng to Eq, Ths step nclude the computaton of the global scalng matrx A The networ weghts vector s updated usng the Kalman gan matrx, the error vectorξ, and the current value of the weght vector as n Eq At the end, the approxmate error covarance matrx s updated as n Eq w In order to apply the optmzaton problem n a form sutable for Kalman lterng n the case of a RB NN, we let the elements of the weght matrx (w) and the elements of the prototypes (v) consttute the state of the nonlnear system And the output of the RB networ consttutes the output of the nonlnear system The state of the nonlnear system model s represented by X [ w wnv v c ], [] The computatonal effort of Kalman s n the order of O [( AB) ], where A s the dmenson of the output dynamc system and B s the number of parameters In the case of concern n ths paper, there are nm outputs and [ n ( c+ ) + mc], e, [ n ( c+) ] weghts and mcprototypes, where n s the dmenson of the RB output, M s the number of tranng samples, c s the number of prototypes (v) --see g, and m s the dmenson of the RB nput Therefore, the computatonal expense of Kalman flter s n the order of O ( nm[ n( c + ) + mc] ) [] C Decoupled Extended Kalman flter (DEK)Tranng Method The classcal dsadvantage of Extended Kalman lter s ts computatonal expense, whch s the obstacle for ts use for larger networs Thus, smplfed varants must be found, that preserve the most useful property of the EK whle requrng less computaton per tme step [3] The parameter-based DEK algorthm s derved from EK by assumng that the teratons between certan weght estmates can be gnored Ths smplfcaton ntroduces many zeros nto the matrx P If the weghts are decoupled n a way such that the weght groups become mutually exclusve of one another, then P can be arranged nto a bloc-dagonal form Let s g refers to the number of such weght groups Then, for group, the vector w refers to the estmated weght parameters, s the sub-matrx of dervatves of networ outputs wth respect to the th group s weghts, p s the weght group s approxmate error covarance matrx, and K s ts Kalman gan matrx The DEK for the th weght group s gven, see [] The computatonal tranng expense of Kalman flter s reduced n the order of O ( nm[ ( c + ) + ( mc) ]), [] The rato between the computatonal tranng expense of the EK and that of the DEK s n the order of: [ n( c+ ) + mc] o( ) ( c+ ) + ( mc) 448

4 parameters for DEK ( P o, Q, R ) were ntalzed n a smlar way [] IV SIMULATION RESULTS gure Radal Bass uncton ( RB) ΙΙΙ SYSTEM DESCRIPTION AND EXPERIMENTAL SETTING Sgnal strength measurements n a GSM moble termnal are the nput data used to tran a RB neural networ usng one of three algorthms (GD, EK, DEK), whereby a target poston (or sector) s gven for every pont Durng the tranng phase, the neural networ does realze a mappng between the sgnal strengths and target postons (sector) for all sample data provded In the testng phase, provdng a set receved sgnal strengths at the nput of the NN, the poston (Sector) of the current moble staton s predcted at the output or the experments conducted n ths paper for llustraton, we do use measurement data collected on one street of the town anover (Schnederberg Str) n Germany The street has a total length of 45 meters Whle movng wth a constant speed, RSSI values were beng collected every 5 seconds Thus, 5 data ponts could be recorded Each data pont contans the RSSI values from the 4 strongest neghborng cells The classfcaton method s used for the NN tranng The street s dvded nto 5 equal parts (segments) Thus, every segment has a length of 3 meters and does contan successve of the collected data ponts The NN used n the experments, conssted of four nput vectors (correspondng to the RSSI measurement for the 4 strongest cells) and 5 outputs, each output correspondng to one of the 5 sectors The number of RB functons has been vared from to 5 The learnng rate n GD s taen equal to, n the frst experment where by a number of teraton s counted for every fxed number of neurons n the hdden layer, as ths has guaranted a monotonc reducton of the error durng the tranng process The RB networ were traned usng the hdden layer functon of Eq3 wth the lnear generator functon of Eq4 The Kalman flter parameters of Eqs9- have been ntalzed wth P o 4I, Q 4I, and R 4I, where I s the dentty matrx of approprate dmensons the g does present a comparson of the postonng performance of NN traned usng the three dfferent schemes The three tranng algorthms (GD, EK and DEK) were termnated when the error functon of Eq8 decreased by less than the gven tranng error The number of neurons n the hdden layer was vared It appeared that the methods based on Kalman lterng (EK, DK) are provdng less errors or example, wth the postonng quadratc average error wth Kalman method s less than 4 meters wth a probablty of 67% or GD, however, the postonng quadratc average error s up to 65 meters wth a probablty of 67% The dfference n accuracy between EK and GD becomes less, as the number of neurons n the hdden layer s ncreased The EK and DEK are provdng the same accuracy as they are both based on Kalman lterng, see gs-3 Snce the DEK algorthm s n prncple derved from EK n that t s assumed that the connectons between certan weght estmates can be gnored, t consequently requres fewer operatons n one teraton (f compared to EK) If one does record the tme needed for the tranng process, the dfference n terms of tranng effort between EK and DEK wll become clearer, especally f the NN s large (a large number of parameters: nputs, output, weghts, prototypes The DK method s generally preferred because of ts savng of tranng tme or g3, for example, the tranng tme could be recorded and s used here for comparson ollowng parameter settng has been used: a RB NN of 3 neurons, wth 4 nputs vectors, each havng 5 samples; and 5 ouputs (classfcaton ones) The computer platform used s a Pentum IV ( GZ and 5 MB RAM) The EK tranng requred mnutes, whereas the DEK ones needed only 8 mnutes, both for teratons g4 compares the number of teratons to converge the tranng error up to a fxed threshold It s seen that methods based on Kalman flterng do converge n fewer teratons compared to the GD, provded a gven tranng error and for a fxed number of neurons n the hdden layer V CONCLUSION Ths paper has compared a well-nown Gradent descent method tranng of a NN to the ones based on Kalman flterng (EK and DEK) The applcaton scenaro n ths wor s a fngerprnt postonng usng GSM RSSI 449

5 data The experments conducted n ths wor have shown that the Kalman flterng based tranng of the NNs does lead to a better postonng performance whle requrng the lowest tranng effort The computatonal savngs acheved by the DEK method compared to EK wll be more sgnfcant for cases where large NN are needed REERENCES [] Z Salcc and E Chan, "Moble staton postonng usng GSM cellular phone and artfcal neural networs," Wreless Personal Communcatons, vol 4, pp 35-54, [] J Par and I W Sanberg, "Unversal approxmaton usng radal-bass functon networs," Neural Computaton, vol 3, pp 46-57, 99 [3] K Kyamaya, DOM-Der orenterte Mensch: SAKER VERLAG, 3 [4] S Sn and R D gueredo, "Effcent learnng procedures for optmal nterpolatve nets," Neural Networs, vol 6, pp 99-3, 993 [5] R Duro and J Reyes, "Dscrete-tme bacpropagaton for tranng synaptc delay-based artfcal neural networs," IEEE Trans on Neural Networs, vol, pp , 999 [6] M Vdyasagar, Learnng and generalzaton wth applcatons to Neural networs, second edton ed: Sprnger, 997 [7] G V Pusorus and L A eldamp, "Neurocontrol of nonlnear dynamcal systems wth Kalman flter traned recurrent networs," IEEE Trans Neural Networs, vol 5, pp 79-97, 994 [8] M Brgmeer, "A fully Kalman-traned radal bass functon networ for nonlnear speech modelng," presented at IEEE Internatonal Conference on Neural Networs, 995 [9] R J Wllams, "Tranng recurrent Networs usng the extended Kalman lter," presented at Internatonal Jont Conference on Neural Networs, 99 [] J Sum, C Leung, G Young, and W Kan, "On the Kalman flterng method n neural networ tranng and prunng," IEEE Transactons on Neural Networs, vol, pp 6-66, 999 [] D Smon, "Tranng Radal Bass Neural Networs wth the Extended Kalman lter," Neurocomputng, vol 48, pp , [] S ayn, Kalman flterng and neural networs: John Wley & Sons, nc, [3] S ayn, Neural networs, a comprehensve foundaton, nd edton ed: Prentce all, 999 CP(%)- Test Cumulatve Probablty uncton (Tran Error, and Number of neurons n hdden layer5) GD-CP 6 EK-CP 5 DK-CP Error(m) gure Cumulatve Probablty unctons (CP) of the postonng error for NNs traned wth the three schemes CP(%) -Test Error (m) GD-CP EK-CP DK-CP gure 3 CP comparson for more neurons n the hdden layer (Tranng error ; 3 neurons) Log Number of Iteratons 3 Trang error, GD EK DK Number of Neurons n RB gure 4 Comparson of the number of teratons n tranng (GD,EK,DK) the neural networs whle varyng the number of neurons n the hdden layer 45