KALMAN FILTER AND NARX NEURAL NETWORK FOR ROBOT VISION BASED HUMAN TRACKING UDC ( KALMAN), ( ), (007.2)

Transcription

1 FACTA UNIERITATI eries: Auomaic Conrol and Roboics ol. 2 N o 23 pp KALMAN FILTER AND NARX NEURAL NETWORK FOR ROBOT IION BAED HUMAN TRACKING UDC (4.42KALMAN) ( ) (7.2) Emina Perović Žaro Ćojbašić Danijela Risić-Durran 2 lasimir Niolić Ivan Ćirić rđan Maić Universiy of Niš Faculy of Mechanical Engineering Deparmen of Mecharonics and Conrol Niš Republic of erbia 2 Universiy of Bremen Insiue of Auomaion Germany Absrac. Tracing human is an imporan and challenging problem in video-based inelligen robo sysems. In his paper a vision-based human racing sysem is supposed o provide sensor inpu for vision-based conrol of a mobile robo ha wors in a eam helping he human co-worer. A comparison beween NARX neural newor and Kalman filer in solving he predicion problem of human racing in robo vision is presened. Afer collecing video daa from a robo simulaion resuls obained from he Kalman filer model are used o compare wih he simulaion resuls obained from he NARX Neural newor. Key words: robo vision Kalman filer neural newors human racing. INTRODUCTION Human racing and recogniion of human acions in he real world environmen is a very imporan percepion as in roboics. Following a specific person is a significan as for service robos. A person-following robo finds a specified person using sensory inpu and follows he person in order o provide various services o humans or o archive differen ass depending on he applicaion []. Neural newors have been widely used for roboic conrol and human racing because of heir abiliy o approximae arbirary linear or non-linear sysems in a compac se. The conrol mehods based on neural newors were demonsraed hrough many heoreical and indusrial soluions [2]. Received March 23 Corresponding auhor: Emina Perović Deparmen of Mecharonics and Conrol Universiy of Niš Faculy of Mechanical Engineering Niš Republic of erbia emina@masfa.ni.ac.rs Acnowledgemens. This research in his paper was suppored by he Minisry of Educaion and cience of Republic of erbia and DAAD hrough he projec "A Novel Approach for Human Deecion and Tracing in Roboics". The research was suppored by he projec "Research and Developmen of New Generaion of Highly Energy Efficien Wind Turbines" (No. 355/2) funded by he Minisry of Educaion and cience of Republic of erbia.

2 44 E. PETROIĆ Ž. ĆOJBAŠIĆ D. RITIĆ-DURRANT. NIKOLIĆ I. ĆIRIĆ. MATIĆ Many researchers have applied arificial neural newors ino effecive human racing sysems and navigaional sysems. An efficien human acion recogniion using NN is presened in [2]. Convolued NN is used for human acion recogniion in videos. The video frames are reaed and his image is used by NN o recognize acions in he individual frame. Also anoher efficien racing sysem using an algorihm for racing a human using NN is developed in [3]. The muli layer feed forward NN is used o esimae he disance of he person from he robo's camera and ha esimaion is combined and used from Kalman filer o rac he posiion of he person over ime. Tracing humans by mobile robos using he Kalman filer is presened in many papers and differen soluions for racing wih robos are repored. ome applicaions use robo's range sensors considering people as moving eniies [4 5]. On he oher hand some approaches use robo's onboard camera o deec people [6 7]. Ohers mae use of muli-sensor sysems in which visual daa is combined wih oher robo's range sensors [8 9]. Exising approaches use color racing a mixure of feaures or sereo vision o idenify single persons and evenually heir gesures or shape conour and differen filering echniques o esimae he arge saes. Mobile robos using 2D or 3D daa by any classical Kalman filer can rac people. The sysem implemened in [] adops Kalman filers o rac people using cameras mouned on mobile robos. The echniques described in [] are designed o improve racing performance using Kalman filer and Opical flow. The approach presened in [2] uses a sereo-based camera o deec and rac persons using Exended Kalman filer. The sysem presened in [3] applies Kalman filer mehod o synchronize he vision based daa and he moion conrol of he robo o preven losing he raced arge due o robo moion. The approach described in [4] presens he self-uning Kalman filer echnique using echo sae newors for mobile robos. In his paper he vision-based human racing sysem should provide sensor inpu for vision-based conrol of a mobile robo which operaes in a eam helping he human coworer. The robo vision sysem for human racing for he considered robo woring scenarios has o be able o deec he human in camera images. Also i should calculae he 3D coordinaes of human's cener of mass in camera coordinae sysem (x y z) and o calculae he disance o he human and o rac he human eeping a consan disance beween he robo and human [5]. Experimens were conduced where a human waling owards he robo was capured by a Poin Grey Bumblebee sereo camera. The frame rae of he used camera a full resoluion is 5 fps (frames per second). Each pair of sereo frames was processed so o exrac informaion for sereo-vision based reconsrucion of human waling wih respec o camera coordinae sysem. This means ha he disance beween he human and he camera coordinae sysem aached o he lef sereo camera was calculaed for every momen of capuring he sereo images according o: D = x + y z () where x y and z are he 3D coordinaes of he human's cener of mass ha are calculaed using a sereo vision based mehod for 3D robo vision reconsrucion of human waling ha is described in deails in [6]. The colleced vision daa such as he disance beween he robo and he human in ime were used for geing he simulaion resuls of KF and NN. The sae of he person is presened in Caresian coordinaes x y z and hree-dimensional velociy. Assuming ha

3 Kalman Filer and NARX Neural Newor for Robo ision Based Human Tracing 45 he velociy is consan he normalized sae ransiion marix can be obained from basic inemaic equaions as: = + = (2) where is posiion of a human is velociy and is a sampling inerval. The comparison beween NARX neural newor and Kalman filer in solving he predicion problem of human racing in robo vision is presened. Afer collecing video daa from a robo simulaion resuls obained from he Kalman filer are used o compare wih he simulaion resuls obained from he NARX Neural newor. The presened mehods are assumed o provide a reliable inpu for he conrol of a mobile robo following human co-worer. 2. KALMAN FILTER In he Kalman Filer approach [7] i is presumed ha he following predefined models of moion and measuremen would characerize he behavior of a moving objec w Ax x + = (3) v Hx z + = (4) and he models can be represened in erms of a sae vecor x ha corresponds o image frame. In (3) A represens he sae ransiion marix which deermines he relaionship beween he presen sae x and he previous one x - and he marix H describes he relaionship beween he measuremen vecor z and he sae vecor x. The vecors w and v are noise erms which are assumed o be independen of each oher Gaussians wih zero mean and covariance marices ] [ T w w Q E = and ] [ T v v R E =. We assume ha he velociy of he raced human is consan beween he subsequen video frames so he sae vecor is simplified and does no include he acceleraion erm. This hree dimensional problem can be expressed in marix form. Kinemaical equaion is rewrien as marix: = z y x z y x z y x z y x (5) where he vecor represens measuremens on 3D posiion of he deeced human in frame and he vecor represen 3D velociies of he cener of mass of he segmened human region. The relaionship beween he sae vecor x and measuremen vecor z is defined by marix H. Given he fac ha he sae vecor is of lengh six and he measuremen vecor is of lengh hree he marix H mus be of lengh six by hree:

4 46 E. PETROIĆ Ž. ĆOJBAŠIĆ D. RITIĆ-DURRANT. NIKOLIĆ I. ĆIRIĆ. MATIĆ H = (6) The complee oupu of human deecion is he hree-dimensional vecor: = x y z (7) The presened Kalman filer based racing has wo sages: he predicion and he correcion sage. In he predicion sage he sae vecor x in he curren frame is prediced using he corresponding sae vecor in he previous frame via marix A. In he correcion sage he measuremen vecor in he curren frame is used o updae he sae esimae and he error covariance marix. The predicion sage can be expressed as follows: T x ˆ = Axˆ (8) T P = AP A + Q (9) where xˆ is he a priori esimae sae and P is he a priori esimae error covariance marix a frame. The correcion sage can be expressed hrough he following hree equaions: K T T = P H ( HP H + R) () x ˆ xˆ + K ( z Hxˆ ) () = P = ( I K H) P (2) where xˆ is he a poseriori esimae sae a frame given he measuremens up o ha ime P is he a poseriori esimae error covariance marix a frame indicaing he accuracy of he esimaed sae vecor and K is he Kalman gain. The racing process is recursive. The esimaes compued from () are used as he sae vecors on he righ hand side of (8) for he nex video frame +. This is exacly he prediced vecor. 3. THE NONLINEAR AUTOREGREIE NETWORK WITH EXOGENOU INPUT (NARX) The nonlinear auoregressive newor wih exogenous inpus (NARX) is a recurren dynamic newor wih feedbac connecions enclosing several layers of he newor. The NARX model is based on he linear ARX model which is commonly used in ime-series modeling. One can implemen he NARX model by using a feed forward neural newor o approximae he funcion f. The defining equaion for he NARX model is [89]: y()=f(y(-) y(-2) y(-n y )u()u(-) u(-n u )) (3) where u() and y()represen he inpu and oupu of he newor a ime and he funcion f is some nonlinear funcion in his case a neural newor. The nex value of he dependen oupu signal y() is regressed on previous values of he oupu signal and previous

5 Kalman Filer and NARX Neural Newor for Robo ision Based Human Tracing 47 values of an independen (exogenous) inpu signal. When he funcion can be approximaed by a Mulilayer Percepron he resuling sysem is called a NARX newor which is a recurren neural newor [8 9]. There are many applicaions of he NARX newor. I can be used as a predicor o predic he nex value of he inpu signal. I can also be used for nonlinear filering in which he arge oupu is a noise-free version of he inpu signal. The oupu of he NARX newor can be an esimae of he oupu of some nonlinear dynamic sysem ha is modeled. The oupu is feedbac o he inpu of he feed forward neural newor as par of he sandard NARX archiecure. Because he rue oupu is available during he raining of he newor i is possible o creae a series-parallel archiecure in which he rue oupu is used insead of feeding bac he esimaed oupu. This has wo advanages he firs is ha he inpu o he feed forward newor is more accurae and he second is ha he resuling newor has a purely feed forward archiecure and saic bac propagaion can be used for raining. To forecas he rajecory of xyz coordinaes in real ime for every prediced sae is esimaed based on he curren inpus in feed forward neural newors and he esimaion in NARX newor is done based on he curren inpus previous inpus and previous saes. In order for he parallel response o be accurae i is imporan ha he NARX newor is properly rained so ha he errors in he series-parallel configuraion are very small. Oherwise he accumulaion of error migh occur during esimaion. The purpose of his simulaion is o forecas he rajecory of xyz coordinaes in real ime. For he simulaions we used feed forward opology of he NARX NN. The NN had en hidden neurons and wo delays and is presened in Fig.. Fig. The used NARX NN opology wih weighs and biases a each node The raining was done by Levenberg-Marquard bac propagaion [8] and regression value was R= The newor also had addiional nodes so more weighing facors needed o be adjused and he raining of NARX newor needed more compuaional ime han i would be needed if convenional neural newor was used. 4. IMULATION REULT In his paper we presen a comparison beween NARX neural newor and Kalman filer in solving he predicion problem of human racing in robo vision. Afer collecing video daa from a robo simulaion resuls obained from he Kalman filer model are

6 48 E. PETROIĆ Ž. ĆOJBAŠIĆ D. RITIĆ-DURRANT. NIKOLIĆ I. ĆIRIĆ. MATIĆ used o compare wih he simulaion resuls obained from he NARX Neural newor. Kalman filer resuls are obained from a mobile robo plaform equipped wih vision sensors ha can be used as human co-warer in he examinaion of environmen. The robo is equipped wih sensors for environmen percepion and wih sensors for plaform navigaion as well [5]. One of he robo's woring scenarios concerns he vision based human racing as he robo wors as a ransporaion robo helping he human o carry conainers wih colleced samples from he invesigaed environmen. Experimens were conduced wih a human waling owards he robo. Robo's vision sensor is a Poin Grey Bumblebee sereo camera ha provides informaion for human racing. The camera is mouned on a rac locaed on he bac side of he robo mobile plaform. Figures 2 3 and 4 presened prediced and measured x y and z coordinaes of he raced human wih Kalman filer and NARX NN respecively. ome examples of processed images by NARX NN and Kalman filer wih superimposed exraced bounding boxes of human are shown in Figure 5. Afer reviewing hese resuls from he figures below we can see ha boh approaches made predicion accuraely and o some exen almos he same. In addiion boh approaches were able o reduce noise in measuremens and o predic sae when he robo's vision sysem did no give measuremens in some inervals of ime. However in case of Kalman filer we mus provide is dynamic model before racing because if he provided model is no he acual model he racing can fail easily wha is he main disadvanage of Kalman filer. Fig. 2 The prediced and he measured x coordinae of he raced human

7 Kalman Filer and NARX Neural Newor for Robo ision Based Human Tracing 49 Fig. 3 The prediced and he measured y coordinae of he raced human Fig. 4 The prediced and he measured z coordinae of he raced human

8 5 E. PETROIĆ Ž. ĆOJBAŠIĆ D. RITIĆ-DURRANT. NIKOLIĆ I. ĆIRIĆ. MATIĆ Fig. 5 KF prediced posiion (blue) and NN prediced posiion (red) 5. CONCLUION In his wor one soluion for human racing using NN is presened. Firs we described he NARX neural newor and is advanages in ime series predicion afer ha we presened he Kalman filer and is conribuion in solving he human racing problem. Finally we compared he resuls of human racing obained from he NN wih he resuls obained from he Kalman filer hrough simulaion. Also we have demonsraed he performance of racing sysem using he resuls obained from he NARX Neural newor which are based on he video sequences and compared wih he resuls obained from he Kalman filer. Experimenal resuls show us ha NARX Neural newors give good resuls elling us ha NN made predicions accuraely and reliably as well as he Kalman filer. On he oher hand NN can wor direcly wih raw inpus and daa. Using ha fac and characerisic of he NN ha hey are able o learn he dynamic model in real ime NN can be easily implemened in human racing algorihm in order o have more robus and reliable racing resul.

9 Kalman Filer and NARX Neural Newor for Robo ision Based Human Tracing 5 REFERENCE [] J. Borensein D. Thomas B. ighs L. Ojedo P. BAnole D. Fellars "Human leader and robo follower eam: correcing leader's posiion from follower's heading" in Proc. of he PIE Defense ecuriy and ensing Orlando Florida 2. [2]. Ji W. Xu M. Yang K. Yu "3D convoluional neural newors for human acion recogniion" in Proc. of he 27h Inernaional Conference on Machine Learning Haifa Israel 2. [3] G. Cielina M. Miladinovic D. Hammarin L. Goranson A. Lilienhal T. Duce "Appearance based racing of persons wih an omnidirecional vision sensor" in Proc. of Compuer ision and Paern Recogniion Worshop 23. [Online]. Available: hp://dx.doi.org/.9/cprw [4] J. Bobru D. Ausin "Laser moion deecion and hypohesis racing from a mobile plaform" in Proc. of he Ausralian Conference on Roboics & Auomaion pp. 24. [5] P. Urcola L. Monano "Adaping robo eam behavior from ineracion wih a group of people" in Proc. Inelligen Robos and ysems pp [Online]. Available: hp://dx.doi.org/.9/ IRO [6] D. Beymer K. Konolige "Tracing people from a mobile plaform" Experimenal Roboics III pp [Online]. Available: hp://dx.doi.org/.7/ _2 [7] M. Bure W. Brin "Gain-scheduling conrol of a monocular vision-based human-following robo" in Proc. of 8 h World Congress of he Inernaional Federaion of Auomaic Conrol Milano Ialy 2 pp [8] N. Belloo H. Hu "A ban of unscened Kalman filers for mulimodal human percepion wih mobile service robos" Inernaional Journal of ocial Roboics vol. 2 no. 2 pp [Online]. Available: hp://dx.doi.org/.7/s x [9] J. A. Corrales F. A. Candelas F. Torres "Kalman filering for sensor fusion in a human racing sysem" Inech pp [] N. Belloo H. Hu "Mulisensor-based human deecion and racing for mobile service robos" IEEE Transacions on sysems Man and Cyberneics-par B: Cyberneics vol. 39 no. pp [Online]. Available: hp://dx.doi.org/.9/tmcb [] R. E. Kalman "A new approach o linear filering and predicion problems" Journal of Basic Engineering vol. 82 no. pp [2] J. aae J. Miura "Robus sereo-based person deecion and racing for a person following robo" in Proc. of he Worshop on People Deecion and Tracing Kobe Japan May 29. [3] Y. Zheng Y Meng "Real-ime people racing and following using a vision-conrolled mobile robo" Robo ision: New Research 29. [4] C.Y. Tsai X. Duoi K.T. ong H.. Brussel M. Nuin "Robus face racing conrol of a mobile robo using self-uning Kalman filer and echo sae newor" Asian Journal of Conrol vol. 2 no. 4 pp [Online]. Available: hp://dx.doi.org/.2/asjc.24 [5] E. Perović D. Risić-Durran A. Leu. Niolić "A novel approach o human racing for roboic Follower" in Proc. XI Inernaional AUM Conference on ysems Auomaic Conrol and Measuremens Niš erbia 22 pp [6] Leu D. Aieanu A. Gräser "A novel sereo camera based collision warning sysem for auomoive applicaions" in Proc of 6h IEEE Inernaional ymposium on applied Compuaional Inelligence and Informaics ACI 2 Timisoara Romania 2 pp [Online]. Available: hp://dx.doi.org/.9/aci [7]. Chen "Kalman filer for robo vision: a survey" IEEE Transacions on Indusrial Elecronics vol. 59 no. pp [Online]. Available: hp://dx.doi.org/.9/tie [8] Pelliccioni T. Tirabassi "Air dispersion model and neural newor: a new perspecive for inegraed models in he simulaion of complex siuaions" Environmenal Modeling & ofware vol. 2 no. 4 pp [Online]. Available: hp://dx.doi.org/.6/j.envsof [9] I. Ćirić Ž. Ćojbašić. Niolić P. Živović M. Tomić "Air qualiy esimaion by compuaional inelligence mehodologies" Thermal cience vol. 6 iss. 2 pp [Online]. Available: hp://dx.doi.org/.2298/tci25386c