NEURAL NETWORK APPROACH TO BAYESIAN BACKGROUND MODELING FOR VIDEO OBJECT SEGMENTATION

Transcription

1 NEURAL NETWORK APPROACH TO BAYESIAN BACKGROUND MODELING FOR VIDEO OBJECT SEGMENTATION Dubravko Ćulibrk, Oge Marques, Daniel Socek, Hari Kalva and Borko Furh Deparmen of Compuer Science and Engineering Florida Alanic Universiy, 777 Glades Rd., Boca Raon, FL 33431,USA Keywords: Absrac: Video, Objec segmenaion, Background modelling, Bayesian modelling, Neural Neworks. Objec segmenaion from a video sream is an essenial ask in video processing and forms he foundaion of scene undersanding, objec-based video encoding (e.g. MPEG4), various surveillance and 2D o pseudo 3D conversion applicaions. The ask is difficul and exacerbaed by he advances in video capure and sorage (e.g. HDTV, QuadHDTV). Increased resoluion of he sequences requires developmen of new, more efficien algorihms for objec deecion and segmenaion. The paper presens a novel neural nework based approach o background modelling for moion based objec segmenaion in video sequences. The proposed approach is designed o enable efficien, highly-parallelized hardware implemenaion. Such a sysem would be able o achieve real ime segmenaion of high-resoluion sequences. 1 INTRODUCTION Objec deecion and segmenaion from a video sream are essenial asks in video processing and form he foundaion of scene undersanding, objecbased video encoding (e.g. MPEG4), various surveillance applicaions, as well as he emerging research ino 2D o pseudo 3D video conversion. The ask is difficul and exacerbaed by he advances in video capure and sorage (e.g. HDTV, QuadHDTV). Increased complexiy of he sequences requires developmen of new, more efficien algorihms for objec deecion and segmenaion. Commonly used approach o exrac foreground objecs from he image sequence is hrough background suppression [39-41], when he video is grabbed from a saionary camera. However, he ask becomes difficul when he background conains shadows and moving objecs, and undergoes illuminaion changes. Significan scienific effor has been spen on he developmen of adapive models of background and segmenaion echniques. A number of proposed echniques are able o achieve real-ime processing of comparaively small video formas(e.g. 120x160 pixels, CIF resoluion) and, usually, a somewha reduced frame raes. I is unlikely, however, ha he exisen objec deecion approaches will be able o efficienly cope wih he increase in he resoluion of video sequences. The developmen of an objec deecion approach, which would allow for efficien hardware implemenaion and objec deecion in real-ime for high-complexiy video sequences (in erms of he frame size as well as background changes), is he focus of his paper. A new neural nework srucure is proposed, o serve boh as an adapive Bayesian model of he background in a video sequence and an algorihm for background subracion and foreground objec deecion and segmenaion. The res of he paper is organized as follows: Nex secion provides a survey of relaed published work. The hird secion describes he main aspecs of he proposed approach. The fourh is dedicaed o he presenaion of simulaion resuls. The las secion holds he conclusions and some direcions for fuure work. 2 RELATED WORK Some of he early objec segmenaion mehods dealing wih he insances of non-saionary background were based on smoohing he colour of a background pixel over ime using differen filering

2 echniques such as Kalman filers[21][22], or Gabor filers [20] o creae a reference background frame. The reference frame is a model of background, which is consanly updaed and used o segmen he foreground objecs by subracing i from he curren frame of he inpu sequence. However, since hese mehods are based on he mos resricive assumpion ha movemens of he background are much slower han hose of he objecs o be segmened, hey are no paricularly effecive for sequences wih high-frequency background changes. Slighly beer resuls were repored for echniques ha rely on a Gaussian -based saisical model whose parameers are recursively updaed in order o follow gradual background changes wihin he video sequence[23]. More recenly, his model was significanly improved by employing a Mixure of Gaussians (MoG), where he values of he pixels from background objecs are described by muliple Gaussian disribuions[24][26]27]. This model was considered promising since i showed good foreground objec segmenaion resuls for many oudoor sequences. However, weaker resuls were repored [28] for video sequences conaining nonperiodical background changes (e.g. due o waves and waer surface illuminaion, cloud shadows, and similar phenomena). These models are parameric in he sense ha hey incorporae underlying assumpions abou he PDFs hey are rying o esimae. In 2003, Li e al. proposed a mehod for foreground objec deecion employing a Bayes decision framework [28][29]. The mehod has shown promising experimenal objec segmenaion resuls even for he sequences conaining complex variaions and non-periodical movemens in he background. In addiion o he generic naure of he algorihm where no a priori assumpions abou he scene are necessary, he auhors claim ha heir algorihm can handle a hroughpu of abou 15 fps for CIF video resoluion. The approach is specific in he fac ha i uses a saisical model of for he changes beween he curren frame and he reference background image mainained by applying an Infinie Impulse Response (IIR) filer o he sequence. A Bayesian classifier is han used o classify he changes, deeced hrough frame differencing beween he curren frame and he reference frame, as perinen o background objecs or foreground objecs. The saisical model is nonparameric since i does no impose any specific shape o he PDFs learned. However, for reasons of efficiency and improving resuls he auhors applied binning of he feaures and assigned single probabiliy o each bin, leading o a discree represenaion of PDFs. The model is general in erms of feaures exraced from he sequence and hey experimened wih he use of differen feaures. The resuls of hese experimens are repored in [29]. Recenly he approach of Li e al. has been adoped and exended o creae a par of a surveillance sysem inended for mariime environmens [30]. The resuls in his domain have been improved by alering he frame differencing sep of he algorihm as well as using a color-based sill image segmenaion insead of he morphological operaions in he pos-processing of he background-subracion resuls. While he use of Bayesian models as basis for background subracion is no new, i has been limied by he fac ha hey are general in he sense ha hey impose no consrains on he shape of he esimaed probabiliy densiy funcion. This ypically makes hem more compuaionally expensive han mos of heir more resricive counerpars (e.g.[23][24][26][27]). However, moving away from he paricle esimaor sysems used ypically o esimae probabiliy densiy funcions in he Bayesian models [29-30] o neural neworks, i is possible o make hem suiable for parallel execuion and increase heir effeciveness. Classical Probabilisic Neural Nework (PNN) archiecure can be used o creae an efficien[1][4][8], bu his exan soluion is a supervised learning classifier and as such unable o cope wih he ask of background subracion wihou a supervisor classifier. In [1] auhors presen an unsupervised video objec (VO) segmenaion and racking algorihm based on an adapable neural-nework archiecure. The proposed scheme comprises a VO racking module and an iniial VO esimaion module. Objec racking is handled as a classificaion problem and implemened hrough an adapive nework classifier, which, however, relies on he resuls of he iniial video objec segmenaion module o adjus iself o he variaions of he sequence. Based on he video objec segmenaion resuls a se is consruced, which is used o rerain he nework, as proposed by he same auhors in [3]. To deermine when he nework should be rerained a decision mechanism is used. I consiss of a sho cu deecion module and an operaional environmen change module. The firs is based on he principle ha all differen poses ha a VO akes wihin a sho are usually srongly correlaed o each oher, while he second is incorporaed as a safey valve o confron gradual bu significan conen changes wihin a sho. To deec sho ransiions an approach proposed in [2] is used while he graduae changes in he environmen are esimaed based on he error of he neural nework wih respec o he resuls achieved by he iniial objec segmenaion algorihm. To his end he auhors exrac feaures of he objecs segmened by

3 he neural nework and compare hem o he feaures of he iniially segmened objecs. The accuracy of he decision module is crucial for he performance of he sysem as a whole, since he reraining of he nework is compuaionally expensive and frequen reraining ruins he compuaional efficiency of he algorihm, while no reraining when needed leads o poor classificaion resuls. The auhors claim improved performance of heir approach over he convenional moion-based racking algorihms. Alhough he whole segmenaion algorihm is an unsupervised learner, clearly he rerainable neural nework is iself a supervised learner, differing from he approach proposed here. An approach employing a Probabilisic Neural Nework (PNN) classifier in a ime varying environmen is proposed in [4] [5]. A PNN was used o classify clouds based on heir specral and emperaure feaures in he visible and infrared GOES 8 (Geosaionary Operaional Environmenal Saellie) imagery daa. A emporal updaing approach for he PNN was developed o increase he classificaion accuracy by accouning for he emporal changes in he daa. The adapaion of he PNN is supervised by Markov chain models of he emporal conexual informaion combined wih he mixure of Gaussians (MoG) maximum likelihood esimaion. The nework iself is a supervised leaner and is updaed every ime a new frame is processed. BACKGROUND MODELLING NEURAL NETWORK (BNN) The proposed background modelling and subracion approach relies on a novel adapive neural nework. The proposed archiecure employs an adaped General Regression Neural Nework (GRNN) [8][9] componen, o serve as an esimaor of he densiy funcion of probabiliy of cerain feaures belonging o background. GRNNs, ypically used as Bayesian classifiers, are supervised classifiers, requiring a raining se. However, in he domain of background modelling i was possible o exend hem o form new neural nework archiecure which is an unsupervised learner. This Background Modelling Neural Nework (BNN) is suiable o serve boh as a saisical model of he background a each pixel in he video sequences and highly parallelized background subracion algorihm. The design of BNN relies on a basic background modelling idea: feaure values corresponding o background objec will occur mos of he ime, i.e. more ofen han hose perinen o he foreground. Three asks, ypical for probabilisic background modelling [MoG][Li], which BNN should perform have been idenified: 1. Soring he values of he feaures and learning he probabiliy wih which each value corresponds o background / foreground, 2. Deermining he sae in which new feaure values should be inroduced in o he model (i.e. when he saisics already learned are insufficien o make a decision),

4 3. Deermining which sored feaure value should be replaced wih he new values. The wo laer requiremens are consequences of he fac ha real sysems are limied in erms of he number of feaure values ha cam be sored o achieve efficien performance. The srucure of BNN, shown in Figure 1, has hree disinc subnes. The classificaion subne is a GRNN [9]. I is a cenral par of BNN concerned wih approximaing he Probabiliy Densiy Funcion (PDF) of pixel feaure values belonging o background/foreground. The GRNN is a neural nework implemenaion of a Parzen esimaor [11]. This class of PDF esimaors asympoically approaches he underlying paren densiy, provided ha i is smooh and coninuous. The classificaion subne conains 3 layers of neurons. Inpu neurons of his ne simply map he inpus of he nework, which are he values of he feaures for a specific pixel. The oupu of he paern neurons is a nonlinear funcion of Euclidean Figure 1 Srucure of Background Modelling Neural Nework. disance beween he inpu of he nework and he sored paern for ha + 1 W jb = (1 β ) W jb specific neuron. The nonlinear funcion used is as proposed by Parzen. The only parameer of his subne is a so called smoohing parameer used o deermine he shape of he nonlinear funcion. The srucure of a paern X 1 X 2 X 3 X p... The classificaion subne requires no raining o sore he paerns (feaure values) represenaive of background. This is accomplished simply by seing he weighs of he connecions beween he inpu and paern neurons o he value of he feaures of he paern o be sored. The classificaion subne diverges from GRNN in he way he weighs beween he paern and summaion neurons are deermined. These values are used o sore he confidence wih which a paern belongs o he background/foreground. The weighs of hese connecions are updaed wih each new value of a pixel a a cerain posiion received (i.e. wih each frame), according o he following recursive equaions: + 1 W = (1 β ) W + β jf (2) + W = (1 β ) W + β jf (1) ib ib W if = (1 β ) Wif when he maximum response is ha of he i-h neuron, and (3) (4) if he maximum response is no ha of he j-h neuron, where: W ib - value of he weigh beween he i-h paern neuron and he background summaion neuron, W if - value of he weigh beween he i-h paern neuron and he foreground summaion neuron, β - learning rae. T - (W - X) (W - X) / 2σ Figure 2 Paern neuron of GRNN. neuron is shown in Figure 2. The oupu of he summaion unis of he classificaion subne is he sum of heir inpus. The subne has wo summaion neurons: one o calculae he probabiliy of pixel values belonging o background and he oher for calculaing he probabiliy of belonging o foreground. T (W - X) (W - X) e 2 Equaions 1-4, express he noion ha whenever an insance perinen o a paern neuron is encounered, he probabiliy ha ha paern neuron is acivaed by a feaure value vecor belonging o he background, is increased. Naurally, if ha is he case he, probabiliy ha he paern neuron is excied by a paern belonging o foreground is decreased. Vice versa, he more seldom a feaure vecor value corresponding o a paern neuron is encounered he more likely i is ha he paerns represened by i belong o foreground objecs. By adjusing he learning raes, i is possible o conrol he speed of he learning process. The oupu of he classificaion subne indicaes wheher he oupu of he background summaion neuron is higher han ha of he foreground summaion neuron, i.e. ha i is more probable ha

5 he inpu feaure value is due o a background objec raher han a foreground objec. The acivaion and replacemen subnes are Winner-Take-All (WTA) neural neworks. A WTA nework is a parallel and fas way o deermine minimum or he maximum of a se of values, consisen wih he ask of doing so wihin a neuralnework based soluion. In paricular, hese subnes are exensions of one-layer feedforward MAXNET (1LF-MAXNET) proposed in [37]. The acivaion subne performs a dual funcion: i deermines which of he neurons of he nework has maximum acivaion (oupu) and wheher ha value exceeds a hreshold replace _ cri = Wib + Wib W provided as a parameer o he algorihm. If i does no, he BNN is considered inacive and replacemen of a paern neuron s weighs wih he values of he curren inpu vecor is required. If his is he case, he feaure is considered o belong o a foreground objec. The firs layer of his nework has he srucure of a 1LF-MAXNET nework and a single neuron is used o indicae wheher he nework is acive. The oupu of he neurons of he firs layer of he nework can be Y j P = X j { F ( X i= 1 form of 1, F( z) = 0, where: expressed in he he following equaion: (5) The oupu of he firs layer of he acivaion subne will differ from 0 only for he neurons wih maximum acivaion and will be equal o he maximum acivaion. In Figure 1 hese oupus are indicaed wih Z 1 Z p. Figure 3 shows he inner srucure of a neuron in he firs layer of he subne. 1, F( z) = 0, j z 0 z < 0 P Z i i= 1 NA = F ( θ ) z 0 z < 0 X ) i i j} if A single neuron in he second layer of he acivaion subne is concerned wih deecing wheher he BNN is acive or no and is funcion can be expressed in he form of he following equaions: (6) where: and θ is he acivaion hreshold, which is provided o he nework as a parameer. Finally, he replacemen subne in figure 1 can be viewed as a separae neural ne wih he uni inpu. However, i is inexricably relaed o he classificaion subne since each of he replacemen subne firs-layer neurons is conneced wih he inpu via synapses ha have he same weigh as he wo oupu synapses beween he paern and summaion neurons of he classificaion subne. Each paern neuron has a corresponding neuron in he replacemen ne. The funcion of he replacemen ne is o deermine he paern neuron ha minimizes he crierion for replacemen, expressed by he following equaion: (7) The crierion is a mahemaical expression of he idea ha hose paerns ha are leas likely o belong o he background and hose ha provide leas confidence o make he decision should be eliminaed from he model. The neurons of he firs layer calculae he negaed value of he replacemen crierion for he paern neuron hey correspond o. The second layer is a 1LF-MAXNET ha yields non-zero oupu corresponding o he paern neuron o be replaced. To form a complee background-subracion soluion a single insance of a BNN is used o model he feaures a each pixel of he image. EXPERIMENTS AND RESULTS The approach is inended o serve as basis for he design of a hardware componen, which would be able o exploi is highly parallel naure. However, as proof of concep, a simulaion applicaion, which can be run on a ypical PC, has been developed. While his simulaion is sequenial in is execuion and canno provide a valid esimae of he speed of he arge hardware sysem, i can demonsrae he segmenaion abiliy of he sysem. In a hardware implemenaion he delay of he nework corresponds o he ime needed by he signal o propagae hrough he nework and ime Figure 3 Srucure of processing neurons of he acivaion subne.

6 required o updae i. In a ypical FPGA implemenaion his can be done in less han 20 clock cycles, which corresponds o a 2ms delay hrough he nework, for a FPGA core running a 100ns clock rae. The simulaion applicaion implemens BNNs conaining 20 processing, wo summaion and one oupu neuron per pixel in he classificaion subne. The acivaion and replacemen subne aribue for addiional 20, i.e. 41 processing unis respecively, bringing up he oal of neurons used per pixel o 84. The inpu neurons of he classificaion shown in Figure 1 jus map he inpu o he oupu and need no be implemened as such. The simulaion is capable of processing a single frame of size 720x480 in 2.25 seconds on average, which ranslaes o 8 frames of 160x120 pixels per second or 2.2 frames per second(fps) for images sized 320x240 pixels. The primary sequence used for esing is a mariime environmen sequence conaining frames of 720x480 pixels, corresponding o a bi more han 10 minues of recording a 30 frames per second. I conains a large number of diverse vessels ha he algorihm ries o segmen and is complex in erms of background changes relaed o he waersurface. Two consecuive frames from he sequence as well as he resuls of segmenaion are given in Figure 3. Coloured pixels correspond o he foreground. Green are classified as foreground due o he BNNs recognizing ha hese are new values no ye sored, while he red ones are sored bu classified as foreground based on he learned PDFs. No morphological operaions, ypically used o remove spurious one pixel effecs and make he objec solid, have been performed on he segmenaion images shown in Figure 3.These are currenly performed as a pos-processing sep, bu will ulimaely be implemened as a neural nework. The learning rae of he neworks was se o and he smoohing parameer for he classificaion subne used was se o 10. The acivaion hreshold of he acivaion subne was se o Figure 4 Two consecuive frames from a es sequence (op) and he segmenaion resul (boom).

7 CONCLUSION AND FURTHER RESEARCH A new moion based objec segmenaion and background modelling approach has been proposed. The basis of he approach is employmen of a novel neural nework archiecure designed specifically o serve as a model of background in video sequences and a Bayesian classifier o be used for objec segmenaion. The new Background Modelling Neural Nework is an unsupervised classifier. The proposed model is independen of he feaures used. The design is inended o be implemened in hardware, allowing for highly-parallelized execuion. We presen resuls of he simulaion of he sysem on a PC, using a complex mariime sequence. The simulaion iself allows for a fairly fas segmenaion of objecs. Fuure work will focus on he use of feaures differen han RGB values, developmen of a FPGA based sysem and enhancing he segmenaion resuls hrough he use of cues no relaed o moion. REFERENCES Hariaoglu, I., Harwood, D., and Davis, L., W 4 Real-ime surveillance of people and heir aciviies. IEEE Trans. Paern Analysis and Machine Inelligence, vol. 22, pp Sauffer, C. and Grimson, W Learning paerns of aciviy using real-ime racking. IEEE Trans. Paern Analysis and Machine Inelligence, vol. 22, pp K. Toyama, J. Krumm, B. Brumi, and B. Meyers. Wallflower: Principles and pracice of background mainenance. In Proceedings of IEEE In l Conf. On Compuer Vision, pp K. P. Karmann, A. von Brand, A.Moving objec recogniion using an adapive background memory, Time-varying Image Processing and Moving Objec Recogniion, 2, pp , Elsevier Publishers B.V., Amserdam, C. Ridder, O. Munkel, H. Kirchner, Adapive background esimaion and foreground deecion using Kalman-filering, in Proc. of Inernaional Conference on Recen Advances in Mecharonics (ICRAM'95), pp , Jain, A.K., Raha, N.K., Lakshmanan, S., Objec deecion using Gabor filers, Journal of Paern Recogniion, vol. 30, pp , T.E. Boul, R. Micheals, X.Gao, P. Lewis, C. Power, W. Yin, A. Erkan: Frame-rae omnidirecional surveillance and racking of camouflaged and occluded arges, in Proc. of IEEE Workshop on Visual Surveillance, pp , T.J. Ellis, M. Xu, Objec deecion and racking in an open and dynamic world, in Proc. of he Second IEEE Inernaional Workshop on Performance Evaluaion on Tracking and Surveillance (PETS'01), C. Sauffer, W.E.L. Grimson, Learning paerns of aciviy using real-ime racking, IEEE Transacions on Paern Analysis and Machine Inelligence, vol. 22, pp , L. Ya, A. Haizhou, X. Guangyou, Moving objec deecion and racking based on background subracion, in Proc. of SPIE Objec Deecion, Classificaion, and Tracking Technologies, pp , L. Li, W. Huang, I.Y.H. Gu, Q. Tian, Foreground objec deecion from videos conaining complex background, in Proc. of he Elevenh ACM Inernaional Conference on Mulimedia (MULTIMEDIA'03), pp. 2-10, L. Li, W. Huang, I.Y.H. Gu, Q. Tian, Saisical Modeling of Complex Backgrounds for Foreground Objec Deecion, IEEE Trans. Image Processing, vol. 13, pp , Nov D. Socek, D. Culibrk, O. Marques, H. Kalva, B. Furh, A Hybrid Color-Based Foreground Objec Deecion Mehod for Auomaed Marine Surveillance, in Proc. of he Advanced Conceps for Inelligen Vision Sysems Conference (ACIVS 2005), (in prin) T.E. Boul, R. Micheals, X.Gao, P. Lewis, C. Power, W. Yin, A. Erkan: Frame-rae omnidirecional surveillance and racking of camouflaged and occluded arges, in Proc. of IEEE Workshop on Visual Surveillance, pp , A. Doulamis, N. Doulamis, K. Nalianis, and S. Kollias, An Efficien Fully Unsupervised Video Objec Segmenaion Scheme Using an Adapive Neural- Nework Classifier Archiecure, IEEE Trans. On Neural Neworks, vol. 14, pp , May B. Tian, M. R. Azimi-Sadjadi, T. H. Vonder Haar, and D. Reinke, Temporal Updaing Scheme for Probabilisic Neural Nework wih Applicaion o Saellie Cloud Classificaion, IEEE Trans. Neural Neworks, vol. 11, pp , July D. F. Spech, Probabilisic neural neworks, Neural New., vol. 3, pp , A. Doulamis, N. Doulamis, and S. Kollias, On line rerainable neural neworks: Improving he performance of neural neworks in image analysis problems, IEEE Trans. Neural Neworks, vol. 11, Jan M. R. Azimi-Sadjadi, W. Gao, T. H. Vonder Haar, and D. Reinke, Temporal Updaing Scheme for Probabilisic Neural Nework Wih Applicaion o Saellie Cloud Classificaion Furher Resuls, IEEE Trans. Neural Neworks, vol. 12, pp , Sepember 2001.

8 D. F. Spech, A general regression neural nework, IEEE Trans. Neural Neworks, pp , Mar E. Parzen, On esimaion of a probabiliy densiy funcion and mode," Ann. Mah. Sa., Vol. 33, pp , Sep H. K. Kwan, One-layer feedforward neural nework fas maximum/minimum deerminaion, Elecronics Leers, pp , 1992.