Object recognition and tracking in video sequences: a new integrated methodology

Transcription

1 Obje reogniion and raking in video sequenes: a new inegraed mehodolog Niolás Amézquia Gómez, René Alquézar 2 and Franes Serraosa Deparamen d'engineria Informàia i Maemàiques, Universia Rovira i Virgili, Campus Seselades, Av. dels Països Caalans 26, 47, Tarragona, Spain 2 Dep. Llenguages i Sisemes Informàis, Universia Poliènia de Caaluna, Campus Nord, Edifii Omega, 84 Barelona, Spain {franes.serraosa, niolas.amezquia}@urv.a,{amezquia, alquezar}@lsi.up.edu. Absra. This paper desribes a mehodolog ha inegraes reogniion and segmenaion, simulaneousl wih image raking in a ooperaive manner, for reogniion of objes (or pars of hem in image sequenes. A probabilisi general approah a pixel level is depied ogeher wih a praial heurisi simplifiaion in whih pixels lass probabiliies are approximaed b a finie small se of lass possibili values. These possibili values are updaed ieraivel along he image sequene for eah lass and eah pixel aking ino aoun boh he prior raking informaion and he spo-based obje reogniion resuls provided b a rained neural nework. A furher segmenaion of he lass possibili images allows he raking of eah obje of ineres in he sequene. The good experimenal resuls obained so far show he viabili of he approah under erain ondiions. Kewords: Obje reogniion, obje raking, image segmenaion, neural neworks, probabilisi approah, video sequenes. Inroduion This work presens a mehodolog ha inegraes segmenaion, reogniion and raking, for reogniion of objes in image sequenes. To he bes of our knowledge here are few exising works ha ombine segmenaion, reogniion and raking in an inegraed framework []. These asks ofen are reaed separael and/or sequeniall on inermediae represenaions obained b he segmenaion and grouping algorihms [2,, 4]. In [5], obje reogniion ehniques are applied o a sene where he objes of ineres do no move mos of he ime and makes raking a disree proess of wahing for obje disappearanes and reappearanes. The proedure ha we used is based on he ieraive and adapive proessing of onseuive frames. A similar mehodolog is presened in [6]. Anoher relaed work is [7], where a probabilisi approah ha ombines segmenaion, obje reogniion, D loalizaion and raking in an inegraed and unified framework is desribed. In our ase, he original images are firsl segmened in homogeneous regions (spos and olor and geomeri feaures are exraed from hese regions. As repored in [8], neural neworks an be rained o lassif spos ino differen objes using he spo

2 feaures as inpu, provided ha an enough large se of labeled spos is given from he supervised segmenaion of represenaive views of hese objes. In [8], he rained neworks were shown o lassif quie orrel es spos loaed in he same regions of ineres ha he raining spos (ROI ha were defined around eah obje. However, he spo lassifiaion performane impairs signifianl ouside hese regions or in differen images han hose used for raining. In he urren work, we address his problem (obje reogniion in full new images hrough he use of a dnami ieraive approah in whih a probabilisi model a pixel level (or an approximaion of i is updaed aking ino aoun boh he neural ne oupus and prior obje raking informaion from he previous image. A sheme of he whole proess inegraing obje reogniion and raking is displaed in Fig.. Video frame (Image Pre-proessing & Feaure Exraion Spos + Feaures Neural Ne Spo Classifiaion Spos + Classes Image spos lassified Compuaion of Possibili images Traking Images Compuaion of Traking images Possibili images Fig.. Blok diagram of he ieraive obje reogniion and raking proess. The res of he paper is organized as follows. A more formal definiion of he addressed problem is given in Seion 2, ogeher wih he enire noaion used hroughou he paper. In Seion, he proposed mehodolog is desribed in more deail. Experimenal resuls are inluded in Seion 4 and, finall, onlusions and fuure work are disussed in Seion 5. 2 Problem saemen and noaion Le us assume ha we have a sequene of 2D olor images I (x, for =,,L, and a orresponding sequene S (x, of segmened images resuling from he appliaion of an image segmenaion algorihm o he former. Also, le us onsider ha here are (or an be N objes of ineres in he sequene of differen pes (assoiaed wih lasses =,,N, and ha a speial lass =N+ is reserved for he bakground. Furhermore, le us assume ha he iniial posiion of eah obje is known and

3 represened b N binar images, p (x,, for =,,N, where p (x,= means ha he pixel (x, belongs o a region overed b an obje of lass in he firs image. We would like o obain N sequenes of binar images T (x,, for =,,N, ha mark he pixels belonging o eah obje in eah image; hese images are he desired oupu of he whole proess and an also be regarded as he oupu of a raking proess for eah obje. Noe ha we an iniialize hese raking images (for = from he given iniial posiions of eah obje, his is T ( x, p ( x, = ( For noaional purposes, le MC, for =,,N, refer o he mass eners of eah obje in he orresponding raking image T (x,. Suppose ha a neural nework has been rained o lassif regions (spos of he same objes using a differen bu similar sequene of labeled segmened images. Hene, he rained nework is able o produe a sequene of lass probabili images Q (x, for =,,L and =,,N+, where he value Q (x, represens he a-poseriori probabili given b he ne oupu ha he pixel (x, of he segmened image S (x, belongs o he lass. From hese probabiliies, a lass an be assigned o eah pixel simpl b hoosing he lass wih maximum probabili: C ( x, arg max ( Q ( x, = (2 In order o obain he raking images, a probabilisi approah ould be followed in whih we would need o sore and updae N+ probabili images p (x,, for =,,N+, where he value p (x, represens he probabili ha he pixel (x, in ime belongs o an obje of lass (for =,,N or o he bakground (for =N+. In general, hese probabiliies should be ompued as a erain funion f of he same probabiliies in he previous sep, he lass probabiliies given b he neural ne for he urren sep and he raking images resuling from he previous sep: ( p ( x,, Q ( x,, T ( x, p ( x, = f ( Now, he raking images ould be ompued dnamiall using hese probabiliies aording o some deision funion d: ( p ( x,, T ( x, T ( x, = d (4 In he presen work, as a firs simple approah o es, we have relaxed he normalizaion onsrain required for probabiliies and have approximaed he probabili values wih a small se of possibili values (e.g., ½, ompued heurisiall. Hene, insead of using he probabili images p (x, we have used he

4 so-alled possibili images H (x,, ha onain he possibili values ha a pixel (x, belongs o a lass in ime. Noe ha hese images an be iniialized as well from he given iniial posiions of eah obje: H ( x, p ( x, = (5 Consequenl, he updae funion f and he deision funion d have been defined in his work using he possibili images H (x, insead of he probabili images p (x,, in he wa desribed in nex seion. In pariular, he lass assignmens given b he ne C (x, have been used insead of he probabiliies Q (x, in he updae funion f. Mehodolog The mehodolog proposed an be spli in wo phases: he obje learning phase and he obje reogniion and raking phase. Nex subseions desribe boh phases.. Obje learning. For obje learning, a sequene of segmened images showing he objes of ineres is required. Furhermore, a subse of he spos (segmenaion regions obained mus be seleed and labeled manuall (or semi-auomaiall as desribed in [8] wih he arge lasses. These arge lasses inlude he differen obje pes and a speial lass for he bakground. In addiion, for eah seleed spo, a number of feaures have o be ompued ha ma inlude boh olor and geomeri properies. The spo feaures and arge lasses are olleed in a paern file. Then, a neural nework is rained o lassif he seleed spos using mos of he paerns as raining se and he res as validaion se. One rained, when a new paern (spo feaure veor is inrodued, he nework is able o esimae he a-poseriori lass probabiliies for his paern, aording o he saisial model i has learn previousl from he given examples. From hese probabiliies, a lass an be assigned o eah spo simpl b hoosing he lass wih maximum probabili. Noe ha if we represen he probabiliies and lasses a pixel level raher han a spo level, all pixels of a given spo will have he same probabiliies and lass ha he enire spo. A more deailed desripion of he learning phase ha inludes he speifi feaures used for he spos is available in [8]..2 Obje reogniion and raking For obje reogniion and raking, anoher sequene of segmened images showing he same objes of ineres is required. Furhermore, for eah obje of ineres, is approximae loaion in he firs image of he sequene is needed. This informaion is

5 supposed o ome as a binar image for eah obje, where he whie pixels represen he obje and he blak pixels represen he bakground or oher objes. These binar images are used o se he iniial values of boh he raking images and he possibili images, as defined in Seion 2, equaions ( and (5. For he following ime seps =, 2,...L, he binar images ha represen he approximae posiions of he objes of ineres (raking images will be ompued as explained laer. The neural nework obained in he learning phase is applied o all he spos of all he images in he reogniion sequene. This means ha all he spo feaures mus be ompued previousl. From he nework oupus, all spos (and heir onsiuen pixels an be lassified aording o equaion (2. In order o updae he raking images, firs a possibili image is ompued for eah lass and ime sep. The updae funion f for he possibili image H (x, is defined heurisiall aking ino aoun he lassifiaion of pixel (x, given b he neuronal nework, C (x,, and he previous values of he pixel in he raking image T (x, and he possibili image H - (x,. Speifiall, we used as funion f he mapping shown in Table. Table. Updae funion for he possibili image H (x,. The wo shadowed enries orrespond o impossible ases, sine T - (x,= H - ½. T - H - C = H No ½ No No Yes ½ ½ Yes ½ Yes ½ No ½ No No ½ Yes ½ ½ Yes Yes Then, his possibili image H (x, is segmened inside a region-of-ineres ROI, - whih is esimaed from he bounding box BB of he previous raking image T -. In order o ompue he bounding box of a binar image, suh as T -, we use he - mehod desribed in [9]. In fa, ROI and BB share he same ener and shape, bu - he size of region ROI is deermined o be greaer han ha of BB aording o a given fixed sale raio r (e.g. r=.25 o ake ino aoun a possible displaemen of he obje beween onseuive frames. The region of ineres ROI is hen passed o a seed-based segmenaion algorihm [] ha ields he nex raking image T b finding a single onneed region of he image H (x,, wihin he limis of ROI, suh ha all heir pixels have a possibili value han a hreshold z (e.g. z=½, where he seed pixel is defined as he mass ener of T -.

6 Summarizing, for he eah ime sep he nex proesses are arried ou sequeniall:. Calulae he lass assignmen C (x, from he oupus given b he neural nework when he feaures of he spo ha inludes he pixel (x, are enered o he ne. 2. Compue H (x, from H - (x,, T - (x, and C., for eah lass =,,N, using he heurisi mapping defined in Table.. Calulae T (x, from H (x, and T - (x,, for eah lass =,,N, b finding he region of ineres ROI and appling wihin i he seed segmenaion algorihm o H (x,. These seps are shown graphiall in figure 2 for a sequene of ime seps. Fig. 2. Dnami alulaion of H (x, and T (x,. Time H ( x, = p ( x, T ( x, = p ( x, - C H T - C - - H T C H T 4 Experimenal resuls We illusrae our mehodolog and approah using wo sequenes of images ha orrespond o he lef and righ image sequenes of a sereo vision ssem insalled on a mobile robo. These sequenes displa an indoor sene where we hose hree objes of ineres (N=: a box, a hair and a pair of adjaen wasebaskes. In our iniial work desribed in [8], onl he lef sequene was used and onl he spos inside some predefined ROIs were seleed for neural nework raining and es; a ross-validaion proedure was followed using 25% of he spos for esing wih a orre lassifiaion performane of around 76%. In a more reen work [], his performane was raised o a 96% b adjusing more aurael he ROIs and o a 99% b ombining he neural ne wih a relassifiaion proess based on lusering. The

7 performane of he seleed neural ne on he righ sequene was a 9% of orrel lassified paerns in he same ROIs. However, for he es phase, i is somewha rik o resri he obje reogniion o predefined ROIs, sine we anno rel on having he ROIs marked on ever frame in a realisi experimenal senario. Hene, in he new experimens repored here, he same neural nework rained from seleed ROIs in he lef sequene was used, bu he whole righ sequene inluding all spos was aken for esing boh obje reogniion and raking. A ROI for eah obje was onl defined in he firs image o iniialize he raking images. To he onrar of he resuls in [8] and [], in his work we were no so ineresed in ahieving a high spo lassifiaion raio bu a sequene of raking images of good quali for eah obje of ineres, as a firs validaion of he mehodolog proposed in Seion. Fig.. Reogniion and raking resuls for lass (he box in he firs 4 frames of he es sequene. H C ( x, H ( x, T ( x, x = (, H C ( x, H ( x, T ( x, x = (, H C 2 ( x, H 2 ( x, T 2 ( x, x =2 (, H C ( x, H ( x, T ( x, x = (,

8 Fig. 4. Reogniion and raking resuls for lass (he wasebaskes in four frames of he es sequene. H C ( x, H ( x, T ( x, x = (, H C 4 ( x, H 4 ( x, T 4 ( x, x =4 (, H C 5 ( x, H 5 ( x, T 5 ( x, x =5 (, H C 6 ( x, H 6 ( x, T 6 ( x, x =6 (, Fig. 5. Traking of he box on par of he original image sequene. Frame 592 Frame 59 Frame 594 Frame 595

9 Fig. 6. Traking of he wasebaskes on par of he original image sequene Frame 595 Frame 596 Frame 597 Frame 598 Figures and 4 illusrae he proess depied in Figure 2 for wo of he objes of ineres (he box and he wasebaskes, respeivel in some onseuive images of he es sequene. Using he obained raking binar images as a visualizaion mask, he resuls of raking boh objes on he original images are displaed in Figures 5 and 6. I an be observed ha he proposed approah obained raher saisfaor resuls on hese images. Similar good resuls were obained for hese wo objes in he res of he sequene, bu hose for he oher obje (he hair were no so sable. 5. Conlusions and fuure work. A dnami ieraive approah for obje reogniion and raking in video sequenes has been presened in whih a probabilisi model a pixel level (or an approximaion of i is updaed aking ino aoun boh he spo lassifiaion given b a rained neural ne and prior obje raking informaion from he previous image. In his work, possibili images for eah obje of ineres have been updaed using a heurisi rule insead of appling a full probabilisi model. The use of he dnami possibili images ombined wih he raking informaion allow he gradual disriminaion of he pixels lassified as belonging o an obje b he neural nework bu whih do no reall belong o i. I also helps o reover obje pixels ha have been lassified as belonging o he bakground b he nework bu ha reall belong o an obje. This an be made beause he values in he possibili images save informaion of how he pixels have been lassified in previous seps. Thus, his helps o deide a eah ieraion if a pixel belongs o an obje or no. The experimens arried ou have indiaed ha he proposed approah is viable and an provide saisfaor resuls. In a fuure work, we would like o subsiue he possibili images b aual probabili images and o define he updae and deision funions in a more prinipled wa. Our final objeive is o design a robus dnami approah o obje reogniion and raking in video sequenes based on unsruured ses of spos, whih an deal wih he variaions in he obje views resuling from he movemen of a mobile robo in an indoor environmen.

10 6. Referenes.. Zhuowen Tu Xiangrong Chen, Yuille, A.L.,ZhuS.-C, Image Parsing: Unifing Segmenaion, Deeion, and Reogniion, Proeedings in Ninh IEEE Inernaional Conferene on Compuer Vision, pp 8-25 ISBN: (2. 2. J. Malik, S. Belongie, T. Leung and J. Shi, Conour and Texure Analsis for Image Segmenaion, IJCV, vol.4, no., (2.. Z. Tu and S.C. Zhu, Image segmenaion b Daa Driven Markov hain Mone Carlo, IEEE Trans. PAMI, vol. 24, no. 5, ( Song Chun Zhu; Yuille, A.; Region ompeiion: unifing snakes, region growing, and Baes/MDL for muliband image segmenaion, IEEE Trans. on Paern Analsis and Mahine Inelligene, Vol: 8, Issue 9, pp:884 9 Sep. ( Nelson, R.C.; Green I.A. Traking objes using reogniion, Proeedings 6h Inernaional Conferene on Paern Reogniion, ISSN: 5-465, ISBN: X Vol: 2, pp:25- ( Yua Iwasa, Ruihi Oka Spoing reogniion and raking of a deformable obje in a ime-varing image using wo-dimensional oninuous dnami programming, CIT '4. The Fourh Inernaional Conferene on Compuer and Informaion Tehnolog, ISBN: , pp: 8. (24 7. Georg von Wiher, A probabilisi approah o simulaneous segmenaion, obje reogniion, d loalizaion, and raking using sereo, in Leure Noes in Compuer Siene, vol no 29, pp, (2. 8. Amezquia Gómez Niolás and Alquézar René, Obje Reogniion in Indoor Video Sequenes b Classifing Image Segmenaion Regions Using Neural Neworks, Proeedings h Iberoamerian Congress on Paern Reogniion, CIARP 25, LNCS Springer Berlin / Heidelberg ISSN: 2-974, Vol. 77, Chaper: pp. 9-2 ( Gerke, M, Heipke, C, Sraub, B.-M. Building exraion from aerial imager using a generi sene model and invarian geomeri momens, in Remoe Sensing and Daa Fusion over Urban Areas, IEEE/ISPRS Join Workshop, ISBN: ,pp: 85-89, (2.. Ballard D.H. and Brown C.M., Compuer Vision, Prenie Hall, New Jerse (982.. Serraosa F., Amezquia Gómez N. and Alquézar R., Combining neural neworks and lusering ehniques for obje reogniion in indoor video sequenes, Submied o h Iberoamerian Congress on Paern Reogniion, CIARP 26, (26.