Chapter 8 Moving Object Detection

Transcription

1 Chaper 8 Moving Objec Deecion One of he main aciviies of visual percepion is ha of being aware of wha is happening in he surrounding world. In mos cases complee knowledge of he environmen is no required for a sensor o be able o achieve a basic level of awareness. And he problem can be reduced o finding ou deecing hose objecs ha undergo changes from he res of he objecs ha do no appear o change he laer can be considered o form par of a saic background scene ha can be ignored. Wihin his cone objecs ha appear o change are labelled foreground objecs and hose ha do no are labelled background objecs or collecivel as he background. Usuall he main reason wh objecs appear o change is because he move and so he process of deecing changes is also called moving objec deecion 1. There are several differen echniques available in compuer vision for moving objec deecion. The main ones are: Opical flow mehods frame difference mehods and background subracion mehods. The deecion mehod chosen for he OmniTracking program uses he background subracion echnique since his amongs ohers is well suied for saionar cameras which is he normal case for omnidirecional cameras since wih heir large fieldof-view he don have o move roae o see he world. 1 In some compuer vision lieraure a disincion is made beween moion deecion and moving objec deecion. Moion deecion is defined as deermining he changes due o moion wihou doing an furher organisaional processing on he changes while moving objec deecion involves deermining he differen objecs ha are moving [SONK93 14]. For his hesis he laer is being aemped so eplaining he name of he chaper. 94

2 This chaper firs sars wih an overview of he oher deecion mehods. Then he background subracion echnique is eamined in more deail. This is followed b a descripion of how moving objec deecion was implemened for he OmniTracking program and ends wih he resuls obained when running he program on he daases menioned in Opical Flow Opical flow mehods use he apparen moion of he image brighness values in a sequence of images o esimae he relaive moion of objecs wih respec o he camera. This apparen moion of piels is used o consruc a 2D vecor field of velociies called a moion field which can be seen as he 2D projecion on he image plane of he 3D veloci field of he objecs in he world [TRUC98 8.3]. The main advanage offered b he opical flow echnique is ha i works when boh he objecs and he camera are moving wih respec o each oher. Bu opical flow is a ver compuaionall inensive and slow process. Anoher disadvanage is ha if he camera is saionar hen objecs are required o move if he sop moving objecs will have a zero moion field and so are undeecable. 8.2 Frame Difference These are he simples mehods and consis of comparing wo adjacen frames from a video sream o find ou hose piels ha have changed. This is usuall done b calculaing he difference beween he brighness values of he piel in he wo frames based on he assumpion ha changes in brighness are due o real changes in he scene. Some hreshold is hen applied o he differences o eliminae small variaions due o sensor acquisiion noise. The resul is a binar image where each piel is labelled as eiher moving piel or background. This process can be epressed b he following equaion: 1 if f1 f 2 > τ oherwise 95

3 for some hreshold and using image frames a ime 1 and 2. The image frames can be eiher consecuive 2 11 or a cerain number n of frames apar. Because of he use of adjacen frames hese mehods are also called emporal differencing [JAIN ]. The main advanage apar from he simplici is ha frame differencing is ver adapive o dnamic environmens. This is because he gap beween frames 1 and 2 is ver shor compared o an usuall slow changes ha migh occur in he saic background. On he oher hand a disadvanage is ha an objec usuall does no move much from frame 1 o 2 and onl pars of he objec he ouer pars will appear as moving as can be seen in Figure 8.1. This is called he foreground aperure problem [TOYA99]. 2 1 Figure 8.1: Frame Differencing echnique The images are pars of frames 460 and 465 of he PETS2001 daase. 8.3 Background Subracion Background subracion mehods can also be loosel classified ino he caegor of difference-based mehods like frame differencing bu insead of using adjacen frames and finding he differences beween he wo a background model is used. Each frame is compared o his background model and he differences from he background are found. The main requiremen for background subracion mehods is ha he camera remains saionar; else he background model will become invalid 2. 2 Alhough here are some implemenaions where background subracion mehods have been adaped o be used for PTZ pan-il-zoom cameras. Such as he Appearance Sphere applicaion quoed in [YAMA02] which builds a background model for a PTZ camera. Bu hese add complei require precise calibraion and camera snchronisaion and can be compuaionall epensive. 96

4 There are several differen varians of he background subracion echnique as described in he ne secion Simples Form Mean and Global Threshold Mehod In is simples form he background model consiss of an image of wha he emp scene ha is wih no moving objecs is epeced o look like. Each frame is hen subraced from he background image B and a hreshold operaion is applied using he value for all image piels. 1 if B f > τ oherwise background model ime Figure 8.2: Background Subracion echnique he basic idea. Compared o frame differencing background subracion mehods have he advanage of deecing he whole objec see Figure 8.2. Bu heir main disadvanage is ha he are ver sensiive o dnamic changes in he scene ha is he background model can become ou-of-dae. In his secion i was menioned ha he background image consiss of an image of he emp scene. Bu in mos cases i s no possible o ge such an image beforehand. Especiall for bus oudoor environmens like he one menioned in 4.2 and being used for his projec. Also he ideal case is for he background model o be buil auomaicall wihou human inervenion. The simples wa of building a background image is o ake he average of several frames: n 1 n f 1 B µ

5 The idea behind he use of averages has is origins in he 19 h cenur when i was discovered in phoograph ha b aking phoos wih ver long eposures moving objecs are eliminaed from he scene [FRIE97]. The ime duraion of n frames used o build he background image is usuall called he raining phase or iniialisaion phase of background subracion Handling Noise Normal Disribuion Mehod One of he limiaions of he previous mehod is he use of he global hreshold for deermining wheher a piel value is similar o he value epeced from he background model or no. Assuming a consan scene he variaion observed for a background piel over a period of ime should onl be caused b camera noise which is usuall modelled b a normal disribuion wih zero mean N noise 0 2 for ha paricular piel [ELGA99]. background model Figure 8.3: Background Subracion echnique normal disribuion model. Adding he camera noise o he piel s mean value from he background model he piel s variaion can hen be modelled b N 2. So ogeher wih form he background model B and is calculaed during he raining phase using he following equaion is sill calculaed using 8.3: n 1 2 n f 1 2 σ µ 8.4 B µ σ 8.5 A piel is hen labelled as foreground if f µ > kσ for some k. 98

6 8.3.3 Handling Background Moion Miure of Gaussians Mehod The assumpion so far has been ha he background scene does no change. Bu i can happen ha background elemens show some movemen for eample rees moving in he wind in oudoor environmens. This unineresing moion should sill be classified as belonging o he background. The brighness variaions ehibied b hese piels is mulimodal and is bes handled b using a miure of Gaussian normal disribuions [STAU99]. The miure model M is defined as: M k i 1 π in i µ σ 8.6 where k is he number of componen disribuions used normall ranging from 3 o 5 N i is he i h individual normal componen disribuion and i is is miing weigh k i 1 wih π 1. i The basic idea of he miure model is ha he differen disribuions will each model a paricular background elemen ha he piel happens o resul from in he case of a moving ree a piel can a one ime be he ligh coming from he ree and a anoher momen i will be he sk for eample. A separae Gaussian miure model is used for each piel. The background model is hen defined as: B π π... π µ µ... µ σ σ... σ k 1 2 k 1 2 k In he above equaion he individual parameers are defined for each piel ha is 1 should read as 1 bu he subscrip has been omied o avoid cluer. Fiing he miure model o he piel daa during he raining phase of background subracion is usuall done b maimising some likelihood funcion of he miure model M. Mehods such as he Epecaion-Maimisaion EM algorihm or he K- means algorihm are usuall used for his process [FRIE97; STAU99]. A Gaussian miure background subracion is ver robus o background scene moion. However fiing he miure model is compuaionall epensive and mainaining a model for each piel requires large amouns of memor. 99

7 background model ƒ Figure 8.4: Background Subracion echnique Gaussian Miure model Pos-Processing The resul of background subracion is a binar image in which each piel is labelled as foreground or background. Normall some sequence of pos-processing operaions is applied o he resul o make he background subracion algorihm more robus. Eamples include: morphological operaions cleaning operaors o fill gaps in he resul and remove noise and shadow removal. 8.4 Background Adapaion One of he disadvanages of background subracion is ha he scene can change over normall long periods of ime and so he background model ma ge ou of dae for eample illuminaion changes during he da. To counerac his he background model can be modified a run-ime o adap o an such changes his is called background adapaion or someimes background mainenance [TOYA99]. A common echnique for performing background mainenance is o emplo a movingwindow average where he background is re-calculaed a ime from he previous n frames from -1 o -1-n. This requires he previous n samples for each piel o be sored. A beer alernaive is o use he emporal inegraion approach where he background model is updaed using he following general equaion: B 1 B 1 f

8 101 where is called he inegraion parameer or blending parameer. More specificall for he single normal disribuion mehod 8.8 is implemened as follows: B f f σ µ µ σ σ µ µ 8.9 In he case of he Gaussian miure model he same blending funcion can be applied o he disribuion componen ha bes maches suppors he piel. Alernaivel he miure model is updaed using a compleel differen procedure such as an incremenal version of he EM algorihm [FRIE97]. The parameer ranges from 0 o 1 and deermines how responsive is he background model o change. A large value means ha a higher conribuion from he curren piel value is added o he background model B and he conribuion of he older values B -1 is reduced. A an ime he reducion of he conribuions of background B follows he sequence: ec f f f B B f f B B f B B I is clear ha his sequence forms a weighed sum of previous piel values wih he weigh 1 being an eponenial funcion. For his reason he background updae process is called an eponenial forgeing process wih 1 being he ime consan of he process [FRIE97]. Figure 8.5: Eponenial forgeing for background updae. 1-

9 8.4.1 Tpes of Background Updae Normall updae of he background model is onl performed for hose piels which a an ime are labelled as background b he deecion process. This implies ha background updae is done as he las sep of he background subracion process afer he deecion and labelling phase is finished. This selecive pe of background updae helps o increase he accurac of deecion since foreground objecs will no corrup he background model. Bu problems ma occur if he decision of wheher a piel is background or no is incorrec. If he background of a piel changes and i is incorrecl labelled as foreground his ma lead o is background model never o be updaed and hence causes persisenl incorrec decisions o be aken b he deecion algorihm. This is a deadlock siuaion [ELGA99]. One wa of solving he deadlock problem is o do a blind updae updaing all he piels regardless of wheher he are foreground or no. The disadvanage of his is he increase in deecion errors. A compromise beween selecive and blind updae is o use wo blending parameers one for background piels and he oher for foreground. Generall he parameer for foreground piels is se o a slower rae of inegraion [BOUL99]. B 1 bkg B 1 bkg f if f is background frg B 1 frg f if f is foreground 8.5 Review of Applicaions using Background Subracion This secion gives a brief review of some applicaions ha use he background subracion echnique for moving objec deecion wih special aenion o hose using omnidirecional cameras. More applicaions can be found in [MCIV00; TOYA99]. [BOUL99] use background subracion for an omnidirecional camera-based surveillance applicaion named LOTS. Background subracion is performed on he raw omnidirecional image. Two separae background models are used 102

10 in cascade fashion. The primar background consiss of a single normal 1s disribuion B µ σ and is updaed using separae foreground and background parameers see The second background is onl applied o piels labelled as foreground b he firs background model and i is updaed using B 2nd 2nd 2nd 1 frg f 1 B if B f is below 2nd some hreshold or B 1 f oherwise. Colour is used for background subracion in he applicaion described in [CUTL98]. Separae averages are kep for he colour componens RGB colour R G B space wih he background model consising of: B µ µ µ. A piel is labelled as foreground if i saisfies he condiion: c R G B > c µ f kσ where is esimaed beforehand. The W 4 applicaion of [HARI98] uses a background model consising of: B fmin fma f dela where he values are he maimum minimum and maimum difference found in he se of piel values during he raining phase. A piel is labelled as foreground if eiher of he condiions f fma f or f fmin f > dela > dela is rue. The background is updaed using he selecive updae approach A Gaussian miure model is used b [STAU99] for modelling he background wih 3 o 5 componen disribuions and using an online K-means algorihm for fiing he miure model o he raining daa. In his applicaion a variaion of he blind background updae pe is used and background updae is done before he foreground is deeced. If he piel value maches a componen disribuion is wihin 2.5 hen ha componen s parameers i i and i are updaed using emporal inegraion 8.8. The oher componens parameers are lef unchanged. If no componen maches he piel s value he leas probable componen is deleed and replaced wih a new componen which has µ σ i f i se o a large value and low π i.for moving objec deecion he componen disribuions in he miure model are sored in order of heir probabili of occurrence and he firs L of hese are chosen o represen he background. If a piel does no fall wihin 2.5 of an of hese L disribuions i is labelled as foreground. 103

11 [HUAN02] use background subracion for an omnidirecional-based applicaion called NOVA used for racking people in a room. Due o he simplified naure of he indoor environmen background subracion is performed on he panoramic image obained from unwrapping he raw omnidirecional image unlike he LOTS applicaion menioned furher above. The single normal disribuion mehod is used wih he addiion ha he background model is augmened wih he brighness disorion and chrominance disorion CD values background model B CD. The las wo are used for doing shadow deecion as a pos-processing sep o background subracion. The differences beween he image and he background panoramic image are collapsed o a 1-D profile b accumulaing a hisogram of piel differences in each column of he panorama. A global hreshold is hen applied o each column o see if enough piels wihin ha column were labelled as foreground. [ZHU99] also use background subracion for an indoor environmen-based applicaion capured wih an omnidirecional camera. Similarl o he previous applicaion background subracion is performed on he dewarped panoramic image. A combinaion of background image subracion and frame differencing is used. The resuls from background subracion s and frame differencing d are combined a a region level raher han a a piel level wih he piel of region R being acceped as foreground if he condiion s min R i i [ d ] i i is saisfied. [YAMA02] uses background subracion for omnidirecional images in which he background is modelled wih B µ σ sin2πω k noise where he erm σ sin 2πω models he flicker of fluorescen ligh and CRT screens and he k noise facor represens he camera sensor noise. If he condiion: µ σ sin 2πω k noise f µ σ sin2πω k noise is saisfied hen he piel is labelled as background. Temporal inegraion is used o updae he background model. [JABR00] implemen a background subracion mehod ha combines colour C C C C and edge informaion. The background is modelled b: B µ σ H V for C RGB where H is he horizonal edge map and V he verical one 104

12 obained from he Sobel edge deecor. The edge maps of he frame a ime are subraced from he edge maps of he background model and he resuling edges are classified as: occluding edge occluded edge or background edge. The edge informaion is combined wih he hresholded colour differences o ge a final measure of change. The idea of using edge informaion is ha his usuall leads o a more accurae eracion of he boundaries of objecs. 8.6 Tpes and Sources of Deecion Errors Moving objec deecion is usuall he firs processing sep performed b compuer vision applicaions. An deecion errors ha occur during his sep will propagae o laer processing phases and affec heir resuls in a negaive wa. Therefore i is imporan ha he algorihms used for moving objec deecion are as accurae as possible and ha deecion errors are eliminaed or minimised. The accurac of hese algorihms is usuall measured b heir deecion rae ha is how man moving objecs in he world he are able o find. And also b he number of deecion errors he generae normall epressed in erms of he number of false posiives and false negaives. A false posiive occurs when an algorihm sas ha a cerain se of piels belong o a moving objec when in reali here is no objec a he indicaed posiion. A false negaive occurs when here is a moving objec in he scene bu he algorihm misses he objec and labels is piels as background [TRUC98 A.1]. In he case of he background subracion echnique several condiions can poeniall give rise o deecion errors and hese are briefl menioned below. The chosen background subracion algorihm should be implemened wih hese condiions in mind wih he aim of ring o achieve a cerain degree of robusness agains hem. Bu because of heir low-level naure ha is independen piel-based processing background subracion echniques ma no be able o solve all of he problems and he soluion o some of hem if a all possible ma require domain knowledge and higher-level processing [TOYA99]. Camera-relaed problems: Noise objecs: Camera noise usuall manifess iself as random flucuaions in he inensi values of piels and hese ma cause he background subracion 105

13 algorihm o misakenl label hem as foreground piels generaing false posiives. The inclusion of a camera noise model in he background see helps o reduce his problem. Appling a size filer o he background subracion resul is also effecive because noise objecs end o be small [COLL00]. Incomplee eracion: Camera noise can also cause an objec no o be full differeniaed from he background leaving gaps especiall in he boundar of he objec. In he wors case an objec ma become fragmened. Appling morphological operaions on he background resul can alleviae his problem. Regisraion errors: The basic assumpion for background subracion echniques is ha he camera is saionar. Bu in some cases he camera ma suffer from vibraions for eample when objecs come ver close o he camera or due o wind in oudoor scenes. Camera moion can cause he background o ge misaligned wih respec o an image and he wrong background values o be used. One wa of minimising his problem is o implemen some form of auomaic regisraion ha brings he background and image back in alignmen wih each oher. Lighing problems: Gradual ligh changes: The appearance of oudoor background scenes is affeced b he slow change of illuminaion caused b daligh 3. This ma also affec indoor scenes where windows are presen. This is normall solved hrough background adapaion 8.4. Sudden ligh changes: Eamples include urning he lighs on and off in indoor scenes and he sun moving behind clouds oudoor scenes. In his case background adapaion migh no help as is rae of updae is much slower. Sudden ligh changes affec he whole scene and cause man false posiives o appear one can deec hem b checking for an abnormal increase in he number of foreground objecs and hen change or reconsruc he background [XU01]. 3 See [FORS ] for a graph showing variaions of daligh measured a differen imes of he da and under differen amospheric condiions. 106

14 Shadow-relaed problems: Objec disorion: Shadows can give rise o a number of differen deecion errors. If shadows are labelled as par of he objec hen he objec s shape will appear disored and his disorion can affec laer processing phases ha use geomerical properies for objec classificaion locaion esimaion ec. [CUCC01]. The effec of shadows can be suppressed b using shadow deecion algorihms or b using informaion ha is invarian o shadows. Objec loss: Shadows can also give rise o objec loss when an objec s shadow is cas upon anoher objec in he scene. Objec under-segmenaion: When wo objecs are close o each oher heir shadow migh cause hem o appear o be conneced. And he wo objecs will be merged ogeher b he algorihm causing under-segmenaion ha is he algorihm repors less objecs han in fac here are in he scene [PRAT01]. Shadow suppression reduces his problem. Objec-relaed problems: Camouflage: The objecs hemselves or heir behaviour ma be he source of errors for background subracion. Camouflage occurs when pars of an objec s colour mach he background so making hem indisinguishable from he background. This resuls in objec fragmenaion. Using clusering or morphological operaions ma help o reduce his problem [COLL00]. Slow-moving objecs: If objecs move a a ver slow rae comparable o he background adapaion rae here is he possibili for hem o be pariall absorbed ino he background. This can cause he objecs o be los or ma give rise o false posiives when he evenuall move awa from he curren posiion. Selecive background updae solves his problem Anoher wa is hrough he use of muliple backgrounds [BOUL99]. Foreground aperure problem: When a uniforml-coloured objec moves i can happen ha he inerior piels of he objec are no deeced [TOYA99]. This problem is more common for frame differencing mehods bu i ma also occur during background subracion if an objec has been saionar for some ime. Moion will onl be perceived a he edges of he objec and no in he cenre. This causes he objec o be fragmened. 107

15 Background-relaed problems: Illegiimae moion: Moion b elemens of he background scene eample rees swaing in he wind causes false posiives o be deeced. Modelling he background wih muliple disribuions as described in helps o reduce his problem. Moved background objecs: Anoher source for false posiives is when objecs change elemens of he scene or remove and inroduce new elemens for eample a person eners a room and moves a chair from one place o anoher [STAU99]. Background adapaion can help o reduce his problem as he affeced elemens will evenuall be absorbed ino he background. Ghoss: When an objec ha has been saionar for a long ime and hence absorbed ino he background moves awa wo objecs are deeced he moving objec and anoher false posiive where he objec was originall locaed. This negaive objec is called a ghos [COLL00]. Use of background updae ecep for he selecive updae pe see will evenuall remove he ghos objec. I is also possible o use some pos-processing logic o pach he hole lef in he background b he objec. This problem migh give rise o deadlock siuaions if selecive background updae is used [CUCC01]. Deadlock problem: As menioned in he previous paragraph if selecive background updae is used ha is he background is no updae for piels labelled as foreground false posiives end o persis indefiniel as he background of hese piels will no be updaed. This problem can be reduced b using he oher updae pes menioned in Quie raining phase: The background model is normall buil auomaicall from he firs few image frames capured b he camera. The ideal condiion is for he scene o be emp of an moving objecs o ge an accurae background represenaion bu in real-life his is no alwas possible. The presence of moving objecs during he raining phase of he algorihm will cause some corrupion of he background which in urn migh give rise o deecion errors. This problem is also called he boosrapping problem b [TOYA99]. 108

16 8.7 Implemenaion The OmniTracking applicaion uses background subracion for deecing moving objecs. Background subracion is performed direcl on he omnidirecional image. I was decided o model each piel wih a single normal disribuion and o perform background model adapaion using wo differen raes for he foreground and background piels respecivel. Background subracion is performed using colour informaion as his is a beer discriminan han jus grescale. To solve he oversegmenaion issue shadow deecion and removal was implemened b working in he HSV colour space. Finall hresholding-wih-hseresis is performed on he background subracion resul o ge he classificaion of piels ino foreground and background Choosing a Background Model The firs decision ha had o be aken was which background model o use. The hree main pes of background models were described in 8.3. The Gaussian miure model is he mos accurae of he hree because i models each piel wih muliple componens and i appears ha i should be he mehod of choice. Bu he problem wih his background model is is compuaional cos and large memor requiremens A Tes using Miure Models To check wheher i was worhwhile using his mehod or no a es program was wrien ha modelled each piel b a miure of 3 Gaussians. The EM algorihm was used o fi he miure model o an iniial se of 100 frames from he PETS2001 daase seleced because his daase shows evidence of moion in he background swaing rees moving clouds sligh camera jier ec. Each piel value f was defined b he colour vecor HS where H and S are is hue and sauraion values as defined in he HSV colour space. Therefore each 2D Gaussian componen N i akes he following parameers: H S 2 H S 2 HS where 2 H S 2 are he individual variances and HS is he covariance. N i is defined as: 109

17 [ H S µ µ ] T 1 [ H S µ ] H S H µ S 2 σ N i e where H σ HS 2 2π σ HS σ S 8.11 The componen miure M of each piel hen consiss of N 1 N 2 N This resuls in a memor requiremen of 72 bes per piel 4. Since he size of each frame of he PETS2001 daase is of piels he oal memor requiremen for he model is of ~31Mb his no couning addiional values such as hresholds and precalculaed values ha can be used o make processing run faser. In addiion running he EM algorihm for each miure model of ever piel is quie epensive even when a fas version of he EM algorihm is used for eample he Incremenal EM algorihm [NG03]. From some iniial ess he sandalone background subracion algorihm wih a miure model was running a a rae of beween 2 and 3 frames per second 5 and he iniial model from he firs 100 frames ook abou 4 minues o be buil using he sandard EM algorihm. One wa of reducing he runime coss is o reduce he size of he image. Bu his is no feasible for omnidirecional images wih heir alread low-resoluion. Oher improvemens include assuming zero covariance independence beween he colour componens HS 0 and using look-up-ables a he cos of more memor [STAU99]. A his poin i is useful o see how man of hese Gaussian componens are acuall used. Figure 8.6 shows he variaions of he HS values for si piels and he number of Gaussian componens ha were required o eplain hese variaions. For eample poin 1 is a piel ha is someimes a ree blue componen and someimes he sk red componen wih weighs 0.57 and 0.43 respecivel. Figure 8.6b shows ha for mos of he piels one componen is enough. As epeced he piels needing all 3 componens are he rees and some pars of he cars wih heir meallic surfaces. Mos of he road and he grass do no change much. There are some variaions in he road surface which are arranged in linear-like srucures hese mos probabl are due o JPEG noise piels ha happen o fall on he edges of he 8 8 compression blocks. 4 The values are sored as floaing-poin variables which in C are represened b 4 bes. Therefore he oal consiss of 20 bes for each of he 3 componens and he res for he miure weighs i. 5 This wihou background updae. 110

18 Mos of he background movemen seen in his daase is quie small a few piels. This is because he background elemens in he oudoor scene are a a cerain disance from he camera and also due o he low resoluion of he omnidirecional camera. In general one ma assume ha his also holds for mos environmens in which omnidirecional-based surveillance applicaions are used. The background movemens manifes hemselves mosl as hin linear-like groups of piels mosl due o piels ha are on he edges beween wo background elemens hese can be eliminaed using echniques such as size filering or hresholding. Using onl hue and sauraion for modelling piels misses objecs ha happen o have he same colour as he background bu differen brighness. To deec hese objecs he brighness V needs o be added o H and S meaning more memor would be required for he miure model. For hese reasons i was decided no o use he Gaussian miure model for background subracion bu he single Gaussian disribuion model see This mehod should give reasonable resuls while a he same ime being fas. Bu given more ime and if he miure model program is opimised enough 6 i could be used insead The Chosen Model The chosen background subracion mehod for he OmniTracking applicaion models each piel b a normal disribuion as follows: µ µ µ σ σ σ B N µ σ N 8.12 H S V H S V And he H S V colour componens are considered o be independen ha is correlaion beween hem is assumed o be zero. Colour is used as his gives a beer deecion rae. The choice of working in he HSV colour space is due o he presence of shadows as will be menioned in he ne secion. 6 For eample memor-wise here s no need o allocae sorage for all 3 componens as few piels will require all 3 of hem as can be seen in Figure 8.6b. Bu hen some form of poiner-based memor srucure will be required which migh add some processing and memor overhead. 111

19 1. Tree & Sk Building S H 3. Road Grass a Si seleced poins and heir HS-scaer plos wih Gaussian componens fied o hem. 5. Tree & Window Car Window b The picure below shows he number of Gaussian componens required b he piels o model heir backgrounds. Componens needed: black 3 componens dark gre 2 componens ligh gre 1 componen Figure 8.6: Miure Models for piels of he PETS2001 daase. 112

20 8.7.2 Robusness o Shadow Shadow is a major source of problems for objec deecion as menioned in 8.6. This is especiall rue wihin he confines of indoor environmens for eample objecs end o be quie close o each oher and shadow migh cause he objecs o appear o merge. Figure 8.7 shows some eamples aken from he PETS-ICVS daase. a Objec Disorion due o shadow par of frame b Under-Segmenaion merging due o shadow frame c False posiive shadow objec frame Figure 8.7: Deecion Errors caused b Shadow columns 1 o 3. Las column is wih shadow deecion enabled gre. Frames from PETS-ICVS daase. For his reason i was decided ha shadow deecion should form an essenial par of he background subracion process. Bu wha is shadow? Firs i can be seen ha when an objec is in shadow is appearance colour does no depend on he objec casing he shadow blocking he ligh [FRIE97]. Furhermore when he objec is in shadow i is no compleel dark bu is li b he ambien ligh generaed b he surrounding scene ha is ligh refleced b he oher objecs in he scene. Normall his ambien ligh is nearl gre colourless as i is he average of he ligh refleced b man of he surrounding objecs [XU01]. 113

21 Combining hese wo observaions ogeher: when an elemen in he background scene is in he shadow cas b a moving objec is colour will no change bu is brighness will be reduced. Therefore if background subracion is o be robus o shadows i mus be able o ignore brighness changes when he colour of he background remains he same his is called illuminaion invariance [XU01]. To do his he algorihm mus be able o separae he illuminaion componen of he piel s ligh from he chromaic componen. This idea is also relaed o an abili of he human vision called colour consanc where humans are able o assign he same colour o an objec under differen levels of illuminaion [HORP99]. There are man differen was of epressing he piel s brighness ha achieve illuminaion invariance. For eample: [ELGA02; XU01] use he values rg where r R R G B g G R G B in he sandard RGB colour space. These are invarian o brighness because of he normalisaion facor in he denominaor. [XU01] menions also he log chromaici differences lnr/g and lnb/g. For he YUV colour space he values U/Y V/Y can be used. The hue H and sauraion S values of he HSI or HSV colour spaces. [HORP00] consider colour values as vecors in he RGB colour space and derive he brighness differences and chromaici disorions CD in erms of vecor geomer. More mehods can be found in [PRAT01] HSV Colour Space and Shadow Deecion For his implemenaion i was decided o use he HSV hue-sauraion-value colour space. This space separaes he chromaici values epressed in erms of hue and sauraion from he illuminaion value V. The values H and S are invarian o illuminaion changes and can be used for suppressing shadows. A piel f HSV is considered o be a shadow piel if i saisfies he following condiion when compared o is background value H B S B V B : 7 In he case of background subracion and man oher compuer vision applicaions shadow is considered o be a nuisance and a source of problems. Bu shadow can be useful in is own righ for eample in shape from shading applicaions [SONK ]. 114

22 H H V B B S S < V < V B B where 0 < The las column in Figure 8.7 shows shadow deecion applied o he PETS-ICVS daase. The global parameer in 8.13 defines he maimum amoun of shadow darkening ha can be epeced o be presen in he scene ha is shadow srengh is epeced o be 1-. This usuall depends on he srengh of he illuminaion sources and he amoun of ambien ligh presen. The reason for including his lower limi on brighness reducion is o avoid missing objecs ha happen o be coloured like he background bu are darker. For eample he person in Figure 8.7a would be classified incorrecl as shadow because he black op appears o be a darker version of he whie-colour of he wall. B defaul he value for parameer is se o 0.7 pical of indoor scenes and is user-configurable 8. Seing o 1 in 8.13 disables shadow deecion. For now his value remains fied while he program is running. This is a limiaion and ideall his value should be adaped depending on he lighing condiions in he scene shadows are sronger a noon Low Chromaici Condiions The HSV colour space is clindrical in naure and he hue componen he main value deermining he chromaici is derived from he RGB space using [JAIN ]: cos H 2 2R G B 2 R G R B G B 8.14 From 8.14 i is eviden ha hue is undefined when R G B 0. In addiion hue is unsable when he colour is near he origin of he RGB space giving rise o wide variaions in hue for small changes in RGB. In Figure 8.8 he range of ellow colours defined in RGB as cc0 for c [0..255] map o he range of colours 60 1c in he HSV colour space he hick black verical line in he diagram. Adding an error of ± 00 o he RGB colours 8 The defaul value of 0.7 worked well for boh of he PETS daases. 115

23 resuls in he error shown in he figure. For eample if is 1 hen he RGB colour c±1c0 maps o HSV colour 60 ± 1c ZKHUH LV IRXQG WR EH 9 : 2c 1 cosδ. The hue error has a maimum value of 30 when c 1 V 2 2 c c 1 showing he insabili of hue for low RGB values. H 60 s s high v v low Figure 8.8: error 30 Low chromaici hresholds for HSV colour space So during background subracion shadow deecion as defined in 8.13 is onl done if he piel s colour saisfies he following hresholds: V > V and S < 8.15 low S high Background Iniialisaion The background model B µ σ is iniialised from he firs n frames of HSV HSV he video sream using 8.3 and 8.4. The defaul duraion is se o 32 frames bu his value is user configurable. Background iniialisaion ma suffer from he problem of having moving objecs wihin he scene while he background is being accumulaed his is he case of he PETS2001 daase where acivi sars from frame 1. Bu no aemp was made a solving his problem as i wasn considered criical o compensae for his one could increase he duraion of background iniialisaion and hope ha here are no much slow-moving objecs a he ime. The values obained from background iniialisaion are checked o make sure ha he are above a cerain global hreshold min. This hreshold represens he esimae for he camera noise and also ensures ha he variances of he background model are 9 Derived b subsiuing he RGB colours c1c0 cc0 ino 8.14 and compuing he difference beween he wo. 116

24 non-zero. One wa of seeing his is as if appling a global hreshold represening he camera noise in addiion o he per-piel hreshold ha represens scene variabili. A value of min 5 was chosen as he defaul for he HSV colour componens. H S V H S V Figure 8.9: Background mean and sandard deviaion maps Figure 8.9 shows he background mean and sandard deviaion maps consruced during he iniialisaion phase. For speed reasons he floaing-poin values and are sored as inegers in 16-bi image maps wih a fied precision of 3 decimal places and calculaions are done using ineger arihmeic. The mask generaed b he calibraion process see is used o reduce he workload per image. The OpenCV and IPL libraries boh have conversion funcions o and from HSV. Bu hese use 8-bi images reduce he hue o a range from 0 o 180 and do he work using floaing-poin arihmeic. Insead he OmniTracking applicaion uses is own conversion funcions based on ineger arihmeic Background Subracion Algorihm Using he background model maps menioned in he previous secion and combining background subracion wih shadow deecion gives he following algorihm. This is combined laer on wih a hseresis hresholding algorihm o ge he final resul. 117

25 The oupu of he algorihm will be he map l where each piel is labelled wih one of he following values: background.. foreground shadow piel probable piel. his piel represens a background elemen. his piel represens a moving objec. his is a background piel under he effec of shadow. his ma be a moving objec or a background shadow. Is final classificaion will be deermined b he hresholding algorihm. Le he colour of piel be HSV. The firs sep is obaining he arihmeic difference for each colour componen from he background value. In he case of hue modular arihmeic is required: H H µ H if H µ H < π H µ H 8.16 π H µ H oherwise S S µ S 8.17 V V µ V 8.18 Then he piel is labelled according o hese condiions: if if V if if > 3σ hen : V µ V < µ and < 3σ and < 3σ hen : V V V H H S S 8.19 < 8.19a l shadow piel > 4σ hen : V l foreground piel else : l probable piel 8.19b 8.19c > 3σ and V > V and S S hen : 8.20 H H low < high if H > 4σ hen : H l foreground piel else : l probable piel oherwise : l background piel 8.20a 8.20b

26 Basicall 8.19 checks if a piel had a large luminance variaion and 8.20 checks if he piel had a large hue variaion. Inside he luminance condiion an era check for shadow is done 8.19a while he hue condiion is onl done if he piel saisfies he low chromaici hresholds V low S high. For boh luminance and hue if he variaion is above 4 100% confidence hen he piel is labelled foreground; if less han 4 bu above % confidence hen i is labelled as probable. These probable piels will hen be eamined laer during he hresholding sep and eiher se o foreground or shadow or jus discarded. From iniial es runs i was found ha because he algorihm checks for boh luminance and chromaici hue variaions beer deecion raes are obained. This is mosl due o he camouflage problem where objecs migh have he same colour or he same inensi as he background Thresholding wih Hseresis The background subracion algorihm labels some of he piels as probable. The idea is ha hese piels show a large difference from he background model >3 so mos probabl are foreground or shadow piels bu heir probabili is no high enough o guaranee his on heir own he mus be suppored b neighbouring foreground or shadow piels. This is wha he hseresis hresholding algorihm of he program does. In addiion i also acs as a noise filer b suppressing an isolaed foreground or shadow piels. The algorihm is described below: For each piel in he label map l consider is 8-neighbours and le: N F number of 8-neighbours labelled foreground and N S number of 8-neighbours labelled shadow piels. Then appl he following condiions o l using hreshold T o demoe an isolaed piels: if l foreground and N F < T hen : 8.22 l probable if l shadow and N S < T hen : 8.23 l probable 119

27 Finall appl he following condiions o l o promoe an probable piels ha are suppored b heir neighbours: if l probable and N F T and N F > N S hen : 8.24 l foreground if l probable and N S T and N S > N F hen : 8.25 l shadow Threshold T above is se o 4 ha is half he neighbouring piels. This process is repeaed unil eiher no piel labels are changed in an ieraion or he maimum number of ieraions are reached. Generall hresholding wih hseresis converges quie rapidl and onl a few ieraions are required [TRUC ]. In his case he maimum number of ieraions was se o 4. Afer hresholding is finished an remaining probable piels are se o background. Oher mehods of achieving similar resuls are menioned in bu compared wih morphological operaions hseresis hresholding was found o give good resuls and o be quie fas i is faser han OpenCV s morphological funcions ha are opimised for MMX processors. Figure 8.10 shows hseresis hresholding applied o pars of wo frames from he PETS-ICVS daase. Noe how he bulk of he changes happen in he firs ieraion for boh frames. This was observed consisenl hroughou he res of he video sream. The figure also shows wha happens if he maimum number ieraions is no limied o 4 he furher refinemen in he labelling is minimal and no reall worh he era processing Background Model Adapaion For he background model o remain useful he moion deecion module of he program uses a background adapaion algorihm based on he wo blending parameer mehod see Parameer bkg is used o updae hose piels ha in he label map l have been se o background or shadow while frg is used for he foregroundlabelled piels. Normall frg bkg and he parameers are in range [0..1]. Seing frg o 0 one ges he selecive updae mehod while frg bkg is he blind updae 120

28 mehod. These wo parameers are user-configurable and he following values were used for he PETS daases: daase bkg frg PETS2001 daase PETS-ICVS daases C and B Background subracion resul frame #10705 Background subracion resul frame #11980 shadow foreground probable background Ieraion 1: 1566 piels changed Ieraion 1: 3260 piels changed Ieraion 2: 300 piels changed Ieraion 2: 846 piels changed Ieraion 3: 134 piels changed Ieraion 3: 405 piels changed Ieraion 4: 37 piels changed Ieraion 4: 228 piels changed Ieraion 11: for ieraions 5 o 11: a oal of 65 piels were changed Ieraion 13: for ieraions 5 o 13: a oal of 422 piels were changed Figure 8.10: Thresholding wih Hseresis some resuls PETS-ICVS daase 121

29 122 For he sandard deviaion map he updae 8.9 was modified from: f f µ σ σ µ µ frg bkg or o: f f µ σ σ µ µ 8.26 o ge rid of he epensive square-roo. The updae parameers end o be small fracions like he values shown above. This can give rise o loss of numerical accurac for floaing-poin arihmeic. In he case of his implemenaion fied-poin ineger mah is used which can mean ha some of he values will be runcaed o 0. To avoid hese problems i was decided o do he background updae onl ever number of N frames se o 6 for his applicaion. To keep he same rae of eponenial forgeing of he old background values 8.26 was modified slighl o ake ino accoun he fac ha no updae is done during he in-beween N-1 frames N i i f f µ σ σ µ µ 8.27 bkg frg bkg frg or and or where is onl calculaed once in he beginning so no era cos is added o he background updae process see 8.4 for how he eponenial weighed sum is derived Background Model Failure Even hough i is adaped regularl he background model ma fail if here is a sudden and large illuminaion change large camera movemen ec. The applicaion should deec such siuaions insead of jus inundaing laer processing phases wih deecion errors and false posiives.

30 The program keeps rack of how man piels are labelled as foreground. If his number eceeds 50% of he poenial piels ha is aking ino accoun he mask buil b he calibraion process o eliminae emp areas an error message is repored on screen and process erminaes. 8.8 Resuls In general he background subracion algorihm appears o perform quie well a leas on he PETS daases. The se of figures below conain seleced images ha give some indicaion of where he algorihm works well and where i fails in erms of he problems menioned in 8.6. The algorihm deecs shadows quie accurael as can be seen from Figure 8.11 where a person moves across he room and is ouline is eraced quie well. Wihou shadow deecion he objec s appearance would have been heavil disored. The algorihm is also robus o small-scale noise and small camera moion ~ SL[HOV Where he background subracion algorihm fails o suppress noise laer on he racking algorihm filers hese ou b using emporal consrains. Figure 8.11: Resuls: deecion of shadows shadow shown in gre; PETS-ICVS daase 123

31 Small scale camera movemen. Ligh gre piels are noise ha has been correcl eliminaed. Figure 8.12: Resuls: noise due o camera movemens camera movemen larger han a few piels cause some deecion errors black areas bu mos noise is eliminaed gre areas. false posiives One problem ha occurs mosl in he indoor scene is objec fragmenaion when objecs have he same colour and brighness as he background camouflage problem. Some of hese errors can be seen in Figure This is solved laer during he racking phase b means of region clusering. Par of he objec has same colour as he background whie wall and so is undeeced causing fragmenaion. In addiion some bis are mislabelled as shadow gre areas. Par of he objec has same colour and nearl same brighness as he background. Figure 8.13: Resuls: Camouflage and objec fragmenaion Oher deecion errors were generaed because of he foreground aperure problem presence of ghoss and background elemens ha were displaced b he objecs. Eamples of hese are shown in he remaining figures. These errors are evenuall eliminaed when he background adaps full o hem. 124

32 The original image wih no moion in i. The blue car moves awa eposing he ground below i. Two objecs deeced: he car on he lef and is ghos on he righ eposed ground. Figure 8.14: Resuls: The Ghos problem. moion cenral piels have he same colour and do no appear o be moving. Figure 8.15: Resuls: Foreground Aperure problem chair scene background moved b person chair s ghos Figure 8.16: Resuls: Moved background elemens 125