Automatic people density maps generation with use of movement detection analysis

Transcription

1 Auomaic people densiy maps generaion wih use of movemen deecion analysis M. Parzych 1, A. Chmielewska 1, T. Marciniak 1, A. Dąbrowski 1, A. Chrosowska 2, M. Klincewicz 2 1 Poznan Universiy of Technology, Poznan, Poland, {marianna.parzych, agaa.chmielewska, omasz.marciniak, adam.dabrowski}@pu.poznan.pl 2 BranchBrohers, Poznan, Poland, {agnieszka.chrosowska,michal.klincewicz}@branchbrohers.pl Absrac. The paper presens echnical aspecs relaed o he video compuer analysis of people movemen in he sales rooms. Auomaic generaion of he people densiy maps (also referred o as he hea maps ) is helpful for he analysis of cusomers and employees behavior in sale and office rooms adverising effeciveness research or racking movemens. Generaion of he densiy maps was realized wih he video recordings acquired by ypical video surveillance cameras. Daa processing was done off-line bu he achieved processing ime allows he real-ime implemenaion. Experimenal ess were carried ou a several poins of he cusomer services. Keywords: densiy (hea) maps, crowd analysis, behavior analysis, people racking I. INTRODUCTION Despie he developmen of online shopping, physical disribuion sale nework remains he mos imporan channel of he cusomer service. Analysis of psychology of he consumer behavior in he sale room aims o increase he sale success. Based on even shor-ime observaions, consuling companies officers can draw conclusions ha can very effecively improve funcioning of he sale area. Convenional soluions for he cusomer behavior analysis in a sore or an office require he use of manual analyical ools like quesionnaires or physical raffic couning by inerviewers. uch mehods are cumbersome and expensive, hus impossible o operae in a large number of rooms over a long ime. An ineresing and aracive opion is he use of he CCTV (closed circui elevision) sysem, since mos commercial spaces are equipped wih a video monioring. By his means manual cusomer behavior observaion can be suppored by advanced sysems for auomaic analysis of video sequences. Auomaic sysems, which are subjec of his paper, can be equipped wih inelligen image analysis faciliies ha allow moion deecion, classificaion of moving objecs, and behavior analysis. For example, deecion of people crossing a line in one or boh direcions can be used for couning hem. In case of he commercial sale space usage analysis, i is useful, from perspecive of he consumer behavior analysis, o generae people densiy maps (referred also o as he hea maps). A densiy map is a graphical represenaion of a able daa wih individual values represened as paricular colors. Very popular are web page hea maps used for displaying areas of web pages mos frequenly scanned by visiors. Generaion of densiy maps based on monioring daa is offered by several companies engaged in CCTV sysems e. g. MxAnalyics [1]. I, however, urns ou ha, due o he naure of regisraion (camera placemen, ype of he room, lighing, condiions, number of he analyzed individuals, ec.), a proper inerpreaion of he image and he corresponding real world is a relaively difficul ask, ofen impossible for users wih he commercial sofware only. Proper selecion of he opimal parameers for he image processing and he densiy map generaion is ypically no available in he commercial sofware. Therefore jus his is he subjec of our research. II. RELATED WORK The people densiy maps generaion is mainly relaed o he crowd analysis. Recenly Davies, Yin, and Velasin esed elemens of he image processing for an esimaion of he crowd densiy and crowd moion [2]. Esimaion of crowd densiy based surveillance monioring is provided in [3] wih he use of he exure informaion based on grey level ransiion probabiliies. Moion hea maps can be buil by accumulaion of binary BLOBs (binary large objecs) of moving objecs. Foreground objecs can be exraced by fusing he classificaion resuls from boh saionary and moving pixels [4]. Generaion of moion hea maps applied o he crowd monioring on he escalaor is presened in paper [5]. III. VIDEO ACQUIITION Proper localizaion of cameras for video sequences acquisiion is very significan for qualiy of he colleced daa. During he recording of es videos, which are used in our experimens wih he Hea Maps Generaor program, i is imporan ha he camera should be placed in he room a an appropriae heigh. The bes place is close o he ceiling, hus he people in he room are no behind each oher and every person is clearly visible. I has a posiive impac on he qualiy of he moving objec deecion algorihm. If angle of he camera view encompasses he enire room properly, he usage of a single camera is possible. In case of long and narrow rooms, wo cameras should be placed close o he ceiling in wo opposie corners of he room. uch a camera arrangemen allows well observaion of people locaion and posiion from boh sides.

2 The acquisiion rae in he Hea Maps Generaor program is irrelevan. The algorihm (in he offline mode) processes daa wo imes faser han he recording speed, which is 25 frames/sec (for pixels resoluion, processed by Inel Core i7-2620m The image resoluion is also no very significan. However, he size of he moving objecs (people) is imporan. According o he PN-EN : CCTV and alarm sysems norm [6], he objec in he moion deecion process should occupy a leas 10% of he heigh of he image (for he CCTV sysems wih he minimum image heigh of 480 lines Fig. 1). B. Gaussian Mixure Model Anoher background esimaion echnique is he Gaussian mixure modeling [10]. I is based on an assumpion ha a pixel has no correlaion wih oher pixels in an image [11]. Each of he background image pixels is esimaed by a mixure of k Gaussian disribuions. The probabiliy of pixels X a ime can be defined as k P( X ) ( X,, ), (2) i1 i, where k is he number of Gaussians (usually k is an ineger beween 3 and 5 [12]), X is a curren pixel value vecor, ω i, is he weigh of i-h Gaussian disribuion and i,, k i1 i, 1. (3) Fig. 1. Illusraion of he recommended minimum size of he observed objec according o PN-EN : CCTV & alarm sysems norm During recordings he KTC3510 color camera from KT-VIION wih TAV308DC 3-8mm IRIU F lens was used. The mos imporan camera feaures are: 1/3'' uperhad CCD (charge coupled device), movemen deecion algorihm, definiions of privacy zones, DNR (dynamic noise reducion) and srong ligh compensaion. For he video signal recording Falcon DDE04 MINI KPL was used. Four cameras can be conneced o his DVR (digial video recorder). The recordings can be saved in he H.264 forma a he maximum processing speed of 100 frames/sec. IV. MOVING OBJECT DETECTION TECHNIQUE Densiy map generaion requires a simple and accurae algorihm for moving objecs deecion. Below we shorly describe basic echniques for video sequence processing. A. Background subracion Background subracion is a popular mehod for moion deecion in video sequences, used, among ohers, by Liying [7]and Azeb [8]. This involves he deerminaion of he difference beween he curren frame and he reference frame (scene wihou moving objecs). Due o possible changes in he lighing condiions and geomery seings, regular updaing of he background image is necessary. Fas and low memory-cos background esimaion is he running Gaussian average [9]. This mehod is based on fiing a Gaussian probabiliy densiy funcion on n las images. The running average is compued as I ( 1), (1) 1 where I is an acual pixel value, µ -1 is a previous compued average and α is an empirically chosen weigh. The i-h Gaussian disribuion a ime is η(x,µ i,,σ, ), where µ i, is he mean, and Σ, represens covariance marix. For each pixel model parameers are adjused. The Gaussians are ordered by he value of ω/ Σ and he firs disribuion of Gaussians is seleced as he background model [11] b B arg min{ T}, (4) b where T is he hreshold, and B is a componen responsible for he background. i1 i, C. ubracion of consecuive video frames A echnique based on subracion of consecuive frames is a very fas, low memory, and low calculaion cos moion deecion algorihm. I has srong adapabiliy. However, his mehod canno exrac all relevan characerisics of pixels [13]. This mehod is ofen used as a supplemen o oher moion deecion algorihms [13]. D. Opical Flow Opical flow is a moion deecion mehod based on exracing he movemen of pixels in he image. a principle of he opical flow algorihms is as follows: by comparing consecuive frames of he video sequence, he correlaion beween hem is found. Then a vecor able called opical flow field is creaed. These vecors define he shif of pixels or regions caused objec moion in relaion o he camera. There are several mehods o esimae he opical flow fields [14]: region-based, energy based, phase-based and differenial. The mos common are: he Lukas-Kanade (LK) and Horn-chunck (H) mehods [15]. The LK mehod is based on comparing consecuive frames of he video, assuming brighness consancy beween hem. The opical flow equaion is as follows [16]: I I xu I yv 0, (5) where I(x,y,) represens he brighness a pixel locaion (x,y) and ime, I x, I y and I are he parial derivaives of I(x,y,), which respec o x,y and, (u,v) is he velociy corresponding o he opical flow of I(x,y,) and can be compued as

3 yy x xy y 2 u xx yy xy, (6) v xy y xx y 2 xx yy xy where xx, xy, yy, x and y are he produc sum i.e. I I over he small range Γ. ab a b as a single objec. I is no necessary o check addiional geomeric parameers in order o disinguish humans from ohers objecs (such as vehicles). The sudy included indoor commercial spaces, where were no moving objecs oher han humans. The racking algorihm is shown in Fig. 3. V. PEOPLE DENITY MAP GENERATOR - OFTWARE A. Algorihm The sofware o generae he densiy maps were prepared wih he use of he Microsof Visual udio 2010 developmen environmen and OpenCV library version 2.3. The program is wrien in C/C++. The algorihm processes he inpu video sequence frame-by-frame. a simplified block scheme of he program sages is shown in Fig. 2. (a) (c) (d) (e) (f) (g) (h) Fig. 3. Moving objecs deecion algorihm: (a) inpu, esimaed background, (c) subraced background, (d) grayscale, (e) hresholding, (f) dilaion, (g) erosion, (h) final resul. Fig. 2. chema of he program processing sages People racking is realized by he background subracion. Consecuive video frames subracion mehod was rejeced because i does no deec a person who has no changed is posiion in relaion o he previous frame. Background esimaion uses he moving average [17] of a variable number of frames. ubracion of consan background (se a beginning of he algorihm) would be sensiive o changes in background due o e.g. moving shadows and sreaks of ligh. Gaussian mixure and he opical flow mehods require a more complex implemenaion han he background esimaion using moving average, which provides a saisfacory high-performance moion deecion. The resul of he background subracion is hen convered o he grayscale and hresholded [18]. A he nex sep he image is dilaed [19] and erosed [20], o ensure ha every person is deeced Each deeced person is described by he coordinaes of he verices of a recangle surrounding i. Then he cener of he recangle is calculaed. In he wo-dimensional accumulaion marix he corresponding cell is incremened. To preven he daaype overflow, he values in he accumulaion marix are divided by 2 afer achieving given hreshold. This has no negaive impac on he final resul, because he algorihm does no calculae he absolue number of people of movemens during he sequence, bu he differences in people densiy in differen pars of he analyzed spaces. Afer he processing of he enire video sequence he densiy map is generaed based on he values in he accumulaion marix. During he accumulaion process he maximum values is searched for. The consan A is calculaed according o equaion A 255/ Maximum _ value (7) This consan is used for scaling he accumulaion marix (zero value excluding) o values (1;255). The nex sage is he reference image loading (Fig. 4 and 5), which has dimensions pixels. Each value in he accumulaion marix (afer scaling) corresponds o color in he adequae line in he reference image. Thanks his operaions we can visualize people densiy maps using differen references in an easy way.

4 (a) (a) (c) Fig. 4. Illusraive color scales for people densiy visualizaion The las sep of he algorihm is he overlay of he previously generaed people densiy map and display of he legend on he lef hand side on he inpu image. Fig. 6. Resuls of he Hea Maps Generaor program wih he use of: (a) logarihmic and inverse logarihmic scale An issue of he appropriae choice of he reference image inervals for long video sequences is illusraed in Fig. 6. Especially for needs of proecion of personal daa, (e.g. faces) he faces in illusraive images are blurred. As shown in Fig. 6 (a) he use of he logarihmic reference image for long video badly affecs on he visibiliy of he maxima, which in his case are locaed in he boom lef corner of he image. When we use he invered logarihmic reference image (Fig. 6 ), he queue in he long recordings is more visible. Thus, we can accuraely deermine where he larges people densiy was. (a) (c) Fig. 5. cale inervals used: (a) equal, logarihmic, (c) inverse logarihmic B. People densiy map visualizaion In order o chose he mos convenien densiy map visualizaion sandard (color scale) hree ses of reference images in differen colors were esed. Fig. 4 shows hree ypical color scale images wih equal inervals. For furher research we decided o use he hird scale shown in Fig. 4 (c). I illusraed he resuls in he bes way and was he mos inuiive. Examples of he use of scale images in he blue-green-yellow-red colors was shown in Fig. 6 and 9. For he previously chosen reference image, he images were prepared (Fig. 5), in which he color ranges are equal (a), logarihmic and inverse logarihmic (c). Each reference images are used for differen ypes of video sequences. a scale wih equal inervals is suiable for shor videos (several minues duraion), in which he moion disribuion is uniform. In he case of queues or narrow passage ways in shor clips we used logarihmic color inervals, while for long sequences (hours or days long) we used inverse logarihmic ranges of colors. C. Inerface for Hea Maps Generaor program The Hea Maps Generaor applicaion is equipped wih he graphical user inerface (GUI) as shown in Fig. 7. The program allows o load a video sequence from a file, choose he proper reference map depending on he inpu video lengh and size of he color poin depending on he inpu video resoluion. Recordings used during he es were equipped wih he imesamp. I was necessary o exclude imesamps from he processing area of he frame. There were also oher areas, like e. g. sliding doors, escalaors ha had o be excluded. Program allows o define he processing area. Upper imesamp and Lower imesamp opions should be used for excluding he appropriae marker. User-defined area opion enables o exclude any oher area from processing. Afer choosing his opion he sample frame is displayed and he user can indicae he excluded area by selecing 10 limiing poins (Fig. 8). The Generae Hea Maps buon sars he processing. The Open he las area coordinaes buon opens he ex file wih he coordinaes of he las excluded area, which can be used if muliple videos from one localizaion are esed. The Open he desinaion folder buon can be used o open he oupu densiy maps desinaion folder.

5 of queues, densiy of people a service poins, ec. Fig. 9 and 10 show he densiy maps compued for he one-hour long ime inervals during he working hours from one day a one locaion. Thanks o one-hour inervals during ess we could find ha he busy hour is beween 12:00 and 13:00. A ha ime he densiy map shows a long queue of people exended across he room. By means of hese resuls we could quickly and precisely deermine, which poin of he cusomer service handled he larges number of people. imulaneously, video sequences have been manually analyzed in erms of he people densiy and disribuion. The auhors have prepared summary describing he locaion and ime of increased raffic, he formaion of queues and occupancy of cusomer service poins. Comparison of he daa obained during he analysis of wached sequence o auomaically generaed hea maps has shown a clear convergence of resuls and same conclusions. Fig. 7. Hea Maps Generaor program inerface VI. REULT During he ess 64 hours of video sequences (saved in H.264 forma) from 4 localizaions were analyzed by our program wih he following opions: Inverse logarihmic reference image due o long recordings - an hour lengh. Poin size: 5 5 pixels, because inpu image had a pixels resoluion. In case of choosing a smaller poin size, unaesheic holes may occur in he densiy maps. Limiing he processing area enable opions: upper imesamp and user-defined area. Figure 8 shows an example of using he Limied user area opion in case of cash regisers and cusomer service poins, which should no be aken ino accoun in he deerminaion of he densiy map. During he ess Inel(R) Core(TM) i7-2620m 2,7 GHz was used. The video sequence processing ime is wo imes smaller han he real video ime. (a) (c) Fig. 9. Resuls for one-hour long ime inervals: (a) 8:00-8:59 am, 9:00-9:59 am, (c) 10:00-10:59 am Fig. 8. Example of using Limied user-defined area opion During he ess he video recordings were divided ino one-hour long sequences and full-day long sequences. Densiy maps analysis wih he one-hour long videos allows o designae busy hours as we can observe changes in he people disribuion in he room during he day: movemen VII. CONCLUION The developed sofware for generaion of he people densiy maps enirely fulfills expecaions defined by he specialiss from he consuling company. During he research a synergy effec among specialiss wih he engineering knowledge relaed o he imaging and video sequences echnologies and people wih he markeing compeencies proved o be visible and very valuable. A combinaion of hese experiences resuled in obaining he opimum densiy maps.

6 (a) (c) (d) (e) Fig. 10. Resuls for one-hour long ime inervals: (a) 11:00-11:59 am, 12:00-12:59 pm, (c) 1:00-1:59 pm, (d) 2:00-2:59 pm, (e) 3:00-3:59pm A helpful elemen in he implemenaion of he algorihms and preparing he program was he use of he OpenCV libraries ha conain basic funcions relaed o he image processing. The curren sofware version developed by he auhors is designed o operae off-line bu he processing speed allows o use i also for he on-line applicaions. This can be used, e. g. in he dynamic allocaion a a cash regisers and operaing saions a he cusomer service poins. REFERENCE [1] MxAnalyics Camera-Inegraed Video Analysis, hp:// (2013) [2] Davies A, Yin JH, Velasin (1995) Crowd monioring using image processing, Elecronics & Communicaion Engineering Journal, vol. 7(1), pp [3] Marana AN, Velasin A, Cosa LF, Loufo RA (1997) Esimaion of crowd densiy using image processing, Image Processing for ecuriy Applicaions, Diges No.: 1997/074, pp 11/1 11/8 [4] Li L, Huang W, Gu IYH, Tian Q (2003) Foreground objec deecion from videos conaining complex background, Mulimedia 03: Proceedings of he elevenh ACM inernaional conference on Mulimedia, pp 2 10 [5] harif MH, Ihaddadene N, Djeraba C (2008) Crowd behaviour monioring on he escalaor exis, ICCIT 2008, 11h Inernaional Conference on Compuer and Informaion Technology, pp [6] PN-EN norm: Alarm sysems CCTV surveillance sysems for use in securiy applicaions Par 7 (2003) [7] Lijing Z, Yngli L (2010) Moion human deecion based on background subracion, 2010 econd Inernaional Workshop on Educaion Technology and Compuer cience (ETC), vol.1, pp [8] Azab MM, hedeed HA, Hussein A (2010) a new echnique for background modeling and subracion for moion deecion in real-ime videos, 17h IEEE Inernaional Conference on Image Processing (ICIP) pp [9] Piccardi M (2004) Background subracion echniques: a review, 2004 IEEE Inernaional Conference on ysems, Man and Cyberneics, vol.4, pp [10] Liu B, Hui X (2010) Objec deecion based on Gaussian mixure background, 2010 Inernaional Conference on Compuaional Problem-olving (ICCP), pp [11] Yunchu Z, Zize L, Zengguang H, Hongming W, Min T (2005) An adapive mixure Gaussian background model wih online background reconsrucion and adjusable foreground mergence ime for moion segmenaion, IEEE Inernaional Conference on Indusrial Technology 2005, pp [12] Rui Y, Xuehua, hu Y (2009) Moving objec deecion based on an improved gaussian mixure background model, Inernaional Colloquium on Compuing, Communicaion, Conrol, and Managemen, vol.1, pp [13] Linkai C, Pinwei Z, Guangping Z (2010) Moving objecs deecion based on background subracion combined wih consecuive frames subracion, 2010 Inernaional Conference on Fuure Informaion Technology and Managemen Engineering, vol.2, pp [14] Barron JL, Flee DJ, Beauchemin, Burki TA (1992) Performance of opical flow echniques, 1992 IEEE Compuer ociey Conference on Compuer Vision and Paern Recogniion, pp [15] Pahirana PN, Lim AEK, Carminai J, Premarane M (2007) imulaneous esimaion of opical flow and objec sae: a modified approach o opical flow calculaion, 2007 IEEE Inernaional Conference on Neworking, ensing and Conrol, pp [16] Lei C, Hua Y, Takaki T, Ishii I (2011) Real-ime frame-sraddling-based opical flow deecion, 2011 IEEE Inernaional Conference on Roboics and Biomimeics (ROBIO), pp [17] AccumulaeWeighed definiion a: OpenCV v2.3 documenaion, OpenCV API Reference, Image Processing, Moion Analysis and Objec Tracking (2011) [18] Threshold definiion a: OpenCV v2.3 documenaion, OpenCV API Reference, Image Processing, Miscellaneous Image Transformaions (2011) [19] Dilae definiion a: OpenCV v2.3 documenaion, OpenCV API Reference, Image Processing, Image Filering (2011) [20] Erode definiion a: OpenCV v2.3 documenaion, OpenCV API Reference, Image Processing, Image Filering (2011) The paper has been parly financed wih he INDECT projec means.