Low Power Motio Detectio with Low Spatial ad Temporal Resolutio for CMOS Image Sesor Araud Verdat (), Atoie Dupret (2), Hervé Mathias (2), Patrick Villard (), Lioel Lacassage (2) () CEA/ LETI, Cetre de Greoble, 7 rue des Martyrs, 38054 Greoble Cedex 9 (2) Istitut d Electroique Fodametale, Bâtimet 220, Uiversité de Paris, 9405 Orsay Cedex araud.verdat@cea.fr Abstract Video surveillace aims at detectig uexpected idividuals or objects itrusio. Whe o motio is observed, commo motio detectio systems iduce huge power cosumptio, regardless of the scee activity. This paper presets algorithms for low power motio detectio, ad their possible implemetatio. The mai iterest is that they are able to adapt the sesor s behavior accordig to the scee activity. Relevat motio iformatio ca be extracted from images with lowered spatial ad temporal resolutio, with specific algorithms. By reducig the amout of data to aalyze ad spatial ad temporal redudacy, a drastic reductio of power cosumptio ca be achieved.. Itroductio I a cotext of embedded video surveillace, with steady camera, power cosumptio reductio costitutes a key poit. The tremedous amout of data processed idepedetly of scee activity i commo image sesors systems (i.e. image sesors associated to a DSP or FPGA), iduce huge waste of power. Hece a promisig approach cosists i wakig up the system whe a evet occurs i the scee. Detectig evets is the equivalet to motio detectio. For several years, may studies have focused o motio detectio problem. Today s submicro processes eable the pixel level implemetatio of some processig steps that used to be doe by exteral circuits while keepig reasoable Fill Factor ad pixel area. May architectures for motio detectio ad trackig have bee proposed. Here are a few examples of possible detectio methods: Optical flow measuremet has bee explored i [] ad [2]. I these approaches, spatial ad temporal gradiet are calculated to solve optical flow equatio. Each pixel is so computed, icreasig at the same time the amout of data for larger images. Frame differece is made i [3]. Each pixel data is coded with matissa ad expoet, allowig wide dyamic rage, ad compariso betwee frames is doe usig expoet bits. Therefore, small amplitude motio is difficult to detect with this method. I the precedet cases, all pixels are used to perform computatios. I order to reduce the amout of data to be processed, ad so to lower power dissipatio, multiresolutio architectures have bee developed leadig to variable spatial acuity imagig, i.e. Regio Of Iterest (ROI). I [4] ad [5], authors preset architectures able to perform multiresolutio outside the pixel array, by sharig charges o baks of capacitors, but icreasig die area. I [6] ad [7], architectures allowig charge biig betwee pixels iside the array are described, with additioal trasistors iside ad betwee pixels. Multiresolutio is achieved by summig pixels currets i [9]. I these approaches with Regio Of Iterest (ROI), [4] to [9], exteral chips are ecessary to compute addresses of foveated area, icreasig power cosumptio. I [0], a computatioal circuit performs variable resolutio by groupig pixels accordig to their brightess, usig quadtree algorithm, but this method requires the whole pixel array to be read. Image sub-samplig, i a cotext of embedded video surveillace, has bee explored i []. A architecture allowig two operatig modes has bee developed. Without scee activity, sesor stays i low power mode, ad checks the brightess of % of all the pixels. If the brightess sigificatly chages, the sesor switches to active mode ad all pixels are read. However, this sub-samplig
techique sigificatly reduces fill factor [8]. O the cotrary, fill factor remais uchaged with macropixels. Our goal is to develop a imager featurig two operatig modes, oe beig a power savig mode with low spatial ad temporal resolutio, able to wake up upo a scee evet, ad the other beig ormal or active mode, performig trackig o target. To face differet situatios foud i video moitorig, a implemetatio o a programmable architecture has bee chose (flexible ad adaptable architecture). This paper presets the studies performed cocerig the wake up fuctio, i.e. the trasitio from power savig to active mode. Figure illustrates the two operatig modes, with a scee with high resolutio o movig cars ad low resolutio o static parts. Low Power Mode Active Mode No Low spatial ad temporal resolutio Motio? High resolutio o ROI Trackig Figure Two modes depedig o scee activity with high resolutio o movig targets ad lowresolutio for fixed part of the scee Part 2 presets our power savig strategy, based o a spatial ad temporal data amout reductio, ad the cosidered low resolutio cofiguratios. I Part 3, we describe the associated motio detectio algorithms that we foud potetially iterestig. We the (part 4) preset the obtaied simulatio results to determie the best solutio. Fially (part 5), we describe the selected implemetatio architecture ad we estimate the power cosumptio of the selected algorithms. 2. Power savig strategy Yes Predefied detectio mode Programmable architecture Withi video moitorig tasks, differet power dissipatio sources may be distiguished: Image acquisitio, image readig ad A/D coversio iduce a power dissipatio that is proportioal to the pixel rate. Image processig: its related power dissipatio ot oly depeds o pixel rate but also o the processed algorithm complexity. The relative impact of the precedet sources o the global power dissipatio depeds o the implemetatio architecture. The power dissipatio lowerig may thus be addressed at software ad hardware level. For our leftbehid video moitorig system, the software-related dissipatio savig is obtaied by reducig the spatial ad/or temporal pixel resolutio i order to tremedously reduce the pixel rate i low power mode. The associated motio detectio algorithm has to be selected so as to have a umber of micro-istructios per pixel as low as possible ad yet beig efficiet eough to detect the aimed evets. Differet ways of lowerig the spatial resolutio have bee studied. They are either based o published works or ew proposals. Figure 2 shows the cosidered cofiguratios for a give image. The processed images are i each case split ito a umber of blocks correspodig to the desired low resolutio. 4 differet types of blocks have bee cosidered: Macropixels (Fig.2 a)): blocks cotaiig the spatial average of their costitutig pixels; Decimated Macropixels (Fig.2 b)): a subset of macropixels are processed; Uiform stripes (Fig.2 c)), cotaiig the average of each colum of pixels. Decimated pixels (Fig.2 d)): blocks where oly the cetral pixel is processed; a c Figure 2 Origial image ad correspodig low-resolutio images obtaied with the differet methods cosidered For all the cosidered cofiguratios, except decimated pixels, all the pixels of a give active block remai active. A spatial average is performed for each block at o extra power cosumptio by coectig the pixel capacitors i parallel. Oly the spatial average (oe value per block) is the processed to perform motio detectio. These three ew cofiguratios have d b
thus a slightly higher power cosumptio tha the decimated pixels solutio, but with far better results i terms of efficiecy, as will be see i part 4. 3. Studied Low Power motio detectio algorithms We chose to focus o algorithms aimig at wakeig Regios Of Iterest (ROI) whe movemet is detected. They ideed allow limitig the power dissipatio durig the active mode ad easig the operator work. The cosidered algorithms are based o Recursive Average (RA), preseted i [2], with two modified versios with improved robustess (Sigma Delta ad Recursive Average with estimator) at the cost of higher complexity ad thus additioal power dissipatio. I order to estimate the associated power dissipatio, each tested algorithm is described so that the umber of ecessary micro-istructios may be determied. The followig otatios have bee used: the idex represets the frame umber, the curret gray level value for the cosidered block is amed S ad a motio estimator is computed ad amed. If this estimator becomes larger tha a predefied threshold, which depeds o the kid of evet to be detected, the correspodig ad some eighborig blocks are switched to high resolutio (all their pixels are processed). Recursive average (RA) The priciple of recursive average is to estimate backgroud (Mrec ()) for each data ad to compare it to the curret value. N is a fixed coefficiet that is chose with respect to the frame rate ad typical target speed: Mrec = Mrec Mrec + S () N N = Mrec S Modified Sigma Delta (SG2) The Σ algorithm allows estimatig backgroud with elemetary icremet ad decremet. Two estimators, Msg ad V, represetig temporal activity ad stadard deviatio of each data, are so geerated: = Msg S if > 0 Msg = Msg if < 0 Msg = Msg + if V < = + N V V if V > V = + if > NV motio for motio detectio is ot fixed here ad depeds o N. Recursive average with estimator(ra+e) To improve robustess, we have developed this origial algorithm based o Recursive Average combied with a estimator, like the oe used i SG2 algorithm, which iduces more stability with respect to oise. The two combied estimators ideed allow pass bad filterig istead of oly low-pass filterig i the case of Recursive Average. Mrec = Mrec Mrec + S N N δ = Msg S if δ if δ 4. Results > < = Msg Msg 0 Msg = Msg 0 + = Mrec Msg Simulatios have bee performed for the cosidered algorithms with each kid of proposed low-resolutio cofiguratios. For these simulatios, a sigle image sequece, that is represetative of the aimed applicatios, has bee used. It takes place i a outdoor area where cars are ridig a street behid barriers ad iside a car park. A tree with rustlig foliage is also preset ad has bee icluded i false alarm parts of the scee. The origial resolutio of the sequece is 480x640. Table Percetage of processed amout of data with respect to origial sequece resolutio for tested low resolutios methods x640 2x6 20x20 40x40 48x64 Macropixels - 0.06% 0.3% 0.52% % Macropixels Decimatio by 2-0.03% 0.065% 0.26% 0.5% Colums 0.2% - - - - Pixel Decim. - 0.06% 0.3% 0.52% % Several levels of low resolutio have bee tested, as show o Table, which idicates, for each cofiguratio, the percet of data amout processed with respect to the origial sequece. Sice, for a give algorithm, the power cosumptio is directly liked to the amout of data processed, this gives a good idea of how much power will be saved with each cosidered cofiguratio. For example, i the case of 2x6 low-resolutio obtaied usig macropixels decimatio by 2, (2x6)/2
data are processed, which represets 0.03% of data with respect to full resolutio. A data amout reductio of 99.97% is so achieved. A comparable power reductio ca be expected The impact of low resolutio o efficiecy is less straightforward sice spatial filterig is performed by the resolutio lowerig. The best compromise betwee power savig ad efficiecy depeds o the cosidered algorithm. I order to have a better efficiecy i motio detectio, a rig of fixed size aroud the processed block has bee defied. If a block becomes active, all the blocks ecompassed by the rig also switch to active mode. Two rig sizes have bee tested: 3x3 ad 5x5 blocks. I the case of stripes cofiguratios, a bad of adjacet stripes has bee used istead. Durig the simulatio, each time a block is activated, the correspodig iformatio is stored i a vector. At the ed of the simulatio, this vector is compared to a referece structure idicatig where ad whe motio should be detected. The umber of false alarms (blocks which crossed threshold i areas where there is o motio) ad of ot detected evets (blocks which should have crossed threshold i areas where there is motio) ca thus be couted. The algorithm latecy ca also be determied i this maer. The efficiecy of the differet tested cofiguratios ca the be estimated by subtractig the umber of false alarms to the umber of rightly activated blocks. A result of 00% correspods to the detectio of all the desired movemets i the sequece ad o false alarm. For these evaluatios, frame rate was 20fps. A compariso of the obtaied results has cofirmed that, depedig o the cosidered algorithm, the chose low resolutio cofiguratio has ot always the same impact o the efficiecy. The best results obtaied for the differet algorithms are show i Table 2, which clearly stresses the efficiecy of RA+E algorithm with Macropixels cofiguratio. For the Recursive Average algorithm (RA), the results obtaied with the decimated pixels ad the macropixels cofiguratios have bee added i order to show the improvemet brought by the proposed low resolutio cofiguratios. Table 2 Results of best algorithms with best cofiguratios (M : Macropixels, DM : Decimated Macropixels, D: Decimated Pixels) 3x3 Rig Algorithm 2x6 20x20 40x40 48x64 Mea value RA (DM) 95.76 85.38 87.20 76.60 86.2 RA (M) 88.07 80.70 8.98 74.68 8.4 SG2 (M) 88.0 8.46 78.95 7.06 79.9 RA+E (M) 85.53 8.72 82.04 77.58 8.7 RA (D) 29.62 3.3 47.78 45.53 38.6 With a rig of 3x3 blocks, RA+E globally produced the best results with a average evaluatio of 86.2%. The best mark obtaied by this algorithm is with a 2x6 resolutio ad Macropixels cofiguratio. I this case, 00% of evets are detected with oly 4.2% of false alarms. Icreasig rigs sizes has globally brought worse results sice larger areas are the switched to high resolutio. As a cosequece, the umber of false alarms icreases. The Decimated Macropixels cofiguratio is more efficiet with the RA algorithm. The choice of the mesh size is however very importat: it must be smaller tha the to-be detected objects, i order to have eough referece poits o the targets, but it must also be larger tha the sources of oise (e.g.: the leaves i the trees for our sequece) i order to reduce the umber of false alarms. Sice more stability is achieved with our ew RA+E algorithm, the Macropixels cofiguratio is preferable i this case i order to maximize the umber of referece poits to switch i high resolutio o targets. Detectio is here excellet with few false alarms sice oisy parts (foliage) are efficietly filtered. Cocerig the modified SG2 algorithm, Macropixels cofiguratio is also required but results are less satisfactory because of a less efficiet detectio. Usig larger resolutios lowers the efficiecy of the spatial filterig performed de facto by the low resolutio cofiguratios. The macropixels are composed of fewer pixels ad the correspodig referece poits are less stable. Most of the time, usig larger resolutios thus gives bad results, with a icreased umber of false alarms. The tred is the opposite i the case of decimated pixels. With the Uiform Stripes cofiguratio associated to the RA algorithm for example, few false alarms are geerated (9.4%) ad detectio is quite performig (89.45%). The mai drawback of this cofiguratio is that, whe switchig to high resolutio, a large part of
the image becomes uecessarily active, thus iducig uecessary power dissipatio. Furthermore, motio detectio is oly efficiet with targets movig perpedicularly to the colums. The Uiform Stripes cofiguratio is also much harder to hadle i terms of threshold determiatio sice a give colum covers a large part of the image ad may cotai static parts ad oisy parts (foliage i our case). I this case, the correspodig threshold has to be higher to get a reasoable umber of false alarms, which reduces the sesitivity for the static parts covered by the give stripes. Sice power dissipatio is a importat issue for our architecture, these evaluatios must be balaced by the power cosumptio associated to each algorithm. If the best results i terms of efficiecy have bee obtaied with the RA+E algorithm ad a 2x6 resolutio, the correspodig power cosumptio is ot the lowest of our tested cofiguratios. Eve if a 99.94% data amout reductio has bee achieved, this algorithm remais more computatioally expesive tha the RA algorithm that requires fewer istructios. Sice the latter gets very good results for the same resolutio ad the decimated macropixels cofiguratio, it might thus be a iterestig choice for our wakig-up algorithm i terms of power cosumptio/efficiecy compromise. The fial choice depeds o the hardware implemetatio ad the applicatio costraits. Tests were performed to determied the respose time, with 20x20 macropixels resolutio. The average time betwee the arisig of targets eter i the macropixels ad the switchig to high resolutio, was respectively of 0.4s for the RA ad 0.63s for the RA+E algorithms. As stated earlier, the RA+E algorithm ideed requires more computatioal steps. However, the larger macropixels sizes, the slower macropixels variatios because of the fewer movig pixels cotributios i blocks spatial averages. As a cosequece, icreasig macropixels size implies to lower the temporal resolutio i order to get higher macropixels variatios betwee each frame. memory poits is associated, where k is the umber of memory elemets per pixel. The so-formed matrix is bordered o oe side by a vector of N processors. A colum of multiplexers selects the colum of pixels or memories to be used by the processor. A sequecer, implemeted for example by a digital IP CPU, delivers the successive processors istructios. Figure 3 Sesor architecture The processor is a switched capacitor aalogue computig uit show o Figure 4. For each processor istructio, the switches cofiguratios for the OTA ad for the associated aalogue registers are fixed. The multiplicatio (MAC) of a aalogue value oly requires 4 clock cycles with the followig sequece of operatios: writig o the left-most capacitor, clearig the other (weighted) capacitors ad settig them ito the feedback loop of the OTA, accordigly to the coefficiet of the MAC. The various operatios required by the cosidered algorithms ca be performed with this parallel architecture. Low-level real-time image processig for trackig i active mode is also possible. I additio, these processors ca be set i idle mode whe they are ot beig used. 5. Chose implemetatio I order to perform both motio detectio i power savig mode ad trackig i active mode, a aalogue programmable computatioal uit is cosidered. Aalogue based computatioal system offer high compactess ad low power cosumptio. The cosidered computatioal uit is based o [3]. The SIMD machie preseted (Figure 3) icludes a NxM photosesors array to which a array of Nx(kM) Figure 4 Processig uit The computatioal cost of the preseted algorithms i power savig mode is proportioal to the amout of data. This power savig must be balaced by the fact that performig a local spatial average has a cost i terms of power dissipatio (charges ad discharges of pixels capacitace), but which is quite egligible compared to power savig iduced by the reductio of
data to compute (charges ad discharges of bus capacitace ad OTA). For the Recursive Average algorithm, our architecture, workig at 40MHz, requires 0 microistructios per pixel. Hece a computatio time of 250s per pixel. For the sequece used i our simulatios, the mea power dissipatio per microistructio is 4W for a frame rate of 40fps. With a 99.94% (2x6) data reductio from the aalyzed full resolutio scee (480x640), the total power dissipatio would thus be of 27µW (92x40W) for Recursive Average algorithm, as log as the system stays i power savig mode. For Recursive Average with Estimator algorithm, our architecture requires 20 micro-istructios per pixel. The correspodig power dissipatio would thus be of 54µW. 6. Coclusio Specific algorithms, which aimed at detectig motio with low spatial ad temporal resolutio, have bee studied o a give video sequece. Origial low resolutio cofiguratios like Decimated Macropixels or Uiform Stripes have bee explored ad a ew algorithm givig excellet results has bee preseted. The cosidered image sesor will allow drastic power cosumptio reductio i the absece of motio, with a more tha 99% reductio of the amout of processed data compared to full resolutio images. A very iterestig hardware implemetatio has also bee proposed with very low power cosumptios for the power savig mode The proposed low resolutio techiques should also be explored for image compressio i order to remove spatial ad temporal redudacy o static part of images. Future works will icorporate trackig i highresolutio mode with actualizatio of ROI o targets (trackig) ad switchig from high resolutio to lowresolutio mode. For example, i order to reduce the processig uit load, local processig allowig iterpixels cooperatio for ROI actualizatio ad i-pixel calculatio of recursive average will be explored. 7. Refereces [] Moii, A. ad al., A isect visio-based motio detectio chip, Solid-State Circuits, IEEE, Volume 32, Issue 2, Feb. 997 Page(s):279 284. [2] Mehta, S.; Etiee-Cummigs, R.; Normal Optical Flow measuremet o a CMOS APS imager, Circuits ad Systems, ISCAS '04, IEEE, Volume 4, 23-26 May 2004 Page(s):IV - 848-5 Vol.4. [3] Milirud, V. ad al., A wide dyamic rage CMOS active pixel sesor with frame differece, Circuits ad Systems, ISCAS 2005, IEEE, 23-26 May 2005 Page(s):588-59 Vol.. [4] Kemey, S.E. ad al., Multiresolutio image sesor, Circuits ad Systems for Video Techology, IEEE, Volume 7, Issue 4, Aug. 997 Page(s):575 583. [5] Pai, B. ad al., Dyamically recofigurable visio with high performace CMOS active pixel sesors (APS) Sesors", Proceedigs of IEEE, Volume, 2-4 Jue 2002 Page(s):2-26 vol. [6] Saffih, F. ad al., Multiresolutio CMOS Image Sesor, Techical Digest of SPIE Opto-Caada 2002, Ottawa, Otario, Caada 9-0, May 2002, p.425. [7] Massie, M. ad al., Visio chip for avigatig ad cotrollig Micro Umaed Aerial Vehicles, Circuits ad Systems. ISCAS '03. Volume 3, 25-28 May 2003 Page(s):III-786 - III-789 vol.3. [8] McCarley, P.L. ad al., Large format variable spatial acuity superpixel imagig: visible ad ifrared systems applicatios, SPIE 2004, Orlado, FL. [9] Coulombe, J. ad al., Variable resolutio CMOS curret mode active pixel sesor, Circuits ad Systems, Proceedigs. ISCAS 2000 Geeva, IEEE, Volume 2, 28-3 May 2000 Page(s):293-296 vol.2. [0] Artyomov, E.; Yadid-Pecht, O.; Adaptive multiple resolutio CMOS active pixel sesor, Circuits ad Systems, 2004. ISCAS '04, Volume 4, 23-26 May 2004 Page(s):IV - 836-9 Vol.4. [] Sug-Mi Soh ad al., A CMOS image sesor (CIS) architecture with low power motio detectio for portable security camera applicatios, Cosumer Electroics, IEEE, Volume 49, Issue 4, Nov. 2003 Page(s):227 233. [2] J. Richefeu; A. Mazaera; Motio Detectio with smart sesor, 9th Cogress Youg Searchers i Computer Visio - ORASIS 2005, Clermot-Ferrad, May 24-27. [3] Moutault, S.; Mathias, H.; Klei, J.-O.; Dupret, A.; A improved aalog computatio cell for Paris II, a programmable visio chip, Circuits ad Systems, 2004. ISCAS '04, Volume 5, 23-26 May 2004 Page(s):V-453 - V-456 Vol.5.