FAST EFFICIENT ALGORITHM FOR ENHANCEMENT OF LOW LIGHTING VIDEO

FAST EFFICIENT ALGORITHM FOR ENHANCEMENT OF LOW LIGHTING VIDEO Xuan Dong, Guan Wang*, Yi (Amy) Pang, Weixin Li*, Jiangao (Gene) Wen, Wei Meng, Yao Lu** Deparmen of Compuer Siene, Tsinghua Universiy, Beijing, China dongxuan8811@gmail.om, pangy@mails.singhua.edu.n, jwen@singhua.edu.n mengw08@mails.singhua.edu.n *Deparmen of Compuer Siene and Tehnology, Beihang Universiy, Beijing, China {wg1989john, lexiwzx}@gmail.om **Deparmen of Eleroni Engineering, Tsinghua Universiy, Beijing, China luyao11175@gmail.om ABSTRACT We desribe a novel and effeive video enhanemen algorihm for low lighing video. The algorihm works by firs invering an inpu low-lighing video and hen applying an opimized image de-haze algorihm on he invered video. To failiae faser ompuaion, emporal orrelaions beween subsequen frames are uilized o expedie he alulaion of key algorihm parameers. Simulaion resuls show exellen enhanemen resuls and 4x speed up as ompared wih he frame-wise enhanemen algorihms. Index Terms low lighing video enhanemen, dehazing 1. INTRODUCTION As video ameras beome inreasingly widely deployed, he problem of video enhanemen for low lighing video has also beome inreasingly aue. This is beause alhough amera and video surveillane sysems are expeed o work in all lighing and weaher ondiions, he majoriy of hese ameras are no designed for low-lighing, and herefore he resuled poor apure qualiy ofen renders he video unusable for riial appliaions. Alhough infrared ameras are apable of enhaning low-ligh visibiliy in [1] and [2], hey suffer from he ommon disadvanage ha visible objes mus have a emperaure higher han is surroundings. In many ases where he riial obje has a emperaure similar o is surroundings, e.g. a big hole in he road, infrared ameras are no very useful. On he oher hand, onvenional image proessing and radio proessing ehniques suh as hisogram equalizaion may no work well eiher in many ases, espeially for real ime proessing of video sequenes. In his paper, we propose a novel, simple and effeive enhanemen algorihm for low lighing video. We show ha afer invering he low-lighing video, he pixels in he sky and disan bakground regions of he invered video always have high inensiies in all olor (RGB) hannels bu hose of non-sky regions have low inensiies in a leas one olor hannel, similar o he ase of video apured in hazy weaher ondiions. Therefore, we apply sae-of-he-ar image de-hazing algorihms o he invered video frames and show ha he ombinaion of invering low-lighing videos and applying de-hazing on hem is an effeive algorihm for enhaning low-lighing videos. To failiae faser implemenaion, we furher uilize emporal orrelaions beween subsequen video frames for he esimaion of key algorihm parameers, resuling in 4x speed-ups as ompared wih he onvenional frame-by-frame approah. 2. ENHANCEMENT OF LOW LIGHTING VISION VIDEO We noie ha low lighing video enhanemen has muh in ommon wih video haze removal, beause afer invering he inpu low-lighing videos, he resuled videos look very muh like videos aquired in hazy lighing ondiions. Figures 1 and 2 provided some examples of low lighing videos, invered low lighing videos and videos aquired in hazy lighing ondiions. As menioned in [3], in hazy ondiions, he inensiies of bakground pixels are always high in all olor hannels, bu he inensiies of he main objes, e.g. houses, vehiles and people are usually low in a leas one hannel beause of he olor, shadow and e.. To es wheher he invered low lighing video has he same propery, we ompue he minimum inensiy of all olor (RGB) hannels for eah pixel of he invered videos/images for 30 randomly seleed (by Google) low lighing ones, some of whih are shown in he op row of Figure 2, as well as 30 random haze videos/images as shown in he boom row of Figure 2. Figure 3 shows he hisogram of he minimum inensiies of all olor hannels of all pixels for he 30 invered low lighing videos/images and Figure 3 is he same hisogram of he 30 haze videos/images. Comparing Figures 3 and 3, we find ha he wo hisograms exhibi grea similariies, and ha more han 80% of pixels in boh he invered and he haze ases have high inensiies in all olor 978-1-61284-350-6/11/$26.00 2011 IEEE

() (d) (e) Fig. 1. low lighing video inpu I. invered resul R from inpu I. () marked image: pixels wih low inensiy in a leas one olor (RGB) hannel are in green. (d) de-haze oupu J using equaion (7). (e) final oupu E afer invering image J. obain he oupu video frames. Thus, for he inpu low lighing video frame I, we inver i using: R ( = 255 I (, (1) where is he olor hannel (RGB). I ( is he inensiy of a olor hannel of pixel x of he low lighing video inpu I. R ( is he same inensiy of invered image R. The haze removal algorihm we use is based on [3], in whih a hazy Fig. 2. Top: low lighing examples. Middle: invered oupus of he Top ones. Boom: haze examples. Fig. 3. hisogram of all pixels olor hannels minimum inensiies of he 30 invered low lighing videos/images and he 30 haze videos/images. hannels (ermed high inensiy pixels ), inluding almos all of he pixels in sky regions. As for he pixels of non-sky regions, he inensiies are always lower in a leas one of he olor hannels (ermed normal inensiy pixels ). In Figure 1(), we mark he normal inensiy pixels in green. A pixel is designaed as a normal inensiy pixel if one of is hree olor hannels inensiies is lower han 180. Clearly, mos of he marked pixels are in a non-sky region or obje, e.g. vehiles, sidewalks, lighs and e. The above saisis srongly suppor our inuiion ha he invered low lighing videos are similar o haze videos. Based on he above observaion, we propose a novel low lighing video enhanemen algorihm by applying he inver operaion on low lighing video frames, and hen performing haze removal on he invered video frames, before performing he inver operaion again o image is modeled as shown in (2): R( = J ( + A(1 ), (2) where A is he global amospheri ligh. R( is he inensiy of pixel x he amera ahes. J( is he inensiy of he original objes or sene. desribes how muh peren of he ligh emied from he objes or sene reahes he amera. The model is also menioned in [4-7]. The riial par of all haze removal algorihms is o esimae A and from he reoded image inensiy I( so ha hey an reover he J( from I(. When he amosphere is homogenous, he an be expressed as: βd ( = e, (3) where β is he saering oeffiien of he amosphere, and d( is pixel x s sene deph. In he same image, where he β is onsan, is deermined by d(, he disane beween he obje and he amera. In [3] is esimaed using: R ( y) = 1 ω min ( min ( )), (4) { r, g, b} y Ω ( A whereω is 0.8 in his paper. Ω( is a loal blok enered a x and he blok size is 9 in his paper. We uilize his algorihm o esimae in his paper. To esimae he global amosphere ligh A, we firs sele 100 pixels whose minimum inensiies in all olor (RGB) hannels are he highes in he image, and hen among he pixels, we hoose he single pixel whose sum of RGB values is he highes. The RGB values of his pixel RGB values are used for A. Thus, aording o he equaion (2), we an of ourse reover he J( as follows:

R( A J ( = + A, (5) however, we find ha dire appliaion of equaion (5) migh lead o under-enhanemen for low-lighing areas. To furher opimize he alulaion of, we fous on enhaning he Region of Ineress suh as houses, vehiles and e. while avoid proessing bakground e.g. sky regions in low lighing videos. To adjus adapively while mainaining is spaial oninuiy, so ha he resuled video beomes smooher visually, we inrodue a muliplier P( ino (5), and hrough exensive experimens, we find ha P( an be se as: 2, 0 < < 0.5 P ( =, (6) 1, 0.5 < < 1 hen he reovery equaion beomes: R( A J ( = + A, (7) P( he idea behind equaion (7) is he following. When is smaller han 0.5, whih means ha he orresponding pixel needs boosing, we assign P( a small value o make P( even smaller so as o inrease he RGB inensiies of his pixel. On he oher hand, when is greaer han 0.5, we refrain from overly boosing he orresponding pixel inensiy. For low-lighing video, one J( is obained, he inver operaion of (1) is performed again o produe he enhaned image E of he original inpu low ligh video frame. This proess is onepually shown in Figure 1. 3. ACCELERATION OF PROPOSED VIDEO ENHANCEMENT PROCESSING ALGORITHM 3.1. Aeleraion Using Moion Esimaion (ME) In his seion we propose an approah o aelerae he proessing of he algorihm in he previous seion. In experimen, we find alulaing oupies nearly 70% of he ompuaion, whih means he fous of aeleraion ehnique beomes aeleraing he alulaion of. Our ehnique is based on he orrelaions beween emporally suessive frames. Sine is deided by he inensiies of pixels of eah frame, he values of of o-loaed pixels should also by emporally orrelaed in emporally neighboring frames. In pariular, if he values of are sored from he previous frame, and if he pixels in he urren frame are deermined o be suffiienly similar o he o-loaed pixels in he previous frame, he sored values ould be used as he for he orresponding pixels in he urren frame, hereby by-passing he alulaion of a signifian porion of he values. To deermine if a pixel is suffiienly similar o is oloaed ouner par in he previous frame, we use he same ehnique in he moion searh proess of video oding. We firs divide he inpu frames ino Groups of Piures (GOPs), wih eah GOP saring using an Inra oded frame (I frame), in whih all values are alulaed. In he remaining frames, i.e. P frames, ME is performed. In deail, eah frame is divided ino non-overlapping 16x16 maro bloks (MBs), for whih a moion searh using he disorion rierion Sum of Absolue Differenes (SAD) are ondued. If he SAD is below he predefined hreshold, he MB will be enoded in iner-mode and he alulaion of for he enire MB is skipped by assigning he bes mah MB s direly. Oherwise, he MB will be enoded in inra-mode and sill needs o be alulaed. In boh ases, he values for he urren frame are sored for possible use in he nex frame. We ake wo ME algorihms in his paper. Firs, we propose a simple ME mehod named Co ME whih only alulaes he disorion value of he (0, 0) moion veor for he urren MB. If he SAD value is larger han a predefined hreshold T, he of he urren MB is alulaed. Oherwise, he alulaion is by-passed. The Co Me algorihm is easy o implemen and os lile sorage spae. The oher ME mehod is EPZS ME algorihm proposed in [12], whih is more diffiul o implemen and oupies more sorage. However, i performs beer han Co ME in oal speedup beause i an provide more aurae ME informaion o mah he referene frames hus o redue more alulaion of. In EPZS ME algorihm, he firssage SAD hreshold T is fixed and predefined. The following hreshold hanges as follow: ( MSAD, MSAD, MSADn ) bk T = a min 2, + k k 1, (8) where a k and b k are fixed value. Based on he above algorihm, we define he speed up raio as: + g, (9) Speedup = = ( k) O oher i g 1 A oher + i + p k = 1 is he ime of enhanemen proessing exep oher alulaing. g is he GOP size. p (k) is he ime of h alulaing for he k frame in a GOP. i is he ime of alulaing for he I frame in a GOP. p (k) varies beause of differen onen of P frames.

To analyze he equaion of speedup more easily, we make wo assumpions. Using he average ime p of alulaing for he k h frame in a GOP subsiues for (k). Comparing he ime of alulaing, we ignore p oher. (8) hen beomes: g i Speedup = = + g Where i =. p i p 2, (10) g + For he same video sequene where i is ons and p is affeed by he SAD hreshold, frame onen and GOP size, speedup is an inverse proporion funion of g. We run he proposed aeleraion algorihm for he same video sequene wih differen GOP sizes, and he resul is shown in Figure 4. The inverse proporion funion fiing urve mahes he experimenal resuls very well. Moreover, equaion (9) explains how he speedup hanges wih differen GOP sizes and ondus us o se he appropriae GOP size for a needed speedup and qualiy of he enhaned video. In addiion, our experimen shows afer adoping he above algorihm he ompuaional ime of dwindles a lo while he ME ime oupies muh of he whole proessing ime. Thus, in nex subseion, we propose a fas SAD algorihm o aelerae ME. Fig. 4. Speedup of proposed fas enhanemen algorihm. 3.2. Aeleraion Using Fas SAD Algorihm Experimenal resuls demonsrae he alulaion of SAD akes 60 peren of he whole ME ime, i.e. 40 peren of ompuaion os of he whole low lighing enhanemen algorihm. Thus, i is desired for us o propose a fas SAD algorihm. Some exising fas SAD algorihms are based on advaned hardware e.g. GPU. However, as disussed in [8, 9, 10], in BBME, all he disorion rierions inluding SAD are based on he inheren assumpion ha moion fields of video sequenes are usually smooh and slowly varying and eah MB onains only one rigid body and radiional moion. In mos ases where he assumpion is orre, he onvenional disorion rierion seems heavy and unneessary. Moreover, in some oher ases, i will no be he rue moion veor when he SAD ges is minimum if here is more han one rigid body or he moion is unradiional. In hese ases, he more pixels are sampled he greaer error he SAD algorihm will probably lead o. However, if we adop sub-sampling mehod in our fas SAD algorihm, no only he unneessary ompuaion in mos ases an be redued bu also he poor performane of BBME in omplex moion ondiions an be improved. Alhough he fewer samples will redue he horizonal and verial soluion and have an effe on he auray, he error urns ou o be sligh and aepable in our algorihm if users do no have sri requiremen of video qualiy bu need he algorihm as fas as possible. () Fig. 5. Fixed subsampling paerns of fas SAD algorihm sandard 4:1 paern. Three-Coeffiien paern. () proposed paern in his paper There are many subsampling paerns proposed e.g. he adapive pixel deimaion in [9], he sandard 4:1 paern in [10], and e.. In experimens, we find fixed pixel subsampling paern is simple and an gain he bes speedup in our algorihm. Some exising fixed paerns are he sandard 4:1 paern in [10] shown in Figure 5, and he Three- Coeffiien paern in [11] shown in Figure 5. We noie in hose inferior enhaned MBs, he drawbak pixels are always he edge pixels hus an unbalaned subsampling paern fousing more on edge pixels suh as he one shown in Figure 5 is more desirable. To make he subsampling paern symmerial, we propose an innovaive paern shown in Figure 5(). The sample we ake are he diagonal pixels and he edge pixels. The oal number is 60 pixels for 16 16 MB. Experimen resuls show our algorihm performs as fas as he 4:1 paern while he effes are no heavily affeed. Moreover, in omparison wih he exhausive pixel-wise SAD mehod, he proposed fas SAD mehod even gains beer enhanemen qualiy wih only 20 peren of he ompuaion. 4. EXPERIMENTAL RESULTS In our experimens, we use Windows PC wih he Inel Core 2 Duo proessor (2.0GHz) wih 3G of RAM. The videos

resoluion is 640 480. Examples of he enhanemen resuls of he proposed algorihm in Seion 2 are shown in Figures 6, 7 and 8. The original low lighing video inpus i.e. Figure 6, 7 and 8 exhibi very lile sene informaion due o he low illuminaion level. The enhaned resuls are shown in Figure 6, 7 and 8 and he red arrows poin o areas wih noable enhanemen. In Figure 6, he improvemen in visibiliy is obvious, inluding four vehiles in he bakground and he liense plae of he vehile in he foreground. More enhanemen resuls are shown in Figure 7 and 8. In addiion, we perform he video enhanemen algorihm using boh he Co ME and he EPZS ME wih differen GOP size, SAD hreshold parameers, a k and b k. In omparison wih he resul of he frame-wise enhanemen algorihm in whih all values of eah frame are alulaed, he speedup resuls are shown in Figure 10, and he orresponding PSNR resuls are shown in Figure 11 (GOP size is 10) and 11 (GOP size is 30). From Figure 11 we an find he PSNR of he EPZS ME wih T=1000, a k =0.8 and b k =40 is even higher han he PSNR of Co ME wih T=500 in he same GOP size. Meanwhile, as shown in Figure 10, he speedup of EPZS ME is muh higher han ha of Co ME oo. This demonsraes ha he EPZS ME helps find more aurae referene MBs, resuling in muh more of he alulaion of bypassed, alhough he EPZS ME akes more ompuaion oss han he Co ME. On he oher hand, we perform he video enhanemen algorihm using he Co and EPZS ME wih he fas SAD algorihm. The speedup resuls are shown in Figure 10. The orresponding PSNR resuls are shown in Figure 12 (GOP size is 10) and 12 (GOP size is 30). From Figure 6, we an learn he fas SAD algorihm enhanes he speedup furher for boh ME algorihms. Meanwhile, he Figure 8 shows he PSNR of EPZS ME using fas SAD algorihm is even enhaned someimes. To sum up, he fas SAD algorihm benefis he EPZS ME in boh speedup and video qualiy hus resuls in our low lighing video enhanemen algorihm faser and higher qualiy. Examples of he resuls of he aeleraion algorihm are shown in Figure 9. Figure 9 is he 16 h frame of he inpu video sequene, Figure 9 and 9() are he enhaned resuls of he same frame using our aeleraion algorihm wih GOP size 1, orresponding o proessing eah frame independenly, and GOP size 16. Alhough his is he las frame of he group wih he GOP size 16, he oupu of our algorihm wih GOP size 16 in Figure 9(), as ompared wih Figure 9 in whih every frame is proessed independenly effeively, gains 4x speedups wihou visible informaion loss. Fig. 6. Inpu low ligh image 1. Enhaned image 1 using proposed low lighing enhanemen algorihm. Fig. 7 Inpu low ligh image 2. Enhaned image 2 using proposed low lighing enhanemen algorihm. Fig. 8. Inpu low ligh image 3. Enhaned image 3 using proposed low lighing enhanemen algorihm. () Fig. 9. The 16h frame of he inpu video sequene. Enhaned frame of wih GOP size 1. () Enhaned frame of wih GOP size 16.

Fig. 10.Speedup of proposed fas enhanemen algorihm using Co ME and EPZS ME. Co ME and EPZS ME wih fas SAD algorihm. Fig. 11.PSNR of proposed low lighing enhanemen algorihm using Co ME and EPZS ME. GOP size is 10. GOP size is 30. Fig. 12. PSNR of proposed low lighing enhanemen algorihm using Co ME and EPZS ME wih fas SAD algorihm. GOP size is 10. GOP size is 30. 5. CONCLUSIONS AND FUTURE WORK In he paper, we desribed a novel, fas and effiien video enhanemen algorihm for low lighing video. Based on he observaion ha invered low lighing video are similar wih haze videos, we hange he de-hazing algorihm properly and apply i on he invered video before invering he dehazed video again o produe he oupu enhaned lowlighing video. 4x speedups is ahieved wih no visible loss of riial informaion by performing ME and skipping he alulaion of for P frames. Areas of furher improvemen inlude improvemens in he de-haze algorihm and enhanemen of videos in oher weaher ondiions. Drivers by Nonlinear Enhanemen of Fused Visible and Infrared Video, in Pro. IEEE Conf. Compuer Vision and Paern Reogniion., San Diego, CA, Jun. 2005, pp.25. [2] Nigh Driver, Making Driving Safer a Nigh Rayheon Company, available a: hp://www.nighdriversysems.om/nighdriver.hml [3] K. He, J. Sun, and X. Tang. Single Image Haze Removal Using Dark Channel Prior, in Pro. IEEE Conf. Compuer Vision and Paern Reogniion., Miami, FL, Jun. 2009, pp. 1956-1963. [4] R. Faal. Single Image Dehazing, in ACM SIGGRAPH 08, Los Angeles, CA, Aug. 2008, pp. 1-9. [5] S. G. Narasimhan, and S. K. Nayar. Chromai Framework for Vision in Bad Weaher, in Pro. IEEE Conf. Compuer Vision and Paern Reogniion., Hilon Head, SC, Jun. 2000, vol. 1, pp. 1598-1605. [6] S. G. Narasimhan and S. K. Nayar. Vision and he Amosphere, in In. Journal Compuer Vision, vol. 48, no. 3, pp. 233 254, 2002. [7] R. Tan. Visibiliy in Bad Weaher from A Single Image, in Pro. IEEE Conf. Compuer Vision and Paern Reogniion., Anhorage, Alaska, Jun. 2008, pp. 1-8 [8] A. Zaarin and B. Liu. Fas Algorihms for Blok Moion Esimaion,in Pro. IEEE In. Conf. Aousis, Speeh and Signal Proessing, San Franiso, CA, 1992, pp. 449-452. [9] Y. L. Chan and W. C. Siu. New Adapive Pixel Deimaion for Blok Moion Veor Esimaion, in IEEE Trans. Ciruis Sys. Video Tehnol., vol. 6, pp. 113-168, 1996. [10] T. Koga, K. Iinuma, A. Hirano, Y. Iijima, and T. Ishiguro. Moion Compensaed Inerframe Coding for Video Conferening, in Pro. Nu. Teleommun. Conf., New Orleans, LA, Nov. 1981, pp. G5.3.1- G5.3.5. [11] B. Girod, and K. W. Suhlm uller. A Conen- Dependen Fas DCT for Low Bi-Rae Video Coding, in Pro. IEEE In. Conf. Image Proess., Chiago, Illinois, O. 1998, vol. 3, pp. 80-83. [12] A. M. Tourapis. Enhaned Prediive Zonal Searh for Single and Muliple Frame Moion Esimaion, in Pro. Visual Communiaions and Image Proessing., San Jose, CA, Jan. 2002, pp. 1069-1079. REFERENCES [1] H. Ngo, L. Tao, M. Zhang, A. Livingson, and V. Asari. A Visibiliy Improvemen Sysem for Low Vision