Correction of Over- and Underexposed Images Using Multiple Lighting System for Exploration Robot in Dark Environments

Transcription

1 Correctio of Over- ad Uderexposed Images Usig Multiple Lightig System for Exploratio Robot i Dark Eviromets Joghoo Im 1, Hiromitsu Fujii 1, Atsushi Yamashita 1 ad Hajime Asama 1 Abstract I this paper, we propose a method to correct the over- ad uderexposed regios i images. For the correctio of over- ad uderexposed regios, multiple light sources are used to obtai several images whose over- ad uderexposed regios are i differet positios. The image processig cosists of four steps. Firstly, multiple images are captured by alterately turig o ad off the illumiatios set i differet positios. Secodly, the lumiace of the images acquired i step 1 is corrected. Thirdly, the over- ad uderexposed regios are extracted from the lumiace-corrected images. Fially, the images are merged except for the over- ad uderexposed regios. The experimet results show that the over- ad uderexposed regios i the iput images are recovered by our proposed method. I. INTRODUCTION Nowadays, remote cotrol robots are used i may dagerous places where people caot eter. Oe such place is dark eviromets such as the Fukushima Daiichi uclear power plat where the recet uclear accidet occurred i 011. I order to ivestigate dark eviromets without the eed for exteral lightig, as show i Fig. 1, it is ecessary to use robots that are equipped with their ow lightig. However, whe lights that are attached to the robot are used, it is difficult to illumiate targets sufficietly as compared to lights that are attached to the ceilig. I this case, the problem is that a portio of the image is uclear because of over- ad uderexposure. I such situatios, the operator may have difficulty i cotrollig the robot. I order to solve this problem, may methods have bee proposed. Oe of the methods is High Dyamic Rage imagig (HDRI) [1-]. This techique obtais oe image by combiig a plurality of images with differet exposure levels. For example, the bright areas are extracted from the images whose exposure values are decreased ad the dark areas are extracted from the images whose exposure values are icreased. The image is the clarified by combiig the extracted areas. Aother method is to use image blocks [3]. This method partitios the image domai ito uiform blocks ad, for each block, selects the image that cotais the most iformatio withi that block. The selected images are the bleded together usig mootoically decreasig bledig fuctios that are cetered at the blocks ad have the sum of 1 everywhere i the image domai. 1 J. Im, H. Fujii, A. Yamashita, ad H. Asama are with the Departmet of Precisio Egieerig, Faculty of Egieerig, The Uiversity of Tokyo, Hogo, Bukyo-ku, Tokyo , Japa. ( {im, fujii, yamashita, asama}@robot.t.u-tokyo.ac.jp). Fig. 1. Ivestigatio usig remote cotrolled robot with attached lightig. These methods use adjustmet of camera parameters such as ISO, shutter speed [4-6], ad aperture, or specific devices such as beam splitters [7-9] for acquirig multi-exposed images. However, it is difficult to correct over- ad uderexposed areas usig the previous methods i dark eviromets with o exteral lightig, such as that show i Fig. 1. Whe the lightig is too strog, the light source is too close to the object, or the light is reflected from a material such as metal or glass, it is difficult to compesate for the irradiated regios usig oly adjustmet of camera parameters. I such cases, it is ecessary to chage the lightig coditio to correct for the irradiated regios. Oe of the methods that utilize the chagig of the light coditios is flash ad o-flash imagig [10]. I this method, reflectios ad highlights from flash images are corrected by comparig the gradiet iformatio of the flash ad o-flash images. However, if the eviromet has o exteral lightig, as is the case withi the Fukushima Daiichi uclear power plat, the o-flash image has o iformatio about brightess ad color. For this reaso, this method has difficulty i correctig the over- ad uderexposed areas i eviromets with o exteral lightig. Aother method is the iteractive digital photomotage [11]. I this method, the user begis with a set of multiple source images take uder differet lightig coditios, ad attempts to create a fial composite image with attractive lightig. However, i this method, the user has to set each area maually for the composite. For this reaso, this method is ot appropriate as a over- ad uderexposure correctio method i the exploratio of dark eviromets where automatic processig is required. To solve these problems, we propose a ew method to correct the over- ad uderexposed regios i images usig o/off cotrol of multiple light sources for exploratio i dark eviromets. I our previous research [1], automatic image processig was successful. However, the acquisitio of multiple images was performed maually. Therefore, the image correctio system could ot be fully automated. I order

2 Fig.. Mai cocept of our proposed method for the correctio of over- ad uderexposed regios usig multiple images obtaied by o/off cotrol of multiple light sources. to overcome this issue, we propose a ew image correctio method that automatically corrects over- ad uderexposed regios from image acquisitio to image processig. The rest of the paper is orgaized as follows. Sectio II describes the proposed method, ad Sectio III depicts the experimet ad its result. Fially, Sectio IV cocludes the paper. II. PROPOSED METHOD We propose a method for the compesatio of over- ad uderexposed regios i a image by alterately turig o ad off multiple light sources set i differet positios. We assume the situatio i which the operator is ivestigatig a dark eviromet usig a remote cotrol robot, such as that show i Fig. 1. Multiple light sources, such as headlights, are attached to the frot of the robot, ad each source ca be tured o ad off by the operator. Figure shows the mai cocept of our correctio method. Whe the over- or uderexposure appears o the moitor that the operator is usig, the operator is able to decide which light sources to tur o or off. Iitially, oly oe light is switched o ad others are switched off. The lights are the cycled through automatically, so that a sigle light at each positio is tured o i successio. This process repeats util a stop commad is issued by the operator. As a result of this process, multiple images whose over- ad uderexposed areas appear at differet positios are acquired. By sythesizig properly exposed areas from each image, composite images with the desired exposure are acquired. The schematic view of our proposed method is show i Fig. 3. The image processig cosists of four steps. Firstly, multiple images are acquired by alterately turig o ad off multiple illumiators that are set i differet positios. Secodly, the lumiace of the iput images is corrected. Thirdly, over- ad uderexposed regios i the lumiace -corrected images are extracted. Fially, the images are merged except i the over- ad uderexposed regios. Each step is described i detail i this paper ad prepared sample images are used to explai the process of our proposed method. A. Image Acquisitio I this process, multiple photographs with over- ad uderexposed areas i differet locatios are captured by chagig the lightig coditio. I this study, multiple images are acquired by alterately turig o ad off each illumiator set i differet positios. Figure 4 shows a example of sychroizig each camera frame with time iterval for switchig o ad off of the lights whe lights are used. The process cosists of 4 step. Firstly, light 1 is oly switched o. Next, all of the lights are switched off. The, light is oly switched o. Fially, all of the lights are switched off oce agai. The time iterval betwee each process is set to be the same. The acquired images whe each light is switched o are Fig. 3. Schematic view of our proposed method for the correctio of over- ad uderexposure usig multiple images obtaied i a multiple lightig system.

3 Fig. 6. Pihole camera model. Fig. 4. Example of sychroizig each camera frame with time iterval for switchig o ad off of the lights whe lights are used. defied iput images ad used i image processig. However, the images obtaied whe all of lights are switched off ot used i image processig. The object of switchig off all of lights is to avoid afterglow which occurs betwee t = t 1 ad t = t 3. This process repeats ad iput images are cotiuously acquired. The first acquired image is set to iput image 1 ( I 0 1 ) ad the ext image is set to iput image (I 0 ), ad so o, util the fial image is set to iput image N (I 0 N ). Figure 4 shows sample images that were obtaied i the image acquisitio step. The over- ad uderexposed areas are show o the left ad right sides of the image i Fig. 5(a), respectively. Similarly, the over- ad uderexposed areas are show o the right ad left sides of the image i Fig. 5(b), respectively. I Fig. 5(c), the overexposed area is show i the middle of the image ad the uderexposed area is show at the edges of the image. B. Correctio of Lumiace The brightess distributio of the iput images is chaged by the illumiatio coditio. It ca be cofirmed by comparig Fig. 5 (a), (b), ad (c) that differet areas of exposure are acquired by chagig the light coditio. If the over- ad uderexposed areas i the iput images are directly compesated by o-over- ad o-uderexposed areas i the other iput images, the brightess betwee the compesated ad o-compesated areas is differet. I order to solve this problem, it is ecessary to correct the brightess distributio of the acquired iput images. As show i Fig. 5(a), (b), ad (c), the regio aroud the ceter of the irradiated area is the brightest, ad the lumiace gradually decreases as the distace of each pixel from the ceter of the irradiated area becomes larger. I this step, we correct the lumiace of the iput images usig the iverse-square law, which meas that the brightess is iversely proportioal to the square of the distace betwee the light source ad the object. Therefore, if the brightess of each pixel ad the square of the distace betwee the light source ad the object are multiplied, the brightess of the iput images becomes uiform. Equatio (1) is used to correct the lumiace of the images. p 0 ( i, I ( i, I d, (1) where is the umber of iput images. I 0 (i, is the lumiace value of the iput image at the coordiates (i,. The lumiace value describes how bright a pixel is. I p (i, is the obtaied image as a result of the lumiace correctio process. α is the correctio factor that is set i advace by cosiderig the evirometal coditios. d is the distace betwee the light source ad the object. I this step, we propose a ew distace estimatio method usig oly the iput image. This method uses the priciples of the straight propagatio of light ad the pihole camera model, as show i Fig. 6. Equatios () ad (3) describe each priciple. r k l, () f l s r c. (3) r' where l is depth ad r is the vertical distace from the ceter of the irradiated area. r is the vertical image distace from the ceter of the irradiated area. k is the proportioal costat. s is the scale factor relatig the pixels to the distace. f is the focal distace. c is the pricipal poit that would ideally be located i the ceter of the image. p is the radius of the irradiated area; the irradiated area is defied as the area for which the lumiace value of each pixel is over the threshold value τc, which is set i advace by cosiderig evirometal coditios. I this step, we assume that the shape of the target object is a plae ad it is i parallel to the image plae. To estimate d, it is ecessary to estimate l ad r. d l r. (4) Equatio (4) represets the relatio betwee d, l, ad r. By substitutig r of Eq. () ito Eq. (3), l ca be calculated from r. If the value of r chages, the value of l also chages. For this reaso, it is ecessary to fix a specific value of r. I this study, we set the value of r as p (Eq. (5)). cp l. (5) p skf (a) (b) (c) Fig. 5. Images obtaied i the image acquisitio step: (a) Iput image 1 I0 1 that is obtaied at t = t1, (b) iput image I0 that is obtaied at t = t, ad (c) iput image I0 that is obtaied at t = t.

4 Fig. 6. Pihole camera model. The, r ca be estimated usig the radius of the irradiated areas. ckr ' r r' skf Next, by substitutig l of Eq. (5) ito Eq. (), r ca be calculated from r (Eq. (6)).., (6) cp ckr' d (7) p skf r' skf i XC j YC. r' (8) Fially, Eq. (7) is acquired by substitutig l of Eq. (5) ad r of Eq. (6) ito Eq. (4). r ca be calculated by Eq. (8), where (i, is the positio of each pixel i the image, ad (X c, Y c) is the positio of the ceter of the irradiated areas i the image. As a result of the lumiace correctio, the lumiace icreases gradually whe the distace betwee the ceter of the irradiated area ad each pixel becomes larger, as show i Fig. 7 (a), (b), ad (c). However, the over- ad uderexposed areas i the lumiace-corrected images are ot perfectly compesated by the correctio of lumiace. C. Extractio of Over ad Uder Exposed Areas This process extracts the over- ad uderexposed areas i the lumiace-corrected images i order to remove these areas. The method usig two threshold values for lumiace has bee widely used i may related publicatios [13-15] to extract over- ad uderexposed areas. I this method, if the lumiace value of the iput image is greater tha the threshold value, it is overexposed. If the lumiace value of the iput image is less tha the threshold value, it is uderexposed. However, there is a problem i that white or black colors are classified as overor uderexposed, respectively. I order to solve this problem, we use lumiace ad spatial iformatio for the extractio of over- ad uderexposed areas. It is possible to estimate the positio of the over- ad uderexposed areas by cosiderig the positio of the irradiated areas i the image. Equatio (9) is used for the extractio of over- ad uderexposed areas. 0 if I p( i, H AND r rl D ( i, 0 if I p( i, L AND r rh (9) 1 otherwise where I p (i, is the lumiace value of the lumiacecorrected image at coordiates (i,. τh ad τl are the threshold values for the lumiace iformatio ad rage from 0 to 55. r is the distace betwee the ceter of the irradiated areas ad each pixel i the image. rh ad rl are threshold values for the spatial iformatio ad rage from 0 to the maximum distace i the image (diagoal distace). τh, τl, rh, ad rl are set i advace by cosiderig the evirometal coditios. D is the referece matrix for determiig over- ad uderexposed areas, ad is obtaied as a result of the extractio of the overad uderexposed areas. This referece matrix is defied as a determiatio image. If the lumiace value at the iput image coordiates (i, is greater tha τh ad if the spatial value of the iput image coordiates (i, is less tha rl, it is determied to be overexposed ad the pixel value is coverted to 0. If the lumiace value of the iput image coordiates (i, is less tha τl ad if the spatial value of the iput image coordiates (i, is greater tha rh, it is determied to be uderexposed ad the pixel value is coverted to 0. Otherwise, it is determied as o-over- or o-uderexposed ad the pixel value is coverted to 1. As a result, the determiatio image D is acquired. The over- ad uderexposed areas are displayed i black ad o-over- ad o-uderexposed areas are displayed i white i Fig. 8(a), (b), ad (c). The areas displayed i black i the ceter of the images are overexposed areas that are caused by direct irradiatio. The areas displayed i black o the edge of the images are uderexposed areas that are caused by lack of light from the light source. D. Image Sythesis I this step, the areas that are ot over- ad uderexposed from the lumiace-corrected images are sythesized. The determiatio image D obtaied i Sectio II-C, show i Fig. 8, is used. The rule of the sythesis is show i Eq. (10). N D ( i, I p ( i, 1 I f ( i,. (10) N D ( i, 1 The areas where D (i, = 0 i the determiatio image D, which are show as black i Fig. 8, are ot used for sythesis. The areas where D (i, = 1 i the determiatio image D, (a) (b) (c) Fig. 7. Images obtaied i lumiace correctio step : (a) Lumiace-corrected image 1 Ip 1, (b) Lumiace-corrected image Ip, (c) Lumiace-corrected image Ip.

5 (a) (b) (c) Fig. 8. Images obtaied i the extractio of over- ad uderexposed areas: (a) Determiatio image 1 D 1, (b) determiatio image D, ad (c) determiatio image D. which are show as white i Fig. 8, are used for sythesis. The areas that are ot over- ad uderexposed are equally sythesized. As a result of this process, a corrected image I f is obtaied. Figure 9 shows the result of the sythesis. As a result of this process, the over- ad uderexposed areas from the iput images, as show i Fig. 5, are corrected. DANGER Radiatio risk o the left sig ad DO NOT ENTER, STAFF ONLY o the right sig became visible. The backgroud of the iput image also became clear. III. EXPERIMENT A. Experimetal Eviromet Three lights were placed o the left, right, ad top of the camera i order to acquire three images that showed over- ad uderexposure i differet locatios. Figure 10 shows the placemet of the camera ad the lights. The height of the camera was 800 mm from the floor. Light 1 was located i 300 mm left of the camera, ad Light was located 300 mm right of the camera. Light 3 was located 300 mm above the camera. All of the photographig coditios were fixed except the distace betwee the camera ad the lights durig the image acquisitio process. The photographig coditios are depicted i Table 1. Fially, three images were acquired by alterately turig o ad off each light. The experimet was performed i four differet locatios. All experimets were simulated o a Itel Core TM i5-50m CPU.5 GHz PC with 8 GB RAM. I this experimet, we set τ C =.0 10, τ H = 1. 10, τ L = 5.0, r L = , r H =.3 10, α= , k = , s = ad c = The time iterval betwee each camera frame ad switchig o ad off of each lights are set to 1/60 s. B. Experimetal Result The objective of the experimet is to verify whether the proposed method is valid i a variety of eviromets. Figures 11, 1, 13, ad 14 describe the experimetal results. Figures 11 ad 1 idicate plae eviromet, Fig. 13 ad Fig. 14 represet o plae eviromet. The average computig time per frame was 0.1 s. Iput image 1 i each figure was acquired whe Light was tured o, ad iput image i each figure was acquired whe Light 1 was tured o. Iput image 3 i each figure was acquired whe Light 3 was tured o. The corrected image i each figure represets the resultig image that was combied by our proposed method. I iput image 1, the overexposure was show o the left side of the image ad the uderexposure was show o the right side of the image. O the other had, the overexposure was show o the right side ad the uderexposure was show o the left side of iput image. I iput image 3, the overexposed areas were show o the top ceter of the image ad the uderexposed areas were show o the edges of the image. The over- ad uderexposed areas of the iput images lost their color iformatio ad, cosequetly, it is difficult to idetify the over- ad uderexposed areas. As a result of applyig the proposed compesatio algorithm to the iput images take i a variety of eviromets, the over- ad uderexposed areas i the iput images became clear ad it was possible to idetify them i the output images. However, if the over- or uderexposed exposed areas i oe iput image correspoded to those i the other iput images, it was ot successfully corrected. I order to solve this problem, it is ecessary to acquire images that do ot have the TABLE 1. Photographig coditios. Camera NIKON D 700 Defiitio ISO 400 F-umber f/8 Shutter speed 1/40 s Focal legth 4 mm Light GENTOS MF-1010G Lumiace of light 1,000 lm Fig. 9. Image obtaied as a result of image sythesis step. Fig. 10. Placemet of camera ad lights.

6 Fig. 11. Experimetal result at locatio 1: (a) Iput image 1, (b) iput image, (c) iput image 3, ad the (d) corrected image. Fig. 1. Experimetal result at locatio : (a) Iput image 1, (b) iput image, (c) iput image 3, ad the (d) corrected image. Fig. 13. Experimetal result at locatio 3: (a) Iput image 1, (b) iput image, (c) iput image 3, ad the (d) corrected image. Fig. 14. Experimetal result at locatio 4: (a) Iput image 1, (b) iput image, (c) iput image 3, ad the (d) corrected image. over- ad uderexposed areas at the same locatios i the iput image ad i the other images. C. Evaluatio We evaluated the proposed method with cotrast compesatio [16] ad HDRI []. Etropy [17], which idicates the iformatio value of the image, was used for the evaluatio. If the image is overexposed or uderexposed, the etropy of the image decreases. O the other had, the etropy of the image icreases if the brightess of the image is evely distributed. The etropy was calculated usig Eq. (11). E p i p i ) L i0 log(, (11) where p i is the probability that a arbitrary pixel i the image has itesity i. L is a gradatio umber ad we set its value to 55. E is the etropy. Figure 15 ad Table show the etropy of the origial image, the result of cotrast compesatio, the result of HDRI, ad the result of our proposed method.the x-axis of the graph represets the images that are corrected by each method ad the y-axis represets the etropy of each image. The blue colum represets the etropy of the origial image. The gree colum represets the etropy of the image that is corrected by cotrast compesatio. The orage colum represets the etropy of the image that is corrected by HDRI. Fially, the red colum represets the etropy of the image that is corrected by the proposed method. I all cases, the etropy of each image that was corrected by our proposed method is larger tha that of the origial image ad of the images that

7 were corrected by cotrast compesatio ad HDRI. This meas that our proposed method is more effective tha the cotrast compesatio ad HDRI methods i dark eviromets. Next, we compared the computig time of each method. Table 3 shows the average computig time of each method. Cotrast compesatio takes 0.1 s for the image processig, while HDRI takes 0.61 s ad the proposed method takes 0.1 s. Therefore, it ca be see that the average computig time of our proposed method is faster tha that of HDRI ad slower tha that of cotrast compesatio. IV. CONCLUSION I this paper, we propose a ew image correctio method that automatically corrects over- ad uderexposed regios from image acquisitio to image processig. We assumed a situatio i which the operator ivestigates a dark eviromet usig a remote cotrol robot. Three lights are attached to the left, right, ad top at the frot of the robot, ad each light ca be tured o ad off by the operator. Next, three images were acquired by alterately switchig o ad off the three lights. Fially, the over- ad uderexposed areas were compesated by the proposed method. The experimetal results showed the effectiveess of our proposed method i a variety of eviromets. For future work, we pla to apply our method to real situatios, such as iside the Fukushima Daiichi uclear power plat. Furthermore, it is ecessary to cotrol the brightess or adjust the agles of the lights i order to acquire more images i which the over- ad uderexposed areas will appear at differet locatios. ACKNOWLEDGMENT This work was fuded by Tough Robotics Challege, ImPACT Program of the Coucil for Sciece, Techology ad Iovatio (Cabiet Office, Govermet of Japa). REFERENCES [1] S. Ma ad R. W. Picard, O beig Udigital with Digital Cameras : Extedig Dyamic Rage by Combiig Differetly Exposed Pictures, Proceedigs of the 48th Aual Coferece of the Imagig Sciece ad Techology, pp , [] P. Debevec ad J. Malik, Recoverig High Dyamic Rage Radiace Maps from Photographs, Proceedigs of ACM SIGGRAPH 1997, pp , [3] A. A. Goshtasby, Fusio of Multi-exposure Images, Joural of Image ad Visio Computig, Vol. 3, No. 6, pp , 005. [4] D. C. H. Schleicher ad B. G. Zagar, High Dyamic Rage Imagig by Varyig Exposure Time, Gai ad Aperture of a Video Camera, 010 IEEE Istrumetatio ad Measuremet Techology Coferece, pp , 010. TABLE. Result of etropy compariso. Method Locatio1 Locatio Locatio3 Locatio4 Origial image Cotrast compesatio HDRI Proposed method TABLE 3. Average computig time of each method. Method Time (s) Cotrast compesatio 0.1 HDRI 0.61 Proposed method 0.1 [5] Y. Piao ad W. Xu, Method of Auto Multi-Exposure for High Dyamic Rage Imagig, 010 Iteratioal Coferece o Computer, Mechatroics, Cotrol ad Electroic Egieerig, Vol. 6, pp , 010. [6] N. Barakat ad T. E. Darcie, Miimal Capture Sets for Multi-Exposure Ehaced-Dyamic-Rage Image, 006 IEEE Iteratioal Symposium o Sigal Processig ad Iformatio Techology, pp , 006. [7] M. D. Tocci, C. Kiser, N. Tocci ad P. se, A Versatile HDR Video Productio System, ACM Trasactios o Graphics, Vol. 30, No. 4, pp. 41, 011. [8] M. Aggarwal ad N. Ahuja, Split Aperture Imagig for High Dyamic Rage, I Proceedigs of IEEE Iteratioal Coferece o Computer Visio 001, pp , 001. [9] M. Aggarwal ad N. Ahuja, Split Aperture Imagig for High Dyamic Rage, Iteratioal Joural of Computer Visio 58, pp. 7 17, 004. [10] A. Agrawal, R. Raskar, S. Nayar ad Y. Li, Removig Photography Artifacts Usig Gradiet Projectio ad Flash-Exposure Samplig, ACM Trasactios o Graphics, Vol. 4, No. 3, pp , 005. [11] A. Agarwala, M. Dotcheva, M. Agarwala, S. Druckers, A. Colbur, D. Salesi ad M. Cohe, Iteractive Digital Photomotage, Proceedigs of the 8th IEEE Iteratioal Coferece o Computer Visio, pp , 001. [1] J. Im, H. Fujii, A. Yamashita ad H. Asama, "Compesatio of Over ad Uder Exposure Image Usig Multiple Light Switchig," Proceedigs of the 014 IEEE/SICE Iteratioal Symposium o System Itegratio (SII014), pp , 014. [13] W. Zhag ad W. K. Cham, Gradiet-Directed Multi-Exposure Compositio, IEEE Trasactios o Image Process, Vol. 1, No. 4, pp , 01. [14] T. Jio ad M. Okuda, Multiple Exposure Fusio for High Dyamic Rage Image Acquisitio, IEEE Trasactios o Image Process, Vol. 1, No. 1, pp , 01. [15] O. Gallo, N. Gelfad, W. Che, M. Tico, ad Kari Pulli, Artifact-Free High Dyamic Rage Imagig, Proceedigs of IEEE Iteratioal Coferece o Computatioal Photography, pp. 1 7, 009. [16] E. Reihard ad K. Devli, "Dyamic Rage Reductio Ispired by Photo receptor Physiology," IEEE Trasactios o Visualizatio ad Computer Graphics, Vol. 11, No. 1,pp. 13-4, 005. [17] C. E. Shao, A Mathematical Theory of Commuic- atio,, Bell System Techical Joural, Vol. 7, No. 3, pp , Fig. 15. Etropy of the origial image (blue), the cotrast compesatio (gree), the HDRI (orage), ad our proposed method (red): (a) Locatio 1, (b) locatio, (c) locatio 3, ad (d) locatio 4.