Light Condition Invariant Visual SLAM via Entropy based Image Fusion

Transcription

1 Light Condition Invariant Visual SLAM via Entropy based Image Fusion Joowan Kim1 and Ayoung Kim1 1 Department of Civil and Environmental Engineering, KAIST, Republic of Korea (Tel : ; jw kim@kaist.ac.kr, ayoungk@kaist.ac.kr) Abstract Cameras have always been a popular type of sensor in simultaneous localization and mapping (SLAM) applications. Images, however, often suffer from light and environment changes. One typical limitation is image saturation; sudden change in lighting conditions causes saturation in the image and prohibits the algorithm from producing meaningful measurements. In this paper, we introduce a stereo vision system with different exposure values. Unlike other image processing approaches, we aim to achieve instant saturation detection and fusion with no delay. When the lighting condition change occurs, we detect the over or under-exposed regions using entropy-based metrics and generate a restored image. The proposed method has been validated using an indoor office environment and outdoor parking lot in which the lighting condition is drastically changed. Keywords Light Condition, Visual SLAM, Image Fusion 1. I NTRODUCTION Visual SLAM estimates the motion of a moving robot in real-time as it continuously observes and creates a map of unknown environment using vision sensors. Two major categories of the visual SLAM include feature-based and direct SLAM. Feature-based SLAM extracts the features of an image and generates descriptors to track the position of the camera. For example, ORB-SLAM [1] is a feature-based monocular SLAM system that operates in real time, in both indoor and outdoor environments. Direct SLAM, on the other hand, uses sensor data directly without extracting features. Semidirect visual odometry(svo) [2] is calculated by optimizing the intensity residual through a sparse model based image alignment using a monocular camera. DTAM [3] provides accurate and dense 3-D maps, and LSD-SLAM [4] provides semi-dense 3-D maps in large scale environments. In this paper, we focus on both types of the visual SLAM that are vulnerable to light and illumination changes. For example, when a robot moves from a dark space to a bright area, the condition of the light changes drastically, causing image matching and pose estimation to fail. When applied to featurebased or keyframe-based direct SLAM in highly dynamic light environments, this issue becomes a major limitation. Overall, the direct method is heavily influenced by over-exposed areas as the intensity value of the image changes dramatically; the feature-based method is affected by under-exposed areas due to the diminished features. The over-exposed or underexposed regions mentioned above frequently occur when the light condition changes suddenly as in two sample figures in Fig. 1. This paper proposes an under-exposed or over-exposed region detection and image pre-processing method using image (a) Over exposed region (b) Under exposed region Fig. 1. Two typical examples of the light condition variation in urban environment. entropy. To test this method, we introduce a stereo sensor system (Fig. 2) with two different exposure values in order to retrieve the image pixel value with no delay. One camera (cam-high) has a high exposure value and the other camera (cam-low) has a low exposure value respectively. 2. R ELATED W ORK Methods have been introduced in order to solve the problem of light condition changes. In [5], the method simply combined sub-images determined by specific conditions of an image with various automatic exposure values. In [6], the images were divided into small blocks and the image that has the most entropy value for each block was selected. This approach, however, resulted in inconsistency despite the smoothing and blending of each block boundary. Optimizing the camera exposure time was introduced by [7] who examined the gradient domain for maximum features. The results were effective when (a) CAD drawing (b) Stereo sensor system Fig. 2. Camera placement for image fusion. Each camera has different exposure time.

2 Brightness Level Relation between BL and ET Exposure Value cam low cam high Fig. 3. Relationship between brightness level and exposure value. applied to robotics applications with the transient time for exposure adjustments in a changing environment Exposure Value Selection 3. PRELIMINARY Since our approach uses different exposure values for the stereo vision system, we first introduce the exposure value (EV) selection criteria used in the method. According to [8] the EV was defined by the following equation EV = log 2 F 2, where F is the ratio of the lens s focal length to the diameter and T is the exposure time. Another factor affecting exposure control is the Brightness Level (BL), which measures the mean brightness level of the image. T (1) Bl = klgt F 2 (2) The BL is a function of constant k, mean luminance of the scene L, and the gain of the Automatic Gain Control (AGC)- circuit G. Then, the next EV is determined by the current EV and BL assuming k, L and G do not change between two frames. EV n+1 = EV n + log 2 Bl n log 2 Bl n+1 (3) Fig. 3 shows the relationship between the brightness level and the exposure time as in (3). As shown in Fig. 3, in an environment in which the light condition changes suddenly, a saturated region occurs because the exposure value rapidly increases or decreases according to the brightness level. Based on this graph, we chose one camera with a high exposure value and another camera with a 2 or 3 step lower exposure value Pixel Activity using Image Entropy We follow image entropy introduced by Shannon [9]. Image entropy for a gray scale image can be written as : 255 H(X) = P (X i )log(p (X i )). (4) i= In the saturation detection, we use pixel activity that captures the entropy variation [1]. For clear derivation of the activity, we first re-write the entropy equation using to indicate an image. In the derivation, represents the normalized brightness level of the pixel at position x, y after t th fine-to-coarse transformation. H(f) = y x ln ()dxdy (5) In [1], pixel activity was defined as a partial differential of (5) with respect to the transformation step. a(x, y) = t H = y x δ(x, y, t)dxdy (6), while delta indicates the density of entropy production as δ(x, y, t) = 2. (7) Using the equation above, we can re-write activity as a(x, y) = = N N δ(x, y, t)dt 2 dt (8) A straightforward solution for this entropy equation is calculated through Fourier-transformation. To apply Fourier s partial differential equation for a conductive diffusion equation, we consider the normalized pixel value of the image after fine to coarse transformation as = f(x, y, ) 1 4πt e x 2 x 2 4t e y2 y 2 4t dx dy (9) which turns out to be the convolution with Gaussian function k(x, y, t) = 1 4πt e x 2 +y 2 4t. = f(x, y, t = ) k(x, y, t). (1) Then, (8) could be directly computed in the frequency domain. { 2 } F = F {} (u 2 + v 2 ). (11) Using Fourier transform of Gaussian function k F {k(x, y, t)} = 1 2 +v 2 ) 2π e t(u (12) and the convolution property, we re-write (8) as equation below. N a = F 1 F {f(x, y, t = )} e t(u2 +v 2) (u 2 + v 2 ) dt (13) 2π }{{} 2 f(x,y,t) f(x,y,t)

3 (a) Image from cam-high (b) Image from cam-low (a) Image from cam-low (b) Image from cam-high (c) Image activity (d) Activity enhancement (c) Image activity (d) Activity enhancement (e) Saturation mask (f) Restored image Fig. 4. The process of creating a saturation mask and restoring an image for over exposure case. (e) Saturation mask (f) Restored image Fig. 5. The process of creating a saturation mask and restoring an image under exposure case. Lastly, integrating over dt results in the following activity that we use in the paper. { } F {f(x, y, t = )} a(x, y) = F 1 (e N(u2 +v 2) 1) (14) 2π In the following section, we explain how we use this activity when detecting regions to be compensated Saturation Detection 4. IMPLEMENTATION If an image is over-exposed it becomes too bright, and too dark if under-exposed. This image loss is represented by the low entropy value accordingly. In contrast to this, we would expect high information entropy for a properly exposed image. Because the distribution of squared gradient values in the differential of entropy refers to the amount of information in the image region, the differential of entropy measures the local information level of the original image. Therefore, it can be regarded as a saturated region without information when the differential of entropy values are close to zero. Using this property, we can detect a region in an image in which the pixels are over-exposed or under-exposed. The differential of entropy can be calculated through the fine-tocoarse transformation using a diffusion process Saturation Mask and Pixel Restoration The saturation mask was determined as an over-exposed or under-exposed region based on the activity measurements. Then, using this saturation mask, we instantly replaced the pixel value from another image to generate the restored image. The overall procedure is described in Fig. 4. Fig. 4 shows a sample stereo image whereas the image from the cam-high shows saturation by over-exposure. As can be seen in Fig. 4(c), the pixel activity map for the saturated image indicates where the pixel information is lost. In order to segment the information regions clearly, the activity enhancement and noise elimination step should be computed using the median filter on the obtained activity map (Fig. 4(d)). In Fig. 4(e), the saturation mask is generated by computing the region with an activity value closer to zero. For this we used threshold.1. Lastly, using this mask, local image fusion is applied by implementing the pixel value from another image (Fig. 4(b)) to generate the restored image, as in Fig. 4(f). In contrast to the over-exposed region, an image may be regarded as under-exposed when certain areas are dark and indistinguishable from black. As can be seen in Fig. 5, image information loss occurs due to the under-exposure. Following the similar sequence as in the over-exposed case, the restored image using our saturation mask is obtained (Fig. 5(f)) Direct SLAM Implementation We apply this image restoration for direct SLAM application. The direct method optimizes the geometry through the image intensities using information from all the images. The direct methods consist of tracking, depth map estimation and map optimization, as show in Fig. 6. The monocular direct SLAM algorithm is the method we used to apply our image processing method. The tracking step of the LSD-SLAM tracks the new camera image, and estimates the camera s pose

4 SLAM Saturated? Main Image Yes Entropy information Create mask No Output image Tracking LE Image Image pre-processing Laplacian Blending Map Estimation Map Optimizati on Output image Fig. 6. System overview. Left module (blue box) generate saturation compensated image using the proposed method. This fused image (output image) is then passed to the SLAM application (right module, green box). through rigid body transformation. However, if a saturated region occurs at this step, the tracking will fail. To avoid this problem, it is important to apply our method and restores the saturated region by pre-processing. After the tracking step, the depth map estimation step applies the tracked frame to modify or replace the current keyframe. Finally, the map optimization step optimizes the global map such as the loop closure through the similarity transformation. The experiment was performed in an office environment in which light conditions change rapidly. In a rapidly changing environment, over-exposure or under-exposure of an image often means that the image fails to generate informative pixels for visual SLAM applications. Fig. 7 presents three sequential frames in an office environment. The left column images are the saturated images, and the right column images are images that have been restored using our method. Fig. 7(a)(left) shows the image is saturated due to the suddenly changing light condition. As shown in Fig. 7(a)(right) the saturated region is detected and restored by the entropy information. Fig. 7(b) shows that the saturated region is reduced by gradually adjusting the exposure time by auto exposure control. In Fig. 7(c), the exposure value returns to the normal state and maintains the optimal brightness level. In short, image information is maintained by iteratively (a) Frame 1 : Saturation region due to illumination change (a) Over exposed image due to illumination change (b) Frame 2 : Reduced saturated region by auto exposure control (b) The result of direct SLAM when the saturation region occurs (c) Frame 3 : Maintain optimized exposure value (c) 3-D point cloud map using LSD-SLAM Fig. 7. Saturation restoration process over time sequence. Fig. 8. Result of image pre-processing and SLAM. 5. E XPERIMENT

5 TABLE 1 SUMMARY OF COMPUTATION TIME FOR EACH IMAGE. Frame No. Mask creation (sec) Image blending(sec) detecting and restoring only when a saturation region occurs. Otherwise, this process is skipped. For this image sequence in Fig. 7, the computational time is summarized as in Table 1. The evaluation was performed in Matlab. The mask generation time is about.13 seconds, and the blending processing time is about.1 second. We expect C implementation would result in faster computational time and real-time performance. Fig. 8 shows the application of the state-of-the-art direct SLAM method, the LSD-SLAM. If a saturated region occurs, it causes a fatal error in the SLAM result, or a failure in tracking and divergence as shown in Fig. 8(b)(left). However, restoring an image using our method indicated that tracking was successful, as shown in Fig. 8(b)(right) because it can find correspondence between the frames. As a result, the final camera pose and 3D point cloud map are shown in Fig. 8(c). 6. CONCLUSION This study provided a method of restoring saturated images for visual SLAM application. We detected the saturated image using image entropy information, and reconstructed using secondary image. We also validated the effectiveness of the proposed method by implementing it to the direct SLAM. [5] Q. K. Vuong, S.-H. Yun, and S. Kim, A new auto exposure system to detect high dynamic range conditions using cmos technology, in Convergence and Hybrid Information Technology. Third International Conference on, vol. 1. IEEE, 28, pp [6] A. A. Goshtasby, High dynamic range reduction via maximization of image information, Preprint of CSE Department, Wright State University, 23. [7] I. Shim, J.-Y. Lee, and I. S. Kweon, Auto-adjusting camera exposure for outdoor robotics using gradient information, in Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 214, pp [8] J. Liang, Y. Qin, and Z. Hong, An auto-exposure algorithm for detecting high contrast lighting conditions, in ASICON 7. 7th International Conference on. IEEE, 27, pp [9] C. E. Shannon, A mathematical theory of communication, ACM SIGMOBILE Mobile Computing and Communications Review, vol. 5, no. 1, pp. 3 55, 21. [1] M. Ferraro, G. Boccignone, and T. Caelli, Entropybased representation of image information, Pattern Recognition Letters, vol. 23, no. 12, pp , 22. ACKNOWLEDGEMENT This material is based upon work supported by the MOTIE, Korea under Industrial Technology Innovation Program (No ) and MOLIT, Korea via U-City Master and Doctor Course Grant Program. This work was also supported by KAIST Institute for Robotics. REFERENCES [1] R. Mur-Artal, J. M. M. Montiel, and J. D. Tardos, ORB-SLAM: a versatile and accurate monocular SLAM system, IEEE Transactions on Robotics, vol. 31, no. 5, pp , 215. [2] C. Forster, M. Pizzoli, and D. Scaramuzza, SVO: Fast semi-direct monocular visual odometry, in Proceedings of the IEEE International Conference on Robotics and Automation. IEEE, 214, pp [3] R. A. Newcombe, S. J. Lovegrove, and A. J. Davison, DTAM: Dense tracking and mapping in real-time, in Proceedings of the IEEE International Conference on Computer Vision. IEEE, 211, pp [4] J. Engel, T. Schöps, and D. Cremers, LSD-SLAM: Large-scale direct monocular SLAM, in European Conference on Computer Vision. Springer, 214, pp