A Real Time Algorithm for Exposure Fusion of Digital Images

Transcription

1 A Real Time Algorithm for Exposure Fusion of Digital Images Tomislav Kartalov #1, Aleksandar Petrov *2, Zoran Ivanovski #3, Ljupcho Panovski #4 # Faculty of Electrical Engineering Skopje, Karpoš II bb, 1000 Skopje, Macedonia 1 kartalov@feit.ukim.edu.mk 3 mars@feit.ukim.edu.mk 4 panovski@feit.ukim.edu.mk * Netcetera, Partizanski Odredi 72a, 1000 Skopje, Republic of Macedonia 2 apetrov@netcetera.com.mk Abstract A real time algorithm for fusion of differently exposed images is proposed in this paper. The algorithm blends the details from two images of high dynamic range scene, acquired with different exposure values, into one output image which can be displayed on low dynamic range devices. The blending is performed in the spatial domain, using pixel by pixel approach, thus eliminating the need for expensive block processing or transform domain coding. The proposed scheme works both on grey and color images. The algorithm shows high efficiency, which make it applicable on low processing power platforms, such as mobile devices. The obtained results are visually comparable with previously published algorithms that are computationally much more expensive. Keywords Image, fusion, dynamic range. I. INTRODUCTION The main objective of the modern electronic devices for digital image and video acquisition is to represent a captured scene as realistic, and as identical to the real eye-observed scene, as possible. With every new model and every new technology, this goal is more achieved. The luminance, the contrast, the saturation of colors, all these parameters of the digital media are continuously getting closer to the psycho physical parameters experienced by human observer in the real scene. However, the photometric quantities existent in the real scene, are differently interpreted by the electronic devices and the human observer, and this still causes in some cases the visual parameters of digital media to be far from visual parameters of the original scene. This especially applies to the real world scenes which have very high ratio between maximum and minimum intensity of the light in the scene, e.g. high dynamic range. Neither image acquisition device, nor display device is manufactured today, which can reproduce the intensity dynamic range that may exist in the real scene, or even the perceived luminance dynamic range of the human eye (brain). This phenomenon often results in loss of detail in the digital reproduction, and overall unpleasant image to the viewer. The detail problem can be addressed by taking multiple snapshots of the scene of interest, using different light sensitivity settings of the capturing device, i.e. different exposure values. That way, multiple digital images can be obtained, in which various segments of the whole light intensity interval are shown. Images taken with longer exposures will reveal the darker objects, while the brighter objects in the scene will be shown on images with shorter exposure values. However, handling multiple images from one scene can be difficult and confusing even if only a still scene is observed. In the case of video material, this approach would be impossible. As a result, a need occurs for combining these multiple snapshots into single understandable image. That image would contain all of the detail in the scene, and would be able of displaying on standard display devices. As pointed above, display devices have lower dynamic range than real world scene, so some dynamic range compression has to take place in the process. Digital image and video community put a lot of effort into optimal solution of the problem of different exposure image fusion, utilizing various concepts and methodologies [1] - [6]. All these proposed solutions use two or more perfectly spatially aligned input images obtained with different exposures, to produce single output image which will contain all the useful parts from the input images. Mann and Picard in [1] propose very complex algorithm which tries to reconstruct point by point the nonlinear response function of an image sensor used in the capturing of the two input images, captured using high and low exposure values, respectively. The selection criteria for combining the luminance of two input images, is based on weighted average, the weights being computed using the previously obtained sensor response function. The luminance values in the result image are mapped to match the low dynamic range of the display devices. In [2], Debevec and Malik use the algorithm for image fusion based on exploiting a physical property of imaging systems, both photochemical and electronic, known as reciprocity. They calculate the characteristic curve of the response of a film to variations in exposure, or Hurter- Driffield curve. That function links the optical density of the film with the logarithm of the exposure to which it has been subjected. The response curve is then used in calculation of the radiance map of the recorded scene, and its mapping on low dynamic range display. Fattal et al. in [3] propose gradient domain dynamic range compression, using the /10/$ IEEE 641

2 property of human visual system to be less sensitive to absolute luminance levels on the retina, but rather responds to local intensity ratio changes. The algorithm is based on spatially variant attenuating mapping to the gradient of the logarithm of the function which represents the luminance range in the scene. The large gradients are more attenuated than the small ones, causing compression of the high dynamic range. In [4], Goshtasby divides the input images on rectangular blocks, calculates the entropy of the luminance within these blocks, and composes the output image from blocks that have highest entropy. In order to solve the imminent occurrence of the tiling effect in the output image, he applies the Gaussian blending function to the image. The size of the blocks and the support of the blending function are iteratively computed on every image, in order to obtain highest entropy of the result, making this algorithm very complex and slow. Using fixed values for block size and blending support speeds up the algorithm, but often results in occurrence of halos around the objects in output image. The authors in [5] and [6] have similar ideas about the method of fusing the images with different exposure values. In the selection part, Mertens in [6] presents more complex scheme for computing the local weights of participants in the output image, while Rubinstein, in [5] uses very simple selection model, producing selection maps as binary, locally choosing only one of the submitted images. The image fusion in both algorithms is performed in a similar manner, using pyramid based image decomposition, as explained in detail in [7] and [8]. Mertens and Rubinstein both employ Gaussian and Laplacian pyramid decomposition of the input images, and of the selection (weight) maps, afterwards building the output image from the useful parts of the pyramids. All explained algorithms perform fairly well, other then being too complex and slow for real time implementation, especially on low power platforms. In this paper, we propose a new algorithm for fusion of the images with different exposures, which visually works comparable with above listed algorithms, and is much less complex in the same time. Our algorithm takes place in spatial domain, in the HSV color space, performs no pre-processing and post processing at all, and every pixel in the output image is calculated only using respective pixels in the input images, thus eliminating the expensive block processing, filtering or blending functions. This paper is organized as follows. In Section II our algorithm is explained in detail, Section III gives some of the obtained experimental results, and the concluding remarks are put in Section IV. Unnumbered sections at the end of the document contain acknowledgments and used references. II. THE ALGORITHM In this paper we propose an algorithm for fusion of multiple images with different exposure values into single image. The design of this algorithm was outlined by the following constraints: The whole procedure should run in low number of operations per pixel, in order to reach the option of embedding this algorithm on low processing power platforms, such as mobile devices. The algorithm should have as lower as possible memory requirement, concerning the usual amount of memory installed on targeted platforms. The algorithm should work both for grayscale and color images. Given the first constraint, we limit our algorithm to two input images, one taken with longer exposure time (overexposed image), and one with shorter exposure time (underexposed image). These two images should be taken consequently by the same device. The period of time between the capturing of the two images should be as short as possible, in order to minimize the changes in the recorded scene. It would be best if the capturing of the two images is automated through the hardware, triggered by single command from the end user. In our algorithm we assume perfect spatial alignment of the overexposed and the underexposed image. The second constraint for low memory consumption is contented by using the pixel by pixel approach, in which only a minimal amount of memory is consumed for processing the pixels with respective places in the overexposed and the underexposed images. Fig. 1. An example of the overexposed (left side), and the underexposed (right side) image taken from the same scene 642

3 Finally, in order to meet the last constraint, this algorithm is performed in the HSV, rather than in the RGB color space. The HSV space allows simple migration between grayscale and color images, in a manner of processing only the V (Value) channel, or all three (Hue, Saturation, Value) channels, respectively. On the other hand, we benefit from the several advantages the HSV space has over RGB space, the most important being the fact that the HSV space is more similar to the psycho physical representation of colors by the human visual system. A. The luminance transfer function The high dynamic range of the light intensity in the real scene cannot be fully represented by a single image with single exposure value. In Fig. 1, an example of one fairly static scene captured with different exposure values is shown. As it can be seen from the figure, the left side image is the overexposed image, and the objects with lower radiance in the scene are clear and rich in detail, while brighter areas and objects can not be observed, due to the white saturation of the image. On the other hand, the right side image, underexposed image, reveal the detail from the objects with higher radiance in the real scene, however the less radiant objects are too dark, and some of them black saturated. If we assume that the normalized average calculated from these two images is a fair representation of the light intensity in the real scene, we can construct a luminance transfer function, Fig. 2. This function shows the manner of mapping the real scene light intensities into image luminance values for the cases of the overexposed and the underexposed images. From Fig. 2 the difference in the global brightness level between the overexposed and the underexposed images is evident, as well as the saturation areas, white for the overexposed and black for the underexposed image. B. The exposure fusion process The main idea in the proposed algorithm is to construct the approximation of the ideal luminance transfer function by translation of certain parts of the transfer functions for the overexposed and the underexposed images. In the following, we explain the procedure for constructing such transfer function, and by itself this procedure is enough to obtain a fused result if the input is grayscale. For color images, few further adjustments should take place, which will be explained in the next subheading. 1) Grayscale images: The processing of a grayscale images is performed using only the V channel of the color image in the HSV color space, or using the whole input image, if it is already grayscale. The procedure for obtaining an ideal luminance transfer function can be understood observing the drawing in Fig. 3. Initially, two threshold values are defined, T H1 and T H2. These values have to enclose the saturation areas of the transfer functions from both images. In our implementation, T H1 is 5% of the maximum possible luminance value in the image, and T H2 equals 95 % of the same value. Then, three different classes of pixel pairs are defined according to the luminance values of the pixels in the respective places from the overexposed and the underexposed images. For every class, different method of constructing the luminance value of the pixel in the fused image is implemented. Luminance Luminance Light intensity Fig. 2. The luminance transfer functions for images shown in Fig. 1 It is clear that in order to get single fused image from the recorded scene, in which all the detail will be present, the ideal luminance transfer function, Fig. 2, must be pursued. In low dynamic range real scenes approximation of the ideal luminance transfer function can be obtained by optimal choice of the exposure value. In high dynamic range scenes, such value does not exist, and the ideal luminance transfer function must be constructed based on the available data in the overexposed and the underexposed images. Light intensity Fig. 3. Approximation of an ideal luminance transfer function The decisions made and the calculations performed during the processing of one pair of pixels are shown in Fig. 4. The two luminance values are read from the respective positions, i- th row and j-th column, in the two input images. This operation is repeated for every pixel in the fused image. The luminance values of the pixels in the fused image are calculated using the formulas (1), (2) and (3). diff f = un+ (1) 4 For the white saturated class, the luminance values f for the pixels in the fused image are obtained with translation of the luminance values in the underexposed image by some portion of the difference between the overexposed and the underexposed image, diff, (1). This is coherent to a slight brightness increase of the underexposed image, in order to match the ideal luminance transfer function. 643

4 The value of the brightness increase should not be too high, because it may impose new white saturation and loss of detail in those areas. Our experiments show that the optimal increase is by quarter of the difference diff. overexposed image. Luminance = ov un < T H1 Calculate the difference diff = ov - un Yes underexposed image. Luminance = un class = black saturated. in the native grayscale images). The equation (3) is constructed so that a smooth luminance transfer function is established. It connects the black saturated and the white saturated classes, and eliminates the abrupt changes in the luminance values. For its border cases, ov = M, or un = 0, the equation (3) reduces into equations (1) or (2), respectively. 2) Color images: In the case when a color output image is required, two input images must be full color images in the HSV color space. The V channel is processed in the same way as explained in the previous subheading. The other two channels are processed as following. The channel S, which represents the color saturation of the observed pixel, imposes a great deal of the subjective quality of a color image, because the standard human observer tends to give higher quality grade to the images with more saturated (pure) colors. To address this fact, in our algorithm we implement the saturation maximization procedure, which is depicted on Fig. 5. No ov > T H2 Yes class = white saturated. overexposed image. Saturation = Sov Hue = Hov underexposed image. Saturation = Sun Hue = Hun No class = weighted average. Pixel (i,j) from the fused image. Luminance = Vfu ov un diff class Calculate the luminance of the pixel (i,j) in the fused image, based on class, diff, ov and un Calculate the visual saturation factor VSFov = Sov(1-Vfu) Calculate the visual saturation factor VSFun = SunVfu Fig. 4. The operations performed on one pair of pixels For the black saturated class, the data from the underexposed image in unusable and the fused image is constructed solely from the pixels in the overexposed image, with slightly lowered luminance values, again by quarter of the difference diff diff f = ov (2) 4 In the regions of the image that are neither black, nor white saturated, the weighted average class, the fused image is constructed as weighted average from overexposed and the underexposed image, using diff diff ov ( M ov) + un + un 4 4 f = (3) M diff where M is the maximum possible luminance value in the images (e.g. 1 for V in the HSV images, or 255 for luminance Sfu = VSFun Hfu = Hun Yes VSFov < VSFun No Sfu = VSFun Hfu = Hov Fig. 5. Saturation maximization and hue selection The information for saturation by itself is not enough to estimate the human observed saturation, because the absolute luminance in the same place is also important. Too bright or too dark regions, although with highly saturated colors, will be perceived as low saturation regions by the human observer. We define the visual saturation factor, VSF, in order to create a measure for the saturation as viewed by the human, taking into account the total amount of luminance too. This factor is calculated differently for the overexposed and for the underexposed image. By comparing the two results and using the higher, visually more saturated colors in the fused image can be achieved. For the saturation of the fused image the VSF values are used rather than the S values from the input images, 644

5 in order to avoid rapid changes in saturation in neighboring pixels, which will create unpleasant noise-like color artifacts in the fused image. The H (Hue) channel from the HSV color images corresponds to the exact tone of the colors in the image, so its value must not be altered by the algorithm, for the fused image to contain the original colors from the scene. Our algorithm is designed simply to choose between two (or in most cases only one) offered values for H, by the overexposed and the underexposed image. The selected value for H is from the pixel with higher VSF value, which contributes to the visual quality of the fused image. III. EXPERIMENTAL RESULTS The proposed algorithm was tested on many images obtained with digital camera using bracketed exposure values. The results were compared to previously published algorithms [4] and [6]. In Fig. 6, the results obtained by processing the images from the example in Fig. 1 are presented. The result using the algorithm [4] with fixed values for block size and blending support for speed is shown in Fig. 6 a). Gradual luminance changes in form of halos are clearly visible around objects and edges in the image. In Fig. 6 b), the result of the algorithm [6] is presented, which has much higher visual quality. The result of the application of our algorithm is shown in Fig 6 c). As it can be observed, our algorithm creates better, or comparable result with the other two algorithms. However, our algorithm is quite less complex, especially compared to the algorithm [6], which calculates two Laplacian decompositions of the input images, and one reconstruction from the fused Laplacian pyramid. The pyramid in the algorithm [6] must be calculated to the level of one pixel, in order to avoid the tiling effect in the output image. The algorithm [4] can produce better visual result if the iterative calculation of the block size and blending support is performed, but in that case this algorithm is even more complex and slow than the algorithm [6]. The moving objects (the car observed through the window) in the scene, are poorly handled with the proposed algorithm. This is due to the fact that this object is not present in one of the input images (see Fig. 1, the car does not exist in the underexposed image) and is present in the other. Mentioned above, our algorithm assumes perfect alignment of the two input images, and there is no procedure implemented that solves this kind of difficulty. On the other hand, for such cases the priority criterion should be defined, because it is unknown which image objects are more important for the end user (e.g. for this case, the car, or the objects hidden by the car). Our algorithm exhibits some sort of priority criterion, in the cases of respective pixels with black and white saturation in the same time, giving priority to the data from the overexposed image, because firstly the black saturation is checked, Fig. 4. This is intentionally designed in such way, in order to obtain generally brighter output image rather than generally darker, which also proves to be more desirable for the visual quality assessment by the human observer. In Fig. 7 few more images are shown depicting the results obtained using the proposed algorithm on different color input images. Alongside the successful luminance fusion, the effect of saturation maximization procedure is evident, resulting in more vivid and colorful fused image. These examples further prove the good performance of the proposed algorithm. IV. CONCLUSIONS In this paper we proposed a new algorithm for fusion of images with different exposure values. The algorithm is fast and simple, and works in the HSV color space. It is designed to process one pixel at a time, operating in very low number of instructions, which makes it a perfect choice for implementation on low memory and low processing power platforms. The algorithm is tested against more complex previously published algorithms on many images. The results show that although simpler, this algorithm produces comparable or sometimes better results than other algorithms. Future work should include development of the procedure for optimal selection of exposure values, targeting the maximization of usable data from the overexposed and the underexposed images. Also, adaptive thresholds T H1 and T H2 should be considered, in order to match the statistical distribution of the luminance values in the given images. Finally, detail or object priority criterion could be implemented, to resolve the non-persistent object problem (like the car in the first example). Of course, all these improvements will make the algorithm more complex and harder to implement on targeted platforms, so further research should be performed in finding the efficient running methods for the intended procedures. ACKNOWLEDGMENT The results were obtained in the course of a research project commissioned and funded by NXP Software B.V., Eindhoven. REFERENCES [1] Steve Mann, Rosalind W. Picard, On being undigital with digital cameras: Extending Dynamic Range by Combining Differently Exposed Pictures, M.I.T. Media Laboratory Perceptual Computing Section Technical Report No. TR-323, 1994, also appears in the Proceedings of the 46th Annual Imaging Science & Technology Conference, Washington D.C., USA, May 1995, pp [2] Paul E. Debevec, Jitendra Malik, Recovering high dynamic range radiance maps from photographs, 24th annual conference on Computer graphics and interactive techniques, SIGGRAPH 97, Los Angeles, California, Aug 1997, pp [3] Raanan Fattal, Dani Lischinski, Michael Werman, Gradient domain high dynamic range compression, Proc. of the 29 th annual conference on Computer graphics and interactive techniques, San Antonio, Texas, July 2002, pp [4] Ardeshir Goshtasby, Fusion of multi-exposure images, Image and Vision Computing, vol. 23, pp , [5] Ron Rubinstein, Fusion of Differently Exposed Images, Technion, Israel Institute of Technology, Final Project Report, Oct [6] Tom Mertens, Jan Kautz, Frank Van Reeth, Exposure Fusion, Proc. of the 15 th Pacific Conference on Computer Graphics and Applications, Maui, Hawaii, Oct/Nov 2007, pp [7] Peter J. Burt, Edward H. Adelson, The Laplacian Pyramid as a Compact Image Code, IEEE Trans. On Communications, vol. com-3l, no. 4, pp , April [8] J. M. Ogden, E. H. Adelson, J. R. Bergen, and P. J. Burt, Pyramidbased computer graphics, RCA Engineer, vol. 30, no. 5,

6 a) Goshtasby, [4] b) Mertens et al, [6] c) The proposed algorithm d) Preview of the input images Fig. 6. Visual comparison of the algorithms applied on grayscale input images Fig. 7. The proposed image fusion algorithm applied on different pairs of color input images 646