REDUCING the backlight of liquid crystal display (LCD)

Transcription

1 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 22, NO. 12, DECEMBER Enhancement of Backlight-Scaled Images Tai-Hsiang Huang, Kuang-Tsu Shih, Su-Ling Yeh, and Homer H. Chen, Fellow, IEEE Abstract Switching the liquid crystal display (LCD) backlight of a portable multimedia device to a low power level saves energy but results in poor image quality especially for the low-luminance image areas. In this paper, we propose an image enhancement algorithm that overcomes such effects of dim LCD backlight by taking the human visual property into consideration. It boosts the luminance of image areas below the perceptual threshold while preserving the contrast of the other image areas. We apply the just noticeable difference theory and decompose an image into an HVS response layer and a background luminance layer. The boosting and compression processes, which enhance the visibility of the low-luminance image areas, are carried out in the background luminance layer to avoid luminance gradient reversal and over-compensation. The contrast of the processed image is further enhanced by exploiting the Craik-O Brein-Cornsweet visual illusion. Experimental results are provided to show the performance of the proposed algorithm. Index Terms LCD backlight, ambient light, image enhancement, human visual system, contrast, JND. (a) Fig. 1. An image displayed on an LCD with (a) 100% backlight and (b) 10% backlight. Note the effect of dim backlight on the color and the detail of the image. (b) I. INTRODUCTION REDUCING the backlight of liquid crystal display (LCD) prolongs the battery life of hand-held electronic devices such as smart phones [1]. However, such energy saving comes at the cost of image quality. An illustration of how dim backlight affects image quality is given in Fig. 1. We can see that the detail of the dark regions becomes less visible and the image color becomes less vivid. The dimmer the backlight, the worse the image quality is. Our goal of this work is to alleviate such undesired effects of dim LCD backlight on image quality for the case where the backlight intensity may be only 10% or less of the full level. Because the perceived image quality is related to the response of human visual system (HVS) to the image signal, a perception-based approach that takes the property of HVS response into consideration is desirable. Manuscript received March 21, 2013; accepted June 25, Date of publication July 9, 2013; date of current version September 26, This work was supported in part by a grant from the National Science Council of Taiwan under Contract NSC E MY3, a grant from National Taiwan University under Contract NTU-CESRP-102R7609-2, and a grant from the Himax Technologies, Inc. The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Jean-Baptiste Thibault. T.-H. Huang and K.-T. Shih are with the Graduate Institute of Communication Engineering, National Taiwan University, Taipei 10617, Taiwan ( tshuang1983@gmail.com; shihkt@gmail.com). S.-L. Yeh is with the Department of Psychology, Graduate Institute of Brain and Mind Sciences, and Neurobiology and Cognitive Science Center, National Taiwan University, Taipei 10617, Taiwan ( suling@ntu.edu.tw). H. H. Chen is with the Department of Electrical Engineering, Graduate Institute of Communication Engineering, and Graduate Institute of Networking and Multimedia, National Taiwan University, Taipei 10617, Taiwan ( homer@cc.ee.ntu.edu.tw). Color versions of one or more of the figures in this paper are available online at Digital Object Identifier /TIP IEEE Among various measurements of HVS response, contrast is an important one. There are various definitions of contrast, such as Michelson contrast, logarithmic ratio, and Weber fraction. The Michelson contrast is defined with a sinusoidal stimulus and the Weber fraction with a step increment stimulus. However defined, the resulting contrast measurements are not linearly correlated to the HVS response. The relationship between the HVS response and the contrast is normally modeled by a transducer function, which transforms an image to a new domain where the HVS response can be better represented. The just noticeable difference (JND) theory [1] is often employed in the derivation of the transducer function. Most previous methods for enhancing dimmed images deal with 50% or more LCD backlight [2] [15]. These methods suffer from detail loss and color degradation (see Section II.A) for the scenario considered in this work. Over-enhancement of dark regions is another common drawback of these methods. When the LCD backlight intensity is sufficiently low, we can hardly see the content of an image. The detail of the image becomes invisible when the luminance is below a certain threshold, which depends on the luminance of the ambient light. The brighter the ambient light, the higher the threshold is. This HVS property serves as the guiding principle for our algorithm, which aims at restoring the detail of dark image regions when illuminated with dim backlight without affecting the appearance of the other regions. The remainder of the paper is organized as follows. We discuss the psychological properties of HVS and the effects of dim backlight on image visibility in Section II. Section III describes the possible methods for dim image enhancement. The proposed algorithm is described in Section IV.

2 4588 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 22, NO. 12, DECEMBER 2013 The experiments, performance evaluations, and sensitivity evaluations are presented in Sections V, VI, and VII, respectively. Evaluation metrics are described in Section VIII. Finally, the conclusion is drawn in Section IX. II. BACKGROUND We give a brief overview of the JND model, HVS response, and the effects of dim backlight in this section. A. Just Noticeable Difference JND is the smallest difference in the sensory input that is discernible by human being. To be specific, given a background luminance L and the corresponding just noticeable difference L, the HVS cannot detect a foreground stimulus if its luminance value is between L L and L + L. There are many models describing the relationship between L and L. The Weber s law, for example, targets the middle luminance range ( cd/m 2 ) [21] and represents the relation between L and L by a linear curve. The luminance of an LCD with 60% to 80% backlight usually lies in the range cd/m 2, and Weber s law works well as most pixels are bright and their luminance values are between 50 cd/m 2 and 240 cd/m 2. However, as the luminance reduces to be within the range 0 30 cd/m 2, which is the case considered here, the Weber s law is no longer appropriate. A more appropriate JND model proposed by Iranli [13] for low dynamic range of luminance describes the relation between L and L by L = J(L) = ( L 0.4 ) 2.5, (1) where J( ) is a function that returns the JND of the given luminance. We adopt this JND model in our work. B. Human Visual Response Model Our perceptual response to an image is a nonlinear function of the image luminance. An HVS model characterizes this nonlinear behavior by taking the luminance value as input and converting it to a nonnegative integer as output such that a difference of 1 in the output corresponds to a just noticeable difference in luminance [14], [15]. To realize this HVS model for practical use, one may compute L 1 = L 0 +J(L 0 ),where L 0 denotes the lower bound of the luminance range under consideration, and continue the recursive procedure L i = L i 1 + J(L i 1 ), i > 0, (2) where i is an integer and J( ) is defined in (1), until L i reaches the upper bound of the luminance range. C. Effects of Dim Backlight on Images Dim backlight affects the visual perception of an LCD image in two ways. First, it causes the image details, especially those of the dark regions, less visible or even imperceptible. This is referred to as the detail loss effect in this paper. Second, it causes color degradation because of the decrease of chrominance intensity. The dimmer the backlight is, the more the color degrades. (a) (b) Fig. 2. (a) The cone response of HVS to an image displayed with 100% backlight, assuming the entire luminance range of the image is [0] [300] /m 2 and the luminance range of the dark region is [0] [20] cd/m 2. In this example, the response of our eyes to the dark region ranges from 0 to 0.1. (b) When displayed with only 10% backlight, the luminance range of the dark region becomes [0] [2] cd/m 2 and the response of our eyes to the dark region falls to zero. We may explain these two effects of dim backlight by examining the response of human cone cells to luminance, as shown in Fig. 2. The cone response is an s-shaped curve [20] described by V I 0.74 = V m I 0.74, (3) + σ 0.74 where V/V m is the normalized response, I is the perceived light intensity, and σ, which is the half saturation parameter, is a function of the adaptation luminance (a larger σ corresponds to a brighter viewing condition). With this cone response model, we now proceed to explain the reason of the detail loss effect theoretically. Suppose the luminance of the LCD ranges from 0 to 300 cd/m 2 at full backlight and 0 to 30 cd/m 2 at 10% backlight. We can see from Fig. 2(a) that the luminance of the dark region at full backlight corresponds to cone response ranging from 0 to 0.1. However, as shown in Fig. 2(b), the cone response of the dark region drops to 0 when the image is illuminated with 10% LCD backlight. This is why the details of the dark region cannot be perceived. III. APPROACHES In essence, we need to boost the luminance of the dark image region to enhance its visibility. There are a number

3 HUANG et al.: ENHANCEMENT OF BACKLIGHT-SCALED IMAGES 4589 This way, the enhancement of the dark region is achieved at only a slight cost of the luminance range of the bright region. Hence, the effect on the perceptual contrast of the bright region is very small. Fig. 3. The luminance ranges of the dark and bright image regions for various methods. of possible ways as illustrated in Fig. 3, where the top dash line represents the maximum luminance supported by the display at 10% backlight and the bottom one represents the minimum perceptible luminance level. Method 1 boosts the dark region in such a way that the resulting luminance range of the boosted dark region is equal to its luminance range at 100% backlight. Although part of the dark region becomes visible, this method has two obvious drawbacks. First, the dark region is only partially visible because a portion of the dark region is still below the minimum perceptible level. Second, as a consequence of the fact that the resulting luminance range of the dark region overlaps with that of the bright region, this method introduces luminance gradient reversal. On the other hand, Method 2 proportionally scales the luminance of the entire image to fit within the maximum and minimum luminance levels such that the resulting image is completely perceptible. Although the dark region becomes visible, the boosting operation degrades the perceptual contrast of the bright region an undesirable effect since the HVS is sensitive to contrast [26]. We observe that the effect of the scaling and boosting operations on the contrast of the bright region is different from that of the dark region. For the bright region, the scaling operation does not change its perceptual contrast, but the boosting operation does. More perceptual contrast is lost for higher boosting level. For the dark region, on the contrary, there is no perceptual contrast loss because the whole region is originally imperceptible. Whether its luminance is boosted above the minimum perceptible luminance is the only thing that matters. Method 3, which is our proposed method, is a variation of Method 2. The luminance of the image is reallocated to the perceptible luminance range as Method 2, but the bright and dark regions are processed with different scaling and boosting factors. To preserve the perceptual contrast of the bright region, its luminance is reallocated to a luminance range slightly smaller than the perceptible luminance range. In the mean while, to enhance the perceptual contrast of the dark region, the luminance of the dark region is compressed to a small range and boosted above the minimum perceptible level. IV. PROPOSED ALGORITHM Up to this point, we have explained the effects of dim backlight using a cone response model. To enhance the backlightscaled images, two issues need to be addressed. First, we need to determine the minimum perceptible luminance threshold of the cone response described earlier. Second, we need to implement the boost and compression idea described in Sec. III. In this section, we introduce two algorithms that are developed to address these two issues. A. Prediction of the Detail Loss Effect As described earlier, the viewing condition affects the cone response of HVS, and this phenomenon is modeled using the parameter σ in (3). Usually, the value of σ is assigned according to the intensity of the ambient light, assuming that the object is not a light source. However, the assumption is not true in our case and the content of the displayed image could impact the value of σ. Besides, even if the value of σ is well determined, it is still difficult to objectively decide a threshold that predicts whether a pixel is visible. To solve this, we adopt a relatively precise model that takes both ambient light and the contrast of the displayed image into account. In this work, the detail loss effect is modeled by exploiting Huang et al. s visibility model [24]. The inputs of the model are an image, a backlight intensity level, and an ambient light intensity level, and the output of the model is a map that represents the probability of visibility of each pixel in the input image. Huang et al. predicted the probability of visibility of each pixel in an input image twice, one with full backlight intensity level and the other one with dim backlight intensity level, and used the difference of the two resulting probabilities (DoP) to model the detail loss effect. However, it is difficult to decide a threshold value for DoP to distinguish detail-loss pixels from others. Therefore, instead of computing DoP, we compute the probability P L that a pixel is visible when illuminated with full backlight but invisible when it is illuminated with dim backlight. Specifically, P L = PL F (1 P L D ), (4) where PL F and PL D, respectively, are the probabilities of visibility for the pixel when the backlight is at full and dim level. In this case, the threshold value for P L can be reasonably set to 0.5 as it indicates that 50% of the viewers are expected to see the detail loss effect. A prediction result of a test image using this modified threshold is shown in Fig. 4. Enhancing only image regions subject to the detail loss effect seems to be an ideal way to enhance the visual quality. However, directly implementing the idea will cause artifacts in the enhanced output as the perception of visibility varies across individuals. For example, one may not think the prediction results in Fig. 4(c) exactly reflect the visibility of each pixel in the image. As a result, if we directly enhance pixels according

4 4590 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 22, NO. 12, DECEMBER 2013 (a) (b) (c) Fig. 4. Prediction of visibility loss for a test image. (a) Original image illuminated with full backlight. (b) Original image illuminated with dim backlight. (c) Prediction of visibility loss due to dim backlight. All visible pixels that become invisible are marked in red. to the prediction, the resulting image would suffer from visual artifacts. Therefore, we take a conservative approach by enhancing only pixels that are darker than the darkest pixel M in the visible regions predicted by Huang et al. s visibility model. We also restrict the luminance of the enhanced pixels to a small range centered at the luminance of M, as described in the next subsection. Fig. 5. The flow chart of the proposed algorithm. B. Enhancement of Invisible Pixels As described earlier in Section III, we propose to boost and scale the luminance of pixels in the invisible regions to enhance a backlight-scaled image. To do this, we need to determine to what extent the luminance range should be compressed and scaled. Note that the boosting operation does affect the contrast and hence should be done with care to avoid contrast lost. To maximally preserve the contrast, we develop a local boosting and scaling algorithm based on the human visual response model described in Section II. C. The flow chart of the enhancement algorithm is shown in Fig. 5. The algorithm first decomposes an input image into two layers: the HVS response layer and the background luminance layer. We boost the luminance of background luminance layer while keeping the HVS response layer unchanged. This way, the HVS response to the image generated by the algorithm is the same as that to the original image. Moreover, the image is further enhanced by exploiting the Craik-O Brein-Cornsweet visual illusion [25]. We now describe the details of each step. 1) JND Decomposition: Before an image is decomposed, we need to construct an HVS response function first. The HVS response function used in this work is developed based on the human visual response model described in Section II, and it models the response of HVS to a foreground luminance L F given a background luminance L B. Denote the HVS response function by f (L F, L B ). When both L F and L B are equal to L 0,wehave f (L 0, L 0 ) = 0 because one cannot perceive the foreground when it has the same luminance as the background. We also have f (L 1, L 0 ) = 1, f (L 2, L 0 ) = 2, and so on because increasing or decreasing the HVS response value by one unit results in a just noticeable change of luminance [15]. Now we have the HVS response for some discrete foreground luminance values L 0, L 1, L 2,etc.Togoonestep further, we make the response function f continuous by linear Fig. 6. Piecewise linear interpolation of the HVS response curve. Fig. 7. Illustration of foreground and background luminance determination: The foreground luminance for the cropped block shown on the right is the pixel value of the central pixel, and the background luminance is the mean of the remaining pixels in the cropped block. interpolation. As illustrated in Fig. 6, for a given background luminance L 0, only the HVS responses of L 0, L 1, L 2,and L 3 are defined originally. The HVS responses of the other luminance values are then interpolated linearly. To apply the HVS response function to find the response of each pixel, we define the background and foreground luminance as follows. Take the selected image area in Fig. 7 as an example. We consider the pixel at the center of this area as foreground, and the remaining pixels of this image area as background. We define the background luminance to be the weighted average of the luminance values of the background pixels (We discuss the weighting in Sec. IV-B-4). Since the

5 HUANG et al.: ENHANCEMENT OF BACKLIGHT-SCALED IMAGES 4591 Fig. 8. Flow chart of the JND decomposition. spatial expansion (view angle), within which the background adaptation level can affect contrast discrimination threshold, of the HVS is about 5 degrees, the edge E of the selected image area is related to the viewing distance D by E = 2D tan 2.5π 180. (5) An image input to the algorithm is divided into the HVS response layer (i.e. f (L F, L B )) and the background luminance layer (i.e. L B ) in this JND decomposition process, which involves the recording of the HVS response and the background luminance for each pixel, as shown in Fig. 8. Note that the JND decomposition is a reversible process [15]. 2) Luminance Boosting and Compression: As described in Section III, we want to boost the luminance of the dark pixels of an image to a perceptible range, and at the same time preserve the perceptual contrast of the bright pixels. These are two conflicting objectives. But the conflict can be resolved because it takes a much smaller range of luminance for the boosted dark region to have the same span of cone response range. In other words, we trade only a small portion of the perceptible luminance of the bright region for a significant improvement of picture quality for the dark region. To achieve the above goal, the background luminance layer image is compressed and boosted as follows: { B BS, BS Bt, = (6) B t, otherwise, where B and B, respectively, are the input and output background luminance, S is the dimming factor of the display, and B t is the luminance value of the pixel M. Then an enhanced luminance layer is generated through the JND composition using the enhanced background luminance layer image and the unaltered HVS response layer image as the inputs. Note that the details of the dark image regions are preserved even under a high compression rate because the HVS responses of pixels in the enhanced image are the same as the original ones. 3) Color Restoration: After boosting, we perform inverse JND decomposition, which takes the HVS response value and the boosted and compressed background luminance value of each pixel as input and generates the enhanced luminance value as output. Denote the enhanced luminance layer image by L e. Then, the enhanced color image is obtained by ( ) 1 Le γ M e = M o, (7) L o Fig. 9. Dynamic range reduction of the lowpassed signal introduces halos in the output image [18]. The image and the corresponding intensity profile at each step of the signal decomposition process are shown. where L o is the luminance value of the original image, γ is the gamma parameter of the display, and M o and M e, respectively, are the original and enhanced pixel values of a color channel. This operation is performed on every pixel in the image. 4) Compensation for the Halo Effect: Fig. 9 shows a simplified view of the algorithm up to this point and illustrates the cause of the halo effect, which refers to the luminance reversal phenomenon at the edge of an object [9]. It would not occur if the lowpassed signal in Fig. 9 is not compressed (recall that compression is performed in the boosting and compression step of the algorithm). The HVS response layer image is a highpassed signal because the perceptual contrast recorded in this layer is mainly contributed by the high frequency components of the image, and the background luminance layer is a lowpassed signal since it is obtained by averaging the luminance of a block of neighboring pixels. Consider a step input signal as shown in Fig. 9. We can see that, after the superposition with the compressed signal, the spikes in the highpassed signal result in the halos in the output signal [18]. One may compensate for the halo effect by applying a bilateral filter, which is a commonly used edge preserving filter [9], to replace the averaging operation in the computation of the background luminance layer. It extends the concept of Gaussian smoothing by weighting the filter coefficients with the corresponding relative pixel intensities. Pixels that are very different from the central pixel in intensity have less weight even though they may be in close proximity to the central pixel. This is equivalent to a convolution with a non-linear Gaussian filter with weights based on pixel intensity. The luminance reversal can be effectively eliminated this way. However, as we can see in Fig. 10, totally eliminating the luminance reversal does not necessarily give better perceptual image quality. In fact, it often results in perceptual contrast degradation. Kingdom et al. [17] found that any form of counter-shading [16] and the combination of them leads to magnification of the perceived contrast, a visual phenomenon called the Craik-O Brien-Cornsweet illusion. The magnification is proportional to the amplitude of counter-shading and has an upper bound. When the counter-shading amplitude exceeds a certain threshold, halo effect appears. Fig. 11 shows that a proper counter-shading at the border of two areas with equal luminance leads to a desirable brightness difference at the border, whereas too much counter shading results in an undesirable luminance reversal. Therefore, the amplitude of a counter shading profile should be properly controlled.

6 4592 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 22, NO. 12, DECEMBER 2013 (a) (b) (c) Fig. 10. Exploiting the halo effect for perceptual contrast improvement of an image. (a) The original image, (b) the enhanced image with halo effect, (c) the enhenced image without halo effect. Fig. 12. Determining the maximum backlight BM and gamma correction coefficient Ɣ by linear regression. In this example, BM = and Ɣ = level) considered in this work. To visualize our results, we overcome the physical limitation by scaling down the range of pixel values to simulate a 10% backlight display scenario. Specifically, the simulated 10% backlight is created by adopting a display model [5] that expresses the luminance of a display as a function of the backlight intensity and the transmittance of LCD. As a color pixel is composed of three sub-pixels (usually red, green, and blue), let us denote a normalized sub-pixel value by x. The normalization factor is 255 for 8-bit color depth per channel. Then the luminance L of a sub-pixel with normalized pixel value x is the product of the transmittance t(x) and the backlight intensity B of the display, L = Bt(x) = SB M t(x), (8) Fig. 11. (a) An image of uniform luminance. (b) Applying a mild countershading horizontally to the image creates a brightness variation. (c) However, if the counter-shading is too strong, a noticeable luminance reversal is created. To have a proper control, we adopt a bilateral filter in the computation of background luminance layer and control the degree of edge preserving by adjusting the variance of the intensity kernel of the bilateral filter. Since the counter shading profile used in this work is generated by subtracting the background luminance layer from the original image (refer to Fig. 9), a bilateral filter with more edge preserving ability leads to smaller amplitude of the counter shading profile, and vice versa. In other words, the more high frequency components preserved in the background luminance layer, the smaller the amplitude of the profile is. To find a reasonable value of the variance of intensity kernel (the control parameter), we heuristically test different intensity kernels on the Kodak image set. The result shows that 95 is a reasonable variance value of the kernel. V. VISUALIZATION OF BACKLIGHT-SCALED IMAGES Normally, the lowest backlight intensity of TFT-LCDs is about 25% of the full backlight intensity, not low enough for the extreme scenario (10% or less of the full backlight intensity where B M is the maximum backlight and S [0, 1] is the backlight dimming ratio. For example, setting S = 0.1 leads to a 10% backlight scenario. The transmittance is a nonlinear mapping from [0, 1] to [0, 1], t(x) = x 1 Ɣ, (9) where Ɣ denotes the gamma value of the display. In this work, a View Sonic VX912 display with backlight intensity ranging from 45.9 to cd/m 2 is used. We measure the corresponding luminance for each normalized pixel and determine Ɣ and B M by linear regression under the uniform backlight assumption that every pixel in the display has the same Ɣ and B M. As shown in Fig. 12, the resulting Ɣ and B M are and , respectively. Substituting (9) into (8) yields L = Bt(x) = (B M S)x 1 Ɣ = B M (S Ɣ x) 1 Ɣ. (10) That is, scaling down the pixel value is equivalent to dimming the backlight based on the display model. Therefore, the luminance intensity of an image illuminated with dim backlight can be simulated by scaling down the normalized pixel value by a factor of S Ɣ. The simulated dim-backlight images enhanced by five methods, which are described in detail in Section VI, are visually inspected. Four example images are shown in Figs Examining the middle left and the top right image regions

7 HUANG et al.: ENHANCEMENT OF BACKLIGHT-SCALED IMAGES 4593 (a) (b) (a) (b) (c) (c) (d) (d) (e) (f) Fig. 13. A test image illuminated with 10% backlight: (a) original, (b) ABIE, (c) CBCS, (d) TABS, (e) GD, and (f) the proposed method. in Fig. 13, the face portrait in the middle of Fig. 14, and the hairs on the left cheek of the dog in Fig. 15, we can see that the proposed algorithm preserves image details better than the other four methods. In addition to luminance, the proposed algorithm enhances the image color; see, for example, the color of the carpet in Fig. 13 and the beach chairs in Fig. 16. Similar findings about the strength of our algorithm are also obtained for other test images not shown here. It should be noted that the simulated backlight-scaled images are slightly different from the real ones for two reasons. First, the display response curve cannot be perfectly fitted by the model we use. Take the fitting result in Fig. 12 for example. The display responses corresponding to the normalized pixel values ranging from 0.75 to 1 are not well fitted. Second, the intensity resolution of a simulated image is smaller than that of the real one. Nevertheless, the simulated images shown here are good enough to help the readers know how the backlightscaled images look like. (e) (f) Fig. 14. A test image illuminated with 10% backlight: (a) original, (b) ABIE, (c) CBCS, (d) TABS, (e) GM, and (f) the proposed method. (a) (b) (c) VI. PERFORMANCE EVALUATIONS We performed a subjective evaluation and an objective evaluation of the proposed method using the images shown in Fig. 17 as test images. Our method was compared with four other methods: the concurrent brightness and contrast scaling (CBCS) method [5], the adaptive backlight image enhancement (ABIE) method [12], the temporally aware backlight scaling (TABS) method [13], and the gradient domain (GD) method [7]. The CBCS method truncates both ends of the dynamic range of an image and stretches it to the full range. The ABIE method applies brightness compensation to the low frequency part of an image and local contrast enhancement to the high frequency part of the image. Similar to the CBCS method, the TABS method scales and clips the dynamic range of an image, but it minimizes the distortion of (d) (e) (f) Fig. 15. A test image illuminated with 10% backlight: (a) original, (b) ABIE, (c) CBCS, (d) TABS, (e) GD, and (f) the proposed method. the perceived brightness of the image based on the JND theory. The GD method operates on the gradient field of an image and attenuates large gradients more than small gradients. Both

8 4594 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 22, NO. 12, DECEMBER 2013 (a) (b) (c) (d) (a) (b) (e) (f) (g) (h) (c) (d) (i) (j) (k) (l) (e) (f) (m) (n) (o) (p) Fig. 17. Test images. The size of images (a)-(i) is 320x240, and the size of images (j) (p) is 240x320. Fig. 16. A test image illuminated with 10% backlight: (a) original, (b) ABIE, (c) CBCS, (d) TABS, (e) GD, and (f) the proposed method. CBCS and TABS are global enhancement methods, whereas ABIE, GD, and the proposed method are local enhancement methods. On the implementation of the methods, we used the original code of ABIE and implemented the other methods ourselves. Fig. 18. The evaluation board used in our experiment. A. Subjective Evaluation We conducted a subjective evaluation of the five methods based on a double stimuli method, using the evaluation board shown in Fig. 18. The evaluation board has two 2 LCD panels illuminated by LED backlight, whose intensity is adjustable from 0 to full. The original image is displayed on the top display and the enhanced image on the bottom display. A test session comprised a number of presentations. In each presentation, a pair of images was shown: One was the original image illuminated with full backlight, and the other was the enhanced image illuminated with dim backlight. The enhanced images were presented in random order to avoid systematic error. Twenty five subjects took part in this evaluation. Fifteen of them were considered experts in color imaging. These subjects were asked to rate the quality of each enhanced image in a ten-point Likert scale [22] with 1 as the lowest quality and 10 as the highest quality, using the original image as a reference. The results are shown in Fig. 19. The mean score of the proposed method is higher than the others for all test images. In addition, the one-way ANOVA test shows that the difference between the five methods is significant (F-value = and P-value = 5E-201). On the other hand, there is a significant difference at the 99% confidence interval if the p-value is smaller than The p-value generated by the Student s t-test for each method is listed in Table I. We can see that the proposed method outperforms the other methods. B. Objective Evaluation Objective image quality evaluation involves the use of an image quality metric to compare one test image against a reference image. Normally the illumination conditions of the two images are identical, thus the two images can be directly compared against each other. In our case, however, we need to evaluate the displayed image quality, but the two images to be compared are illuminated with different illumination

9 HUANG et al.: ENHANCEMENT OF BACKLIGHT-SCALED IMAGES 4595 (a) (b) Fig. 19. Mean subjective evaluation scores of ABIE, CBCS, TABS, GD, and the proposed method for the sixteen test images. The standard deviation of the subjective evaluation of each method is indicated by the error bar. TABLE I P-VALUE OF THE PAIRED T-TEST COMPARING VARIOUS METHODS AGAINST OUR PROPOSED METHOD (c) (d) (e) conditions: dim backlight for the enhanced image and full backlight for the original image. Therefore, we cannot directly compare the image data. Instead, we apply the method described in Section V to generate a simulated dimbacklight image for the enhanced image, while the original image data remains unchanged. The simulated dimbacklight image is then compared with the original image (the reference). We adopted the dynamic range independent (DRI) metric [19] and the visual information fidelity (VIF) metric [23] for the objective evaluation. In the DRI-based objective evaluation, contrast distortions were classified into three categories: contrast loss, over-enhancement, and contrast reversal. To distinguish the contrast distortions, we marked contrast loss by green, over-enhancement by blue, and contrast reversal by red on the output image. The results of the five methods under comparison for one test image are shown in Fig. 20. We see that contrast loss is the major distortion introduced by dim backlight. In particular, the images generated by ABIE and TABS have more contrast loss in the bright region, whereas the images generated by the other three methods have more contrast loss in the dark region. Among these five methods, CBCS and the proposed method have the least overall contrast loss. VIF measures the similarity between images by the amount of information that can be extracted from an image by the brain. The value of VIF is equal to 1 when the test image is a copy of the reference image. The results of VIF-based evaluation are summarized in Table II, which shows that, on the average, the proposed method has the best performance, and that, if the proposed method does not receive the top score in an individual test, it is very close to the best one. Fig. 20. Evaluation of an image by the dynamic range independent metric. Image pixels with degraded contrast are marked green. (a) CBCS, (b) ABIE, (c) TABS, (d) GD, and (e) Proposed method. VII. SENSITIVITY OF THE PROPOSED METHOD We analyze the sensitivity of the proposed method to the display gamma and bilateral filter parameters. For the display gamma, the possible error comes from the luminance measurement and curve fitting process. Usually, the variance of the estimated value is within 0.1. Suppose that the display gamma is γ. We generate the enhanced images with Ɣ ranging from γ 0.1 to γ The results in Fig. 21 show that the proposed method is not sensitive to the display gamma. For the bilateral filter parameters (window size, spatial variance, and range variance), the results are generated using several different combinations of these parameters. Figs. 22(a) and 22(b) show that the image contrast becomes stronger as the range variance increases from 5 to 35. Beyond 35, the image contrast is not affected. The results also show that the image contrast becomes stronger as the spatial variance increases from 3 to 9. Beyond 9, the image contrast is not affected. On the other hand, the method is not sensitive to the window size, as shown in Fig. 22(c). Overall, the above evaluation shows that the proposed method is stable because it is not sensitive to small changes of the display gamma and bilateral filter parameters. VIII. DISCUSSION In Section VI, the subjective evaluation presented shows that CBCS receives the lowest score among the five methods under comparison. However, the objective evaluation shows

10 4596 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 22, NO. 12, DECEMBER 2013 TABLE II IMAGE QUALITY EVALUATION BY VISUALINFORMATIONFIDELITY MEASURE (a) (b) Fig. 21. Evaluation of the sensitivity of the proposed method to the parameter Ɣ. that it introduces less distortion than ABIE, TABS, and GD. We investigate the inconsistency between the subjective and the objective evaluation results and find that image details of the dark regions displayed with dim backlight are more likely to become invisible for CBCS. It is the invisibility that causes CBCS to receive the lowest subjective evaluation score. However, this weakness is not reflected in the objective evaluation. More specifically, the visibility of dark regions is not considered in the image quality metrics adopted for the objective evaluation. To make this point clear, let us compare the original image (Fig. 17(k)) with the enhanced image (Fig. 14(c)). We see that the details of ceiling and wall are visible in the original image but not in the enhanced image. However, Fig. 20(a) shows this difference in visibility is not detected by the DRI metric. One may fix the inconsistency by incorporating the visibility of dark pixels into the objective quality metric. This is an issue for future research. Fig. 22. Evaluation of the sensitivity of the proposed method to the parameters in the bilateral filtering process. (a) Spatial variance and range variance in coarse level, (b) Spatial variance and range variance in fine level, and (c) Window size and range variance. IX. CONCLUSION In this paper, we have described a perceptual algorithm to enhance backlight-scaled images for extremely dim backlight scenarios where the LCD backlight may drop to 10% or even 5% of the full intensity level. Based on the JND theory and the HVS response model, the algorithm effectively enhances the visibility of image details of dark regions without affecting (c)

11 HUANG et al.: ENHANCEMENT OF BACKLIGHT-SCALED IMAGES 4597 the perceptual contrast of bright regions. The algorithm also applies counter shading to eliminate halo effect and, meanwhile, enhance perceptual contrast of the backlight-scaled image. Both subjective and objective evaluations show the superiority of the proposed algorithm. REFERENCES [1] H. Wilson, A transducer function for threshold and suprathreshold human vision, Biol. Cybern., vol. 38, no. 3, pp , Oct [2] Y. Rao and L. Chen, A survey of video enhancement techniques, J. Inf. Multimedia Signal Process., vol. 3, no. 1, pp , Jan [3] T. H. Kim, K. S. Choi, and S. J. Ko, Backlight power reduction using efficient image compensation for mobile devices, IEEE Trans. Consum. Electron., vol. 56, no. 3, pp , Aug [4] Y. K. Lai, Y. F. Lai, and P. Y. Chen, Content-based LCD backlight power reduction with image contrast enhancement using histogram analysis, J. Display Tech., vol. 7, no. 10, pp , Oct [5] W. C. Cheng and M. Pedram, Power minimization in a backlit TFT- LCD display by concurrent brightness and contrast scaling, IEEE Trans. Consum. Electron., vol. 50, no. 1, pp , Feb [6] L. Cheng, S. Mohapatra, M. E. Zarki, N. Dutt, and N. Venkatasubramantan, A backlight optimization scheme for video playback on mobile devices, in Proc. Consum. Commun. Netw. Conf., Jan. 2006, pp [7] R. Fattal, D. Lischinski, and M. Werman, Gradient domain high dynamic range compression, ACM Trans. Graph., vol. 21, no. 3, pp , Jul [8] S. I. Cho, S. J. Kang, and Y. H. Kim, Image quality-aware backlight dimming with color and detail enhancement techniques, J. Display Tech., vol. 9, no. 2, pp , Feb [9] F. Durand and J. Dorsey, Fast bilateral filtering for the display of highdynamic-range images, ACM Trans. Graph., vol. 21, no. 3, pp , Jul [10] Z. Farbman, R. Fattal, D. Lischinski, and R. Szeliski, Edge-preserving decompositions for multi-scale tone and detail manipulation, ACM Trans. Graph., vol. 27, no. 3, pp. 1 7, Aug [11] S. N. Pattanaik, J. A. Ferwerda, M. D. Fairchild, and D. P. Greenberg, A multiscale model of adaptation and spatial vision for realistic image display, in Proc. SIGGRAPH, 1998, pp [12] P.-S. Tsai, C.-K. Liang, T.-H. Huang, and H. H. Chen, Image enhancement for backlight-scaled TFT-LCD displays, IEEE Trans. Circuits Syst. Video Tech., vol. 19, no. 4, pp , Apr [13] A. Iranli, W. Lee, and M. Pedram, HVS-aware dynamic backlight scaling in TFT-LCDs, IEEE Trans. Very Large Scale Integr. (VLSI) Syst., vol. 14, no. 10, pp , Oct [14] M. Ashikhmin, A tone mapping algorithm for high contrast images, in Proc. Eurograph., 2002, pp [15] R. Mantiuk, K. Myszkowski, and H. P. Seidel, A perceptual framework for contrast processing of high dynamic range images, ACM Trans. Appl. Perception, vol. 3, no. 3, pp , Jul [16] G. Krawczyk, K. Myszkowski, and H. P. Seidel, Contrast restoration by adaptive countershading, in Proc. Eurograph., vol , pp [17] F. Kingdom and B. Moulden, Border effects on brightness: A review of findings, models, and issues, Spatial Vis., vol. 3, no. 4, pp , Aug [18] Y. Li, L. Sharan, and E. H. Adelson, Compressing and companding high dynamic range images with subband architecture, ACM Trans. Graph., vol. 24, no. 3, pp , Jul [19] T. O. Aydin, R. Mantiuk, K. Myszkowski, and H. P. Seidel, Dynamic range independent image quality assessment, ACM Trans. Graph., vol. 27, no. 3, pp. 1 10, Aug [20] J. M. Valeton and D. V. Norren, Light adaptation of primate cones: An analysis based on extracellular data, Vis. Res., vol. 23, no. 12, pp , Dec [21] A. Valberg, Light Vision Color, New York, NY, USA: Wiley, 2005, p [22] R. Likert, A technique for the measurement of attitudes, Archives Psychol., vol. 26, no. 140, pp. 1 55, Jun [23] H. R. Sheikh, A. C. Bovik, and C. D. Veciana, An information fidelity criterion for image quality assessment using natural scene statistics, IEEE Trans. Image Process., vol. 14, no. 12, pp , Dec [24] T.-H. Huang, C.-T. Kao, Y.-C. Chen, S.-L. Yeh, and H. H. Chen, A visibility model for quality assessment of dimmed images, in Proc. 4th IEEE Int. Workshop Multimedia Signal Process., Sep. 2012, pp [25] Wikimedia Found. (2013, Feb. 3). Cornsweet Illusion, Petersburg, FL, USA [Online]. Available: Cornsweet_illusion [26] E. P. Ong, X. Yang, W. Lin, Z. Lu, S. Yao, X. Lin, S. Rahardja, and B. C. Seng, Perceptual quality and objective quality measurements of compressed videos, J. Vis. Commun. Image Represent., vol. 17, no. 4, pp , Aug Tai-Hsiang Huang received the B.S. degree in electrical engineering from National Taiwan University, Taipei, Taiwan, in 2006, where he is currently pursuing the Ph.D. degree with the Graduate Institute of Communication Engineering. His current research interests include perceptual based image and video processing. Kuang-Tsu Shih received the B.S. degree in electrical engineering from National Taiwan University, Taipei, Taiwan, in He is currently pursuing the Ph.D. degree with the Graduate Institute of Communication Engineering. His current research interests include color image processing, color science, and computer vision. Su-Ling Yeh received the Ph.D. degree in psychology from the University of California, Berkeley, CA, USA. Since 1994, she has been with the Department of Psychology, National Taiwan University, Taipei, Taiwan. She is a recipient of Distinguished Research Award of National Science Council of Taiwan, and is a Distinguished Professor of National Taiwan University. Her current research interests include multisensory integration and effects of attention on perceptual processes. She is an Associate Editor of Chinese Journal of Psychology, and serves in the editorial board of Frontiers in Perception Science. Homer H. Chen (S 83 M 86 SM 01 F 03) received the Ph.D. degree in electrical and computer engineering from the University of Illinois at Urbana-Champaign, Urbana, IL, USA. Since August 2003, he has been with the College of Electrical Engineering and Computer Science, National Taiwan University, Taipei, Taiwan, where he is an Irving T. Ho Chair Professor. He held various research and development management and engineering positions with U.S. companies over a period of 17 years, including AT&T Bell Labs, Holmdel, NJ, USA, Rockwell Science Center, Thousand Oaks, CA, USA, ivast, and Digital Island. He was a U.S. delegate of the ISO and ITU standards committees and contributed to the development of many new interactive multimedia technologies that are now part of the MPEG-4 and JPEG-2000 standards. His current research interests include multimedia processing and communications. Dr. Chen was an Associate Editor of the IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY from 2004 to 2010, the IEEE TRANSACTIONS ON IMAGE PROCESSING from 1992 to 1994, and Pattern Recognition from 1989 to He served as a Guest Editor for the IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY in 1999 and the IEEE TRANSACTIONS ON MULTIMEDIA in 2011.