Correction of Clipped Pixels in Color Images

Transcription

1 Correction of Clipped Pixels in Color Images IEEE Transaction on Visualization and Computer Graphics, Vol. 17, No. 3, 2011 Di Xu, Colin Doutre, and Panos Nasiopoulos Presented by In-Yong Song School of Electrical Engineering and Computer Science Kyungpook National Univ.

2 Proposed method Abstract Enhancement of luma and chroma of clipped pixels Based on the strong chroma spatial correlation Between clipped pixels and surrounding unclipped area Procedure Identifying the clipped areas Partition of the clipped areas Estimating chroma Correcting Clipped R, G, or B color channels Smoothing of the boundaries 2 / 27

3 Introduction Representation of high dynamic range (HDR) Limited dynamic range of conventional display How to properly display HDR content on a conventional low dynamic range (LDR) media Using tone mapping Reducing strong contrast from the HDR scene Preserving the image details and color apperance Appreciation of the original scene content Lost of Information Very bright or dark parts of a scene Clipping upper or lower displayable limits 3 / 27

4 Proposed method Color distortion due to the clipping For a clipped pixel, often not clipping all three red, green, and blue channels, nor occurring the same amount of clipping in each channel Fig. 1. Example of color distortion due to the clipping. (a) A correct exposure image, (c) the corresponding overexposed image, and (e) the enhanced image with corrected luma for clipped pixels. (b), (d), and (f) The plots of the RGB intensities versus the pixel index along the highlighted horizontal lines of images (a), (c), and (e), respectively. 4 / 27

5 Proposing effective clipped-pixel-enhancing algorithm Automatically restoring both the luma and chroma of the clipped pixels Exploiting strong correlation in chroma» Between saturated pixels and surrounding unsaturated pixels Experiment of proposed method Objective and subjective quality evaluations 5 / 27

6 Our Proposed Method Aim of proposed method Restoring lost information in overexposed color images Based on the strong spatial correlation in the chroma channels Using YCbCr color space Fig. 2. Normalized autocorrelation of R, G, B, Y, Cb, and Cr signals (average over 24 true-color Kodak images). 6 / 27

7 Flowchart Fig. 3. Flowchart of our proposed method. 7 / 27

8 Identify clipped areas Simply selecting pixels from all three color channels Often generating very small isolated clipped areas and large clipped areas with small holes Due to image noise light from the scene, camera response, sensor noise, and color filter array interpolation Slightly lower than the maximum value Proposed method Apply of bilateral filter Removing noise Applying threshold τ to each color channel of pixel Experimentally selecting τ to for 8-bit per channel images 8 / 27

9 Fig. 4. Example of clipped areas. (a) Clipped image with a rectangular region of interest,(b) clipped areas in the region of interest identified with a simple threshold, and (c) clipped areas in the region of interest identified with our proposed method. 9 / 27

10 Partition clipped areas Partitioning group the clipped pixels into regions with similar chroma Partitioning clipped areas into spatially disconnected regions Belonging different surfaces and different chroma Containing multiple clipped objects (different colors) Segmenting each region according to chroma Eliminating illumination difference on the same color surface Considering only chroma quantities (Cb and Cr) 10 / 27

11 2 or 3-channel saturated pixels (pixels with heavier saturation) Considered as one subregion 1-channel saturated pixels Further segmented by using histogram based multithreshold algorithm Fig. 5. Example of clipped area partition. (a) The clipped areas before partition (white pixels) and (b) the clipped areas partitioned into regions with similar chroma. 11 / 27

12 Correct chroma Finding an unclipped region with similar chroma Selecting neighboring pixels with similar color to the clipped region as seed points Apply of region growing algorithm from each seed point Union of all surrounding areas obtained from different seed points as the surrounding region for a clipped region Fig. 6. Example of a surrounding region. (a) Clipped image, (b) clipped areas (in color) superimposed on the image luma, and (c) the surrounding region (white pixels) for the clipped ara on the girl s arm. 12 / 27

13 Problem of missing pixels Not used in the interpolation Clipped pixels or nonclipped pixels with different chroma to the current clipped region Not applying traditional convolution Solve of problem Use of normalized convolution Adjusting the filter weights Certainty component m 1, m( x, y) = 0, cxy (, ) = for unsaturated pixels in the surrouding region and saturated pixels that have been corrected otherwise [ ] cxy (, ) mxy (, ) hxy (, ) mxy (, ) hxy (, ) 13 / 27

14 Ω : the saturated pixel set {( xy, ) : Rxy (, ) τ, orgxy (, ) τ, orbxy (, ) τ} Ω= Ω1, Ω2, and Ω : set of clipped pixels with 1, 2, and 3 3 saturated channel Closing clipped pixel to the surrounding region Strong correlation with the surrounding unclipped pixels Fig. 7. Example of chroma correction. The Cr channel of (a) unclipped image, (b) clipped image, and (c) corrected image using our proposed algorithm. The difference between (a) and (b) is shown in (d), and the difference between (a) and (c) is shown in (e). 14 / 27

15 Correct RGB values Correct 2-channel saturated pixels Use of conversion from RGB to YCbCr RGB and YCbCr values» Range of 0.0 ~ 1.0 Y R (2) Cb = G Cr B [ ] [ R G B] [ ] [ ] Cb = (3) Cr = R G B (4) U : known (unsaturated channel) S and S : two clipped channel ( saturated channel) 1 2 T T 15 / 27

16 Solving for using the two equation (3) and (4) f 2 S 1 = f 1 ( U, Cb, Cr) S 2 = f 2 ( U, Cb, Cr) where 1 and f are functions of U, Cb, and Cr, and S denotes the corrected value of color channel S Example of correct 2-channel saturated pixels R and G : two clipped unknown channels B : unclipped channel R = ( Cb B) ( ) ( Cr B) ( ) / ( ) ( ) ( ) G = ( Cb B) ( Cr B) ( ) / ( ) ( ) ( ) 16 / 27

17 Correct 1-channel saturated pixels Similar to correcting 2-channel saturated pixels S : clipped channel (saturated channel) U1andU 2 : two known channel ( unsaturated channel) S1+ S2 S =, where 2 S1 = f3( U1, U2, Cb) S = f ( U, U, Cr) Example of correct 2-channel saturated pixels R : clipped unknown channels G and B : two unclipped channel R1+ R2 R =, where 2 Cb ( ) G B R1 = Cr ( ) G ( ) B S2 = / 27

18 Correct 3-channel saturated pixels Needed to solve three equations First, estimating luma Y value of the 3-channel saturated pixels based on the surrounding region 2D Gaussian function Not assuming center of the Gaussian function as the centroid of the clipped region 2 2 a( x x0) + 2 b( x x0)( y y0) + c( y y0) g( x, y) = Ae + B where ABabcx,,,,, 0, and y a b 0 are the parameters, and is b c positive definite Solving least-squares surface-fitting problem» Use of optimization form n 2 a b arg min [ Yi g( xi, yi) ], Subject to : 0 ABabcx,,,,, 0, y0 i= 1 b c where ( xi, yi, Yi) is the i th the surrounding area, xi and yi represent the pixel location, Y i is the luma at pixel ( xi, y i), and the symbol " " stands for positive definite. 18 / 27

19 Decomposing symmetric and positive definite matrix M = LL where L is a lower triangular matrix T M» For the matrix a b b c» In the constraint, we have a b l11 0 l11 l21 b c = l l 0 l Substituting a, b, and c using the new variables l11, l21, and l22 n,, 11, 21, 22, 0, 0 = 1 [ Y gx y ] arg min (, ) ABl l l x y i i i i 2 where l11 2 l21+ l22 2 ( x x0 ) l11l21( x x0 )( y y0 ) ( y y0 ) g( x, y) = Ae + B 19 / 27

20 Solving optimization Standard least-squares fitting algorithm In the end, Solving corrected RGB values by using (2) R Y G Cb = B Cr Lower bound α d ( α 1) + 1, when 0 < d < d α = α 0, when d d0 0 0 d0 where the enhancement factor α0 is a constant and α 0 > 1, the constant d0 is the transition width, and d denotes the distance between the current saturated pixel and the closest unsaturated pixel in the surrounding region. 20 / 27

21 Smooth enhancement Small jumps of enhance values between adjacent 1, 2, and 3-channel saturated regions Use of different strategies Calculating saturated RGB channel Reduction of disturbing contours Obtainment of natural looking enhanced images Fig. 8. Illustration of the smoothing process between 1, 2, and 3- channel saturated regions 21 / 27

22 Adjusted saturated channel value ω ω 1 Px ( 0, y0) = Px ( 0, y0) + 1 Pxy (, ) (5) ω0 ω0 N ( xy, ) A x 0, y 0 where ω0 is the width of the transition band, ω is the distance between ( x0, y0) and ( x, y ), and N is the number of pixels in the area A x 0, y Choosing ω 0 Reasonable range of» 3 ~ 10 pixels to adjust the amount of the smoothing ω 0 Fig. 9. Example of the smoothing effect. (a) The saturation category map, (b) enhanced image before smoothing, and (c) enhanced image after smoothing. 22 / 27

23 Experimental Results Experiment of proposed method Objective and subjective evaluation Fig. 10. Thumbnails of our test images. In reading order: girl, landscape, baby_girl, mountain, shoes, sunset, kodim03(caps), kodim05(motor-cycles), kodim06(boat), kodim12(beach), kodim16(lake), kodim21(light-house), and kodim23(parrots). 23 / 27

24 Objective evaluation Table 1. Quality comparison between different algorithms 24 / 27

25 Result image (1) Fig. 11. Results of clipped pixel enhancement. For each row, we show (from left to right) the original image, clipped image, clipped areas superimposed on the image luma, enhanced image using Bayesian algorithm, and enhanced image using our proposed algorithm. 25 / 27

26 (2) Fig. 12. Two virtual exposures of HDR images, The four column of images correspond to (from left to right) the original unclipped LDR image, the clipped LDR image, the enhanced image obtained by the Bayesian algorithm, and the enhanced image produced by our proposed algorithm, respectively. 26 / 27

27 Proposed method Conclusion Effective method for enhancing clipped pixels in color image Use of the strong correlation between the chroma of the clipped pixels and surrounding unclipped pixels Greatly reducing color distortion caused by clipping Objective and subjective evaluation Visually more plausible than clipped images More enhanced images produced by previous algorithm 27 / 27