Automatic High Dynamic Range Image Generation for Dynamic Scenes IEEE Computer Graphics and Applications Vol. 28, Issue. 2, April 2008 Katrien Jacobs, Celine Loscos, and Greg Ward Presented by Yuan Xi School of Electrical Engineering and Computer Science Kyungpook National Univ.
Abstract Requirement of HDRI generation Multi-exposure LDR images Static scene throughout LDR images Proposed method Ghost-free LDRI alignment Focus on camera Deleting moving objects Without camera curve Independent from contrast between background and moving object 2/26
High dynamic range Weakness of LDR images Introduction Loss of information in under or over exposed area High dynamic range image Combining details in multi-exposure LDR images Problems of High dynamic range image processing Acquirement of exposure time Changing exposure time manually Ghost Camera movement Object movement 3/26
Instance for ghost free method Fig. 1. (a) A sequence of LDRIs captured with different exposure times. Several people walk through the viewing window. (b) An HDRI created from the sequence shown in (a) using conventional methods, showing ghosting effects (black squares). (c) Uncertainty image (UI) shows regions of high uncertainty (bright) due to the dynamic behaviour of those pixels. (d) HDRI of the same scene after applying movement removal using UI. 4/26
Debevec s method Background Depend on good alignment between LDRIs Rescal s method Providing manual image alignment method Reinhard s method Generating binary-map with weighted variance between LDRIs Kang s method Using camera response function to normalize LDRIs Performing local image alignment using gradient-based optical flow 5/26
MTB method Median threshold bitmap Threshold Median pixels value Ghost detection Getting summation of all bitmap image Sand s method Classifying pixel neither 0 or N(number of LDR images) as movement Basis on feature-based method Without camera response curve Khan s method(segmentation method) Using kernel density estimation method to estimation motion area 6/26
HDRI Generation: An Overview Flowchart of proposed method Fig. 2. HDRI generation methodology: the rounded white and grey boxes are processes that operate on input data and produce output data. The rounded grey boxes are modules developed for this paper. 7/26
Camera Alignment Generate alignment method Calculating camera transformation with scene features Euclidean transformation Rotation and translation Scene features Weakness Different intensity Different color Edge Impossible find edges perfectly in every image 8/26
Failing instance of edge finding Fig. 3. (a,b) Two LDRIs captured with different exposures. (c,d) Edge images of two LDRIs. (e,f) Bitmap images of two LDRIs after applying MTB transformation. 9/26
Proposed alignment method Using MTB method instead of canny edge detection Threshold set as median intensity of each image Using optimization method XOR E N D Euclidean transformation NO! Minima? YES! 10/26
Movement Detection Generate alignment method Detecting movement clusters Clusters of pixels affected by movement in any of LDRIs Using two evaluation method Movement detection based on variance Contrast-independent movement detection 11/26
Movement detection based on variance Focus on variance over irradiance images Pixels affected by movement showing large irradiance variation Movement derived from variance image Achieving variance image Definition of weighted variance: Weighted sum of squares at each pixel over the square of weighted average, the quantity minus 1 N N 2 Wi(,) klei(,) / Wi(,) kl i= 0 i= 0 N N 2 2 Wi klei kl Wi kl i= 0 i= 0 VI(,) k l = 1 ( (,) (,)) /( (,)) where Wi (,) kl is weight used during HDRI generation. (1) Aim of weight-processing Decreasing effect of redundancy information stored in distortion pixels 12/26
Weight-function Hat function 13/26
Binary image making with threshold Setting threshold to 0.18 Flowchart of proposed method Fig. 4. An adaptation of figure 2 illustrates where movement detector based on variance fits inside general HDRI generation framework. Variance detector requires knowledge of camera curve, and therefore movement detector takes place after camera curve calibration. 14/26
Weakness of movement detection based on variance Other influences existing, besides remaining camera misalignment and movement object Camera curve» Fail to convert intensity values to irradiance values Weighting factors Inaccuracies in exposure speed and aperture width 15/26
Contrast-independent movement detection Definition of entropy In information theory Uncertainty that remains about a system, after having taken into account observable properties Entropy H( X) of variable X is given by: HX ( ) = PX ( = x)log( PX ( = x)) x where X is random variable with probability function px ( ) = PX ( = x). x ranging over a certain interval, for instance [0,255]. (2) 16/26
Entropy providing information Entropy of image has positive value between [0, log( M ) ] The lower entropy, the more less different intensity values Actual order or organization of pixel intensities in image does not influence entropy Only focus on entirety, ignore specific distribution Applying scaling factor on intensity values of image does not change its entropy, if intensity values do not saturate Entropy of image gives measure of uncertainty of pixels in image All intensity values are equal: entropy is zero All intensity values are different: entropy is one 17/26
Calculating UI(uncertainty image) with 2D window Generating entropy image from LDR image One pixel value in entropy image corresponding to entropy value of around region(2d window) at same position M 1 H( kl, ) = PX ( = x)log( PX ( = x)) i (3) x= 0 where PX ( = x) is derived from normalized histogram constructed from the intensity values of pixels within 2D window. and, over all pixels p in: { p L( k ω: k+ ω, l ω: l+ ω} (4) i 18/26
Uncertainty image Local weighted entropy difference υ UIkl h kl N 1 j< i i, j (,) 1, (,) N j< i i j i= 0 j= 0 υi, j i= 0 j= 0 = (5) hi, j(,) kl = Hi(,) kl H j(,) kl (6) υ i, j= min( Wi( kl, ), Wj( kl, )) (7) 19/26
HDRI Generation Final flowchart for proposed method Fig. 5. An adaptation of figure 2 illustrates where contrast-independent movement detector, explained in section V-B, fits inside general HDRI generation framework. Movement detector does not require knowledge about camera curve, therefore movement detector can take place before camera calibration. 20/26
Results Comparison of resulting aligned or not (a) (d) Fig. 6. HDRI generation and the influence of camera movement. The left column shows the entire HDRI, the right column shows an image detail in close-up for the following scenarios: no image alignment (a,d), translational alignment (b,e), translational and rotational alignment (c,f). 21/26
(b) (e) Fig. 6. HDRI generation and the influence of camera movement. The left column shows the entire HDRI, the right column shows an image detail in close-up for the following scenarios: no image alignment (a,d), translational alignment (b,e), translational and rotational alignment (c,f). 22/26
(c) (f) Fig. 6. HDRI generation and the influence of camera movement. The left column shows the entire HDRI, the right column shows an image detail in close-up for the following scenarios: no image alignment (a,d), translational alignment (b,e), translational and rotational alignment (c,f). 23/26
HDRI generation and movement(general) removal (a) (b) Fig. 7. HDRI generation and movement removal for the exposure sequence shown in figure 1 (a). (a) HDRI after object movement removal using variance detector discussed in section V-A. (b) HDRI after object removal using uncertainty detector discussed in section V-B. (c) Variance image V I used to generate (a). (d) Uncertainty image UI used to generate (b). (c) (d) 24/26
HDRI generation and movement(fluid) removal (a) (b) Fig. 8.(a) HDRI without movement removal: leaves on left hand side show considerable ghosting. (b) HDRI after movement removal using uncertainty image UI shown in (d). (c) Variance image VI. (d) Uncertainty image UI used to generate (b). (c) (d) 25/26
Conclusion And Future Work Proposed method Camera alignment Using MTB method More better performance than simple edge based methods Novel criterion : Entropy Contrast-independent movement detection Proposed definition of UI(uncertainty image) Experimental results More stable than existing method Pleasing resulting for fluid ghosts Future work Research for more better alignment method 26/26