Fixing the Gaussian Blur : the Bilateral Filter Lecturer: Jianbing Shen Email : shenjianbing@bit.edu.cnedu Office room : 841 http://cs.bit.edu.cn/shenjianbing cn/shenjianbing Note: contents copied from siggraph course notes <A Gentle Introduction to Bilateral Filtering and its Applications >
Blur Comes from Averaging g across Edges input * output * * Same Gaussian kernel everywhere. 2
Bilateral Filter [Aurich 95, Smith 97, Tomasi 98] No Averaging g across Edges input * output * * The kernel shape depends d on the image content. t 3
Bilateral Filter Definition: an Additional Edge Term Same idea: weighted average of pixels. BF new 1 W not new new [ I ] G G I I p s p q p q [ r p q S I q normalization factor space weight range weight I 4
Illustration a 1D Image 1D image = line of pixels Better visualized as a plot pixel intensity pixel position 5
Gaussian Blur and Bilateral Filter Gaussian blur p q GB I] G p q space Bilateral filter [Aurich 95, Smith 97, Tomasi 98] [ p q S space I q q p space ra ange 1 BF [ I ] p G W s r p q p q S space range normalization p q G I I I q
Bilateral Filter on a Height Field BF 1 [ I ] p G W s r p q G I I p q q Wp q S I output input reproduced from [Durand 02]
Space and Range Parameters 1 BF [ r W [ I ] G G I I p s p q p q p q S I q space s : spatial extent of the kernel, size of the considered neighborhood. range r : minimum amplitude of an edge 8
Influence of Pixels Only pixels close in space and in range are considered. space range p
Exploring the Parameter Space r = 0.1 r = 0.25 r = r (Gaussian blur) input s = 2 s = 6 s = 18 10
Varying the Range Parameter r = 0.1 r = 0.25 r = r (Gaussian blur) input s = 2 s = 6 s = 18
input 12
r = 0.1 13
r = 0.25 14
r = (Gaussian blur) 15
Varying the Space Parameter r = 0.1 r = 0.25 r = r (Gaussian blur) input s = 2 s = 6 s = 18
input 17
s = 2 18
s = 6 19
s = 18 20
How to Set the Parameters Depends on the application. For instance: space parameter: proportional to image size eg e.g., 2% of image diagonal range parameter: proportional p to edge amplitude e.g., mean or median of image gradients independent of resolution and exposure 21
A Few More Advanced Remarks 22
Bilateral Filter Crosses Thin Lines Bilateral filter averages across features thinner than ~2 s Desirable for smoothing: more pixels = more robust Different from diffusion that stops at thin lines close-up kernel
Iterating the Bilateral Filter I BF [ I ] [ ( n 1) ( n ) Generate more piecewise-flat images Often not needed in computational photo. 24
input 25
1 iteration 26
2 iterations 27
4 iterations 28
Bilateral Filtering Color Images For gray-level images 1 BF [ I ] p G r W s p q S S intensity difference p q G I I p q I q scalar input For color images 1 BF [ I ] p G r W s Wp q S S color difference p q G C C p q C q 3D vector (RGB, Lab) output The bilateral filter is extremely easy to adapt to your need.
Hard to Compute 1 Nonlinear BF [ I ] p G s p q G I p I q r Complex, spatially varying kernels W p q S S Cannot be precomputed, no FFT I q Brute-force implementation is slow > 10min 30
Questions? 31
Applications
Overview Denoising Tone mapping Relighting & texture editing 33
Overview Not most powerful application Not best denoising, but good & simple Denoising Tone mapping Relighting & texture editing
Basic denoising Noisy input Bilateral filter 7x7 window
Basic denoising Bilateral filter Median 3x3 36
Basic denoising Bilateral filter Median 5x5 37
Basic denoising Bilateral filter Bilateral filter lower sigma 38
Basic denoising Bilateral filter Bilateral filter higher sigma 39
Denoising Small spatial sigma (e.g. 7x7 window) Adapt range sigma to noise level l Maybe not best denoising method, but best simplicity/quality tradeoff No need for acceleration (small kernel) But the denoising feature in e.g. Photoshop is better
Overview Denoising Tone mapping Relighting & texture editing 41
Real world dynamic range Eye can adapt from ~ 10-6 to 10 6 cd/m 2 Often 1 : 10,000 in a scene 10-6 10 6 Real world High dynamic range 42
Picture dynamic range Typically 1: 20 or 1:50 Real world Black is ~ 50x darker than white 10-6 10 6 Picture 10-6 10 6 Low contrast 43
Multiple exposure photography Real world 10-6 High dynamic range 10 6 Merge HDR Merge multiple l exposure to cover full range We obtain one single image with floats per pixel But we still can t display it
The future: HDR Cameras HDR sensors using CMOS Use a log response curve e.g. SMaL, Assorted pixels Fuji Nayar et al. Per-pixel exposure Filter Integration time Fuji SuperCCD Multiple cameras using beam splitters Other computational photography tricks
Problem: Contrast reduction Match limited contrast of the medium Preserve details 10-6 High dynamic range 10 6 Real world Picture 10-6 10 6 Low contrast 46
Tone mapping Input: high-dynamic-range image (floating point per pixel)
Naïve technique Scene has 1:10,000 contrast, display has 1:100 Simplest contrast reduction?
Naïve: Gamma compression X X (where =0.5 in our case) But colors are washed-out. Why? Input Gamma 49
Gamma compression on intensity Colors are OK, but details (intensity high-frequency) are blurred Gamma on intensity Intensity Color
Oppenheim 1968, Chiu et al. 1993 Reduce contrast of low-frequencies (log domain) Keep high frequencies Low-freq freq. Reduce low frequency High-freq. Color
The halo nightmare For strong edges Because they contain high frequency Low-freq freq. Reduce low frequency High-freq. Color
Bilateral filtering to the rescue Large scale = bilateral (log intensity) Detail = residual [Durand & Dorsey 2002] Large-scale Output Detail Color
Contrast reduction Input HDR image Contrast too high! 54
Input HDR image Contrast reduction Intensity Color 55
Input HDR image Contrast reduction Intensity Large scale Color Bilateral Filter (in log domain!) Spatial sigma: 2% image size Range sigma: 0.4 (in log 10) 56
Input HDR image Contrast reduction Intensity Large scale Bilateral Filter Detail Color Detail = log intensity large scale (residual) 57
Input HDR image Contrast reduction Intensity Large scale Reduce contrast Large scale Bilateral Filter Detail Color 58
Input HDR image Contrast reduction Intensity Large scale Reduce contrast Large scale Bilateral Filter Detail Preserve! Detail Color 59
Input HDR image Contrast Output reduction Intensity Large scale Reduce contrast Large scale Bilateral Filter Detail Preserve! Detail Color Color 60
Contrast reduction in log domain Set target large-scale contrast (e.g. log 10 10) In linear output, we want 1:10 contrast for large scale Compute range of input large scale layer: largerange = max(inloglarge) min (inloglarge) Scale factor k= log 10 (10) / largerange Normalize so that the biggest value is 0 in log outlog= inlogdetail + inloglarge * k max(inloglarge)
Alternative explanation Explanation 1 (previous slides): outlog =k inloglarge + inlogdetail (ignoring offset) Explanation 2 outlog = k inlogintensity + (1-k) detail Reduce contrast of full intensity layer Add back some detail Same final effect since inlogdetail+inloglarge scale = inlogintensity But different philosophy: decomposition vs. add back detail
Demo Don t try at home without FAST bilateral filtering 63
Denoising vs. tone mapping Denoising: decompose into noise+signal si n Throw away noise, keep signal Small kernel Tone mapping Decompose into large scale + detail Preserve detail, reduce large scale Large kernel because detail=high+medium frequency computation challenge 64
Crossing lines The bilateral filter is influenced by pixels across thin line Good for tone mapping
What matters Spatial sigma: not very important Range sigma: quite important Use of the log domain for range: critical Because HDR and because perception sensitive to multiplicative contrast CIELab might be better for other applications Luminance computation Not critical, i but has influence see our Flash/no-flash paper [Eisemann 2004] for smarter r functionn 66
Tone mapping evaluation Recent user experiments to evaluate competing tone mapping Ledda et al. 2005 http://www.cs.bris.ac.uk/publications/ Papers/2000255.pdf Interestingly, the former concludes bilateral is the worst, the latter that it is the best! They choose to test a different criterion: fidelity vs. preference More importantly, they focus on algorithm and ignore parameters From Kuang et al. Adapted from Ledda et al. 67
Dirty vision for cool graphics Three wrongs make one right Analyze image Intrinsic image: albedo & illumination Simple bilateral filter Modify In our case, reduce contrast of large-scale (illumination) Recombine Get final image
Overview Denoising Tone mapping Relighting & texture editing 69
HDR hallucination [Lvdi Wang, Liyi Wei, Kun Zhou, Baining Guo, Heung-Yeung Shum, EGSR 2007] Low-dynamic-range images have under- and over-exposed exposed parts Information missing Underexposed Over- exposed 70
HDR hallucination Separate illumination and texture (Bilateral!) Fit smooth function to illumination Use texture t synthesis
HDR hallucination input output 72
HDR hallucination input output 73
Recap Decompose into Large scale (with bilateral l filter) Detail (residual: medium+high frequencies) Use big kernels Use appropriate domain (log for HDR) Manipulate/process independently Tone mapping Relighting, HDR hallucination HDR hallucination 74
Input Denoising with the BF output Images courtesy of Ce Liu 75
Noise level adaptation [Liu et al. 06] The noise of digital cameras varies with intensity Adapt sigma range to noise level Main topic of the paper: noise level inference Red Green Blue noise Intensity
Adaptive bilateral filter Adapt sigma range to intensity Input Bilateral Adaptive bilateral 77
Joint bilateral upsampling p (JBU) First publication on bilateral filters for upsampling at SIGGRAPH 2007 J. Kopf, Univ. of Konstantz (Germany) provided reference sw. [Kopf2007] solution: High resolution image in range term Low resolution input high resolution output high resolution result low resolution result (input) high resolution image (input)