Fixing the Gaussian Blur : the Bilateral Filter

Transcription

1 Fixing the Gaussian Blur : the Bilateral Filter Lecturer: Jianbing Shen shenjianbing@bit.edu.cnedu Office room : cn/shenjianbing Note: contents copied from siggraph course notes <A Gentle Introduction to Bilateral Filtering and its Applications >

2 Blur Comes from Averaging g across Edges input * output * * Same Gaussian kernel everywhere. 2

3 Bilateral Filter [Aurich 95, Smith 97, Tomasi 98] No Averaging g across Edges input * output * * The kernel shape depends d on the image content. t 3

4 Bilateral Filter Definition: an Additional Edge Term Same idea: weighted average of pixels. BF new 1 W not new new [ I ] G G I I p s p q p q [ r p q S I q normalization factor space weight range weight I 4

5 Illustration a 1D Image 1D image = line of pixels Better visualized as a plot pixel intensity pixel position 5

6 Gaussian Blur and Bilateral Filter Gaussian blur p q GB I] G p q space Bilateral filter [Aurich 95, Smith 97, Tomasi 98] [ p q S space I q q p space ra ange 1 BF [ I ] p G W s r p q p q S space range normalization p q G I I I q

7 Bilateral Filter on a Height Field BF 1 [ I ] p G W s r p q G I I p q q Wp q S I output input reproduced from [Durand 02]

8 Space and Range Parameters 1 BF [ r W [ I ] G G I I p s p q p q p q S I q space s : spatial extent of the kernel, size of the considered neighborhood. range r : minimum amplitude of an edge 8

9 Influence of Pixels Only pixels close in space and in range are considered. space range p

10 Exploring the Parameter Space r = 0.1 r = 0.25 r = r (Gaussian blur) input s = 2 s = 6 s = 18 10

11 Varying the Range Parameter r = 0.1 r = 0.25 r = r (Gaussian blur) input s = 2 s = 6 s = 18

12 input 12

13 r =

14 r =

15 r = (Gaussian blur) 15

16 Varying the Space Parameter r = 0.1 r = 0.25 r = r (Gaussian blur) input s = 2 s = 6 s = 18

17 input 17

18 s = 2 18

19 s = 6 19

20 s = 18 20

21 How to Set the Parameters Depends on the application. For instance: space parameter: proportional to image size eg e.g., 2% of image diagonal range parameter: proportional p to edge amplitude e.g., mean or median of image gradients independent of resolution and exposure 21

22 A Few More Advanced Remarks 22

23 Bilateral Filter Crosses Thin Lines Bilateral filter averages across features thinner than ~2 s Desirable for smoothing: more pixels = more robust Different from diffusion that stops at thin lines close-up kernel

24 Iterating the Bilateral Filter I BF [ I ] [ ( n 1) ( n ) Generate more piecewise-flat images Often not needed in computational photo. 24

25 input 25

26 1 iteration 26

27 2 iterations 27

28 4 iterations 28

29 Bilateral Filtering Color Images For gray-level images 1 BF [ I ] p G r W s p q S S intensity difference p q G I I p q I q scalar input For color images 1 BF [ I ] p G r W s Wp q S S color difference p q G C C p q C q 3D vector (RGB, Lab) output The bilateral filter is extremely easy to adapt to your need.

30 Hard to Compute 1 Nonlinear BF [ I ] p G s p q G I p I q r Complex, spatially varying kernels W p q S S Cannot be precomputed, no FFT I q Brute-force implementation is slow > 10min 30

31 Questions? 31

32 Applications

33 Overview Denoising Tone mapping Relighting & texture editing 33

34 Overview Not most powerful application Not best denoising, but good & simple Denoising Tone mapping Relighting & texture editing

35 Basic denoising Noisy input Bilateral filter 7x7 window

36 Basic denoising Bilateral filter Median 3x3 36

37 Basic denoising Bilateral filter Median 5x5 37

38 Basic denoising Bilateral filter Bilateral filter lower sigma 38

39 Basic denoising Bilateral filter Bilateral filter higher sigma 39

40 Denoising Small spatial sigma (e.g. 7x7 window) Adapt range sigma to noise level l Maybe not best denoising method, but best simplicity/quality tradeoff No need for acceleration (small kernel) But the denoising feature in e.g. Photoshop is better

41 Overview Denoising Tone mapping Relighting & texture editing 41

42 Real world dynamic range Eye can adapt from ~ 10-6 to 10 6 cd/m 2 Often 1 : 10,000 in a scene Real world High dynamic range 42

43 Picture dynamic range Typically 1: 20 or 1:50 Real world Black is ~ 50x darker than white Picture Low contrast 43

44 Multiple exposure photography Real world 10-6 High dynamic range 10 6 Merge HDR Merge multiple l exposure to cover full range We obtain one single image with floats per pixel But we still can t display it

45 The future: HDR Cameras HDR sensors using CMOS Use a log response curve e.g. SMaL, Assorted pixels Fuji Nayar et al. Per-pixel exposure Filter Integration time Fuji SuperCCD Multiple cameras using beam splitters Other computational photography tricks

46 Problem: Contrast reduction Match limited contrast of the medium Preserve details 10-6 High dynamic range 10 6 Real world Picture Low contrast 46

47 Tone mapping Input: high-dynamic-range image (floating point per pixel)

48 Naïve technique Scene has 1:10,000 contrast, display has 1:100 Simplest contrast reduction?

49 Naïve: Gamma compression X X (where =0.5 in our case) But colors are washed-out. Why? Input Gamma 49

50 Gamma compression on intensity Colors are OK, but details (intensity high-frequency) are blurred Gamma on intensity Intensity Color

51 Oppenheim 1968, Chiu et al Reduce contrast of low-frequencies (log domain) Keep high frequencies Low-freq freq. Reduce low frequency High-freq. Color

52 The halo nightmare For strong edges Because they contain high frequency Low-freq freq. Reduce low frequency High-freq. Color

53 Bilateral filtering to the rescue Large scale = bilateral (log intensity) Detail = residual [Durand & Dorsey 2002] Large-scale Output Detail Color

54 Contrast reduction Input HDR image Contrast too high! 54

55 Input HDR image Contrast reduction Intensity Color 55

56 Input HDR image Contrast reduction Intensity Large scale Color Bilateral Filter (in log domain!) Spatial sigma: 2% image size Range sigma: 0.4 (in log 10) 56

57 Input HDR image Contrast reduction Intensity Large scale Bilateral Filter Detail Color Detail = log intensity large scale (residual) 57

58 Input HDR image Contrast reduction Intensity Large scale Reduce contrast Large scale Bilateral Filter Detail Color 58

59 Input HDR image Contrast reduction Intensity Large scale Reduce contrast Large scale Bilateral Filter Detail Preserve! Detail Color 59

60 Input HDR image Contrast Output reduction Intensity Large scale Reduce contrast Large scale Bilateral Filter Detail Preserve! Detail Color Color 60

61 Contrast reduction in log domain Set target large-scale contrast (e.g. log 10 10) In linear output, we want 1:10 contrast for large scale Compute range of input large scale layer: largerange = max(inloglarge) min (inloglarge) Scale factor k= log 10 (10) / largerange Normalize so that the biggest value is 0 in log outlog= inlogdetail + inloglarge * k max(inloglarge)

62 Alternative explanation Explanation 1 (previous slides): outlog =k inloglarge + inlogdetail (ignoring offset) Explanation 2 outlog = k inlogintensity + (1-k) detail Reduce contrast of full intensity layer Add back some detail Same final effect since inlogdetail+inloglarge scale = inlogintensity But different philosophy: decomposition vs. add back detail

63 Demo Don t try at home without FAST bilateral filtering 63

64 Denoising vs. tone mapping Denoising: decompose into noise+signal si n Throw away noise, keep signal Small kernel Tone mapping Decompose into large scale + detail Preserve detail, reduce large scale Large kernel because detail=high+medium frequency computation challenge 64

65 Crossing lines The bilateral filter is influenced by pixels across thin line Good for tone mapping

66 What matters Spatial sigma: not very important Range sigma: quite important Use of the log domain for range: critical Because HDR and because perception sensitive to multiplicative contrast CIELab might be better for other applications Luminance computation Not critical, i but has influence see our Flash/no-flash paper [Eisemann 2004] for smarter r functionn 66

67 Tone mapping evaluation Recent user experiments to evaluate competing tone mapping Ledda et al Papers/ pdf Interestingly, the former concludes bilateral is the worst, the latter that it is the best! They choose to test a different criterion: fidelity vs. preference More importantly, they focus on algorithm and ignore parameters From Kuang et al. Adapted from Ledda et al. 67

68 Dirty vision for cool graphics Three wrongs make one right Analyze image Intrinsic image: albedo & illumination Simple bilateral filter Modify In our case, reduce contrast of large-scale (illumination) Recombine Get final image

69 Overview Denoising Tone mapping Relighting & texture editing 69

70 HDR hallucination [Lvdi Wang, Liyi Wei, Kun Zhou, Baining Guo, Heung-Yeung Shum, EGSR 2007] Low-dynamic-range images have under- and over-exposed exposed parts Information missing Underexposed Over- exposed 70

71 HDR hallucination Separate illumination and texture (Bilateral!) Fit smooth function to illumination Use texture t synthesis

72 HDR hallucination input output 72

73 HDR hallucination input output 73

74 Recap Decompose into Large scale (with bilateral l filter) Detail (residual: medium+high frequencies) Use big kernels Use appropriate domain (log for HDR) Manipulate/process independently Tone mapping Relighting, HDR hallucination HDR hallucination 74

75 Input Denoising with the BF output Images courtesy of Ce Liu 75

76 Noise level adaptation [Liu et al. 06] The noise of digital cameras varies with intensity Adapt sigma range to noise level Main topic of the paper: noise level inference Red Green Blue noise Intensity

77 Adaptive bilateral filter Adapt sigma range to intensity Input Bilateral Adaptive bilateral 77

78 Joint bilateral upsampling p (JBU) First publication on bilateral filters for upsampling at SIGGRAPH 2007 J. Kopf, Univ. of Konstantz (Germany) provided reference sw. [Kopf2007] solution: High resolution image in range term Low resolution input high resolution output high resolution result low resolution result (input) high resolution image (input)