Tone mapping. Digital Visual Effects, Spring 2009 Yung-Yu Chuang. with slides by Fredo Durand, and Alexei Efros

Transcription

1 Tone mapping Digital Visual Effects, Spring 2009 Yung-Yu Chuang 2009/3/5 with slides by Fredo Durand, and Alexei Efros

2 Tone mapping How should we map scene luminances (up to 1:100,000) 000) to display luminances (only around 1:100) to produce a satisfactory image? Real world radiance Display Linear scaling?, thresholding? h dynamic range intensity Pixel value 0 to 255 CRT has 300:1 dynamic range

3 The ultimate goal is a visual match visual adaption We do not need to reproduce the true radiance as long as it gives us a visual match.

4 Eye is not a photometer! Dynamic range along the visual pathway is only around 32:1. The key is adaptation

5 Eye is not a photometer! Are the headlights different in two images? Physically, they are the same, but perceptually different.

6 We are more sensitive to contrast Weber s law Just-noticeable Difference (JND) ΔI b I b ~ 1% background intensity flash

7 How humans deal with dynamic range We're more sensitive to contrast (multiplicative) A ratio of 1:2 is perceived as the same contrast as a ratio of 100 to 200 Makes sense because illumination has a multiplicative effect Use the log domain as much as possible Dynamic adaptation (very local in retina) Pupil (not so important) Neural Chemical Different sensitivity to spatial frequencies

8 Preliminaries For color images w d L R L = w d w d d d L G L L G R w d w d L B L L B Log domain is usually preferred. w L

9 HDR Display Once we have HDR images (either captured or synthesized), how can we display them on normal displays? diffuser LCD 300:1 Theoretically, 240,000:1. DLP 800:1 Due to imperfect optical depth, 54,000:1 measured HDR display system, Sunnybrook Technology, SIGGRAPH2004

10 Sunnybrook HDR display Slide from the 2005 Siggraph course on HDR

11 How it works Slide from the 2005 Siggraph course on HDR

12 Brightside HDR display :1 Acquired cqu ed by Dolby

13 Tone mapping operators Spatial (global/local) Frequency domain Gradient domain 3 papers from SIGGRAPH 2002 Photographic Tone Reproduction for Digital Images Fast Bilateral Filtering for the Display of High- Dynamic-Range Images Gradient Domain High Dynamic Range Compression

14 Photographic Tone Reproduction for Digital Images Erik Reinhard Mike Stark Peter Shirley Jim Ferwerda SIGGRAPH 2002

15 Photographic tone reproduction Proposed by Reinhard et. al. in SIGGRAPH 2002 Motivated by traditional i practice, zone system by Ansel Adams and dodging and burning It contains both global and local operators

16 Zone system

17 The Zone system Formalism to talk about exposure, density Zone = intensity range, in powers of two In the scene, on the negative, on the print Source: Ansel Adams

18 The Zones

19 The Zone system You decide to put part of the system in a given zone Decision: exposure, development, print

20 Dodging and burning During the print Hide part of the print during exposure Makes it brighter From The Master Printing Course, Rudman

21 Dodging and burning dodging burning From Photography by London et al.

22 Dodging and burning Must be done for every single print! Straight print After dodging and burning

23 Global operator 1 L = w exp log( δ + Lw( x, y)) N x, y User-specified; high key or low key L a ( x, y) Lw( x, y) L m = w Approximation of scene s key y( (how light or dark it is). Map to 18% of display range for average-key scene L d ( x, y) Lm ( x, y) ) = 1+ L ( x, y) m transfer function to compress high luminances

24 Global operator It seldom reaches 1 since the input image does not have infinitely large luminance values ), ( 1 ) ( y x L y x L m + not have infinitely large luminance values. ) ( 1 ), ( 1 ), ( ), ( 2 y x L y x L y x L y x L white m d + + = ), ( 1 y x + L m L hit is the smallest luminance L white is the smallest luminance to be mapped to 1

25 low key (0.18) high key (0.5)

26 Dodging and burning (local operators) Area receiving a different exposure is often bounded by sharp contrast Find largest surrounding area without any sharp contrast t L blur s V ( x, y ) = L ( x, y ) G ( x, y ) m blur blur Ls ( x, y ) Ls+ 1 ( x, y ) s ( x, y) = φ 2 blur 2 a s + Ls s s : V (x, (, y) max s max < ε

27 Dodging and burning (local operators) L d ( x, y ) = L 1+ L ( x, y) ( x, y) m blur s A darker pixel (smaller than the blurred average of its surrounding area) is divided by a larger number and become darker (dodging) A brighter pixel (larger than the blurred average of its surrounding area) is divided id d by a smaller number and become brighter (burning) Both increase the contrast max

28 Dodging and burning

29 Fast Bilateral Filtering for the Display of High-Dynamic-Range Images Frédo Durand & Julie Dorsey SIGGRAPH 2002

30 Frequency domain First proposed by Oppenheim in 1968! Under simplified assumptions, image = illuminance i * reflectance low-frequency attenuate more high-frequency attenuate t less

31 Oppenheim Taking the logarithm to form density image Perform FFT on the density image Apply frequency-dependent attenuation filter s ( f ) = (1 c ) + kf c 1+ kf Perform inverse FFT Take exponential to form the final image

32 A typical photo Sun is overexposed Foreground is underexposed d

33 Gamma compression X > X γ Colors are washed-out Input Gamma

34 Gamma compression on intensity Colors are OK, but details (intensity high- frequency) are blurred Intensity Gamma on intensity Color

35 Chiu et al Reduce contrast of low-frequencies Keep high h frequencies Low-freq. Reduce low frequency High-freq. Color

36 The halo nightmare For strong edges Because they contain high h frequency Low-freq. Reduce low frequency High-freq. Color

37 Durand and Dorsey Do not blur across edges Non-linear filteringi Large-scale Output Detail Color

38 Edge-preserving filtering Blur, but not across edges Input Gaussian blur Edge-preserving Anisotropic diffusion [Perona & Malik 90] p [ ] Blurring as heat flow LCIS [Tumblin & Turk] Bilateral filtering [Tomasi & Manduci, 98]

39 Start with Gaussian filtering Here, input is a step function + noise J = f I output input

40 Start with Gaussian filtering Spatial Gaussian f J = f I output input

41 Start with Gaussian filtering Output is blurred J = f I output input

42 Gaussian filter as weighted average Weight of ξ depends on distance to x J (x) = ξ f ( x, ξ ) I(ξ ) x ξ x output input

43 The problem of edges I(ξ ) Here, pollutes our estimate J(x) It is too different J (x) = ξ f ( x, ξ ) I(ξ ) I(x) ( ) x output input

44 Principle of Bilateral filtering [Tomasi and Manduchi 1998] Penalty g on the intensity difference 1 J (x) = f ( x, ξ ) g( I( ξ ) I( x)) I(ξ ) k ( x ) ξ x I(x) ( ) I (ξ ) output input

45 Bilateral filtering [Tomasi and Manduchi 1998] Spatial Gaussian f 1 J (x) = f ( x, ξ ) g( I( ξ ) I( x)) I(ξ ) k ( x ) ξ x output input

46 Bilateral filtering [Tomasi and Manduchi 1998] Spatial Gaussian f Gaussian g on the intensity difference 1 J (x) = f ( x, ξ ) g( I( ξ ) I( x)) I(ξ ) k ( x ) ξ x output input

47 Normalization factor [Tomasi and Manduchi 1998] k(x)= f ( x, ξ ) g ( I ( ξ ) I ( x )) ξ J (x) = 1 k ( x ) ξ f ( x, ξ ) g( I( ξ ) I( x)) I(ξ ) x output input

48 Bilateral filtering is non-linear [Tomasi and Manduchi 1998] The weights are different for each output pixel 1 J (x) = f ( x, ξ ) g( I( ξ ) I( x)) I(ξ ) k ( x ) ξ x x output input

49 Contrast reduction Input HDR image Contrast too high!

50 Contrast reduction Input HDR image Intensity Color

51 Contrast reduction Input HDR image Intensity Large scale Fast Bilateral Filter Color

52 Contrast reduction Input HDR image Intensity Large scale Fast Bilateral Filter Detail Color

53 Contrast reduction Input HDR image Scale in log domain Intensity Large scale Reduce contrast Large scale Fast Bilateral Filter Detail Color

54 Contrast reduction Input HDR image Intensity Large scale Reduce contrast Large scale Fast Bilateral Filter Detail Preserve! Detail Color

55 Contrast reduction Input HDR image Output Intensity Large scale Reduce contrast Large scale Fast Bilateral Filter Detail Preserve! Detail Color Color

56 Bilateral filter is slow! Compared to Gaussian filtering, it is much slower because the kernel is not fixed. Durand and Dorsey proposed an approximate approach to speed up Paris and Durand proposed an even-faster approach in ECCV We will cover this one when talking about computational photogrphy.

57 Oppenheim bilateral

58 GradientDomainHighDynamic Dynamic Range Compression Raanan Fattal Dani Lischinski Michael Werman SIGGRAPH 2002

59 Log domain Logorithm is a crude approximation to the perceived brightness Gradients in log domain correspond to ratios (local l contrast) t) in the luminance domain

60 The method in 1D log derivative attenuate exp integrate

61 The method in 2D Given: a log-luminance image H(x,y) Compute an attenuation map Φ( H ) Compute an attenuated gradient field G: G( x, y) = H ( x, y) Φ ( ) H Problem: G is not integrable!

62 Solution Look for image I with gradient closest to G in th l t the least squares sense. I minimizes the integral: ( ) dxdy G I F, g 2 2 ( ) y, ( ) 2 2 2, + = = y x G y I G x I G I G I F y x G G I I 2 2 Poisson y G x G y I x I y x + = Poisson equation

63 Solve G G I I y x + = Solve y x y x ), ( ), ( ) 1, ( ), ( + y x G y x G y x G y x G y y x x ), ( ), ( ), ( ), ( y y y y y y x x ), ( 4 1), ( 1), ( ) 1, ( ) 1, ( y x I y x I y x I y x I y x I = I

64 Solving Poisson equation No analytical solution Multigrid id method Conjugate gradient method

65 Attenuation Any dramatic change in luminance results in large luminance gradient at some scale Edges exist in multiple scales. Thus, we have to detect t and attenuate t them at multiple l scales Construct a Gaussian pyramid H i

66 Attenuation ϕ ( x, y ) k = H k ( x, α y) β 1 β ~ 0.8 α = 0. 1 H log(luminance) gradient magnitude attenuation map

67 Multiscale gradient attenuation interpolate X = interpolate X =

68 Final gradient attenuation map

69 Performance Measured on 1.8 GHz Pentium 4: x 384: 1.1 sec 1024 x 768: 4.5 sec Can be accelerated using processor-optimized optimized libraries.

70 Informal comparison Gradient domain [Fattal et al.] Bilateral [Durand et al.] Photographic [Reinhard et al.]

71 Informal comparison Gradient domain [Fattal et al.] Bilateral [Durand et al.] Photographic [Reinhard et al.]

72 Informal comparison Gradient domain [Fattal et al.] Bilateral [Durand et al.] Photographic [Reinhard et al.]

73 Evaluation of Tone Mapping Operators using a High Dynamic Range Display Patrick Ledda Alan Chalmers Tom Troscinko Helge Seetzen SIGGRAPH 2005

74 Six operators H: histogram adjustment B: bilateral l filter P: photographic reproduction I: icam L: logarithm mapping A: local eye adaption

75 23 scenes

76 Experiment setting tonemapping result HDR display tonemapping result

77 Preference matrix Ranking is easier than rating. 15 pairs for each person to compare. A total of 345 pairs per subject. preference matrix (tmo2->tmo4 >tmo4, tom2 is better than tmo4)

78 Statistical measurements Statistical measurements are used to evaluate: Agreement: whether most agree on the ranking between two tone mapping operators. Consistency: no cycle in ranking. If all are confused in ranking some pairs, it means they are hard to compare. If someone is inconsistent alone, his ranking could be droped.

79 Overall similarity Scene 8

80 Summary

81 Not settled yet! Some other experiment said bilateral are better than others. For your reference, photographic reproduction performs well in both reports. There are parameters to tune and the space could be huge.

82 References Raanan Fattal, Dani Lischinski, Michael Werman, Gradient Domain High Dynamic Range Compression, SIGGRAPH Fredo Durand, Julie Dorsey, Fast Bilateral Filtering for the Display of High Dynamic Range Images, SIGGRAPH Erik Reinhard, Michael Stark, Peter Shirley, Jim Ferwerda, Photographics Tone Reproduction for Digital Images, SIGGRAPH Patrick Ledda, Alan Chalmers, Tom Troscianko, Helge Seetzen, Evaluation of Tone Mapping Operators using a High Dynamic Range Display, SIGGRAPH Jiangtao Kuang, Hiroshi Yamaguchi, Changmeng Liu, Garrett Johnson, Mark Fairchild, Evaluating HDR Rendering Algorithms, ACM Transactions on Applied Perception, 2007.