Flash Photography Enhancement via Intrinsic Relighting

Size: px

Start display at page:

Download "Flash Photography Enhancement via Intrinsic Relighting"

Bonnie Singleton
6 years ago
Views:

1 Flash Photography Enhancement via Intrinsic Relighting Elmar Eisemann MIT/Artis-INRIA Frédo Durand MIT

2 Introduction Satisfactory photos in dark environments are challenging!

3 Introduction Available light: + nice lighting - noise/blurriness - color No-flash

4 Introduction Flash: + details + color - flat/artificial - flash shadows - red eyes Flash

5 Introduction Our approach: Use no-flash image relight flash image No-flash Flash Result

6 Introduction Our approach: Use no-flash image relight flash image No-flash + original lighting + details/sharpness + color Result

7 Introduction One approach: Blend the two photos No-flash + Flash Blending

8 Introduction One approach: Blend the two photos Blending Our result

9 Introduction One approach: Blend the two photos Blending Our solution: more details and less noise Our result

10 Overview Related Work Our Approach Results Conclusion and Future Work

11 Overview Related Work Our Approach Results Conclusion and Future Work

12 Related Work Relighting: Decouple luminance and texture Barrow and Tennenbaum [1978] Weiss [2001] using image sequences Tappen et al. [2003] from a single image Oh et al. [2001] Input Image Luminance Texture Images courtesy of Oh et al.

13 Related Work Tone Mapping of High Dynamic Range Images Decouple detail / large-scale information Tumblin et al. [1999] Durand et al. [2002] Choudhury et al. [2003] + Input Detail Large-Scale Output

14 Related Work Jia et al. [2004]: Solve problem of blur in long exposures Image pair with different exposure times Short exposure Color manipulation Result Long exposure Details from short exposure Images courtesy of Jia et al.

15 Related Work Petschnigg et al.[2004]: many similarities previous talk discussion at the end

16 Overview Related Work Our Approach Results Conclusion and Future Work

17 Our Approach - Main Idea

18 Our Approach

19 Our Approach

20 Our Approach Registration Registration

21 Registration Compensate for camera motion (image translation) Difficult because lighting changes Edge detection No-flash See also Ward[2004], Kang[2003] Flash

22 Our Approach Registration Registration

23 Our Approach Our Approach Decomposition

24 Decomposition Color / Intensity: = * original intensity color

25 Our Approach Our Approach Decomposition

26 Our Approach Decoupling

27 Decoupling Lighting : Large-scale variation Texture : Small-scale variation Lighting Texture

28 Large-scale Layer Bilateral filter edge preserving filter Smith and Brady 1997; Tomasi and Manducci 1998; Durand et al Input Output

29 Large-scale Layer Bilateral filter

30 Cross Bilateral Filter Similar to joint bilateral filter by Petschnigg et al. When no-flash image is too noisy Borrow similarity from flash image edge stopping from flash image See detail in paper Bilateral Cross Bilateral

31 Detail Layer / = Intensity Large-scale Detail Recombination: Large scale * Detail = Intensity

32 Recombination * = Large-scale No-flash Detail Flash Intensity Result Recombination: Large scale * Detail = Intensity

33 Recombination shadows ~ * ~ Intensity Result Color Flash Result Recombination: Intensity * Color = Original

34 Our Approach

35 Our Approach Shadow Detection/Treatment

36 Shadow Correction Why? No flash Flash Global Straightforward white balance recombination shadows

37 Shadow Correction Why? Global white balance in shadows Several artifacts: at shadow boundary inside shadows (color bleeding) shadows need to be corrected!... and detected

38 Shadow Detection Flash Umbra Penumbra Detection in two steps

39 Shadow Detection Umbra detection No direct light from flash Difference of the two photos Ι reveals these regions - = Flash No-flash Ι

40 Shadow Detection Umbra detection Difference Ι = light from the flash? Goal: Find a threshold for Ι Automatic Threshold Detection (see paper) Ι

41 Shadow Detection Penumbra detection strong gradient at boundary no strong gradient in no-flash image connected to umbra No-flash flash Umbra Penumbra

42 Shadow Correction Correct color and detail Shadow mask

43 Shadow Color Correction Correct color Wrong color Flash color Fill in shadow from similar surrounding No-flash colors

44 Shadow Color Correction No-flash Flash

45 Shadow Color Correction No-flash Flash

46 Shadow Color Correction No-flash Flash Outside shadow

47 Shadow Color Correction No-flash Flash Inside shadow Outside shadow

48 Shadow Color Correction No-flash Flash Inside shadow Outside shadow Select pixel in shadow

49 Shadow Color Correction No-flash Flash Corresponding pixel Inside shadow Outside shadow

50 Shadow Color Correction No-flash Flash Spatial weights Inside shadow Outside shadow

51 Shadow Color Correction No-flash Flash Spatial and Color weights Inside shadow Outside shadow

52 Shadow Color Correction No-flash Flash Spatial and Color weights Inside shadow Outside shadow

53 Shadow Color Correction No-flash Flash Use shadow mask Inside shadow Outside shadow

54 Shadow Color Correction No-flash Flash Use shadow mask Inside shadow Outside shadow

55 Shadow Color Correction No-flash Flash Use shadow mask Inside shadow Outside shadow

56 Shadow Color Correction No-flash Flash Use shadow mask Inside shadow Outside shadow

57 Shadow Color Correction No-flash Flash Use shadow mask Inside shadow Outside shadow

58 Shadow Color Correction No-flash Flash Use shadow mask Inside shadow Outside shadow

59 Shadow Color Correction No-flash Flash Use weights on flash color Inside shadow Outside shadow

60 Shadow Color Correction No-flash Flash Replace shadow pixel Inside shadow Outside shadow

61 Shadow Color Correction No-flash Flash Inside shadow Proceed for all shadow pixels Outside shadow

62 Our Approach

63 Our Approach

64 Overview Related Work Our Approach Results Conclusion and Future Work

65 Results No-flash Flash

66 Results No correction Our result

67 Results No-flash Flash

68 Results No-flash Flash

69 Results No-flash Flash Our result

70 Results Our result

71 Results Our result

72 Emphasize Foreground Our result Deduce distance to camera Exploit 1/r 2 flash intensity falloff (see paper) Emphasized foreground

73 (Inverse) White Balance No-flash Flash Retain warm tones from available lighting (see paper)

74 Results No-flash Flash Result

75 Overview Related Work Our Approach Results Conclusion and Future Work

76 Conclusion Improving photography in dim environments Capture original lighting Add sharpness/details Cross bilateral filter Correct flash shadows No-flash Result Pseudo distance (emphasize foreground) (Inverse) white balance

77 Future Work Local coherence for shadow detection Different recombinations Bilateral filter parameters Infra-red flash

78 Aknowledgements École Normale Supérieure - Paris Deshpande Center MIT-France NSF CISE

79 Thank you very much for your attention!

80 Comparison: Similarities Petschnigg et al. Eisemann et al. Detail transfer or relighting Use bilateral filter Joint or cross bilateral filter

81 Comparison: Color Petschnigg et al. Eisemann et al. Color from no-flash More noisy Less need for shadow correction White balance no-flash image Color from flash Less noisy Need correction in shadow Inverse white balance flash image

82 Comparison: Differences Petschnigg et al. Eisemann et al. Color from no-flash Semi-manual shadows/highlights Continuous flash Red-eye removal Algorithmic differences Additional tools Color from flash Automatic shadows Pseudo distance, emphasize foreground

83 Thank you again for your attention!

Computational Illumination Frédo Durand MIT - EECS

Computational Illumination Frédo Durand MIT - EECS Some Slides from Ramesh Raskar (MIT Medialab) High level idea Control the illumination to Lighting as a post-process Extract more information Flash/no-flash