Prof. Trevor Darrell Lecture 23: Segmentation II & Computational Photography Teaser

Transcription

1 C280, Computer Vision Prof. Trevor Darrell Lecture 23: Segmentation II & Computational Photography Teaser

2 Two presentations today:

3 Contours and Junctions in Natural Images Jitendra Malik University of California at Berkeley (with Jianbo Shi, Thomas Leung, Serge Belongie, Charless Fowlkes, David Martin, Xiaofeng Ren, Michael Maire, Pablo Arbelaez) 3

4 From Pixels to Perception Water Tiger Grass outdoor wildlife Sand back Tiger head eye tail legs mouth shadow 4

5 I stand at the window and see a house, trees, sky. Theoretically I might say there were 327 brightnesses and nuances of colour. Do I have "327"? No. I have sky, house, and trees Max Wertheimer,

6 Perceptual Organization Grouping Figure/Ground 6

7 Key Research Questions in Perceptual Predictive power Organization Factors for complex, natural stimuli? How do they interact? Functional significance Why should these be useful or confer some evolutionary advantage to a visual organism? Brain mechanisms How are these factors implemented given what we know about V1 and higher visual areas? 7

8 Attneave s Cat (1954) Line drawings convey most of the information 8

9 Contours and junctions are fundamental Key to recognition, inference of 3D scene properties, visually- guided manipulation and locomotion This goes beyond local, V1-like, edge-detection. Contours are the result of perceptual organization, grouping and figure/ground processing 9

10 Some computer vision history Local Edge Detection was much studied in the 1970s and early 80s (Sobel, Rosenfeld, Binford- Horn, Marr-Hildreth, Canny ) Edge linking exploiting curvilinear continuity was studied as well (Rosenfeld, Zucker, Horn, Ullman ) In the 1980s, several authors argued for perceptual organization as a precursor to recognition (Binford, Witkin and Tennebaum, Lowe, Jacobs ) 10

11 However in the 90s 1. We realized that there was more to images than edges Biologically inspired filtering approaches (Bergen & Adelson, Malik & Perona..) Pixel based representations for recognition (Turk & Pentland, Murase & Nayar, LeCun ) 2. We lost faith in the ability of bottom-up vision Do minimal bottom up processing, e.g. tiled orientation histograms don t even assume that linked contours or junctions can be extracted Matching with memory of previously seen objects then becomes the primary engine for parsing an image.? 11

12 At Berkeley, we took a contrary view 1. Collect Data Set of Human segmented images 2. Learn Local Boundary Model for combining brightness, color and texture 3. Global framework to capture closure, continuity 4. Detect and localize junctions 5. Integrate low, mid and high-level information for grouping and figure-ground segmentation 12

13 Berkeley Segmentation DataSet [BSDS] D. Martin, C. Fowlkes, D. Tal, J. Malik. "A Database of Human Segmented Natural Images and its Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics", ICCV,

14 14

15 Contour detection ~

16 Contour detection ~

17 Contour detection ~

18 Contour detection ~2008 (gray) 18

19 Contour detection ~2008 (color) 19

20 Outline 1. Collect Data Set of Human segmented images 2. Learn Local Boundary Model for combining brightness, color and texture 3. Global framework to capture closure, continuity 4. Detect and localize junctions 5. Integrate low, mid and high-level information for grouping and figure-ground segmentation 20

21 Contours can be defined by any of a number of cues (P. Cavanagh) 21

22 Cue-Invariant Representations Gray level photographs Objects from motion Objects from luminance Objects from disparity Line drawings Objects from texture Grill-Spector et al., Neuron

23 Image Martin, Fowlkes, Malik PAMI 04 Boundary Cues Cue Combination Brightness P b Color Model Texture Challenges: texture cue, cue combination Goal: learn the posterior probability bili of a boundary P b (x,y, ) from local information only 23

24 Individual Features 1976 CIE L*a*b* colorspace Brightness Gradient BG(x,y,r, ) Difference of L* distributions Color Gradient CG(x,y,r, ) Difference of a*b* distributions (x,y) r Texture Gradient TG(x,y,r, ) Difference of distributions of V1-like filter responses These are combined using logistic regression 24

25 Various Cue Combinations 25

26 Outline 1. Collect Data Set of Human segmented images 2. Learn Local Boundary Model for combining brightness, color and texture 3. Global framework to capture closure, continuity 4. Detect and localize junctions 5. Integrate low, mid and high-level information for grouping and figure-ground segmentation 26

27 Exploiting global constraints: Image Segmentation as Graph Partitioning Build a weighted graph G=(V,E) from image V: image pixels E: connections between pairs of nearby pixels Partition graph so that similarity within group is large and similarity between groups is small -- Normalized Cuts [Shi & Malik 97] 27

28 Wij small when intervening contour strong, small when weak.. Cij = max Pb(x,y) for (x,y) on line segment ij; Wij = exp ( - Cij / 28

29 Eigenvectors carry contour information 29

30 We do not try to find regions from the eigenvectors, so we avoid the broken sky artifacts of Ncuts 30

31 Key idea compute edges on ncut eigenvectors, sum over first k: where is the output of a Gaussian derivative on the j-th eigenvector of 31

32 The Benefits of Globalization Maire, Arbelaez, Fowlkes, Malik, CVPR 08 32

33 Comparison to other approaches 33

34 34

35 Outline 1. Collect Data Set of Human segmented images 2. Learn Local Boundary Model for combining brightness, color and texture 3. Global framework to capture closure, continuity 4. Detect and localize junctions 5. Integrate low, mid and high-level information for grouping and figure-ground segmentation 35

36 Detecting Junctions 36

37 37

38 Benchmarking corner detection 38

39 39

40 Better object recognition using previous version of Pb Ferrari, Fevrier, Jurie and Schmid (PAMI 08) Shotton, Blake and Cipolla (PAMI 08) 40

41 Outline 1. Collect Data Set of Human segmented images 2. Learn Local Boundary Model for combining brightness, color and texture 3. Global l framework to capture closure, continuity it 4. Detect and localize junctions 5. Integrate low, mid and high-level cues for grouping and figure-ground segmentation 1. Ren, Fowlkes, Malik, IJCV Fowlkes, Martin, Malik, JOV Ren, Fowlkes, Malik, ECCV 06 41

42 Power laws for contour lengths 42

43 Convexity [Metzger 1953, Kanizsa and Gerbino 1976] Conv G = percentage of straight lines that lie completely within region G G p F Convexity(p) = log(conv F / Conv G ) 43

44 Figural regions tend to be convex 44

45 Lower Region [Vecera, Vogel & Woodman 2002] θ p center of mass LowerRegion(p) = θ G 45

46 Figural regions tend to lie below ground regions 46

47 Ren, Fowlkes, Malik ECCV 06 Object and Scene Recognition Grouping / Segmentation Figure/Ground Organization Human subjects label groundtruth figure/ground assignments in natural images. Shapemes encode high-level knowledge in a generic way, capturing local figure/ground cues. A conditional random field incorporates junction cues and enforces global consistency. 47

48 Forty yyears of contour detection Roberts Sobel Prewitt Marr Canny Perona Martin Maire (1965) (1968) (1970) Hildreth (1986) Malik (1980) (1990) Fowlkes Malik (2004) Arbelaez Fowlkes Malik (2008) 48

49 Forty yyears of contour detection Roberts Sobel Prewitt Marr Canny Perona Martin Maire (1965) (1968) (1970) Hildreth (1986) Malik (1980) (1990) Fowlkes Malik (2004) Arbelaez Fowlkes Malik (2008)??? (2013) 49

50 Curvilinear Grouping Boundaries are smooth in nature! A number of associated visual phenomena Good continuation Visual completion Illusory contours 50

51 51

52 Computational Photography Computer Vision CSE 576, Spring 2008 Richard Szeliski Microsoft Research

53 Computational ti Photography h photometric camera calibration high-dynamic range imaging & tone mapping flash photography h Richard Szeliski Computational Photography 53

54 Readings Debevec and Malik, Recovering High Dynamic Range Radiance Maps from Photographs. In SIGGRAPH 97. S. B. Kang et al. High dynamic range video. SIGGRAPH D. Lischinski. Interactive local adjustment of tonal values. SIGGRAPH G. Petschnigg et al. Digital photography with flash and no-flash image pairs. SIGGRAPH P. Pérez et al. Poisson image editing. SIGGRAPH 2003 Richard Szeliski Computational Photography 54

55 Sources Some of my slides are from: Bill Freeman and Frédo Durand Richard Szeliski Computational Photography 55

56 Sources Some of my slides are from: Alexei (Alyosha) Efros cmu Richard Szeliski Computational Photography 56

57 But first, for something (a little) different

58 Panography - Richard Szeliski Computational Photography 58

59 Panography - Richard Szeliski Computational Photography 59

60 Panography What kind of motion model? What kind of compositing? Can you do global alignment? Richard Szeliski Computational Photography 60

61 High Dynamic Range Imaging (HDR) slides borrowed from : Computational Photography Alexei Efros, CMU, Fall 2007, Paul Debevec, and my talks

62 Problem: Dynamic Range The real world is high dynamic range. 25, ,000 2,000,000,000 Richard Szeliski Computational Photography 62

63 Problem: Dynamic Range Typical cameras have limited dynamic range What can we do? Solution: merge multiple exposures Richard Szeliski Computational Photography 63

64 Varying Exposure Richard Szeliski Computational Photography 64

65 HDR images multiple l inputs Pixel count Radiance Richard Szeliski Computational Photography 65

66 HDR images merged Pixel count Radiance Richard Szeliski Computational Photography 66

67 Camera is not a photometer! t Limited dynamic range Use multiple exposures? Unknown, nonlinear response Not possible to convert pixel values to radiance Solution: Recover response curve from multiple exposures, then reconstruct the radiance map Richard Szeliski Computational Photography 67

68 Imaging system response function 255 Pixel value 0 log Exposure = log (Radiance * t) (CCD photon count)

69 Camera Calibration Geometric How pixel coordinates relate to directions in the world Photometric How pixel values relate to radiance amounts in the world Per-pixel transfer and blur Richard Szeliski Computational Photography 69

70 Camera sensing pipeline Camera Irradiance Optics Aperture Shutter Camera Body Sensor (CCD/CMOS) Gain (ISO) A / D RAW Sensor chip Demosaic (Sharpen) White Balance Gamma/curve Compress JPEG DSP Richard Szeliski Computational Photography 70

71 Camera sensing pipeline Camera Irradiance Optics Aperture Shutter Blur kern. & RD F-stop Camera & Body Vignette Exposure T Sensor (CCD/CMOS) Gain (ISO) A / D RAW AA CFA Noise ISO Sensor Gainchip Q1 Demosaic? (Sharpen)? White Balance Gamma/curve Compress JPEG RGB Gain DSP Q2 Richard Szeliski Computational Photography 71

72 Recovering High Dynamic Range Radiance Maps from Photographs Paul Debevec Jitendra Malik Computer Science Division University of California at Berkeley SIGGRAPH 97, August 1997

73 Ways to vary exposure Shutter Speed (*) F/stop (aperture, iris) Neutral Density (ND) Filters Richard Szeliski Computational Photography 73

74 Shutter Speed Ranges: Canon D30: 30 to 1/4,000 sec. (1997) Sony VX2000: ¼ to 1/10,000 sec. Pros: Directly varies the exposure Usually accurate and repeatable Issues: Noise in long exposures Richard Szeliski Computational Photography 74

75 Shutter Speed Note: shutter times usually obey a power series each stop is a factor of 2 ¼, 1/8, 1/15, 1/30, 1/60, 1/125, 1/250, 1/500, 1/1000 sec Usually really is: ¼, 1/8, 1/16, 1/32, 1/64, 1/128, 1/256, 1/512, 1/1024 sec Richard Szeliski Computational Photography 75

76 The Algorithm t t = t t = t t = t t = t t = 1/64 sec 1/16 sec 1/4 sec 1 sec 4 sec Pixel Value Z = f(exposure) Exposure = Radiance t log Exposure = log Radiance log t Richard Szeliski Computational Photography 76

77 Response Curve Assuming unit radiance for each pixel After adjusting radiances to obtain a smooth response curve ixel valu ue P Pixel valu ue log Exposure log Exposure Richard Szeliski Computational Photography 77

78 The Math Let g(z) be the discrete inverse response function For each pixel site i in each image j, want: ln Radiance i ln t j g(z ij ) Solve the over-determined d linear system: N P Z max ln Radiance 2 2 i ln t j g(z ij ) g (z) i 1 j 1 z Z min fitting i term smoothness term Richard Szeliski Computational Photography 78

79 MatLab code function [g,le]=gsolve(z,b,l,w) n = 256; A = zeros(size(z,1)*size(z,2)+n+1,n+size(z,1)); b = zeros(size(a,1),1); k = 1; %% Include the data-fitting equations for i=1:size(z,1) for j=1:size(z,2) wij = w(z(i,j)+1); A(k,Z(i,j)+1) = wij; A(k,n+i) = -wij; b(k,1) = wij * B(i,j); k=k+1; end end A(k,129) = 1; %% Fix the curve by setting its middle value to 0 k=k+1; k+1 for i=1:n-2 %% Include the smoothness equations A(k,i)=l*w(i+1); A(k,i+1)=-2*l*w(i+1); A(k,i+2)=l*w(i+1); k=k+1; end x = A\b; %% Solve the system using SVD Richard Szeliski Computational Photography 79 g = x(1:n); le = x(n+1:size(x,1));

80 Results: digital it camera Kodak DCS460 1/30 to 30 sec Recovered response curve Pi ixel val lue Richard Szeliski Computational Photography log Exposure 80

81 Reconstructed t Radiance Map Richard Szeliski Computational Photography 81

82 Results: Color Film Kodak Gold ASA 100, PhotoCD Richard Szeliski Computational Photography 82

83 Recovered Response Curves Red Green Blue RGB Richard Szeliski Computational Photography 83

84 The Radiance Map Richard Szeliski Computational Photography 84

85 The Radiance Map Linearly scaled to display device Richard Szeliski Computational Photography 85

86 Portable FloatMap (.pfm) 12 bytes per pixel, 4 for each channel sign exponent mantissa Text header similar to Jeff Poskanzer s.ppm image format: Floating Point TIFF similar PF <binary image data> Richard Szeliski Computational Photography 86

87 Radiance Format (.pic,.hdr) 32 bits / pixel Red Green Blue Exponent (145, 215, 87, 149) = (145, 215, 87, 103) = (145, 215, 87) * 2^( ) = (145, 215, 87) * 2^( ) = ( , , ) ( , , ) Ward, Greg. "Real Pixels," in Graphics Gems IV, edited by James Arvo, Academic Press, 1994 Richard Szeliski Computational Photography 87

88 ILM s OpenEXR (.exr) 6 bytes per pixel, 2 for each channel, compressed sign exponent mantissa Several lossless compression options, 2:1 typical Compatible with the half datatype in NVidia's Cg Supported natively on GeForce FX and Quadro FX Available at Richard Szeliski Computational Photography 88

89 High Dynamic Range Video Sing Bing Kang, Matt Uyttendaele, Simon Winder, Rick Szeliski [SIGGRAPH 2003]

90 HDR images merged Pixel count Radiance Richard Szeliski Computational Photography 90

91 What about scene motion? Inputs Tonemapped output (no compensation or consistency ste cy check) Richard Szeliski Computational Photography 91

92 With motion compensation Inputs Tonemapped output (global+local compensation) Richard Szeliski Computational Photography 92

93 Registration ti (global) l) After global registration Richard Szeliski Computational Photography 93

94 Registration ti (local) l) After local registration Richard Szeliski Computational Photography 94

95 Now What? Richard Szeliski Computational Photography 95

96 Tone Mapping

97 Tone Mapping How can we do this? Real World Ray Traced World (Radiance) Linear scaling?, thresholding? Suggestions? 10-6 High dynamic range 10 6 Display/ Printer to 255 Richard Szeliski Computational Photography 97

98 Simple Global l Operator Compression curve needs to Bring everything within range Leave dark areas alone In other words Asymptote at 255 Derivative of 1 at 0 Richard Szeliski Computational Photography 98

99 Global l Operator (Reinhart et al) L display L 1 L world world Richard Szeliski Computational Photography 99

100 Global l Operator Results Richard Szeliski Computational Photography 100

101 Reinhart Operator Darkest 0.1% scaled to display device Richard Szeliski Computational Photography 101

102 What do we see? Vs. Richard Szeliski Computational Photography 102

103 What does the eye sees? The eye has a huge dynamic range Do we see a true radiance map? Richard Szeliski Computational Photography 103

104 Metamores Can we use this for range compression? Richard Szeliski Computational Photography 104

105 Fast bilateral filtering for the display of high-dynamic-range images Frédo Durand and Julie Dorsey SIGGRAPH 2002.

106 Input Naïve: Gamma compression X X colors are washed-out. Why? Gamma Richard Szeliski Computational Photography 106

107 Gamma compression on intensity it Colors are OK, details are blurred Intensity Gamma on intensity Color Richard Szeliski Computational Photography 107

108 Oppenheim 1968, Chiu et al Reduce contrast of low-frequencies, keep high Low-freq. Reduce low frequency High-freq. Color Richard Szeliski Computational Photography 108

109 Halos Strong edges contain high frequency Low-freq. Reduce low frequency High-freq. Color Richard Szeliski Computational Photography 109

110 Our approach Do not blur across edges: non-linear filtering Large-scale Output Detail Color Richard Szeliski Computational Photography 110

111 Bilateral l filter Tomasi and Manduci Related to SUSAN filter [Smith and Brady 95] Digital-TV [Chan, Osher and Chen 2001] sigma filter Richard Szeliski Computational Photography 111

112 Start t with Gaussian filteringi Output t is blurred J f I output input Richard Szeliski Computational Photography 112

113 Bilateral l filtering i is non-linear The weights are different for each output t pixel 1 J (x) f ( x, ) g( I( ) I( x)) I( ) k(x) ( ) x x output input Richard Szeliski Computational Photography 113

114 Other view The bilateral filter uses the 3D distance Richard Szeliski Computational Photography 114

115 Contrast reduction Input HDR image Output Intensity Large scale Reduce Large scale contrast Fast Bilateral Filter Detail Preserve! Detail Color Color Richard Szeliski Computational Photography 115

116 Dynamic range reduction To reduce contrast of base layer scale in the log domain exponent in linear Set a target range: log 10 (5) Compute range in the log layer: (max-min) Deduce using divisioni i Normalize so that the biggest value in the (linear) base is 1 (0 in log): offset the compressed based by its max Richard Szeliski Computational Photography 116

117 Summary of approach Do not blur base/gain layer: non-linear filtering Large-scale Output Detail Color Richard Szeliski Computational Photography 117

118 Gradient domain high dynamic range compression Raanan Fattal, Dani Lischinski, and Michael Werman SIGGRAPH 2002.

119 Gradient Tone Mapping Slide from Siggraph 2005 by Raskar (Graphs by Fattal et al.) Richard Szeliski Computational Photography 119

120 Gradient attenuation ti From Fattal et al. Richard Szeliski Computational Photography 120

121 Interactive Local Adjustment of Tonal Values Dani Lischinski Zeev Farbman The Hebrew University Matt Uyttendaele Rick Szeliski Microsoft Research SIGGRAPH 2006

122 Tonal Manipulation brightness exposure contrast saturation color temperature Richard Szeliski Computational Photography 122

123 Interpretation 1: Richard Szeliski Computational Photography 123

124 Interpretation 2: Richard Szeliski Computational Photography 124

125 Interpretation 3: Richard Szeliski Computational Photography 125

126 This Work is About: New tool for interactive e tonal manipulation: developing negatives in the digital darkroom. Target material: HDR images: the ultimate digital negative. Camera RAW images: the most common digital negative. Ordinary snapshots. Richard Szeliski Computational Photography 126

127 Existing Tools Automatic tone mapping algorithms Why do we need yet another tone mapping approach? Why interactive rather than automatic? Image manipulation and editing packages, e.g., Adobe Photoshop. Richard Szeliski Computational Photography 127

128 Tone Reproduction Operators Bilateral Filtering Gradient Domain Photographic Durand & Dorsey 2002 Fattal et al Reinhard et al Richard Szeliski Computational Photography 128

129 Automatic ti vs. Interactive ti Bilateral Filtering Interactive Tone Photographic Durand & Dorsey 2002 Mapping Reinhard et al Richard Szeliski Computational Photography 129

130 Automatic ti vs. Interactive ti Existing automatic TM operators are black boxes No direct control over the outcome No local adjustment Not suitable for creative/artistic work Results do not always look photographic Most operators not really automatic Richard Szeliski Computational Photography 130

131 But What About Photoshop? h You can do just about everything er Adjustment Layers Layer Masks Select regions Paint blending weights but you need a lot of experience, patience, and time! Richard Szeliski Computational Photography 131

132 Example 15 minutes in Photoshop: Our 3 minutes: approach Richard Szeliski Computational Photography 132

133 Approach User indicates regions using scribbles. User adjusts tonal values using sliders. Scribbles + tonal values are interpreted as soft constraints. t Optimization framework propagates the constraints to the entire image. Richard Szeliski Computational Photography 133

134 User interface Richard Szeliski Computational Photography 134

135 Input: constraints t +0.5 f-stops -10f-stops f-stops +1.2 f-stops Richard Szeliski Computational Photography 135

136 Result: adjustment t map Richard Szeliski Computational Photography 136

137 Constraint t Propagation Approximate constraints with a function whose smoothness is determined by underlying image: data term smoothness term Our smoothness term: Richard Szeliski Computational Photography 137

138 Influence Functions Richard Szeliski Computational Photography 138

139 Influence Functions Richard Szeliski Computational Photography 139

140 Automatic ti Initialization Inspired by Ansel Adams Zone System. Segment image (very crudely) into brightness zones Determine the desired exposure for each zone Let the image-guided optimization produce a piecewise smooth exposure map Richard Szeliski Computational Photography 140

141 Results Automatic ti mode Richard Szeliski Computational Photography 141

142 Results Automatic ti Mode Richard Szeliski Computational Photography 142

143 Results Automatic ti mode Richard Szeliski Computational Photography 143

144 Richard Szeliski Computational Photography 144

145 Richard Szeliski Computational Photography 145

146 Richard Szeliski Computational Photography 146

147 Snapshot Enhancement Richard Szeliski Computational Photography 147

148 Snapshot Enhancement Richard Szeliski Computational Photography 148

149 Spatially Variant White Balance Richard Szeliski Computational Photography 149

150 Comparison of tone mappers Durand and Dorsey. Fast bilateral filtering for the display of high-dynamic-range images. SIGGRAPH Fattal, Lischinski, and Werman. Gradient domain high dynamic range compression. SIGGRAPH Li, Sharan, and Adelson. Compressing and Companding High Dynamic Range Images with Subband Architectures. SIGGRAPH Richard Szeliski Computational Photography 150

151 Li et al Fattal et al Lischinski et al Reinhard et al Durand & Dorsey 2002 Richard Szeliski Computational Photography 151

152 Merging flash and non-flash images Georg Petschnigg, Maneesh Agrawala, Hugues Hoppe, Rick Szeliski, Michael Cohen, Kentaro Toyama [SIGGRAPH 2004]

153 Flash + non-flash images Flash photos have less noise, more detail Non-flash photos have better color Idea: merge them togetherth But how? + = non-flash flash merged Richard Szeliski Computational Photography 153

154 Flash + non-flash images Smooth non-flash photo using flash photo s edge information Add high-frequency details from flash image + = non-flash flash merged Richard Szeliski Computational Photography 154

155 Joint bilateral l filter Richard Szeliski Computational Photography 155

156 Bilateral l detail filter Richard Szeliski Computational Photography 156

157 Final result merged non-flash Richard Szeliski Computational Photography 157

158

159 Image Formation Color Filters Pyramids Local Features Texture Alignment Flow Stereo SFM Coda Recognition Intro. Topic Models Recognition Kernels Voting and Parts Context Articulated Recognition Photometric Stereo Tracking MRFS Segmentation Comp. Photography