CSCI 1290: Comp Photo

Transcription

1 CSCI 29: Comp Photo Fall Brown University James Tompkin Many slides thanks to James Hays old CS 29 course, along with all of its acknowledgements.

2 Things I forgot on Thursday Grads are not required to do pts of extra credit - This is not a 2-level course Question: How many of you do not have laptops (for in-class labs)?

3 What is an image?

4 >> I = rand(256,256); Think-Pair-Share: - What is this? What does it look like? - Which values does it take? - How many values can it take? - Is it an image?

5 >> I = rand(256,256); >> imshow(i); Danny Alexander

6 Dimensionality of an 8bit = 256 values ^ 65,536 Computer says Inf combinations. Some depiction of all possible scenes would fit into this memory.

7 Dimensionality of an 8bit = 256 values ^ 65,536 Computer says Inf combinations. Some depiction of all possible scenes would fit into this memory. We have to operate or reason over this extremely high-dimensional space. Subspace of natural images. Deriving low-dimensional, explainable, physically-meaningful models.

8 Each part of an image is a pixel y x

9 Each part of an image is a pixel Pixel -> picture element 38 y I(x,y) x

10 Image as a 2D sampling of signal Signal: function depending on some variable with physical meaning. Image: sampling of that function. 2 variables: xy coordinates 3 variables: xy + time (video) Brightness is the value of the function for visible light Can be other physical values too: temperature, pressure, depth Danny Alexander

11 Example 2D Images Danny Alexander

12 Sampling in D f(x) [2,6,5,2] x x Sampling in D takes a function, and returns a vector whose elements are values of that function at the sample points. Danny Alexander

13 Sampling in 2D Sampling in 2D takes a function and returns a matrix. Danny Alexander

14 Grayscale Digital Image as Height Map Brightness or intensity x y Danny Alexander

15 What is each part of a photograph? Pixel -> picture element 27 y I(x,y) x

16 Integrating light over a range of angles. Camera Sensor Output Image James Hays

17 Resolution geometric vs. spatial resolution Both images are ~5x5 pixels

18 Quantization

19 Quantization Effects Radiometric Resolution 8 bit 256 levels bit 2 levels 2 bit 4 levels 4 bit 6 levels

20 Images in Matlab NxM image im im(,) = top-left pixel value im(y, x) = y pixels down, x pixels to right im(n, M) = bottom-right pixel row column James Hays

21 Grayscale intensity

22 James Hays Color R G B

23 Images in Matlab NxM RGB image im im(,,) = top-left pixel value in R-channel im(y, x, b) = y pixels down, x pixels to right in the b th channel im(n, M, 3) = bottom-right pixel in B-channel row column G R B James Hays

24 Cameras with Three Sensors Lens [Edmund Optics; Adam Wilt]

25 Cheaper/More Compact Color Sensing: Bayer Filter Estimate RGB at G cells from neighboring values Steve Seitz

26 Camera Color Response Canon 45D What s going on over here? MaxMax.com

27 How to capture color? Back in the old days Take three photographs each with a different color filter How to display? Light is additive (linear), so just project three photographs on top of each other! [High Point Scientific]

28 The Prokudin-Gorsky Photo Collection Prokudin-Gorsky ( ) Similar design camera of Adolf Miethe [Nadezhda Stanulevich, Russian Academy of Fine Arts]

29 The Prokudin-Gorsky Photo Collection Prokudin-Gorsky ( ) Similar design viewing projector of Adolf Miethe [Nadezhda Stanulevich, Russian Academy of Fine Arts]

30 Alim Khan, emir of Bukhara, 9. [Wikipedia]

31 Good result Six or seven hours of manual work Geometric correction Radiometric (color) correction Blemish removal Too much! Let s go computational! [Wikipedia]

32 Project : Image Alignment to Recombine the Prokudin-Gorsky Photo Collection How to align R,G,B channels? Think-pair-share!

33 Project : Image Alignment to Recombine the Prokudin-Gorsky Photo Collection Approach we will take: sliding window.

34 Project : Image Alignment to Recombine the Prokudin-Gorsky Photo Collection Approach we will take: sliding window. Compute metric for similarity at each position.

35 Project : Image Alignment to Recombine the Prokudin-Gorsky Photo Collection Approach we will take: sliding window. Compute metric for similarity at each position. What is a good metric? (open research problem)

36 Project : Image Alignment to Recombine the Prokudin-Gorsky Photo Collection Approach we will take: sliding window. Compute metric for similarity at each position. Candidate metrics: Sum of Squared Differences (SSD): I = Image to align P = Image to compare x,y = Pixel coordinates u,v = Alignment offset coordinates N = Neighbourhood of pixels to compare (e.g., whole image)

37 Project : Image Alignment to Recombine the Prokudin-Gorsky Photo Collection Approach we will take: sliding window. Compute metric for similarity at each position. Candidate metrics: Normalized Cross Correlation (NCC): I = Image to align Ī = Mean of I P = Image to compare P= Mean of P x,y = Pixel coordinates u,v = Alignment offset coordinates N = Neighbourhood of pixels to compare (e.g., whole image)

38 Project : Image Alignment to Recombine the Prokudin-Gorsky Photo Collection Next problem: It s expensive! Especially for high-resolution images. Coarse-to-fine strategy:. Compare at low resolution; find alignment 2. At higher resolution, search around this alignment 3. Repeat

39 Multi-resolution Image Pyramids Good for: Search over translations Like project Classic coarse-to-fine strategy Search over scale Template matching E.g. find a face at different scales Pre-computation Real-time graphics (texture mapping) Compositing In a couple of lectures Wikipedia Image Pyramids

40 A bar in the big images is a hair on the zebra s nose; in smaller images, a stripe; in the smallest, the animal s nose Figure from David Forsyth

41 Algorithm for downsampling by factor of 2. Start with image(h, w) 2. Sample every other pixel im_small = image( :2:end, :2:end ); 3. Repeat until im_small is pixel large. Hays

42 Image sub-sampling /8 /4 Throw away every other row and column to create a /2 size image. Steve Seitz

43 Subsampling without filtering /2 /4 (2x subsample) /8 (4x subsample) Steve Seitz

44 Subsampling and aliasing

45 Aliasing problem D example (sinewave): Source: S. Marschner

46 Aliasing problem D example (sinewave): Source: S. Marschner

47 What s happening? Input signal: Plot as image: x = :.5:5; imagesc(sin((2.^x).*x)) Aliasing! Not enough samples

48 Aliasing problem Sub-sampling may be dangerous. Characteristic errors may appear: Car wheels rolling the wrong way in movies Striped shirts look funny on color television Checkerboards disintegrate in ray tracing Moiré patterns! Source: D. Forsyth; Wikipedia

49 Aliasing in graphics Source: A. Efros

50 Gong 96, 932, Claude Tousignant, Musée des Beaux-Arts de Montréal

51 Aliasing in video Slide by Steve Seitz

52 Videos [YouTube; JoinBuzzirk; phrancque]

53

54 The blue and green colors are actually the same

55 Nyquist-Shannon Sampling Theorem When sampling a signal at discrete intervals, the sampling frequency must be 2 f max f max = max frequency of the input signal This will allows to reconstruct the original perfectly from the sampled version good bad

56 How to fix aliasing? Solutions?

57 Better sensors Solutions: Sample more often

58 Anti-aliasing Solutions: Sample more often Get rid of all frequencies that are greater than half the new sampling frequency Will lose information But it s better than aliasing Apply a smoothing (low pass) filter Hays

59 Anti-aliasing Forsyth and Ponce 22

60 Subsampling without filtering /2 /4 (2x subsample) /8 (4x subsample) Steve Seitz

61 Subsampling with Gaussian pre-filtering Gaussian /2 G /4 G /8 Gaussian Pyramid [Burt and Adelson, 983] Steve Seitz

62 IMAGE FILTERING

63 Image filtering Image filtering: Compute function of local neighborhood at each position h[ m, n] = k, l f [ k, l] I[ m + k, n + l] James Hays

64 Image filtering Image filtering: Compute function of local neighborhood at each position h=output h[ m, n] = f [ k, l] I[ m + k, n + l] k, l f=filter I=image 2d coords=k,l 2d coords=m,n [ ] [ ] [ ]

65 Example: box filter f [, ] Slide credit: David Lowe (UBC)

66 Image filtering f [, ] I[.,.] h[.,.] h[ m, n] = k, l f [ k, l] I[ m + k, n + l] Credit: S. Seitz

67 Image filtering f [, ] I[.,.] h[.,.] h[ m, n] = k, l f [ k, l] I[ m + k, n + l] Credit: S. Seitz

68 Image filtering f [, ] I[.,.] h[.,.] h[ m, n] = k, l f [ k, l] I[ m + k, n + l] Credit: S. Seitz

69 Image filtering f [, ] I[.,.] h[.,.] h[ m, n] = k, l f [ k, l] I[ m + k, n + l] Credit: S. Seitz

70 Image filtering f [, ] I[.,.] h[.,.] h[ m, n] = k, l f [ k, l] I[ m + k, n + l] Credit: S. Seitz

71 Image filtering f [, ] I[.,.] h[.,.] ? 9 h[ m, n] = k, l f [ k, l] I[ m + k, n + l] Credit: S. Seitz

72 Image filtering f [, ] I[.,.] h[.,.] ? h[ m, n] = k, l f [ k, l] I[ m + k, n + l] Credit: S. Seitz

73 Image filtering f [, ] I[.,.] h[.,.] h[ m, n] = k, l f [ k, l] I[ m + k, n + l] Credit: S. Seitz

74 Box Filter What does it do? Replaces each pixel with an average of its neighborhood f [, ] Achieve smoothing effect (remove sharp features) Slide credit: David Lowe (UBC)

75 Box Filter What does it do? Replaces each pixel with an average of its neighborhood f [, ] Achieve smoothing effect (remove sharp features) Why does it sum to one? Slide credit: David Lowe (UBC)

76 Smoothing with box filter James Hays

77 Image filtering Image filtering: Compute function of local neighborhood at each position h[ m, n] = k, l f [ k, l] I[ m + k, n + l] Really important! Enhance images Denoise, resize, increase contrast, etc. Extract information from images Texture, edges, distinctive points, etc. Detect patterns Template matching James Hays

78 Think-Pair-Share time

79 . Practice with linear filters? Original Source: D. Lowe

80 . Practice with linear filters Original Filtered (no change) Source: D. Lowe

81 2. Practice with linear filters? Original Source: D. Lowe

82 2. Practice with linear filters Original Shifted left By pixel Source: D. Lowe

83 3. Practice with linear filters 2 Sobel Vertical Edge (absolute value) David Lowe

84 3. Practice with linear filters Sobel - Horizontal Edge (absolute value) David Lowe

85 4. Practice with linear filters 2 -? Original (Note that filter sums to ) Source: D. Lowe

86 4. Practice with linear filters 2 - Original Sharpening filter - Accentuates differences with local average Source: D. Lowe

87 4. Practice with linear filters Source: D. Lowe

88 Correlation and Convolution 2d correlation h[ m, n] = f [ k, l] I[ m + k, n + l] k, l h=filter2(f,i); or h=imfilter(i,f); James Hays

89 Correlation and Convolution 2d correlation h[ m, n] = f [ k, l] I[ m + k, n + l] k, l h=filter2(f,i); or h=imfilter(i,f); 2d convolution h[ m, n] = k, l f [ k, l] I[ m k, n l] h=conv2(f,i); or h=imfilter(i,f, conv ); conv2(i,f)is the same as filter2(rot9(f,2),i) Correlation and convolution are identical when the filter is symmetric. James Hays

90 Key properties of linear filters Linearity: imfilter(i, f + f 2 ) = imfilter(i,f ) + imfilter(i,f 2 ) Shift invariance: Same behavior regardless of pixel location imfilter(i,shift(f)) = shift(imfilter(i,f)) Any linear, shift-invariant operator can be represented as a convolution. Source: S. Lazebnik

91 Convolution properties Commutative: a * b = b * a Conceptually no difference between filter and signal But particular filtering implementations might break this equality, e.g., image edges Associative: a * (b * c) = (a * b) * c Often apply several filters one after another: (((a * b ) * b 2 ) * b 3 ) This is equivalent to applying one filter: a * (b * b 2 * b 3 ) Source: S. Lazebnik

92 Convolution properties Commutative: a * b = b * a Conceptually no difference between filter and signal But particular filtering implementations might break this equality, e.g., image edges Associative: a * (b * c) = (a * b) * c Often apply several filters one after another: (((a * b ) * b 2 ) * b 3 ) This is equivalent to applying one filter: a * (b * b 2 * b 3 ) Correlation is _not_ associative (rotation effect) Why important? Source: S. Lazebnik

93 Convolution properties Commutative: a * b = b * a Conceptually no difference between filter and signal But particular filtering implementations might break this equality, e.g., image edges Associative: a * (b * c) = (a * b) * c Often apply several filters one after another: (((a * b ) * b 2 ) * b 3 ) This is equivalent to applying one filter: a * (b * b 2 * b 3 ) Correlation is _not_ associative (rotation effect) Why important? Distributes over addition: a * (b + c) = (a * b) + (a * c) Scalars factor out: ka * b = a * kb = k (a * b) Identity: unit impulse e = [,,,, ], a * e = a Source: S. Lazebnik

94 Important filter: Gaussian Weight contributions of neighboring pixels by nearness x y x y 5 x 5, = Slide credit: Christopher Rasmussen

95 Smoothing with box filter f [, ] James Hays

96 Smoothing with Gaussian filter James Hays

97 Gaussian filters Remove high-frequency components from the image (low-pass filter) Images become more smooth Gaussian convolved with Gaussian is another Gaussian So can smooth with small-width kernel, repeat, and get same result as larger-width kernel would have Convolving two times with Gaussian kernel of width σ is same as convolving once with kernel of width σ 2 Separable kernel Factors into product of two D Gaussians Source: K. Grauman

98 Separability of the Gaussian filter Source: D. Lowe

99 Separability example 2D convolution (center location only) = The filter factors into a product of D filters: Perform convolution along rows: * = Followed by convolution along the remaining column: * = Source: K. Grauman

100 Separability Why is separability useful in practice?

101 Separability Why is separability useful in practice? MxN image, PxQ filter 2D convolution: ~MNPQ multiply-adds Separable 2D: ~MN(P+Q) multiply-adds Speed up = PQ/(P+Q) 9x9 filter = ~4.5x faster

102 Practical matters How big should the filter be? Values at edges should be near zero Gaussians have infinite extent Rule of thumb for Gaussian: set filter half-width to about 3 σ James Hays

103 Practical matters What about near the edge? the filter window falls off the edge of the image need to extrapolate methods: clip filter (black) wrap around copy edge reflect across edge Source: S. Marschner

104 Project : Image Alignment to Recombine the Prokudin-Gorsky Photo Collection Make alignment operation fast using image pyramid. Remove aliasing from pyramid by filtering. Coarse-to-fine strategy:. Build Gaussian Pyramid. Compare at low resolution; find alignment 2. At higher resolution, search around this alignment 3. Repeat