CSCI 29: Comp Photo Fall 28 @ Brown University James Tompkin Many slides thanks to James Hays old CS 29 course, along with all of its acknowledgements.
Things I forgot on Thursday Grads are not required to do pts of extra credit - This is not a 2-level course Question: How many of you do not have laptops (for in-class labs)?
What is an image?
>> I = rand(256,256); Think-Pair-Share: - What is this? What does it look like? - Which values does it take? - How many values can it take? - Is it an image?
>> I = rand(256,256); >> imshow(i); Danny Alexander
Dimensionality of an Image @ 8bit = 256 values ^ 65,536 Computer says Inf combinations. Some depiction of all possible scenes would fit into this memory.
Dimensionality of an Image @ 8bit = 256 values ^ 65,536 Computer says Inf combinations. Some depiction of all possible scenes would fit into this memory. We have to operate or reason over this extremely high-dimensional space. Subspace of natural images. Deriving low-dimensional, explainable, physically-meaningful models.
Each part of an image is a pixel y x
Each part of an image is a pixel Pixel -> picture element 38 y I(x,y) x
Image as a 2D sampling of signal Signal: function depending on some variable with physical meaning. Image: sampling of that function. 2 variables: xy coordinates 3 variables: xy + time (video) Brightness is the value of the function for visible light Can be other physical values too: temperature, pressure, depth Danny Alexander
Example 2D Images Danny Alexander
Sampling in D f(x) [2,6,5,2] x x Sampling in D takes a function, and returns a vector whose elements are values of that function at the sample points. Danny Alexander
Sampling in 2D Sampling in 2D takes a function and returns a matrix. Danny Alexander
Grayscale Digital Image as Height Map Brightness or intensity x y Danny Alexander
What is each part of a photograph? Pixel -> picture element 27 y I(x,y) x
Integrating light over a range of angles. Camera Sensor Output Image James Hays
Resolution geometric vs. spatial resolution Both images are ~5x5 pixels
Quantization
Quantization Effects Radiometric Resolution 8 bit 256 levels bit 2 levels 2 bit 4 levels 4 bit 6 levels
Images in Matlab NxM image im im(,) = top-left pixel value im(y, x) = y pixels down, x pixels to right im(n, M) = bottom-right pixel row column.92.93.94.97.62.37.85.97.93.92.99.95.89.82.89.56.3.75.92.8.95.9.89.72.5.55.5.42.57.4.49.9.92.96.95.88.94.56.46.9.87.9.97.95.7.8.8.87.57.37.8.88.89.79.85.49.62.6.58.5.6.58.5.6.45.33.86.84.74.58.5.39.73.92.9.49.74.96.67.54.85.48.37.88.9.94.82.93.69.49.56.66.43.42.77.73.7.9.99.79.73.9.67.33.6.69.79.73.93.97.9.94.89.49.4.78.78.77.89.99.93 James Hays
Grayscale intensity.92.93.94.97.62.37.85.97.93.92.99.95.89.82.89.56.3.75.92.8.95.9.89.72.5.55.5.42.57.4.49.9.92.96.95.88.94.56.46.9.87.9.97.95.7.8.8.87.57.37.8.88.89.79.85.49.62.6.58.5.6.58.5.6.45.33.86.84.74.58.5.39.73.92.9.49.74.96.67.54.85.48.37.88.9.94.82.93.69.49.56.66.43.42.77.73.7.9.99.79.73.9.67.33.6.69.79.73.93.97.9.94.89.49.4.78.78.77.89.99.93
James Hays Color R G B
Images in Matlab NxM RGB image im im(,,) = top-left pixel value in R-channel im(y, x, b) = y pixels down, x pixels to right in the b th channel im(n, M, 3) = bottom-right pixel in B-channel row column.92.93.94.97.62.37.85.97.93.92.99.95.89.82.89.56.3.75.92.8.95.9.89.72.5.55.5.42.57.4.49.9.92.92.93.94.97.62.37.85.97.93.92.99.96.95.88.94.56.46.9.87.9.97.95 G.95.89.82.89.56.3.75.92.8.95.9.7.8.8.87.57.37.8.88.89.79.85.89.72.5.55.5.42.57.4.49.9.92.49.62.6.58.92.5.93.6.94.58.97.5.62.6.37.45.85.33.97.93.92.99.96.95.88.94.56.46.9.87.9.97.95.86.84.74.58.95.5.89.39.82.73.89.92.56.9.3.49.75.74.92.8.95.9.7.8.8.87.57.37.8.88.89.79.85.96.67.54.85.89.48.72.37.5.88.55.9.5.94.42.82.57.93.4.49.9.92.49.62.6.58.5.6.58.5.6.45.33.69.49.56.66.96.43.95.42.88.77.94.73.56.7.46.9.9.99.87.9.97.95.86.84.74.58.5.39.73.92.9.49.74.79.73.9.67.7.33.8.6.8.69.87.79.57.73.37.93.8.97.88.89.79.85.96.67.54.85.48.37.88.9.94.82.93.9.94.89.49.49.4.62.78.6.78.58.77.5.89.6.99.58.93.5.6.45.33.69.49.56.66.43.42.77.73.7.9.99.86.84.74.58.5.39.73.92.9.49.74.79.73.9.67.33.6.69.79.73.93.97.96.67.54.85.48.37.88.9.94.82.93.9.94.89.49.4.78.78.77.89.99.93.69.49.56.66.43.42.77.73.7.9.99.79.73.9.67.33.6.69.79.73.93.97.9.94.89.49.4.78.78.77.89.99.93 R B James Hays
Cameras with Three Sensors Lens [Edmund Optics; Adam Wilt]
Cheaper/More Compact Color Sensing: Bayer Filter Estimate RGB at G cells from neighboring values Steve Seitz
Camera Color Response Canon 45D What s going on over here? MaxMax.com
How to capture color? Back in the old days Take three photographs each with a different color filter How to display? Light is additive (linear), so just project three photographs on top of each other! [High Point Scientific]
The Prokudin-Gorsky Photo Collection Prokudin-Gorsky (863-944) Similar design camera of Adolf Miethe [Nadezhda Stanulevich, Russian Academy of Fine Arts]
The Prokudin-Gorsky Photo Collection Prokudin-Gorsky (863-944) Similar design viewing projector of Adolf Miethe [Nadezhda Stanulevich, Russian Academy of Fine Arts]
Alim Khan, emir of Bukhara, 9. [Wikipedia]
Good result Six or seven hours of manual work Geometric correction Radiometric (color) correction Blemish removal Too much! Let s go computational! [Wikipedia]
Project : Image Alignment to Recombine the Prokudin-Gorsky Photo Collection How to align R,G,B channels? Think-pair-share!
Project : Image Alignment to Recombine the Prokudin-Gorsky Photo Collection Approach we will take: sliding window.
Project : Image Alignment to Recombine the Prokudin-Gorsky Photo Collection Approach we will take: sliding window. Compute metric for similarity at each position.
Project : Image Alignment to Recombine the Prokudin-Gorsky Photo Collection Approach we will take: sliding window. Compute metric for similarity at each position. What is a good metric? (open research problem)
Project : Image Alignment to Recombine the Prokudin-Gorsky Photo Collection Approach we will take: sliding window. Compute metric for similarity at each position. Candidate metrics: Sum of Squared Differences (SSD): I = Image to align P = Image to compare x,y = Pixel coordinates u,v = Alignment offset coordinates N = Neighbourhood of pixels to compare (e.g., whole image)
Project : Image Alignment to Recombine the Prokudin-Gorsky Photo Collection Approach we will take: sliding window. Compute metric for similarity at each position. Candidate metrics: Normalized Cross Correlation (NCC): I = Image to align Ī = Mean of I P = Image to compare P= Mean of P x,y = Pixel coordinates u,v = Alignment offset coordinates N = Neighbourhood of pixels to compare (e.g., whole image)
Project : Image Alignment to Recombine the Prokudin-Gorsky Photo Collection Next problem: It s expensive! Especially for high-resolution images. Coarse-to-fine strategy:. Compare at low resolution; find alignment 2. At higher resolution, search around this alignment 3. Repeat
Multi-resolution Image Pyramids Good for: Search over translations Like project Classic coarse-to-fine strategy Search over scale Template matching E.g. find a face at different scales Pre-computation Real-time graphics (texture mapping) Compositing In a couple of lectures Wikipedia Image Pyramids
A bar in the big images is a hair on the zebra s nose; in smaller images, a stripe; in the smallest, the animal s nose Figure from David Forsyth
Algorithm for downsampling by factor of 2. Start with image(h, w) 2. Sample every other pixel im_small = image( :2:end, :2:end ); 3. Repeat until im_small is pixel large. Hays
Image sub-sampling /8 /4 Throw away every other row and column to create a /2 size image. Steve Seitz
Subsampling without filtering /2 /4 (2x subsample) /8 (4x subsample) Steve Seitz
Subsampling and aliasing
Aliasing problem D example (sinewave): Source: S. Marschner
Aliasing problem D example (sinewave): Source: S. Marschner
What s happening? Input signal: Plot as image: x = :.5:5; imagesc(sin((2.^x).*x)) Aliasing! Not enough samples
Aliasing problem Sub-sampling may be dangerous. Characteristic errors may appear: Car wheels rolling the wrong way in movies Striped shirts look funny on color television Checkerboards disintegrate in ray tracing Moiré patterns! Source: D. Forsyth; Wikipedia
Aliasing in graphics Source: A. Efros
Gong 96, 932, Claude Tousignant, Musée des Beaux-Arts de Montréal
Aliasing in video Slide by Steve Seitz
Videos [YouTube; JoinBuzzirk; phrancque]
The blue and green colors are actually the same http://blogs.discovermagazine.com/badastronomy/29/6/24/the-blue-and-the-green/
Nyquist-Shannon Sampling Theorem When sampling a signal at discrete intervals, the sampling frequency must be 2 f max f max = max frequency of the input signal This will allows to reconstruct the original perfectly from the sampled version good bad
How to fix aliasing? Solutions?
Better sensors Solutions: Sample more often
Anti-aliasing Solutions: Sample more often Get rid of all frequencies that are greater than half the new sampling frequency Will lose information But it s better than aliasing Apply a smoothing (low pass) filter Hays
Anti-aliasing Forsyth and Ponce 22
Subsampling without filtering /2 /4 (2x subsample) /8 (4x subsample) Steve Seitz
Subsampling with Gaussian pre-filtering Gaussian /2 G /4 G /8 Gaussian Pyramid [Burt and Adelson, 983] Steve Seitz
IMAGE FILTERING
Image filtering Image filtering: Compute function of local neighborhood at each position h[ m, n] = k, l f [ k, l] I[ m + k, n + l] James Hays
Image filtering Image filtering: Compute function of local neighborhood at each position h=output h[ m, n] = f [ k, l] I[ m + k, n + l] k, l f=filter I=image 2d coords=k,l 2d coords=m,n [ ] [ ] [ ]
Example: box filter f [, ] Slide credit: David Lowe (UBC)
Image filtering f [, ] I[.,.] h[.,.] 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 h[ m, n] = k, l f [ k, l] I[ m + k, n + l] Credit: S. Seitz
Image filtering f [, ] I[.,.] h[.,.] 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 h[ m, n] = k, l f [ k, l] I[ m + k, n + l] Credit: S. Seitz
Image filtering f [, ] I[.,.] h[.,.] 2 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 h[ m, n] = k, l f [ k, l] I[ m + k, n + l] Credit: S. Seitz
Image filtering f [, ] I[.,.] h[.,.] 2 3 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 h[ m, n] = k, l f [ k, l] I[ m + k, n + l] Credit: S. Seitz
Image filtering f [, ] I[.,.] h[.,.] 2 3 3 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 h[ m, n] = k, l f [ k, l] I[ m + k, n + l] Credit: S. Seitz
Image filtering f [, ] I[.,.] h[.,.] 2 3 3 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9? 9 h[ m, n] = k, l f [ k, l] I[ m + k, n + l] Credit: S. Seitz
Image filtering f [, ] I[.,.] h[.,.] 2 3 3 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9? 9 9 9 9 9 9 9 9 9 5 9 h[ m, n] = k, l f [ k, l] I[ m + k, n + l] Credit: S. Seitz
Image filtering f [, ] I[.,.] h[.,.] 2 3 3 3 2 9 9 9 9 9 2 4 6 6 6 4 2 9 9 9 9 9 3 6 9 9 9 6 3 9 9 9 9 9 3 5 8 8 9 6 3 9 9 9 9 3 5 8 8 9 6 3 9 9 9 9 9 2 3 5 5 6 4 2 2 3 3 3 3 2 9 h[ m, n] = k, l f [ k, l] I[ m + k, n + l] Credit: S. Seitz
Box Filter What does it do? Replaces each pixel with an average of its neighborhood f [, ] Achieve smoothing effect (remove sharp features) Slide credit: David Lowe (UBC)
Box Filter What does it do? Replaces each pixel with an average of its neighborhood f [, ] Achieve smoothing effect (remove sharp features) Why does it sum to one? Slide credit: David Lowe (UBC)
Smoothing with box filter James Hays
Image filtering Image filtering: Compute function of local neighborhood at each position h[ m, n] = k, l f [ k, l] I[ m + k, n + l] Really important! Enhance images Denoise, resize, increase contrast, etc. Extract information from images Texture, edges, distinctive points, etc. Detect patterns Template matching James Hays
Think-Pair-Share time. 2. - -2 2-2 - 3. 4.
. Practice with linear filters? Original Source: D. Lowe
. Practice with linear filters Original Filtered (no change) Source: D. Lowe
2. Practice with linear filters? Original Source: D. Lowe
2. Practice with linear filters Original Shifted left By pixel Source: D. Lowe
3. Practice with linear filters 2 Sobel - -2 - Vertical Edge (absolute value) David Lowe
3. Practice with linear filters - 2-2 Sobel - Horizontal Edge (absolute value) David Lowe
4. Practice with linear filters 2 -? Original (Note that filter sums to ) Source: D. Lowe
4. Practice with linear filters 2 - Original Sharpening filter - Accentuates differences with local average Source: D. Lowe
4. Practice with linear filters Source: D. Lowe
Correlation and Convolution 2d correlation h[ m, n] = f [ k, l] I[ m + k, n + l] k, l h=filter2(f,i); or h=imfilter(i,f); James Hays
Correlation and Convolution 2d correlation h[ m, n] = f [ k, l] I[ m + k, n + l] k, l h=filter2(f,i); or h=imfilter(i,f); 2d convolution h[ m, n] = k, l f [ k, l] I[ m k, n l] h=conv2(f,i); or h=imfilter(i,f, conv ); conv2(i,f)is the same as filter2(rot9(f,2),i) Correlation and convolution are identical when the filter is symmetric. James Hays
Key properties of linear filters Linearity: imfilter(i, f + f 2 ) = imfilter(i,f ) + imfilter(i,f 2 ) Shift invariance: Same behavior regardless of pixel location imfilter(i,shift(f)) = shift(imfilter(i,f)) Any linear, shift-invariant operator can be represented as a convolution. Source: S. Lazebnik
Convolution properties Commutative: a * b = b * a Conceptually no difference between filter and signal But particular filtering implementations might break this equality, e.g., image edges Associative: a * (b * c) = (a * b) * c Often apply several filters one after another: (((a * b ) * b 2 ) * b 3 ) This is equivalent to applying one filter: a * (b * b 2 * b 3 ) Source: S. Lazebnik
Convolution properties Commutative: a * b = b * a Conceptually no difference between filter and signal But particular filtering implementations might break this equality, e.g., image edges Associative: a * (b * c) = (a * b) * c Often apply several filters one after another: (((a * b ) * b 2 ) * b 3 ) This is equivalent to applying one filter: a * (b * b 2 * b 3 ) Correlation is _not_ associative (rotation effect) Why important? Source: S. Lazebnik
Convolution properties Commutative: a * b = b * a Conceptually no difference between filter and signal But particular filtering implementations might break this equality, e.g., image edges Associative: a * (b * c) = (a * b) * c Often apply several filters one after another: (((a * b ) * b 2 ) * b 3 ) This is equivalent to applying one filter: a * (b * b 2 * b 3 ) Correlation is _not_ associative (rotation effect) Why important? Distributes over addition: a * (b + c) = (a * b) + (a * c) Scalars factor out: ka * b = a * kb = k (a * b) Identity: unit impulse e = [,,,, ], a * e = a Source: S. Lazebnik
Important filter: Gaussian Weight contributions of neighboring pixels by nearness x y.3.3.22.3.3.3.59.97.59.3.22.97.59.97.22.3.59.97.59.3.3.3.22.3.3 x y 5 x 5, = Slide credit: Christopher Rasmussen
Smoothing with box filter f [, ] James Hays
Smoothing with Gaussian filter James Hays
Gaussian filters Remove high-frequency components from the image (low-pass filter) Images become more smooth Gaussian convolved with Gaussian is another Gaussian So can smooth with small-width kernel, repeat, and get same result as larger-width kernel would have Convolving two times with Gaussian kernel of width σ is same as convolving once with kernel of width σ 2 Separable kernel Factors into product of two D Gaussians Source: K. Grauman
Separability of the Gaussian filter Source: D. Lowe
Separability example 2D convolution (center location only) = The filter factors into a product of D filters: Perform convolution along rows: * = Followed by convolution along the remaining column: * = Source: K. Grauman
Separability Why is separability useful in practice?
Separability Why is separability useful in practice? MxN image, PxQ filter 2D convolution: ~MNPQ multiply-adds Separable 2D: ~MN(P+Q) multiply-adds Speed up = PQ/(P+Q) 9x9 filter = ~4.5x faster
Practical matters How big should the filter be? Values at edges should be near zero Gaussians have infinite extent Rule of thumb for Gaussian: set filter half-width to about 3 σ James Hays
Practical matters What about near the edge? the filter window falls off the edge of the image need to extrapolate methods: clip filter (black) wrap around copy edge reflect across edge Source: S. Marschner
Project : Image Alignment to Recombine the Prokudin-Gorsky Photo Collection Make alignment operation fast using image pyramid. Remove aliasing from pyramid by filtering. Coarse-to-fine strategy:. Build Gaussian Pyramid. Compare at low resolution; find alignment 2. At higher resolution, search around this alignment 3. Repeat