CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt. Session 7 Pixels and Image Filtering Mani Golparvar-Fard Department of Civil and Environmental Engineering 329D, Newmark Civil Engineering Lab e-mail: mgolpar@illinois.edu Department of Civil and Environmental Engineering, University of Illinois at Urbana-Champaign
Outline What we have learned so far: Image formation Camera Calibration and Single View Metrology Introduction to Pixels and Image Filtering Review of lighting Reflection and absorption What is image filtering and how do we do it? Color models (if time allows) Reading: [FP] Chapters 7,8 Some slides in this lecture are courtesy to Prof. S. Savarese, Prof. D. Hoiem, prof F. Li, prof S. Lazebnik, and various other lecturers CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 2
From the 3D to 2D P = [x,y,z] p = [x,y] 3D world Image Let s now focus on 2D Extract building blocks CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 3
Extract useful building blocks 4 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23
The big picture Feature Detection e.g. DoG Feature Description e.g. SIFT database of local descriptors Matching / Indexing / Detection CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 5
Images as functions We can think of an image as a function f, from R 2 to R: Defined over a rectangle, with a finite range: f: [a, b] [c, d] [,255] f( x, y ) gives the intensity at position ( x, y ) A color image: Source: S. Seitz r( x, y) f ( x, y) g( x, y) b ( x, y) CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 6
Images as functions 7 Source: S. Seitz CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23
Images as functions Images are usually digital (discrete): Sample the 2D space on a regular grid The image can now be represented as a matrix of integer values pixel Source: S. Seitz CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 8
Pixels and Reflection Models Quick Overview CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 9
Reflection models Albedo: fraction of light that is reflected Determines color (amount reflected at each wavelength) Very low albedo (hard to see shape) Higher albedo Credit: D. Hoiem CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23
Reflection models Specular reflection: mirror-like Light reflects at incident angle Reflection color = incoming light color Credit: D. Hoiem CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23
Reflection models Diffuse reflection Light scatters in all directions (proportional to cosine with surface normal) Observed intensity is independent of viewing direction Reflection color depends on light color and albedo Credit: D. Hoiem CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 2
Surface orientation and light intensity Amount of light that hits surface from distant point source depends on angle between surface normal and source 2 I x = ρ x S N(x) Credit: D. Hoiem prop to cosine of relative angle CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 3
Reflection models Lambertian: reflection all diffuse Mirrored: reflection all specular Glossy: reflection mostly diffuse, some specular Specularities Credit: D. Hoiem CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 4
Questions How many light sources are in the scene? How could I estimate the color of the camera s flash? Credit: D. Hoiem CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 5
The plight of the poor pixel A pixel s brightness is determined by Light source (strength, direction, color) Surface orientation Surface material and albedo Reflected light and shadows from surrounding surfaces Gain on the sensor A pixel s brightness tells us nothing by itself 6 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23
Basis for interpreting intensity images Key idea: for nearby scene points, most factors do not change much The information is mainly contained in local differences of brightness 7 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23
Darkness = Large Difference in Neighboring Pixels 8 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23
Next few classes: different views of filtering Image filters in spatial domain Filter is a mathematical operation of a grid of numbers Smoothing, sharpening, measuring texture Image filters in the frequency domain Filtering is a way to modify the frequencies of images Denoising, sampling, image compression Templates and Image Pyramids Filtering is a way to match a template to the image Detection, coarse-to-fine registration 9 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23
The raster image (pixel matrix).92.93.94.97.62.37.85.97.93.92.99.95.89.82.89.56.3.75.92.8.95.9.89.72.5.55.5.42.57.4.49.9.92.96.95.88.94.56.46.9.87.9.97.95.7.8.8.87.57.37.8.88.89.79.85.49.62.6.58.5.6.58.5.6.45.33.86.84.74.58.5.39.73.92.9.49.74.96.67.54.85.48.37.88.9.94.82.93.69.49.56.66.43.42.77.73.7.9.99.79.73.9.67.33.6.69.79.73.93.97.9.94.89.49.4.78.78.77.89.99.93 Credit: D. Hoiem CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 2
Image filtering Image filtering: compute function of local neighborhood at each position Linear filtering: function is a weighted sum/difference of pixel values Credit: D. Hoiem Really important! Enhance images Denoise, resize, increase contrast, etc. Extract information from images Texture, edges, distinctive points, etc. Detect patterns Template matching CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 2
Convolution Let f be the image and g be the kernel. The output of convolving f with g is denoted by f * g. ( f g)[ m, n] k, l f [ k, l] g[ m k, n Weighted product of f(k,l) by g(-(k,l))) computed at different locations m,n l] MATLAB: conv2 vs. filter2 (also imfilter) f 22 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23
Example: box filter g[, ] Slide credit: David Lowe (UBC) CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 23
Image filtering g[, ] f [.,.] h[.,.] 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 Credit: S. Seitz h[ m, n] g[ k, l] k, l f [ m k, n l] CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 24
Image filtering g[, ] f [.,.] h[.,.] 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 Credit: S. Seitz h[ m, n] g[ k, l] k, l f [ m k, n l] CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 25
Image filtering g[, ] f [.,.] h[.,.] 2 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 Credit: S. Seitz h[ m, n] g[ k, l] k, l f [ m k, n l] CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 26
Image filtering g[, ] f [.,.] h[.,.] 2 3 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 Credit: S. Seitz h[ m, n] g[ k, l] k, l f [ m k, n l] CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 27
Image filtering g[, ] f [.,.] h[.,.] 2 3 3 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 Credit: S. Seitz h[ m, n] g[ k, l] f [ m k, n l] 28 k, l
Image filtering g[, ] f [.,.] h[.,.] 2 3 3 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9? 9 Credit: S. Seitz h[ m, n] g[ k, l] f [ m k, n l] 29 k, l
Image filtering g[, ] f [.,.] h[.,.] 2 3 3 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9? 9 9 9 9 9 5 9 Credit: S. Seitz h[ m, n] g[ k, l] f [ m k, n l] 3 k, l
Image filtering g[, ] f [.,.] h[.,.] 2 3 3 3 2 9 9 9 9 9 2 4 6 6 6 4 2 9 9 9 9 9 3 6 9 9 9 6 3 9 9 9 9 9 3 5 8 8 9 6 3 9 9 9 9 3 5 8 8 9 6 3 9 9 9 9 9 2 3 5 5 6 4 2 2 3 3 3 3 2 9 h[ m, n] g[ k, l] f [ m k, n l] 3 k, l Credit: S. Seitz
Example: box filter Kernel k with positive entries, that sum to. g[, ] Notice: all weights are equal Slide credit: David Lowe (UBC) CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 32
Smoothing with box filter 33
Practice with linear filters? Original 34 Source: D. Lowe CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23
Practice with linear filters Original Filtered (no change) Source: D. Lowe CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 35
Practice with linear filters? Original Source: D. Lowe CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 36
Practice with linear filters Original Shifted left By pixel Source: D. Lowe CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 37
Practice with linear filters 2 -? Original (Note that filter sums to ) 38 Source: D. Lowe CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23
Practice with linear filters 2 - Original Sharpening filter - Accentuates differences with local average Source: D. Lowe CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 39
Sharpening Source: D. Lowe CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 4
Other filters 2 - -2 - Sobel Source: D. Hoiem Vertical Edge (absolute value) CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 4
Other filters - 2-2 - Sobel Source: D. Hoiem Horizontal Edge (absolute value) CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 42
Basic gradient filters Horizontal Gradient Vertical Gradient - - or - or - 43 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23
How could we synthesize motion blur? theta = 3; len = 2; fil = imrotate(ones(, len), theta, 'bilinear'); fil = fil / sum(fil(:)); figure(2), imshow(imfilter(im, fil)); 44 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23
Filtering vs. Convolution g=filter f=image 2d filtering h=filter2(g,f); or h=imfilter(f,g); h[ m, n] g[ k, l] k, l f [ m k, n l] 2d convolution h=conv2(g,f); h[ m, n] g[ k, l] k, l f [ m k, n l] 45 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23
Key properties of linear filters Linearity: filter(f + f 2 ) = filter(f ) + filter(f 2 ) Shift invariance: same behavior regardless of pixel location filter(shift(f)) = shift(filter(f)) Any linear, shift-invariant operator can be represented as a convolution 46 Source: S. Lazebnik CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23
More properties Commutative: a * b = b * a Conceptually no difference between filter and signal Associative: a * (b * c) = (a * b) * c Often apply several filters one after another: (((a * b ) * b 2 ) * b 3 ) This is equivalent to applying one filter: a * (b * b 2 * b 3 ) Distributes over addition: a * (b + c) = (a * b) + (a * c) Scalars factor out: ka * b = a * kb = k (a * b) Identity: unit impulse e = [,,,, ], a * e = a Source: S. Lazebnik CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 47
Important filter: Gaussian Spatially-weighted average.3.3.22.3.3.3.59.97.59.3.22.97.59.97.22.3.59.97.59.3.3.3.22.3.3 5 x 5, = Slide credit: Christopher Rasmussen CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 48
Smoothing with Gaussian filter 49 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23
Smoothing with box filter 5 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23
Gaussian filters Remove high-frequency components from the image (low-pass filter) Images become more smooth Convolution with self is another Gaussian So can smooth with small-width kernel, repeat, and get same result as larger-width kernel would have Convolving two times with Gaussian kernel of width σ is same as convolving once with kernel of width σ 2 Separable kernel Factors into product of two D Gaussians Source: K. Grauman CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 5
Separability of the Gaussian filter 52 Source: D. Lowe CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23
Separability example 2D filtering (center location only) The filter factors into a product of D filters: Perform filtering along rows: * = Followed by filtering along the remaining column: * = Source: K. Grauman CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 53
Separability Why is separability useful in practice? 54 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23
Some practical matters 55 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23
Practical matters How big should the filter be? Values at edges should be near zero important! Rule of thumb for Gaussian: set filter half-width to about 3 σ 56 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23
Practical matters What about near the edge? the filter window falls off the edge of the image need to extrapolate methods: clip filter (black) wrap around copy edge reflect across edge 57 Source: S. Marschner CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23
Practical matters methods (MATLAB): clip filter (black): imfilter(f, g, ) wrap around: imfilter(f, g, circular ) copy edge: imfilter(f, g, replicate ) reflect across edge: imfilter(f, g, symmetric ) 58 Source: S. Marschner CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23
Practical matters What is the size of the output? MATLAB: filter2(g, f, shape) shape = full : output size is sum of sizes of f and g shape = same : output size is same as f shape = valid : output size is difference of sizes of f and g g full same valid g g g g g f f f g g g g g g Source: S. Lazebnik CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 59
A little more about color 6 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23
Digital Color Images Bayer Filter CMOS sensor 6 Source: D. Hoiem CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23
Color Image R G B 62 Source: D. Hoiem CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23
Images in Matlab Images represented as a matrix Suppose we have a NxM RGB image called im im(,,) = top-left pixel value in R-channel im(y, x, b) = y pixels down, x pixels to right in the b th channel im(n, M, 3) = bottom-right pixel in B-channel imread(filename) returns a uint8 image (values to 255) Convert to double format (values to ) with im2double row column.92.93.94.97.62.37.85.97.93.92.99.95.89.82.89.56.3.75.92.8.95.9.89.72.5.55.5.42.57.4.49.9.92 G.92.93.94.97.62.37.85.97.93.92.99.96.95.88.94.56.46.9.87.9.97.95.95.89.82.89.56.3.75.92.8.95.9.7.8.8.87.57.37.8.88.89.79.85.89.72.5.55.5.42.57.4.49.9.92.49.62.6.58.92.5.93.6.94.58.97.5.62.6.37.45.85.33.97.93.92.99.96.95.88.94.56.46.9.87.9.97.95.86.84.74.58.95.5.89.39.82.73.89.92.56.9.3.49.75.74.92.8.95.9.7.8.8.87.57.37.8.88.89.79.85.96.67.54.85.89.48.72.37.5.88.55.9.5.94.42.82.57.93.4.49.9.92.49.62.6.58.5.6.58.5.6.45.33.69.49.56.66.96.43.95.42.88.77.94.73.56.7.46.9.9.99.87.9.97.95.86.84.74.58.5.39.73.92.9.49.74.79.73.9.67.7.33.8.6.8.69.87.79.57.73.37.93.8.97.88.89.79.85.96.67.54.85.48.37.88.9.94.82.93.9.94.89.49.49.4.62.78.6.78.58.77.5.89.6.99.58.93.5.6.45.33.69.49.56.66.43.42.77.73.7.9.99.86.84.74.58.5.39.73.92.9.49.74.79.73.9.67.33.6.69.79.73.93.97.96.67.54.85.48.37.88.9.94.82.93.9.94.89.49.4.78.78.77.89.99.93.69.49.56.66.43.42.77.73.7.9.99.79.73.9.67.33.6.69.79.73.93.97.9.94.89.49.4.78.78.77.89.99.93 R B 63 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23
Color spaces How can we represent color? Source: D. Hoiem CEE598 Visual Sensing for http://en.wikipedia.org/wiki/file:rgb_illumination.jpg Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 64
Color spaces: RGB Default color space,, R (G=,B=),, G (R=,B=),, Some drawbacks Strongly correlated channels Non-perceptual Image from: http://en.wikipedia.org/wiki/file:rgb_color_solid_cube.png B (R=,G=) CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 65
Color spaces: HSV Intuitive color space H (S=,V=) S (H=,V=) V (H=,S=) 66 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23
Color spaces: YCbCr Fast to compute, good for compression, used by TV Y= Y=.5 Y (Cb=.5,Cr=.5) Cr Cb Y= Cb (Y=.5,Cr=.5) Cr (Y=.5,Cb=5) 67 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23
Color spaces: CIE L*a*b* Perceptually uniform color space L (a=,b=) a (L=65,b=) Luminance = brightness Chrominance = color b (L=65,a=) CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 68
Which contains more information? (a) intensity ( channel) (b) chrominance (2 channels) 69 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23
Most information in intensity Source: D. Hoiem Only color shown constant intensity CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 7
Most information in intensity Source: D. Hoiem Only intensity shown constant color CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 72
Most information in intensity Source: D. Hoiem Original image CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23 73
.92.93.94.97.62.37.85.97.93.92.99.95.89.82.89.56.3.75.92.8.95.9.89.72.5.55.5.42.57.4.49.9.92.96.95.88.94.56.46.9.87.9.97.95.7.8.8.87.57.37.8.88.89.79.85.49.62.6.58.5.6.58.5.6.45.33.86.84.74.58.5.39.73.92.9.49.74.96.67.54.85.48.37.88.9.94.82.93.69.49.56.66.43.42.77.73.7.9.99.79.73.9.67.33.6.69.79.73.93.97.9.94.89.49.4.78.78.77.89.99.93 Important Take Home Messages Image is a matrix of numbers (light intensities at different orientations) Interpretted mainly through local comparisons = Linear filtering is sum of dot product at each position Can smooth, sharpen, translate (among many other uses) Attend to details: filter size, extrapolation, cropping Color spaces beyond RGB sometimes useful 74 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. Mani Golparvar-Fard, 23