Image Analysis & Retrieval CS/EE 559 Special Topics (Class Ids: 44873, 44874) Fall 26, M/W 4-5:5pm@Bloch 2 Lec 4: Image Filtering and Edge Features Zhu Li Dept of CSEE, UMKC Office: FH56E, Email: lizhu@umkc.edu, Ph: x 2346. http://l.web.umkc.edu/lizhu Outline Recap of Lec 3 Perspective Projection &Homography SVD and an example Image Filtering A Quick Review Image Filtering and Correlation Edge Detection Edge Descriptors HOG SIFT a taste of it Z. Li, Image Analysis & Retrieval, 26 p. Z. Li, Image Analysis & Retrieval, 26 p.2 Camera Projection: Intrinsic + Extrinsic Parameters Camera Intrinsic + Extrinsic Parameters: x K R t X u w v ur v r x tx y t y z r 3 r32 r33 tz Intrinsic: camera center:[u, v ], focus length/aspect ratio: [kf, lf] Extrinsic: Rotation R=R x R y R z, translation t=[t x, t y, t z ] R,t 2 r r 2 22 r r 3 23 j w Homography In general, homography H maps 2d points according to, x =Hx x y w = h h 2 h 3 h 4 h 5 h 6 h 7 h 8 h 9 x y w Up to a scale, as [x, y, w]=[sx, sy, sw], so H has 8 DoF Affine Transform: 6 DoF: Contains a translation [t, t 2 ], and invertable affine matrix A [2x2] x X i w O w k w a a 2 t H = a 3 a 4 t 2 Similarity Transform, 4DoF: a rigid transform that preserves distance if s=: s cos θ s sin(θ) t H = s sin θ s cos(θ) t 2 Image Analysis & Retrieval, 26 p.3 Image Analysis & Retrieval, 26 p.4
Homography Estimation SVD Pseudo Inv In matrix form, we have, Ah = If we have more than 4, then the system is over-determined, we will find a solution by least squares, computationally via SVD Recap of Lec 3 SVD and an example Outline Image Filtering A Quick Review Image Filters Edge Detection Edge Feature Pseudo Inverse: SVD: A = VSU T Pseudo inverse (minimizing least square error) A + = US + V T Image Analysis & Retrieval, 26 p.5 Z. Li, Image Analysis & Retrieval, 26 p.6 The famous Lenna. What is an Image? What is an image? We can think of a (grayscale) image as a function, f, from R 2 to R (or a 2D signal): f (x,y) gives the intensity at position (x,y) f (x, y) x y Lenna, 52x52 Lena at ICIP 25, Quebec City A digital image is a discrete (sampled, quantized) version of this function Z. Li, Image Analysis & Retrieval, 26 p.7 Z. Li, Image Analysis & Retrieval, 26 p.8
Image as a function: I=f(x,y); A grid (matrix) of intensity values Some use cases Motivation: Why Filtering? Edge detection 92 [R(x,y), G(x,y), B(x,y)]=[9,, 54] 22 [249, 25, 23] 44 [22, 5, 77] (common to use one byte per value: = black, 255 = white) Z. Li, Image Analysis & Retrieval, 26 p.9 Z. Li, Image Analysis & Retrieval, 26 p. Image filtering Looking at pixel neighbors Modify the pixels in an image based on some function of a local neighborhood of each pixel Discrete form: Linear filtering One simple version: linear filtering (cross-correlation, convolution) Replace each pixel by a linear combination of its neighbors The prescription for the linear combination is called the kernel (or mask, filter ) 5 3 4 6.5 8 8.5 5 3 Some function Local image data kernel Modified image data 4 5 7 7 Local image data Modified image data Source: L. Zhang Z. Li, Image Analysis & Retrieval, 26 p. Z. Li, Image Analysis & Retrieval, 26 p.2
Cross-Correlation: No flipping of kernel Cross-correlation Let be the image, be the kernel (of size 2k+ x 2k+), and be the output image Convolution Same as cross-correlation, except that the kernel is flipped (horizontally and vertically), called convolution G = H F This is called a cross-correlation operation: Convolution is commutative and associative F=G*h + G*f = G*(h+f) F=G*h*f = G*(h*f) Z. Li, Image Analysis & Retrieval, 26 p.3 Z. Li, Image Analysis & Retrieval, 26 p.4 Find the average Mean filtering Identity filter: Linear filters: examples /9 /9 /9 /9 /9 /9 /9 /9 /9 * = 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 2 3 3 3 2 2 4 6 6 6 4 2 3 6 9 9 9 6 3 3 5 8 8 9 6 3 3 5 8 8 9 6 3 2 3 5 5 6 4 2 2 3 3 3 3 2 Original = * Identical image Source: D. Lowe Z. Li, Image Analysis & Retrieval, 26 p.5 Z. Li, Image Analysis & Retrieval, 26 p.6
Shift Filter Linear filters: examples Average/Blurring Linear filters: examples = * = * Original Shifted left By pixel Original Blur (with a mean filter) Source: D. Lowe Source: D. Lowe Z. Li, Image Analysis & Retrieval, 26 p.7 Z. Li, Image Analysis & Retrieval, 26 p.8 Sharpen: remove the average. Linear filters: examples Box Filter: Smoothing with box filter * 2 - = Original Sharpening filter (accentuates edges) Source: D. Lowe Source: D. Forsyth Z. Li, Image Analysis & Retrieval, 26 p.9 Z. Li, Image Analysis & Retrieval, 26 p.2
Gaussian Kernel with scale σ Gaussian Kernel Gaussian Filters A Scale Space Approximation Gaussian Blur = pixel = 5 pixels = pixels = 3 pixels Z. Li, Image Analysis & Retrieval, 26 p.2 Z. Li, Image Analysis & Retrieval, 26 p.22 Gaussian filter Removes high-frequency components from the image (lowpass filter) Convolution with self is another Gaussian What does blurring take away? Sharpening revisited * = Convolving an image twice with Gaussian, by association rule, is equivalent to convolve with a single Gaussian with kernel: σ = σ + σ 2 original Let s add it back: + α smoothed (5x5) = = detail original detail sharpened Source: K. Grauman Source: S. Lazebnik Z. Li, Image Analysis & Retrieval, 26 p.23 Z. Li, Image Analysis & Retrieval, 26 p.24
Sharpen filter - LoG Sharpen filter image blurred image unit impulse (identity) unfiltered filtered scaled impulse Gaussian Laplacian of Gaussian Z. Li, Image Analysis & Retrieval, 26 p.25 Z. Li, Image Analysis & Retrieval, 26 p.26 Physical world convolutions Convolution in the real world Filtering in Matlab Area of support for the operations Camera Shake = Bokeh: Blur in out of focus regions of an image. * Source: Fergus, et al. Removing Camera Shake from a Single Photograph, SIGGRAPH 26 Source: http://lullaby.homepage.dk/diy camera/bokeh.html To give the same n x m output, need to padding the edge Default is zero padding Also replicate the last edge pixel Or, mirroring (used in MPEG codec) Z. Li, Image Analysis & Retrieval, 26 p.27 Z. Li, Image Analysis & Retrieval, 26 p.28
Image Filtering, Sweet Deal with Matlab It is such a nice tool Main filter operation: im2 = imfilter(im, h, replicate ) Design your filter: h= fspecial( filter_type, kernel_size, options) Filter Design Examples: Sobel Laplacian, Laplacian of Gaussian Gaussian, Difference of Gaussian (SIFT) Matlab Image Filtering Example %%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % image filters %%%%%%%%%%%%%%%%%%%%%%%%%%%%%% im = imread('../pics/lenna.png'); im = rgb2gray(im); im = double(im(2:32, 2:32)); % edge filters h{} = fspecial('sobel'); h{2} = fspecial('laplacian',.25); h{3} = fspecial('log', 7,.25); % gaussians sigmas = [.226,.554,.946]; for k=:length(sigmas) h{3+k} = fspecial('gaussian',, sigmas(k)); end % diff of gaussian h{7} = (h{6} - h{4}); h{7} = h{7}/sum(sum(h{7})); h{8} = h{5} - h{4}; h{8} = h{8}/sum(sum(h{8})); for k=:8 fprintf('\n k=%d', k); figure(26); subplot(2,4,k); grid on; hold on; colormap('gray'); imagesc(h{k}); figure(27); subplot(2,4,k); imshow(imfilter(im, h{k}, 'replicate')); end Z. Li, Image Analysis & Retrieval, 26 p.29 Z. Li, Image Analysis & Retrieval, 26 p.3 Filtering Properties Linear operation that is Shift-Invariant: f(m-k, n-j)*h = g(m-k, n-j), if f*h=g Associative: f*h *h 2 = f*(h *h 2 ) this can save a lot of complexity Distributive: f*h + f*h2 = f*(h+h2) useful in SIFT s DoG filtering. Non-Linear Filters Linear Filters usually remove details from image in smoothing/denoising Non-linear filters like the median filter does better in this Median Filtering in Matlab: I = imread('cameraman.tif'); J = imnoise(i,'salt & pepper',.2); K = medfilt2(j); subplot(2);imshow(j); subplot(22);imshow(k); I u, v = median{i j, k R u, v } Z. Li, Image Analysis & Retrieval, 26 p.3 Z. Li, Image Analysis & Retrieval, 26 p.32
Recap of Lec 3 Outline Image Filtering A Quick Review Image Filtering and Correlation Edge Descriptors Edge detection Edge description Edge Detection Convert a 2D image into a set of curves Extracts salient features of the scene More compact than pixels Z. Li, Image Analysis & Retrieval, 26 p.33 Z. Li, Image Analysis & Retrieval, 26 p.34 Edge Mechanism in Images Various ways of having edges in images. Characterizing edges An edge is a place of rapid change in the image intensity function image intensity function (along horizontal scanline) first derivative surface normal discontinuity depth discontinuity surface color discontinuity illumination discontinuity edges correspond to extrema of derivative Z. Li, Image Analysis & Retrieval, 26 p.35 Z. Li, Image Analysis & Retrieval, 26 p.36
Image derivatives Image Gradient The gradient of an image: How can we differentiate a digital image F[x,y]? Option : reconstruct a continuous image, f, then compute the derivative Option 2: take discrete derivative (finite difference) The gradient points in the direction of most rapid increase in intensity The edge strength is given by the gradient magnitude: How would you implement this as a linear filter? : - : - The gradient direction is given by: Source: S. Seitz how does this relate to the direction of the edge? Source: Steve Seitz Z. Li, Image Analysis & Retrieval, 26 p.37 Z. Li, Image Analysis & Retrieval, 26 p.38 Image Gradient Example Unreliable Gradient due to Nose Compute gradient energy Noisy input image Where is the edge? Source: S. Seitz Z. Li, Image Analysis & Retrieval, 26 p.39 Z. Li, Image Analysis & Retrieval, 26 p.4
Smoothing before gradient computing Instead smooth the image with a Gaussian Filter Then do gradient (difference) operation Is this good enough? Associative Rule Comes to the Rescue Differentiation is convolution, and convolution is associative: This saves us one operation: f h f f * h Source: S. Seitz Z. Li, Image Analysis & Retrieval, 26 p.4 Z. Li, Image Analysis & Retrieval, 26 p.42 DoG filter 2D edge detection filters Directive Gradient Detector How to figure out dominant direction of the edge? Gaussian derivative of Gaussian (x) =? (From vector calculus) Directional deriv. is a linear combination of partial derivatives x direction y direction + = Z. Li, Image Analysis & Retrieval, 26 p.43 Z. Li, Image Analysis & Retrieval, 26 p.44
Derivative of Gaussian filter with a direction To detect edge along a particular angle: Sobel Filter An Approximation of DoG Filter - -2 2-2 - -2 - x direction y direction + = Matlab Implementation: Use a threshold to throw away edge strength not higher than bw_im = edge( sobel, thres) Z. Li, Image Analysis & Retrieval, 26 p.45 Z. Li, Image Analysis & Retrieval, 26 p.46 Sobel operator: example Example original image (Lena) Before Y direction Gradients After X direction Gradients Source: Wikipedia Z. Li, Image Analysis & Retrieval, 26 p.47 Z. Li, Image Analysis & Retrieval, 26 p.48
Finding edges Finding edges where is the edge? gradient magnitude Thresholding(also thinning) thinning Z. Li, Image Analysis & Retrieval, 26 p.49 Z. Li, Image Analysis & Retrieval, 26 p.5 Canny edge detector. Filter image with derivative of Gaussian Canny Edge Detector Canny edge detector MATLAB: edge(image, canny ) 2. Find magnitude and orientation of gradient 3. Non-maximum suppression 4. Linking and thresholding (hysteresis): Define two thresholds: low and high Use the high threshold to start edge curves and the low threshold to continue them original Canny with Canny with The choice of depends on desired behavior large detects large scale edges small detects fine edges Source: D. Lowe, L. Fei Fei Source: S. Seitz Z. Li, Image Analysis & Retrieval, 26 p.5 Z. Li, Image Analysis & Retrieval, 26 p.52
Outline HOG Pipeline Recap of Lec 3 SVD and an example Image Filtering A Quick Review Image Filtering and Correlation Edge Detection Edge Descriptors HOG Histogram of Oriented Gradient SIFT a taste of it HoG Generation: In practice, effect is very small (about %) while some computational time is required* *Navneet Dalal and Bill Triggs. Histograms of Oriented Gradients for Human Detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, SanDiego, USA, June 25. Vol. II, pp. 886 893. Z. Li, Image Analysis & Retrieval, 26 p.53 Z. Li, Image Analysis & Retrieval, 26 p.54 Computing Gradients- various choice of Filters Making HoG Features Different block structure for Histogram Mask Type D centered D uncentered D cubic-corrected 2x2 diagonal 3x3 Sobel Operator [-,, ] [-, ] [, -8,, 8, -] 2 2 2 2 Miss rate at 4 FPPW % 2.5% 2% 2.5% 4% Variants of HOG descriptors. (a) A rectangular HOG (R-HOG) descriptor with 3 3 blocks of cells. (b) Circular HOG (C-HOG) descriptor with the central cell divided into angular sectors as in shape contexts. (c) A C-HOG descriptor with a single central cell. Z. Li, Image Analysis & Retrieval, 26 p.55 Z. Li, Image Analysis & Retrieval, 26 p.56
HOG example Generate local histogram per block 2x2 blocks of 4x4 pixels Each block generate gradient magnitude and angle data at each pixel location Histogram is computed for each block which has 9 bins. 2 2 3 3 4 4 f h,..., h, h,..., h, h,..., h, h h 9 9 9,..., 9 HoG Properties Fixed vector over image blocks (similar to pooled histogram) Not scale invariant. Not rotation invariant Angle Magnitude 5 25 25 5 2 2 5 25 3 5 5 45 47 95 97 2 2 5 3 7 3 7 4 8 5 Binary voting 6 Magnitude voting 4 4 2 3 8 2-9 2-39 4-59 6-79 8-99 -9 2-39 4-59 6-79 6 4 2-9 2-39 4-59 6-79 8-99 -9 2-39 4-59 6-79 In each triplet: () the input image, (2) the corresponding R-HOG feature vector (only the dominant orientation of each cell is shown), (3) the dominant orientations selected by the SVM (obtained by multiplying the feature vector by the corresponding weights from the linear SVM). Z. Li, Image Analysis & Retrieval, 26 p.57 Z. Li, Image Analysis & Retrieval, 26 p.58 VL_FEAT: cellsize = 8 ; hog = vl_hog(im, cellsize, 'verbose') ; imhog = vl_hog('render', hog, 'verbose') ; clf ; imagesc(imhog) ; colormap gray ; VL_FEAT implementation Z. Li, Image Analysis & Retrieval, 26 p.59 Lec 4 Summary Covered basic filtering operation, which is An operation on a neighborhood of pixels (details in DIP/DSP) Are linear operations that can be communicative, distributive and associative o Implications: combine filtering operations. Edge detection Mostly Gradient operation based detection Smoothing is necessary for robust performance Combine smoothing and differentiation, you get DoG derivative of Gaussian (not Difference of Gaussian) Many variations of DoGs, Canny, Sobel etc. Edge Description Histogram of Oriented Gradients: very successful feature, simple to compute, spatial partitioning of images blocks, and then compute gradient stats. Think: how about a Histogram of Spatially Pooled Color? Z. Li, Image Analysis & Retrieval, 26 p.6
Homework - HW- part a): Basic Image Features [ch]=getpooledcolorhist(im, codebook, opt): compute pooled color histogram for given images with a given codebook. Opt will specify the color space used [hog] = gethog(im, opt): compute HoG features from an image Z. Li, Image Analysis & Retrieval, 26 p.6