Achim J. Lilienthal Mobile Robotics and Olfaction Lab, AASS, Örebro University

Achim J. Lilienthal Mobile Robotics and Olfaction Lab, Room T29, Mo, -2 o'clock AASS, Örebro University (please drop me an email in advance) achim.lilienthal@oru.se

4.!!!!!!!!! Pre-Class Reading!!!!!!!!! Pre-Class Reading Schedule o Class "Course Introduction" (Mar 29, 22) o Class 2 "Introduction" (Apr 2, 22)» Gonzalez/Woods Chapter "Introduction"» Gonzalez/Woods Chapter 2 "Fundamentals"» Lecture Notes from last year o Class 3 "" (Apr 2, 22)» Gonzalez/Woods Chapter 3 "Intensity Transformations and "» Lecture Notes from last year o Class 4 "Bilateral Filtering/Fourier Domain" (Apr 7, 22)» "A Gentle Introduction to Bilateral Filtering and its Applications", Sylvain Paris, Pierre Kornprobst, Jack Tumblin, and Frédo Durand, SIGGRAPH 28» "Bilateral Filtering for Gray and Color Images", C. Tomasi, R. Manduchi, Proc. Int. Conf. Computer Vision» Gonzalez/Woods Chapter 4 "Filtering in the Frequency Domain"» Lecture Notes from last year DIP'2 A. J. Lilienthal (Apr 2, 22) 2

2. General Introduction Schedule Lectures. week 3: Thu, Mar 29, 22, :5-2: o'clock, T27 2. week 4: Tue, Apr 3, 22, 5:5-7: o'clock, T 3. week 5: Thu, Apr 2, 22, :5-2: o'clock, T27 4. week 6: Tue, Apr 7, 22, 5:5-7: o'clock, T week 6: Thu, Apr 9, 22, :5-2: o'clock, T27 6. week 7: Tue, Apr 24, 22, 5:5-7: o'clock, T 7. week 7: Thu, Apr 26, 22, :5-2: o'clock, T27 8. week 8: Thu, May 3, 22, :5-2: o'clock, T27 9. week 9: Tue, May 8, 22, 5:5-7: o'clock, T. week 9: Thu, May, 22, :5-2: o'clock, T27 Exercises. week 4: Thu, Apr 5, 22, 3:5-7: o'clock, T2 2. week 5: Thu, Apr 2, 22, 3:5-7: o'clock, T2 3. week 6: Thu, Apr 9, 22, 3:5-7: o'clock, T 3. week 8: Thu, May 3, 22, 3:5-7: o'clock, T2 DIP'2 A. J. Lilienthal (Apr 2, 22) 3

Contents. Image Enhancement in the Spatial Domain 2. Grey Level Transformations 3. Histogram Processing 4. Operations Involving Multiple Images Applications People Tracking DIP'2 A. J. Lilienthal (Apr 2, 22) 4

Image Enhancement in the Spatial Domain DIP'2 A. J. Lilienthal (Apr 2, 22) 5

. Image Enhancement in the Spatial Domain Image Enhancement o image processing o the result is supposed to be "more suitable"» "more suitable" according to a certain application more suitable for visual interpretation DIP'2 A. J. Lilienthal (Apr 2, 22) 6

. Image Enhancement in the Spatial Domain We want to create an image which is "better" in some sense o helps visual interpretation (brightening, sharpening ) subjective o pre-processing for a subsequent image analysis algorithm performance metric (performance of a task) o make the image more "specific" application dependent T f(x,y) g(x,y) original image (or set of images) new image DIP'2 A. J. Lilienthal (Apr 2, 22) 7

. Image Enhancement in the Spatial Domain Spatial Domain versus Frequency Domain o spatial domain» direct manipulation of the pixels discussed in this lecture» two types of transformations in the spatial domain: pixel brightness transformations, point processing (depend only on the pixel value itself) spatial filters, local transformations or local processing (depend on a small neighbourhood around the pixel) o frequency domain: modifications of the Fourier transform» discussed in coming lectures DIP'2 A. J. Lilienthal (Apr 2, 22) 9

. Image Enhancement in the Spatial Domain Transformations in the Spatial Domain g ( x, y) = T[ f ( x, y)] o standard approach: T is applied to a sub-image centred at (x,y) o sub-image is called mask (kernel, filter, template, window) o mask processing or filtering o T can operate on a set of images DIP'2 A. J. Lilienthal (Apr 2, 22)

. Image Enhancement in the Spatial Domain Transformations in the Spatial Domain g ( x, y) = T[ f ( x, y)] o fill new array with weighted sum of pixel values from the locations surrounding the corresponding location in the image using the same set of weights each time DIP'2 A. J. Lilienthal (Apr 2, 22)

2 Gray Level Transformations DIP'2 A. J. Lilienthal (Apr 2, 22) 2

2. Grey Level Transformations Grey Level Transformations o simplest case: each pixel in the output image depends only on the corresponding pixel in the input image o x neighbourhood (point processing) o example: contrast stretching s = T (r) s = T (r) DIP'2 A. J. Lilienthal (Apr 2, 22) 3

2. Grey Level Transformations Grey Level Transformations contrast stretching thresholding DIP'2 A. J. Lilienthal (Apr 2, 22) 4

2. Grey Level Transformations Grey Level Transformations f = imread('bubbles.tif'); fp = imadjust(f, [..9], [..],.5); imshow(fp); o imadjust» parameters always specified in [,]» values below. clipped to.» values above.9 clipped to.» image intensity reversed since. <. DIP'2 A. J. Lilienthal (Apr 2, 22) 5

2. Grey Level Transformations Grey Level Transformations f = imread('bubbles.tif'); fp = imadjust(f, [..9], [..],.5); imshow(fp); fp = imadjust(f, [.55.9], [..], 3); DIP'2 A. J. Lilienthal (Apr 2, 22) 7

2. Grey Level Transformations Grey Level Transformations f = imread('bubbles.tif'); fp = imadjust(f, [..9], [..],.5); imshow(fp); fp = imadjust(f, [.55.9], [..], 3); o imadjust» gamma function parameter > g = f γ DIP'2 A. J. Lilienthal (Apr 2, 22) 8

2. Grey Level Transformations Contrast Stretching o piecewise linear function o power law transformation (gamma transformation) γ s = cr DIP'2 A. J. Lilienthal (Apr 2, 22) 9

2. Grey Level Transformations Common Grey Level Transformations (Single Image) o linear» identity» inverse (negative) o power law» n. power» n. root o logarithmic DIP'2 A. J. Lilienthal (Apr 2, 22) 2

2. Grey Level Transformations Common Grey Level Transformations (Single Image) o inverse transform DIP'2 A. J. Lilienthal (Apr 2, 22) 2

2. Grey Level Transformations Common Grey Level Transformations (Single Image) o linear» identity» inverse o piecewise linear o power law (gamma)» n. power» n. root o logarithmic... with more than one input image o sum, mean o transformation based on statistical operations (variance, t-test ) DIP'2 A. J. Lilienthal (Apr 2, 22) 23

3 Histogram Processing DIP'2 A. J. Lilienthal (Apr 2, 22) 24

3. Histogram Processing Grey Scale Histogram o shows the number of pixels per grey level f = imread('bubbles.tif'); imhist(f); % displays the histogram % histogram display default DIP'2 A. J. Lilienthal (Apr 2, 22) 26

3. Histogram Processing Grey Scale Histogram o shows the number of pixels per grey level f = imread('bubbles.tif'); h = imhist(f); % default number of bins = 256 imhist(f,8); % number of bins = 8 DIP'2 A. J. Lilienthal (Apr 2, 22) 27

3. Histogram Processing Grey Scale Histogram o shows the number of pixels per grey level f = imread('bubbles.tif'); h = imhist(f); % default number of bins = 256 h = imhist(f,6); % number of bins = 6 hn = h/numel(f); % normalized histogram % numel num. of elements (pixels) bar(hn) % normalized histogram DIP'2 A. J. Lilienthal (Apr 2, 22) 28

3. Histogram Processing Grey Scale Histogram o neutral transform DIP'2 A. J. Lilienthal (Apr 2, 22) 3

3. Histogram Processing Grey Scale Histogram o neutral transform o inverse transform DIP'2 A. J. Lilienthal (Apr 2, 22) 3

3. Histogram Processing Grey Scale Histogram o neutral transform o inverse transform o logarithmic transform DIP'2 A. J. Lilienthal (Apr 2, 22) 32

3. Histogram Processing Histogram Equalization o contrast / brightness adjustments sometimes need to be automatised o "optimal" contrast for an image? flat histogram DIP'2 A. J. Lilienthal (Apr 2, 22) 36

3. Histogram Processing Histogram Equalization o consider the continuous case: s, r [,] o probability density functions (PDFs) of s and r are related by gray levels as random variables! s = T (r) p s ( s) = p r ( r) dr ds = p r ( r) T ( r) o transformation function = cumulative density function (CDF) ds dr r T ( r) p r ( ω) dω r d = T ( r) = pr ( ω) dω = pr ( r) p s ( s) = dr DIP'2 A. J. Lilienthal (Apr 2, 22) 37

3. Histogram Processing Histogram Equalization o discrete case pr rk ) = nk n ( s = = = k T ( rk ) pr ( rj ) j= o does not generally produce a uniform PDF o tends to spread the histogram o enables automatic contrast stretching k k j= n j n DIP'2 A. J. Lilienthal (Apr 2, 22) 38

3. Histogram Processing Histogram Equalization CDF DIP'2 A. J. Lilienthal (Apr 2, 22) 39

3. Histogram Processing Histogram Equalization DIP'2 A. J. Lilienthal (Apr 2, 22) 4

3. Histogram Processing Histogram Equalization f = imread('bubbles.tif'); g = histeq(f, 256); imshow(g); f = imread('bubbles.tif'); g = histeq(f, 4); % 4 output levels imshow(g); DIP'2 A. J. Lilienthal (Apr 2, 22) 4

5 DIP'2 A. J. Lilienthal (Apr 2, 22) 69

Neighbourhood Relations Between Pixels o a pixel has 4 or 8 neighbours in 2D depending on the neighbour definition:» 4-neighborhood each neighbor must share an edge with the pixel» 8- neighborhood each neighbor must share an edge or a corner with the pixel DIP'2 A. J. Lilienthal (Apr 2, 22) 7

Basics of o the pixel value in the output image is calculated from a local neighbourhood in the input image o the local neighbourhood is described by a mask with a typical size of 3x3, 5x5, 7x7, pixels o filtering is performed by moving the mask over the image o the centre pixel in the output image is given a value that depends on the input image and the weights of the mask DIP'2 A. J. Lilienthal (Apr 2, 22) 7

Basics of o filter subimage defines coefficients w(s,t) o used to update pixel at (x,y) DIP'2 A. J. Lilienthal (Apr 2, 22) 72

Linear o filter subimage defines coefficients w(s,t) o response of the filter at point (x,y) is given by a sum of products g ( x, y) = w( s, t) f ( x + s, y + t) s= at= b o also called convolution (convolution kernel) a b (-,-) (,-) (,-) (-,) (,) (,) (-,) (,) (,) DIP'2 A. J. Lilienthal (Apr 2, 22) 73

Linear Implementation o generic code: for P(x,y) in image for F(u,v) in filter Q(x,y) += F(u,v) P(x-u,y-v) end end How to Deal With the Border? o limit excursion of the centre of the mask smaller image o set outside pixel value zero border effects o mirroring border pixel values border effects o modify filter size along the border slower DIP'2 A. J. Lilienthal (Apr 2, 22) 75

Smoothing Spatial Filters (Averaging Filters) o for blurring» removal of small (irrelevant) details, bridging small gaps o for noise reduction» but: edges are also blurred DIP'2 A. J. Lilienthal (Apr 2, 22) 79

Smoothing Spatial Filters (Averaging Filters) o for blurring» removal of small (irrelevant) details, bridging small gaps o for noise reduction Smoothing Spatial Filters 3x3 Mean Filter / Box Filter o need for normalization to conserve the total energy of the image x /9 (sum of all greylevels) o can cause "ringing" o no good model of blurring in a defocused camera» turns a single "point" into a "box" DIP'2 A. J. Lilienthal (Apr 2, 22) 8

Smoothing Spatial Filters Mean Filter original Mean 5x5 Mean x DIP'2 A. J. Lilienthal (Apr 2, 22) 8

Linear in Matlab f = imread('bubbles.tif'); g = imfilter(f, w, filtering_mode, boundary_options, size_options); o filter matrix w o filtering modes» 'corr' or 'conv'» only important in the case of asymmetric filters» 'corr' (no mirroring) is the default DIP'2 A. J. Lilienthal (Apr 2, 22) 82

Linear in Matlab f = imread('bubbles.tif'); g = imfilter(f, w, filtering_mode, boundary_options, size_options); o boundary options» P padding with (default)» 'replicate' replicate values at the outer border» 'symmetric' mirror reflecting across the outer border» 'circular' repeating the image like a periodic function DIP'2 A. J. Lilienthal (Apr 2, 22) 83

Linear in Matlab f = imread('bubbles.tif'); g = imfilter(f, w, filtering_mode, boundary_options, size_options); o size options» 'same' same size as the input image (cropped padded image)» 'full' full size of the padded image» default is 'same' DIP'2 A. J. Lilienthal (Apr 2, 22) 84

Linear in Matlab f = imread('bubbles.tif'); g = imfilter(f, ones(8)/64, 'replicate'); imshow(g); 72 px DIP'2 A. J. Lilienthal (Apr 2, 22) 85

Linear in Matlab f = imread('bubbles.tif'); g = imfilter(f, fspecial('average',32), 'replicate'); imshow(g); DIP'2 A. J. Lilienthal (Apr 2, 22) 86

Smoothing Spatial Filters Mean Filter o square box filter generates defects» axis aligned streaks» blocky results output input example from "A Gentle Introduction to Bilateral Filtering and its Applications", Sylvain Paris, Pierre Kornprobst, Jack Tumblin, and Frédo Durand, SIGGRAPH 28 DIP'2 A. J. Lilienthal (Apr 2, 22) 87

box profile pixel weight pixel position unrelated pixels related pixels unrelated pixels from "A Gentle Introduction to Bilateral Filtering and its Applications", S. Paris, P. Kornprobst, J. Tumblin, and F. Durand, SIGGRAPH 28 DIP'2 A. J. Lilienthal (Apr 2, 22) 88

strategy to solve problems with box filters o use an isotropic (i.e. circular) window o use a window with a smooth falloff box kernel Gaussian kernel from "A Gentle Introduction to Bilateral Filtering and its Applications", S. Paris, P. Kornprobst, J. Tumblin, and F. Durand, SIGGRAPH 28 DIP'2 A. J. Lilienthal (Apr 2, 22) 89

Smoothing Spatial Filters Gaussian Filter o weighted average o 2D Gaussian kernel o higher weight in the centre to decrease blurring Why a Gaussian? o simple model of blurring in optical systems o smooth o also a Gaussian in the frequency domain /6 x 2 2 4 2 2 DIP'2 A. J. Lilienthal (Apr 2, 22) 9

5 input from "A Gentle Introduction to Bilateral Filtering and its Applications", S. Paris, P. Kornprobst, J. Tumblin, and F. Durand, SIGGRAPH 28 DIP'2 A. J. Lilienthal (Apr 2, 22) 9

5 box average input from "A Gentle Introduction to Bilateral Filtering and its Applications", S. Paris, P. Kornprobst, J. Tumblin, and F. Durand, SIGGRAPH 28 DIP'2 A. J. Lilienthal (Apr 2, 22) 92

5 Gaussian box average input blur from "A Gentle Introduction to Bilateral Filtering and its Applications", S. Paris, P. Kornprobst, J. Tumblin, and F. Durand, SIGGRAPH 28 DIP'2 A. J. Lilienthal (Apr 2, 22) 93

Gaussian profile pixel weight pixel position unrelated pixels uncertain pixels related pixels uncertain pixels unrelated pixels from "A Gentle Introduction to Bilateral Filtering and its Applications", S. Paris, P. Kornprobst, J. Tumblin, and F. Durand, SIGGRAPH 28 DIP'2 A. J. Lilienthal (Apr 2, 22) 94

Gaussian profile o spatial parameter σ input small σ large σ limited smoothing strong smoothing from "A Gentle Introduction to Bilateral Filtering and its Applications", S. Paris, P. Kornprobst, J. Tumblin, and F. Durand, SIGGRAPH 28 DIP'2 A. J. Lilienthal (Apr 2, 22) 95

Gaussian profile o spatial parameter σ o how to set σ?» depends on the application» common strategy: proportional to image size e.g. 2% of the image diagonal property: independent of image resolution» depends on image content smooth "object areas" larger σ but don't smooth edges smaller σ from "A Gentle Introduction to Bilateral Filtering and its Applications", S. Paris, P. Kornprobst, J. Tumblin, and F. Durand, SIGGRAPH 28 DIP'2 A. J. Lilienthal (Apr 2, 22) 96

Smoothing Spatial Filters Median Filter o take the values of the input image corresponding to the desired sub-window (3x3, 5x5, ) o sort them o take the middle value (example: 3x3 the 5th largest) o forces pixels with distinct grey levels to be more like their neighbours o very good at reducing salt-and-pepper noise o less blurring than linear filters of the same size DIP'2 A. J. Lilienthal (Apr 2, 22) 97

Smoothing Spatial Filters Median Filter o take the median value over the sub-window X ray image of a circuit board Average 3x3 Median 3x3 DIP'2 A. J. Lilienthal (Apr 2, 22) 98

Median Filter in Matlab f = imread('bubbles.tif'); g = medfilt2(f, [32 32]); imshow(g); original image median 8x8 median 32x32 median 32x2 DIP'2 A. J. Lilienthal (Apr 2, 22) 99

Order Statistics Filters (Fractile Filters) o median o min, max» useful in mathematical morphology o percentile» generalization of median, min, max 3 7 25% percentile 2 4 5 2 2 3 4 5 7 8 8 2 min (%) median (5%) max (%) DIP'2 A. J. Lilienthal (Apr 2, 22)

Sharpening Spatial Filters o highlight fine detail (also noise) o enhance edges o use image differentiation ( st order) f f( x+ ε, y) f( x, y) = lim x ε ε DIP'2 A. J. Lilienthal (Apr 2, 22) 3

Sharpening Spatial Filters o highlight fine detail (also noise) o enhance edges o use image differentiation ( st order) f f( x+ ε, y) f( x, y) = lim x ε ε f f x f i+, j i, j DIP'2 A. J. Lilienthal (Apr 2, 22) 4

Sharpening Spatial Filters o highlight fine detail (also noise) o enhance edges o use image differentiation Sharpening Spatial Filters D o approximation to st order derivation» equivalent to the D convolution mask f x f ( x + ) f ( x) - DIP'2 A. J. Lilienthal (Apr 2, 22) 5

Sharpening Spatial Filters o highlight fine detail (also noise) o enhance edges o use image differentiation Sharpening Spatial Filters D o approximation to st order derivation» equivalent to the D convolution mask f f( x + ) f( x ) x - DIP'2 A. J. Lilienthal (Apr 2, 22) 6

7 DIP'2 A. J. Lilienthal (Apr 2, 22) Gradient and Magnitude of the Gradient Sharpening Spatial Filters Based on the Gradient o Prewitt o Sobel o y f x f y f x f f + + = 2 2 = y f x f f,

Sharpening Spatial Filters o Prewitt gradient edge detector» 2 masks approximate G x and G y in - - - f f f + = G x + G y x y - - DIP'2 A. J. Lilienthal (Apr 2, 22) 9

Sharpening Spatial Filters o Sobel Operators» 2 masks approximate G x and G y in» detects horizontal and vertical edges f f f + = G x + G y x y - 2-2 2 - - -2 - DIP'2 A. J. Lilienthal (Apr 2, 22)

Sharpening Spatial Filters Sobel Operators o weight 2 is supposed to smooth by emphasizing the centre - -2-2 - 2-2 DIP'2 A. J. Lilienthal (Apr 2, 22)

Sharpening Spatial Filters Sobel Operators o detection of vertical dark-light edges - -2-2 DIP'2 A. J. Lilienthal (Apr 2, 22) 2

Sharpening Spatial Filters Sobel Operators o combination of all the directional responses DIP'2 A. J. Lilienthal (Apr 2, 22) 3

Sharpening Spatial Filters o comparison between Sobel and Prewitt operator Sobel (~ G x + G y ) Prewitt (~ G x + G y ) DIP'2 A. J. Lilienthal (Apr 2, 22) 4

Sharpening Spatial Filters o highlight fine detail (also noise) o enhance edges o uses image differentiation Sharpening Spatial Filters D o approximation to st order derivation o approximation to 2 nd order derivation 2 f = f ( x + ) + f ( x ) 2 f ( x) 2 x» equivalent to the D convolution mask -2 DIP'2 A. J. Lilienthal (Apr 2, 22) 6

7 DIP'2 A. J. Lilienthal (Apr 2, 22) Sharpening Spatial Filters Laplace Filter o Laplacian (second order derivative) 2 2 2 2 2 y f x f + = ) ( 2 ) ( ) ( 2 2 x f x f x f x f + + =

Sharpening Spatial Filters Laplace Filter o Laplacian (second order derivative) 2 2 2 2 f f f = + = f ( x + ) + f ( x ) 2 2 2 2 x y x o filter masks to implement the Laplacian» add the "digital implementation" of the two terms in the Laplacian (9 rotation symmetry) f ( x) -4 DIP'2 A. J. Lilienthal (Apr 2, 22) 8

Sharpening Spatial Filters Laplace Filter o detection of edges independent of direction o isotropic with respect to 9 rotations - - 4 - - DIP'2 A. J. Lilienthal (Apr 2, 22) 2

Sharpening Spatial Filters Laplace Filter o Laplace filter + original image sharpening - - 5 - - DIP'2 A. J. Lilienthal (Apr 2, 22) 22

Sharpening Spatial Filters o Laplace filter + original image sharpening DIP'2 A. J. Lilienthal (Apr 2, 22) 23

Sharpening Spatial Filters 2 nd order vs. st order o Laplacian (second order derivative) 2 2 2 2 f f f = + = f ( x + ) + f ( x ) 2 f 2 2 2 x y x o thinner edges o not so strong response to a step o better response to fine details o double response to edges o rotation independent one mask for all edges ( x) DIP'2 A. J. Lilienthal (Apr 2, 22) 24

Sharpening Spatial Filters Unsharp Masking o analog equivalent used in publishing industry o basic idea: subtract blurred version of an image from original image to generate the edges DIP'2 A. J. Lilienthal (Apr 2, 22) 25

Sharpening Spatial Filters in Matlab f = imread('bubbles.tif'); g = imfilter(f, fspecial('laplacian',.5)); g2 = imfilter(f, fspecial('unsharp',.5)); %... DIP'2 A. J. Lilienthal (Apr 2, 22) 26

6 Preparation for Next Class DIP'2 A. J. Lilienthal (Apr 2, 22) 29

6.!!!!!!!!! Preparation for Next Class!!!!!!!!! Pre-Class Reading Schedule o Class "Course Introduction" (Mar 29, 22) o Class 2 "Introduction" (Apr 2, 22)» Gonzalez/Woods Chapter "Introduction"» Gonzalez/Woods Chapter 2 "Fundamentals"» Lecture Notes from last year o Class 3 "" (Apr 2, 22)» Gonzalez/Woods Chapter 3 "Intensity Transformations and "» Lecture Notes from last year o Class 4 "Bilateral Filtering/Fourier Domain" (Apr 7, 22)» "A Gentle Introduction to Bilateral Filtering and its Applications", Sylvain Paris, Pierre Kornprobst, Jack Tumblin, and Frédo Durand, SIGGRAPH 28» "Bilateral Filtering for Gray and Color Images", C. Tomasi, R. Manduchi, Proc. Int. Conf. Computer Vision» Gonzalez/Woods Chapter 4 "Filtering in the Frequency Domain"» Lecture Notes from last year DIP'2 A. J. Lilienthal (Apr 2, 22) 3

Achim J. Lilienthal Mobile Robotics and Olfaction Lab, Room T29, Mo, -2 o'clock AASS, Örebro University (please drop me an email in advance) achim.lilienthal@oru.se 3