ELEC Dr Reji Mathew Electrical Engineering UNSW

Transcription

1 ELEC 4622 Dr Reji Mathew Electrical Engineering UNSW

2 Multi-Resolution Processing Gaussian Pyramid Starting with an image x[n], which we will also label x 0 [n], Construct a sequence of progressively lower resolution images x k [n], k = 0, 1, 2,..., K Where each lower resolution image is obtained by: low-pass filtering & decimating previous higher resolution image h[n] is the impulse response of a relevant low-pass filter and the resolution is reduced by a factor of 2 in each direction

3 Multi-Resolution Processing Gaussian Pyramid The low pass filter h[n] can be a Gaussian Hence the name Gaussian Pyramid However we are not restricted to Gaussian low pass filters Remember properties of Gaussian filters Frequency response and appropriate choice of Sigma (variance)

4 Gaussian Pyramid Multi-Resolution Processing

5 Gaussian Pyramid Multi-Resolution Processing The transform is obviously redundant Image is perfectly represented by the first level alone, x 0 [n] Pyramid requires more samples than the original input image Usefulness of the Gaussian pyramid arises in computationally intensive tasks usually involving image matching Searching for inter image displacement due to disparity Motion estimation between frames of a video Image fusion applications Image registration

6 Multi-Resolution Processing Gaussian Pyramid Coarse to fine search strategy using the pyramid representation Performing the search first in the lowest resolution image, x K [n] Low computation cost at the coarsest level Propagate the result up the pyramid to each successively higher resolution image, Refine the precision of the search results to take advantage of the higher resolution information refinement is typically conducted within a small search window

7 Multi-Resolution Processing k=0 k=1 k=2 Gaussian Pyramid Laplacian Pyramid

8 Laplacian Pyramid Multi-Resolution Processing Consists of high-pass filtered versions of the image Laplacian operator may be understood as a high-pass filter Starting from the Gaussian pyramid images, x k [n], k = 0, 1,, K, Laplacian pyramid is constructed by: interpolating lower resolution sub-image x k [n], k 1, to obtain a predictor x k 1 [n], for the next higher resolution image, x k 1 [n] How to go about obtaining the predictor x k 1 n? Next.

9 Laplacian Pyramid Multi-Resolution Processing g(s) is an interpolation kernel Convolution operation Choose filters that are separable Examples: Ideal bandlimited sinc interpolation Linear interpolation Interpolation by a factor of 2 Need new interpolated values at odd pixel locations Even locations - equal to pixel values from lower resolution

10 Laplacian Pyramid Multi-Resolution Processing g(s): interpolation kernel. y[n]: interpolated output (x2) Even locations: just copy from pixel values from lower resolution image. Odd locations: Interpolate values. Convolution of low resolution image with interpolation kernel.

11 Laplacian Pyramid Multi-Resolution Processing Convolution of x[n] with Linear interpolation example:

12 Laplacian Pyramid Multi-Resolution Processing Linear interpolation example: Draw on board linear interpolation example 1 PSF for liner interpolation Inner product with input image

13 Laplacian Pyramid Multi-Resolution Processing Having got a prediction for the higher resolution image Prediction based on the lower Resolution image. Interpolation kernel We can go about constructing the Laplacian pyramid.

14 Laplacian Pyramid Multi-Resolution Processing We start with the Gaussian pyramid images And interpolate each lower resolution image, x k [n], k 1, by a factor of 2 in each direction to form a prediction x k 1 [n], of higher resolution image x k 1 [n] Final step is to replace all but the lowest resolution image x K [n], by the difference between x k [n], and its prediction x k 1 [n], We write x k 1[n], for this difference image

15 Multi-Resolution Processing For notational consistency, we write x K[n] = x k [n], (lowest level K) Original image x[n], represented by: Laplacian pyramid images, x k[n], k = 0, 1,..., K. Laplacian pyramid consists of difference images; apart from the lowest resolution image, Difference images contain details required to augment a lower resolution image to the next resolution level

16 Formation of the Laplacian pyramid representation of an image from its Gaussian pyramid representation

17 Recovery of the Gaussian pyramid from the Laplacian pyramid

18 Multi-Resolution Processing The Laplacian pyramid is a redundant image representation, Since it represents the image using the same number of samples as the Gaussian pyramid which is obviously redundant

19 Multi-Resolution Processing After Image registration need to stitch together multiple images Example panoramic view construction stitching one large image from smaller images Need to avoid blurring however achieve a seamless transition An elegant solution to the stitching problem is to perform the stitching in the Laplacian pyramid representation

20 Multi-Resolution Processing Distinct boundary between two images Left of the boundary we take pixels from image A and on the right of the boundary we take pixels from image B How to avoid discontinuity at the selected boundary, due to differences between the two images variations in illumination Camera automatic gain control misalignment of edges due to imperfect registration Solution: perform stitching in the Laplacian pyramid representation

21

22 Multi-Resolution Processing

23 Multi-Resolution Processing Most smoothing is applied to the boundary transition artefacts introduced in the lowest resolution component Successively less smoothing is applied to the boundary transition artefacts introduced by successively higher frequency components

24 Automatic panoramic image stitching using invariant features M Brown, DG Lowe - International journal of computer vision, 2007

25 Automatic panoramic image stitching using invariant features M Brown, DG Lowe - International journal of computer vision, 2007

26 Mathematical Morphology Mathematical tools for the manipulation of shape Morph meaning or equivalent to shape Will consider bi-level images first Remember that we are interested in LSI systems Bi-level images linearity is meaningless Grey-level images linearity is inherently inconsistent with the concept of processing shape. Shift Invariance remains important We will consider shift invariant operators

27 Bi-Level Image Morphology Notation used for images so far Where the function takes on values only in the set {0, 1} Notation used for Sets Set of all points for which the image function evaluates to 1

28 Notation used for Sets Bi-Level Image Morphology Image is represented by the set Where A contains all locations for which the image function = 1 f is referred to as the indicator function of the Set A Can think of A as the set of foreground pixel locations

29 Notation used for Sets Bi-Level Image Morphology Translation A h = {n + h n A} Complement A c Flipping A A = { n n A} All the elements of A have been shifted by h Bi-level images exchanges the roles of foreground and background pixels Mirror image of A; foreground pixels are flipped about the origin A is symmetric if A=A

30 Erosion X A

31 Erosion X A When A is centred about the origin, we can think of erosion as a process of moving A all over the image and keeping only those pixels which lie at the centre of shifted structuring elements which fit inside the image

32 Erosion Erosion affects only the boundaries of objects in the image, causing those boundaries to shrink. If A is not centred about the origin, then an overall shift will also be involved.

33 Dilation X A

34 Dilation X A

35 Common structuring elements Erosion and Dilation

36 Erosion and Dilation Properties of Erosion and Dilation Translation Invariance Increasing Increasing set operator (erosion or dilation)

37 Erosion and Dilation Properties of Erosion and Dilation Duality: Erosion and dilation are dual operators. Compliment: foreground/background switch Reflection of A about the origin 0 Dilating the foreground by A is the same as eroding the background by the mirror image of A

38 Erosion and Dilation Properties of Erosion and Dilation Two successive erosions are equivalent to a single erosion by the dilation of the two structuring sets

39 Erosion and Dilation Properties of Erosion and Dilation Two successive dilations are equivalent to a single dilation by the dilation of the two structuring sets

40 Erosion and Dilation Properties of Erosion and Dilation Symmetry of dilation Only for dilation NOT true for erosion

41 Erosion and Dilation Properties of Erosion and Dilation Commutativity Any two erosion operators commute with each other Similarly, any two dilation operators commute with each other

42 Opening and Closing Opening & Closing Operators constructed from erosion and dilation primitives Opening Opening of an image X, by a structuring element A First erode X using the structuring element A, and then dilate the result using the same structuring element

43 Opening and Closing Opening Union of all shifted copies of the structuring set A, which entirely fit inside the image X Opening X by any structuring element makes X smaller

44 Important properties of Opening Opening and Closing Translation invariance dilation and erosion are also translation invariant Idempotence Opening twice in succession - the result is the same as that obtained from a single application So repeated applications of opening operator has NO impact Remember: opening is a union of all shifted copies of the structuring element which fit inside X

45 Important properties of Opening Translation invariance Opening and Closing Idempotence Increasing Remember that erosion and dilation are both increasing operators

46 Closing Opening and Closing Closing of an image X, by a structuring element A First dilate X using the structuring element, A, and then erode the result using the same structuring element

47 Opening and Closing Closing Opening can be implemented using closing And similarly, closing can be expressed in terms of opening Duals with respect to set complementation and reflection Closing X by any structuring element makes X larger

48 Important properties of Closing Opening and Closing Translation invariance dilation and opening are also translation invariant Idempotence second application of operator to the image has no further effect Increasing Erosion and dilation are both increasing operators Important property for morphological filters

49 Alternating Sequential Filters Morphological filter: a morphological operator which is idempotent Filter removes or filters out unwanted parts of the image For example filter out noise What is a Morphological filter removing? Idempotence property: Means that once the undesired parts of the image have been removed, the filter will have no further impact upon the image

50 Alternating Sequential Filters Alternating sequential filter One structuring element Opening: removes noise from background Closing removes noise from foreground Shape of the image changed High curvature regions smoothed. What happens if we repeat the opening and closing with the same structuring element?

51 Alternating Sequential Filters

52 Filtering Curvature Filtering out curvature using Opening and Closing For discussions we consider spatially continuous domain X now represents all locations in the spatially continuous image which belong to foreground regions Can consider X as being objects or shapes Definitions of erosion, dilation, opening & closing are unchanged

53 Filtering Curvature

54 Filtering Curvature Euclidean ball of radius r is the smallest object whose boundaries all have curvature of at most 1 r Curvature of a circular boundary = 1 r Small circles are thus highly curved Curvature at any point on an arbitrary boundary circle which exhibits the same first and second derivatives as the boundary, at the point in question

55 Filtering Curvature Inward curvature: Centre of the circle lies to the side of the boundary that we associated with the foreground region Outward curvature: Centre of the circle lies to the side of the boundary that we associate with the background region

56 Filtering Curvature

57 Filtering Curvature Opening X with B(r) yields the largest image Y such that Y is contained in X and no boundary in Y has inward curvature greater than 1 r. carves away the smallest possible amount of the image Closing X with B(r) yields the smallest image Y such that Y contains X and no boundary in Y has outward curvature greater than 1 r. fills in the smallest possible amount of background

58 Filtering Curvature

59 Morphology How do we extend morphological operators to grey-scale? How to manipulate shapes in grey-scale images? Work with level sets..