ECE 484 Digital Image Processing Lec 09 - Image Resampling Zhu Li Dept of CSEE, UMKC Office: FH560E, Email: lizhu@umkc.edu, Ph: x 2346. http://l.web.umkc.edu/lizhu slides created with WPS Office Linux and EqualX equation editor Z. Li, Digital Image Processing, 2018 p.1
Outline Recap of Lec 08 Image Sampling subsampling interpolation Summary Z. Li, Digital Image Processing, 2018 p.2
Fourier Series Any periodic contiunous signals can be expressed as linear combinations of sinusoids While coefficients are: Z. Li, Digital Image Processing, 2018 p.3
Fourier Transform Any energy bounded continuous function, can be expressed in its frequency domain representation as And its inverse transform from freq domain is given as, Z. Li, Digital Image Processing, 2018 p.4
DFT Discrete Fourier Transform real(a) Forward transform imag(a) u=0 Inverse transform basis u=7 n Z. Li, Digital Image Processing, 2018 n p.5
DFT vs. DCT 1D-DCT 1D-DFT a real(a) u=0 u=0 u=7 u=7 Z. Li, Digital Image Processing, 2018 imag(a) n=7 p.6
DFT spectrum DFT domain representation and processing real-valued input Z. Li, Digital Image Processing, 2018 p.7
2-D Fourier basis Fourier basis: low freq is in the middle real imag real( ) imag( ) Z. Li, Digital Image Processing, 2018 p.8
LP, HP and BP in Freq Domain Filtering in frequency domain BP filter to remove repetitive noises LP filter and smoothing Z. Li, Digital Image Processing, 2018 p.9
Outline Recap of Lec 08 Image Sampling subsampling interpolation Summary Z. Li, Digital Image Processing, 2018 p.10
Image Scaling Need to resample images This image is too big to fit on the screen. How can we reduce it? How to generate a halfsized version? Z. Li, Digital Image Processing, 2018 p.11
Image sub-sampling sub sample without filtering, what is wrong? 1/8 1/4 Throw away every other row and column to create a 1/2 size image - called image sub-sampling Z. Li, Digital Image Processing, 2018 p.12
Image sub-sampling Aliasing... 1/2 Why does this look so crufty? 1/4 (2x zoom) 1/8 (4x zoom) Z. Li, Digital Image Processing, 2018 p.13
Even worse for synthetic images Aliasing effect Z. Li, Digital Image Processing, 2018 p.14
Sampling and the Nyquist rate Aliasing can arise when you sample a continuous signal or image occurs when your sampling rate is not high enough to capture the amount of detail in your image Can give you the wrong signal/image an alias formally, the image contains structure at different scales called frequencies in the Fourier domain the sampling rate must be high enough to capture the highest frequency in the image To avoid aliasing: sampling rate > 2 * max frequency in the image This minimum sampling rate is called the Nyquist rate Z. Li, Digital Image Processing, 2018 p.15
Sampling in Time Domain Computer needs a discrete representation of signals Many signals originate as continuous-time signals, e.g. conventional music or voice By sampling a continuous-time signal at isolated, equally-spaced points in time, we obtain a sequence of numbers s [ n] = s( ) n T s n {, -2, -1, 0, 1, 2, } T s is the sampling period. s ( t) = s( t) d ( t - n ) å sampled T s n= - impulse train s sampled Sampled analog waveform Z. Li, Digital Image Processing, 2018 p.16 T s ( t) T s t s(t)
Spectrum of several special functions Spatial-Frequency signatures Z. Li, Digital Image Processing, 2018 p.17
Consequence in Freq Domain Multiplication with sampling train function, is convolving in freq domain Replicates spectrum of continuous-time signal At offsets that are integer multiples of sampling frequency Fourier series of impulse train where s = 2 f s 1 2 2 ( ) = å dt s t d ( t - n Ts ) = + cos( s t) + cos(2 s t) +... n= - Ts Ts Ts 1 g( t) = f ( t) dt s ( t) = f ( t) + 2 f ( t)cos( s t) + 2 f ( t)cos(2 s t) T Example F( ) s ( +... ) Modulation by cos( s t) G( ) Modulation by cos(2 s t) -2 f max 2 f max - s - s s s gap if and only if 2 f max < 2 f s - 2 f max Û fs > 2 f max Z. Li, Digital Image Processing, 2018 p.18
Shannon/Nyquist Sampling Theorem What is the sampling rate to recover the original continuous signal w/o loss? A continuous-time signal x(t) with frequencies no higher than f max can be reconstructed from its samples x[n] = x(n T s ) if the samples are taken at a rate f s which is greater than 2 f max. Nyquist rate = 2 f max Nyquist frequency = f s /2. What happens if f s = 2f max? Consider a sinusoid sin(2 p f max t) Use a sampling period of T s = 1/f s = 1/2f max. Sketch: sinusoid with zeros at t = 0, 1/2f max, 1/f max, Z. Li, Digital Image Processing, 2018 p.19
Illustration of Sampling Theorem 1-D example sampling pattern w 1/w sampled signal Spatial domain Frequency domain Z. Li, Digital Image Processing, 2018 p.20
Reconstruction Recon via convolving with Sinc function (low pass filtering) sinc function w 1/w reconstructed signal Spatial domain Frequency domain Z. Li, Digital Image Processing, 2018 p.21
Aliasing from under-sampled reconstruction Aliasing effect (a) f(x) and its spectrum (b) sampling train (c) aliasing from under sampling (d) reconstruction via convolving with sinc function (e) aliasing (a) (b) (c) (d) Z. Li, Digital Image Processing, 2018 p.22 (e)
Anti-Aliasing Filtering Apply low pass filtering to limit the band of signal before filtering ideally need to use sinc filter infinite taps, not practical use Gaussian instead or LP FIR Z. Li, Digital Image Processing, 2018 p.23
Image Anti-Aliasing Filtering Images are not bandlimited as we would like 2D sampling effects Z. Li, Digital Image Processing, 2018 p.24
2D image anti aliasing example Pre-Low Pass Filter before subsampling Z. Li, Digital Image Processing, 2018 p.25
Subsampling with Gaussian pre-filtering Pre LP filtering with Gaussian G 1/4 G 1/8 Gaussian 1/2 Solution: filter the image, then subsample Filter size should double for each ½ size reduction. Why? Z. Li, Digital Image Processing, 2018 p.26
Subsampling with Gaussian pre-filtering Gaussian 1/2 G 1/4 G 1/8 Solution: filter the image, then subsample Filter size should double for each ½ size reduction. Why? How can we speed this up? Z. Li, Digital Image Processing, 2018 p.27
Aliasing w/o pre filtering 1/2 1/4 (2x zoom) 1/8 (4x zoom) Z. Li, Digital Image Processing, 2018 p.28
Outline Recap of Lec 08 Image Sampling subsampling interpolation Summary Z. Li, Digital Image Processing, 2018 p.29
Image Resampling/Interpolation Reconstruct the continuous signal with different filters Z. Li, Digital Image Processing, 2018 p.30
0-th and 1-st Order Interpolation Simple interpolation schemes 0-th order: introduced discontinuity 1st order: not differentiable Z. Li, Digital Image Processing, 2018 p.31
Linear Image resampling/interpolation So what to do if we don t know Answer: guess an approximation Can be done in a principled way: filtering 1 d = 1 in this example 1 2 2.5 3 4 5 Image reconstruction Convert to a continuous function Reconstruct by convolution: Z. Li, Digital Image Processing, 2018 p.32
Cubic Interpolation Linear, Quadratic and Cubic Interpolation Cubic is popular, supported in Matlab x = -3:3; y = [-1-1 -1 0 1 1 1]; xq1 = -3:.01:3; s = spline(x,y,xq1); plot(x,y,'o',xq1,p,'-',xq1,s,'-.') legend('sample Points','spline','Location','SouthEast') Z. Li, Digital Image Processing, 2018 p.33
2D Interpolation Divide & conquer 1/2 pel position interpolation Z. Li, Digital Image Processing, 2018 p.34
Bilinear Interpolation 2D linear interpolation Z. Li, Digital Image Processing, 2018 p.35
Interpolation Results NN, Bilinear vs Bicubic Original image: x 10 Nearest-neighbor interpolation Bilinear interpolation Bicubic interpolation Z. Li, Digital Image Processing, 2018 p.36
DCTIF Interpolation Filter DCTIF interpolation filters 1/4 pel interpolations row first then column Z. Li, Digital Image Processing, 2018 p.37
DCTIF Filter Filter derived from DCT transforms row interpolation column interpolation Performance better than bicubicle! Z. Li, Digital Image Processing, 2018 p.38
Deep Learning Interpolation Filter High performing solution in HEVC for sub-pel motion estimation half pixel location images: Deep learning ground truth: Network: Z. Li, Digital Image Processing, 2018 p.39
Summary Downsampling images: downsampling is multiplication with the impulse train in time domain it is convolution with signal spectrum in freq domain if not sampling at 2fmax, aliasing. Interpolation Ideal reconstruction via Sinc function, infinite support not practical Other form of interpolation filters, best performing for images, DCTIF Deep learning solution for interpolation: CNN directly learn fractional pel location images. Z. Li, Digital Image Processing, 2018 p.40