Sampling, Frequency downsampled experiment in Python: 0::N matrix of zeros

Transcription

1 Sampling, Frequency We have seen that the color components can be "downsampled" horizontally and vertically by a factor of two. The same happens in the retina: 110 million rods and 6 million cones are fed into only about 1 million nerve fibers. -How do we make this downsampling correct? What can go wrong? To do this, we do an experiment in Python: We are again streaming our video signal, but horizontally and vertically we show only every Nth pixel. In this way we get a downsample by the factor N in both directions. In this example, N = 4. Important in the Python Script videoresampley.py : -Initialize the downsampled frame: [ret, frame] = cap.read() [rows,cols,c]=frame.shape; Ds=np.zeros((rows,cols)); -Creating the downsampled Frame: Ds[0::N,::N]=Y[0::N,::N]; The abbreviation 0::N means: The indices from 0 to the end in steps of N, thus here in the horizontal and vertical direction. So we have a matrix of zeros of the size of

2 our video frame, and then only every Nth pixel is written horizontally and vertically. We can try it out with: python videoresampley.py Note: The video appears to consist of many small points, but the content is still clearly recognizable. Next we will produce a test image, consisting of strips which we create by means of a sine function for the brightness. We produce a sine wave with 100 cycles over 1000 pixels, which is shifted to the positive range and scaled to the range of 0,..., 1: f=100 sinewave=(1.0+np.sin(np.linspace(0,2*np. pi*f,1000)))/2 The addition of 1.0 is to obtain only positive brightness values and to divide by 2 to obtain a maximum brightness value of 1.0, which expects cv2.imwrite at float values. Subsequently, we produce 500 equal lines

3 with this sinusoidal oscillation of 1000 pixels. To do this we apply a matrix "trick": We set the sine values on the diagonal of a 1000x1000 diagonal matrix, which we multiply from the left with a 500x1000 matrix consisting of ones: d=np.diag(sinewave) #Matrix with sinewave from left to right, #identically on each row, with 500 rows: A=np.dot(np.ones((500,1000)),d) This results in a matrix of 500 lines of the sine wave. This matrix is then displayed as a frame and stored in a jpg file (for printing). Try out with: python horizortsfreq.py Note: We get a field with 100 vertical black/white stripes with smooth transitions. Note: Our used sine wave has a frequency, namely these 100 cycles across the width of the lines. In order to distinguish this frequency from time frequencies (in oscillations per second or hertz), we call it "spatial frequency". Units: "cycles per length" (here

4 the width of our lines). Instead of the length, it is often more practical to specify the angle under which our lines appear to the viewer. Then we get the unit "cycles per degree" We only print out this image and use it as a test image by holding the strip image at a medium distance in front of the camera and watching the downsampled image. We see: On the downsampled image, the original stripe pattern now appears, but a new pattern of wave lines and slower light-dark fluctuations, which have no resemblance to the original! We call these new patterns "Moire" or "Aliasing" in digital signal processing. Note: If we hold the strips closer to the camera so that they appear larger (less brightness cycles per degree appear on the camera), the aliasing or moire effects disappear. Example of an image:

5 Image without Moire Image with Moire from: How does our retina perform its type of "downsampling", the reduction to only 1 million nerve fibers? We assume that here the problem of aliasing or Moire was solved. When we look at the retina, we recognize so-called receptive fields of the visual nerve cells. These represent the interconnection of many rods and cones to a central optic nerve. This interconnection is such that it has a so-called lateral inhibition, which is outlined below. (from:

6 We see: light incidence on light cells on the edge of these receptive fields reduce the output of the visual nerve cell! Hence the name "lateral inhibition or inhibition".

7 Further presentation of the receptive fields on the retina: Retina nerve to brain Note: Light in center, in + area, leads to increase of the signal in nerve to brain, light on the edge, in - area, leads to reduction of Signal. Note: The recipe fields overlap.

8 Illustration from the side : example: edge weights spatia l-filter Spatial frequency Center sensor weight negative weights weights Visual cells Weber`s Low Δ w w =konst Visual cells Delta perception proportional perception Compression of value range for nerves. From: Chapter7.ppt Mult. by factor corresponds to constant change in perception.

9 example: Small brightness big brightness ΔA(x) mittlere Aktivität A 0 An edge produces greater values than a constant surface by this function Raising edges detection of objects Output of the Yellen from: H. Weisleder

10 brightness Position in room Actual britghness perceived brightness -> Effect of highpass Filtering position in Room perceived brightness From: 7.ppt The weight distribution of the receptive fields is similar to a so-called Si function (see, for example, Definition: si(x)= sin(x) x An alternative definition is the so called Sinc

11 Function: sinc(x)= sin(π x) π x We plot a Si function in python: ipython --pylab x=linspace(-6.28,6.28,100) plot(x,sin(x)/x) title("si-funktion") It shows that this function has special properties in the so-called Discrete Fourier Transformation, or DFT for short. Discrete Fourier Transform We have seen that in the case of aliasing the number of brightness oscillations per

12 degree has an influence. Therefore, it would be helpful if we could present our image or signal by a sum of different sinusoidal sweeps. This allows us to use this Discrete Fourier Transform. We start from a signal x (n), e.g. The intensity values of a line, or also e.g. The samples of an audio signal having a number (length) of N values in the range 0 n N 1. The DFT of x (n) is defined as: N 1 X (k)= n=0 j 2 π N x(n) e n k (1) X (k) are the resulting N coefficients of the DFT decomposition, with the frequency index k in range 0 k N 1. The inverse DFT takes these N coefficients to form a weighted sum of oscillations representing the original signal x (n): N 1 x (n)= 1 N k=0 X (k) e j 2 π N n k (2) In this case e j ω =cos(ω)+ j sin(ω) (3) and j is the complex unit with the property j 2 = 1.

13 Note: The coefficients are usually complex valued. We see: The inverse DFT describes every signal x(n) as sum of weighted sine and cosine osscilations of different Frequency. There are fast, very efficient algorithms for calculating the DFT and the inverse DFT. These are called "Fast Fourier Transform", in short "FFT". In Python it is included in Package numpy.fft. Example in ipython: ipython pylab In [17]: x=sign(sin(2.0*pi/8*arange(0,8))) In [18]: x Out[18]: array([ 0., 1., 1., 1., 1., -1., -1., -1.]) In [19]: X=fft.fft(x) In [20]: X Out[20]: array([ 1.+0.j, j, 1.+0.j, j, 1.+0.j, j, 1.+0.j, j]) In [21]: ifft(x) Out[21]: array([ e e+00j, e e-16j, e e-16j, e e-16j, e e+00j, e e-16j, e e-16j,

14 e e-16j]) Note: The coefficients X are actually complex. Furthermore, they are symmetrical about the center, but with the opposite sign in the imaginary parts (conjugated complex). In order to see where this symmetry comes from, we plot the sinusoidal osscilations of the above DFT (equations (1), (2), (3)) for the frenquency indices k = 1 and k = 7 in ipython: x1=sin(2.0*pi/8*1*arange(0,8)) plot(x1) x7=sin(2.0*pi/8*7*arange(0,8)) plot(x7) title('sinus Basisfunktionen 1 und 7')

15 We see that these two cycles (also called "basis functions") are indeed identical, except for the opposite sign. Hence the symmetry. The corresponding cosine functions would be identical. Hence the observed symmetry. For this reason, we reach the highest frequency, most oscillations, at an average frequency index k, in our example k = 4. In ipython we take the cosine term, because the sine term disappears here: x=cos(2.0*pi/8*4*arange(0,8))

16 plot(x) title('cos Basisfunktion fuer k=4') Illustration 1: X-axis: location n, y-axis: sample value Python real time example: The audio signal from the microphone is divided into 1024 samples (blocks), the FFT is applied to each of these blocks, and then the amount of the coefficients X(k) is plotted live on the plot. python pyrecfftanimation.py Note: We see again the symmetry around the

17 center. When whistling, we see larger coefficients at the corresponding frequency index k. The highest frequencies appear in the middle. If we represent the magnitude of the DFT coefficients as different colors; On the x-axis the first half of the DFT coefficients so that the coefficients with the lowest frequencies are on the left and those with the highest frequencies on the right, and we apply the time to the y-axis, we get the so-called "Waterfall Animation", Also called" Spectrogram ". So: x-axis: Frequency, y-axis: time Color: value (magnitude) of current DFT coefficients X(k). We can see it through calling: python pyrecspecwaterfall.py Attention: When we whistle a tone, it appears like yellow stripe, horizontally at the corresponding frequency of tone. High tones appear on the right. We can also easily apply the DFT or FFT to images by first applying them to each line of the image, and applying the DFT to each column on the result.

18 The corresponding function in python is fft2. Python live video example: In python we use the function numpy.fft.fft2(..) For calculating the 2D FFT for each frame of the green component of our video stream. To display in the video window, use the amount of FFT coefficients with numpy.abs (..). The script "videorecfftdisp.py" is shown below: #Program to capture a video from the default camera (0), compute the 2D FFT #on the Green component, take the magnitude (phase) and display it live on the screen #Gerald Schuller, Nov import cv2 import numpy as np cap = cv2.videocapture(0) while(true): # Capture frame-by-frame [retval, frame] = cap.read() #compute magnitude of 2D FFT of green component #with suitable normalization for the display: frame=np.abs(np.fft.fft2(frame[:,:,1]/255.0))/512.0 #angle/phase: #frame=(3.14+np.angle(np.fft.fft2(frame[:,:,1]/ #255.0)))/6.28 # Display the resulting frame cv2.imshow('frame',frame) #Keep window open until key 'q' is pressed: if cv2.waitkey(1) & 0xFF == ord('q'): break # When everything done, release the capture cap.release() cv2.destroyallwindows()

19 We start it with: python videorecfftdisp.py and then keep our test image at different distances in front of the camera. We observe: When the test image is held in front of the camera, a series of more or less thick and bright dots are formed, as on a line. The farther we remove the image from the camera (the greater the number of brightness cycles per degree for the camera), the farther away the dots appear. Note: The location of the points is dependent on the spatial frequency that the camera perceives. The highest spatial frequencies appear in the center of the image, with a symmetry around this center. The lowest horizontal spatial frequencies are located on the left and right edges; the lowest vertical spatial frequencies are at the upper and lower edge. Note: Together with the phase of the FFT components, this picture of the amounts still contains the entire information of the original video (since the FFT is invertible).

20 Avoidance of Aliasing/Moire We saw in our experiment that Aliasing/Moire occurred only at many brightness cycles per degree, ie at high spatial frequencies. We could avoid it if we would set these high spatial frequencies in the image to zero before we downsample the image. Since the DFT is invertible, we can achieve this by setting the corresponding DFT coefficients of the high spatial frequencies to zero and then using the IDFT to produce the image or video again. We do this with a Mask that we multiply to the 2D DFT of our image / video. Let us assume that 1 is the highest frequency, and we only want to keep the DFT coefficients of the 1/4 lowest frequencies. One line has e.g. 640 pixels (and thus the DFT 640 coefficients per line). Because of the symmetry around the center, the highest frequency is in the center, and we get 2 halves with the same content. Therefore we have to keep only one-eighth of the total coefficients at each end, and set the remainder to zero. A corresponding mask for 1 dimension would be the following: ipython --pylab M=ones((640)) M[(640.0/8):( /8)]=zeros((3.0/4.0*640)) plot(m) axis([0, 640, -0.1, 1.1]) title("maske fuer tiefe Frequenzen")

21 Since this mask indicates which frequencies are transferred as strongly, it is also called the "Transfer Function". Illustration 2: X-axis: frequency index k, y-axis: value of the mask

22 For 2 dimensions, as in our case for pictures, we apply the mask simply for each dimension. If r is the number of rows and c is the number of the columns, we take accordingly in python: #For rows: Mr=np.ones((r,1)) Mr[(r/8.0):(rr/8.0),0]=np.zeros((3.0/4.0*r)) #For columns: Mc=np.ones((1,c)) Mc[0,(c/8.0):(c-c/8)]=np.zeros((3.0/4.0*c)); #Together: M=np.dot(Mr,Mc) plt.plot(m) Illustration 3: X- and y-axis: frequency index k, color value: value of the mask

23 If we multiply this mask (transfer function) with our DFT coefficients (element-wise), all the coefficients are set to zero at high spatial frequencies. From the property that this mask is a "Filter" which only "passes" the lower frequencies, it is also called "Lowpass Filter". We create the 2D FFT and set the mask to zero: X=np.fft.fft2(frame[:,:,1]/255.0) #Set to zero the 3/4 highest spacial frequencies in each direction: X=X*M If we then transform these new coefficients by means of the inverse DFT, we get an image / video without these high spatial frequencies: x=np.abs(np.fft.ifft2(x)) Note: Here, we use the command "abs" to get again real numbers, because the inverse FFT produces complex numbers, although here the imaginary parts become zero to accuracy. The following Python script shows this process: python videorecfft0ifftdisp.py Note: The reconstructed image / video after the inverse FFT contains, in fact, less high spatial

24 frequencies, and therefore looks more unclear. We know from "Signals and Systems": A multiplication in the Fourier domain corresponds to a so-called convolution of the corresponding signals in the spatial or time domain. Since we have here finite signals whose length we call N, we obtain a socalled circular convolution. The filter (or the mask) as a signal in spatial domain (by inverse FFT) is also called "Impulse Response" of the filter. Suppose that x(n) is our signal in the spatial domain, and h(n) is our filter in the spatial domain (the impulse response), then the circular convolution: N 1 y(n)=x(n) N h(n):= i=0 x(i) h((n i)mod N) where "mod" is the "modulus" function, the remainder after integer division of (n-i) by N (on N-1 again follows 0), i. This convolution sum runs in a circle". The h(n) is obtained by the inverse FFT of our mask M (here in the example for better visibility for the 1/8 lowest frequencies). In ipython: ipython --pylab M=ones((640)) M[(640.0/16):( /16)]=zeros((7.0/8.0*640))

25 h=fft.ifft(m) plot(h) Illustration 4: x-achse: Orts-Index n, y-achse: Abtastwert Due to the circular convolution, the signal (the impulse response) runs practically "in a circle", so we can also consider a version cyclically shifted by 320 values: hc=concatenate((h[320:640],h[0:320])) plot(hc) title("zyklisch verschobene Inv FFT unserer Tiefpass Maske")

26 Note: This is now like the Sinc or Si-function with which the weights of the retinal receptors had similarity. If, after this low-pass filtering, we perform the downsampling by the factor N, this means that we only take every Nth pixel or Nth value. This corresponds to the replacement of our index n in the convolution equation by mn, where m is an integer: N 1 y(mn)= i=0 x (i) h((mn i)mod N) This sum shows us: for each assumed value y (mn),

27 we form a sum over the shifted impulse response of the filter at the position mn: h((mn i)mod N ) Example in ipython: We take our already calculated impulse response hc and set N = 8. Then we get for 2 adjacent positions m and m + 1, for a section with the approximately 40 largest values: plot(hc[300:340]) plot(hc[308:348]) title("2 benachbarte Impulsantworten fuer Abtastfaktor N=8" ) Illustration 5: x-achse: Orts-Index n, y-achse: Abtastwert Note: We see an overlapping of the neighboring

28 impulse responses. This corresponds in the 2- dimensional and on the retina to the overlapping of the receptive fields! Also we can see the inverse FFT of our 2D mask M in ipython: ipython pylab r=480 c=640 #For rows: Mr=ones((r,1)) Mr[(r/8.0):(r-r/8.0),0]=zeros((3.0/4.0*r)) #For columns: Mc=ones((1,c)) Mc[0,(c/8.0):(c-c/8)]=zeros((3.0/4.0*c)); #Together: M=dot(Mr,Mc) h=fft.ifft2(m) #Rotieren: hc=concatenate((h[:,320:640],h[:,0:320]),axi s=1) hc2=concatenate((hc[240:480,:],hc[0:240,:]), axis=0) imshow(real(hc2[230:250,310:330])/0.07) title('ausschnitt der Impulsantwort unseres 2D Tiefpass Filters')

29 Illustration 6: x- und y-achse: Orts-Indizes, y-achse: Farbwert entspricht Aptastwert Deep blue are the smallest (negative) values, red the largest (positive) values. Note: This false colors illustration of inverse FFT resp. 2D-impulse response now also resembles to the observed receptive fields in retina. Python example: The Library scipy.signal contains function for 2D convolution: scipy.signal.convolve2d(x,h)

30 convolves two 2-dimensional signals x and h, where x is e.g. our frame and h is our filter impulse response. We use these in the script videofiltdisp.py. We calculate the impulse response h as an inverse FFT of our transfer function M. We must limit the size of the 2D impulse response drastically, so that a real-time processing is possible. For FFT filters with a large impulse response, the calculation using FFT is much more efficient than the convolution. We run the script with: python videofiltdisp.py Note: The filtered image / video looks similar to the implementation using FFT. Thus we see that the receptive fields on the retina correspond to a filtering of the image, and their distance corresponds to the downsampling for the smaller number of optic nerves. The "transfer function" of the human eye can also be measured experimentally by visual tests. This results in the so-called "Contrast Sensitivity Function" (CSF). example:

31 Basicbrightness Max. for bright light: 2 period/degree (cycles=periods) Max. for dark light: 0.7 per./degree fine details of images: eye is les sensitive Spatial frequency (periods/degree) degree: Independent from viewing distance bright-dark oscillation (From: We see:the Transfer function is not exactly a low pass filter, but rather a kind of low-pass with an increase of center frequencies. Our eye is most sensitive to these center spatial frequencies. By means of refined tests the CSF can also be determined from the eyes of other living creatures:

32 Comparison of Contrast Sensitivity Function of different creatures from: Harmening, Vossen, Wagner, van der Willigen The disparity sesitivity function compared new insights from barn owl vision, EVCP09

33 The different number of rods and cones on the retina is noticeable by different CSFs for color and brightness: Differences between maximums correspond appr. differences of density of rods(brightness) and cones (color):

34 Reconstruction of image in Decoder We can now transfer the downsampled image to the decoder with fewer pixels and thus a small bit rate. How do we make a "impressive" picture? We have seen that if we just put the transferred pixels in the right place, leaving the other pixels at zero value, we get a picture which consists of many small points. It is interesting to look at the spectrum of this point image. This is done with the following python script. Here are the following steps: Frame-low pass filter-sampling (Remove zeroestransfer set zeroes) Low pass filter- Reconstructed Frame. We start with: python videofft0ifftresampleykey.py Note: The video in "Decoder: FFT range of Sampled Frames" shows: Unlike the original picture, periodic sequels of the spectrum arise in the x- and y- direction. Since these were not present in the original, and thus they represent artifacts (namely, our aliasing), we should simply set them to zero. This works again with our mask or the low-pass filter M, which we apply again in the FFT range.

35 To this end, we look at the outputs of our Python script. Note: The image reconstructed in the decoder in the window "Decoder: Filtered Sampled Frames" actually looks almost like our encoder in the lowpass filtered image! Summary: We are able to reduce the number of pixels to be extracted by the factor N * N, with the same image size, but at the expense of the image quality. This represents a further compression capability. The art is now to choose the number of transmitted pixels (the resolution) so that the eye does not perceive the uncertainties. Processing-Sequence: -Encoder: Filtering- downsampling- Removing zeros -Decoder: Filling zeros- Filtering References: Circular convolution, zyklische Faltung, sh. S. 542, Spectrogram: S. 742: A. Openheim, R. Schafer: Discrete Time Signal Processing, Prentice Hall.

36 Quantiziation In a Video-Coder we must quantize our values, for which we need to know how many intensity steps of the human eye can distinguish: -ca. 20 brightness changes in a small area in complex perceptible image. -ca. 100 grayscale levels required (7 bit/pixel) to avoid artificial contours. From :