DESIGN OF GUIDED FILTER FOR IMAGE FILTERING

Transcription

1 DESIGN OF GUIDED FILTER FOR IMAGE FILTERING A PROJECT REPORT Submitted by ANITHA.P Register No: 14MAE001 in partial fulfillment for the requirement of award of the degree of MASTER OF ENGINEERING in APPLIED ELECTRONICS Department of Electronics and Communication Engineering KUMARAGURU COLLEGE OF TECHNOLOGY (An autonomous institution affiliated to Anna University, Chennai) COIMBATORE ANNA UNIVERSITY: CHENNAI APRIL i

2 BONAFIDE CERTIFICATE Certified that this project report titled Design of Guided Filter for Image Filtering is the bonafide work of ANITHA.P (Reg. No. 14MAE001) who carried out the project under my supervision. Certified further, that to the best of my knowledge the work reported here in does not form part of any other project or dissertation on the basis of which a degree or award was conferred on an earlier occasion on this or any other candidate. SIGNATURE Dr.G.AMIRTHA GOWRI PROJECT SUPERVISOR Associate Professor Department of ECE Kumaraguru College of Technology SIGNATURE Dr.A.VASUKI PROFESSOR AND HEAD Department of ECE Kumaraguru College of Technology Coimbatore Coimbatore The candidate with Register No. 14MAE001 was examined by us in the project viva-voce examination held on.. INTERNAL EXAMINER EXTERNAL EXAMINER ii

3 ACKNOWLEDGEMENT First, I would like to express my praise and gratitude to the Lord, who has showered his grace and blessings enabling me to complete this project in an excellent manner. I express my sincere thanks to the management of Kumaraguru College of Technology and Joint Correspondent Shri Shankar Vanavarayar for his kind support and for providing necessary facilities to carry out the work. I would like to express my sincere thanks to our beloved Principal Dr.R.S.Kumar M.E., Ph.D., Kumaraguru College of Technology, who encouraged me with his valuable thoughts. I would like to thank Dr.A.Vasuki M.E., Ph.D., Head of the Department, Electronics and Communication Engineering, for her kind support and for providing necessary facilities to carry out the project work. In particular, I wish to thank with everlasting gratitude to the project coordinator Ms.S.Umamaheswari M.E.,(Ph.D) Associate Professor, Department of Electronics and Communication Engineering, throughout the course of this project work. I am greatly privileged to express my heartfelt thanks to my project guide Dr.G.Amirtha Gowri M.E., Ph.D., Associate Professor, Department of Electronics and communication Engineering, who encouraged me in each and every step of the project work and I wish to convey my deep sense of gratitude to all teaching and nonteaching staff of ECE Department for their help and cooperation. Finally, I thank my parents and my family members for giving me the moral support and abundant blessings in all of my activities and my dear friends who helped me to endure my difficult times with their unfailing support and warm wishes. iii

4 ABSTRACT Image processing plays a major role in many applications like medical, satellite communication, multimedia, etc. The images taken in real time will have additional disturbance like noise. So, it is necessary to filter these parameters. The filters used for this purpose are sobel filter, bilateral filter, joint bilateral filter, etc. Among these filters, guided filter plays a major role due to its better performance around the edges. Guided image filtering is used to smooth the result of transferred colors. Guided filter has the edge-preserving smoothing property but does not suffer from the gradient reversal artifacts near the edges. A reformation of guided filter formula is used to prevent the error resulted from the truncation. The guided filter has an O(N) time complexity (in the number of pixels) exact algorithm for both gray-scale and color images. Guided filter has the nonapproximation characteristic and offers an ideal option for real-time filter applications. Guided filter can be embedded in mobile devices. Guided filter has better performance near the edges than bilateral filter. Guided filter is applied to the different images to remove the disturbances and obtained results are compared with the results of the bilateral filter and the performance of guided filter is analyzed for noise removal. iv

5 TABLE OF CONTENTS CHAPTER NO. TITLE ABSTRACT LIST OF FIGURES LIST OF NOMENCLATURE LIST OF TABLES PAGE NO. iv vi Viii x 1 INTRODUCTION DEFINITION OF A 1 DIGITAL IMAGE 1.2 CHARACTERISTICS OF 1 IAMGE 1.3 IMAGE DENOISING GENERAL 2 CLASSIFICATION OF IMAGE FILTERING TECHNIQUES 1.4.1Spatial domain filtering Linear Filters Non-Linear Filters Transform domain filtering Data adaptive transform Non-data adaptive transform LITERATURE SURVEY 6 v

6 3 DENOISING USING GUIDED FILTER BILATERAL FILTERING Functional Units of Bilateral Filter Merits of bilateral filter Limitations of bilateral filter Applications of bilateral filter GUIDED FILTER Algorithm Merits of Guided filter 3.2.3Applications of guided filter Methodology RESULTS AND DISCUSSION 4.1 INTRODUCTION 4.2PERFORMANCE METRICS 4.3SIMULATION RESULTS 4.3.1Results of Guided filter Results of Bilateral filter SCHEMATIC OF GUIDED FILTER CONCLUSION 36 REFERENCES 37 vi

7 LIST OF FIGURES FIGURE NO. NAME OF FIGURE PAGE NO. 1.1 Basic model of image donoising Illustration of bilateral filter Example for Guided Filter Work flow Results of Guided Filter Results of Bilateral Filter Simulink Model of Guided Filter Simulink Model for calculating mean to Guided Image 4.19 Simulink Model for calculating mean to Input Image 4.20 Simulink Model for calculating mean to correlation of Input and guided image 4.21 Simulink model for calculating mean to correlation of Guided image and Guided image 4.22 Simulink model for calculating co-efficient 31 a and b 4.23 Simulink Model for viewing filtered output 32 image 4.24 RTL Schematic for subsystem 33 vii

8 4.25 Guided Image Input image Output image 35 viii

9 LIST OF NOMENCLATURE BF Bilateral filtering DSP Digital Signal Processors CCD Charge Coupled Device CWM Central Weighted Median SFR Special Function Register BF Bilateral Filter RGB Red Green Blue PSNR SDK FPGA MSE Peak Signal to Noise Ratio Software Development Kit Field Programmable Gate Array Mean Square Error ix

10 LIST OF TABLES TABLE NO. CAPTION PAGE NO. 5.1 Parameters Used Comparison of Performance Metrics of Guided Filter and Bilateral Filter 28 x

11 CHAPTER 1 INTRODUCTION The main aim of this project is to design the guided filter for image filtering. Filtering is widely used in many applications. Image processing is one the main application of image filtering. More specifically, filtering can be applied in other applications such as noise reduction, texture editing, detail smoothing/enhancement, colorization, relighting tone mapping, haze/rain removal, and joint upsampling. The Guided filter has the good edge preservation quality and the noise removal property. The most popular technique is the edge-preserving bilateral filter for image filtering and image noise reduction. Bilateral filter can be used on high dynamic range (HDR) images. Based on bilateral filter, joint bilateral filter is developed and used in flash/noflash denoising. Joint bilateral filter can be used for upsampling problems. Although a bilateral filter has a good edge-preserving characteristic, it has been noticed that it may have artifacts in detail decomposition and HDR compression. Artifacts are resulted from those pixels around the edge that may have an unstable Gaussian weighted sum. To overcome this problem a guided filter is designed which can filter output by considering the content of the guiding image. Compared to a bilateral filter, the guided filter can perform better at the pixels near edges. Moreover, the guided filter is a nonapproximate linear-time algorithm, which is a very important strength for real-time applications. 1.1 Definition of a Digital Image A digital image (also called a discrete image) is obtained from an analogue image by sampling and quantization. This process depends on the acquisition device and depends, for instance, on CCDs for digital cameras. Basically, the idea is to superimpose a regular grid on an analogue image and to assign a digital number to each square of the grid, for example the average brightness in the square. Each square is called a pixel, for picture element, and its value is the gray-level brightness. 1

12 1.2 Characteristics of Image The space domain S, which is the set of possible positions in an image. This is related to the resolution, i.e., the number of rows and columns in the image. Consumer-grade cameras now give images with several megapixels (i.e. millions of pixels), typically between 5 and 10, professional cameras provided up to 16 megapixels, and some prototypes reach several hundreds of megapixels or even a few gig pixels.the range domain R, which is the set of possible pixel values. The number of bits used to represent the pixel value may vary. Common pixel representations are unsigned bytes (0 to 255) and floating point. To describe a pixel, one may also need several channels (or bands): for example, a vector field has two components; a color image is described with three channels, red, green and blue (or any other color space such as hue, saturation, value, namely HSV). 1.3 Image denoising One of the fundamental challenges in the field of image processing and computer vision is image denoising, where the underlying goal is to estimate the original image by suppressing noise from a noise-contaminated version of the image. Image noise may be caused by different intrinsic (i.e., sensor) and extrinsic (i.e., environment) conditions which are often not possible to avoid in practical situations. Therefore, image denoising plays an important role in a wide of applications such as image restoration, visual tracking, image registration, image segmentation, and image classification, where obtaining the original image content is crucial for strong performance. While many algorithms have been proposed for the purpose of image denoising, the problem of image noise suppression remains an open challenge, especially in situations where the images are acquired under poor conditions where the noise level is very high. So, the challenge of good image denoising model is that it has to remove noise while preserving edges. 2

13 (Input Image + Noise)= noisy image Apply wavelet Transform to decompose image Denoised image Inverse Wavelet Transform Apply filter & threshold to shrink Figure 1.1.Basic model of image denoising 1.4 General classification of image filtering techniques There are two basic approaches of the image filtering: spatial domain filtering and transform domain filtering Spatial domain filtering A traditional way to remove noise from image data is to employ spatial filters. Spatial domain filtering methods take original noisy image into consideration and apply filtering processing on it. Spatial filters are high speed processing tools Linear Filters Linear filters like mean filter, wiener filter too tend to blur sharp edges, destroy lines and other fine image details, and perform poorly in the presence of signaldependent noise. Mean filtering is simple, intuitive and easy to implement method of reducing noise in images. The idea of mean filtering is simply to replace each pixel value in an image with the mean ( average ) value of its neighbors, including itself. The Wiener filtering method requires the information about the spectra of the noise and original signal and it works well only if the underlying signal is smooth Non-Linear Filters With non-linear filters, the noise is removed without any attempts to explicitly identify it. To resolve the issues raised with linear filters, a variety of non-linear filters such as median, weighted median, rank conditioned rank selection, and relaxed median have been developed. 3

14 1.4.2 Transform domain Filtering In contrast with spatial domain filtering methods, transform domain filtering methods first obtain some transform of given noisy image and then apply denoising procedure on transformed image. The transform domain filtering methods were subdivided according to the choice of the basis transform functions which may be data adaptive or non-data adaptive Data adaptive transform The transform domain filtering methods that made choice of data adaptive transform functions inside a popular example of Independent component analysis (ICA) method. This method is successfully implemented for denoising for non- Gaussian data. This method assumes the signal should be non-gaussian. This assumption helps to denoising images with non-gaussian as well as Gaussian distribution. The main drawback with ICA method is its computational cost because it uses a sliding window and requires sample of noise free data or at least two image frames of the same scene. But in some applications, it might be difficult to obtain the noise free data Non-data adaptive transform The transform domain filtering methods that made choice of non-data adaptive transform functions were further subdivided into two domains namely spatialfrequency domain and wavelet domain. Spatial frequency domain Filtering methods in spatial-frequency domain refer use of low pass filtering by designing a frequency domain filter that passes all the frequencies lower than and attenuates all frequencies greater than a cut-off frequency. Before applying the filtering method, domain of given noisy image is changed from spatial to frequency using Fast Fourier Transform (FFT). These methods are time consuming and depend on the cut-off frequency and the filter function behavior. Furthermore, they may produce artificial frequencies in the processed image. 4

15 Wavelet transform For denoising in wavelet transform, various algorithms based on wavelet transform have been developed. The focus was shifted from the spatial and Fourier transform domain to the wavelet transform domain. It has been proved that the use of wavelets successfully removes noise while preserving the signal characteristics, regardless of its frequency content. Similar to spatial domain filtering, filtering operations in the wavelet domain can also be subdivided into linear and non-linear methods. Linear wavelet transform methods include the most popular example of Wiener filters while non-linear wavelet transform methods include coefficient thresholding based methods. 5

16 CHAPTER 2 LITERATURE SURVEY A temporal information considerably improves the frame-by-frame approach of [3] for both stereo and optical flow estimation and outperform the current state-of-theart in local space-time stereo matching. However, it is not applicable for CPU computation. Although GPU provides an alternative solution to a high-throughput guided filter, it has higher cost and power demand that is not suitable for mobile devices like digital cameras or mobile phones. Quantitative and qualitative results demonstrate that the approach (i) considerably improves over frame-by-frame methods for both stereo and optical flow; and (ii) outperforms the state-of- the- art for local space-time stereo approaches. Discrete label-based approaches have been successfully applied to many computer vision problems such as stereo, optical flow, interactive image segmentation or object recognition. In a typical labeling approach, the input data is used to construct a three-dimensional cost volume, which stores the costs for choosing a label at image co-ordinates. Because of the degradation in quality caused by fast approximation bilateral filter, a guided filter was used for fast cost volume filtering. To achieve this (i) disparity maps in real-time, whose quality exceeds those of all other fast (local) approaches on the Middlebury stereo benchmark, and (ii) optical flow fields with very fine structures as well as large displacements. [4] An efficient and scalable design for histogram-based bilateral filtering (BF) and joint BF (JBF) by memory reduction methods [17] and architecture design techniques to solve the problems of high memory cost, high computational complexity, high bandwidth, and large range table. The presented memory reduction methods exploit the progressive computing characteristics to reduce the memory cost to 0.003% 0.020%, as compared with the original approach. The architecture design techniques adopt range domain parallelism and take advantage of the computing order and the numerical properties to solve the complexity, bandwidth, and range-table problems. 6

17 A novel filtering-based technique to tackle this issue, called importance filtering. It uses a guided filter to filter out the image saliency under the guidance of the original image [16]. It avoids undesired distortion such as pixel swap that occurs to many earlier methods. The importance filtering operations are highly efficient and ready for real-time applications. The simple nature of filter operations allows highly efficient implementation for real-time applications and easy extension to video retargeting, as the structural constraints from the original image naturally convey the temporal coherence between frames. An optical degradation model which enables us to adopt a point operation scheme to realize image multi-focusing [15]. It can effectively reduce halo artifacts in the refocused image and greatly improve the computational efficiency. A two-step approach is applied to estimate the blur map of the input image. i) A sparse blur map is obtained by estimating the amount of defocus blur at edge locations. ii) The guided image filtering method is applied to propagate the value from edge locations into the unknown regions. A simple geometry prior of photograph to eliminate the ambiguity over the focal plane. Based on the obtained depth map, we can directly produce different styles of images by multi-focusing with the adjustment to the camera parameters. The VLSI architecture to achieve high-throughput and improved-quality stereo vision for real applications. The stereo vision processor generates gray-scale output images with depth information from input images taken by two CMOS Image Sensors (CIS). The depth estimator using the sum of absolute differences (SAD) algorithm as stereo matching technique is implemented on hardware by exploiting pipelining and parallelism. To produce depth maps with improved-quality at real-time, pre- and postprocessing units are adopted, and to enhance the adaptability of the system to real environments, special function registers (SFRs) are assigned to vision parameters [14]. Automatically generated depth maps from video are usually not aligned with the objects in the original image and produced at lower resolutions. In order to apply a joint-bilateral filter to smoothen the depth map within the objects and up sample it to the original image resolution while keeping object edges in the depth map aligned 7

18 with the original image. The performed algorithmic and DSP specific optimizations to achieve the real-time implementation on an embedded DSP processor, TM3270, while preserving high quality results [13]. A novel explicit image filter called guided filter. Derived from a local linear model, the guided filter computes the filtering output by considering the content of a guidance image, which can be the input image itself or another different image. The guided filter can be used as an edge-preserving smoothing operator like the popular bilateral filter [12], but it has better behaviors near edges. The guided filter is also a more generic concept beyond smoothing: It can transfer the structures of the guidance image to the filtering output, enabling new filtering applications like dehazing and guided feathering. Moreover, the guided filter naturally has a fast and no approximate linear time algorithm, regardless of the kernel size and the intensity range. The guided filter is both effective and efficient in a great variety of computer vision and computer graphics applications, including edge-aware smoothing, detail enhancement, HDR compression, image matting/feathering, dehazing, joint up sampling. Bilateral filtering smooth's images while preserving edges, by means of a nonlinear combination of nearby image values. The method is non iterative, local, and simple. It combines gray levels or colors based on both their geometric closeness and their photometric similarity, and prefers near values to distant values in both domain and range. In contrast with filters that operate on the three bands of a colour image separately, a bilateral filter can enforce the perceptual metric underlying the CIE-Lab color space, and smooth colors and preserve edges in a way that is tuned to human perception [5]. Digital photography has made it possible to quickly and easily take a pair of images of low-light environments: one with flash to capture detail and one without flash to capture ambient illumination. The variety of applications is presented that analyze and combine the strengths of such flash/no-flash image pairs. The applications include denoising and detail transfer (to merge the ambient qualities of the no-flash image with the high-frequency flash detail), white-balancing (to change the color tone of the ambient image), continuous flash (to interactively adjust flash intensity), and red-eye removal (to repair artifacts in the flash image) [7]. 8

19 CHAPTER 3 DENOISING USING GUIDED FILTER Digital images can be corrupted by noise during the process of the acquisition and transmission, degrading their quality. Image denoising is one of the fundamental challenges in the field of image processing and computer vision, where the underlying goal is to estimate the original image by removing noise from a noisy version of the image. A major challenge is to remove noise as much as possible without eliminating the most representative characteristics of the image, such as edges, corners and other sharp structures. Ideally denoising is all about filtering noise from degraded image while keeping other details unchanged. Indeed, filtering is the most fundamental operation of image processing and computer vision and it is used extensively in a wide range of applications, including image smoothing and sharpening, noise removal, resolution enhancement and reduction, feature extraction and edge detection. In the broadest sense of the term "filtering", the value of the filtered image at a given location is a function of the values of the input image in a small neighborhood of the same location. Filtering is an image processing technique widely adopted in computer vision, computer graphics, computational photography, etc. More specifically, filtering can be applied in many applications such as noise reduction, texture editing, detail smoothing/enhancement, colorization, relighting tone mapping, haze/rain removal, and joint up sampling. The most popular technique is the edge-preserving bilateral filter. A bilateral filter has a good edge-preserving characteristic, it has been noticed that it may have artifacts in detail decomposition and HDR compression. Artifacts are resulted from those pixels around the edge that may have an unstable Gaussian weighted sum. To overcome this problem, a guided filter is designed, which can filter output by considering the content of the guiding image. Compared to a bilateral filter, the guided filter can perform better at the pixels near edges. Moreover, the guided filter is a non-approximate linear-time algorithm, which is a very important strength 9

20 for real-time applications. Guided filter can prevent the error resulted from truncation. It can be embedded in mobile devices to achieve real-time HD applications. The designed guided filter architecture greatly reduces the usage not only in equivalent gate counts but also in on chip memory. Therefore, the architecture can achieve lower implementation costs. 3.1 BILATERAL FILTERING Figure 3.1. Illustration of bilateral filter Bilateral filtering (BF) is widely adopted in image and video processing such as denoising, texture editing and relighting tone management stylization, and optical flow estimation due to its texture preserving capabilities during processing. Figure 3.1. shows the illustration of bilateral filter. The intensity value at each pixel in an image is replaced by a weighted average of intensity values from nearby pixels. This weight can be based on a Gaussian distribution. Crucially, the weights depend not only on Euclidean distance of pixels, but also on the radiometric differences (e.g. range differences, such as color intensity, depth distance, etc.). This preserves sharp edges by systematically looping through each pixel and adjusting weights to the adjacent pixels accordingly Functional Units of Bilateral Filter The image data, as well as all constants and coefficients used in the following design concept, are integer numbers. There is no need to implement floating-point computation. With the aid of the presented design concept, the bilateral filter can be realized as a highly parallelized pipeline structure giving great importance to the 10

21 effective re-source utilization. In this project, the data paths are detailed. The description of the control signals is not addressed here. Figure 3.2. Order of the functional units of the bilateral filter For the design description, a window size of 5 5 is chosen. This window size is the tradeoff between high noise reduction and low blurring effect. The design concept for the implementation of the bilateral filter is subdivided into three functional blocks. The block-based design approach reduces design complexity and simplifies validation. Figure 3.2. presents these units and their order in the concept. The input data marked by Data in are read line by line and arranged for further processing in the register matrix. The second unit is the photometric filter which weights the input data according to the intensity of the processed pixels. The filtering is completed by the geometric filter, and the filtered data are marked by Data out Merits of bilateral filter 1. In contrast with filters that operate on the three bands of a color image separately, a bilateral filter can smooth colors and preserve edges in a way that is tunes to human perception. 2. Bilateral filtering produces no phantom colors along edges in color images, and reduces phantom colors where they appear in the original image Limitations of bilateral filter 1. Staircase effect - intensity plateaus that lead to images appearing like cartoons. 2. Gradient reversal introduction of false edges in the image. There exist several extensions to the filter that deal with these artifacts. Alternative filters, like the guided filter, have also been designed as an efficient alternative without these limitations. 11

22 3.1.4 Applications of bilateral filter 1. Denoising - This is of course the primary goal of bilateral filter, and it has been used in several applications such as medical images, movie restoration, etc. Some fields of applications are described. An extension of the bilateral filter will be presented in the cross bilateral filter. 2. Contrast Management Bilateral filtering has been particularly successful as a tool for contrast management tasks such as detail enhancement or reduction. The bilateral filter is used to separate an image into a large-scale component and a small-scale component by subtracting filtered results. 3. Data Fusion These applications use bilateral filtering to decompose several source images into components and then recombine them as a single output image that inherits selected. 4. Texture and Illumination Separation, Tone Mapping, Retinex, and Tone Management - Based on a large-scale / small-scale decomposition of images, these applications edit texture and manipulate the tonal distribution of an image to match the capacities of a given display or achieve photographic stylization. 5. Three dimensional Fairing This is the counterpart of image denoising for three-dimensional meshes and point clouds. Noise is removed from these data sets. 3.2 GUIDED FILTER A novel explicit image filter called guided filter. Derived from a local linear model, the guided filter computes the filtering output by considering the content of a guidance image, which can be the input image itself or another different image. The Figure 3.3. shows that the process of guided filter. It can be used as an edgepreserving smoothing operator like the popular bilateral filter, but it has better behaviors near edges. The guided filter is also a more generic concept beyond smoothing: It can transfer the structures of the guidance image to the filtering output, enabling new filtering applications like dehazing and guided feathering. Moreover, the guided filter naturally has a fast and non-approximate linear time algorithm, regardless of the kernel size and the intensity range. Currently, it is one of the fastest edgepreserving filters. Experiments show that the guided filter is both effective and efficient in a great variety of computer vision and computer graphics applications, 12

23 including edge-aware smoothing, detail enhancement, HDR compression, image matting/feathering, dehazing, joint up sampling. Guided filter has the non-approximation characteristic and offers an ideal option for real-time filter applications on HD videos. Recently, many applications adopted a guided filter as the filtering method. Because of the degradation in quality caused by fast approximation bilateral filter, a guided filter was used for fast cost volume filtering. In order to suppress color noise while preserving color structures, guided image filtering was used to smooth the result of transferred colors. Figure 3.3. Example for Guided Filter In guided filter the input is represented as p and the output is represented as q and I is nothing but the guided image representation. Compared to a bilateral filter, the guided filter can perform better at the pixels near edges. Compared with joint bilateral filter, a guided filter has the edge-preserving smoothing property but does not suffer from the gradient reversal artifacts near edges. First define a general linear translation-variant filtering process, which involves a guidance image I, an filtering input image p, and an output image q. Both I and p are given beforehand according to the application, and they can be identical. The filtering output at a pixel is expressed as weighted average: Qi =ΣjWij (I) Pj [3.1] Where i and j are pixel indexes. The filter kernel Wij is a function of the guidance image I and independent of p. This filter is linear with respect to p. The key assumption of the guided filter is a local linear model between the guidance I and the filtering output q. Assume that q is a linear transform of I in a 13

24 window centered at the pixel k: q i a k I i b k i w k [3.2] a k b k Where (, ) are some linear coefficients. To determine the linear coefficients a k b k (, ) we need constraints from the filtering input p. Then model the output q as the input p subtracting some unwanted components n like noise/textures. q i p n [3.3] i i So after computing coefficients for all windows filtering output by, q i 1 w i w k a k I i b k w k in the image, then compute the [3.4] Algorithm Input: Filtering input image p, guidance image I, radius r, regularization Output: Filtering output q. 1. Compute following mean values by applying averaging filter fmean : meani = fmean(i) meanp = fmean(p) meanip = fmean(i.* p) meanii = fmean(i.* I) 2. Compute covariance of (I,p) using formula: covip = meanip meani.* meanp 3. Compute variance using formula: vari = meanii meani.* meani 7. Compute linear coefficients a and b as: a = covip/ (vari + ε ) b = meanp a.* meani 8. Compute mean of a and b as: meana = fmean(a) meanb = fmean(b) 14

25 9. Compute filtered output as: q = meana.* I + meanb Merits of Guided Filter 1. This filter has the edge-preserving smoothing property like the bilateral filter, but does not suffer from the gradient reversal artifacts. 2. The guided filter has an O(N) time (in the number of pixels) exact algorithm for both gray-scale and color images. 3. The guided filter performs very well in terms of both quality and efficiency Applications of guided filter 1. Image smoothing / enhancement. 2. HDR compression The HDR compression is done in a similar way, but compressing the base layer instead of magnifying the detail layer. 3. Flash/no-flash imaging Denoise a no-flash image under the guidance of its flash version. 4. Matting/feathering combined with the global sampling, the guided filter is the best performing filtering based matting method in the alphamatting benchmark. 5. Dehazing. 6. Joint upsampling - joint upsampling is to upsample an image under the guidance of another image Methodology The main aim of this project is to design guided filter for image filtering. In order to design the guided filter, the graphical block diagramming tool Simulink is used. Based on the algorithm the guided filter is designed by using the blocks provided by the Simulink. In the algorithm of guided filter the mean, the correlation and covariance of both the guided image and input image are calculated. The mathematical blocks in Simulink are used to calculate these parameters. There is a inbuilt block to calculate the mean. For calculating the mean the image is converted from 2D to 1D. And then the mean is calculated. After that the value is displayed by 15

26 using frame rate display. The HDL code will be generated directly from the Simulink by using HDL Generation option. But the mean block doesn t support for the HDL generation. So, the Simulink model for designing the guided filter is subdivided into three stages. In the first stage the mean of the guided image and input image is calculated. The correlation of input image and guided image is calculated. Also, the correlation guided image and guided image is calculated. The mean block is also take place in these calculations also. So, these models are doesn t converted to HDL code directly from Simulink. The values of these parameters are noted. In the second stage, the coefficients a and b are calculated. In this stage, there is no mean block. By using all the values which is obtained from the first stage the coefficients a and b are calculated. After that the Simulink model for calculating coefficients is put into a subsystem. The subsystem is then converted as HDL code by using the option HDL Generation. In the third stage, the mean for coefficients of a and b are calculated and by using that values the output filtered image is obtained. Guided filter is the one, which reduces noise, preserves edges in image and videos. Here, the image is taken as a input to the guided filter to obtain the high quality output image. By analysis, it is known that an image cannot be given as an input in Xilinx directly. Hence, the process is carried out with the use of Simulink. Simulink is an integrated tool with the MATLAB software. For testing purpose, different images are given directly in Simulink and received the higher quality image. The sub-system is converted HDL code by Simulink. The obtained HDL code is given to Xilinx to convert it into RTL model. 16

27 MATLAB ENVIRONMENT Simulink model Code Generation System Generator Xilinx Implmentation Flow RTL Schematic Figure 3.4. Work flow The Figure 3.4. Shows the methodology used to design the guided filter. The Simulink and system generator is available in MATLAB environment. After converting the Simulink model to HDL the code is processed in Xilinx and the RTL schematic will be generated. 17

28 CHAPTER 4 RESULTS AND DISCUSSION 4.1 INTRODUCTION The filtered output of Guided filter and RTL schematic of subsystem for generating coefficient a and b are presented in this chapter. The performance measure that has been used to analyze results is Peak Signal to Noise Ratio (PSNR) and Mean Square Error (MSE). The Simulink tool is used to view the result and the Xilinx tool is used to generate the RTL model. 4.2 PERFORMANCE METRICS The results are analyzed using different quality metrics which are detailed below Mean Square Error Mean Square Error is the average squared difference between a reference image and reconstructed image. For a m x n reference image I and reconstructed image K, the MSE is given by MSE mn m n i 0 j 0 [( i, j) K( i, j)] Peak Signal to Noise Ratio Peak Signal to Noise Ratio is the ratio between the reference image and the reconstructed image, given in decibels. The higher the PSNR value, the closer the reconstructed image is to the reference image. 2 PSNR 10 log 10 MAX I MSE 2 18

29 4.3 SIMULATION RESULTS RESULTS OF GUIDED FILTER A. Visual Results Test Image: Tulip Image. a) Guided Image b) Input image c) After applying guided filter Figure 4.1. Result of Guided Filter Figure 4.1 b) Input image (Noisy Image) has the Salt and Pepper Noise with noise density 0.02(PSNR : 27.47). Figure 4.1 c) shows the output image after applying Guided Filter (PSNR : ) a) Guided image b) Input image c) After Applying Guided Filter Figure 4.2. Result of Guided Filter Figure 4.2 b) Input image (Noisy Image) has the Salt and Pepper Noise with noise density 0.04 (PSNR : 23.21). Figure 4.2 c) shows the output image after applying Guided Filter ( PSNR : 41.73). 19

30 Test Image: Academy a) Guided image b) Input image c) After applying Guided Filter Figure 4.3. Result of Guided Filter Figure 4.3 b) Input image (Noisy Image) has the Salt and Pepper Noise with noise density (PSNR : 20.84) Figure 4.3 c) shows the output image after applying Guided Filter (PSNR : 32.65). a) Guided image b) Input image c) After applying Guided Filter Figure 4.4. Result of Guided Filter Figure 4.4 b) Input image (Noisy Image) has the Salt and Pepper Noise with noise density 0.04 (PSNR : 18.96). Figure 4.4 c) shows the output image after applying Guided Filter (PSNR : 30.75). 20

31 Test image : Mandrill a) Guided image b) Input image c) After applying Guided Filter Figure 4.5. Result of Guided Filter Figure 4.5 b) Input image (Noisy Image) has the Salt and Pepper Noise with noise density 0.02 (PSNR : 25.53). Figure 4.5 c) shows the output image after applying Guided Filter (PSNR : 38.31). a) Guided image b) Input image c) After applying Guided Filter Figure 4.6. Result of Guided Filter Figure 4.6 b) Input image (Noisy Image) has the Salt and Pepper Noise with noise density 0.04 (PSNR : 18.71). Figure 4.6 c) shows the output image after applying Guided Filter (PSNR : 34.92). 21

32 Test image : Einstein a) Guided image b) Input image c) After applying Guided Filter Figure 4.7. Result of Guided Filter Figure 4.7 b) Input image (Noisy Image) has the Salt and Pepper Noise with noise density (PSNR : 22.50) Figure 4.7 c) shows the output image after applying Guided Filter (PSNR : 38.31) a) Guided image b) Input image c) After applying Guided Filter Figure 4.8. Result of Guided Filter Figure 4.8 b) Input image (Noisy Image) has the Salt and Pepper Noise with noise density 0.04 (PSNR : 19.75). Figure 4.8 c) shows the output image after applying Guided Filter (PSNR : 33.69). 22

33 5.3.2RESULTS OF BILATERAL FILTER Test Image: Tulip Image. a) Input image b) Noisy image c) After applying Bilateral Filter Figure 4.9. Result of Bilateral Filter Figure 4.9 b) Noisy Image has the Salt and Pepper Noise with noise density 0.02 (PSNR: 27.47). Figure 4.9 c) Shows the output image after applying Bilateral Filter (PSNR: 26.71). a) Input image b) Noisy image c) After applying Bilateral Filter Figure Result of Bilateral Filter Figure 4.10 b) Noisy Image has the Salt and Pepper Noise with noise density 0.04 (PSNR: 23.21). Figure 4.10 c) Shows the output image after applying Bilateral Filter (PSNR : 24.43) 23

34 Test Image: Academy a) Input image b) Noisy image c) After applying Bilateral Filter Figure Result of Bilateral Filter Figure 4.11 b) Noisy Image has the Salt and Pepper Noise with noise density 0.02 (PSNR: 20.84). Figure 4.11 c) Shows the output image after applying Bilateral Filter (PSNR: 21.65). a) Input image b) Noisy image c) After applying Bilateral Filter Figure Result of Bilateral Filter Figure 4.12 b) Noisy Image has the Salt and Pepper Noise with noise density 0.04 (PSNR: 18.96). Figure 4.12 c) Shows the output image after applying Bilateral Filter (PSNR : 22.75). Test image: Mandrill 24

35 a) Input image b) Noisy image c) After applying Bilateral Filter Figure Result of Bilateral Filter Figure 4.13 b) Noisy Image has the Salt and Pepper Noise with noise density 0.02 (PSNR: 25.53). Figure 4.13 c) Shows the output image after applying Bilateral Filter (PSNR : 27.38). b) Input image b) Noisy image c) After applying Bilateral Filter Figure Result of Bilateral Filter Figure 4.14 b) Noisy Image has the Salt and Pepper Noise with noise density 0.04 (PSNR: 18.71). Figure 4.14 c) Shows the output image after applying Bilateral Filter (PSNR : 25.75). 25

36 Test image: Einstein a) Input image b) Noisy image c) After applying Bilateral Filter Figure Result of Bilateral Filter Figure 4.15 b) Noisy Image has the Salt and Pepper Noise with noise density 0.02 (PSNR: 22.50) Figure 4.15 c) Shows the output image after applying Bilateral Filter (PSNR: 27.61) c) Input image b) Noisy image c) After applying Bilateral Filter Figure Result of Bilateral Filter Figure 4.16 b) Noisy Image has the Salt and Pepper Noise with noise density 0.04 (PSNR: 19.75) Figure 4.16 c) Shows the output image after applying Bilateral Filter (PSNR: 29.81) 26

37 The visual results obtained for the test images indicate that the proposed Guided Filter which makes use of guided image is better compared with other filters like bilateral filter. Table 5.1: Parameters Used Component Parameter Value of Parameter Image Image size 1920 x x 256 Type Gray and color Image Noise Type Salt and Pepper Noise Variance of Noise Software MATLAB R2012a VIVADO Table 5.2: Comparison of Performance Metrics of Guided Filter and Bilateral Filter (Salt and Pepper Noise with Noise Density 0.02 and 0.04) TECHNIQUE NOISE METRIC IMAGES DENSITY Tulip Academy Mandrill Einstein Guided Filter 0.02 PSNR PSNR Bilateral Filter 0.02 PSNR PSNR

38 5.3.3 SCHEMATIC OF GUIDED FILTER The RTL Schematic is obtained by using Simulink and Xilinx tool. In Simulink, the Guided Filter based on its algorithm is designed. In this design of guided filter, the mean for input image and guided image as well as co-efficient a and b has been calculated. The mean block doesn t support to generate the HDL code directly from the Simulink blocks. For this reason, the architecture is subdividedd into three parts: The subsystem which is used for generating the co-effeicients only converted into HDL code and RTL Schematic also obtained for the same. Figure Simulink Model of Guided filter The design of guided filter using Simulink blocks is depicted in Figure The mean of guided image and input image and the correlation of input image and guided image, correlation of guided image and guided image will be given as the input to the subsystem. The coefficients a and b are generated by the subsystem. Again, the mean for coefficient 'a' and 'b' is calculated and by using that value the filtered output image will be obtained. 28

39 Figure Simulink model for calculating mean to Guided image The Figure 4.18 shows the Simulink model for calculating the mean for guided image. The image is taken by using Image from file block. Then the image is converted from 2D to 1D by using Convert 2-D to 1-D block. After that by using the Mean block the mean will be calculated and the value displayed using Frame Rate Display block. The mean of guided image is Figure Simulink model for calculating mean to Input image The Figure 4.19 shows the Simulink model for calculating the mean for input image. The image is taken by using Image from file block. Then the image is converted from 2D to 1D by using Convert 2-D to 1-D block. After that by using the Mean block the mean will be calculated and the value displayed using Frame Rate Display block. The mean of input image is

40 Figure Simulink model for calculating correlation of Input image and Guided image The Figure 4.20.Shows the Simulink model for calculating correlation of Input image and Guided image. The input image and guided image is taken by using Image From File block. After that the product block is used to multiply the two images. After that the mean is calculated by using Mean block. The value is displayed by Frame Rate Display. The correlation value is Figure Simulink model for calculating mean to correlation of Guided image and Guided image The Figure Shows the Simulink model for calculating correlation of Guided image and Guided image. The input image and guided image is taken by using Image From File block. After that the product block is used to multiply the two 30

41 images. After that the mean is calculated by using Mean block. The value is displayed by Frame Rate Display. The correlation value is Figure Simulink model for calculating co-efficient a and b The coefficient 'a' and 'b' are calculated by Figure This Simulink model has been inserted in subsystem. The subsystem is converted as HDL code and the RTL Schematic is generating for that subsystem. The mean of input image and guided image is used as the input to this subsystem. Also the correlation of input image and 31

42 guided image is given as the input to the subsystem. The correlation of guided image and guided image also given as the input to the subsystem. The values of coefficients a and b are and This subsystem is directly converted to the HDL code by using HDL Generation option. Figure Simulink model for viewing filtered output image The Figure shows the Simulink model for viewing filtered output image. The Image From File block is used to take the guided image. And the values of coefficients a and b are given by using constant block. After that these values are multiplied by using Product block. The product of guided image I and coefficient a is added with coefficient b by using Add block. The filtered output image is viewed from Video Viewer block. 32

43 Figure RTL Schematic for Subsystem The HDL code which is automatically generated for subsystem in the simulink is then given to the xilinx software. The RTL Schematic which is shown in Figure is generated from VHDL code through xilinx software. The RTL schematic has eight inputs and two outputs. The two outputs are coefficients a and b. The Inputs will be given by as the values which is obtained from the first stage (i.e.) The mean of Input image, mean of guided image, correlation of input and guided image and also the correlation of guided image and guided image. 33

44 Figure Guided Image Figure Input Image 34

45 Figure Output Image The guided image which is shown in Figure and the input image which is shown in Figure 4.26 are given as the inputs to the guided filter and the output image which is shown in the Figure is obtained as the filtered output image which has the sharp edges than the input image is obtained from the guided filter. 35

46 CHAPTER 5 CONCLUSION In the field of image processing, the filters with high performance and low hardware cost plays a vital role. In this project, the guided filter is designed using Simulink block set and it is converted into VHDL code for Schematic generation. Two images are given as input which obtains highly sharp edged output image with absence of noise and improved quality. The guided filter performs edge preserving, smoothening on an image using the content of the second image called a guidance image to influence the filtering. The guidance image can be an input image or a different version of an input image or a completely different image. The performance of guided filter is compared with bilateral filter and it is observed that the guided filter image has high PSNR value. Finally the RTL schematic for the subsystem is generated by using the HDL code which is generated from the Simulink using the Xilinx tool. Also, the same process can be applied for HD videos. 36

47 REFERENCES [1] A. Y.-S. Chia, S. Zhuo, R. K. Gupta, Y.-W. Tai, S.-Y. Cho, P. Tan, and S. Lin, Semantic colorization with internet images, ACM Trans.Graph., vol. 30, no. 6, pp. 156:1 156:8, Dec [2] C. Liu, W. Freeman, R. Szeliski, and S. B. Kang, Noise estimation from a single image, in Proc. IEEE CVPR, vol. 1, 2006, pp [3].C. Rhemann, M. Bleyer, and M. Gelautz, Temporally consis-tent disparity and optical flow via efficient spatio-temporal filtering, in Advances in Image and Video Technology, LNCS. vol. 7087, Y.-S. Ho,Ed. Berlin, Germany: Springer, 2012, pp [4] C. Rhemann, A. Hosni, M. Bleyer, C. Rother, and M. Gelautz, Fast cost-volume filtering for visual correspondence and beyond, in Proc.IEEE Conf. CVPR, 2011, pp [5] C. Tomasi and R. Manduchi, Bilateral filtering for gray and color images, in Proc. IEEE 6th ICCV, 1998, pp [6] F. Durand and J. Dorsey, Fast bilateral filtering for the display of high-dynamicrange images, ACM Trans. Graph., vol. 21, no. 3, pp , Jul [7] G. Petschnigg, R. Szeliski, M. Agrawala, M. Cohen, H. Hoppe, and K. Toyama, Digital photography with flash and no-flash image pairs, ACM Trans. Graph., vol. 23, no. 3, pp , Aug [8] J. Kopf, M. F. Cohen, D. Lischinski, and M. Uyttendaele, Joint bilateral upsampling, ACM Trans. Graph., vol. 26, no. 3, article no. 96, Jul [9] J.-W. Van de Waerdt, S. Vassiliadis, S. Das, S. Mirolo, C. Yen, B. Zhong, C. Basto, J.-P. van Itegem, D. Amirtharaj, K. Kalra, P. Rodriguez, and H. van Antwerpen, The TM3270 media-processor, in Proc. 38th Annu.IEEE/ACM Int. Symp. MICRO-38, 2005, pp [10] J. Zhang, L. Li, Y. Zhang, G. Yang, X. Cao, and J. Sun, Video dehazing with spatial and temporal coherence, Vis. Comput., vol. 27, pp , Jun [11] K. He, C. Rhemann, C. Rother, X. Tang, and J. Sun, A global sampling method for 37