The Use of Non-Local Means to Reduce Image Noise By Chimba Chundu, Danny Bin, and Jackelyn Ferman ABSTRACT Digital images, such as those produced from digital cameras, suffer from random noise that is difficult to reduce without losing details and textures. This report describes a number of computer programs that use the Non-Local Means algorithm to successfully de-noise images while retaining image integrity. INTRODUCTION Digital images, especially those produced by digital cameras, contain undesirable, unwanted noise. Image noise is created when random pixels fluctuate in value and can result in grainy, unclear pictures. This is a common occurrence that can be caused by the exposure setting of the digital camera, limited lighting, shadows, and more. This problem is as old as digital photography, but it has been difficult to solve. There are many different methods currently being used to reduce the amount of noise in images, such as DUDE and UINTA. However, many of these methods fail at keeping certain textures and details intact as the image is de-noised. To overcome these problems we have investigated how effective a relatively new algorithm, Non- Local Means (NL Means), is with the hope that it will effectively reduce noise while keeping the rest of the image intact. The NL Means algorithm is based on concept that different areas of an image are likely to have similar characteristics so that one area can be used to de-noise another. Simply put, each pixel is represented by a matrix of surrounding pixels its neighborhood. Neighborhoods are compared to calculate a weight based on their similarity:
w ij ( k, l) = e N1( i, j ) N 2( k, l ) 2 A where ( k, l) are the weights, and N are neighborhoods, and A is a constant. w ij N 1 2 The new, denoised value of the pixel, u( i, j), is determined by the sum of the neighborhood center pixel multiplied by the weight and divided by the sum of the weights, u( i, j) = k, l f ( k, l) * w ( k, l) k, l ij ij w ( k, l) PROCEDURE We started with test images that were on the order of 100x100 pixels and analyzed them using Matlab. The idea was to change each pixel in the image to a new value that takes into account other pixels in the image by using different types of averaging in order to reduce and cancel out noise. We used simple test images and implemented the algorithms in both Matlab and C++. Our initial attempts involved straightforward programs written in Matlab, which handles images well and converts them easily into matrices containing the images intensity values. Once the image is in matrix form, it is possible to manipulate the values to reduce noise before outputting the image. It is harder to work with images in C++ than in Matlab, but C++ is much faster at running loops and other calculations. Since it is important to process images quickly, we therefore began writing all our code in C++, but continued using Matlab to read and write the images. The general procedure was to start with an image and read it into Matlab as a
matrix of integers in the range 0-255. The C++ program then input the integer matrix, performed various calculations, and created a second matrix outputted to Matlab, which created a new image. INITIAL MATLAB PROGRAMS We first began coding an elementary method of reducing noise and worked our way up to the NL Means algorithm. To become more familiar with Matlab and coding with images, we started by writing code that averages each pixel with its surrounding pixels. It loops through the entire image and changes each pixel to the average value of that pixel and its surrounding neighborhood. For any given pixel, its neighborhood consists of all pixels found in each direction over a specified radius, r. The borders create a small issue because there does not exit enough pixels on all four sides of pixels along the edges. For these cases, we took averages of smaller neighborhoods, as to not go off the edge of the image. The smaller neighborhoods are still found by adding on pixels found by going the radius amount in each direction, but if an edge is hit first it stops. Each neighborhood is a matrix of size (2 1) 2 r + entries, except at the borders which obviously result in smaller neighborhoods of various sizes. A larger radius value will reduce more noise, but results in a blurred image, as each neighborhood spans larger portions of the image and averages a larger variety of values. To effectively reduce noise with this simple averaging method it is not possible to retain details and textures. The reason NL Means is a good choice is that it makes it possible to retain the details and textures when de-noising. This is because most images contain some degree of replication. For example, brick walls, zebras, and grass are just a few of the numerous instances where distinct patterns can be found in repeated pictures. The idea is that if two separate neighborhoods of
pixels in an image have similar pixel values, then the center pixels of the two neighborhoods should have similar values as well. The algorithm compares a given pixel s neighboring pixels to other same sized neighborhoods throughout the image and computes a weight for each comparison. The weights are then used to determine how much the center pixel of each comparing neighborhood should contribute to the given pixel we are trying to de-noise. We used Matlab to write the first draft of NL Means algorithm. With this algorithm it is important to always compare neighborhoods of the same size, 2 (2r + 1), meaning the borders again created a problem, this time a little more complicated. To evaluate a pixel less than r pixels from the border it is not possible to go a value of radius in each direction. We solved this problem initially by adding rows and columns onto the image a border of size radius on all sides. Initially we added on by duplicating a portion of the image on each size of amount radius. This was not the best idea because it results visible lines around the edges in the de-noised image. When the image is simply duplicated it can result in areas around the edges where pixels of extremely different values are placed next to each other. This creates a problem when neighborhoods are compared because since this does not actually occur in the image, these areas could adversely affect the weights. We improved this by using periodic boundaries in the next version of the code. Instead of duplicating the image we reflected a radius amount of the image onto each side. This way the pixel values being added on do not create drastically different neighborhoods. We tried testing a variety of images but found that it took at least 20 minutes to run small 100x100 pixel images. Therefore it was necessary to rewrite the code in C++.
FIRST C++ CODE The first copy of our C++ code uses Matlab to read in the image and write it out to a text file saved to the Desktop. Then the C++ code reads in the file, which also contains the dimensions of the image. The user enters the desired radius and a value for the constant variable A, which determines the magnitude the weights. Larger A values cause the weights to have a greater effect on the pixels. Different images require different A values depending on the degree of noise in the image. Then we again used periodic boundaries to add a border of size radius. Next we wrote a variety of functions: zeros, nbr, vecops, matops, sqmag, and nonlocalmeans. Zeros initializes arrays to contain all 0s, nbr creates the neighborhoods and N, vecops adds, subtracts, multiplies, divides, or sums up to one dimensional arrays, matops does the same for two dimensional arrays, sqmag squares the values within an array, and nonlocalmeans performs the algorithm. In main, we created arrays for each neighborhood, and N, and used the zeros function to initialize them to zero. Next nonlocalmeans is called. First it loops through all the pixels in the image and for each pixel creates. For each N it again loops through the whole image and for each pixel creates. Vecops is used to subtract each from N, then sqmag squares the values in the new array, then the sum of the values in the array are divided by the constant A, and then raise e to the negative value of that. Theses values are the weights by which we multiply the center pixels of N1 1 this value becomes the new value of the center pixel of N1 2 N1 2 N2 N2 1. The sum of that is divided by the sum of the weights and. Finally, once we ve looped through the entire image the new pixel values are written out into a new document. Again we use Matlab and read in the document to view the new image. N 1
IMPROVEMENTS IN EXECUTION SPEED REDUCED NEIGHBORHOOD COMPARISON Looping through the entire image takes a long time - especially when the image is large. We needed to improve the code to run faster, so we created a new variable rad2. The user enters a value for this variable that is used to reduce the size of the window from which the neighborhoods are calculated. This way the first time nonlocalmeans loops through the image it still goes through the entire image, but the second time it only has to loop through a much smaller region. With this method there are also much fewer calculations, as there are fewer neighborhoods. Each pixel is no longer compared to all the other pixels in the image, but with rad2 values of roughly 10 the new image is still effectively de-noised and the run time is reduced exponentially. Example images are shown on the next page.
Figure 1: Original Image Figure 2: Original Image Figure 3: Image with added Gaussian noise Figure 4: Image with added Gaussian noise Figure 5: De-noised Image (r = 3, rad2 = 10, A = 3000) Figure 6: De-noised Image (r = 3, rad2 = 10, A = 3000)
RANDOM COMPARISON Another method we investigated to improve the code s runtime is by comparing each pixel to a random assortment of other pixels in the image. We created a user defined variable, randpts, for the number of comparison points for calculating neighborhoods. Depending on the number of random points, this made it possible to improve the time immensely. We often used between 500 and 1000 points and found this to be very effective in reducing the noise. Based on the results from both this method and the reduced window method, we found it unnecessary to ever loop through the entire image again for. The miniscule difference in how well the image was de-noised was not worth the exponential time difference. Example images are shown below. Figure 7: Original Image Figure 8: Image with added Gaussian noise Figure 9: De-noised Image (r = 2, rad2 = 500, A = 2000)
NEIGHBORHOOD COMPARISON: An additional approach we tried to reduce the time was to compare entire neighborhoods and change the entire N 1 at one time instead of just the center pixel. This way we could loop through the image neighborhood by neighborhood, instead pixel by pixel. For radii of size 3, and the resulting 7x7 neighborhoods, we managed about a 98% reduction in execution time. However this method results in a crisscross pattern in the new image produced where the borders between the different neighborhoods were (in the original image). We therefore needed to loop though the original image a second time to eliminate the lines. The second time we looped through we made the center pixels of each neighborhood to be the bottom right pixel of the previous neighborhoods. Finally, we averaged two resulted images. The result was an end image without any crisscrossing lines. In the end the time was reduced by roughly 25 times. Example images are shown below.
Figure 10: Original Image Figure 11: Image with added Gaussian noise Figure 12: New De-noised Image (r = 3, A = 2000, 1 loop) Figure 13: New De-noised Image (r = 3, A = 2000, 2 loops) Reduced window, random comparison and neighborhood comparison are all methods to speed up the processing time of the program, and as a result, compromise the quality of the resulting images. In a real life image, identical objects in the same image may have different intensity levels due to shadow effects. The previously mentioned routines are not capable of recognizing this when comparing pixel values and, instead, assign smaller weights to an otherwise similar neighborhood. So in order to resolve this issue, for a given neighborhood N k, we interpreted the value of the pixel p (at row i and column j) as the sum of the base image and the kij, b kij,
illumination at that pixel, g k ( i, j). In order to find g, we assumed a linear relationship for simplicity, and used linear regression to find the best-fit plane through the neighborhood pixels. We then applied the following formula: N (, i j) = N (, i j) g (, i j) + g (, i j ) * 2 2 2 1 * Where is adjusted to the illumination of N, allowing for a more accurate comparison. N2 N2 1 Below are a few examples. Figure 14: Original Shadow Image Figure 15: Shadow image with Gaussian Noise Figure 16: New De-noised Image (r = 3, A = 1000, 10000 points)
So far, we have only addressed two types of noise, both generated by either the rand() or randn() functions. The rand() and randn() functions generate uniformly and normally distributed numbers, respectively; normal being the more realistic distribution of noise in real life images. Our other main focus was salt and pepper noise. This type of noise either corrupts a pixel or it doesn t in a salt and pepper manner, as the name suggests. As a result, the values of these noisy pixels are either very small or very large. The previously mentioned routines work best for Gaussian noise since the noise levels are relatively close to the value of the original pixel. Applying these routines results in correctly assigned weights (according to the noise level). However, in the case of salt and pepper noise, if the center pixel of is a corrupted pixel, namely a pepper, then when multiplied by its weight the result is just large weight being assigned to a noisy pixel. Due to this unique characteristic of salt and pepper noise, we are not able to successfully de-noise the image by using the center pixels of neighborhood. Since the pixel values of salt and pepper noises are either very large or very small, we can actually use the median pixel value of the selected neighborhood to replace the center pixel in our calculation. The idea is to sort the pixels of in order of increasing magnitude. Note: the value of the noisy pixels will be near the very beginning and the very end of the sequence. The median value of that sequence is taken as the best approximation of the center. The standard procedure for calculating the weights is followed, replacing the usual center with the median. Examples of this method are on the next page.
Figure 17: Original Image Figure 18: Image with added Salt and Pepper noise Figure 19: New De-noised Image (r= 7, A= 1000) Figure 20: New De-noised Image (r= 5, A= 1000) In our aims to implement the fastest version of the NL Means algorithm, we were able to arrive at a few new ideas in the process. First, was a unique neighborhood comparison where distinct, non-overlapping neighborhoods were used to obtain and N, followed by two N1 2 iterations using a smaller radius. Each of the centers was spaced by a length of 2r pixels, compared to the 0 pixel spacing that is used for the original algorithm. This is followed by repeating the procedure with an r pixel shift in the i direction, the j direction, and then the i and j directions. The final image is obtained by taking the arithmetic mean of the four matrices.
Figure 21: Original Image Figure 22: Image with Gaussian noise Figure 23: Regular NLM (r= 1, A= 0.5) Figure 24: Single N'hood (r=1, r=2, r=2, A=0.5) Figure 25: Regular NLM (r=2, A= 0.5) Figure 26: Single N'hood (r=2, r=2, r=2, A=0.5) Figure 27: Regular NLM (r= 3, A= 0.5) Figure 28: Single N'hood (r=3, r=2, r=2, A=0.5)
The second idea involved using a more statistical approach through K-Means clustering. This was motivated by the fact that an image, such as that of a face, contains groups of very similar pixels, i.e. eyes, eyebrows, ears, lips, etc. We wanted to take advantage of this to reduce the amount of time that would be spent looking for similar regions. A Matlab implementation of the K-Means algorithm was used to find the user-defined number of centroids in a given image. This image was then passed to the C++ implementation of the NL Means algorithm (including the array containing the k-centers), and neighborhoods were taken from a 9-by-9 pixel window about each center. Figure 29: Face 1 Original Figure 30: Face 1 Original Figure 31: Face 1 with Gaussian noise Figure 32: Face 2 with 5 points Figure 33: Face 2 with 10 points Figure 34: Face 2 with 25 points
Figure 35: Face 1 (r=1, A= 0.5, 5 pts) Figure 36: Face 1 (r=1, A= 0.5, 10 pts) Figure 37: Face 1 (r=1, A= 0.5, 25 pts) Figure 38: Face 1 (r=2, A= 0.5, 5 pts) Figure 39: Face 1 (r=2, A= 0.5, 10 pts) Figure 40: Face 1 (r=2, A= 0.5, 25 pts) Figure 41: Face 1 (r=3, A= 0.5, 5 pts) Figure 42: Face 1 (r=3, A= 0.5, 10 pts) Figure 43: Face 1 (r=3, A= 0.5, 25 pts)
CONCLUSION In this paper, we have mentioned various ways in which the NL Means algorithm can be implemented to de-noise several types of noise. Gaussian, Salt & Pepper (S&P), Gaussian with shading, and S&P with shading were just some of the conditions to which the images were subjected. Our implementation was, however, limited by the amount of processing time required to perform the calculations. Fortunately, for the faster variations performance did not always lead to a compromise of calculation speed. For instance, the non-overlapping version of NL Means resulted in de-noising nearly identical to the original implementation in a fraction of the time. The NL Means algorithm is definitely an algorithm with many possibilities. It is applicable not only for the purpose of de-noising images, but also for creative and artistic applications. Since the algorithm is exceptionally good at preserving patterns in images, it is only natural to extend its capabilities to something more artistic in nature. In future investigations we look to combine images such as paintings and photographs to create photorealistic paintings. This could be achieved by using single images, or even image libraries to recreate a painted replica of a photo.