Demosaicking using Adaptive Bilateal Filtes Ekine Akuiyibo Timothy Atobatele EE 362, Winte 2006 0. Abstact Digital cameas sample the continuous spectum using an aay of colo filtes such that each pixel samples only one colo band poducing what is commonly efeed to as a colo mosaic. Demosaicking is then the pocess of filling in the lost colo infomation to estoe the oiginal image. In this poject, we exploed the novel concept of bilateal filteing [2] and its use fo the demosaicking pocess. We pesent the esults of ou expeiments. Figue 1: a) Oiginal Lighthouse Pictue, b) Lighthouse Mosaic 1. Intoduction Demosaicking o colo filte aay intepolation is one of seveal image pocessing steps a digital camea pefoms to povide the use a viewable image. Digital still colo cameas poduce a colo mosaic duing image captue as the light sensos sample the image though a colo filte aay (CFA). Thee ae diffeent CFA s used in diffeent cameas but the most popula till date is the Baye CFA [1]. The Baye patten shown in Figue 2 below typifies the design of most CFA pattens. Luminance (epesented by geen) occus at twice the spatial fequency of the chominance (ed and blue), accounting fo two facts i) the luminance esponse of the human vision peaks aound the fequency of geen light and, ii) the human vision is moe sensitive to luminance than chominance. The challenge becomes the best way to epoduce the oiginal full-colo esolution image. 1
Figue 2: Baye Colo Filte Aay With the advent of cheape digital camea manufactuing ove the last 10 yeas, eseach into moe efficient ways to demosaic has been a focus of the image pocessing community. This has esulted in a plethoa of techniques being poposed fo demosaicking fom simple data eplication techniques to moe complex fouie domain algoithms. We will only include efeences to some of these othe methods pefeing instead to concentate on poviding backgound into demosaicking via bilateal filteing. 2. Bilateal Filteing Bilateal filteing smoothes images while peseving edges, by means of a nonlinea combination of neaby image values. And even though bilateal filteing is a nonlinea technique it is non-iteative, local and simple. The nonlineaity aises due to the nonlinea elationship of pixel values of an image. Bilateal filtes compise of two component filtes: a domain filte and a ange filte. I. Domain: the domain filte component efes to the taditional low-pass filte used to aveage values of the image that ae close in space. Ou implementation of the domain filte utilizes a Gaussian blu kenel fo filte weights. Figue 3 shows an example 5X5 Gaussian blu kenel. Figue 3: 5X5 Gaussian Blu Example 2
II. Range Filte: the ange component aveages pixel values based on similaity (photometic closeness). We say two pixels ae simila to each othe if thei photometic values ae close. Unlike the domain component, similaity is dependent on image chaacteistics. This adaptive quality is illustated in the figues below. Figue 4: a) Slanted Line Image, b) Staight Line Image Figue 5: a) 5X5 Slanted Line Kenel, b) 5X5 Staight Line kenel The Bilateal filte is the esultant poduct (spatial convolution) of the domain and ange filtes, which esults in an aveaging of image pixels based on spatial and photometic closeness. This is the cental idea undelying the bilateal filteing intepolation method. Notice that the nonlineaity as a esult of pixel values does not necessaily add computational complexity. Figue 4 shows the oiginal images while the coesponding ange filte kenels geneated along the edge of the image is depicted in Figue 5. Figue 6 below shows the combined Gaussian blu kenel with the slant and staight-line kenels discussed above. As mentioned befoe the ange kenel is geneated accoding to image content and thus the esulting bilateal filte has adaptive weights. 3
Figue 6: Bilateal Filte kenels fo a) Slant line, b) Staight line Note that ange filteing alone does not povide us with the desied aveaging [2]. We do not want to aveage pixels close in value fom one cone of the image to the othe cone. The domain component thus ensues the aveaging is local which is the desied effect we seek. 3. Application to Demosaicking The pincipal chaacteistic of any good demosaicking technique is edge detection. Filteing in geneal is a fundamental opeation that is vey well undestood: value of the filteed image is some weighted aveage of neaby image values. The poblem is we don t want to aveage acoss edges in ou filteed image. Thus, we have to find a way to detect edges, and in the case of demosaicking intepolate along the edges and not acoss them. Bilateal filtes povide this capability with the combined domain and ange components. Demosaicking via bilateal filteing occus essentially as a two-step pocess: Estimate the bilateal kenel fo the desied Image pixel: as we showed in pevious examples the bilateal kenel adapts to the image chaacteistics. Lets say we want to intepolate the missing blue values of a mosaic that utilized the Baye CFA. Figue 7 shows an m x n aay whee the blank spaces efe to the missing blue values. 4
B1 X B2 B3 B4 B5 Figue 7: Mosaic showing only blue pixels To intepolate fo a missing blue pixel value say at X, we detemine ou bilateal kenel (bilateal filte weights) fom the suounding pixels B1-B5. The esulting intepolation thus is the weighted sum of blue pixel weights suounding pixels used to estimate X. We explain ou implementation pecisely in the next sections. Cuent bilateal filte eseach (fo demosaicking) is focused on moe efficient penalty functions fo weight adaptation. 4. Bilateal Filte Design [3][8] Reiteating: Bilateal filte adaptively smoothes an input image ove neighboing pixels with edge detection capability. Thoughout this section the input image is denoted f while the output image is denoted g. Conside a pixel of an image with coodinates (x, y) epesented as z = $ x% ' and a # y& neighboing pixel of the image with coodinates (x+δx, y+δy) epesented as $ = x + #x ' & ). The contents of the input and output images at the pixel (x, y) ae f % y + #y( and g z ( ) espectively. The output image content at pixel (x, y) is computed as a weighted aveage of the input image contents of all pixels within a (2W+1)-by- (2W+1) window aound the pixel (x, y), whee W is pedefined. This is expessed mathematically as below g( z ) = whee b W % W % #y=$w #x=$w W W f ( )b z,, f ( z ), f ( ( )) b z,, f ( z ), f % % ( ( )) #y=$w #x=$w z,, f ( z ( ), f ( )) z ( ) (4.1) is the bilateal filte kenel used to weight each of the pixels in the aveaging pocess. The bilateal filte kenel is constucted such that the neaby pixels contibute moe to the aveage than the pixels that ae fathe away. 5
The cental idea in the design of any image-smoothing filte is that pixels that ae in close geometic poximity have simila contents. Thus, it is assumed safe to aveage ove close pixels. Howeve, this cental idea beaks down at the edges of an image. In this context, edges efe to those points on an image whee thee ae discontinuities o shap contasts between a pixel s content and its immediate neighbo s. The bilateal filte accounts fo the edges by weighting pixels based on thei photometic similaity in addition to geometic poximity. Theefoe, the bilateal filte kenel is a composite kenel consisting of a domain filte kenel and a ange filte kenel. The domain filte kenel weights pixel contents based on geometic poximity to the cente pixel while the ange filte kenel weights the pixel contents based on photometic similaity to the cente pixel. These ideas ae expessed mathematically as below b z,, f ( z ), f ( ( )) = d whee d z, z, ( ) f ( z ), f ( ( )) (4.2) ( ) is the domain filte kenel and ( f ( z ), f ( )) is the ange filte kenel. As mentioned ealie, the domain filte kenel weights pixels based on thei geometic poximity to the cente pixel while the ange filte kenel weights the pixels based on thei photometic similaity (poximity) to the cente pixel with the nea pixels contibuting moe than the fa pixels. The Gaussian function is a natual candidate fo implementing such a weighting scheme. The Gaussian has most of its weight at o nea the cente and exponentially diminishes away fom the cente. Thus, both the domain filte kenel and the ange filte kenel have been designed with the Gaussian as follows d z, % ( ) = exp # s d ( z, ) 2 ( ' * 2 ' 2$ d * & ) % f ( z ( ), f ( )) = exp # s ' f ( z ), f 2 ' 2$ & ( ( )) 2 ( * * ) (4.3) (4.4) The aguments of the Gaussians above expess the notion of poximity between the cente pixel and its neighbos in the geometic and photometic sense espectively. The tems 2 d and 2 ae the geometic and photometic vaiances espectively. Roughly speaking, the vaiances detemine how much weight fa pixels (geometic o photometic) contibute to the aveaging pocess. The tem s d z, squae of the Euclidean distance between two points in a plane given by s d z, ( ) 2 is the familia ( ) 2 = ( ) T ( ) (4.5) z # z # 6
The tem s f ( z ( ), f ( )) 2 is a measue of the photometic distance between the cente pixel and its neighbos. This measue has been defined in moe than one way in liteatue. Howeve, the two definitions that ae adopted in this wok ae the squae of the ΔE diffeence (expessed in Eq. 4.6 below) and the squaed diffeence (SD) in the pixel contents (expessed in Eq. 4.7 below). f ( z ( ), f ( )) 2 = f ( z ( ) # f ( )) T f ( z ( ) # f ( )) (4.6) s s f ( z ( ), f ( )) 2 = f ( z ( ) # f ( )) 2 (4.7) The ΔE metic is well defined in the CIELAB and HSV colo spaces while the SD metic was used in the RGB and YCbC colo spaces in the bilateal filte implementation in this wok. 5. Bilateal Filte Implementation (MATLAB) [3][8] A MATLAB file (bilateal _ filte3.m ) that implements the above design of the bilateal filte has been witten. A sample function call to the bilateal filte file is as below >> [g,n] = bilateal _ filte3( f, d 2, s 2,space1,space2); The inputs to the filte ae an M-by-N-by-3 RGB image, the geometic and photometic vaiances, the colo space in which the aveaging pocess descibed in the bilateal filte design section is to take place and the colo space in which photometic similaity is defined. The outputs of the filte ae an M-by-N-by-3 RGB image, which is the bilatealfilteed vesion of the input image and M-by-N-by-3 nomalization matix used in the aveaging pocess but not eally useful aftewad. We have attempted to lucidly document the MATLAB code making it easy to use. To see the documentation, you could just type >>help bilateal_filte3 at the MATLAB command pompt. 6. Expeiments/Results We an the bilateal filte (bilateal_filte3) on test images to poduce a sampling of the esults that follow. 7
Animals: a) Mosaic b) Demosaicked (a) (b) Macbeth: a) Mosaic, b) Demosaicked 8
(a) (b) Mackay: a) Mosaic, b) Demosaicked Slanted Ba: a) Mosaic, b) Demosaicked 9
Lighthouse Image Analysis: (a) (b) (c) (d) Figue 8: a) Oiginal Image, b) Mosaic, c) Bilinea Image, d) Bilateal Image MSE vs Domain Vaiance Expeiment: Accoding to ou domain filte definition, we expect eo of the demosaiced image to incease with incease domain filte vaiance. Figue 9 shows this tend is tue fo the example lighthouse image. 10
Figue 9: Lighthouse Eo Plot 7. De-noising As a side note, we also like to point out that an added benefit of bilateal filteing is the inheent de-noising capability that occus due to the aveaging pocess. Since noise in pixel values ae mutually less coelated than images values, any aveaging pocess automatically pefoms a smoothing opeation while peseving image values. Due to a lack of time, we did not pefom expeiments to show this effect although it is well documented in the liteatue [2]. 8. Conclusion We ae happy to show that the elatively simple idea behind the bilateal filteing pocess is indeed capable of demosaicking. Unfotunately we didn t have enough time duing this poject to quantify ou filte s pefomance, do a simple PSNR evaluation of the 4 diffeent kenels we implemented o do a compaative study with othe standad techniques available today. Nevetheless, we hope that we have shown the exceptional vesatility of the concept of bilateal filteing. A natual follow on could be paamete sensitivity analysis testing to detemine just how effective bilateal filtes can be fo demosaicking. ACKNOWLEDGEMENTS 11
We would like to thank Pof. Bian Wandell and Geg Ng fo thei valuable contibutions to this poject. 1. Bilateal Filte: bilateal_filte3.m 2. Expeiment scipts: filte_tests.m 3. Images: data APPENDIX I APPENDIX II Ekine Akuiyibo 1. Bilateal Filte and Demosaicking Reseach 2. Pesentation: Making Slides, Reheasal 3. Matlab: Example Scipts / Analysis 4. Poject wite-up Timothy Atobatele 1. Bilateal Filte and Demosaicking Reseach 2. Pesentation: Making Slides, Reheasal 3. Matlab: Bilateal Filte Implementation / Analysis 4. Poject wite-up REFERENCES [1] BE Baye, Colo imaging aay, US Patent, no.3 971 065, 1976 [2] R. Ramanath and W. E. Snyde, Demosaicking as a Bilateal Filteing Pocess, Poc. SPIE, Image Pocessing 4667, pp 236-244, 2002 [3] C. Tomasi and R. Manduchi, Bilateal Filteing fo Gay and Colo images, Sixth Intenational Confeence on Compute Vision, 1998. [4] P. Peona and J. Malik, Scale-space and edge detection using anisotopic diffusion, IEEE Tans., PAMI-12 (7): 629 639, 1990. [5] M. Gupta and T Chen, Vecto Colo Filte Aay Demosaicing, Poc. SPIE, Sensos and Camea Systems 4306, pp 374-382, 2001. [6] EE 362: Tutoial 2: Colo Matching, Tutoial 3: Colo Metics, Winte, 2006. [7] N. Kehtanavaz, H. Oh, and Y. Yoo, Colo filte aay intepolation using colo coelations and diectional deivatives JEI, 12(4), 621 632, 2003. [8] D. Baash, A Fundamental Relationship between Bilateal Filteing, Adaptive Smoothing and the Nonlinea Diffusion Equation, IEEE Tansactions on Patten Analysis and Machine Intelligence, 24(6), pp 1-5, 2002. 12