Comparison of color demosaicing methods

Transcription

1 Comparison of color demosaicing metods Olivier Losson, Ludovic Macaire, Yanqin Yang To cite tis version: Olivier Losson, Ludovic Macaire, Yanqin Yang. Comparison of color demosaicing metods. Advances in Imaging and Electron Pysics, Elsevier, 21, 162, pp <1.116/S (1)625-8>. <al > HAL Id: al ttps://al.arcives-ouvertes.fr/al Sumitted on 28 Mar 212 HAL is a multi-disciplinary open access arcive for te deposit and dissemination of scientific researc documents, weter tey are pulised or not. Te documents may come from teacing and researc institutions in France or aroad, or from pulic or private researc centers. L arcive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recerce, puliés ou non, émanant des étalissements d enseignement et de recerce français ou étrangers, des laoratoires pulics ou privés.

2 Comparison of color demosaicing metods O. Losson a,, L. Macaire a, Y. Yang a a Laoratoire LAIS UMR CNRS 8146 Bâtiment P2 Université Lille1 Sciences et Tecnologies, Villeneuve d Ascq Cedex, France Keywords: Demosaicing, Color image, Quality evaluation, Comparison criteria 1. Introduction Today, te majority of color cameras are equipped wit a single CCD (Carge- Coupled Device) sensor. Te surface of suc a sensor is covered y a color filter array (CFA), wic consists in a mosaic of spectrally selective filters, so tat eac CCD element samples only one of te tree color components Red (R), reen () or Blue (B). Te Bayer CFA is te most widely used one to provide te CFA image were eac pixel is caracterized y only one single color component. To estimate te color (R,,B) of eac pixel in a true color image, one as to determine te values of te two missing color components at eac pixel in te CFA image. Tis process is commonly referred to as CFA demosaicing, and its result as te demosaiced image. In tis paper, we propose to compare te performances reaced y te demosaicing metods tanks to specific quality criteria. An introduction to te demosaicing issue is given in section 2. Besides explaining wy tis process is required, we propose a general formalism for it. Ten, two asic scemes are presented, from wic are derived te main principles tat sould e fulfilled in demosaicing. In section 3, we detail te recently pulised demosaicing scemes wic are regrouped into two main groups : te spatial metods wic analyze te image plane and te metods wic examines te frequency domain. Te spatial metods exploit assumptions aout eiter spatial or spectral correlation etween colors of neigors. Te frequency-selection metods apply specific filters on te CFA image to retrieve te color image. Since tese metods intend to produce perceptually satisfying demosaiced images, te most widely used evaluation criteria detailed in section 4 are ased on te fidelity to te original images. enerally, te Mean Square Error (MSE) and te Peak Signalto-Noise Ratio (PSNR) are used to measure te fidelity etween te demosaiced image and te original one. Te PSNR criterion cannot distinguis te case wen tere are Corresponding autor addresses: olivier.losson@univ-lille1.fr (O. Losson), ludovic.macaire@univ-lille1.fr (L. Macaire), yanqin.yang@etudiant.univ-lille1.fr (Y. Yang) Preprint sumitted to Advances in imaging and electron pysics 18 décemre 29

3 a ig numer of pixels wit sligt estimation errors from te case wen only a few pixels ave een interpolated wit severe demosaicing artifacts. However, te latter case would more significantly affect te result quality of a low-level analysis applied to te estimated image. Terefore, we propose new criteria especially designed to determine te most effective demosaicing metod for furter feature extraction. Te performances of te demosaicing metods are compared in section 5 tanks to te presented measurements. For tis purpose, te demosaicing scemes are applied to twelve images of te encmark Kodak dataase. 2

4 2. Color Demosaicing Digital images or videos are currently a preeminent medium in environment perception. Tey are today almost always captured directly y a digital (still) camera, rater tan digitized from a video signal provided y an analog camera as tey used to e several years ago. Acquisition tecniques of color images in particular ave involved muc researc work and undergone many canges. Despite major advancements, mass-market color cameras still often use a single sensor and require susequent processing to deliver color images. Tis procedure, named demosaicing, is te key point of our study and is introduced in te present section. Te demosaicing issue is first presented in detail, and a formalism is introduced for it Introduction to te Demosaicing Issue Te demosaicing issue is ere introduced from tecnological considerations. Two main types of color digital cameras are found on te market, depending on weter tey emed tree sensors or a single one. Usually known as mono-ccd cameras, te latter are equipped wit spectrally-sensitive filters arranged according to a particular pattern. From suc color filter arrays (CFA), an intermediate gray-scale image is formed, wic ten as to e demosaiced into a true color image. In te first susection are compared te major implementations of tree-ccd and mono-ccd tecnologies. Ten are presented te main types of colors filter arrays released y te various manufacturers. Proposed y Bayer from Kodak in 1976, te most widespread CFA is considered in te following, not only to formalize demosaicing ut also to introduce a pioneer metod using ilinear interpolation. Tis asic sceme generates many color artifacts, wic are analyzed to derive two main demosaicing rules. Spectral correlation is one of tem, and will e detailed in te last susection. Te second one, spatial correlation, is at te eart of edge-adaptive demosaicing metods, tat will e presented in te next section Mono-CCD vs. Tree-CCD Color Cameras Digital area scan cameras are devices ale to convert color stimuli from te oserved scene into a color digital image (or image sequence) tanks to potosensors. Suc an output image is spatially digitized, eing formed of picture elements (pixels). Wit eac pixel is generally associated a single potosensor element, wic captures te incident ligt intensity of te color stimulus. A digital color image I can e represented as a matrix of pixels, eac of tem eing denoted as P(x,y), were x and y are te spatial coordinates of pixel P witin te image plane of size X Y, ence (x,y) N 2 and x X 1, y Y 1. Wit eac pixel P is associated a color point, denoted as I(x,y) or I x,y. Tis color point is defined in te RB tree-dimensional color space y its tree coordinates I k x,y, k {R,,B}, wic represent te levels of te tricromatic components of te corresponding color stimulus. Te color image I may also e split into tree component planes or images I k, k {R,,B}. In eac component image I k, te pixel P is caracterized y level I k (P) for te single color component k. Tus, tree component images I R, I and I B, must e acquired in order to form any digital color image. 3

5 R dicroic prism stimulus CCD sensor B (a) Beam splitting y a tricroic prism assemly. Spectral sensitivity B(λ) (λ) R(λ) Wavelengt λ(nm) () Relative spectral sensitivity of te Kodak KLI sensor. Spectral sensitivity B c (λ) [B c ] c (λ) [ c ] R c (λ) [R c ] Wavelengt λ(nm) (c) CIE 1931 RB color matcing functions. [R c ], [ c ] and [B c ] are te monocromatic primary colors. FI. 1: Tree-CCD tecnology. Te two main tecnology families availale for te design of digital camera potosensors are CCD (Carge-Coupled Device) and CMOS (Complementary Metal-Oxide Semiconductor) tecnologies, te former eing te most widespread one today. Te CCD tecnology uses te potoelectric effect of te silicon sustrate, wile CMOS is ased on a potodetector and an active amplifier. Bot potosensors overall convert te intensity of te ligt reacing eac pixel into a proportional voltage. Additional circuits ten converts tis analog voltage signal into digital data. For illustration and explanation purposes, te following text relates to te CCD tecnology. Te various digital color cameras availale on te market may also e distinguised according to weter tey incorporate only a single sensor or tree. In accordance wit te tricromatic teory, tree-ccd tecnology incorporates tree CCD sensors, eac one eing dedicated to a specific primary color. In most devices, te color stimulus from te oserved scene is split onto te tree sensors y means of a tricroic 4

6 B R stimulus CMOS sensor (a) Wavelengt asorption witin te Foveon X3 sensor R(λ) Spectral sensitivity B(λ) (λ) Wavelengt λ(nm) () Relative spectral sensitivity of te Foveon X3 sensor endowed wit an infrared filter (Lyon and Huel, 22) FI. 2: Foveon X3 tecnology prism assemly, made of two dicroic prisms (see figure 1a)(Lyon, 2). Alternately, te incident eam may e dispatced on tree sensors, eac one eing covered wit a spectrally selective filter. Te tree component images I R, I and I B are simultaneously acquired y te tree CCD sensors, and teir comination leads to te final color image. Eac digital tree-ccd camera is caracterized y its own spectral sensitivity functions R(λ), (λ) and B(λ) (see figure 1 for an example), wic differ from te CIE color matcing functions of te standard oserver (see figure 1). Since 25, Foveon Inc. as een developing te X3 sensor, wic uses a multilayer CMOS tecnology. Tis new sensor is ased on tree superimposed layers of potosites emedded in a silicon sustrate. It takes advantage of te fact tat ligts of different wavelengts penetrate silicon to different depts (see figure 2a)(Lyon and Huel, 22). Eac layer ence captures one of te tree primary colors, namely lue, green and red, in te ligt incidence order. Te tree potosites associated wit eac pixel 5

7 tus provide signals from wic te tree component values are derived. Any camera equipped wit tis sensor is ale to form a true color image from tree full component images, as do tree-ccd-ased cameras. Tis sensor as een first used commercially in 27 witin te Sigma SD14 digital still camera. According to its manufacturer, its spectral sensitivity (see figure 2) etter fits wit te CIE color matcing functions tan tose of tree-ccd cameras, providing images tat are more consistent wit uman perception. Altoug tree-ccd and Foveon tecnologies yield ig quality images, te manufacturing costs of te sensor itself and of te optical device are ig. As a consequence, cameras equipped wit suc sensors ave not een so far affordale to everyone, nor widely distriuted. In order to overcome tese cost constraints, a tecnology using a single sensor as een developed. Te solution suggested y Bayer from te Kodak company in 1976 (Bayer, 1976) is still te most widely used in commercial digital cameras today. It uses a CCD or CMOS sensor covered y a filter (Color Filter Array, or CFA) designed as a mosaic of spectrally selective color filters, eac of tem eing sensitive to a specific wavelengt range. At eac element of te CCD sensor, only one out of te tree color components is sampled, Red (R), reen () or Blue (B) (see figure 3a). Consequently, only one color component is availale at eac pixel of te image provided y te CCD carge transfer circuitry. Tis image if often related to as te raw image, ut CFA image is preferred ereafter in our specific context. In order to otain a color image from te latter, two missing levels must e estimated at eac pixel tanks to a demosaicing algoritm (sometimes spelled demosaicking). As sown in figure 3, many oter processing tasks are classically acieved witin a mono-ccd color camera (Lukac and Plataniotis, 27). Tey consist for instance in raw sensor data correction or, after demosaicing, in color improvement, image sarpening and noise reduction, so as to provide a visually pleasing color image to te user. Tese processing tasks are essential to te quality of te provided image and, as a matter of fact, discriminate te various models of digital cameras, since manufacturers and models of sensors are not so numerous. Te related underlying algoritms ave common features or asis, and parameter tuning is often a key step leading to more or fewer residual errors. Togeter wit noise caracteristics of te imaging sensor, suc artifacts may incidentally e used to typify eac camera model (Bayrama et al., 28) Color Filter Arrays Several configurations may e considered for te CFA, and figure 4 sows some examples found in te literature. A few mono-ccd cameras use a CFA ased on complementary color components (Cyan, Magenta and Yellow), wit a 2 2 pattern wic also sometimes includes a filter sensitive to te green ligt. But te very large majority of cameras are equipped wit filter arrays ased on R, and B primary color components. Regardless of teir arrangement and design, tese arrays often include twice as many filters sensitive to te green primary as filters sensitive to lue or red ligt. Tis stems from Bayer s oservation tat te uman eye as a greater resolving power for green ligt. Moreover, te potopic luminous efficiency function of te uman retina also known as te luminosity function is similar to te CIE 1931 green matcing function c (λ), wit a maximum reaced in te same spectral domain. Bayer 6

8 CCD (Bayer) CFA stimulus B R CFA image Color Filter Array (CFA) (a) Mono-CCD tecnology outline, using te Bayer Color Filter Array (CFA). defective pixel correction linearization dark current compensation wite alance image compression EXIF file formatting CFA data image processing preprocessing postprocessing storage image CFA estimated color image CFA image digital zoom on te CFA image demosaicing estimated image post-processing digital zoom on te estimated image color correction sarpening and noise reduction () Image acquisition witin a mono-ccd color camera (detailed scema). Dotted steps are optional. FI. 3: Internal structure of a mono-ccd color camera. 7

9 (a) Vertical stripes () Bayer (c) Pseudo-random (d) Complementary colors (e) Pancromatic, or CFA2. (Kodak) (f) Burtoni CFA FI. 4: Configuration examples for te mosaic of color filters. Eac square depicts a pixel in te CFA image, and its color is tat of te monocromatic filter covering te associated potosite. terefore ot makes te assumption tat green potosensors capture luminance, wereas red and lue ones capture crominance, and suggests to fill te CFA wit more luminance-sensitive (green) elements tan crominance-sensitive (red and lue) elements (see figure 4). Te CFA using alternating vertical stripes (see figure 4a) of te RB primaries as een released first, since it is well suited to te interlaced television video signal. Neverteless, considering te Nyquist limits for te green component plane, Parulski (1985) sows tat te Bayer CFA as larger andwidt tan te latter for orizontal spatial frequencies. Te pseudo-random filter array (see figure 4c) as een inspired y te uman eye pysiology, in an attempt to reproduce te spatial repartition of te tree cone cell types on te retina surface (Lukac and Plataniotis, 25a). Its irregularity acieves a compromise etween te sensitivity to spatial variations of luminance in te oserved scene (visual acuity) and te aility to perceive tin ojects wit different colors (Roorda et al., 21). Indeed, optimal visual acuity would require potosensors wit identical spectral sensitivities wic are constant over te spectrum, wereas te perception of tin color ojects is etter ensured wit sufficient local density of different types of cones. Despite pseudo-random color filter arrays sow interesting properties (Alleysson et al., 28), teir design and exploitation ave not muc een investigated so far ; for some discussions, see e.g. Condat (29) or Savard (27) aout CFA design and Zapryanov and Nikolova (29) aout demosaicing of Bayer CFA pseudo-random variations. Among oter studies drawing teir inspiration from natural pysiology for CFA design, Kröger s work (24) yields a new mo- 8

10 1. Y(λ).8 Spectral sensitivity C(λ) (λ) M(λ) Wavelengtλ(nm) FI. 5: Relative spectral sensitivity of te JAI CV-S33P camera sensor (Jai Corporation, 2). saic wic mimics te retina of a ciclid fis, Astatotilapia urtoni (ünter, 1894). It is sown tat tis particular arrangement (see figure 4f), wic includes many spatial frequencies and different geometries for color components, generates weak aliasing artifacts. Tis complex mosaic configuration efficiently enances te simulated image quality (Medjeldi et al., 29), ut te effective implementation of suc a sensor, and te demosaicing step of te corresponding CFA images, are open and callenging prolems. Color filter arrays ased on complementary primary colors ave also een designed and used, wit two main advantages. First, tey own iger spectral sensitivity and wider andwidt tan RB filters, wic is of particular interest in noisy environments and/or wen te frame rate imposes low integration period (Hirakawa, 28). Figure 5 sows te spectral sensitivity of te JAI CV-S33P camera sensor, equipped wit te CFA of figure 4d. A few years ago, some professional still cameras used complementary color filter arrays to ensure ig ISO sensitivity, as te Kodak DCS-62x model equipped wit a CMY filter (Nole, 2). As a second advantage, tese CFAs make te generation of television luminance/croma video signal almost immediate, and are sometimes emedded in PAL or NTSC color video cameras (Sony Corporation, 2). Teir usage is owever largely restricted to television, since te strong mutual overlapping of C,M,Y spectral sensitivity functions makes te conversion into R,,B primaries unsatisfactory. New types of CFA ave recently een released, and used in camera models released y two major manufacturers. Since 1999, Fuji develops a new so-called Super CCD sensor, ased on potosites in a 45-degree oriented oneycom lattice (see figure 6). Te HR version of 23 (see figure 6a) allows to optimize te occupancy on te CCD surface, ence to potentially capture more ligt. Square pixels are otained from octagonal potosites y comining te four neigors in part, so tat new pixels are created and te resolution is douled. An alternative version of tis sensor 9

11 Created pixel R pixel Potosite S pixel R pixel S pixel Coupled pixels (a) Super CCD HR (23) () Super CCD SR (23) (c) Super CCD SRII (24) (d) Super CCD EXR (28) FI. 6: Super CCD tecnology. For clarity sake, potosites are represented furter apart from eac oter tan at teir actual location. (SR, see figure 6) as expanded dynamic range, y incorporating ot ig-sensitivity large potodiodes ( S-pixels ) used to capture normal and dark details, and smaller Rpixels sensitive to rigt details. Te EXR version (see figure 6d) takes advantage of same idea, ut extra efforts ave een conducted on noise reduction tanks to pixel inning, resulting in a new CFA arrangement and its exploitation y pixel coupling. As a proprietary tecnology, little tecnical detail is availale on ow Super CCD sensors turn te image into an orizontal/vertical grid witout interpolating, or on ow demosaicing associated wit suc sensors is acieved. A few ints may owever e found in a patent using a similar imaging device (Kuno and Sugiura, 26). In 27, Kodak develops new filter arrays (Hamilton and Compton, 27) as anoter alternative to te widely used Bayer CFA. Te asic principle of tis so-called CFA2. family of color filters is to incorporate transparent filter elements (represented as wite squares on figure 4e), tose filters eing ence also known as RBW or pancromatic ones. Tis property makes te underlying potosites sensitive to all wavelengts of te visile ligt. As a wole, te sensors associated wit CFA2. are terefore more sensitive to low-energy stimuli tan tose using Bayer CFA. Suc increase of gloal sensitivity leads to etter luminance estimation, ut at te expense of cromatic information estimation. Figure 7 sows te processing steps required to estimate a full color image from te data provided y a CFA2.-ased sensor. By modifying te CFA arrangement, manufacturers primarily aim at increasing te spectral sensitivity of te sensor. Lukac and Plataniotis (25a) tackled te CFA design issue y studying te influence of te CFA configuration on demosaicing results. Tey considered ten different RB color filter arrays, tree of tem eing sown on figures 4a to 4c. A CFA image is first simulated y sampling one out of te tree color components at eac pixel in an original color image, according to te considered CFA pattern. A universal demosaicing framework is ten applied to otain a full-color image. Te quality of te demosaiced image is finally evaluated y comparing it to te original image tanks to several ojective error criteria. Te autors conclude tat te CFA design is critical to demosaicing quality results, ut cannot advise any CFA tat would yield est results in all cases. Indeed, te relative performance of filters is igly dependent on te tested image. 1

12 P P B P P P P B averaging B demosaicing B B R R R RB B R R R B P B P B P B P CFA image P P P P B P B P interpolation Color component (reduced resolution) Color image B P B P B P B P R P R P R P R P P P P P P P pixels (reduced resolution) R PR P B B P R P B PR P B PR P B P P P P P P R P R P R P B PR P B PR P B P P P P P P P Crominance-luminance R P R P R P R P image P P averaging (reduced resolution) Crominance-luminance Luminance image image (reduced resolution) (full resolution) P interpolation P P P Pancromatic pixels P P P P P P P P P P P P P P Luminance image (full resolution) + B B B B R RB RB RB B R RB RB RB B R RB RB RB B R R R R Color image (full resolution) FI. 7: Processing steps of te raw image provided y a CFA2.-ased sensor. Pancromatic pixels are tose associated wit potosites covered wit transparent filters. All in all, te Bayer CFA acieves a good compromise etween orizontal and vertical resolutions, luminance and crominance sensitivities, and terefore remains te favorite CFA in industrial applications. As tis CFA is te most commonly used and as inspired some more recent ones, it will e considered first and foremost in te following text. Demosaicing metods presented ereafter are notaly ased on te Bayer CFA Demosaicing Formalization Estimated colors ave less fidelity to color stimuli from te oserved scene tan tose provided y a tree-ccd camera. Improving te quality of color images acquired y mono-ccd cameras is still a igly relevant topic, investigated y researcers and engineers (Lukac, 28). In tis paper, we focus on te demosaicing step and examine its influence on te estimated image quality. In order to set a formalism for te demosaicing process, let us compare te acquisition process of a color image in a tree-cdd camera and in a mono-ccd camera. Figure 8a outlines a tree-ccd camera arcitecture, in wic te color image of a scene is formed y comining te data from tree sensors. Te resulting color image I is composed of tree color component planes I k, k {R,,B}. In eac plane I k, a given pixel P is caracterized y te level of te color component k. A tree-component vector defined as I x,y (R x,y, x,y,b x,y ) is terefore associated wit eac pixel located at spatial coordinates (x,y) in image I. In a color mono-ccd camera, te color image generation is quite different, as sown in figure 8 : te single sensor delivers a raw image, ereafter called CFA image and denoted I CFA. If te Bayer CFA is considered, to eac pixel wit coordinates (x,y) in image I CFA is associated a single color 11

13 R sensor R image scene optical device sensor B sensor image color image I (a) Tree-CCD camera B image scene optical device cf sensor demosaicing CFA filter () Mono-CCD color camera CFA image I CFA estimated color image Î FI. 8: Color image acquisition outline, according to te camera type. component R, or B (see figure 9) : Ix,y CFA = R x,y if x is odd and y is even, (1a) B x,y if x is even and y is odd, (1) x,y oterwise. (1c) Te color component levels range from to 255 wen tey are quantized wit 8 its. Te demosaicing sceme F, most often implemented as an interpolation procedure, consists in estimating a color image Î from I CFA. At eac pixel of te estimated image, te color component availale in I CFA at te same pixel location is picked up, wereas te oter two components are estimated : I CFA x,y F Î x,y = (R x,y,ĝ x,y, ˆB x,y ) if x is odd and y is even, (2a) ( ˆR x,y,ĝ x,y,b x,y ) if x is even and y is odd, (2) ( ˆR x,y, x,y, ˆB x,y ) oterwise. (2c) Eac triplet in equations (2) stands for a color, wose color component availale at pixel P(x,y) in I CFA is denoted R x,y, x,y or B x,y, and wose oter two components among ˆR x,y, Ĝ x,y and ˆB x,y are estimated for Î x,y. Before we get to te eart of te matter, let us still precise a few notations tat will e most useful later in tis section. In te CFA image (see figure 9), four different 12

14 , R 1, 2, R 3,... 4, B,1 1,1 B 2,1 3,1 B... 4,1,2 R 1,2 2,2 R 3,2... 4,2 B,3 1,3 B 2,3 3,3 B... 4,3...,4 R... 1,4... 2,4 R... 3,4... 4,4 FI. 9: CFA image from te Bayer filter. Eac pixel is artificially colorized wit te corresponding filter main spectral sensitivity, and te presented arrangement is te most frequently encountered in te literature (i.e. and R levels availale for te first two row pixels). B 1, 1, 1 B 1, 1 R 1, 1, 1 R 1, 1 1, R, 1, 1, B, 1, B 1,1,1 B 1,1 R 1,1,1 R 1,1 (a) {R} () {B} 1, 1 B, 1 1, 1 1, 1 R, 1 1, 1 R 1,, R 1, B 1,, B 1, 1,1 B,1 1,1 1,1 R,1 1,1 (c) {RR} (d) {BB} FI. 1: 3 3 neigorood structures of pixels in te CFA image. 13

15 structures are encountered for te 3 3 spatial neigorood, as sown on figure 1. For eac of tese structures, te pixel under consideration for demosaicing is te central one, at wic te two missing color components sould e estimated tanks to te availale components and teir levels at te neigoring pixels. Let us denote te aforementioned structures y te color components availale on te middle row, namely {R}, {B}, {RR} and {BB}. Notice tat {R} and {B} are structurally similar, apart from te sligt difference tat components R and B are excanged. Terefore, tey can e analyzed in te same way, as can {RR} and {BB} structures. A generic notation is ence used in te following : te center pixel is considered aving (,) spatial coordinates, and its neigors are referred to using teir relative coordinates (δx,δy). Wenever tis notation ears no amiguity, (,) coordinates are omitted. Moreover, we also sometimes use a letter (e.g. P) to generically refer to a pixel, its color components eing ten denoted as R(P), (P) and B(P). Te notation P(δx,δy) allows to refer to a pixel tanks to its relative coordinates, its colors components eing ten denoted R δx,δy, δx,δy and B δx,δy, as in figure Demosaicing Evaluation Outline Demosaicing ojective is to generate an estimated color image Î as close as possile to te original image I. Even tis image is unavailale effectively, I is generally used as a reference to evaluate te demosaicing quality. Ten, one eiter strive to otain as a low value as possile for an error criterion, or as a ig value as possile for a quality criterion comparing te estimated image and te original one. A classical evaluation procedure for te demosaicing result quality consists in (see figure 11) : 1. simulating a CFA image provided y a mono-ccd camera from a color original image provided y a tree-ccd camera. Tis is acieved y sampling a single color component R, or B at eac pixel, according to te considered CFA arrangement (Bayer CFA of figure 9, in our case) ; 2. demosaicing tis CFA image to otain an estimated color image ; 3. comparing te original and estimated color images, so as to igligt artifacts affecting te latter. Tere is no general agreement on te demosaicing quality definition, wic is igly dependent upon te estimated color image exploitation as will e detailed in te next sections. In a first time, we will rely on visual examination, or else on te most used quantitative criterion (signal-to-noise ratio) for a quality result evaluation, wic ot require a reference image. As in most works related to demosaicing, we will ere use te Kodak image dataase (Kodak, 1991) as a encmark for performance comparison of te various metods, as well as for illustration purposes. More precisely, to avoid overloaded results, a representative suset of twelve of tese images as een picked up as te most used set in literature. Tese natural images contain ric colors and textural regions, and are fully reproduced in figure 37 so tat tey can e referred to in te text. 14

16 Original image I 1. Color sampling (simulated) CFA image I CFA 2. Demosaicing Estimated image Î 3. Comparison according to criteria FI. 11: Classical evaluation procedure for te demosaicing result quality (example of ilinear interpolation on an extract from te Kodak encmark image Ligtouse ) Basic Scemes and Demosaicing Rules Bilinear Interpolation Te first solutions for demosaicing were proposed in te early eigties. Tey process eac component plane separately and find te missing levels y applying linear interpolation on te availale ones, in ot main directions of te image plane. Suc a ilinear interpolation is traditionally used to resize gray-level images (rion and Bailey, 24). Considering te {R} structure, te missing lue and green values at te center pixel are respectively estimated y ilinear interpolation tanks to te following equations : ˆB = 1 4 (B 1, 1 + B 1, 1 + B 1,1 + B 1,1 ), (3) Ĝ = 1 4 (, 1 + 1, + 1, +,1 ). (4) As for te {RR} structure, te missing red and lue component levels are estimated as follows : ˆR = 1 2 (R 1, + R 1, ), (5) ˆB = 1 2 (B, 1 + B,1 ). (6) Alleysson et al. (28) notice tat suc interpolation is acievale y convolution. For tat purpose, consider te tree planes formed of te sole levels of component k, k {R,,B}, availale in te CFA image, oter component levels eing set to zero. Let 15

17 R R R R B B B B B B R R R R B B B B B B R R (a) I CFA R R () ϕ R ( I CFA) (c) ϕ ( I CFA) (d) ϕ B ( I CFA) FI. 12: Definition of planes ϕ k ( I CFA) y sampling te CFA image according to eac color component k, k {R,,B}. Te CFA image and planes ϕ k ( I CFA) are ere colorized for illustration sake. us denote ϕ k (I) te function sampling a gray-level image I according to te locations of te availale color component k in te CFA : ϕ k (I)(x,y) = { I(x,y) if component k is availale at pixel P(x,y) in I CFA, oterwise. (7) Figure 12 illustrates te special cases of planes ϕ k (I CFA ) otained y applying functions ϕ k to I CFA. Let us also consider te convolution filters defined y te following kernels : H R = H B = (8) and H = (9) In order to determine te color image Î, eac color component plane Î k can now e estimated y applying te convolution filter of kernel H k on te plane ϕ k ( I CFA), respectively : Î k = H k ϕ k (I CFA ), k {R,,B}. (1) Bilinear interpolation is easy to e implemented and not processing time consuming, ut generates severe visile artifacts, as also sown in figure 11. Te aove sceme provides satisfying results in image areas wit omogeneous colors, ut many false colors in areas wit spatial ig frequencies as for te fence ars in tis extract. Following Cang and Tan (26), a deep study of te causes of teses artifacts can e acieved y simulating teir generation on a syntetic image (see figure 13a). In tis original image, two omogeneous areas are separated y a vertical transition, wic recreates te oundary etween two real ojects wit different gray levels. At eac pixel, te levels of all tree color components are ten equal. Levels of pixels depicting te darker left oject (laeled as ) are lower tan tose of pixels depicting te ligter rigt oject (laeled as ). Figure 13 sows te CFA image I CFA yielded y sampling 16

18 R R R B B R R R B B R R R (a) Original image () CFA image (c) Estimated image (d) ˆR plane (e) Ĝ plane (f) ˆB plane FI. 13: Demosaicing y ilinear interpolation of an gray-level image wit a vertical transition. Te CFA image and ˆR, Ĝ and ˆB planes are ere colorized for illustration sake. a single color component per pixel according to te Bayer CFA. Te result of ilinear interpolation demosaicing applied to tis image is given on figure 13c. Figures 13d to 13f give details on te tree estimated color planes ˆR, Ĝ and ˆB. On ˆR and ˆB planes, tis demosaicing algoritm generates a column of intermediate-level pixels, wose value is te average of te two oject levels. On te green plane, it produces a jagged pattern on ot edge sides, formed of pixels alternating etween two intermediate levels a low one (3+)/4 and a ig one (3+)/4. As a wole, te edge area is formed of a square 2 2 pattern of four different colors repeated alongside te transition (see te estimated image in figure 13c). Tis demosaicing procedure as ence generated two types of artifacts : erroneously estimated colors (ereafter referred to as false colors ), and an artificial jagged pattern (so-called zipper effect ), wic are ot studied in section 4.2. According to te orizontal transition location relative to te CFA mosaic, te generated pattern may e eiter orange-colored as in figure 13c or wit luis colors as in figure 14c. Tese two dominant-color patterns may e actually oserved in te estimated image of figure Main Demosaicing Rules Let us examine te component-wise profiles of te middle pixel row in te original image 13a and its corresponding estimated image 13c. Dissimilarities etween tese profiles on R, and B planes are underlined on figure 15 : te transition occurs at identical orizontal locations on te tree original image planes, ut tis is no more te case for te estimated image. Suc inconsistency among te demosaicing results for different components generates false colors in te estimated image formed from 17

19 R R R B B R R R B B R R R (a) Refrence image () CFA image (c) Estimated image FI. 14: Variant version of image 13a, demosaiced y ilinear interpolation as well. teir comination. It can also e noticed tat te transition corresponds, in eac color plane of te original image, to a local cange of omogeneity along te orizontal direction. Bilinear interpolation averages te levels of pixels located on ot sides of te transition, wic makes te latter less sarp. In accordance wit te previous oservations, we can state tat two main rules ave to e enforced so as to improve demosaicing results : spatial correlation and spectral correlation. Spectral correlation. Te transition profiles plotted in figure 15 are identical for te original image component planes, wic conveys strict correlation etween components. For a natural image, unturk et al. (22) sow tat te tree color components are also strongly correlated. Te autors apply a idimensional filter uilt on a low-pass filter = [1 2 1]/4 and a ig-pass one 1 = [1 2 1]/4, so as to split eac color component plane into four suands resulting from row and column filtering : (LL) ot rows and columns are low-pass filtered ; (LH) rows are low-pass and columns ig-pass filtered ; (HL) rows are ig-pass and columns low-pass filtered ; (HH) ot rows and columns are ig-pass filtered. For eac color component, four suand planes are otained in tis way, respectively representing data in rater omogeneous areas (low-frequency information), orizontal detail (ig-frequency information in te orizontal direction), vertical detail (ig-frequency information in te vertical direction) and diagonal detail (ig-frequency information in ot main directions). Te autors ten compute a correlation coefficient r R, etween red and green components over eac suand according to te following formula : r R, = X 1 x= X 1Y 1 x= y= Y 1 y= ( Rx,y µ R)( x,y µ ) (R x,y µ R ) 2 X 1Y 1 x= y= ( x,y µ ) 2, (11) in wic R x,y (respectively x,y ) is te level at (x,y) pixel in te red (respectively green) component plane witin te same suand, µ R and µ eing te average of R x,y and x,y levels over te same suand planes. Te correlation coefficient etween te lue and green components is similarly computed. Test results on 18

20 A A A A R B (a) Original image () Estimated image FI. 15: Component-wise profiles of middle pixel row levels A-A in te original and estimated images. Black dots stand for availale levels, and wite dots for estimated levels. 19

21 twenty natural images sow tat tose coefficients are always greater tan.9 in suands carrying spatial ig frequencies at least in one direction (i.e. LH, HL and HH). As for te suand carrying low frequencies (LL), coefficients are lower ut always greater tan.8. Tis reveals a very strong correlation etween levels of different color components in a natural image, especially in areas wit ig spatial frequencies. Lian et al. (26) confirm, using a wavelet coefficient analysis, tat ig-frequency information is not only strongly correlated etween te tree component planes, ut almost identical. Suc spectral correlation etween components sould e taken into account to retrieve te missing components at a given pixel. Spatial correlation. A color image can e viewed as a set of adjacent omogeneous regions wose pixels ave similar levels for eac color component. In order to estimate te missing levels at eac considered pixel, one terefore sould exploit te levels of neigoring pixels. However, tis task is difficult at pixels near te order etween two distinct regions due to ig local variation of color components. As far as demosaicing is concerned, tis spatial correlation property avoids to interpolate missing components at a given pixel tanks to neigor levels wic do not elong to te same omogeneous region. Tese two principles are generally taken into account sequentially y te demosaicing procedure. In te first step, demosaicing often consists in estimating te green component using spatial correlation. According to Bayer s assumption, te green component as denser availale data witin te CFA image, and represents te luminance of te image to e estimated. Estimation of red and lue components (assimilated to crominance) is only acieved in a second step, tanks to te previously interpolated luminance and using te spectral correlation property. Suc a way of using ot correlations is used y a large numer of metods in te literature. Also notice tat, altoug red and lue component interpolation is acieved after te green plane as een fully populated, spectral correlation is also often used in te first demosaicing step to improve te green plane estimation quality Spectral Correlation Rules In order to take into account te strong spectral correlation etween color components at eac pixel, two main ypoteses are proposed in te literature. Te first one assumes a color ratio constancy and te second one is ased on color difference constancy. Let us examine te underlying principles of eac of tese assumptions efore comparing ot. Interpolation ased on color ue constancy, suggested y Cok (1987), is istorically te first one ased on spectral correlation. According to Cok, ue is understood as te ratio etween crominance and luminance, i.e. R/ or B/. His metod proceeds in two steps. In te first step, missing green values are estimated y ilinear interpolation. Red (and lue) levels are ten estimated y weigting te green level at te given pixel wit te ue average of neigoring pixels. For instance, interpolation of te lue level at te center pixel of {R} CFA structure (see figure 1a) uses te four diagonal 2

22 neigors were tis lue component is availale : ˆB = Ĝ 1 4 [ B 1, 1 Ĝ 1, 1 + B 1, 1 Ĝ 1, 1 + B 1,1 Ĝ 1,1 + B 1,1 Ĝ 1,1 ]. (12) Tis ilinear interpolation etween color component ratios is ased on te local constancy of tis ratio witin an omogeneous region. Kimmel (1999) justifies te color ratio constancy assumption tanks to a simplified approac tat models any color image as a Lamertian oject surface oservation. According to te Lamertian model, suc a surface reflects te incident ligt to all directions wit equal energy. Te intensity I(P) received y te potosensor element associated to eac pixel P is terefore independent of te camera position, and can e represented as : I(P) = ρ N(P), l, (13) were ρ is te aledo (or reflection coefficient), N(P) is te normal vector to te surface element wic is projected on pixel P, and l is te incident ligt vector. As te aledo ρ caracterizes te oject material, tis quantity is different for eac color component (ρ R ρ ρ B ), and te tree color components may e written as : I R (P) = ρ R N(P), l, (14) I (P) = ρ N(P), l, (15) I B (P) = ρ B N(P), l. (16) Assuming tat any oject is composed of one single material, coefficients ρ R, ρ and ρ B are ten constant at all pixels representing an oject. So, te ratio etween two color components is also constant : K k,k = Ik (P) ρk N(P), l I k (P) = = ρk = constant, (17) ρ k N(P), l ρ k were (k,k ) {R,,B} 2. Altoug tis assumption is simplistic, it is locally valid and can e used witin te neigorood of te considered pixel. Anoter simplified and widely used model of correlation etween components relies on te color difference constancy assumption. At a given pixel, tis can e written as : D k,k = I k (P) I k (P) = ρ k N(P), l ρ k N(P), l = constant, (18) were (k,k ) {R,,B} 2. As te incident ligt direction and amplitude are assumed to e locally constant, te color component difference is also constant witin te considered pixel neigorood. 21

23 As a consequence, te crominance interpolation step in Cok s metod may e rewritten y using component difference averages, for instance : ˆB = Ĝ+ 1 [ (B 1, 1 Ĝ 1, 1 )+(B 1, 1 Ĝ 1, 1 )+(B 1,1 Ĝ 1,1 )+(B 1,1 Ĝ 1,1 ) ], 4 (19) instead of equation (12). Te validity of tis approac is also justified y Lian et al. (27) on te ground of spatial ig frequency similarity etween color components. Te color difference constancy assumption is gloally consistent wit te ratio rule used in formula (12). By considering te logaritmic non-linear transformation, te difference D k,k 2,(k,k ) {R,,B} 2, can e expressed as : ( I k ) D k,k (P) ( ) ( ) 2 = log 1 = log I k (P) 1 I k (P) log 1 I k (P). (2) Furtermore, we propose to compare tose two assumptions expressed y equations (17) and (18). In order to take into account spectral correlation for demosaicing, it turns out tat te difference of color components presents some enefits in comparison to teir ratio. Te latter is indeed error-prone wen its denominator takes low values. Tis appens for instance wen saturated red and/or lue components lead to comparatively low values of green, making te ratios in equation (12) very sensitive to red and/or lue lue small variations. Figure 16a is a natural image example wic is igly saturated in red. Figures 16c and 16d sow te images were eac pixel value is, respectively, te component ratio R/ and difference R (pixel levels eing normalized y linear dynamic range stretcing). It can e noticed tat tese two images actually carry out less ig-frequency information tan te green component plane sown on figure 16. A Soel filter is ten applied to tese two images, so as to igligt te igfrequency information location. Te Soel filter output module is sown on figures 16e and 16f. In te rigt-and parrot plumage area were red is saturated, te component ratio plane contains more ig-frequency information tan te component difference plane, wic makes it more artifact-prone wen demosaiced y interpolation. Moreover, ig color ratio values may yield to estimated component levels eyond te data ounds, wic is undesirale for te demosaicing result quality. To overcome tese drawacks, a linear translation model applied on all tree color components is suggested y Lukac and Plataniotis (24a, 24). Instead of equation (17), te autors reformulate te color ratio rule y adding a predefined constant value β to eac component. Te new constancy assumption, wic is consistent wit equation (17) in omogeneous areas, now relies on te ratio : K k,k 2 = Ik + β I k + β, (21) were (k,k ) {R,,B} 2, and were β N is a ratio normalization parameter. Under tis new assumption on te normalized ratio, te lue level interpolation formulated in 22

24 (a) Original image () plane (c) R/ ratio plane (d) R difference plane (e) Soel filter output on te R/ plane (f) Soel filter output on te R plane FI. 16: Component ratio and difference planes on a same image ( Parrots from te Kodak dataase). 23

25 equation (12) under te ratio rule now ecomes 1 : ˆB = β + ( Ĝ+β ) 1 [ 4 B 1, 1 + β Ĝ 1, 1 + β + B 1, 1 + β Ĝ 1, 1 + β + B 1,1 + β Ĝ 1,1 + β + B ] 1,1 + β. (22) Ĝ 1,1 + β In order to avoid too different values for te numerator and denominator, Lukac and Plataniotis advise to set β = 256, so tat te normalized ratios R/ and B/ range from.5 to 2. Tey claim tat tis assumption improves te interpolation quality in areas of transitions etween ojects and of tin details. In our investigation of te two main assumptions used for demosaicing, we finally compare te estimated image quality in ot cases. Te procedure depicted on figure 11 is applied on twelve natural images selected from Kodak dataase : te demosaicing scemes presented aove, respectively using component ratio and difference, are applied to te simulated CFA image. To evaluate te estimated color image quality in comparison wit te original image, we ten compute an ojective criterion, namely te peak signal-to-noise ratio (PSNR) derived from te mean square error (MSE) etween te two images. On te red plane for instance, tese quantities are defined as : MSE R = 1 XY X 1 x= Y 1 y= PSNR R = 1 log 1 ( ( I R x,y Îx,y R ) 2, (23) MSE R ). (24) As te green component is ilinearly interpolated witout using spectral correlation, only red and lue estimated levels vary according to te considered assumption. Te PSNR is ence computed on tese two planes. Results displayed in tale 1 sow tat using te color difference assumption yields etter results tan using te simple ratio rule K, wic is particularly noticeale for image Parrots of figure 16a. Te normalized ratio K 2, wic is less prone to large variations tan K in areas wit spatial ig frequencies, leads to iger values for PSNR R and PSNR B. However, te color difference assumption generally outperforms ratio-ased rules according to te PSNR criterion, and is most often used to exploit spectral correlation in demosaicing scemes. 1 Te autors use, in tis interpolation formula, extra weigting factors depending on te local pattern and dropped ere for conciseness. 24

26 Image PSNR R PSNR B D K K 2 D K K 2 1 ( Parrots ) ( Sailoats ) ( Windows ) ( Houses ) ( Race ) ( Pier ) ( Island ) ( Ligtouse ) ( Plane ) ( Cape ) ( Barn ) ( Calet ) Average TAB. 1: Peak signal-to-noise ratios (in deciels) for red (PSNR R ) and lue (PSNR B ) planes of twelve Kodak images (Eastman Kodak and various potograpers, 1991), demosaiced under te color difference D (see equation (18) and interpolation formula (19)), under te color ratio K (see equation (17) and interpolation formula (12)) and under te normalized ratio K 2 (β = 256) (see equation (21) and interpolation formula (22)) constancy rules. For eac color component and image, te value printed in old typeface igligts te est result. 25

27 3. Demosaicing Scemes In tis section, te main demosaicing scemes proposed in te literature are descried. We distinguis two main procedures families, according to weter tey scan te image plane or ciefly use te frequency domain Edge-adaptive Demosaicing Metods Estimating te green plane efore R and B ones is mainly motivated y te doule amount of samples in te CFA image. A fully populated component plane will susequently make te R and B plane estimation more accurate. As a consequence, te component estimation quality ecomes critical in te overall demosaicing performance, since any error in te plane estimation is propagated in te following crominance estimation step. Important efforts are terefore devoted to improve te estimation quality of te green component plane usually assimilated to luminance, especially in ig-frequency areas. Practically, wen te considered pixel lies on an edge etween two omogeneous areas, missing components sould e estimated along te edge rater tan across it. In oter words, neigoring pixels to e taken into account for interpolation sould not elong to distinct ojects. Wen exploiting te spatial correlation, a key issue is to determine te edge direction from CFA samples. As demosaicing metods presented in te following text generally use specific directions and neigoroods in te image plane, some useful notations are introduced in figure radient-ased Metods radient computation is a general solution to edge direction selection. Hiard s metod (1995) uses orizontal and vertical gradients, computed at eac pixel were te component as to e estimated, in order to select te direction wic provides te est green level estimation. Let us consider te {R} CFA structure for instance (see figure 1a). Estimating te green level Ĝ at te center pixel is acieved in two successive steps : 1. Approximate te gradient module (ereafter simply referred to as gradient for simplicity) according to orizontal and vertical directions, as : 2. Interpolate te green level as : x = 1, 1,, (25) y =, 1,1. (26) Ĝ = ( 1, + 1, )/2 if x < y, (27a) (, 1 +,1 )/2 if x > y, (27) (, 1 + 1, + 1, +,1 )/4 if x = y. (27c) Laroce and Prescott (1993) suggest to consider a 5 5 neigorood for partial derivative approximations tanks to availale surrounding levels, for instance x = 2R R 2, R 2,. Moreover, Hamilton and Adams (1997) comine ot approaces. 26

28 2 x x x x y y 2 y 2 y (a) Directions () N 4 neigorood (c) N 4 neigorood P(δx,δy) N 4 (δx,δy) {(, 1), ( 1,), (1,), (,1)} P(δx,δy) N 4 (δx,δy) {( 1, 1), (1, 1), ( 1,1), (1,1)} x x y (d) N 8 neigorood (N 8 N 4 N 4 ) y (e) N 9 pixel set (N 9 N 8 P(,)) FI. 17: Notations for te main spatial directions and considered pixel neigoroods. R 2, 2 1, 2 R, 2 1, 2 R 2, 2 2, 1 B 1, 1, 1 B 1, 1 2, 1 R 2, 1, R, 1, R 2, 2,1 B 1,1,1 B 1,1 2,1 R 2,2 1,2 R,2 1,2 R 2,2 FI. 18: 5 5 neigorood wit central {R} structure in te CFA image. 27

29 To select te interpolation direction, tese autors take into account ot gradient and Laplacian second-order values y using te green levels availale at neary pixels and red (or lue) samples located 2 pixels apart. For instance, to estimate te green level at {R} CFA structure (see figure 18), Hamilton and Adams use te following algoritm : 1. Approximate te orizontal x and vertical y gradients tanks to asolute differences as : x = 1, 1, + 2R R 2, R 2,, (28) y =, 1,1 + 2R R, 2 R,2. (29) 2. Interpolate te green level as : ( 1, + 1, )/2+(2R R 2, R 2, )/4 if x < y, (3a) (, 1 +,1 )/2+(2R R, 2 R,2 )/4 if x > y, (3) Ĝ = (, 1 + 1, + 1, +,1 )/4 +(4R R, 2 R 2, R 2, R,2 )/8 if x = y. (3c) Tis proposal outperforms Hiards metod. Indeed, precision is gained not only y comining two color component data in partial derivative approximations, ut also y exploiting spectral correlation in te green plane estimation. It may e noticed tat formula (3a) for te orizontal interpolation of green component may e split into one left Ĝ g and one rigt Ĝ d side parts : Ĝ g = 1, +(R R 2, )/2, (31) Ĝ d = 1, +(R R 2, )/2, (32) (Ĝg Ĝ = + Ĝ d) /2. (33) Suc interpolation is derived from te color difference constancy assumption, and ence exploits spectral correlation for green component estimation. Also notice tat, in tese equations, orizontal gradients are assumed to e similar for ot red and lue components. A complete formulation as een given y Li and Randawa (25). As tese autors sow esides, te green component may more generally e estimated y a Taylor series as long as green levels are considered as a continuous function g wic is differentiale in ot main directions. Te aove equations (31) and (32) may ten e seen as first-order approximations of tis series. Indeed, in Ĝ g case for instance, te orizontal approximation is written as g(x) = g(x 1) + g (x 1) g(x 1)+(g(x) g(x 2))/2. Using te local constancy property of color component difference yields Ĝ x Ĝ x 2 = R x R x 2, from wic expression (31) is derived. Li and Randawa suggest an approximation ased on te second-order derivative, g estimation ecoming : Ĝ g = 1, +(R R 2, )/2+(R R 2, )/4 ( 1, 3, )/4, (34) 28

30 for wic a neigorood size of 7 7 pixels is required. Te additional term compared to (31) enales to refine te green component estimation. Similar reasoning may e used to select te interpolation direction. According to te autors, increasing te approximation order in suc a way improves estimation results under te mean square error (MSE) criterion. Anoter proposal comes from Su (26), namely to interpolate te green level as a weigted sum of values defined y equations (3a) and (3). Naming te latter respectively Ĝ x = ( 1, + 1, )/2 + (2R R 2, R 2, )/4 and Ĝ y = (, 1 +,1 )/2 + (2R R, 2 R,2 )/4, orizontal and vertical interpolations are comined as : { w1 Ĝ x + w Ĝ = 2 Ĝ y if x < y, (35a) w 1 Ĝ y + w 2 Ĝ x if x > y, (35) were w 1 and w 2 are te weigting factors. Expression (3c) remains uncanged (i.e. Ĝ = ( Ĝ x + Ĝ y) /2 if x = y ). Te smallest level variation term must e weigted y te igest factor (i.e. w 1 > w 2 ) ; expressions (3a) and (3) incidentally correspond to te special case w 1 = 1, w 2 =. Incorporating terms associated to ig level variations allows to undertake ig-frequency information in te green component interpolation expression itself. Su sets w 1 to.87 and w 2 to.13, since tese weigting factor values yield te minimal average MSE (for te tree color planes) over a large series of demosaiced images. Oter researcers, like Hirakawa and Parks (25) or Menon et al. (27), use te filterank approac in order to estimate missing green levels, efore selecting te orizontal or vertical interpolation direction at {R} and {B} CFA structures. Tis enales to design five-element mono-dimensional filters wic are optimal towards criteria specifically designed to avoid interpolation artifacts. Te proposed optimal filters (e.g. opt = [ ] for Hirakawa and Parks sceme) are close to te formulation of Hamilton and Adams Component-consistent Demosaicing Hamilton and Adam s metod selects te interpolation direction on te asis of orizontal and vertical gradient approximations. But tis may e inappropriate, and unsatisfying results may e otained in areas wit textures or tin ojects. Figure 19 sows an example were orizontal x and vertical y gradient approximations do not allow to take te rigt decision for te interpolation direction. Wu and Zang (24) propose a more reliale way to select tis direction, still y using a local neigorood. Two candidate levels are computed to interpolate te missing green value at a given pixel : one using orizontal neigors, te second using vertical neigoring pixels. Ten, te missing R or B value is estimated in ot orizontal and vertical directions wit eac of tese candidates. A final step consists in selecting te most appropriate interpolation direction, namely tat minimizing te gradient sum on te color difference planes (R and B ) in te considered pixel neigorood. Tis interpolation direction 2 No detail will e ere given aout ow R and B components are estimated y te aove metods, for teir originality mainly lies in te component estimation. 29

31 25 2 R 2, R R 2, , 1, x = 1, 1, + 2R R 2, R 2, = x 25 2 R, 2 R R, , 1,1 y =, 1,1 + 2R R 2, R 2, = y FI. 19: Direction selection issue in Hamilton and Adams interpolation sceme (1997), on an extract of te original image Ligtouse containing tin details. Plots igligt te R and component values used for orizontal and vertical gradient computations : color dots represent availale levels in te CFA image, wereas wite dots are levels to e estimated. As x < y, orizontal neigoring pixels are wrongly used in Ĝ estimation. Tis is sown on te lower rigt sufigure, togeter wit te erroneous demosaicing result (at center pixel only). 3

32 allows to select te levels computed eforeand to e taken into account for te missing component estimation. More precisely, Wu and Zang s approac proceeds in te following steps : 1. At eac pixel were te green component is missing, compute two candidate levels : one denoted as Ĝ x y using te orizontal direction (according to equation (3a)), and anoter Ĝ y y using te vertical direction (according to (3)). For oter pixels, set Ĝ x = Ĝ y =. 2. At eac pixel were te green component is availale, compute two candidate levels (one orizontal and one vertical) for eac of te missing red and lue components. At {RR} CFA structure tese levels are expressed as (see figure 1c) : ˆR x = (R 1, Ĝ x 1, + R 1, Ĝ x 1, ), (36) ˆR y = (R 1, Ĝ y 1, + R 1, Ĝ y 1, ), (37) ˆB x = (B, 1 Ĝ x, 1 + B,1 Ĝ x,1 ), (38) ˆB y = (B, 1 Ĝ y, 1 + B,1 Ĝ y,1 ). (39) 3. At eac pixel wit missing green component, compute two candidate levels for te missing crominance component (i.e. ˆB at R samples, and conversely). At {R} CFA structure, te lue levels are estimated as (see figure 1a) : ˆB x = Ĝ x (B(P) Ĝ x (P)), (4) P N 4 ˆB y = Ĝ y (B(P) Ĝ y (P)), (41) P N 4 were N 4 is composed of te four diagonal pixels (see figure 17c). 4. Acieve te final estimation at eac pixel P y selecting one component triplet out of te two candidates computed eforeand in ot orizontal and vertical directions. So as to use te direction for wic variations of (R ) and (B ) component differences are minimal, te autors suggest te following selection criterion : { ( ˆR ( ˆR,Ĝ, ˆB) x,ĝ x, ˆB x ) if x < y, (42a) = ( ˆR y,ĝ y, ˆB y ) if x y, (42) were x and y are, respectively, te orizontal and vertical gradients on te difference plane of estimated colors. More precisely, tese gradients are computed y considering all distinct (Q,Q ) pixel pairs, respectively row-wise and column-wise, witin te 3 3 window centered at P (see figure 17e) : 31

33 x = (Q,Q ) N 9 N 9 y(q)=y(q ) y = (Q,Q ) N 9 N 9 x(q)=x(q ) ( ˆR x (Q) Ĝ x (Q) ) ( ˆR x (Q ) Ĝ x (Q ) ) + ( ˆB x (Q) Ĝ x (Q) ) ( ˆB x (Q ) Ĝ x (Q ) ), (43) ( ˆR y (Q) Ĝ y (Q) ) ( ˆR y (Q ) Ĝ y (Q ) ) + ( ˆB y (Q) Ĝ y (Q) ) ( ˆB y (Q ) Ĝ y (Q ) ). (44) Tis metod uses te same expressions as Hamilton and Adams ones in order to estimate missing color components, ut improves te interpolation direction decision y using a 3 3 window rater tan a single row or column in wic te gradient of color differences (R and B ) is evaluated so as to minimize its local variation. Among oter attempts to refine te interpolation direction selection, Hirakawa and Parks (25) propose a selection criterion wic uses te numer of pixels wit omogeneous colors in a local neigorood. Te autors compute te distances etween te color point of te considered pixel and tose of its neigors in te CIE L a color space (defined in section 4.3.2), wic etter fits wit te uman perception of colors tan RB space. Tey design an omogeneity criterion wit adaptive tresolding wic reduces color artifacts due to incorrect selection of te interpolation direction. Cung and Can (26) nicely demonstrate tat green plane interpolation is critical to te estimated image quality, and suggest to evaluate te local variance of color difference as an omogeneity criterion. Te selected direction corresponds to minimal variance, wic yields green component refinement especially in textured areas. Omer and Werman (24) use a similar way to select te interpolation direction, except tat te local color ratio variance is used. Tese autors also propose a criterion ased on a local corner score. Under te assumption tat demosaicing generates artificial corners in te estimated image, tey apply te Harris corner detection filter (Harris and Stepens, 1988), and select te interpolation direction wic provides te fewest detected corners Template Matcing-ased Metods Tis family of metods aims at identifying a template-ased feature in eac pixel neigorood, in order to interpolate according to te locally encountered feature. Suc strategy as een first implemented y Cok in a patent dating ack to 1986 (Cok, 1986)(Cok, 1994), in wic te autor classifies 3 3 neigoroods into edge, stripe or corner features (see figure 2). Te algoritm original part lies in te green component interpolation at eac pixel P were it misses (i.e. at center pixel of {R} or {B} CFA structures) : 1. Compute te average green level availale at te four nearest neigor pixels of P (i.e. elonging to N 4, as defined on figure 17). Examine weter eac of tese four green levels is lower (), iger (), or equal to teir average. Sort tese four 32

34 values in descending order, let , and compute teir median M = ( )/2. 2. Classify P neigorood as : (a) edge if 3 and 1 are present, or 1 and 3 (see figure 2a) ; () stripe if 2 and 2 are present and opposite y pairs (see figure 2) ; (c) corner if 2 and 2 are present and adjacent y pairs (see figure 2c). In te special case wen two values are equal to te average, te encountered feature is taken as : (a) a stripe if te oter two pixels and are opposite ; () an edge oterwise. 3. Interpolate te missing green level according to te previously identified feature : (a) for an edge, Ĝ = M ; () for a stripe, Ĝ = CLIP 2 3 (M (S M)), were S is te average green level over te eigt neigoring pixels laeled as Q in figure 2d ; (c) for a corner, Ĝ = CLIP 2 3 (M (S M)), were S is te average green level over te four neigoring pixels laeled as Q in figure 2e, wic are located on ot sides of te orderline etween and pixels. Function CLIP 2 3 simply limits te interpolated value to range [ 3, 2 ] : α if 3 α 2, α R, CLIP 2 3 (α) = 2 if α > 2, 3 if α < 3. Tis metod, wic classifies neigorood features into tree groups, encompasses tree possile cases in an image. But te criterion used to distinguis te tree features is still too simple, and comparing green levels wit teir average may not e sufficient to determine te existing feature adequately. Moreover, in case of a stripe feature, interpolation does not take into account tis stripe direction. Cang and Tan (26) also implement a demosaicing metod ased on templatematcing, ut apply it on te color difference planes (R and B ) in order to interpolate R and B color components, eing estimated eforeand tanks to Hamilton and Adams sceme descried aove. Te underlying strategy consists in simultaneously exploiting te spatial and spectral correlations, and relies on a local edge information wic causes fewer color artifacts tan Cok s sceme. Altoug color difference planes carry less ig-frequency information tan color component planes (see figure 16), tey can provide relevant edge information in areas wit ig spatial frequencies Adpative Weigted-Edge Metod Metods descried aove, as template-ased or gradient-ased ones, acieve interpolation according to te local context. Tey ence require prior neigorood classification. Te adaptive weigted-edge linear interpolation, first proposed y Kimmel (45) 33

35 P (a) Edge P () Stripe P (c) Corner Q Q Q Q P Q Q Q Q (d) Stripe neigorood Q Q P Q Q (e) Corner neigorood FI. 2: Feature templates proposed y Cok to interpolate te green component at pixel P. Tese templates, wic are defined modulo π/2, provide four possile Edge and Corner features, and two possile Stripe features. (1999), is a metod wic merges tese two steps into a single one. It consists in weigting eac locally availale level y a normalized factor as a function of a directional gradient. For instance, interpolating te green level at center pixel of {R} or {B} CFA structures is acieved as : Ĝ = w, 1, 1 + w 1, 1, + w 1, 1, + w,1,1 w, 1 + w 1, + w 1, + w,1, (46) were w δx,δy coefficients are te weigting factors. In order to exploit spatial correlation, tese weigts are adjusted according to te locally encountered pattern. Kimmel suggests to use local gradients to acieve weigt computation. In a first step, directional gradients are approximated at a CFA image pixel P y using te levels of its neigors. radients are respectively defined in orizontal, vertical, x -diagonal (top-rigt to ottom-left) and y -diagonal (top-rigt to ottom-rigt) directions (see figure 17a) over a 3 3 neigorood y te following generic expressions : x (P) = (P 1, P 1, )/2, (47) y (P) = (P, 1 P,1 )/2, (48) x (P) = (1, 1 max( )/ 2, ( 1,1 )/ ) 2 at locations, (49a) (P 1, 1 P 1,1 )/2 2 elsewere, (49) 34

36 y (P) = ( 1, 1 max( )/ 2, ( 1,1 )/ ) 2 at locations, (5a) (P 1, 1 P 1,1 )/2 2 elsewere, (5) were P δx,δy stands for te neigoring pixel of P, wit relative coordinates (δx,δy), in te CFA image. Here, R, or B is not specified, since tese generic expressions apply to all CFA image pixels, watever te considered availale component. However, we notice tat all differences involved in equations (47) and (48) imply levels of a same color component. Te weigt w δx,δy in direction d, d {x,y,x,y }, is ten computed from directional gradients as : w δx,δy = 1 1+ d (P) 2 + d (P δx,δy ) 2, (51) were direction d used to compute te gradient d is defined y te center pixel P and its neigor P δx,δy. At te rigt-and pixel (δx,δy) = (1,) as an example, te orizontal direction x is used for d ; d (P) and d (P 1, ) are terefore ot computed y expression (47) defining x, and te weigt is expressed as : w 1, = 1 1+(P 1, P 1, ) 2 /4+(P 2, P) 2 /4. (52) Definition of weigt w δx,δy is uilt so tat a local transition in a given direction yields a ig gradient value in te same direction. Consequently, weigt w δx,δy is close to for te neigor P δx,δy and does not contriute muc to te final estimated green level according to equation (46). On te opposite, weigt w δx,δy is equal to 1 wen te directional gradients are equal to. Adjustments in weigt w computation are proposed y Lu and Tan (23), wo use a Soel filter to approximate te directional gradient, and te asolute instead of square value of gradients in order to oost computation speed. Suc a strategy is also implemented y Lukac and Plataniotis (25). Once ten green plane as een fully populated tanks to equation (46), red and lue levels are estimated y using component ratios R/ and B/ among neigoring pixels. Interpolating te lue component is for instance acieved according to two steps (te red one eing processed in a similar way) : 1. Interpolation at red locations (i.e. for {R} CFA structure) : ˆB=Ĝ P N 4 w(p) B(P) Ĝ(P) w(p) P N 4 w 1, 1 B 1, 1 + w Ĝ 1, 1 B1, 1 + w 1, 1 Ĝ 1,1 B 1,1 + w 1, 1 Ĝ 1,1 B1,1 1,1 Ĝ 1,1 = Ĝ. w 1, 1 + w 1, 1 + w 1,1 + w 1,1 (53) 35

37 2. Interpolation at oter CFA locations wit missing lue level (i.e. at {RR} and {BB} structures) : w(p) ˆB(P) P N Ĝ(P) ˆB= 4 w(p) P N 4 w, 1 ˆB, 1 + w Ĝ 1, ˆB 1, + w, 1 Ĝ 1, ˆB 1, + w 1, Ĝ,1 ˆB,1 1, Ĝ,1 =. w, 1 + w 1, + w 1, + w,1 Once all missing levels ave een estimated, Kimmel s algoritm (1999) acieves green plane refinement y using te color ratio constancy rule. Tis iterative refinement procedure is taken up y Muresan et al. (2) wit a sligt modification : instead of using all N 8 neigoring pixels in step 1 elow, only neigoring pixels wit green availale component are considered. Te following steps descrie tis refinement sceme : 1. Correct te estimated green levels wit te average of two estimations (one on te lue plane, te oter on te red one), so tat te constancy rule is locally enforced for color ratio /R : were : Ĝ R R Ĝ = 1 2 w(p) (P) P N ˆR(P) 4 (54) (ĜR + Ĝ B), (55) w(p) and Ĝ B B P N 4 w(p) (P) P N ˆB(P) 4 w(p), P N 4 B and R standing eiter for an estimated level or an availale CFA value, according to te considered CFA structure ({R} or {B}). 2. Correct ten red and lue estimated levels at green locations, y using weigted R/ and B/ ratios at te eigt neigoring pixels : ˆR = w(p) P N 8 R(P) (P) w(p) P N 8 3. Repeat te two previous steps twice. (56) and ˆB = w(p) P N 8 w(p) P N 8 B(P) (P). (57) Tis iterative correction procedure gradually enforces more and more omogeneous /R and /B color ratios, wereas te green component is estimated y using spectral correlation. Its convergence is owever not always guaranteed, wic may cause troule for irrelevant estimated values. Wen a level occurring in any color ratio denominator is very close or equal to zero, te associated weigt may not cancel te resulting ias. Figure 21c sows some color artifacts wic are generated in tis case. In pure yellow areas, quasi-zero lue levels cause a saturation of te estimated green component at R and B locations, wic ten alternate wit original green levels. Smit (25) suggests to compute adaptive weigts as w δx,δy = d (P) +4 d (P δx,δy ), in order to reduce te division ias and contriution of pixels on ot edge sides. 36

38 (a) Original image () Estimated image efore correction (c) Estimated image after correction FI. 21: Demosaicing result acieved y Kimmel s metod (1999), efore and after te iterative correction steps. enerated artifacts are pointed out on image (c). Lukac et al. (26) coose to apply adaptive weigting on color difference planes for R and B component estimations, wic avoids te aove-mentioned artifacts during te iterative correction step. Tsai and Song (27) take up te latter idea, ut enance te green plane interpolation procedure : weigts are adapted to te local topology tanks to a preliminary distinction etween omogeneous and edge areas Local Covariance-ased Metods In is PD dissertation, Li (2) presents an interpolation sceme to increase te resolution of a gray-level image. Classical interpolation metods (ilinear and icuic), ased on spatial invariant models, tend to lur transitions and generate artifacts in ig-frequency areas. Li s approac exploits spatial correlation y computing a local level covariance, witout relying on directional gradients as do te aove-mentioned metods in tis section. Beyond resolution enancement, te autor applies tis approac to demosaicing (Li and Orcard, 21). In te CFA image, eac of R, or B color component plane may e viewed as a su-sampled version of its respective, fully-populated estimated color plane. According to tis consideration, a missing level in a given color plane is interpolated y using local covariance, preliminarily estimated from neigoring levels availale in te same plane. Te underlying principle of tis metod may e etter understood y considering te resolution enancement prolem first. More precisely, figure 22 illustrates ow te resolution of a gray-level image can e douled tanks to geometric duality, in a twostep procedure. Te first step consists in interpolating P 2i+1,2 j+1 level (represented y a wite dot in figure 22a) from availale P 2(i+k),2( j+l) levels (lack dots). Te following linear comination of N 4 neigors is used ere : ˆP 2i+1,2 j+1 = 1 1 k= l= α 2k+l P 2(i+k),2( j+l), (58) in wic α m coefficients, m 3, of α are computed as follows (see justification 37

39 Â 3 2i 2 2i 2i+2 2i 1 2i+1 2j 2 2j 1 2j 2j+1 2j+2 â â 1 â 2 â 3 a A 3 a 3 a 1 a 2 (a) Interpolating lattice P 2i+1,2 j+1 from lattice P 2i,2 j 2i 2 2i 2i+2 2i 1 2i+1 2j 2 2j 1 2j 2j+1 2j+2 Â 1 â 1 â A 1 a 1 â 2 a a 3 â 3 a 2 () Interpolating lattice P i, j (i+ j odd) from lattice P i, j (i+ j even) FI. 22: eometric duality etween te low-resolution covariance and te igresolution covariance. Black dots are te availale levels at low resolution, and te wite dot is te considered pixel to e interpolated. In sufigure (), diamonds represent pixels estimated in te previous step. 2i 2 2i 1 2i 2i+1 2i+2 2j 2 â 2j 1 a 2j 2j+1 2j+2 Â 1 A a 1 1 a 3 a 2 â 1 â2 â 3 (a) Interpolating at R or B locations 2i 1 2i+1 2i+3 2i 2i+2 2j 2 2j 1 2j 2j+1 2j+2 R â Â 3 â 1 â2 â 3 a A 3 a 3 a 1 a 2 Ĝ B () Interpolating R at B locations 2i 1 2i+1 2i+3 2i 2i+2 2j 2 Â 1 â 2j 1 A a a 1 2j 1 a 3 â 1 a â 3 2 2j+1 2j+2 R ˆR â2 Ĝ B (c) Interpolating R at locations FI. 23: eometric duality etween covariances used in demosaicing. Color dots are te availale components in te CFA image, and te wite dot is te considered pixel to e interpolated. In sufigures () and (c), diamonds represent pixels estimated in te previous step, and spatial coordinates are sifted one pixel rigt. 38

40 and details in Li and Orcard, 21) : α = A 1 a. (59) Tis expression incorporates te local covariance matrix A [A m,n ], m,n 3 etween te four neigoring levels considered pair-wise (e.g. A 3 in figure 22a), and te covariance vector a [a m ], m 3 etween te pixel level to e estimated and tose of its four availale neigors (see figure 22a) 3. Te main issue is to get tese covariances for te ig-resolution image from levels wic are availale at low resolution. Tis is acievale y using te geometric duality principle : once covariance is computed in a local neigorood of te low-resolution image, te equivalent covariance at ig resolution is estimated y geometric duality wic considers pixel pairs in te same direction at ot resolutions. Under tis duality principle, a is for instance estimated y â, A 3 eing replaced y  3 (see figure 22). Te underlying assumption to approximate a m y â m and A m,n y  m,n, is tat te local edge direction is invariant to image resolution. Te second step consists in estimating remaining unavailale levels, as for te wite dot on figure 22. Interpolation ten relies on exactly te same principle as aove, except tat te availale pixel lattice is now te previous one rotated y π/4. Applying tis metod to demosaicing is rater straigtforward : 1. Fill out te green plane at R and B locations y using : Ĝ = P N 4 α(p)(p), (6) were α coefficients are computed according to expression (59) and figure 23a. 2. Fill out te two oter color planes, y exploiting te assumption of color difference (R and B ) constancy. For te red plane as example : (a) At B locations, interpolate te missing red level as : ( ) ˆR = Ĝ+ α(p) R(P) Ĝ(P), (61) P N 4 were α coefficients are computed according to figure 23. () At locations, interpolate te missing red level as : ( R(P) ) ˆR = + α(p) Ĝ(P), (62) P N 4 were α coefficients are computed according to figure 23c, R eing a value eiter availale in I CFA or estimated. 3 Notations used ere differ from tose in te original pulication (i.e. R and r for covariances) in order to avoid any confusion. 39

41 Altoug tis metod yields satisfying results (see next susection), some limits may e pointed out. First, it requires te covariance matrix A to e invertile so tat α coefficients can e computed. Li sows tat tis condition may not e verified in omogeneous areas of te image. Second, computing covariance matrices is a greedy processing task. To overcome tose drawacks, te autor proposes a yrid approac y using covariance-ased interpolation only in edge areas, and a simple metod (like ilinear interpolation) in omogeneous areas. Tis sceme avoids te covariance matrix invertiility issue, wile decreasing computation time since edge areas generally take up a small part of te wole image. Leitão et al. (23) oserve tat tis metod performs worse in textured areas tan edge areas. Tey advise, for covariance estimation, to avoid considering pixels wic are too far from te pixel to e interpolated. Asuni and iacetti (28) refine te detection sceme of areas in wic te covariance estimation is appropriate for interpolation. Tese autors also improve te covariance matrix conditioning y adding a constant to pixel levels were tey reac very low values. Tam et al. (29) raise te covariance mismatc prolem, wic occurs wen te geometric duality property is not satisfied, and solve it y extending te covariance matcing into multiple directions. Multiple low-resolution training windows are considered, and te one tat yields te igest covariance energy is retained to apply te linear interpolation according to generic equation (58). Lukin and Kuasov (24) incorporate covariance-ased interpolation for te green plane estimation, in a demosaicing algoritm comining several oter tecniques notaly Kimmel s. In addition, it is suggested to split nonomogeneous areas into textured and edge ones. Te interpolation step is ten acieved specifically to eac kind of ig-frequency contents Comparison Between Edge-adaptive Metods. Finally, it is relevant to compare results acieved y te main exposed propositions wic exploit spatial correlation. Te key ojective of tese metods is to acieve te est estimation of green plane as possile, on wic relies susequent estimation of red and lue ones. Hence, we propose to examine te peak signal-to-noise ratio PSNR (see expression (24)) of te estimated green plane, according to te experimental procedure descried on figure 11. Tale 2 sows te corresponding results, togeter wit tose acieved y ilinear interpolation for comparison. It can e noticed tat all metods ased on spatial correlation provide significant improvement in regard to ilinear interpolation. Among te six tested metods, Cok s (1986) and Li s (21) estimate missing green levels y using only availale green CFA samples, like ilinear interpolation ; all tree generally provide te worst results. Te green plane estimation may terefore e improved y using information from R and B components. In Kimmel s algoritm for instance (1999), green plane quality is noticealy enanced, for 1 images out of 12, tanks to corrective iterations ased on spectral correlation (see results of columns Kimmel and Kimmel 1 ). From tese results may e asserted tat any efficient demosaicing metod sould take advantage of ot spatial and spectral correlations, simultaneously and for eac color plane interpolation. Bot metods proposed y Hamilton and Adams (1997) and y Wu and Zang (24) use te same expression to interpolate green levels, ut different rules to select te interpolation direction. A comparison of respective results 4

42 Image Bilinear Hamilton Kimmel Kimmel 1 Wu Cok Li Average TAB. 2: Peak Signal-to-Noise Ratio (in deciels) of te green plane (PSNR ), estimated y various interpolation metods. For eac image, te est result is printed in old typeface. Tested metods are ere referred to ciefly y teir first autor s name : 1. Bilinear interpolation 2. Hamilton and Adams gradient-ased metod (1997) 3 and 4. Kimmel s adaptive weigted-edge metod (1999), efore (Kimmel ) and after (Kimmel 1 ) corrective iterations 5. Wu and Zang s componentconsistent sceme (24) 6. Cok s metod ased on template matcing (1986) 7. Li s covariance-ased metod (21). sow tat careful selection of te interpolation direction is important for overall performance. Tis is all te most noticeale tat, compared to oter algoritms, computation complexity is rater low for ot Hamilton and Adams and Wu and Zang s metods. Indeed, tey do not require any corrective iteration step nor covariance matrix estimation step, wic are computation-expensive operations Estimated Color Correction Once te two missing components ave een estimated at eac pixel, a post-processing step of color correction is often applied to remove artifacts in te demosaiced image. To remove false colors in particular, a classical approac consists in strengtening spectral correlation etween te tree estimated color components. Suc a goal may e reaced first y median filtering, as descried elow. An iterative update of initial interpolated colors is also sometimes acieved, as Kimmel s corrective step (1999) presented in susection A still more sopisticated algoritm proposed y unturk et al. (22) is descried in detail in te second part of tis section. Among oter correction tecniques of estimated colors, Li (25) uilds a demosaicing sceme y using a iterative approximation strategy wit a spatially-adaptive stopping criterion ; e also studies te influence of te numer of corrective iteration steps on te estimated image quality. Let us also mention ere regularization scemes ased on te Bayesian framework, as Markov Random Fields (see e.g. Mukerjee et al., 21), wic are owever poorly adapted to real-time implementation. 41

43 Median Filtering One of te most widespread tecniques in demosaiced image post-processing is median filtering. Suc a filter as een used for years to remove impulse noise in graylevel images, ut also efficiently removes color artifacts witout damaging local color variations. Freeman (1988) was te first person to take advantage of te median filter to remove demosaicing artifacts. Applied to te estimated planes of color differences R and B, tis filter noticealy improves te estimation provided y ilinear interpolation. As sown on figure 16d, tese planes contain little ig-frequency information. False estimated colors, wic result from inconsistency etween te local interpolation and tose acieved in a neigorood, may ence e more efficiently corrected on tese planes wile preserving oject edges. Median filtering is implemented in several works of te demosaicing literature. For instance, Hirakawa and Parks (25) propose to iterate te following correction witout giving more details aout te numer of iteration steps nor te filter kernel size, defined at eac pixel as : ˆR = Ĝ+M R, (63) Ĝ = 1 ( ˆR+M R + ˆB+M B), 2 (64) ˆB = Ĝ+M B, (65) were ˆR, Ĝ and ˆB denote te filtered estimated components, and M kk is te output value of te median filter applied on estimated planes of color differences Î k Î k, (k,k ) {R,,B} 2. Lu and Tan (23) use a sligt variant of te latter, ut advise to apply it selectively, since median filtering tends to attenuate color saturation in te estimated image. An appropriate strategy is proposed for te pre-detection of artifactprone areas, were median filtering is ten solely applied. However, Cang and Tan (26) notice tat median filtering applied to color difference planes, wic still ear some textures around edges, tends to induce zipper artifact in tese areas. In order to avoid filtering across edges in te color difference planes, edge areas are preliminarily detected tanks to a Laplacian filter. Some artifacts may owever remain in te median filtered image, wic is mainly due to separate filtering of color difference planes (R and B ). An alternative may e to apply a vector median filter on te estimated color image wile exploiting spectral correlation. Te local output of suc a filter is te color vector wic minimizes te sum of distances to all oter color vectors in te considered neigorood. But according to Lu and Tan (23), te vector filter rings out little superiority if any in artifact removal, compared wit te median filter applied to eac color difference plane. Te autors justification is tat te estimation errors may e considered as additive noise wic corrupts eac color plane. Tese noise vector components are loosely correlated. In suc conditions, Astola et al. (199) sow tat vector median filtering does not acieve etter results tan marginal filtering on te color difference planes. 42

44 Alternating Projection Metod As previously mentioned in section 2.2.3, pixel levels ear strong spectral correlation in ig spatial frequency areas of a natural color image. From tis oservation, unturk et al. (22) aim at increasing te correlation of ig-frequency information etween estimated ˆR, Ĝ and ˆB component planes, wile keeping te CFA image data. Tese two ojectives are enforced y using two convex constraint sets, on wic te algoritm alternately projects estimated data. Te first set is named Oservation and ensures tat interpolated data are consistent wit tose availale in te CFA image. Te second set, named Detail, is ased on a decomposition of eac R, and B plane into four frequency suands tanks to a filterank approac. A filterank is a set of passand filters wic decompose (analyze) te input signal into several suands, eac one carrying te original signal information in a particular frequency suand. On te opposite, a signal may e reconstructed (syntesized) in a filterank y recomination of its suands. Te algoritm uses an initially estimated image as starting point ; it may ence e considered as a sopisticated refinement sceme. To get te initial estimation Î, any demosaicing metod is suitale. Te autors suggest to use Hamilton and Adams sceme to estimate te green plane Î, and a ilinear interpolation to get te red ÎR and lue Î B planes. Two main steps are acieved ten, as illustrated on figure 24a : 1. Update te green plane y exploiting ig-frequency information of red and lue planes. Tis enances te initial green component estimation. (a) Use availale red levels of te CFA image (or Î R ) to form a downsampled plane I R of size X/2 Y/2, as illustrated on figure 24. () Sample, at te same R locations, green levels from te initial estimation Î to form a downsampled plane Î (R), also of size X/2 Y/2. (c) Decompose te downsampled plane I R into four suands : I R,LL (x,y) = (x) [ (y) I R (x,y) ], (66) I R,LH (x,y) = (x) [ 1 (y) I R (x,y) ], (67) I R,HL (x,y) = 1 (x) [ (y) I R (x,y) ], (68) I R,HH (x,y) = 1 (x) [ 1 (y) I R (x,y) ], (69) and do te same wit plane Î (R). In teir proposition, unturk et al. use a low-pass filter H (z) and a ig-pass filter H 1 (z) to analyze eac plane respectively in low and ig frequencies, as descried aove in susection (d) Use te low-frequency suand (LL) of Î (R) and te tree suands of wit ig frequencies (LH, HL and HH) to syntesize a re-estimated I R 43

45 } } Î R I R I R,HL Î1 R Î Î B Initial estim. Î Î (R) Î (B) I B I R,LL I R,LH I R,HH Î (R),LL Î (R),LH Î (R),HL Î (R),HH Î (B),LL Î (B),LH Î (B),HL Î (B),HH I B,LL I B,LH I B,HL I B,HH Ĩ (R) Ĩ (B) plane update Î R Î 1 Î B Î B 1 Î R,LL 1 Î R,LH 1 Î R,HL 1 Î R,HH 1 Î,LL 1 Î,LH 1 Î,HL 1 Î,HH 1 Î B,LL 1 Î B,LH 1 Î B,HL 1 Î B,HH 1 Ĩ R 1 I CFA Ĩ B 1 Iterations Î R Î 1 Intermediate estim. estim. Final Î 1 Alternating projection of R and B Î (a) Procedure outline. cannel update : 1 Extraction of downsampled X/2 Y/2 planes (see details on figure 24) 2 Suand analysis of eac downsampled plane 3 Syntesis of re-estimated downsampled green planes Ĩ (R) and Ĩ (B) at R and B locations 4 Insertion of tese planes into Î Î Î B (see details on figure 24c). Alternating projection of R and B components : 5 Suand analysis of intermediate estimation Î 1 planes 6 Syntesis of re-estimated red and lue planes 7 Projection of tese planes onto te Oservation constraint set (see details on figure 24d). ˆR R ˆR R ˆR ˆR ˆR ˆR ˆR R ˆR R ˆR ˆR ˆR ˆR Î R Ĝ R Ĝ R Ĝ B Ĝ B Ĝ R Ĝ R Ĝ B Ĝ B Î R R R R I R Ĝ R Ĝ R Ĝ R Ĝ R Î (R) R R R R Ĩ (R) Ĝ R Ĝ R Ĝ B Ĝ B Ĝ R Ĝ R Ĝ B Ĝ B R R B B R R B B R R R R R R R R Ĩ R 1 R R R R R B B R B B I CFA R R R R R R R R R R R R R R Î R R R R R R R R R ˆB ˆB B ˆB ˆB ˆB B ˆB Î B ˆB ˆB B ˆB ˆB ˆB B ˆB B B I B B B () Extraction of downsampled planes from initial estimation. Î B B B B Ĩ (B) (c) Insertion of re-estimated downsampled green planes into Î. Î 1 B B B B B B B B Ĩ B 1 B B B B B B B B B B B B B B B B Î B B B B B B B B B (d) Projection of re-estimated red and lue planes onto te Oservation set. FI. 24: Demosaicing procedure proposed y unturk et al. (22) from an initial estimation Î. 44

46 downsampled green plane Ĩ (R) : [ Ĩ (R) (x,y) = g (x) g (y) Î (R),LL (x,y) [ +g 1 (x) g (y) I R,HL (x,y) ] ] [ ] + g (x) g 1 (y) I R,LH (x,y) [ + g 1 (x) g 1 (y) I R,HH ] (x,y). (7) Filters 1 (z) and (z) used for tis syntesis ave impulse responses g 1 = [ ]/8 and g = [ ]/8, respectively. (e) Apply aove instructions (a)-(d) similarly on te lue plane Î B, wic yields a second re-estimated downsampled green plane Ĩ (B). (f) Insert tese two re-estimated downsampled estimations of te green plane at teir respective locations in plane Î (i.e. Ĩ(R) at R locations, and Ĩ (B) at B locations, as illustrated on figure 24c). A new full-resolution green plane Î1 is otained, wic forms an intermediate estimated color image Î 1 togeter wit planes Î R and ÎB from te initial estimation. 2. Update red and lue planes y alternating projections. (a) Projection onto te Detail set : tis step insures tat ig-frequency information is consistent etween te tree color planes, wile preserving as muc details as possile in te green plane. To acieve tis, a) analyze te tree color planes Î1 R, Î 1 and ÎB 1 of te intermediate image Î 1 into four suands y using te same filterank as previously (composed of H (z) and H 1 (z)) ; ) use te low-frequency suand of te red plane and te tree ig-frequency suands of te green plane to syntesize a re-estimated red plane Ĩ1 R, similarly to equation (7). At last, c) repeat te same operations on te lue plane to estimate Ĩ1 B. () Projection onto te Oservation set : tis step insures tat estimated values are consistent wit te ones availale ( oserved ) in te CFA. Te latter are simply inserted in re-estimated planes Ĩ1 R and ĨB 1 at corresponding locations, as illustrated on figure 24d. (c) Repeat aove instructions (a) and () several times (te autors suggest to use eigt iterations). In sort, ig-frequency suands at red and lue CFA locations are used first to refine te initial estimation of green color plane. Te ig-frequency information of red and lue planes is ten determined y using green plane details so as to remove color artifacts. Tis metod acieves excellent results, and is often considered as a reference in demosaicing encmarks. However, its computation cost is rater ig, and its performance depends on te quality of initial estimation Î. A non-iterative implementation of tis algoritm as een recently proposed (Lu et al., 29), wic acieves te same results as alternating projection at convergence, ut at aout eigt times faster speed. Cen et al. (28) exploit ot suand cannel decomposition and median filtering : a median filter is applied on te difference planes Î R,LL Î,LL and Î B,LL Î,LL 45

47 of low-frequency suands. Components are updated tanks to formulas proposed y Hirakawa and Parks (see equations (63) to (65)), ut on eac low-frequency suand. Hig-frequency suands are not filtered, in order to preserve spectral correlation. Te final estimated image is syntesized from te four frequency suands, as in te alternating projection sceme of unturk et al.. Compared to te latter, median filtering mainly improves te demosaicing result on crominance planes. Menon et al. (26) notice tat unturk et al. s metod tends to generate zipper effect along oject oundaries. To avoid suc artifact, a corrective tecnique is proposed, wic uses te same suand decomposition principle ut pre-determines te local edge direction (orizontal or vertical) on te estimated green plane. Te autors suggest to use tis particular direction to correct green levels y replacing ig-frequency components wit tose of te availale component (R or B) at te considered pixel. As te same direction is used to correct estimated ˆR and ˆB levels at locations on te color difference planes, tis tecnique insures interpolation direction consistency etween color components, wic as een sown to e important in susection Demosaicing using te Frequency Domain Some recent demosaicing scemes rely on a frequency analysis, y following an approac originated y Alleysson et al. (25). Te fundamental principle is to use a frequency representation of te Bayer CFA image 4. In te spatial frequency domain, suc a CFA image may e represented as a comination of a luminance signal and two crominance signals, all tree eing well localized. Appropriate frequency selection terefore allows to estimate eac of tese signals, from wic te demosaiced image can e retrieved. Notice tat frequency-ased approaces do not use Bayer s assumption tat assimilates green levels to luminance, and lue and red levels to crominance components Frequency Selection Demosaicing A simplified derivation of Alleysson et al. s approac as een proposed y Duois (25), wose formalism is retained ere to present te general framework of frequencydomain representation of CFA images. Let us assume tat, for eac component k of a color image, k {R,,B}, tere exists an underlying signal f k. Demosaicing ten consists in computing an estimation ˆf k (coinciding wit Î k ) at eac pixel. Let us assume similarly tat tere exists a signal f CFA wic underlies te CFA image. Tis signal is referred to as CFA signal and coincides wit I CFA at eac pixel. Te CFA signal value at eac pixel wit coordinates (x,y) may e expressed as te sum of spatially sampled f k signals : f CFA (x,y) = k=r,,b f k (x,y)m k (x,y), (71) 4 Let us make ere clear tat frequency (i.e. spatial frequency), expressed in cycles per pixel, corresponds to te inverse numer of adjacent pixels representing a given level series according to a particular direction in te image (classically, te orizontal or vertical direction). 46

48 were m k (x,y) is te sampling function for te color component k, k {R,,B}. For te Bayer CFA of figure 9, tis set of functions is defined as : Wit te definition ecomes : m R (x,y) = 1 ( 1 ( 1) x)( 1+( 1) y), (72) 4 m (x,y) = 1 ( 1+( 1) x+y), (73) 2 m B (x,y) = 1 ( 1+( 1) x)( 1 ( 1) y). (74) 4 f L f C1 f C = f R f f B, te expression of f CFA f CFA (x,y) = f L (x,y)+ f C1 (x,y)( 1) x+y + f C2 (x,y) (( 1) x ( 1) y) ( = f L (x,y)+ f C1 (x,y)e j2π(x+y)/2 + f C2 (x,y) e j2πx/2 e j2πy/2).(75) Te CFA signal may terefore e interpreted as te sum of a luminance component f L at aseand, a crominance component f C1 modulated at spatial frequency (orizontal and vertical) (.5,.5), and of anoter crominance component f C2 modulated at spatial frequencies (.5,) and (,.5). Suc interpretation may e easily cecked on an acromatic image, in wic f R = f = f B : te two crominance components are ten equal to zero. Provided tat functions f L, f C1 and f C2 can e estimated at eac pixel from te CFA signal, estimated color levels ˆf R, ˆf and ˆf B are simply retrieved as : ˆf R ˆf ˆf B = ˆf L ˆf C1 ˆf C2. (76) To acieve tis, te autors take te Fourier transform of te CFA signal (75) : F CFA (u,v) = F L (u,v)+f C1 (u.5,v.5)+f C2 (u.5,v) F C2 (u,v.5), (77) expression in wic terms are, respectively, te Fourier transforms of f L (x,y), of f C1 (x,y)( 1) x+y, and of te two signals defined as f C2a (x,y) = f C2 (x,y)( 1) x and f C2 (x,y) = f C2 (x,y)( 1) y. It turns out tat te energy of a CFA image is concentrated in nine zones of te frequency domain (see example of figure 25), centered on spatial frequencies according to equation (77) : energy of luminance F L (u,v) is mainly concentrated at te center of tis domain (i.e. at low frequencies), wereas tat of crominance is located on its order (i.e. at ig frequencies). More precisely, te energy of F C1 (u.5,v.5) is located around diagonal zones ( corners of te domain), tat of F C2 (u.5,v) along 47

49 v +.5 C1 C2 C1 L C2a C2a (a) Ligtouse CFA image C1 C2 C () Normalized energy (frequencies in cycles/pixel) u FI. 25: Localization of te energy (Fourier transform module) of a CFA signal in te frequency domain (Alleysson et al., 25). u axis of orizontal frequencies, and tat of F C2 (u,v.5) along v axis of vertical frequencies. Tese zones are quite distinct, so tat isolating te corresponding frequency components is possile y means of appropriately designed filters. But teir andwidt sould e carefully selected, since te spectra of te tree functions mutually overlap. In tese frequency zones were luminance and crominance cannot e properly separated, te aliasing penomenon migt occur and color artifacts e generated. In order to design filter andwidts wic acieve te est possile separation of luminance (L) and crominance (C1, C2), Duois (25) proposes an adaptive algoritm tat mainly andles te spectral overlap etween crominance and ig-frequency luminance components. Te autor oserves tat spectral overlap etween luminance and crominance ciefly occurs according to eiter te orizontal or te vertical axis. Hence e suggests to estimate f C2 y giving more weigt to te su-component of C2 (C2a or C2) tat is least prone to spectral overlap wit luminance. Te implemented weigt values are ased on an estimation of te average directional energies, for wic aussian filters (wit standard deviation σ = 3.5 pixels and modulated at spatial frequencies (,.375) and (.375,) cycles per pixel) are applied to te CFA image Demosaicing y Joint Frequency and Spatial Analyses Frequency selection is also a key feature used y Lian et al. (27), wo propose a yrid metod ased on an analysis of ot frequency and spatial domains. Tey state tat te filter used y Alleysson et al. for luminance estimation may not e optimal. Moreover, since te parameters defining its andwidt (see figure 26a) depend on te image content, tey are difficult to e adjusted (Lian et al., 25). Altoug lowpass filtering te CFA image allows to extract te luminance component, it removes 48

50 1 1.5 C1 C2.5 v.5 C2 L Amplitude.5.5 u.5 v.5.5 u.5 C1 C2.5 v.5 C2 L Amplitude u.5 v.5.5 u.5.5 r2 r1.5 r1 (a) Alleysson et al. () Lian et al. (filter used at locations) FI. 26: Filters (andwidt and spectrum) used to estimate luminance, as proposed y Alleysson et al. (25) and Lian et al. (27). te ig-frequency information along orizontal and vertical directions. As te uman eye is igly sensitive to te latter, suc loss is prejudicial to te estimation quality. Lian et al. ten notice tat F C2 components in orizontal and vertical directions ave same amplitudes ut opposite signs 5. Consequently, te luminance spectrum F L at locations is otained as te CFA image spectrum from wic C1 ( corner ) component as een removed (see details in Lian et al., 27). A low-pass filter is proposed to tis purpose, wic cancels C1 wile preserving te ig-frequency information along orizontal and vertical axes. Tis filter is inspired from Alleysson et al. s, reproduced on figure 26a, ut its andpass is designed to remove C1 component only (see figure 26). Te main advantage of tis approac is tat luminance L spectrum ears less overlap wit te spectrum of C1 tan tat of C2 (see example of figure 25), wic makes te filter design easier. From tese oservations, Lian et al. propose a demosaicing sceme wit tree main steps (see figure 27) : 1. Estimate te luminance (denoted as ˆL) at locations, y applying a low-pass filter on te CFA image to remove C1. Practically, te autors suggest to use te following 5 5 kernel, wic gives very good results at low computational cost : H = (78) 2. Estimate te luminance at R and B locations y a spatial analysis. As isolating te spectrum of component C2 is rater difficult, te autors suggest an adaptive algoritm ased on color difference constancy (exploiting spectral correlation) and adaptive weigted-edge linear interpolation (exploiting spatial correlation) : 5 We keep ere notations used y Alleysson et al. for C1 and C2, altoug switced y Lian et al. 49

51 R B B R ˆL ˆL ˆL ˆL ˆL (a) ˆL ˆR, ˆB ˆL ˆR, ˆB ˆL ˆR, ˆB ˆL ˆR, ˆB ˆL ˆR, ˆB () 2.(c) Répétition ˆL ˆL ˆL ˆL ˆL ˆL ˆL ˆL ˆL I CFA Î L() Î L Î 3. ˆB ˆB ˆB Ĝ Ĝ Ĝ... ˆR ˆR ˆR ˆR ˆR ˆR ˆR ˆR ˆR FI. 27: Demosaicing sceme proposed y Lian et al. (27) : 1. Luminance estimation at locations 2.(a) Pre-estimation of R and B components at locations 2.() Luminance estimation at R and B samples 2.(c) Repetition of steps (a) and () 3. Final color image estimation from te fully-populated luminance plane. Notation Î L used ere for illustration sake coincides at eac pixel wit te luminance signal of expression (75), namely ˆL x,y Î L (x,y) ˆf L (x,y). (a) Pre-estimate R and B components at locations, y simply averaging te levels of te two neigoring pixels at wic te considered component is availale. () Estimate te luminance at R and B locations y applying, on te component difference plane L R or L B, a weigted interpolation adapted to te local level transition. For instance, luminance ˆL at R locations is estimated as follows : w(p) (ˆL(P) ˆR(P) ) P N 4 ˆL = R+ w(p) P N 4. (79) For te same {R} CFA structure, weigts w(p) w δx,δy are expressed y using te relative coordinates P δx,δy of te neigoring pixel as : w δx,δy = 1 1+ R, R 2δx,2δy + ˆL δx,δy ˆL δx, δy, (8) wic acieves an adaptive weigted-edge interpolation, as in Kimmel s metod (see section 3.1.4). (c) Repeat te previous steps to refine te estimation : a) re-estimate R component (ten B similarly) at locations, y averaging L R levels at neigoring R locations ; ) re-estimate L at R (ten B) locations according to equation (79) (weigts w(p) remaining uncanged). 3. From te fully-populated luminance plane Î L, estimate te two missing components at eac pixel of te CFA image y using ilinear interpolation : ( Îx,y k = Îx,y L + H k ϕ k ( I CFA Î L)) (x,y), (81) 5

52 were ϕ k (I)(x,y), k {R,,B} is te plane defined y expression (7) and sown on figure 12, and were convolution kernels H k wic acieve ilinear interpolation are defined y expressions (8) and (9) 6. Te aove approac does not require to design specific filters in order to estimate C1 and C2 components, as do metods using te frequency domain only (Duois uses for instance complementary asymmetric filters). Lian et al. sow tat teir metod gloally outperforms oter demosaicing scemes according to MSE (or PSNR) criterion. Te key advantage seems to lie in exploiting te frequency domain at locations only. According to results presented y Lian et al. (27), luminance estimations are less error-prone tan green level estimations provided y metods wic ciefly scan te spatial image plane (sown in tale 2) Conclusion An introduction to te demosaicing issue and to its major solutions as een exposed in te aove section. After aving descried wy suc a processing task is required in mono-ccd color cameras, te various CFA solutions ave een presented. Focusing on te Bayer CFA, we ave detailed te formalism in use trougout te paper. Te simple ilinear interpolation as allowed us to introduce ot artifact generation tat demosaicing metod ave to overcome, and two major rules widely used in te proposed approaces : spatial and spectral correlations. Te vast majority of demosaicing metods strive to estimate te green plane first, wic ear te most ig-frequency information. Te quality of tis estimation strongly influences tat of red and lue planes. Wen exploiting spatial correlation, we experimentally sow tat a correct selection of te interpolation direction is crucial to reac a ig interpolation quality for green levels. Moreover, component-consistent directions sould e enforced in order to avoid color artifact generation. Spectral correlation is often taken into account y interpolating on te difference, rater tan ratio, of component planes. An iterative post-processing step of color correction is often acieved, so as to improve te final result quality y reinforcing spectral correlation. Demosaicing metods may exploit spatial and/or frequency domains. Te spatial domain as een istorically used first, and many studies are ased on it. More recently, autors exploit te frequency domain, wic opens large perspectives. Suc approaces indeed allow to avoid using at least partially or in a first step te euristic rule of color difference constancy to take spectral correlation into account. In all cases were suc assumptions are not fulfilled, even locally, exploiting te frequency domain is an interesting solution. Duois foresaw several years ago (25) tat frequency selection approaces are preeminently promising. Tis will e corroorated is te next sections, dedicated to te ojective quality evaluation of images demosaiced y te numerous presented metods. Already mentioned criteria (MSE and PSNR) will e completed y measures suited to uman color perception, and new specific ones dedicated to te local detection of demosaicing artifacts. 6 Notice tat ϕ k (I) may equally e expressed as ϕ k (I)(x,y) = I(x,y)m k (x,y), were sampling functions m k are defined y (72) to (74). 51

53 4. Ojective Evaluation Criteria for Demosaiced Images 4.1. Introduction Te performances reaced y different demosaicing scemes applied to te same CFA image can e very different. Indeed, different kinds of artifacts wic alter te image quality, can e generated y demosaicing scemes. A description of tese artifacts is given in susection 4.2. Measuring te performance reaced y a demosaicing sceme requires to evaluate te quality of its output image. Indeed, suc a measurement elps to compare te performances of te different scemes. For tis purpose, we always follow te same experimental procedure (see figure 11). First, we simulate te color sampling y keeping only one out of te tree color components at eac pixel of te original image I, according to te Bayer CFA mosaic. Ten, we apply te considered demosaicing sceme to otain te estimated color image Î (ereafter called demosaiced image) from te CFA samples. Finally, we measure te demosaicing quality y comparing te original and demosaiced images. Te main strategy of ojective comparison is ased on error estimation etween te original and demosaiced images. In susection 4.3, we present te most used criteria for ojective evaluation of te demosaiced image. Te ojective criteria are generally ased on a pixel-wise comparison etween te original and te estimated colors. Tese fidelity criteria are not specifically sensitive to one given artifact. Hence, in susection 4.5, we present new measurements wic quantify te occurrences of demosaicing artifacts. Since demosaicing metods intend to produce perceptually satisfying images, te most widely used evaluation criteria are ased on te fidelity to te original images. Rater tan displaying images, our goal is to apply automatic image analysis procedures to te demosaiced images in order to extract features. Tese extracted features are mostly derived from eiter colors or detected edges in te demosaiced images. Since te quality of features is sensitive to te presence of artifacts, we propose to quantify te demosaicing performance y measuring te rates of erroneously detected edge pixels. Tis evaluation sceme is presented in te last susection Demosaicing Artifacts Te main artifacts caused y demosaicing are lurring, false colors and zipper effect. In tis part, we present tose artifacts on examples and explain teir causes y considering te spatial and frequency domains Blurring Artifact Blurring is located in areas were ig frequency information, representing precise details or edges, is altered or erased. Figure 28, illustrates different lurring levels according to te used demosaicing sceme. A visual comparison etween te original image 28 and image 28c wic as een demosaiced y ilinear interpolation, sows tat tis sceme causes severe lurring. Indeed, some details of te parrot plumage are not retrieved y demosaicing and lurring is generated y low-pass filtering. As stated in section 2.2.1, tis interpolation can e acieved y a convolution applied to 52

54 (a) Original Image. () (c) (d) FI. 28: Blurring in te demosaiced image. Image () is an extract from te original image (a), located y a lack ox. Images (c) and (d) are te corresponding extracts of te images respectively demosaiced y ilinear interpolation and y Hamilton and Adams (1997) scemes. eac sampled color component plane (see expression (1)). Te corresponding filters, wose masks H k are given y expressions (8) and (9), reduce ig frequencies. Hence, fine details may e not properly estimated in te demosaiced image (see figure 28c). Tis artifact is less visile in image 28d, wic as een demosaiced y Hamilton and Adams sceme (1997). A visual comparison wit image 28c sows tat tis sceme, presented in section 3.1.1, generates a small amount of visile lurring. It first estimates vertical and orizontal gradients, ten interpolates te green levels along te direction wit te lowest gradient module, i.e. y using as omogeneous levels as possile. Tis selection of neigors used to interpolate te missing green level at a given pixel, tends to avoid lurring Zipper Effect Let us examine figure 29, and more precisely images 29 and 29d wic are extracted from te original Ligtouse image 29a. Images 29c and 29e are te corresponding extracts from te demosaicing result of Hamilton and Adams sceme (1997). On image 29e, one can notice repetitive patterns in transition areas etween omogeneous ones. Tis penomenon is called zipper effect. Te main reason for tis artifact is te interpolation of levels wic elong to omogeneous areas representing different ojects. It occurs at eac pixel were te interpolation direction (orizontal or vertical) is close to tat of te color gradient computed in te original image. Image 29c does not contain any zipper effect, since te interpolation direction is overall ortogonal to tat of a color gradient, ence close to te transition direction etween omogeneous areas. Oppositely, image 29e contains strong zipper effect. In tis area wit ig spatial frequencies along te orizontal direction, te sceme often fails to determine te correct gradient direction (see section and 53

55 () (c) () (c) (d) (e) (a) Original image (d) (e) (f) Demosaiced image FI. 29: Zipper effect due to erroneous selection of te interpolation direction. Images () and (d) are two extracts from te original image (a), located y lack oxes. Images (c) and (e) are te corresponding extracts from te image (f) demosaiced y Hamilton and Adams s sceme (1997). () (c) (a) Original image () (c) (d) Demosaiced image FI. 3: False colors on a diagonal detail. Image () is an extract from te original image (a), located y a lack ox. Image (c), on wic artifacts are circled in lack, is te corresponding extract from image (d) demosaiced y Hamilton and Adams s sceme (1997). figure 19). Te oter main reason is related to te arrangement, in te CFA image, of pixels wose green level is not availale. Indeed, tese pixels were te green levels can e erroneously estimated, are arranged in staggered locations False Colors False color at a pixel corresponds to a large distance etween te original color and te estimated one, in te acquisition color space RB. Figures 3c and 31c sow tat tis penomenon is not caracterized y a specific geometrical structure in te image. Incorrect estimation of te color components may cause perceptile false colors, in particular in areas wit ig spatial frequencies. 54

56 (a) Original Image () (c) FI. 31: False colors generated on a textured area. () Extract from te original image (a), located y a lack ox. (c) Extract demosaiced y Wu and Zang sceme (24), wit artifacts circled in lack Artifacts Descried in te Frequency Domain Te representation of te CFA color samples in te frequency domain, proposed y Alleysson et al. (25), also allows to explain te reasons wy artifacts are generated y demosaicing scemes. As seen in section 3.3.1, te CFA image signal is made up of a luminance signal, mainly modulated at low spatial frequencies, and of two crominance signals, mainly modulated at ig frequencies (see figure 25 page 48). Terefore, demosaicing can e considered as an estimation of luminance and crominance components. Several scemes wic analyze te frequency domain (Alleysson et al., 25; Duois, 25; Lian et al., 27) estimate te missing levels y selective filters applied to te CFA image. Te four possile artifacts caused y frequency analysis are sown in figure 32 extracted from (Alleysson et al., 25) : excessive lurring, grid effect, false colors and watercolor. Wen te andwidt of te filter applied to te CFA image to estimate te luminance is too narrow, an excessive lurring occurs in te demosaiced image (see figure 32a). Wen te andwidt of tis filter is too wide, it may select ig frequencies in zones of crominance. Suc a case can result in a grid effect, especially visile in flat (omogeneous) areas of te image (see figure 32). Moreover, false colors appear wen te crominance filters overlap wit te luminance filter in te frequency domain (see figure 32c). Finally, wen te crominance filter is too narrow, watercolor effect may appear as colors wic are spread eyond te edges of an oject (see figure 32d). Tese artifacts are caused y a ad conception of te selective filters used to estimate luminance and crominance. Tey can also e generated y demosaicing metods wic spatially scan te image. Indeed, several spatial demosaicing scemes generate lurring and false colors since tey tend to under-estimate luminance and over-estimate crominance. Kimmel s (1999) and unturk et al. s (25) scemes also generate grid effect and watercolor Classical Ojective Criteria All te descried artifacts are due to errors in color component estimation. Te classical ojective evaluation criteria sum up te errors etween levels in te original and demosaiced images. At eac pixel, te error etween te original and demosaiced images is quantized tanks to a distance etween two color points in a treedimensional color space (Busin et al., 28). In tis susection, we regroup te most 55

57 (a) Blurring () rid effect (c) False color (d) Watercolor FI. 32: Four kinds of artifacts caused y demosaicing (Alleysson et al., 25). 56

58 widely used measurements into two categories, namely te fidelity and perceptual criteria Fidelity Criteria Tese criteria use colors coded in te RB acquisition color space in order to estimate te fidelity of te demosaiced image compared wit te original image. 1. Mean Asolute Error. Tis criterion evaluates te mean asolute error etween te original image I and te demosaiced image Î. Denoted y MAE, it is expressed as (Cen et al., 28; Li and Randawa, 25) : MAE(I,Î) = 1 3XY X 1Y 1 k=r,,b x= y= Ix,y k Îx,y k, (82) were I k x,y is te level of te color component k at te pixel wose spatial coordinates are (x,y) in te image I. X and Y are respectively te numer of columns and rows of te image. Te MAE criterion can e used to measure te estimation errors of a specific color component. For example, tis criterion is evaluated on te red color plane as : MAE R (I,Î) = 1 XY X 1 x= Y 1 y= I R x,y Îx,y R. (83) MAE values range from to 255, and te demosaicing quality is considered as etter as its value is low. 2. Mean Square Error. Tis criterion measures te mean quadratic error etween te original image and te demosaiced image. Denoted y MSE, it is defined as (Alleysson et al., 25) : MSE(I,Î) = 1 3XY X 1 k=r,,b x= Y 1 y= (I k x,y Î k x,y) 2. (84) Te MSE criterion can also measure te error on eac color plane, as in equation (23). Te optimal quality of demosaicing is reaced wen MSE is equal to, wereas te worst is measured wen MSE is close to Peak Signal-to-Noise Ratio. Te PSNR criterion is a widely used distortion measurement to estimate te quality of image compression. Many autors (e.g. Alleysson et al., 25; Hirakawa and Parks, 25; Lian et al., 27; Wu and Zang, 24) use tis criterion to quantify te performance reaced y demosaicing scemes. Te PSNR is expressed in deciels as : PSNR(I,Î) = 1 log 1 ( 57 d 2 MSE(I,Î) ), (85)

59 were d is te maximum color component level. Wen te color components are quantized wit 8 its, d is set to 255. Like te preceding criteria, PSNR can e applied to a specific color plane. For te red color component, it is defined as : PSNR R (I,Î) = 1 log 1 ( d 2 MSE R (I,Î) ). (86) Te iger te PSNR value is, te etter is te demosaicing quality. Te PSNR measured on demosaiced images generally ranges from 3 to 4 db (i.e. MSE ranges from 65.3 to 6.5). 4. Correlation. A correlation measurement etween te original image and te demosaiced image is used y Su and Willis (23) to quantify te demosaicing performance. Te correlation criterion etween two gray-level images I and Î is expressed as : C(I,Î) = [( X 1 x= Y 1 y= I x,y 2 ( X 1Y 1 x= y= ) XY µ 2 ] 1/2 [( I x,y Î x,y ) XY µ ˆµ X 1Y 1 2 Î x,y x= y= ) ] 1/2, (87) XY ˆµ 2 were µ and ˆµ are te mean gray levels in te two images. Wen a color demosaiced image is considered, one estimates te correlation level C k ( I k,î k), k {R,,B}, etween te original and demosaiced color planes. Te mean of te tree correlation levels is used to measure te quality of demosaicing. Te correlation levels C range etween and 1, and a measurement close to 1 can e considered as a satisfying demosaicing quality Perceptual Criteria Te preceding criteria are not well consistent wit quality estimation provided y te uman visual system. Tat is te reason wy new measurements ave een defined, wic operate in perceptually uniform color spaces (Cung and Can, 26). 1. Estimation Error in te CIE L a color space. Te CIE L a color space is recommended y te International Commission on Illumination to measure te distance etween two colors (Busin et al., 28). Tis space is close to a perceptually uniform color space wic as not een completely defined yet. So, te Euclidean distance in te CIE L a color space is a perceptual distance etween two colors. Te tree color components (R,,B) at a pixel are first transformed into (X,Y,Z) components according to a CIE XY Z linear operation. Ten, te color components CIE L a are expressed as : 58

60 L = { Y/Y n 16 if Y/Y n >.8856, (88a) 93.3 Y/Y n oterwise, (88) a = 5 (f(x/x n ) f(y/y n )), (89) = 2 (f(y/y n ) f(z/z n )), (9) wit : { 3 x if Y/Yn >.8856, (91a) f(x) = 7.787x oterwise, (91) were te used reference wite is caracterized y te color components (X n,y n,z n ). We can notice tat L represents te eye response to a specific luminance level, wereas a and components correspond to crominance. Te component a represents an opposition of colors Red reen, and corresponds to an opposition of colors Blue Yellow. Te color difference is defined as te distance etween two color points in tis color space. Ten, te estimation error caused y demosaicing is te mean error processed wit all image pixels : E L a (I,Î) = 1 XY X 1 x= Y 1 y= ( ) I k x,y Î k 2 x,y. (92) k=l,a, Te lower E L a is, te lower is te perceptual difference etween te original and demosaiced images, and te iger is te demosaicing quality. 2. Estimation Error in te S-CIE L a color space. In order to introduce spatial perception properties of te uman visual system, Zang and Wandell (1997) propose a new perceptual color space, called S-CIE L a. Te color components (R,,B) are first transformed into te color space XY Z wic does not depend on te acquisition device. Ten, tese color components are converted into te antagonist color space AC 1 C 2, were A represents te perceived luminance and C 1, C 2, te crominance information in terms of opposition of colors Red reen and Blue Yellow, respectively. Te tree component planes are ten separately filtered y aussian filters wit specific variances, wic approximate te contrast sensitivity functions of te uman visual system. Te tree filtered components A, C 1 and C 2 are converted ack into (X,Y,Z) components, wic are ten transformed into CIE L a color space tanks to equations (88) and (89). Once te color components L, a and ave een computed, te estimation error E in S-CIE L a is defined y equation (92). Tis measurement was used y Li (25), Su (26) and Hirakawa and Parks (25) to measure te demosaicing quality. 59

61 3. Normalized Color Difference in te CIE L u v color space. Te CIE proposes anoter perceptually uniform color space called CIE L u v, wose luminance L is te same as tat of CIE L a color space. Te crominance components are expressed as : u = 13 L (u u n), (93) v = 13 L (v v n), (94) wit : u 4X = X + 15Y + 3Z, (95) v 9Y = X + 15Y + 3Z, (96) were u n et v n are te crominance of te reference wite. Te criterion of normalized color difference NCD is expressed as (Li and Randawa, 25; Lukac and Plataniotis, 24) : NCD(I,Î) = X 1Y 1 x= y= X 1 x= ( I k x,y Î k k=l,u,v x,y ( ) I k 2 x,y k=l,u,v Y 1 y= ) 2, (97) were I k x,y is te level of color component k, k {L,u,v }, at te pixel aving (x,y) spatial coordinates. Tis normalized measurement ranges from (optimal demosaicing quality) to 1 (worst demosaicing quality). Among oter measurements found in te literature, let us also mention Buades et al. (28). Tese autors first consider artifacts as noise wic corrupts te demosaiced image, and propose an evaluation sceme ased on specific caracteristics of wite noise. Unfortunately, te evaluation is only acieved y sujective appreciation. More interesting is te suggestion to use gray-level images for demosaicing evaluation. Indeed, color artifacts are ten not only easily visually identified, ut may also also e analyzed y considering te cromaticity. Te rate of estimated pixels wose cromaticity is iger tan a tresold reflects te propensity of a given demosaicing sceme to generate false colors Artifact-sensitive Measurements Te ojective measurements presented aove are ased on an evaluation of te color estimation error. None of tese measurements quantify te specific presence of eac kind of artifact witin te demosaiced images. Toug, it would e interesting to isolate specific artifacts during te evaluation process. In tis part, we present measurements wic are sensitive to specific kinds of artifacts y taking teir properties into account. 6

62 L level 1 5 P 1 P r 1 Pl 2 P 2 P r 2 P l 3 P 3 P r 3 Pl 4 P 4 P r 4 P 1 l x spatial coordinate FI. 33: Vertical edge pixels associated wit teir left and rigt pixels. Vertical edge pixels P 1, P 2, P 3 and P 4 are represented y solid lines, wile pixels corresponding to extrema are located y dased lines. Te left (resp. rigt) extremum of a vertical edge pixel P i is denoted Pi l (resp. Pi r ). One single extremum may e associated wit two different vertical edge pixels, for example P1 r Pl Blurring Measurement Te lurring measurement proposed y Marziliano et al. (24) is sensitive to te decrease of local level variations in transition areas. Te autors notice tat lurring corresponds to an expansion of tese transition areas, and propose to measure te transition widts to quantify tis artifact. Te evaluation sceme analyzes te luminance planes of te original and demosaiced images, respectively denoted as L and ˆL. Te transition widt increase, evaluated at te same pixel locations in ot images, yields an estimation of te lurring caused y demosaicing. Tis lurring measurement consists in te following successive steps : 1. Apply te Soel filter to te luminance plane L according to te orizontal direction, and tresold its output. Te pixels detected in tis way are called vertical edge pixels. 2. At eac vertical edge pixel P, examine te luminance levels of pixels located on te same row as P in te luminance plane L. Te pixel P l (resp. P r ) corresponds to te first local luminance extremum located on te left (resp. te rigt) of P. To eac vertical edge pixel P, associate in tis way a pair of pixels P l and P r, one of tem corresponding to a local luminance maximum and te oter one to a minimum (see figure 33). 3. Te transition widt at P is defined as te difference etween te x coordinates 61

63 of pixels P l and P r. 4. Compute te lurring measurement as te mean transition widt estimated over all vertical edge pixels in te image. 5. From te spatial locations of vertical edge pixels in L wic ave een detected in step 1, steps 2 to 4 are performed on te luminance plane ˆL of te demosaiced image. A lurring measurement is ten otained for tis plane. 6. Te two measurements, otained respectively for te original and demosaiced images, are compared in order to estimate lurring caused y te considered demosaicing sceme Zipper Effect Measurements As far as we know, te single proposition for zipper effect measurement was given y Lu and Tan (23). Tis artifact is caracterized at a pixel y an increase of te minimal distance etween its color and tose of its neigors. Tis measurement terefore relates to te original color image. Te zipper effect measurement in a demosaiced image Î, compared wit te original image I, is computed y tese successive steps : 1. At eac pixel P in te original image I, identify te neigoring pixel P wose color is te closest to tat of P in CIE L a color space : P = argmin I(P) I(Q), (98) Q N 8 were N 8 is te 8-neigorood of P and is te Euclidean distance in CIE L a color space. Te color difference is ten computed as : I(P) = I(P) I(P ). (99) 2. At te same locations as P and P, compute teir color difference in te demosaiced image Î : Î(P) = Î(P) Î(P ). (1) 3. Compute te color difference variation ϕ(p) = Î(P) I(P). 4. Tresold te color difference variation, in order to detect te pixels P were zipper effect occurs. If ϕ(p) > T ϕ, te pixel P in te demosaiced image presents a ig variation of te difference etween its color and tat of P. More precisely, wen ϕ(p) is lower tan T ϕ, te demosaicing sceme as reduced te color difference etween pixels P and P. On te oter and, wen ϕ(p) > T ϕ, te difference etween te color of P and tat of P as een igly increased in Î compared wit I ; so, te pixel P is considered as affected y zipper effect. Te autors propose to set te tresold T ϕ to Compute te rate of pixels affected y zipper effect in te demosaiced image : ZE % = Card { P(x,y) ϕ(p) > T ϕ }. (11) 62

64 (a) Original image I () Demosaiced image Î (c) Zipper effect map FI. 34: Over-detection of te zipper effect y Lu and Tan s measurement (23), in a syntetic image. In te detection map (c), pixels affected y zipper effect are laeled as, and te ground-trut (determined y visual examination) is laeled as gray. A pixel laeled ot as and gray corresponds to a correct detection, wereas a pixel laeled only as corresponds to an over-detection of te zipper effect. Te effectiveness of tis measurement was illustrated y its autors wit a syntetic image 7. However, y applying it to images of Kodak Dataase, we will sow in section 5.2 tat it tends to over-detect zipper effect in te demosaiced images. Two reasons explain tis over-detection. First, a pixel wose color is correctly estimated and wic as neigoring pixels wose colors are erroneously estimated, can e considered as eing affected y zipper effect (see figure 34). Second, we notice tat all te pixels detected y Lu and Tan s measurement are not located in areas wit perceptile alternating patterns wic correspond to zipper effect. Indeed, all te artifacts wic can increase te minimal difference etween te color of a pixel and tose of its neigors do not always ear te geometric properties of zipper effect. An example of tis penomenon is found on te zipper effect detection result of figure 38c4 : almost all pixels are detected as affected y zipper effect, altoug te demosaiced image 384 does not contain tis repetitive and alternating pattern. To avoid over-detection, we propose a sceme ereafter referred to as directional alternation measurement wic quantifies te level variations over tree adjacent pixels along te orizontal or vertical direction in te demosaiced image. Two reasons explain wy te direction of zipper effect is mainly orizontal or vertical. Demosaicing scemes usually estimate te green color component first, ten te red and lue ones y using color differences or ratios. However, along a diagonal direction in te CFA image, all te green levels are eiter availale or missing. Since tere is no alternating pattern etween estimated and availale levels along tis diagonal direction, tere are few alternating estimation errors wic caracterize zipper effect. Secondly, edges of ojects in a natural scene tend to follow orizontal and vertical directions. We propose to modify te selection of neigoring pixels used to decide, tanks to Lu and Tan s criterion, weter te examined pixel is affected y zipper effect. We 7 Tis image is not availale. 63

65 require te selected adjacent pixels to present a green alternating pattern specific to zipper effect. Moreover, tis series of tree adjacent pixels as to e located along transitions etween omogeneous areas, so tat te variations of levels associated wit tis transition are not taken into account. Te zipper effect detection sceme ased on directional alternation, wic provides a measurement for tis artifact, consists in te following successive steps : 1. At a give pixel P, determine te local direction (orizontal or vertical) along wic te green variations are te lowest in te original image. Tis direction is selected so tat te green level dispersion is te lowest : σ x (P)= i= 1 ( I x+i,y µ x (P) ) 2 (12) and σ y (P)= i= 1 ( I x,y+i µ y (P) ) 2, (13) were µ x (P) (respectively, µ y (P)) is te mean of te green levels Ix+i,y (respectively, Ix,y+i ), i { 1,,1}, in te original image I. Te determined direction δ is tat for wic te directional variance is te lowest : ( ) δ = argmin d {x,y} σ d (P). (14) Tanks to tis step, te green levels of te tree selected adjacent pixels are locally te most omogeneous. 2. Evaluate te alternation amplitude at pixel P, etween te tree adjacent pixels along direction δ, in te original and estimated images. Wen δ is orizontal, te amplitude on a plane I is computed as : α x (I,P) = I x 1,y I x,y + I x,y I x+1,y I x 1,y I x+1,y, (15) Wen δ is vertical, te amplitude is computed as : α y (I,P) = Ix,y 1 I x,y + Ix,y I x,y+1 Ix,y 1 I x,y+1. (16) Wen te tree green levels present an alternating ig-low-ig or low-iglow pattern, α δ (I,P) is strictly positive, oterwise zero. 3. Compare te alternation amplitudes on te plane of te original image I and tat of te demosaiced image Î. Wen α δ ( Î,P ) > α δ ( I,P ), te alternation amplitude of green levels as een amplified y demosaicing along te direction δ. Te pixel P is retained as a candidate pixel affected y zipper effect. 4. Apply to tese candidate pixels a modified te sceme proposed y Lu and Tan, except tat te neigoring pixel P wose color is te closest to P as to e one of te two neigoring pixels along te selected direction δ. 64

66 False Colors Measurement We also propose a measurement for te false color artifact (Yang et al., 27). At a pixel in te demosaiced image, any mere error in te estimated value a color component can e considered as a false color. However, te uman visual system cannot actually distinguis any sutle color difference wic is lower tan a specific tresold (Faugeras, 1979). We consider tat te estimated color at a pixel is false wen te asolute difference etween an estimated color component and te original one is iger tan a tresold T. Te proposed measurement FC % is te ratio etween te numer of pixels affected y false colors and te image size : FC % = 1 { ( I XY Card k P(x,y) max x,y Î k x,y k=r,,b ) } > T. (17) FC % is easy to e implemented and expresses te rate of pixels affected y false colors as a measurement of te performance reaced y a demosaicing sceme. Moreover, tis criterion can e also used to locate pixels affected y false colors. However, as classical fidelity criteria, it requires te original image in order to compare te efficiency of demosaicing scemes Measurements Dedicated to Low-level Image Analysis Since te demosaicing metods intend to produce perceptually satisfying demosaiced images, te most widely used evaluation criteria are ased on te fidelity to te original images. Rater tan displaying images, our long-term goal is pattern recognition y means of feature analysis. Tese features extracted from te demosaiced images are mostly derived from eiter colors or detected edges. Artifacts generated y demosaicing (mostly lurring and false colors) may affect te performance of edge detection metods applied to te demosaiced image. Indeed, lurring reduces te sarpness of edges, and false colors can give rise to irrelevant edges. Tat is wy we propose to quantify te demosaicing performance y measuring te rates of erroneously detected edge pixels Measurements of Su- and Over-detected Edges. Te edge detection procedure is sensitive to te alteration of ig spatial frequencies caused y demosaicing. Indeed, low-pass filtering tends to generate lurring, and so to smoot edges. Moreover, wen te demosaicing sceme generates false colors or zipper effect, it may give rise to anormally ig values of a color gradient module. Te respective expected consequences are su- and over-detection of edges. Notice tat te different demosaicing algoritms are more or less efficient in avoiding to generate lurring, false color and zipper effect artifacts. So, we propose a new evaluation sceme wic performs tese successive steps (Yang et al., 27) : 1. Apply a ysteresis tresolding of te module of te color gradient proposed y Di Zenzo (1986), in order to detect edges in te original image I. Te same edge detection sceme wit te same parameters is applied to te demosaiced image Î. Edge detection is performed as follows : 65

67 (a) Compute te square module of te Di Zenzo gradient at eac pixel in image I as : I 2 = 1 ( ) a+c+ (a c) , (18) θ = 1 ( ) 2 2 arctan, (19) a c were coefficients a, and c are computed y approximating te partial derivatives of te image function I : a = ( I x) 2 ( x (I R ) ) 2 + ( x (I ) ) 2 + ( x (I B ) ) 2, = I I x y x (I R ) y (I R )+ x (I ) y (I )+ x (I B ) y (I B ), c = ( I y) 2 ( y (I R ) ) 2 + ( y (I ) ) 2 + ( y (I B ) ) 2. Eac approximative partial derivative d (I k ), d {x,y}, k {R,,B}, is computed tanks to te Derice operator (Derice, 1987). () Find te local maxima of te vector gradient module I. (c) Among pixels wic are associated wit local maxima, detect te edge pixels tanks to a ysteresis tresolding, parametrized y a low tresold T l and a ig tresold T. 2. Store te edge detection result for te original image in a inary edge map B, and similarly for te demosaiced image edges in ˆB. Notice tat tese two maps, in wic edge pixels are laeled as wite, may e different due to artifacts in te demosaiced image. 3. In order to quantify te influence of demosaicing on edge detection quality, we propose to follow te strategy developed y Martin et al. (24). Edge maps B and ˆB are compared y means of two successive operators (see figure 35a) : (a) Apply te XOR logical operator to edge maps B and ˆB, in order to enance te differences etween tem in a new inary map J ; () Apply te AND logical operator to maps J and B, wic results in te inary su-detected edge map SD. Similarly, te AND logical operator is applied to maps J and ˆB, wic results in te inary over-detected edge map OD. Pixels laeled as wite in map SD are edge pixels wic are detected in te original image I ut undetected in te demosaiced image Î. Pixels laeled as wite in te image OD are edge pixels erroneously detected in te demosaiced image Î, compared wit edge pixels detected in I. 66

68 Original image I Demosaiced image Î Edge detection Edge detection Original edges B XOR Demosaiced edges ˆB Difference map J = B XOR ˆB AND AND Su-detection map SD = J AND B Over-detection map OD = J AND ˆB (a) eneral sceme SD I B J OD Î ˆB () Example FI. 35: Steps to measure te quality of edge detection. In sufigure (), over-detected edge pixels are laeled as (in old typeface) in ˆB, J and OD, in order to distinguis tem from su-detected edge pixels (laeled as ). 67

69 SD SD (X) OD (X) ÕD FI. 36: Computing SD and ÕD from SD and OD, on an example. Pixels laeled as dotted elong to pairs of sifted edge pixels, and are dropped out in te final detection maps. 4. Compute te rates of su- and over-detected edge pixels respectively as : SD % = 1 XY Card{ P(x,y) SD x,y }, (11) OD % = 1 XY Card{ P(x,y) OD x,y }. (111) Finally, te rate of erroneously detected edge pixels is expressed as ED % = SD % + OD % Measurements Based on Sifted Edges By visually examining te map J in figure 35, we notice te presence of many pairs of adjacent edge pixels. In suc edge pairs, one pixel is detected in B only (i.e. sudetected), and te oter one in ˆB only (i.e. over-detected). For example, te map J of figure 35 presents five pairs of adjacent pixels composed of a su-detected edge pixel (laeled as ) and an over-detected edge pixel (laeled as in old typeface). Tese cases do not result from a ad edge detection, ut from a spatial sift of edge pixels etween te original and demosaiced images. A su-detected (respectively, overdetected) edge pixel is sifted wen at least one of its neigors is an over-detected (respectively, su-detected) edge pixel. Suc pairs of pixels are ereafter called pairs of sifted (edge) pixels. In order to caracterize te effect of demosaicing on edge detection precisely, we want to distinguis pairs of sifted edge pixels from oter edge pixels. For tis purpose, we represent unsifted su- and over-detected edge pixels as two inary maps respectively denoted as SD and ÕD, and defined as : 68

70 SD x,y SD x,y ( Q(x,y ) N 8 (P(x,y)) ODx,y ), (112) ÕD x,y OD x,y ( Q(x,y ) N 8 (P(x,y)) SD x,y ), (113) were symol represents te logical AND operator. Figure 36 illustrates, from te example of figure 35, ow maps SD and ÕD are otained. In tis figure, maps SD and OD used to uild SD and ÕD are superimposed in order to igligt te pairs of sifted edge pixels. From te two inary maps SD and ÕD, we compute te rates of su- and overdetected unsifted edge pixels as : SD % = 1 { } XY Card P(x,y) SD x,y, (114) ÕD % = 1 { } XY Card P(x,y) ÕD x,y. (115) Tese rates are used to evaluate precisely te quality of edge detection in demosaiced images Conclusion In tis section, we ave presented te tecniques of ojective evaluation of demosaicing quality. For tis purpose, we ave first presented te most occurred artifacts caused y demosaicing. Blurring, false colors and zipper effect damage te quality of te demosaiced images. Ten, we ave presented classical criteria wic total te errors etween te original and estimated colors troug te image. Tese criteria ave some limits since tey provide a gloal estimation of te demosaicing quality and do not reflect te judgment of an oserver. Indeed, tey do not quantify te occurrences of artifacts wic can e identified y an oserver. Terefore, we ave descried measurements dedicated to tree kinds of artifacts. In te computer vision context, most images are acquired y color mono-sensor cameras in order to e automatically processed. So, te quality of demosaicing affects te quality of low-level image analysis scemes. Tat is te reason wy we ave proposed criteria wic are ased on te quality of edge detection. 69

71 5. Quality Evaluation Results 5.1. Results of Classical Criteria Te quality of demosaicing results, acieved y te ten metods detailed in section 2, as een first evaluated tanks to classical criteria. For tis purpose, te twelve mostly used images of Kodak encmark dataase are considered (see figure 37) 8. Tese images, all of size pixels, ave een selected in order to present a significant variety of omogeneous regions, colors and textured areas. Tale 3 displays te results otained wit criteria wic measure te fidelity of eac demosaiced image to its corresponding original image, namely te mean asolute error (MAE, expression (82)), te peak signal-to-noise ratio (PSNR, expression (85)) and te correlation criterion (C, expression (87)). Tale 4 sows, for te same images and demosaicing scemes, te results otained wit perceptual criteria, namely te estimation error in CIE L a color space ( E L a, expression (92)), te estimation error in S-CIE L a color space ( E S-L a ) and te criterion of normalized color difference (NCD, expression (97)) etween te demosaiced image and its original image. Tese two tales sow tat for a given metod, te performances measured wit a specific criterion vary from an image to anoter one. Tis confirms tat otaining a good color estimation from te CFA image is all te more difficult as te image is ric in ig spatial frequency areas. For instance, te PSNR of images demosaiced y ilinear interpolation ranges from 24.5 db for image 4 ( House ), wic contains a lot of ig frequency areas, to 36 db for image 1 ( Parrots ), wic contains a lot of omogeneous regions. It can e noticed tat te two metods wic ciefly use te frequency domain provide etter results tan tose wic only scan te spatial domain. Moreover, te metod proposed y Duois (25) acieves te est average results over te twelve images, watever te considered criterion. We also notice tat te different criteria provide similar performance rankings for te metods on a given image Results of Artifact-sensitive Measurements Zipper Effect Measurements In order to compare te relevance of te results provided y te two zipper effect measurements descried in section 4.4.2, we propose to use te following procedure. First, a ground trut is uilt for te zipper effect y visually examining te demosaiced image and defining weter eac pixel is affected y zipper effect or not. Ten, te two measurements are applied, in order to provide inary maps were pixels wic are affected y zipper effect are laeled as wite. A final comparison step of tese inary maps wit te ground trut quantifies te performance of eac ojective measurement, y counting pixels were zipper effect is correctly detected, su-detected and overdetected. Figure 38 displays te results on four image extracts of size 1 1 pixels. It sows tat te directional alternation measurement generally fits etter wit te ground trut 8 Tis dataase is availale at ttp:// 7

72 1. Parrots 2. Sailoat 3. Windows 4. Houses 5. Race 6. Pier 7. Island 8. Ligtouse 9. Plane 1. Cape Barn 12. Calet FI. 37: Te twelve encmark images picked up from Kodak dataase. Images 5 and 8 are presented vertically for illustration purpose, ut ave een analyzed in landscape orientation.

73 Image Criterion Bilinear Cst. Hue Hamilton Wu Cok Kimmel Li unturk Duois Lian Avg. MAE PSNR C MAE PSNR C MAE PSNR C MAE PSNR C MAE PSNR C MAE PSNR C MAE PSNR C MAE PSNR C MAE PSNR C MAE PSNR C MAE PSNR C MAE PSNR C MAE PSNR C TAB. 3: Demosaicing quality results, for twelve color images from Kodak dataase, according to fidelity criteria : mean asolute error (MAE), peak signal-to-noise ratio (PSNR, in deciels), and correlation (C) etween te original image and te demosaiced image. For eac image and eac criterion, te est result is written in old typeface. Te tested metods are : 1. Bilinear interpolation 2. Constant-ue-ased interpolation (Cok, 1987) 3. radient-ased metod (Hamilton and Adams, 1997) 4. Component-consistent sceme (Wu and Zang, 24) 5. Metod ased on template matcing (Cok, 1986) 6. Adaptive weigted-edge metod (Kimmel, 1999) 7. Covariance-ased metod (Li and Orcard, 21) 8. Alternating projection metod (unturk et al., 22) 9. Frequency selection metod (Duois, 25) 1. Metod ased on frequency and spatial analyses (Lian et al., 27). 72

74 Image Criterion Bilinear Cst. Hue Hamilton Wu Cok Kimmel Li unturk Duois Lian Avg. E L a E S-L a NCD E L a E S-L a NCD E L a E S-L a NCD E L a E S-L a NCD E L a E S-L a NCD E L a E S-L a NCD E L a E S-L a NCD E L a E S-L a NCD E L a E S-L a NCD E L a E S-L a NCD E L a E S-L a NCD E L a E S-L a NCD E L a E S-L a NCD TAB. 4: Demosaicing quality results, for twelve color images from Kodak dataase, according to perceptual criteria : estimation error in CIE L a color space ( E L a ), estimation error in S-CIE L a color space ( E S-L a ), and criterion of normalized color difference (NCD). For eac image and eac criterion, te est result (i.e. lowest value) is written in old typeface. Images and tested metods are te same as in tale 3. Te illuminant used for (X,Y,Z) transform is te standard CIE D65, wic corresponds to dayligt. 73

75 (a1) (1) (c1) (d1) (a2) (2) (c2) (d2) (a3) (3) (c3) (d3) (a4) (4) (c4) (d4) FI. 38: Zipper effect detection in four Kodak image extracts, according to two measurements. (a1) (a4) : original extracts. (1) (4) : demosaiced extracts. Last two columns : pixels affected y zipper effect, according to Lu and Tan s criterion (c1) (c4) and to te directional alternation (d1) (d4). Pixels affected y zipper effect are laeled as. Tey correspond to ground trut in images (1) (4). In images (c1) (d4), te ground trut is reproduced as gray-laeled pixels. So, pixels were te zipper effect is well detected are ot laeled as and gray. Pixels were te zipper effect is su-detected (respectively, over-detected) are laeled only as (respectively, only as gray). Images (1) and (2) are estimated y ilinear interpolation, (3) and (4) y Hamilton and Adams (1997) gradient-ased metod. 74

76 Well-detected Su-detected Over-detected Image Lu and Directional Lu and Directional Lu and Directional Tan alternation Tan alternation Tan alternation (a1) 1 1 (a2) (a3) (a4) Total TAB. 5: Comparison etween te measurements quantifying zipper effect, proposed y Lu and Tan (23) and ased on te directional alternation. Values correspond to te numers of well-detected, su-detected and over-detected pixels affected y tis artifact in te four image extracts of figure 38. tan Lu and Tan s measurement does. Tis remark is confirmed numerically y comparing te numers of well-detected, su-detected and over-detected pixels affected y zipper effect in te four images. Te results in tale 5 sow tat te measurement ased on directional alternation generally provides iger well-detected pixel rates tan te one proposed y Lu and Tan. Indeed, te latter over-detects zipper effect wereas te measurement ased on directional alternation tends to sligtly su-detect tis artifact. Finally, we ave compared te demosaicing scemes according to te measurement ased on directional alternation. Tale 6 sows tat te results are similar to tose otained wit classical criteria, presented in tales 3 and 4 : ilinear interpolation always generates te igest amount of zipper effect, wereas te sceme proposed y Lian et al. (27) is overall te most efficient. However, y examining tale 6 in detail, we notice tat in images wit few ig spatial frequencies (numer 2- Sailoat and 7- Island ), te metod proposed y Duois tends to generate less zipper artifact tan Lian et al. s metod does. enerally speaking, tese results sow tat te metods wic analyze te frequency domain generate less zipper effect tan tose wic scan te image plane (Menon et al., 26) False colors As descried in section 4.4.3, te estimated color at a pixel is taken as false wen te asolute difference etween an estimated color component and te original one is iger tan a tresold T (see equation (17)). Since adjusting tis tresold is not easy, we compare te performance reaced y a set of ten demosaicing scemes applied to twelve images of te Kodak dataase, wen T varies from 1 to 25 wit an incremental step of 5. Figure 39 sows ot te evolution of te average rate of false colors wit respect to T for a given sceme, and te rates of false colors generated y te considered scemes for a given value of T. As expected, te rate of false colors decreases wen T increases. More interestingly, te relative ranking of demosaicing metods wit respect to te numer of false colors is consistent wit ot rankings provided y classical fidelity criteria and y measurements ased on zipper effect. 75

77 Image Bilinear Cst. Hue Hamilton Wu Cok Kimmel Li unturk Duois Lian Avg TAB. 6: Rates ZE % of pixels affected y zipper effect, according to te measurement ased on directional alternation. Te images and tested metods are te same as in tale 3. FC% Bilinear Cst. Hue Hamilton Wu Cok Kimmel Li unturk Duois Lian T 2 25 FI. 39: Average rates of false colors FC % wit respect to te detection tresold T. Te twelve considered images and ten tested metods are te same as in tale 3. 76

78 5.3. Discussion Te most widely used criteria for te evaluation of demosaicing quality are MSE and PSNR, te latter eing a logaritmic form of te MSE criterion. Several reasons explain wy most of te autors use tese criteria (Wang and Bovik, 26). First, tese functions are easy to e implemented and teir derivatives can e estimated. Tey may terefore e integrated into an optimization sceme. Second, te PSNR criterion as a real pysical meaning namely, te maximal energy of te signal wit respect to errors generated y demosaicing, wic can also e analyzed in te frequency domain. However, te PSNR criterion provides a general estimation of te demosaicing quality, ut does not really reflect te uman judgment. For example, an oserver would prefer an image containing a large numer of pixels wit estimated colors close to te original ones, tan an image containing a reduced numer of pixels affected y visile artifacts. But MSE and PSNR criteria could provide identical values in ot cases, since tey do not discriminate te caracteristics of different artifacts in te demosaiced image. Tese ojective measurements ave een criticized (Wang and Bovik, 29) since tey cannot evaluate te image alteration as a uman oserver does (Eskicioglu and Fiser, 1995). Te alternative criteria E of estimation errors in te CIE L a and S-CIE L a color spaces are te most widely used perceptual criteria (Zang and Wandell, 1997). Tey are ased on perceptually uniform color spaces as an attempt to represent te uman perception, ut require prior knowledge aout te illuminant and te reference wite used during image acquisition. Since te acquisition conditions are not always known, te quality of tese measurements may e iased Experimental Results for Edge Detection Te demosaicing performance as een evaluated wit respect to te quality of edge detection tanks to measurements detailed in section 4.5. Tale 7 displays te average rates of su-detected (SD % ), over-detected (OD % ) and erreneously detected (ED % = SD % + OD % ) edge pixels. Tese values ave een computed over te twelve Kodak images previously considered, and for te ten classical demosaicing scemes. Moreover, tis tale displays te average rates SD %, ÕD % and ẼD % wic take into account only unsifted edge pixels. Te lowest values correspond to te est demosaicing quality according to tese edge-dedicated measurements. By examining te average rates ED % and ẼD %, similar conclusions can e drawn aout te performances of demosaicing scemes. Te metods wic privilege te frequency domain allow to otain etter edge detection quality tan te oter metods do. Besides, te metods proposed y Duois and y Lian et al. provide te lowest error rates in ot edge and unsifted edge detection. Tese demosaicing scemes are terefore te most apt to e coupled wit edge detection procedures ased on color gradient. Moreover, we notice tat te ranking of te ten tested demosaicing scemes wit respect to OD % and SD % is relatively consistent wit te ranking otained wit measurements ÕD % and SD %. However, te rate of over-detected unsifted pixels is te lowest for ilinear interpolation. Tis suprising performance result can e explained 77

79 Meas. Bilinear Cst. Hue Hamilton Wu Cok Kimmel Li unturk Duois Lian SD % OD % ED % SD % ÕD % ẼD % TAB. 7: Average rates of su-detected edge pixels (SD % ), of over-detected edge pixels (OD % ) and erreneously detected pixels (ED % = SD % + OD % ). Average rates of sudetected unsifted edge pixels ( SD % ), of over-detected unsifted edge pixels (ÕD % ) and of unsifted edge pixels tat are erreneously detected (ẼD % = SD % + ÕD % ). Te low and ig tresolds used for ysteresis tresolding are set to 1 and 6, respectively. Te twelve considered images and ten tested metods are te same as in tale 3. y ot strong lurring and zipper effect generated y tis demosaicing metod. Indeed, lurring induces fewer detected edge pixels, and zipper effect mainly induces pairs of sifted edge pixels. For eac of te oter metods, te rates of su- and over-detected edge pixels are overall similar. Moreover, teir ranking is almost te same as te one otained wit te previous criteria. In tale 7, we also notice tat more tan te alf of su- and over-detected edge pixels according to measurements SD % and OD % are not retrieved wit measurements SD % and ÕD %. Tat means tat sifted edges strongly contriute to te dissimilarity etween edges detected in te original and demosaiced images. Edge pixels are su-detected ecause te color gradient module used to detect edges decreases wit lurring in demosaiced images. Te over-detected edge pixels correspond to an increase of te color gradient module in case of zipper effect or false colors. Tese new rates of su- and over-detected pixels SD % and ÕD % are ale to reflect te artifacts caused y demosaicing. From tale 7, we can evaluate te influence, on edge detection, of te demosaicing strategies implemented in te tested metods. Bot metods using ilinear interpolation and ue constancy estimate te pixel colors witout exploiting spatial correlation. Hence, tey generate more artifacts tan te tree oter metods wic exploit spatial correlation, and provide iger rates of su- and over-detected edge pixels. All in all, su- and over-detected edge pixels often coincide wit artifacts. Figure 4 sows images wic are demosaiced y two different scemes, and te respective maps of su- and over-detected unsifted edge pixels ( SD and ÕD). We notice tat demosaicing influences te edge detection more significantly in areas wit ig spatial frequencies and tat te artifacts are also mainly located in tese areas. Zipper effect often decreases te variation of levels in transition areas etween omogeneous regions. Hence, zipper effect tends to decrease te gradient module, so tat te norm of local maxima ecomes lower tan te ig tresold T used y te 78

80 (a) Image demosaiced y ilinear interpolation () Image demosaiced y Hamilton and Adams (c) Su-detected edge pixels SD in image (a) (d) Su-detected edge pixels SD in image () (e) Over-detected edge pixels ÕD in image (a) (f) Over-detected edge pixels ÕD in image () FI. 4: Su- and over-detected unsifted edge pixels, for two demosaicing scemes : ilinear interpolation and te gradient-ased metod proposed y Hamilton and Adams (1997). 79