International Journal O Computational Engineering Research (ceronlinecom) Vol Issue Optimized Color ransorms or Image Demosaicing Evgen Gershiov Department o Electrical Engineering, Ort Braude Academic College o Engineering, Karmiel 98, Israel and Department o Electrical Engineering, echnion - II, Haia 000, Israel Abstract: Most demosaicing algorithms toda are based on irst reconstructing the green (G) color component ollowed b the reconstruction o the red (R) and the blue (B) components based on the green his approach and the associated methods o using the dierences R G and B G are arbitrar and in most cases not optimal Instead, we propose optimal color transorms or demosaicing his optimization is based on energ compactness and smoothness o the color components We compare the perormance o the proposed algorithms to presentl available demosaicing methods and show that the new optimized approaches are superior both visuall and quantitativel Our conclusion is that the proposed color transorms improve the perormance o demosaicing algorithms Kewords: Baer pattern, Color transorm, Demosaicing, Energ compactness, Image interpolation, Optimization, Smoothness Introduction Since man acquisition devices are based on a single sensor using a color ilter arra (CFA), onl partiall sampled versions o the primar colors R, G, B are recorded his is done in most cases according to the Baer pattern [], as shown in Fig In this case, the green has twice as much samples as the red and the blue, maing the green interpolation easier to accomplish due to reduced potential o aliasing [,] hen the red and the blue components can be reconstructed based on inter-color correlations, which are usuall high in natural images [4-8] Straightorward algorithms or demosaicing, such as bilinear or bicubic interpolation methods, however, do not use these inter-color correlations and operate on each color component independentl Better perormance is achieved b algorithms that are based on the above sequential scenario o reconstructing G irst, ollowed b the reconstruction o R and B, eg, [9-4] In such algorithms the inter-color correlations are usuall eploited b interpolating the dierences R G and B G However, since no optimization is perormed, it can be shown that using these dierences is not the best method to perorm the tas eicientl For the sae o completeness, we should add that non-sequential demosaicing methods have also been proposed, eg the iterative techniques o [5] or [6] as well as vector CFA demosaicing [7] In this wor we propose methods o choosing other color transorms or the interpolation o the red and the blue according to dierent optimization criteria We consider the ollowing demosaicing algorithm he basic demosaicing algorithm he stages o the algorithm are: he green color component is interpolated using the method in [9] It consists o iltering the CFA pattern horizontall and verticall, then choosing the direction o interpolation corresponding to the smaller estimated gradient (to avoid interpolation across edges): horizontal or vertical In case o equal gradients the average o the horizontal and vertical interpolators is taen his technique o interpolation was chosen because it provides good perormance at low compleit he interpolated green component Ĝ is used in the reconstruction o the red and the blue colors he color RG dierences RGˆ BG, BGˆ are calculated at the nown piels o the red and the blue colors, respectivel hen the red-green dierence is interpolated at the locations o the nown blue samples and the blue-green dierence at the locations o the red samples using the local polnomial approimation (LPA) ilter [] Better perormance can be achieved b this ilter compared to simple bilinear interpolation Issn 50-005 (Online) March 0 wwwceronlinecom
he missing piels in the red and blue - those at the locations o the nown green piels are interpolated using simple averaging o their two vertical and two horizontal piels (bilinear interpolation) he RG BG interpolation is perormed once again on the and dierences resulting in ull images ˆ RG and ˆ BG he inal red and blue components are calculated according to ˆ ˆ ˆ RG R G and ˆ ˆ ˆ BG B G Figure he Baer CFA pattern he structure o this wor is as ollows Color transorms or demosaicing based on optimization o dierent properties o the image are presented in Section Demosaicing results or the proposed method and comparison to other available techniques are given in Section Section 4 provides summar and conclusions Optimal Color ransorms All printed material, including tet, illustrations, and charts, must be ept within a print area o 6-/ inches (65 cm) wide b 8-7/8 inches (5 cm) high Do not write or print anthing outside the print area All tet must be in a single-column ormat Columns are to be -/6 inches (785 cm) wide, with he basic algorithm perorms its interpolation in the G, R G, B G color space his choice is not necessaril optimal and thus other color transorms can be considered ollowing an optimization process [8] he change o the color space is not possible prior to the reconstruction o the green since at each piel o the image onl one o the primar colors is available However, it becomes possible ater the reconstruction o G in Subsection, Step We thus propose a new general color space: () C G, C a R a G, C d B d G or some constants a, a, d, d instead o the regular choice o avoid the solution o a a 0 in the optimization problems presented below, a constraint has to be added orcing the L norm, or eample, o the a coeicients to be (similarl or the d coeicients) hus a a and d d () Energ compactness and non-singularit A Rate-Distortion model or color image coders was developed and optimized in [9] As a result the optimal Color ransorm (C) was derived Denoting the C matri b M, the target unction to be minimized or the optimal C was ound to be MM GM, where GM is the geometric mean o the subband variances Based on this result, the ollowing target unction can be proposed or our demosaicing algorithm: var C () MM where C and the RGB components are connected b, Issn 50-005 (Online) March 0 wwwceronlinecom
(4) C R 0 0 C M G, M a a 0 C B 0 d d he epression in () is made o two terms: var C new color space and MM coeicients minimizing () under the norm constraint o () are (5), which is a measure o the energ compactness in the, which is a measure o the non-singularit o M he optimal var( G) cov( R, G) var( G) cov( B, G) a, a, d, d var( G) cov( R, G) var( G) cov( R, G) var( G) cov( B, G) var( G) cov( B, G) Since the target unction o () combines the properties o energ compactness and non-singularit o the color transorm, we reer to this algorithm as ECNS, which is the acronm o Energ Compactness and Non- Singularit Smoothness o the C and C components RG BG he energ o the dierence signals R Gand B G is mostl concentrated in the low requencies [] his is the reason or the good perormance achieved b interpolating these dierences using RG BG averaging [-] his also means that and are smooth o urther impose this smoothness on C and C, the ollowing methods are proposed Minimal high pass energ he idea here is to minimize the energ o C and C, iltered b a two dimensional High Pass () ilter We denote the iltered color components at piel (, ) M N i j C i j b C and minimize,, or an image o size M N Alternativel, a pair o one dimensional ilters and can be used to ilter C or C horizontall and verticall, respectivel Usuall, hen the epression to be minimized becomes C C where C is C iltered b where (7) and similarl or (6) a is chosen as,, i j i j, C he optimal a, a coeicients or this problem are, a, R R G G i j i j R G R G i j,, he solution or the d and d coeicients is the same as the solution or a and a, respectivel, in (6) with B replacing R everwhere in (7) For simple choices o, such as the bacward/orward approimation o the horizontal derivative Issn 50-005 (Online) March 0 wwwceronlinecom
Optimized Color ransorms For Image [ ], the calculations can be perormed on the available small images obtained rom the CFA (Fig ) Alternativel, R and B can be irst reconstructed using a simple technique, such as bilinear iltering o the RGand BGdierences, and then used or the estimation o the derivatives In this wor we use the Sobel 0 gradient operator given b 0 0 Figure he Baer pattern components: RR, BB, GR and GB (rom let to right) Minimal energ in the wavelet domain Another approach is to consider the energ o the C and C color components in the high requencies and to search or the coeicients ormulation or this problem is to minimize a and d, C W,, { LH, HL, HH } m n that minimize this energ One possible C, where W is the one level Discrete Wavelet ransorm (DW) o C at position ( mn, ) in subband, which is one o the high requenc subbands (LH, HL or HH) above he solution or this problem is as in (6), but now R G W, W, (8) { LH, HL, HH } m n { LH, HL, HH } m n { LH, HL, HH } m n he solution or d and d is similar R G W ( m, n) W Minimal relative energ in the Fourier domain he energ in the high requencies can be epressed in the requenc domain o the Discrete Fourier ransorm (DF) as well In this case taing the relative energ o C or C provides better results hereore, we see to minimize (or an image o size M Nassuming M and N are multiples o 4) (9) E M/ N/ mm /4 nn /4 M/ N/ m0 n0 DF DF C C,, C DF here denotes the DF coeicient o C at position ( mn, ) in the requenc domain he solution o this problem requires solving third order polnomial equations resulting in long epressions or the a and d coeicients For simplicit we do not provide them here Demosaicing Results he basic algorithm (Section ) was implemented with and without the optimization techniques o Section We also added the reinement method [0] as post-processing his method provides urther utilization o the inter-color and intra-color correlations and wors well with our algorithms he set o images given in Fig was used in our simulations,ie, or each one the Baer pattern was built, interpolated b the dierent algorithms and compared to the original image he distortion measure used was the S-CIELAB metric [] he comparison o the proposed algorithms is given on the let side o able We can see that all the proposed algorithms achieve better results than the basic algorithm and the bilinear interpolation he best perormance with respect to the S-CIELAB metric is achieved b the minimal high pass energ (Min ) 4 Issn 50-005 (Online) March 0 wwwceronlinecom
algorithm (Section ) his shows the importance o the smoothness o the C or C color components or our interpolation process he second best algorithm is ECNS, which means energ compactness and nonsingularit o the color transorm are important or demosaicing as well as image coding [8]It is o interest to compare the proposed methods to other available algorithms We have chosen some o the available state o the art techniques: Alternating Projections (AP [0]), Directional Linear Minimum Mean Square Error (DLMMSE []), Variance o Color Dierences (VCD []), Local Polnomial Approimation (LPA []) and Successive Approimation (SAP [6]) he results or these algorithms can be seen on the right side o able From the table we can see that the Min algorithm is superior to the other methods, while the perormance o ECNS is similar to that o the VCD method that provides the best results among the algorithms chosen or comparison Visual results or the new methods as well as eisting algorithms or part o Image 8 are given in Fig 4 As can be seen, the proposed methods provide better results than the other algorithms (including VCD that provides the most competitive perormance) he values o the a and d coeicients or these algorithms are given in able Note that even i the values are close to a a 05 and d d 05, which would result in taing the common R G and BGdierences (ater scaling), the new methods outperorm the basic algorithm 4 Summar And Conclusions An optimization approach to demosaicing has been introduced Instead o using the common choice o the R G and BGdierences or the reconstruction process, better perormance can be achieved b choosing an optimized color space according to the desired properties o the image Such properties can be energ compactness as in the ECNS algorithm or smoothness as in the Min algorithm A basic demosaicing algorithm has been optimized to achieve these properties and compared to other available demosaicing methods Our results show that the proposed optimization method signiicantl improves the interpolation perormance and that the best perormance is achieved b minimizing the high pass energ in the new color space he second best is the algorithm that combines energ compactness and non-singularit o the color transorm, providing better results also in the case o color image coding [8] Our conclusion is that the proposed optimization approach is useul or demosaicing o color images able S-CIELAB results or the algorithms (rom let to right): minimal High Pass energ, ECNS, minimal DW energ, minimal relative DF energ, the basic algorithm, bi-linear interpolation, AP, SAP, DLMMSE, LPA and VCD Image Min Proposed Algorithms ECNS Min DW Min Rel DF Basic BL AP SAP Other Algorithms DL MMSE LPA VCD 07 079 075 070 0769 505 085 0897 07 0758 0850 0779 0786 0794 080 0796 0 0 5 0749 0766 0778 0747 07 079 08 0808 50 65 77 08 08 0795 4 0645 0654 0650 0659 0656 08 0787 0877 0644 06 0687 5 0579 0595 057 059 0578 098 088 088 056 050 059 6 0576 06 0594 0577 0606 49 0654 0760 054 058 056 7 408 475 456 467 488 7 86 80 4 8 490 508 569 569 608 74 867 68 689 54 56 Mean 0870 0886 0897 0905 094 64 6 0894 0944 0888 5 Issn 50-005 (Online) March 0 wwwceronlinecom
able a and d coeicients or Image 8 or the proposed algorithms (same order o columns as in able ) Even i the values are close to a a 05 and d d 05 as in the basic method, the new methods outperorm the basic algorithm Min ECNS Min DW Min Rel DF a 059 0578 0506 0460 a -046-04 -0494-0540 d 055 0597 0509 0559 d -0449-040 -049-044 Original AP SAP DLMMSE LPA VCD New Alg : Min New Alg : ECNS 6 Issn 50-005 (Online) March 0 wwwceronlinecom
New Alg : Min DW New Alg 4: Min Rel DF Figure 4 Demosaicing results or the dierent algorithms or part o Image 8 New Alg -4 are the new algorithms Reerences [] B E Baer Color imaging arra US Patent 97065, Jul 976 [] B Guntur, X Li and L Zhang Image demosaicing: A sstematic surve In Proc o SPIE, 68J-68J-5, 008 [] H Kirshner and M Porat On the Approimation o L Inner Products rom Sampled Data IEEE rans on Signal Processing 55:6-44, 007 [4] H Kotera and K Kanamori A Novel Coding Algorithm or Representing Full Color Images b a Single Color Image J Imaging echnolog 6:4-5, Aug 990 [5] J O Limb and CB Rubinstein Statistical Dependence Between Components o A Dierentiall Quantized Color Signal IEEE rans on Communications 0:890-899, Oct 97 [6] H Yamaguchi Eicient Encoding o Colored Pictures in R, G, B Components IEEE rans on Communications :0-09, Nov 984 [7] Y Roterman and M Porat Color image coding using regional correlation o primar colors Image and Vision Computing 5:67-65, 007 [8] E Gershiov and M Porat Optimal color image compression using localized color component transorms In Proc EUSIPCO 008, Lausanne, Switzerland, Aug 008 [9] J F Hamilton and J E Adams Adaptive Color Plane Interpolation in Single Sensor Color Electronic Camera US Patent 56974, 997 [0] B K Guntur, Y Altunbasa, and R M Mersereau Color plane interpolation using alternating projections IEEE rans Image Processing :997-0, 00 [] L Zhang and X Wu Color demosaicing via directional linear minimum mean square-error estimation IEEE rans on Image Processing 4:67-78, 005 [] K-H Chung and Y-H Chan Color demosaicing using variance o color dierences IEEE rans on Image Processing 5:944-955, 006 [] D Pali, V Katovni, R Bilcu, S Alenius, and K Egiazarian Spatiall adaptive color ilter arra interpolation or noiseless and nois data Int Journal o Imag Ss and echnolog 7:05-, 007 [4] R Sher and M Porat CCD Image Demosaicing using Localized Correlations In Proc o EUSIPCO, Poznan, Poland, Sept 007 [5] R Kimmel Demosaicing: Image reconstruction rom color ccd samples IEEE rans on Image Processing 8:- 8, 999 [6] X Li Demosaicing b successive approimation IEEE rans Image Processing 4:70-79 [7] M R Gupta and Chen Vector color ilter arra demosaicing In Proc o SPIE, Sensors and Camera Sstems or Scientiic, Industrial, and Digital Photograph Applications II 406:74-8, 00 [8] E Gershiov and M Porat A rate-distortion approach to optimal color image compression In Proc EUSIPCO, Florence, Ital, Sept 006 [9] E Gershiov and M Porat On color transorms and bit allocation or optimal subband image compression Signal Processing: Image Communication :-8,007 [0] L Chang and Y P am Eective use o spatial and spectral correlations or color ilter arra demosaicing IEEE rans Consumer Electronics, 50:55-65, Feb 004 [] X Zhang and B A Wandell A spatial etension o CIELAB or digital color image reproduction SID Journal 5:6-6, 997 7 Issn 50-005 (Online) March 0 wwwceronlinecom