Adaptive Fuzzy Color Interpolation

Transcription

1 Joural of Electroic Imagig Vol. (3), July 00. Adaptive Fuzzy Color Iterpolatio Pig-Sig Tsai, Tiku Acharya,, Ajay K. Ray 3 Itel Corporatio Desktop Architecture Lab 5000 West Chadler Boulevard Mail Stop: CH7- Chadler, Arizoa 856, USA Departmet of Electrical Egieerig Arizoa State Uiversity Tempe, Arizoa 8587, USA 3 Departmet of Electroics ad Electrical Commuicatio Egieerig Idia Istitute of Techology Kharagpur, Idia pstsai@ieee.org

2 Joural of Electroic Imagig Vol. (3), July 00. Abstract I a electroic color imagig device, such as a digital camera, usig a sigle CCD or CMOS sesor, the color iformatio is usually acquired i sub-sampled patters of Red (R), Gree (G) ad Blue (B) pixels. Full resolutio color is subsequetly geerated from this sub-sampled image. This is popularly called Color Iterpolatio or Color Demosaicig. I this paper, we preset a color iterpolatio algorithm usig a method of fuzzy membership assigmet alog with the cocept of smooth hue trasitio. The algorithm is adaptive i ature ad produces superior quality full resolutio color images compared to most of the popularly kow color iterpolatio algorithms i the literature. We preset the results of compariso with some challegig subsampled images for color iterpolatio. Keywords: Color Iterpolatio, Color Demosaicig, Fuzzy Membership, Hue, Digital Imagig. Itroductio Due to the cost ad packagig cosideratio, i digital imagig devices such as a digital camera, the image color is captured i a sub-sampled patter. Typically the raw image is captured with each "pixel'' locatio composed of oly oe of the three primary color compoets, R (Red), G (Gree), or B (Blue). This sub-sampled color image is geerated usig certai patter of a Color Filter Array (CFA). This CFA is realized by coatig the surface of the electroic sesor array usig some optical material that acts as bad-pass filter. This coatig o each pixel locatio of the sesor array permits the photos correspodig to oly oe color compoet (frequecy rage) to be trasmitted to the sesor ad the other two color compoets are blocked. A typical ad widely used CFA patter is called Bayer Patter [], as show i Figure. It should be oted that there are may other patters kow i the literature. However, we oly cosider the Bayer patter i this paper. m G R G R G B G B G B 3 G R G R G 4 B G B G B 5 G R G R G Figure : Bayer CFA patter A perfect, rather efficiet full color represetatio of a image eeds the iformatio of all the three colors i each pixel locatio. As a result, each pixel eeds to be represeted by 4 bit color, assumig 8 bits for each of R, G ad B. To achieve this, it is essetial to iterpolate the missig two colors i each pixel locatio usig the iformatio of the eighborig pixels. The methodology to recover or iterpolate these missig colors is popularly kow as Color Iterpolatio or Color Demosaicig. We used the termiology Color Iterpolatio i this paper, istead of Color Demosaicig. I this paper, we preset a brief review of the color iterpolatio algorithms existig i the literature i sectio. We proposed a ew color iterpolatio algorithm usig the cocept of Fuzzy membership assigmet based scheme ad smooth hue trasitio i sectio 3. We preset the experimetal results ad compariso of our

3 Joural of Electroic Imagig Vol. (3), July 00. proposed techique with other popular techiques i sectio 4. We coclude this paper i sectio 5.. Related Works I the past decade, may color iterpolatio algorithms have bee proposed i the literature. This area of research ad developmet is gettig more ad more attetio, thaks to the boomig market of digital imagig devices. These color iterpolatio algorithms ca be broadly classified ito o-adaptive algorithms ad adaptive algorithms, as described below. () No-adaptive algorithms: I o-adaptive color iterpolatio algorithms, a fixed patter of computatio is applied i every pixel locatio i the sub-sampled color image i order to recover the missig two color compoets. Usually, this type of algorithms is easy to implemet with low cost i terms of computatioal requiremets. () Adaptive algorithms: I adaptive color iterpolatio algorithms, itelliget processig is applied i every pixel locatio based o the characteristics of the image i order to recover the missig color compoets. This type of algorithms yield better results i terms of quality as compared with the o-adaptive algorithms. However, effective algorithms i this category are usually more computatioally itesive. We review some algorithms from both categories i order to itroduce some flavor ad characteristics of the color iterpolatio methodologies proposed i the literature. However, there are other algorithms that we did ot iclude i this paper, such as Cubic Covolutio type methods [, 3], Patter Recogitio Iterpolatio method [4], Patter Classificatio method [5], gradiet based method [6], ad so o. I this paper, we oly give a review to some basic methods that are related to the thought process of our proposed ew algorithm. Iterested readers are also suggested to visit the web-site at Staford Uiversity [7].. No-adaptive algorithms Nearest Neighbor Replicatio: I this simple color iterpolatio method [8, 9], each missig color i a pixel is allocated by the value of the earest pixel of the same color i the iput image. The earest eighbor ca be ay oe of the upper, lower, left ad right pixels. As discussed by Adams [8], the oly advatage of this approach is that computatioal requiremet is very small ad suitable for applicatios where speed is very crucial. However, the sigificat color errors make it uacceptable for still imagig syste such as high-resolutio digital cameras. Biliear Iterpolatio: Istead of replicatig the earest eighbors, biliear iterpolatio [8] allocates the missig color compoet with the liear average of the adjacet pixels with same color compoet. For example, the pixel locatio (, 3) i Figure cotais BLUE compoet oly. Hece the missig GREEN compoet ca be estimated as average of the left, right, top ad bottom GREEN pixel values. The missig RED compoet ca be estimated as average of the four diagoally adjacet corer eighbors cotaiig RED pixels. This method is very simple ad ca be easily implemeted. However, experimetal results show that ew kid of pixel artifacts, e.g. zipper effect, is itroduced i the eighborhood of the iterpolated pixels i the iterpolated full color image. This artifact may be acceptable i a video applicatio because the artifact may ot be visible by the huma eye due to effect of motio blur betwee video frames, but these artifacts are ot acceptable for a still imagig system. 3

4 Joural of Electroic Imagig Vol. (3), July 00. Media Iterpolatio: The media iterpolatio [0] allocates the missig color compoet with the media value of the adjacet pixels of same color compoet, as opposed to the liear average used i biliear iterpolatio. This provides a slightly better result i terms of visual quality as compared with the biliear iterpolatio. However, the resultat images are still blurry for images with high frequecy cotets, ad for high resolutio still imagig syste this is still ot acceptable. Smooth Hue Trasitio Iterpolatio: The key problem of the color artifacts i both biliear ad media iterpolatio is that the hue values of adjacet pixels chage suddely (o-smoothly). O the other had, the Bayer CFA patter ca be cosidered as combiatio of a lumiace chael (gree pixels) ad two chromiace chaels (red ad blue pixels). The smooth hue trasitio iterpolatio method [] iterpolates these chaels differetly. The missig GREEN compoet i every RED ad BLUE pixel locatios i the Bayer patter ca first be iterpolated usig biliear iterpolatio as discussed before. The idea of chromiace chael iterpolatio is to impose a smooth trasitio i hue value from pixel to pixel. I order to do so, it defies blue "hue value" as B/G, ad red "hue value" as R/G. For iterpolatio of the missig blue pixel values B m, i pixel locatio ( ) i the Bayer patter, the followig three differet cases may arise. Case : the pixel locatio ( ) cotais GREEN color compoet oly ad the adjacet left ad right pixel locatios cotai BLUE color compoet oly. For example, the pixel locatio (, ) i Figure cotais GREEN compoet oly ad its adjacet pixels i left ad right cotai oly BLUE iformatio. The BLUE iformatio i locatio ( ) ca be estimated as follows: B B,, * = G m G G B. m Case : the pixel locatio ( ) cotais GREEN color compoet oly ad the adjacet top ad bottom pixel locatios cotai BLUE color compoet oly. The pixel at locatio (3, 3) i Figure is such a example. The BLUE iformatio i locatio ( ) ca be estimated as follows: Bm, Bm,,, * = G m Gm, Gm, B. m Case 3: the pixel locatio ( ) cotais RED color compoet oly. Obviously, four diagoally eighborig corer pixels cotai BLUE color oly. For example, the pixel locatio (3, ) i Figure cotais RED color compoet oly. The BLUE iformatio i locatio ( ) ca be estimated as follows: Bm, Bm, Bm, Bm,,, * = G m Gm, Gm, Gm, Gm, B. m The iterpolatio of missig RED pixel values ca be carried out similarity. As metioed by Adams [8], depedig o where the iterpolatio step happes i the image processig chai the defiitio of "hue value" chages. For example, if the pixel value is trasformed ito logarithmic exposure space from liear space before iterpolatio, istead of B/G or R/G, oe ca ow defie the "hue value" as B-G or R-G. 4 4

5 Joural of Electroic Imagig Vol. (3), July 00. This is comig from the fact that log(x/y) = log(x) log(y) = X' Y'. Here X ad Y are the logarithmic values of X ad Y respectively. Sice the liear/oliear trasformatio ca be doe usig a simple table look-up ad all the divisio for calculatig hue value is replaced by subtractio, this helps reduce computatioal complexity for implemetatio.. Adaptive algorithms Patter Matchig based Iterpolatio Algorithm: I the Bayer patter, a BLUE or RED pixel has four eighborig GREEN pixels. A simple patter matchig techique for recostructig the missig color compoets based o the pixel cotexts was proposed by Wu, et. al. []. This patter matchig algorithm defies a gree patter for the pixel at locatio ( ) cotaiig a o-green color compoet as a four-dimesioal iteger-valued vector: g( ) = ( Gm,, Gm,, G, G ). The similarity (or differece) betwee two gree patters g ad g is defied as the vector -orm g g = g g. 0 i< 4 Whe the differece betwee two gree patters is small, it is likely that the two pixel locatios where the two gree patters are defied will have similar RED ad BLUE color compoets. A weighted average proportioal to degree of similarity of the gree patters is used to calculate the missig color compoet based o the gree patter cotexts. For example, the missig BLUE color value B m, i pixel locatio ( ) cotais oly RED color compoet is estimated by comparig the gree patter g ( ) with the four eighborig gree patters g ( m, ), g ( m, ), g ( m, ) ad g ( m, ). If all the differeces betwee g ( ) ad other four gree patters are uiformly small, the a simple average is used to estimated the missig BLUE color compoet, Bm, Bm, Bm, Bm, B =. 4 Otherwise, whe the largest differece is above certai threshold, oly the top two bestmatched gree patters iformatio are used. If g ( ) g( m, ) ad g ( ) g( m, ) are the two smallest differeces, the the missig BLUE color is estimated as follows. g( ) g( m, ) Bm, g( ) g( m, ) Bm, Bm, = g( ) g( m, ) g( ) g( m, ) The missig RED color values ca be carried out aalogously. This algorithm is simple ad efficiet. However, as poited out by Wu, et. al. [], the quality of recostructed images is still udesirable. Block Matchig Based Algorithm: Similar to the patter matchig algorithm described previously, the block matchig algorithm by Acharya, et. al. [3] defies the Color Block of a o-gree pixel as the vector formed by the four eighborig gree pixels. However, istead of usig vector orm to measure the similarity, a ew metrics Color i i 5

6 Joural of Electroic Imagig Vol. (3), July 00. Gravity is defied as the mea value of the four vector compoets for measurig the similarity betwee color blocks. Let x = ( x, x, x3, x4) be a vector of a Color Block, x x x3 x4 the the Color Gravity for this color block will be x = ( ). The 4 similarity betwee two color blocks is defied as the absolute differece of the two color gravity values. Also istead of usig the weighted average proportioal to the similarity of the gree patters for estimatig the missig color values, the block matchig algorithm is developed based o the selectio of a eighborig block whose Color Gravity is closest to the Color Gravity of the color block uder cosideratio. For a o-gree pixel i the Bayer patter image, there are four eighborig gree pixels G (the North eighbor), Gs (South), Ge (East), ad Gw (West) which form the Color Block g = ( G, Gs, Ge, Gw) whose Color Gravity is g. The missig GREEN value G is simply computed by the media of the four eighborig gree pixel value. If the pixel locatio cotais BLUE color value, it will have four diagoally RED pixels Re (North-East), Rse (South-East), Rsw (South-West), ad Rw (North-West) whose color blocks are g e, g se, g sw, ad g w. The correspodig color gravity are g e, g se, g sw, ad g w respectively. The missig RED value R is replaced usig oe of the four diagoally red pixels based o best match of their color gravity values. The best match or miimal differece mi is the miimum amogst the four absolute differeces = g ge, = g g se, 3 = g g sw, ad 4 = g gw respectively. Similarly, we ca estimate the missig BLUE color value, if the pixel locatio cotais RED color value due to the symmetry of red ad blue samplig positio i a Bayer patter image. For the gree pixel locatio, oly two color blocks (either up-bottom or left-right positios) are cosidered for the missig RED or BLUE color. Similar operatio ca be carried out aalogously. The algorithm ca be described as follows: Begi if the pixel locatio is ot GREEN the { G media{g, Gs, Ge, Gw }; compute = g ge, = g g se, 3 = g g sw, ad 4 = g gw ; Fid mi = mi{,, 3, 4} ; If ( mi = ) the R Re if Red is missig, B Be if Blue is missig; If ( = mi ) the R Rse if Red is missig, B Bse if Blue is missig; If ( = mi 3 ) the R Rsw if Red is missig, B Bsw if Blue is missig; If ( = mi 4 ) the R Rw if Red is missig, B Bw if Blue is missig; }; if the pixel locatio is GREEN the { compute u = G gu, b = G gb, l = G gl ad r = G gr ; If ( u < b ) the B Bu else B Bb ; 6

7 Joural of Electroic Imagig Vol. (3), July 00. } If ( l < r ) the R Rl else R Rr ; Ed. This method provides a much sharper image as compared with the simple media or biliear iterpolatio type methods ad the simple patter matchig method. However, sice this method does ot cosider smooth hue trasitio, the color bleedig artifacts may still be a problem for some images such as image of a Zebra. Edge Sesig Iterpolatio: Depedig o lumiace gradiets, differet predictors are used for the missig GREEN values i the edge sesig iterpolatio method [8, 4]. First, two gradiets are defied, oe i horizotal directio, the other i vertical directio, for each RED or BLUE oly pixel locatio. For istace, cosider the pixel "b8" as show i Figure. We defie two gradiets as H = g7 g9 ad V = g3 g3, where x deotes absolute value of x. Based o these gradiet values ad a certai threshold (T ), the iterpolatio algorithm the ca be described as follows if H < T ad V > T the G8 = (g7g9)/; else if H > T ad V < T the G8 = (g3g3)/; else G8 = (g3g7g9g3)/4; edif edif g r g3 r4 g5 b6 g7 b8 g9 b0 g r g3 r4 g5 b6 g7 b8 g9 b0 g r g3 r4 g5 Figure A slightly differet edge sesig iterpolatio algorithm is described i [5]. Istead of lumiace gradiets, chromiace gradiets are used. The two gradiets, refer to Figure 3 below, are defied as: b3 b7 H = b5, b b9 V = b5. 7

8 Joural of Electroic Imagig Vol. (3), July 00. b g b3 g4 b5 g6 b7 g8 b9 Figure 3 Liear Iterpolatio with Laplacia secod-order Correctio terms: This algorithm [6] is desiged for optimizig performace i terms of the visual quality of the iterpolated image whe applied o images with sharp edges. We illustrate this algorithm usig a example. Missig color compoets are estimated by followig steps. The first step i this algorithm is to estimate the missig GREEN color compoets at the pixel locatios cotaiig RED or BLUE color compoet oly. Let us cosider estimatig the gree value at a blue pixel locatio (usig Figure 3) as a example. Iterpolatio at a red pixel locatio ca be doe i the similar fashio. Let's fid out the missig GREEN compoet (G5) at pixel locatio b5. We defie horizotal ad vertical gradiets i this pixel locatio as follows: H = g4 g6 ( b5 b3) ( b7 b5), V = g g8 ( b5 b) ( b9 b5). ad V Ituitively, we ca cosider H above as combiatio of the lumiace gradiet ad the chromiace gradiet as described i edge sesig iterpolatio algorithm i the previous sectio. I the expressio of H above, as a example, the first term g4 g6 is the first-order differece of the eighborig gree values, cosidered to be the lumiace gradiet ad the secod term ( b5 b3) ( b7 b5) is the secod-order derivative of the eighborig blue values, cosidered as the chromiace gradiet. Usig these two gradiet values, the missig gree compoet G5 at pixel locatio b5 is estimated as follows. if H < V the G5 = (g4 g6)/ (-b3 *b5 - b7)/4; else if H > V the G5 = (gg8)/ (-b *b5 - b9)/4; else G5 = (gg4g6g8)/4 (-b - b3 4*b5 - b7 - b9)/8; edif edif The iterpolatio step for G5 has two parts. The first part is the liear average of the eighborig gree values, ad the secod part ca be cosidered as a secod-order correctio term based o the eighborig blue (red) values. 8

9 Joural of Electroic Imagig Vol. (3), July 00. The missig red (or blue) color compoets are estimated i every pixel locatio after estimatio of the missig gree compoets i every pixel locatio. Depedig o the positio, refer to Figure 4 below, we have three cases: r g r3 g4 b5 g6 r7 g8 r9 Figure 4. Estimate red (blue) color compoet at a gree pixel where earest eighbors of red (blue) pixels are i the same colum, e.g. pixel locatio g4 as show i Figure 4 above. We estimate the red compoet R4 at pixel locatio g4 as follows. R4 = (r r7) / (g4 - G g4 - G7) / 4. Estimate red (blue) color compoet at a gree pixel where earest eighbors of red (blue) pixels are i the same row, e.g. pixel locatio g as show i Figure 4. We estimate the red compoet R at pixel locatio g as follows. R = (r r3) / (g - G g - G3) / 4 3. Estimate red (blue) color compoet at a blue (red) pixel. For istace, estimate red compoet R5 at pixel locatio b5 as show i Figure 4. Here we first defie two diagoal gradiets as follows: N = r r9 G5 g G5 g9, P = r3 r7 G5 g3 G5 g7, Usig these diagoal gradiets, the algorithm for estimatig the missig color compoets is described as: if N < P the R5 = (rr9)/ (-G *G5 G9)/; else if H > V the R5 = (r3r7)/ (-G3 *G5 G7)/; else R5 = (rr3r7r9)/4 (-G - G3 4*G5 - G7 - G9)/4; edif edif. This method provides much better visual quality of the recostructed image cotaiig a lot of sharp edges. However, the secod-order derivative for calculatig the gradiets makes the algorithm quite sesitive to oise. Sice oly the color iformatio i the same directio (vertical, horizotal, or oe of the diagoal directios based o the gradiet iformatio) is used for iterpolatio, we believe that it is still possible to further improve the visual quality of the recostructed image. 9

10 Joural of Electroic Imagig Vol. (3), July Proposed Fuzzy Assigmet Based Adaptive Method 3. Fuzzy membership assigmet strategy I our proposed approach, we have assiged a fuzzy membership [7, 8, 9] to all the surroudig four coected eighborig pixels ( G, G, G3, G4 as show i Figure 5, as a example) i order to estimate the missig color compoet (say gree) i a pixel locatio (R i Figure 5). The membership assigmet strategy differs depedig upo characteristic of a possible edge at the pixel locatio. For example, let us cosider estimatio of the missig gree color compoet at a pixel locatio, say cotaiig R as show i Figure 5. B G 3 B G R G B G 4 B Figure 5 Depedig upo the correlatio amogst the surroudig pixels, we have formulated a strategy to assig membership grades to the surroudig horizotal ad vertical pixels. The membership grades have bee experimetally derived through exhaustive subjective visual ispectio, takig ito cosideratio the exhaustive set of images havig possible edges i the horizotal ad vertical directios. We have cosidered the followig four cases, where there is a possible edge alog the horizotal directio. I a likewise maer we have also cosidered the cases where there are possible edges i the vertical directio. Case : G G is small while 3 G4 G is arbitrarily large, subject to the coditio that G 3 G4 G G >> 0. Here we have assumed the existece of a horizotal edge while the horizotal eighborig pixels G ad G have approximately the same itesity. Case : G G is small ad 3 G4 G is arbitrary ad G G G4. I this case also there is clearly a possible edge at the pixel locatio R, ad the itesity of this edge depeds upo the surroudig pixel values G 3 ad G 4. Case 3: This case is similar to case with a differece that here the pixels G, G ad G 3 to be approximately of similar pixel itesity, i.e. G G is small ad G3 G4 is arbitrary ad G G. G3 0

11 Joural of Electroic Imagig Vol. (3), July 00. Case 4: I this case we have cosidered that all the four coectig eighborig pixels G, G, G 3 ad G 4 that are all differet subject to the coditio that G 3 G4 G G >> 0. I each of the above cases we have computed the horizotal ad vertical membership fuctios ad a similar logic has bee applied to the images havig a edge i the vertical directio. While computig the membership assigmets, we have estimated the likelihood of the occurreces of each of the four cases as described above usig a large umber of images. We have also observed that the assumptio of equal a- priori probabilities of occurrece of differet types of edges, viz., horizotal ad vertical edges is ot quite true. Thus we have also foud out the likelihood of occurrece of each of the types of edges from our database images. Fially from the average of these membership grades, the fuzzy membership value of 0.5 has bee assiged to the horizotal gree pixel values G ad G. Similarly the two vertical gree pixel value G 3 ad G 4 have bee assiged membership grade of 0.. We have observed that the above assigmet of membership values have resulted i very good color demosaicig, yieldig visually pleasig color recostructio. The membership assigmet has bee decided after performig exhaustive experimets o a large umber of images. O the basis of above discussio, the missig gree value at pixel locatio R ca be iterpolated as follows: Missig 0.5* G 0.5* G 0.* G3 0.* G4 G = = ( G G ) ( G * * G4 ) Usig this fuzzy membership assigmet as a weighted-average tool for missig color iterpolatio, we ca fully utilize all the eighborig iformatio for estimatig the missig color iformatio. 3. Proposed three-steps iterpolatio algorithm The proposed iterpolatio algorithm is a three-step algorithm as summarized below. Step : Estimatio of all missig Gree values. After completio of this step, each ad every pixel locatio has a value for the gree color compoet. Step : Estimatio of missig Blue (Red) color compoet at each pixel locatio cotaiig Red (Blue) color compoet oly. The gree values estimated i the previous step are used i this step. The decisio is based o the chage of hue values. Step 3: Estimatio of missig Red ad Blue at gree pixels. The estimated Red/Blue at blue/red pixels i the previous step have bee utilized for iterpolatio of the missig Red ad Blue at gree pixels. The details of above steps i the proposed color iterpolatio algorithm have bee described below. It should be oted that by lower case r, g or b, we represet the red (r), gree (g) or blue

12 Joural of Electroic Imagig Vol. (3), July 00. (b) values preset i each pixel i the sub-sampled Bayer patter color image. Ad by upper case letters R, G ad B, we deote the estimated values of Red (R), Gree (G) ad Blue (B) i each pixel locatio. Step : Estimatio of all missig Gree values From the Bayer patter, the arragemet of the eighborig pixels i a 5x5 widow with a red pixel at the ceter of the widow is show below. We estimate the missig gree color compoet at this pixel locatio as follows. r m-,- g m-,- r m-, g m-, r m-, g m-,- b m-,- g m-, b m-, g m-, r - g - r g r g m,- b m,- g m, b m, g m, r m,- g m,- r m, g m, r m, First, we estimate two parameters i terms of chages i the Hue values, oe i horizotal directio ad the other i vertical directio, usig the followig two equatios. C hor ( R G ) ( R G ) ( r r ) ( r r ) g ( r g g r ) = ( Rm, Gm, ) ( Rm, Gm ) ( r r ) ( r r ) C ver, m, = m g m, m, ( g g r ) r, m, m, m, g g m, As metioed earlier i the smooth hue trasitio iterpolatio, if the pixel value is already trasformed from liear exposure space ito logarithmic exposure space, oe ca defie the RED hue value as R G. Whe R is ot available, we used a simple average of eighborig R values to approximate it. As show i the above two equatios, the two chages of hue values, C hor ad C ver, ca be easily obtaied usig the filterig operatio at the pixel locatio ( ) i the iput Bayer patter image with a five taps filter, (-0.5,, 0, -, 0.5). The, depedig upo the values of these two parameters, differet fuzzy memberships umbers are used as weightig factors to estimate the missig Gree value (as described i the followig "if the- else" statemet).

13 Joural of Electroic Imagig Vol. (3), July 00. if ( else edif C < C ) the hor ver * I hor G = * I ; if ( Cver < Chor ) G = * I * I ; else edif hor hor ver G = 0.5* I 0.5* I ; Where the two variables I hor ad I ver are estimated as show i equatios () ad () below based o the assumptio of smooth hue trasitio as discussed earlier with a weightig factor (0.5) i the secod term to reduce the sesitivity of oise i the image. ( gm, gm, ) 0.5 ( rm, rm, rm, ) 4 ( g g ) 0.5 ( r r r ) 4 I hor ver = = () I () The two scalig factors, ad 0.667, i above expressios have bee derived experimetally based o the fuzzy membership assigmet strategy. A similar strategy is applied for estimatio of the missig Gree value at the blue pixels i Bayer patter image. Step : Estimatio of the missig Blue/Red value at red/blue pixel From the Bayer patter, the arragemet of the eighborig pixels i a 3x3 widow with a red pixel at the ceter of the widow is show below. ver ver b m-,- g m-, b m-, g - r g b m,- g m, b m, The hue values of the four corer pixels i the widow ad the differece of hues alog the diagoals are estimated usig followig equatios. hue b G ; w = m, m, sw = bm, Gm, e = bm, Gm, se = bm, Gm, hue ; hue ; hue ; hue md = hue hue ; w se 3

14 Joural of Electroic Imagig Vol. (3), July 00. hue sd = hue hue ; e sw Where each G mi,j at r has bee estimated i step. Here hue_md ad hue_sd idicate the differece of hues alog the diagoals. The procedure for estimatio of the missig Blue value i the red pixel is show below. if ( else hue < hue ) the md B B sd ( b G ) ( b G ).8333*( ) = G 0 m, m, m, m, 0 ( bm, Gm, ) ( b Gm, ) m,.667 * ( ).667 *(( b G ) ( b G )) = G 0 m, m, m, m, 0 ( bm, Gm, ) ( b Gm, ) m,.8333* ( ) ; ; edif Estimatio of the missig Red value at a blue pixel may similarly be obtaied. Step 3: Estimatio of the missig Blue/Red value at gree pixel From the Bayer patter, the arragemet of the eighborig pixels i a 3x3 widow with a gree pixel at the ceter of the widow is show below. g m-,- b m-, g m-, r - g r g m,- b m, g m, Here the four hue values adjacet to the ceter pixel i the above widow ad the differece of hues alog the horizotal ad vertical directios are estimated usig followig equatios. hue = b G, ; m, m e = bm, G w = bm, G s bm, Gm hue ; hue ; hue =, ; hue = hue hue ; hue hor ver e w = hue hue ; s Both the color values G mi,j ad B mi,j at the pixel locatio g i above equatios have bee estimated i the previous steps. Here hue_hor ad hue_ver idicate the differece of hues alog 4

15 Joural of Electroic Imagig Vol. (3), July 00. the horizotal ad vertical directios. The procedure for estimatio of the missig Blue value i the gree pixel is show below. if ( hue < hue ) the hor B ver ( B G ) ( B G ).8333*( ) = g 0 0 ( bm, Gm, ) ( b Gm, ) m,.667 * ( ) ; else B.667 *(( B G ) ( B G )) = g 0 0 ( bm, Gm, ) ( b Gm, ) m,.8333* ( ) ; edif Similarly, the missig Red color compoet is estimated i each pixel locatio cotaiig oly the gree color compoet i the Bayer patter image. 4. Experimetal Results Sice it is difficult to grab the sub-sampled Bayer patter images directly from the digital cameras available i the market, we have sythetically geerated the sub-sampled Bayer patter images from 4-bit full color RGB high quality images by simply droppig two color values i each pixel locatio. We have geerated more tha 5 such test images of differet types ad characteristics of the cotets. However i this paper, we have reported results of performace of the iterpolatio algorithms applied o four Bayer patter images that have bee sythetically geerated from four origial full color test images show i Figure 6. We carefully have chose these challegig images of differet characteristics to demostrate performace of differet color iterpolatio algorithms. As show i Figure 6(a), the STAR image cotais lots of high frequecy patters i the form of black ad white sharp edges i differet agles. As show i Figure 6(b), ZEBRA is a image with black ad white strips i the scee. As show i Figure 6(c), the TOWN image cotais a lot of sharp ad colorful edges, ad the NEWENG image i Figure 6(d) is a atural outdoor scee. We compared performace of the proposed color iterpolatio algorithm with three methods, biliear iterpolatio, block matchig based algorithm by Acharya, et. al. [3] ad combiatio of Smooth Hue Trasitio ad Edge Sesig iterpolatio as described i sectio. The recostructed full color images usig these algorithms are show i Figures 7-. I all of these images, our proposed algorithm produces much better result i terms of subjective quality ad objective measure of the peak sigal-to-oise ratio (PSNR). Table shows the PSNR results for these test images. As we ca see, we ca get up to about 0 db improvemet by fuzzy color iterpolatio algorithm as compared with the simple biliear iterpolatio o the STAR image, ad o the average, our ew method ca achieve more tha 5 db improvemet as compared with the other three methods we tested. This STAR image is very useful for testig color iterpolatio algorithms. We ca easily see the blurrig ad color bleedig artifacts, as show i Figure 7(b), (c) ad (d), produced by the biliear iterpolatio, Block Matchig method, ad combiatio of Smooth Hue Trasitio ad Edge Sesig iterpolatio. The proposed fuzzy color iterpolatio algorithm produces the best result i terms of visual quality, as show i Figure 7(a), i term of those blurrig ad color bleedig 5

16 Joural of Electroic Imagig Vol. (3), July 00. artifacts. Figure 8 shows a zoomed versio of those results i Figure 7. I Figures 9, we show three sets of recostructed ZEBRA, TOWN, NEWENG images usig the same algorithms. Image STAR Table : PSNR compariso of differet algorithms PSNR Color Chael Biliear Block Smooth Hue iterpolatio Matchig Edge Sesig Proposed method R G B R ZEBRA G B R TOWN G B R NEWENG G B

17 Joural of Electroic Imagig Vol. (3), July Coclusio ad Summary Due to the cost ad packagig cosideratio, i digital imagig device such as a digital camera, oly a sigle electroic sesor is used ad the eed for color iterpolatio will remai critical util other techologies such as multi-chael color moiré free sesor [0] is mature. I this paper, we preseted a ew color iterpolatio algorithm for Bayer patter sub-sampled color images. The proposed algorithm utilizes the fuzzy membership assigmet as a weightig factor alog with the cocept of smooth hue trasitio for estimatig the missig colors i each pixel. This algorithm sigificatly improves the overall visual quality of the recostructed color images. The experimetal results show that the algorithm preserves colors o the edges with miimal or o visual artifacts. We have preseted the objective quality metrics i terms of PSNR to show the performace of the algorithm with four challegig images for color iterpolatio. Ackowledgemet The authors solemly ackowledge the cotributio of Late A. K. V. Subba Rao for his dedicated work i the area of Color Iterpolatio ad his collaboratio i early phase of this research. Refereces. Bryce E. Bayer, "Color imagig array," U.S. Patet 3,97,065, Eastma Kodak Compay, Do P. Mitchell et. al, "Recostructio Filters i Computer Graphics," Computer Graphics, (SIGGRAPH'88 Proceedigs), Vol., No.4, P.-8, August Hsieh S. Hou et. al., "Cubic Splies for Image Iterpolatio ad Digital Filterig," IEEE Trasactios o Acoustic, Speech ad Sigal Processig, Vol. ASSP-6, pp , David R. Cok, "Sigal processig method ad apparatus for sampled image sigals," U.S. Patet 4,630,307, Eastma Kodak Compay, Eiichi Shimizu et. al., "The Digital Camera Usig New Compressio ad Iterpolatio Algorith" IS&T 49th Aual Coferece, pp:68-7, Tmoasz A. Matraszek, David R. Cok, ad Robert T. Gray, "Gradiet based method for providig values for ukow pixels i a digital image," U.S. Patet 5,875,040, Eastma Kodak Compay, James E. Adams, Jr., Iteractios betwee color plae iterpolatio ad other image processig fuctios i electroic photography, Proceedigs of the SPIE Electroic Imagig Coferece, Vol. 46, pp:44-5, Tadashi Sakamoto, Chikako Nakaishi ad Tomohiro Hase, "Software Pixel Iterpolatio for Digital Still Cameras Suitable for A 3-bit MCU," IEEE Trasactios o Cosumer Electroics, Vol. 44, No. 4, pp 34-35, Nov

18 Joural of Electroic Imagig Vol. (3), July William T Freema, "Method ad apparatus for recostructig missig color samples," U.S. Patet 4,663,655, Polaroid Corporatio, David R Cok, "Sigal processig method ad apparatus for producig iterpolated chromiace values i a sampled color image sigal," U.S. Patet 4,64,678, Eastma Kodak Compay, X. Wu, W. K. Choi, ad P. Bao, Color Restoratio from Digital Camera Data by Patter Matchig, Proceedigs of the SPIE s Electroic Imagig Coferece, Color Imagig: Device- Idepedet Color, Color Hardcopy, ad Graphic Arts II, Vol. 308, pp. -7, Tiku Acharya ad Pig-Sig Tsai, "A New Block Matchig Based Color Iterpolatio Algorith'' Proceedigs of the SPIE Electroic Imagig Coferece, Color Imagig: Device- Idepedet Color, Color Hardcopy, ad Graphic Arts IV, Vol. 3648, pp , Robert H. Hibbard, "Apparatus ad method for adaptively iterpolatig a full color image utilize lumiace gradiets," U.S. Patet 5,38,976, Eastma Kodak Compay, Glaude A. Laroche ad Mark A. Prescott, Apparatus ad method for adaptively iterpolatig a full color image utilize chromiace gradiets," U.S. Patet 5,373,3, Eastma Kodak Compay, Joh F. Hamilto, Jr. ad James E. Adams, Jr., "Adaptive color pla iterpolatio i sigle sesor color electroic camera," U.S. Patet 5,69,734, Eastma Kodak Compay, Zadeh, L. A. Fuzzy Sets, Iformatio ad Cotrol 8; , Zimmerma, H. J. Fuzzy Set Theory ad Its Applicatios, d ed. Norwell, MA: Kluwer Academic, Kazuo, T: traslated by Niimura, T. A Itroductio to Fuzzy Logic for Practical Applicatios, Spriger-Verlag New York, Ic P. G. Herzog, D. Kipp, H. Stiebig ad F. Koeig, "Characterizatio of ovel three ad six chael color moire free sesors,'' Proceedigs of the SPIE Electroic Imagig Coferece, Color Imagig: Device-Idepedet Color, Color Hardcopy, ad Graphic Arts IV, Vol. 3648, pp 48-59,

19 Joural of Electroic Imagig Vol. (3), July 00. (a) (b) (c) (d) Figure 6: Origial 4 bits full color images: (a) Star (b) Zebra (c) Tow (d) Neweg. 9

20 Joural of Electroic Imagig Vol. (3), July 00. (a) (b) (c) (d) Figure 7: STAR image: (a) Recostructed image usig proposed algorith (b) Recostructed image usig Biliear color iterpolatio, (c) Recostructed image usig Block Matchig based method, ad (d) Recostructed image usig combiatio of Smooth Hue ad Edge Sesig iterpolatio. 0

21 Joural of Electroic Imagig Vol. (3), July 00. (a) (b) (c) (d) Figure 8: STAR image (ceter portio oly): (a) Recostructed image usig proposed algorith (b) Recostructed image usig Biliear color iterpolatio, (c) Recostructed image usig Block Matchig based method, ad (d) Recostructed image usig combiatio of Smooth Hue ad Edge Sesig iterpolatio.

22 Joural of Electroic Imagig Vol. (3), July 00. (a) (b) (c) (d) Figure 9: Zebra image: (a) Recostructed image usig proposed algorith (b) Recostructed image usig Biliear color iterpolatio, (c) Recostructed image usig Block Matchig based method, ad (d) Recostructed image usig combiatio of Smooth Hue ad Edge Sesig iterpolatio.

23 Joural of Electroic Imagig Vol. (3), July 00. (a) (b) (c) (d) Figure 0: Tow image: (a) Recostructed image usig proposed algorith (b) Recostructed image usig Biliear color iterpolatio, (c) Recostructed image usig Block Matchig based method, ad (d) Recostructed image usig combiatio of Smooth Hue ad Edge Sesig iterpolatio. 3

24 Joural of Electroic Imagig Vol. (3), July 00. (a) (b) (c) (d) Figure : Neweg image: (a) Recostructed image usig proposed algorith (b) Recostructed image usig Biliear color iterpolatio, (c) Recostructed image usig Block Matchig based method, ad (d) Recostructed image usig combiatio of Smooth Hue ad Edge Sesig iterpolatio. 4