ADAPTIVE JOINT DEMOSAICING AND SUBPIXEL-BASED DOWN-SAMPLING FOR BAYER IMAGE

ADAPTIVE JOINT DEMOSAICING AND SUBPIXEL-BASED DOWN-SAMPLING FOR BAYER IMAGE Lu Fang, Oscar C. Au Dept. of Electronic and Computer Engineering Hong Kong Univ. of Sci. and Tech. {fanglu, eeau}@ust.hk Aggelos K. Katsaggelos Dept. of EECS Northwestern University {aggk}@eecs.northwestern.edu ABSTRACT A digital camera provided with a Bayer pattern single sensor needs color interpolation to reconstruct a full color image. To show high resolution image on a lower resolution display, it must then be downsampled. These two steps influence each other, i.e., the color artifacts introduced in demosaicing may be magnified in subsequent down-sampling process and vice versa. Thanks to the fact that LCD displays are actually composed of separable subpixels, which can be individually addressed to achieve a higher effective apparent resolution. This paper presents an Adaptive Joint Demosaicing and Subpixel-based Down-sampling scheme (AJDSD) for singlesensor camera image, where the subpixel-based down-sampling is adaptively and directly applied in Bayer domain, without the process of demosaicing. Simulation results demonstrate that when compared with conventional demosaicing-first and downsamplinglater methods, AJDSD achieves superior performance improvement in terms of computational complexity. As for visual quality, AJDSD is more effective in preserving high frequency details, leading to much sharper and clearer results. 1. INTRODUCTION Color image processing has aroused much interest and acclaim over the past few years. Digital cameras for still images are among the most popular acquisition devices. Instead of three CCD/CMOS sensors, a single sensor array based on the Bayer CFA structure [1] constitutes a cost-effective tool to capture the visual scene [2]. Fig. 1(a) illustrates the Bayer CFA structure, where half of the pixels are assigned to G channel due to the fact that green (G) channel is the most important factor to determine the luminance of the color image, and R and B channels share the remaining parts evenly. With Bayer CFA structure, each pixel location of the mosaic image has only one of the three primary colors. Since the full color image is more acceptable than the mosaic image for the human visual system, the two missing color components for each pixel in the mosaic image should be recovered as best as possible and such a recovering process is called demosaicing. It is well known that 1 : 1 zoom level is not typical for digital photographs. Currently, available entry-level digital cameras are capable of multi-megapixel resolution, while display resolutions have remained relatively static in the recent past. Without any resampling, even relatively high resolution 100 1200 displays are not capable of displaying an entire megapixel image. Camera viewfinder or live preview requires much greater levels of down-sampling to fit the full image. Although camera viewfinders and hand-held devices have displays of vastly low resolution, higher apparent resolution is always attractive to consumers because, for a given physical size of (a) Fig. 1. (a) The Bayer arrangement of color filters on the pixel array of an image sensor, (b) Subpixel arrangement on the pixel array of an image sensor. display, higher resolutions can make images more realistic by rendering more details. For example, for two displays with the same physical size of 4 in in, a 704 57 display would convey more details than a 52 288 resolution. This is also why the HDTV looks nicer than SDTV, on displays with the same physical size. Conventional pixel-based down-sampling with an anti-aliasing prefilter causes severe blurring artifacts, as only the low frequency information can be retained in the process [] [4]. Thanks to the fact that each pixel on a color LCD is actually composed of individual red, green and blue subpixel stripes, as shown in Fig. 1(b), the number of individual reconstruction points in a LCD can be increased by three times. Researchers in [5] [] [7] [8] have shown that subpixelbased down-sampling is effective in providing higher apparent resolution than conventional pixel-based down-sampling methods. Traditionally, to produce a down-sampled full color image, the CFA image is first recovered to a full color image using conventional demosaicing methods, then the demosaiced full color image is subsampled, as shown in Fig. 2(a). Unfortunately, such approach requires extra storage of full color images, and the color artifacts caused by demosaicing may be magnified in following subsampling. Intuitively, an alternative approach in Fig. 2(b) is down-sampling CFA image followed by conventional demosaicing process. However, it may cause inefficient utilization of the raw sensor data, and the blurring artifacts caused by down-sampling may be magnified during the later demosaicing process. Since demosaicing and downsampling influence each other, it may be advantageous to perform down-sampling and demosaicing jointly and directly in the Bayer domain when possible. Lukac et al. [2] have shown that joint demosaicing and zooming produces good results and is computationally efficient. In this paper, an adaptive joint demosaicing and subpixel-based down-sampling method is proposed, as shown in Fig. 2(c). By shar- (b) 978-1-1284-50-/11/$2.00 2011 IEEE

(a) (b) (c) Fig. 2. Image down-sampling methods: (a) down-sampling after demosaicing, (b) demosaicing after down-sampling, (c) proposed joint demosaicing and subpixel-based down-sampling. (a) Fig.. (a) The horizontal and vertical indexes of the (i, j)th block in original CFA image; (b) The (m, n) th 2 2 sub-block in the (i, j) th block of mosaic image, m = i 2, i 1, i and n = j 2, j 1, j. (b) ing the color information extracted directly from the raw sensor data, a subpixel-based down-sampling scheme is specifically designed. Simulation results show that the proposed method is superior to conventional general combinations of different demosaicing and downsampling algorithms, in producing down-sampled full-color images in terms of computational complexity and visual quality. The remainder of this paper is organized as follows. Section 2 introduces the details of AJDSD. In Section, the proposed algorithm is tested on a number of 24-bit full color images. Evaluations of performance as well as complexity analysis are provided. Finally, conclusions are drawn in Section 4. 2. PROPOSED ADAPTIVE JOINT DEMOSAICING AND SUBPIXEL-BASED DOWN-SAMPLING (AJDSD) For a mosaic image, each color plane is sampled at a rate less than once per pixel, i.e., 1 Bayer samples/subpixel in Fig. 1. Thus at 1 : 1 scale, there is not enough input information available for subpixel rendering to take advantage of the increased resolution. Downsampling by a factor of three effectively increases this to the requisite 1 Bayer sample/subpixel for subpixel rendering. A factor of six (in each dimension) is used here since it simplifies the implementation - adjacent blocks of Bayer data are identical, while blocks contain alternately more red or blue samples. The approach could be used for any higher down-sampling factor as well. For the simple of presentation, let us suppose the original Large CFA image (L CFA) is of size M N, and the down-sampled Small full color image (S) is of size M N. We divide L CFA into blocks so that there are M N blocks, one for each pixel in the down-sampled low resolution full color images S. Note that in this paper, we use (i, j) (i = 1, 2,, M, and j = 1, 2,, N) to index the pixel location is S. In other words, any (i, j) th pixel of S corresponds to (i, j) th block of L CFA, as shown in Fig. (a). We further divide one block into nine sub-blocks, which is of size 2 2 and indexed by (m, n). (For down-sampling ratio other than six, the size of sub-block can be chosen accordingly.) It is assumed that image is stationary within a small region, i.e., a 2 2 sub-block in Fig. (b), and so adjacent Red, Green and Blue color components can be combined into color vectors [2], which we denote as pseudo-pixel. Thus a 2 2 sub-block in Fig. (b) will have two pseudo-pixels. Since we do not know which pseudo-pixel (if any) corresponds to the actual local color information of the subblock, we can form an unbiased estimate by the average of surrounding pseudo-pixel values. However, merely taking an arithmetic av- erage of the pseudo-pixels is equivalent to bilinear interpolation and may lead to smearing. Instead, to preserve edges, we use the maximum likelihood estimate approach under the constraint that one of the input vectors is chosen to be the output, minimize K k=1 ˆP P k 2 ˆP subject to ˆP {Pk k = 1, 2,, K} where P k = [p k1, p k2, p k ] T represents a three-component (R, G and B) pseudo-pixel. ˆP is the estimated pseudo-pixel that may best represent the local information of current sub-block. For the (m, m) th sub-block in Fig. (b), there are two pseudo-pixels: P 1(m, n) = [R(2m 1, 2n), G(2m 1, 2n 1), B(2m, 2n 1)] T and P 2(m, n) = [R(2m 1, 2n), G(2m, 2n), B(2m, 2n 1)] T. Given only two vectors, the solution of (1) is not unique. As is commonly down with scalar medians, in those cases we use the mean of the two vectors, given by ˆP(m, n) = ˆR(m, n) Ĝ(m, n) ˆB(m, n) = R(2m 1, 2n) G(2m 1,2n 1)+G(2m,2n) 2 B(2m, 2n 1) (1) (2) Recall that one pixel in S corresponds to one block (nine sub-blocks) in L CFA, and according to the pattern of subpixel-based decimation we will take three subpixels of a pixel from different subblocks, each from one sub-block. Most of existing subpixel-based down-sampling methods perform horizontal subsampling [9], due to the fact that the red, green, blue subpixels of a typical LCD display are arranged in a horizontal manner. However, as Lu noted in [8], if the edge direction is parallel to the sampling direction, there is little apparent resolution improvement. To achieve improved resolution in all four directions (Horizontal, Vertical, Diagonal, and Anti- Diagonal), we will choose the sampling direction adaptively according to the local edge information. Traditionally, we can use indirect approach to extract gradient information on mosaic image, i.e., apply existing demosaicing methods and convert the demosaiced full color image to the luminance map, then a Sobel-based edge detector is applied on the luminance map [10] []. However, such approach may lead to inaccurate estimation of gradient and high computational complexity. To extract more accurate edge information, we will directly embed Sobel operator into Bayer image. Recall that the estimated pseudo-pixel of the (m, n) th sub-block is given in (2), and hence the estimated luminance value may be approximated by (). Given the estimated luminance map, the gradient response of the (i, j) th block in hori-

zontal direction is given in (4). Ŷ (m, n) = ˆR(m, n) + 1 ˆB(m, n) + Ĝ(m, n) 10 10 10 = 1 R(2m 1, 2n) + B(2m, 2n 1) () 10 10 + (G(2m 1, 2n 1) + G(2m, 2n)) 10 YCF H A(i, j) = (Ŷ (i 2, j) + 2Ŷ (i 1, j) + Ŷ (i, j)) (Ŷ (i 2, j 2) + 2Ŷ (i 1, j 2) + Ŷ (i, j 2)) (4) Fig. 5. Pattern of Adaptive Joint Demosaicing and Subpixel-based Down-sampling scheme. (a) (b) (c) (d) Fig. 4. SL-based mask in four directions: (a) horizontal mask, (b) vertical mask, (c) diagonal mask, (d) anti-diagonal mask. Combining () and (4), we have YCF H A(i, j) in (5), which is directly applied on Bayer image. Similarly, we can obtain YCF V A(i, j), YCF D A(i, j), and YCF AD A(i, j). For avoiding the floating point computation, the coefficients in the masks are normalized into integers in advance and the four normalized Sobel- Luminance-based masks are shown in Fig. 4. These four masks are directly applied on CFA image, which can effectively determine the local edge direction for any (i, j) th block, denoted as Dir(i, j). We further denote the direction of subpixel-based decimation in (i, j) th block as Dir (i, j), which is adaptively chosen to be the perpendicular direction of Dir(i, j), H, if Dir(i, j) = V V, if Dir(i, j) = H Dir (i, j) = () D, if Dir(i, j) = AD AD, if Dir(i, j) = D One of the four cases in () is demonstrated in Fig. 5, where the local edge direction (Dir(i, j)) is anti-diagonal direction, and the direction for subpixel-based decimation (Dir (i, j)) is diagonal direction. In other words, we will copy the red, green, blue components of the (i, j) th pixel in S from three different sub-blocks of (i, j) th block in L CFA along diagonal direction. r i,j = ˆR i 2,j 2, g i,j = Ĝi 1,j 1, bi,j = ˆB i,j (7) Fig.. Diagram of adaptive joint demosaicing and subpixel-based down-sampling scheme.. EXPERIMENTAL RESULTS In this section, simulation is carried out to evaluate the performance of various methods. Fig. 7 shows 12 true color (24-bit) testing images from Kodak PhotoCD. In our experiments, the original testing images (M N ) are first down-sampled to a CFA image (M N) using the Bayer pattern. The result full color images of size M N are then produced by demosaicing-first and downsampling-later methods as well as AJDSD. The demosaicing algorithms used in demosaicing-first and downsampling-later methods are Bilinear Interpolation by T. Sakamoto et al. [11], Alternating Projection by Gunturk et al. [12], and Edge-based demosaicing by King-Hong Chung and Yuk-Hee Chan [1]. The downsampling algorithm used in demosaicing-first and downsamplinglater methods is bicubic interpolation []. We denote these three demosaicing-first and downsampling-later methods as BI, AP and EDGE, respectively. Unfortunately, direct application of subpixel-based decimation may cause local color unbalance, introducing severe color fringing artifacts. Several algorithms have been proposed to suppress color distortion. However, to the best of our knowledge, most of existing methods work merely in horizontal direction [5] [] [7]. In [8], an optimal 2-D minimum mean square error subpixel-based downsampling scheme is proposed and the corresponding solution can be implemented by a 2-D linear filter followed by subpixel-based downsampling. Such 2-D linear prefilter is effective in suppressing color fringing artifacts and can be well implemented in our adaptive directional subpixel-based decimation. Therefore, as shown in Fig., the proposed AJDSD is implemented by firstly extracting pseudopixels from the corresponding sub-blocks based on the maximum likelihood estimation, then the extracted pseudo-pixels are filtered by the 2-D optimal linear filter, followed by adaptive subpixel-based down-sampling processing. Fig. 7. True color (24-bit) Kodak testing images..1. Speed Improvement To measure the computational complexity of AJDSD and three demosaicing-first and downsampling-later methods (BI, AP and EDGE), in Table 1 we depict the CPU running time when processing 712 512 full color images. The concerned algorithms are implemented on the Mac OS X with Intel Core 2 Duo CPU 2.1 GHz and 1GB RAM. The operating system used is Mac OS X 10.5.8 and the program developing environment is Matlab R2010a. It should

[ G(i 5, j 1) + R(i 5, j) + 1 B(i 4, j 1) + G(i 4, j)] YCF H A(i, j) = + [ G(i, j 1) + 2 R(i, j) + B(i 2, j 1) + G(i 2, j)] + [ G(i 1, j 1) + 1 [ R(i 1, j) + B(i, j 1) + G(i, j)] G(i 5, j 5) + R(i 5, j 4) + 1 B(i 4, j 5) + G(i 4, j 4)] + [ G(i, j 5) + R(i, j 4) + B(i 2, j 5) + G(i 2, j 4)] + [ G(i 1, j 5) + 1 R(i 1, j 4) + B(i, j 5) + G(i, j 4)] (5) be stressed that this is only a rough complexity comparison since the running time also depends on the optimization of the program codes. It is obvious that AJDSD performs favorably compared to BI, AP and EDGE. Generally speaking, the CPU time consuming of AJDSD is about 0% of the EDGE, and is merely 10% of the AP. This is mainly because BI, AP and EDGE are demosaicing-first and downsampling-later methods, and the demosaicing methods used in AP and EDGE are of high computational complexity, while AJDSD does not need to demosaic each and every pixel. Table 1. CPU time (seconds) of BI/EDGE/AP and AJDSD. Image BI EDGE AP AJDSD 1 0.45 0.5.5 0.7 2 0. 0.54.8 0.9 0.40 0.54.75 0. 4 0.9 0.5.71 0.7 5 0.2 0.5.72 0. 0.2 0.52.7 0. 7 0. 0.52.74 0.8 8 0.1 0.54.7 0. 9 0.47 0.54.70 0.7 10 0.8 0.5.70 0.7 11 0. 0.54.8 0.7 12 0.2 0.59.78 0.5 ave 0.7 0.54.7 0.7.2. Visual Quality Since subpixel-based down-sampling methods barely provide improvement in smooth regions with low frequency details, and the major differences happen in their high frequency details. We use a sharpness measure which computes the average of Directional Highfrequency Energy (DHE) in (8). For any image X to be measured, we compute the average of directional high-frequency energy by convolving a 1-D high pass filter in four directions. Basically, the four filters are simply the same 1-D high pass filter H k = [1 1] applied in the horizontal (k=1), vertical (k=2), diagonal (k=) and anti-diagonal (k=4) directions, though other high pass filters can be used also. Note that the reference images are computed by applying Pixel-based Down-sampling with Anti-aliasing Filter (PDAF) to the original true color (non Bayer-filtered) images. By definition, the values of DHE are positive numbers. A higher value indicates that there is more luminance high frequency energy which suggests higher apparent luminance resolution. DHE(X) = 1 4k=1 H k X 4 1 4k=1 (8) H k PDAF 1 1 4 Table 2 depicts that the DHE values of BI, AP and EDGE are smaller than 1, indicating that BI, AP and EDGE retain less details than our reference PDAF. This is mainly because the deci- Table 2. Sharpness Measure: Directional High-frequency Energy Image BI EDGE AP AJDSD 1 0.95 0.991 0.985 1.2 2 0.979 0.99 0.997 1.4 0.981 0.998 0.995 1.254 4 0.975 1.002 0.995 1.2 5 0.981 1.00 1.001 1.2 0.971 0.995 0.988 1.0 7 0.979 1.001 0.99 1.270 8 0.97 0.998 0.995 1.290 9 0.97 0.997 0.992 1.282 10 0.981 1.002 1.000 1.222 11 0.974 0.99 0.99 1.28 12 0.977 0.997 0.99 1.279 ave 0.97 0.998 0.994 1.289 mation in PDAF is applied on the original true color (non Bayerfiltered) images, while in BI, AP and EDGE the decimation is applied on demosaiced images. We also observed that the DHE values of AJDSD is larger than 1, which suggests that AJDSD provides clearer and sharper down-sampled images than PDAF, due to the fact that subpixel-based decimation performs superior in retaining high frequency details. In other words, generally combination of demosaicing and down-sampling algorithms causes severe loss of details, resulting in burring artifacts, as verified in Fig. 8. This is despite the fact that the subpixel-based algorithm implicitly uses a linear interpolation step, which generally performs worse than adaptive demosaicing. As expected, the improvements are especially obvious in edge regions, i.e., the window and roof in Fig. 8(d), the hat and scarf in Fig. 8(h). Similar conclusion can be obtained from Fig. 8(l), where the details of trees are rendered much clearer and sharper than those in Fig. 8(i), 8(j) and 8(k). As Platt [14] notes, showing the results in printed form may have problem. By the nature of subpixel rendering, the result images are suggested to be viewed on an LCD panel to fully appreciate its benefits. Some reviewers may find that there exists somewhat color fringing artifacts in AJDSD. However, it is less objectionable than the loss of detail caused by not taking advantage of subpixel resolution, as demonstrated in Fig. 8. This is due to the superior sharper and clearer luminance details produced by taking advantage of subpixel resolution, and the fact that our human eyes are much more sensitive to the high frequency of luminance than to that of chrominance [5] [15]. 4. CONCLUSION To show single-sensor camera images on a lower resolution display, two steps are needed: demosaicing and down-sampling. Unfortunately, the color artifacts introduced in demosaicing may be magnified in subsequent down-sampling process and vice versa.

(a) BI (b) EDGE This paper presents a joint demosaicing and subpixel-based downsampling scheme for Bayer images, where subpixel-based decimation is applied adaptively and directly in Bayer domain. Simulation results show that the proposed algorithm is superior to conventional demosaicing-first and downsampling-later methods, in terms of highly reducing computational complexity and preserving more high frequency details. 5. ACKNOWLEDGMENT This work has been supported in part by the Research Grants Council (GRF 10109) and the Innovation and Technology Commission (GHP/048/08) of Hong Kong, China. (c) AP (d) AJDSD (e) BI (f) EDGE (g) AP (h) AJDSD (i) BI (j) EDGE (k) AP (l) AJDSD Fig. 8. Down-sampled images for test image House.. REFERENCES [1] B.E. Bayer, Color imaging array, July 20 197, US Patent,971,05. [2] R. Lukac, K.N. Plataniotis, and D. Hatzinakos, Color image zooming on the Bayer pattern, Circuits and Systems for Video Technology, IEEE Transactions on, vol. 15, no. 11, pp. 1475 1492, 2005. [] R.C. Gonzalez and E. Richard, Woods, digital image processing, Beijlng: Publishing House of Electronics Industry, pp. 420 450, 2005. [4] P.S.R. Diniz, Digital signal processing, Cambridge University Press, 2010. [5] S. Daly, Analysis of subtriad addressing algorithms by visual system models, in SID International Symposium Digest of Technical Papers. Society for Information Display, 2001, vol. 2, pp. 1200 1204. [] C. Betrisey, J.F. Blinn, B. Dresevic, B. Hill, G. Hitchcock, B. Keely, D.P. Mitchell, J.C. Platt, T. Whitted, et al., Displaced Filtering for Patterned Displays, in SID International Symposium Digest of Technical Papers. Society for Information Display, 2000, vol. 1, pp. 29 01. [7] M.A. Klompenhouwer, G. De Haan, and R.A. Beuker, Subpixel image scaling for color matrix displays, Journal of the Society for Information Display, vol. 11, no. 1, pp. 99, 200. [8] L. Fang and O.C. Au, Novel 2-D MMSE subpixel-based image down-sampling for matrix displays, in Acoustics Speech and Signal Processing (ICASSP), 2010 IEEE International Conference on. IEEE, 2010, pp. 98 989. [9] S. Gibson, Sub-pixel font rendering technology, from http://www. grc. com/cleartype. htm. [10] K.L. Chung, W.J. Yang, W.M. Yan, and C.C. Wang, Demosaicing of color filter array captured images using gradient edge detection masks and adaptive heterogeneity-projection, Image Processing, IEEE Transactions on, vol. 17, no. 12, pp. 25 27, 2008. [11] T. Sakamoto, C. Nakanishi, and T. Hase, Software pixel interpolation for digital still cameras suitable for a 2-bit MCU, Consumer Electronics, IEEE Transactions on, vol. 44, no. 4, pp. 142 152, 2002. [12] B.K. Gunturk, Y. Altunbasak, and R.M. Mersereau, Color plane interpolation using alternating projections, Image Processing, IEEE Transactions on, vol. 11, no. 9, pp. 997 101, 2002.

[1] K.H. Chung and Y.H. Chan, Color demosaicing using variance of color differences, Image Processing, IEEE Transactions on, vol. 15, no. 10, pp. 2944 2955, 200. [14] J. C. Platt, Optimal filtering for patterned displays, Signal Processing Letters, IEEE, vol. 7, no. 7, pp. 179 181, 2000. [15] B.A. Wandell, Foundations of vision., Sinauer Associates, 1995.