Analysis on Color Filter Array Image Compression Methods

Analysis on Color Filter Array Image Compression Methods Sung Hee Park Electrical Engineering Stanford University Email: shpark7@stanford.edu Albert No Electrical Engineering Stanford University Email: albertno@stanford.edu Abstract In most imaging systems, image compression is carried out after demosaicing process and compression algorithms usually have focused on compressing threechannel color images. Recently, several image compression methods are proposed to directly handle CFA images that are captured before color interpolation. These CFA compression schemes exploit the fact that CFA images do not contain the redundant information introduced by demosaicing. In this paper, we will compare the performance of various CFA compression methods and describe how visual qualities are affected by the choice of demosaic algorithm. Index Terms CFA image compression, lossy compression, lossless compression I. INTRODUCTION The digital color imaging systems generally use a single image sensor to capture full color images using a color filter array (CFA). A color filter array enables each pixel to capture the intensity of light with different color spectrum. The most common design of CFA is the 2 2 Bayer pattern [1] which consists of two greens, one red, and one blue component. Since each pixel has only one color component, demosaic operation is performed to interpolate all three color components to estimate full color RGB images. Then, the demosaiced image can be sent to image display devices for display. Figure 1(a) shows the overall diagram of most widely used digital imaging system. When images are transmitted or stored from the image capture module, the connection between two systems is almost always bandwidth-limited and becomes the bottleneck of whole system performance. Thus, efficient image compression methods are employed to send less amount of data to the link and they are commonly applied after demosaicing operation [2], [3]. Recently, an alternative imaging system is proposed as shown in Figure 1 [4] [7]. The main difference of this system is that it directly compresses CFA image and demosaic operation is done after decompression. By compressing the CFA data, the compressor can deal with only one-third of data compared to the case for compressing three-channel RGB images. Thus, it does not have to fight with the redundancy information that (a) Fig. 1. Illustration of digital color imaging systems. (a) Conventional RGB image compression. CFA image compression. are introduced by color interpolation, which enables more efficient compression. In addition, CFA compression scheme offloads computational burden to the decoder side that usually has much more computation power and memory. Not only reducing the load on the compressor, it also passes whole demosaic stage to the other side. Many image capturing systems have very strong constraints in power and processing time in order to work as real-time mobile devices. Thus, the proposed pipeline has great advantage in minimiz-

(a) Fig. 2. Illustration of the rearranging structure scheme. ing the work load on power-limited image capturing module. From the motivations mentioned above, various approaches are suggested to compress CFA data. We would like to go through various approaches in CFA data compression and analyze their performances in the following sections. II. EXISTING CFA COMPRESSION METHODS There are several different color filter arrays, but the Bayer CFA [1] is most popular. In this CFA, the green filters are placed in a quincunx pattern (interlaced), and red and blue filters are in the remaining locations. The compression methods assume the implementation precision is 8-bit and it can be expanded to other precisions. A. Naive Method The naive method is simply applying the existing compression algorithm, such as JPEG, to the original CFA images. Since the CFA image does not look like natural images and contains very specific Bayer structure, the JPEG compressor is not expected to work very well. B. Rearranging Structure In order to handle the problem of the naive method, several solutions have been proposed [4]. One main approach is rearranging structure methods. Figure 2 shows basic pipeline of this method. Each color component of CFA image is separated into different channels, and reshaped into the rectangular form so that regular compression algorithm can deal with. However, handling green patterns is not trivial because of its quincunx pattern and numerous ways are proposed to reshape it into rectangular form. Figure 3 shows several rearranging structure methods. Figure 3(a) shows the simple merging method which only merges every neighboring odd and even columns. Figure 3 shows the structure separation method which separates quincunx array into two arrays. Figure 3(c) describes the structure conversion method which converts two columns into a single column by applying weighted-averaging. These methods can be also applied in different color spaces, such as both RGB and YCbCr. Even though (c) Fig. 3. The rearranging structure scheme: (a) Simple merging, Structure separation, (c) Structure conversion. Fig. 4. Illustration of color conversion for the Bayer pattern. the CFA image contains only one color component per pixel, we can still apply color conversion by considering a 2 2 block at the same time as described in Figure 4 [5]. The color conversion matrix from RGB to YCbCr can be represented as the following. [ 128.6 0 25 65.5 ] = Y ul Y lr Cb Cr 0 128.6 25 65.5 37.1 37.1 112 37.8 46.9 46.9 18.2 112 C. Filtering and Conversion Y ul Y lr Cb Cr +[ 0 0 128 128 Another approach is to apply filtering followed by downsampling [7]. The CFA image includes more high frequency components than the full color image data, which causes compression artifacts. By applying the low pass filter, we can reduce these high frequency components. Figure 5 shows several low pass filters which can be applied to CFA images. These methods are different from the rearranging case because they are irreversible algorithms. Thus, we cannot apply them into lossless case. III. IDEAL ENTROPY CODING ESTIMATION FOR LOSSLESS CASES Now we are going to find theoretical bounds of rate in lossless case by considering the ideal entropy ]

(a) (a) Fig. 6. The prediction methods : (a) Simple prediction, Adaptive prediction. Fig. 5. The low-pass filters used in the filtering and conversion methods. coding. These bounds will give idea about how much redundancy is removed by each compression scheme. Each method can be easily implemented by adopting efficient encoders that closely achieve theoretical bounds. A. Single Pixel Entropy Coding First, we can find the entropy of each pixel. This case considers neither spatial nor color correlation between pixels. Thus, this entropy coding is expected to contain much redundant information. B. Joint 2 2 Bayer Block Entropy Coding In order to consider the color correlation between pixels, we can consider evaluating joint entropy of 2 2 Bayer block as following. However, as the order of joint entropy increases, the memory requirement explodes exponentially for 8-bit precision. Thus, we used the following to get approximate values for 4th order. H(Y 1, Y 2, Cb, Cr) = H(Y 1, Cb, Cr) + H(Y 2 Y 1, Cb, Cr) H(Y 1, Cb, Cr) + H(Y 2 Y 1 ) Actually, it is reasonable to assume that Y2 and Cb, Cr components are independent given Y1 because the luminance component will be mainly dependent on the neighboring luminance value rather than chrominance value. C. Simple Prediction The simple prediction method is trying to use spatial correlation to reduce the rate. We applied the one used in lossless JPEG case as shown in Figure 6(a). The prediction is done on each color channel separately, Fig. 7. Experimental results of lossy CFA image compression. with two greens separated as the structure separation method described in section II-B. D. Adaptive Prediction To do better prediction, we tried a context matching based prediction [5]. This method predicts the value by evaluating the weighted sum of four candidate pixels that are shown in Figure 6. The weights are determined by how the support regions around the pixel are similar to the one around the pixel that you are predicting. The candidate pixel with the similar support will get higher weight. IV. EXPERIMENTAL RESULTS FOR LOSSLESS CFA COMPRESSION METHODS In this section, we will apply various lossless CFA image compression methods to 24 color images in the Kodak image set [8]. In the lossless case, reconstructed images will be exactly the same as the input images and only the rate, which can be measured in bits per pixel, will matter. Figure 7 shows the rate for several lossless compression methods. The red bars represent the cases of applying JPEG and PNG directly to the CFA image. We can see that these cases do not compress well

Three color channels are equally weighted in distortion calculation. Fig. 8. The pipeline for evaluating error component. A. Comparison between CFA Images Figure 9 shows the rate-psnr curves evaluated by comparing CFA images before and after the compression. As we expected, the naive method, which applies JPEG directly to the CFA data, shows the worst tradeoff line. All other CFA methods usually work better than the naive method in most region. The filtering and conversion methods tend to give lower PSNR in high rate since the high-frequency details lost from low-pass filtering can not be recovered. Fig. 9. Experimental results of lossy CFA image compression. The error is estimated between input and reconstructed CFA images. for CFA images. The blue bars represent the rearranging methods, such as simple merging, structure conversion and structure separation and they clearly work better than the naive methods. Especially, the structure conversion method performs the best among the rearranging methods. The green bars represent ideal entropy estimations. The single pixel entropy coding is the reference case and joint 2 2 Bayer entropy coding performs better, since they consider color correlation. Also, prediction methods show improvements by exploiting spatial correlation, and it is clear that adaptive prediction works better than simple prediction. V. EXPERIMENTAL RESULTS FOR LOSSY CFA COMPRESSION METHODS In this section, we will discuss about lossy CFA image compression cases. Six different CFA methods are implemented based on lossy JPEG compressor. By changing the Quality parameter in JPEG, the rate-distortion curves are obtained and the results are averaged through all 24 images in the Kodak set. B. Comparison between Demosaiced Images Since it is hard to evaluate the visual qualities between CFA images, we only compared with a quantitative measure in section V-A. In practice, it is the final RGB image that the users will see and evaluate the visual qualities. Thus, we apply demosaic algorithm to both input CFA image and the reconstructed CFA image to get RGB images as described in Figure 8. We choose three demosaic algorithms to test: simple bilinear interpolation and two state-of-the-art demosaic methods, adaptive homogeneity method [9] and adaptive frequency domain method [10]. Before comparing the performance of each method, let s see what kinds of visual artifacts you get after CFA compression and demosaicing. When the naive method is used, we get very severe color artifacts because the JPEG compression is messing up inherent CFA structure as shown in Figure 11(a). The error in a pixel spreads to neighboring pixels, which causing undesired big color blobs. However, this artifact is not present in other CFA compression methods since they handle each channel separately. But, as in Figure 11, they still show blocking artifact at low rate because the CFA compression methods are designed based on JPEG compressor. The spatial CIELAB [11] is a measure that can be used to evaluate perceptual difference between two images. Figure 10 shows that the naive method gives significant loss in visual quality while other methods perform relatively similar. Figure 10 shows the rate-distortion curves of each CFA compression methods for three different demosaic methods. In addition, the conventional method, which applies demosaic first and then do RGB JPEG compression, is also drawn as the bright red line with circles. It is interesting to see that in Figure 10(a), the CFA methods work better than the conventional method for bilinear demosaicing while it is not true for the other sophisticated demosaic algorithms. It results from the fact that the conventional method does not work well for bilinear case as well as CFA methods work better for bilinear case.

(a) Fig. 10. Experimental results of lossy CFA image compression. The error is estimated between demosaiced RGB images. First, let s explain why bilinear demosaic method is not good for the conventional compression way. The bilinear method is a simple linear operation that causes severe color artifacts and zipper effects along strong edges. These artifacts result in very high frequency components in chrominance channels. However, these components are easily lost during the compression stage because JPEG assumes less high-frequency values in color and performs aggressive subsampling in the chrominance channels. This does not happen when complicated demosaic is used since those methods effectively suppress color artifacts from the beginning. In addition, we can explain why CFA compression methods give higher PSNR for bilinear interpolation. Good demosaic algorithms try to recover high frequency data from green channels and spread them to other color channels so that edges can be reconstructed sharply without causing color artifacts. However, the compression stage introduces various artifacts including blocking, which confuses demosaic algorithm. The demosaic algorithm can not tell whether the edge is from the image itself or it is an false edge caused by compression artifact. Thus, it amplifies compression artifacts which results in lower PSNR. As a result, the performance between the conventional method and CFA image compression methods is highly dependent on the choice of demosaic algorithm used afterwards. Many CFA methods [4], [7] assume bilinear demosaicing is used and claim victory over RGB image compression, but it may not be the case if we select better engineered demosaic methods. VI. CONCLUSIONS We have shown that CFA data can be efficiently compressed by using existing image compression algorithms such as JPEG. However, it is important to note that the overall performance of CFA compression methods is highly dependent on the choice of demosaic algorithm. When simple bilinear demosaic method is used, CFA image compression results in better images compared to conventional compression scheme. However, it may not be the case if you choose more sophisticated demosaic methods that minimize color artifacts. Additional studies will be required to come up with better solutions to improve the performance of CFA data compression. Another interesting direction of future works will be considering denoising, demosaicing, deblocking and image compression at the same time. There are several

are taken into account. All four operations are closely related and strongly affect the performance each other. We believe there is room for big improvement in this direction. (a) (c) Fig. 11. Various artifacts after compression. (a) Color artifacts (Direct JPEG, Bilinear). Blocking artifacts (Structure Conversion, Adaptive Frequency Domain). (c) Amplification of compression artifacts (Structure Conversion, Adaptive Homogeneity). works on switching the order of some operations or trying some of them jointly, but not all of the operations VII. APPENDIX: WORK DISTRIBUTION Sung Hee Park: ideal entropy estimation, pipeline simulation, presentation slides, final report. Albert No: lossy CFA compression methods, presentation slides, final report. REFERENCES [1] B. Bayer, Color imaging array, U.S. Patent 3971065, July 1976. [2] A. Gentile and F. Sorbello, Image processing chain for digital still cameras based on the simpil architecture, in ICPPW 05: Proceedings of the 2005 International Conference on Parallel Processing Workshops. Washington, DC, USA: IEEE Computer Society, 2005, pp. 215 222. [3] R. Ramanath, W. E. Snyder, Y. Yoo, and M. S. Drew, Color image processing pipeline, IEEE Signal Process. Mag, vol. 22, pp. 34 43, 2005. [4] K. Nallaperumal, S. Christopher, S. S. Vinsley, and R. K. Selvakumar, New efficient image compression method for single sensor digital camera images, in ICCIMA 07: Proceedings of the International Conference on Computational Intelligence and Multimedia Applications (ICCIMA 2007). Washington, DC, USA: IEEE Computer Society, 2007, pp. 113 117. [5] K.-H. Chung and Y.-H. Chan, A lossless compression scheme for bayer color filter array images. IEEE Trans Image Process, vol. 17, no. 2, pp. 134 44, 2008. [Online]. Available: http://www.biomedsearch.com/nih/lossless-compressionscheme-bayer-color/18270106.html [6] S. Battiato, A. Buemi, L. D. Torre, and A. Vitali, Fast vector quantization engine for cfa data compression, in In Proceedings of IEEE-EURASIP Workshop on Nonlinear Signal and Image Processing, NSIP, 2003. [7] X. Xie, G. Li, Z. Wang, C. Zhang, D. Li, and X. Li, A novel method of lossy image compression for digital image sensors with bayer color filter arrays, in ISCAS (5). IEEE, 2005, pp. 4995 4998. [8] Eastman Kodak Company, PhotoCD PCD0992, (http://r0k.us/graphics/kodak/). [9] K. Hirakawa and T. W. Parks, Adaptive homogeneity-directed demosaicing algorithm, IEEE Transactions on Image Processing, vol. 14, no. 3, pp. 360 369, 2005. [10] E. Dubois, Frequency-domain methods for demosaicking of Bayer-sampled color images, IEEE Signal Process. Lett., vol. 12, no. 12, pp. 847 850, 2005. [11] X. Zhang, B. A. Wandell, and B. A. W, A spatial extension of cielab for digital color image reproduction, in Proceedings of the SID Symposiums, 1996, pp. 731 734.