A robust, cost-effective post-processor for enhancing demosaicked camera images

Transcription

1 ARTICLE IN PRESS Real-Time Imaging 11 (2005) A robust, cost-effective post-processor for enhancing demosaicked camera images Rastislav Lukac,1, Konstantinos N. Plataniotis Multimedia Laboratory, BA 4157, The Edward S. Rogers Sr. Department of Electrical and Computer Engineering, University of Toronto, 10 King s College Road, Toronto, Ont., M5S 3G4, Canada Available online 31 May 2005 Abstract This paper presents an efficient post-processing/enhancement solution capable of reducing visual artifacts introduced during the image demosaicking process. Edge-sensing weights and the original color filter array data are used to detect structural elements in the captured image, and correct color components generated by the demosaicking process using adaptive, spectral model-based enhancement operations. The solution produces excellent results in terms of both objective and subjective image quality measures. r 2005 Elsevier Ltd. All rights reserved. 1. Introduction Color filter array (CFA) interpolation or demosaicking is a necessary stepin single-sensor imaging solutions [1 3]. The sensor, usually a charge-coupled device (CCD) [4] or complementary metal oxide semiconductor (CMOS) sensor [5], is essentially a monochromatic device and thus, the raw data that acquires in conjunction with the CFA constitute a mosaic-like gray-scale image. This so-called CFA image has only a single color component available at each spatial position of the CFA output. Therefore, the two missing color components must be estimated from the adjacent pixels using the demosaicking process to produce the full-color Red Green Blue (RGB) image [6 8]. The Bayer pattern (Fig. 1) [9], the most popular CFA solution, is commonly used in image-enabled wireless phones, pocket devices and visual sensors for surveillance and automotive applications [6,10]. Most of the available demosaicking designs, such as those in [2,3,7,8,11 13], introduce visual artifacts in the Corresponding author. address: lukacr@dsp.utoronto.ca (R. Lukac). URL: 1 Partially supported by a NATO/NSERC Science award. form of blurred edges and false colors, [1,10,14 16]. Therefore, demosaicked color image post-processing or enhancement, implemented either directly in hardware or as an additive software module, should be used to reduce the visual impairments introduced during the demosaicking process [16 18]. It was argued in [16] that demosaicking and postprocessing of demosaicked images are two operations performed at different stages in a single-sensor camera image processing pipeline. As it can be seen in Fig. 2, any demosaicking solution is used before post-processing or enhancement operations. In addition, demosaicking is considered a stand-alone procedure, while post-processing complements the available demosaicking solutions, not replacing them. Other essential differences, related to the nature, functionality, and design characteristics of the camera image processing steps are summarized below: Demosaicking is an integral and probably the most common processing step used in digital cameras. Demosaicking (Fig. 3a) performs spectral interpolation as it transforms a gray-scale (scalar) image to a three-channel, full color output. Namely, it rearranges the acquired gray-scale sensor data to the RGB vectorial field, and completes missing color /$ - see front matter r 2005 Elsevier Ltd. All rights reserved. doi: /j.rti

2 140 ARTICLE IN PRESS R. Lukac, K.N. Plataniotis / Real-Time Imaging 11 (2005) process improves both the color appearance and the sharpness of the demosaicked image by localizing and eliminating false colors created during demosaicking. Finally, it should be noted that unlike demosaicking, post-processing can be applied iteratively until certain quality criteria are met. Fig. 1. Bayer CFA pattern with the GRGR phase in the first row. Fig. 2. Simplified digital camera pipeline. Fig. 3. Fundamental differences between (a) demosaicking and (b) demosaicked image post-processing: (a) demosaicking transforms a gray-scale CFA image to a full color image, (b) demosaicked image post-processing enhances the quality of a previously demosaicked color image. components from the adjacent sensor data using an interpolator operating in the spectral (color) domain. Post-processing the demosaicked image is an optional step, implemented mainly in software and activated by the end-user. It performs full color image enhancement (Fig. 3b). The input of the solution is a fully restored RGB color image and the output is an enhanced RGB color image. The post-processing From the above listing it is evident that the two processing steps are fundamentally different, although may employ similar, if not identical, signal processing concepts. It should also be mentioned that the research in the area of demosaicking has recently culminated due to commercial proliferation of digital still cameras, which are the most popular acquisition devices. However, post-processing of demosaicked images is a novel application of great importance to both the end-users and the camera manufacturers. The proposed post-processor is a fully automated solution constructed using both an edge-sensing mechanism and a spectral model. The paper extends the preliminary results presented in [17,18]. In the postprocessing solution discussed here the color-difference model [19] is used instead of the color-ratio models [16] due to its simplicity and ease in implementation [8]. Moreover, the employed simple, fully adaptive, edgesensing mechanism based on the aggregated absolute differences between the CFA inputs makes the postprocessing procedure capable of tracking the spatial content of the demosaicked image. This results in naturally colored, sharpimages without color shifts. The proposed technique exhibits robust performance by yielding improvements for both cost-effective and sophisticated demosaicking algorithms used on a variety of natural and artificial images. It should be noted that the proposed procedure requires only modest computational resources and is thus appropriate for most, if not all, single-sensor imaging systems. It can be treated as either a post-processing procedure incorporated as the last step in a demosaicking pipeline or as an independent post-processing operation. The post-processor can be implemented in hardware or software by the digital camera. Alternatively, the post-processing of the demosaicked image can be performed in a personal computer (PC) using software distributed by the camera manufacturers or by utilizing conventional, public image processing tools. Thus, the proposed post-processing concept has potential to complete the existing CFAbased processing pipelines. The rest of the paper is organized as follows. In Section 2, the fundamentals of Bayer CFA-based demosaicking are briefly described for completeness. The proposed post-processing framework is introduced in Section 3. Motivation and design characteristics are discussed in detail and analyzed with respect to their properties. In Section 4, the proposed framework is tested using a variety of demosaicked images. Extended

3 ARTICLE IN PRESS R. Lukac, K.N. Plataniotis / Real-Time Imaging 11 (2005) simulation studies are included in order to demonstrate the effectiveness of the proposed scheme. Both subjective and objective criteria are utilized in order to quantify the improvement afforded by the employed post-processing module. Finally, the paper concludes in Section Single-sensor imaging basics Let us consider a Bayer CFA (Fig. 1)-based digital camera solution. It was explained before that the sensor is a monochromatic device and thus, the raw CFA data constitute a K 1 K 2 gray-scale image z : Z 2! Z. This CFA image represents a two-dimensional matrix of integer samples z ðp;qþ with p ¼ 1; 2;...; K 1 and q ¼ 1; 2;...; K 2 denoting the image rows and columns, respectively. Since spatial locations of z correspond to different (RGB) spectral bands determined by the underlying structure of the Bayer CFA, the sensor image z depicted in Fig. 4a has a mosaic-like structure with 25%, 50%, and 25% amounts of the z ðp;qþ values corresponding to R, G and B information, respectively. By allocating the double number of G locations compared to those assigned R or B components, the Bayer CFA improves the perceived sharpness of the digital image since it is well-known that the human visual system is more sensitive to luminance which is composed primarily of green light [14]. Based on the arrangements of color filters in the Bayer CFA, the gray-scale image z can be transformed to a K 1 K 2 color (RGB) image x : Z 2! Z 3 which represents a two-dimensional matrix of threecomponent samples (Fig. 4b). Each sample x ðp;qþ ¼ ½x ðp;qþ1 ; x ðp;qþ2 ; x ðp;qþ3 Š denote an RGB vector in x with x ðp;qþk indicating the R (k ¼ 1), G (k ¼ 2) and B (k ¼ 3) component. The CFA color vectors can be obtained from the acquired sensor values z ðp;qþ as follows [6,10]: 8 ½z ðp;qþ ; 0; 0Š for p odd and q even; >< x ðp;qþ ¼ ½0; 0; z ðp;qþ Š for p even and q odd; >: ½0; z ðp;qþ ; 0Š otherwise: The demosaicking procedure recovers missing color components of x ðp;qþ from the adjacent Bayer data using an interpolator operating in the spectral domain [6,10]. As shown in Fig. 5, a generalized demosaicking solution is constructed using the spectral model and the edgesensing mechanism [10]. The use of the spectral model, such as the color-ratio model [20], the normalized colorratio models [6,16], and the color-difference model [19], allows for the utilization of the essential spectral information and thus, the processing solution is capable of preserving the spectral correlation that exists between the color components of the natural image. Through the use of the parameters, usually weighting coefficients, the edge-sensing mechanism is used to direct the postprocessing process along edges thus preserving the sharpness and structural information of the demosaicked image [1,21]. Most of the demosaicking solutions such as those in [1,2,7,15,21] control the edge sensitivity by the parameters calculated from the sensor values within a relatively large 5 5or7 7 neighborhood. Our recent works [10,17] decrease the cost of the solution by calculating the edge-sensing weights using the aggregated difference concept defined over the four surrounding CFA samples only. In order to meet the objective of this paper and design a computationally efficient post-processing solution, the most cost-effective construction elements, namely, the color-difference model and the aggregated difference (1) Fig. 4. Example of a CFA image: (a) captured gray-scale image, (b) its color image variant obtained using (1). Fig. 5. Block scheme of a generalized demosaicking architecture. A similar architecture serves as the base of the proposed here demosaicked image post-processor.

4 142 ARTICLE IN PRESS R. Lukac, K.N. Plataniotis / Real-Time Imaging 11 (2005) concept-based edge-sensing mechanism, are used in the sequence as the base for the proposed post-processor. 3. Proposed post-processor for demosaicked image enhancement Since there is no method to objectively determine whether or not a color component is inaccurate, our post-processing framework utilizes the differences between the color components generated by a demosaicking solution and the original Bayer CFA data included in the restored color vector x ðp;qþ 2 x to post-process the demosaicked images x [17]. According to (1), the demosaicked image x, generated by a Bayer CFA (Fig. 1)-based demosaicking scheme, is obtained using the original R and B CFA components located at ðodd p; even qþ and ðeven p; odd qþ, respectively, as well as the original G components which occupy the rest of the locations in the CFA. The spectral correlation between the G and R (or B) components of the restored, full-color image x is utilized in the proposed post-processing process to further improve color appearance of x. Based on the colordifference model [19], the proposed post-processor 1 reevaluates the G components x ðp;qþ2 produced by the demosaicking process as follows: P x ðp;qþ2 ¼ x ðp;qþk þ ði;jþ2z w ði;jþðx ði;jþ2 x ði;jþk Þ P ði;jþ2z w, (2) ði;jþ where z ¼fðp 1; qþ; ðp; q 1Þ; ðp; q þ 1Þ; ðp þ 1; qþg denotes the spatial locations of the original G CFA components which surround the interpolated location ðp; qþ. As can be seen in Fig. 6a, the original G CFA components x ðp 1;qÞ ; x ðp;q 1Þ ; x ðp;qþ1þ ; x ðpþ1;qþ form a diamond-shaped mask on the image lattice. In Eq. (2), the quantities w ði;jþ signify the edge-sensing weights, x ðp;qþk denotes the original R (or B) component at the position under consideration, and ðx ði;jþ2 x ði;jþk Þ denotes the differences between the surrounding original G components and the R (or B) values produced by a demosaicking solution. If ðp; qþ is a position which corresponds to an R location in the Bayer CFA, then k ¼ 1 is used. Otherwise, the position which corresponds to a B location in the Bayer CFA is identified by k ¼ 3. Through the normalization procedure of (2), two constraints necessary to ensure that the output x ðp;qþ2 is an unbiased solution are satisfied. Namely, (i) each weight is a positive number, w ði;jþ X0, and (ii) the summation of all the weights w ði;jþ = P ðg;hþ2z w ðg;hþ, for ði; jþ 2z, is equal to unity. Note that each x ði;jþ2 is associated with a positive, real-valued, edge-sensing 1 Following the structure shown in Fig. 5, our framework uses both an edge-sensing mechanism and a spectral model to enhance the demosaicked image. Fig. 6. Spatial arrangements of the pixels and shape-masks obtained during the proposed post-processing. weight w ði;jþ which can be defined in many different ways. The interested reader should refer to [10] for additional information on the weights that edge-sensing mechanism can be realized. Cost considerations necessitate the utilization of a simple and easy to implement solution such as w ði;jþ ¼½1þd ði;jþ Š 1, where d ði;jþ is the aggregated absolute difference between the G CFA values x ði;jþ2 and x ðg;hþ2 : d ði;jþ ¼ X jx ði;jþ2 x ðg;hþ2 j. (3) ðg;hþ2z The weights w ði;jþ, for ði; jþ 2z, are used to regulate the contribution of the neighboring input components x ði;jþ2. When no edge is positioned across the directions in which we post-process the image, the corresponding aggregated absolute difference d ði;jþ is small and the CFA component x ði;jþ2, via its corresponding weight w ði;jþ, contributes greatly in (2). If the x ði;jþ2 and x ðg;hþ2 considered in calculating d ði;jþ are located across an edge, the corresponding absolute difference d ði;jþ increases resulting in the small value of w ði;jþ. That in return decreases the contribution of x ði;jþ2 in the output value calculated in (2). The color difference ðx ði;jþ2 x ði;jþk Þ used in (2) reduces the high-frequency components when compared to the individual color planes [1,8]. In this way, the averaging operation in (2) introduces in a smaller estimation error compared to the error introduced by a processing solution operating directly on the original R or B components. The utilization of x ðp;qþk in (2) preserves the high-frequency components of the original the G color

5 ARTICLE IN PRESS R. Lukac, K.N. Plataniotis / Real-Time Imaging 11 (2005) channel since the addition of x ðp;qþk to the normalized weighted sum scales the post-processor s output to the desired intensity range. After completing the post-processing of the G color plane via (2) the method proceeds by improving the contrast of the demosaicked R and B channels. First, the R (or B) components occupying the original B (or R) CFA locations are post-processed using the following equation: P x ðp;qþk ¼ x ðp;qþ2 þ ði;jþ2z w ði;jþðx ði;jþk x ði;jþ2 Þ P ði;jþ2z w. (4) ði;jþ The edge-sensing weights w ði;jþ are defined as w ði;jþ ¼ ½1þd ði;jþ Š 1 with d ði;jþ ¼ P ðg;hþ2z jx ði;jþk x ðg;hþk j where z ¼ fðp 1; q 1Þ; ðp 1; q þ 1Þ; ðp þ 1; q 1Þ; ðp þ1; q þ1þg. The parameter k is used to indicate R ðk ¼ 1Þ or B ðk ¼ 3Þ color components. The sample x ðp;qþ2 is the postprocessed G component which is located at the center of a square-shaped mask (Fig.6b) formed by four surrounding R (or B) components x ðp 1;q 1Þk, x ðp 1;qþ1Þk, x ðpþ1;q 1Þk, x ðpþ1;qþ1þk. The utilization of the colordifference quantities ðx ði;jþk x ði;jþ2 Þ reduces processing errors in both R and B channels. Note that each weight w ði;jþ in (4) is calculated using original R (or B) components x ðp 1;q 1Þk, x ðp 1;qþ1Þk, x ðpþ1;q 1Þk, and x ðpþ1;qþ1þk. Since (4) with z ¼fðp 1; q 1Þ; ðp 1; q þ 1Þ; ðp þ 1; q 1Þ; ðp þ 1; q þ 1Þg allows only for post-processing R components which are co-located with the original B CFA components, and B components co-located with the original R CFA values, an additional post-processing stepusing (4) is needed in order to process the R and B components x ðp;qþk co-located with the original G CFA data. It can be seen that the process forms a spatial arrangement (Figs.6c and d) similar to the one used in the G component post-processing (Fig. 6a). As before, the G component x ðp;qþ2 is located at the center of a diamond-shaped mask created by four surrounding R (or B) components x ðp 1;qÞk, x ðp;q 1Þk, x ðp;qþ1þk, x ðpþ1;qþk in the patterns shown in Figs.6c and d. Thus, the weighting coefficients w ði;jþ are calculated similarly using the R (or B) components x ði;jþk, for z ¼fðp 1; qþ; ðp; q 1Þ; ðp; q þ 1Þ; ðp þ 1; qþg and ði; jþ 2z. As shown in Fig. 7, the proposed post-processor can be implemented in a conventional digital camera and/or in a companion personal computer (PC) which interfaces with the digital camera. If the conventional architecture (Fig.7a) is considered, the post-processor, similarly to other camera s functions, can be activated by the end-user in the camera software interface. In the case of the personal computer-based architecture (Fig.7b), the post-processor can be used to automatically enhance the quality of the captured images downloaded from the camera using the software distributed by camera manufacturers. Note that both processing pipelines depicted in Fig. 7 can use the same post-processor, however, the approach depicted in Fig.7b allows for the utilization of sophisticated color image enhancement schemes which cannot, due to their complexity, be embedded in the conventional camera image processing pipeline (Fig.7a) due to the real-time constraint limitations. 4. Experimental results A number of natural and artificial color images have been used to evaluate the proposed post-processing framework with representative examples shown in Fig. 8. All images have been normalized to the standard , 8-bit per channel RGB representation, except for the Lighthouse image which is in size. The images depicted in Figs. 8a and b are the artificial test images circular zone plate (CZP) [8] and Pattern, respectively, which facilitate the analysis of aliasing effects resulting from demosaicking algorithms. These images contain sharp, high-contrast edges running in various directions and are thus well suited for analyzing the deficiencies that are common in demosaicking algorithms. The tests were performed by sampling the images with the Bayer CFA pattern to obtain a Bayer pattern image [3], demosaicking with a number of different schemes, then applying the proposed method to the restored images. Performance was measured by comparing the original full-color images with the demosaicked images obtained with and without the post-processing step. To facilitate the objective comparisons [6], the mean absolute error (MAE), the mean square error (MSE) and the normalized color difference (NCD) criterion are used. Fig. 7. Camera image processing architectures: (a) the solution with the post-processing operations performed by the digital camera and (b) the solution with the post-processing operations performed by the personal computer.

6 144 ARTICLE IN PRESS R. Lukac, K.N. Plataniotis / Real-Time Imaging 11 (2005) Fig. 8. Test color images: (a) CZP, (b) Pattern, (c) Lighthouse, (d) Parrots, (e) Window, (f) Train, (g) Sydney and (h) Bikes. Table 1 Objective results for the image CZP SAIG SHT MFI API EMI PVM C2D KA DAD BI BD ADS Table 2 Objective results for the image Pattern SAIG SHT MFI API EMI PVM C2D KA DAD BI BD ADS The MAE and MSE measures are defined as follows [6]: MAE ¼ MSE ¼ 1 X 3 3K 1 K 2 k¼1 1 X 3 3K 1 K 2 k¼1 X K 1 p¼1 X K 1 p¼1 X K 2 q¼1 X K 2 q¼1 jo ðp;qþk x ðp;qþk j, ðo ðp;qþk x ðp;qþk Þ 2, where o ðp;qþ ¼½o ðp;qþ1 ; o ðp;qþ2 ; o ðp;qþ3 Š is the original RGB pixel, x ðp;qþ ¼½x ðp;qþ1 ; x ðp;qþ2 ; x ðp;qþ3 Š is the processed pixel ð5þ ð6þ with ðp; qþ denoting a spatial position in a K 1 K 2 color image and k characterizing the color channel. The perceptual similarity between the original and the processed image is quantified using the normalized color difference (NCD) criterion [6]: NCD ¼ P K1 P K2 p¼1 q¼1 P K1 p¼1 qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi P 3 k¼1 ðō ðp;qþk ȳ ðp;qþk Þ 2 q ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi P, (7) 3 k¼1 ðō ðp;qþkþ 2 P K2 q¼1

7 ARTICLE IN PRESS R. Lukac, K.N. Plataniotis / Real-Time Imaging 11 (2005) Table 3 Objective results for the image Lighthouse SAIG SHT MFI API EMI PVM C2D KA DAD BI BD ADS Table 6 Objective results for the image Train SAIG SHT MFI API EMI PVM C2D KA DAD BI BD ADS Table 4 Objective results for the image Parrots SAIG SHT MFI API EMI PVM C2D KA DAD BI BD ADS Table 7 Objective results for the image Sydney SAIG SHT MFI API EMI PVM C2D KA DAD BI BD ADS Table 5 Objective results for the image Window SAIG SHT MFI API EMI PVM C2D KA DAD BI BD ADS Table 8 Objective results for the image Bikes SAIG SHT MFI API EMI PVM C2D KA DAD BI BD ADS

8 146 ARTICLE IN PRESS R. Lukac, K.N. Plataniotis / Real-Time Imaging 11 (2005) where ō ðp;qþ ¼½ō ðp;qþ1 ; ō ðp;qþ2 ; ō ðp;qþ3 Š and ȳ ðp;qþ ¼½ȳ ðp;qþ1 ; ȳ ðp;qþ2 ; ȳ ðp;qþ3 Š are the vectors representing the RGB vectors o ðp;qþ and y ðp;qþ, respectively, in the CIE LUV color space. Since our post-processor operates on demosaicked images, the overall final results depends critically on the preceding demosaicking solutions. In the experiments reported here, the following demosaicking schemes were used for the tests: the saturation-based adaptive inverse gradient (SAIG) scheme [2], the bilinear interpolation (BI) [3], the Kimmel s algorithm (KA) [7], the bilineardifference interpolation (BD) [8], the data-adaptive demosaicking (DAD) scheme [10], the edge map interpolation (EMI) [11], the median filter interpolation (MFI) [13], the adaptive demosaicking scheme (ADS) [15], the smooth hue transition approach (SHT) [20], the color correlation-directional derivatives (C2D2) [21], the adaptive color plane interpolation (API) [22], and the principle vector method (PVM)-based demosaicking scheme [23]. The numerical results are summarized in Tables 1 8, and some visual results in critical image regions with fine details and color edges are shown in Figs As can be seen from the numerical results, the proposed method provides significant improvement for almost all cases. In Fig. 9, the sharp, high-contrast edges result in false color artifacts which in turn produce moire effects in the images output from the various demosaicking algorithms. After applying the post-processing operation, the effects are either completely removed or significantly reduced. The improvement is most dramatic when it comes to the BI demosaicked images. Fig. 10 shows Fig. 9. Enlarged region of the original CZP image (a), and the restored outputs (b) (f) before (top image) and after (bottom image) post-processing: (b) SAIG, (c) API, (d) EMI, (e) C2D2 and (f) BI.

9 ARTICLE IN PRESS R. Lukac, K.N. Plataniotis / Real-Time Imaging 11 (2005) Fig. 10. Enlarged region of the original Pattern image (a), and the restored outputs (b) (f) before (top image) and after (bottom image) postprocessing: (b) MFI, (c) KA, (d) PVM, (e) BI and (f) SHT. similar results where the demosaicking algorithms produce significant blurring and false color artifacts surrounding the text and lines. After applying the proposed post-processing step, these effects are significantly reduced or removed altogether. Figs depict results obtained using natural test images. It is evident that the proposed post-processing scheme: (i) excellently eliminates color shifts in natural images, (ii) reduces visual impairments introduced during demosaicking, and (iii) never decreases the visual quality of the acquired image. All the post-processed images shown visual improvement with the most significant enhancement achieved when the post-processor was used to enhance demosaicked images obtained using the BI and PVM schemes. In summary, the overall best results in terms of both objective and subjective image evaluation measures, were those obtained by post-processing demosaicked images generated using the DAD and C2D2 methods. Apart from the numerical behavior (actual performance) of any algorithm, its computational complexity is a realistic measure of its practicality and usefulness. Therefore, the proposed post-processor is analyzed here in terms of normalized operations, such as additions (ADDs), subtractions (SUBs), multiplications (MULs), divisions (DIVs) and absolute values (AVs). The analysis reveals that 38 ADDs, 20 SUBs, 8 MULs, 16 DIVs, and 12 AVs per two components in any postprocessing locations are needed to perform by a postprocessor. The values-listed above suggest that the proposed post-processor is an efficient and cost-effective camera solution. Moreover, additional computational

10 148 ARTICLE IN PRESS R. Lukac, K.N. Plataniotis / Real-Time Imaging 11 (2005) Fig. 11. Enlarged region of the original Train image (a), and the restored outputs (b) (f) before (top image) and after (bottom image) post-processing: (b) API, (c) MFI, (d) KA, (e) C2D2 and (f) DAD. saving may occur when the aggregated differences d ði;jþ and/or edge-sensing weights w ði;jþ are implemented in parallel and shared forms. When implemented in software, on a PC equipped with an Intel Pentium IV 2.40 GHz CPU, 512 MB RAM, Windows XP operating system and MS Visual C programming environment, the proposed post-processor required (on average) seconds to process a demosaicked image. It should be noted that the objective of this analysis is to provide benchmark information regarding implementation issues and not to exhaustively cover all possible implementations. The development of software-optimized realizations of the algorithm under consideration is beyond the scope of this paper. 5. Conclusions A post-processing solution for the enhancement of demosaicked images was introduced in this paper. Color images restored using a demosaicking scheme are post-processed in order to reduce color artifacts and blurring. Edge-sensing weights and colordifference-based post-processing achieves both robust performance and excellent results, in terms of both objective and subjective image quality measures. The proposed solution can be implemented as either a post-processing step incorporated directly into a single-sensor imaging pipeline, or as a separate processing module. Under either scenario, the technique is capable of significantly increasing the quality of the

11 ARTICLE IN PRESS R. Lukac, K.N. Plataniotis / Real-Time Imaging 11 (2005) Fig. 12. Enlarged region of the original Sydney image (a), and the restored outputs (b) (f) before (top image) and after (bottom image) postprocessing: (b) SHT, (c) BI, (d) C2D2, (e) KA and (f) MFI. color images captured using single-sensor consumer imaging devices. References [1] Lukac R, Plataniotis KN, Hatzinakos D, Aleksic M. A novel cost effective demosaicing approach. IEEE Transactions on Consumer Electronics 2004;50: [2] Cai C, Yu TH, Mitra SK. Saturation-based adaptive inverse gradient interpolation for Bayer pattern images. IEE Proceedings - Vision, Image, Signal Processing 2001;148: [3] Longere P, Zhang X, Delahunt PB, Brainard DH. Perceptual assessment of demosaicing algorithm performance. Proceedings of the IEEE 2002;90: [4] Dillon PLP, Lewis DM, Kaspar FG. Color imaging system using a single CCD area array. IEEE Journal of Solid-State Circuits 1978;13: [5] Doswald D, Haflinger J, Blessing P, Felber N, Niederer P, Fichtner W. A 30 frames/s megapixel real-time CMOS image processor. IEEE Journal of Solid-State Circuits 2000;35: [6] Lukac R, Plataniotis KN. Normalized color-ratio modelling for CFA interpolation. IEEE Transactions on Consumer Electronics 2004;50: [7] Kimmel R. Demosaicing: image reconstruction from color CCD samples. IEEE Transactions on Image Processing 1999;8: [8] Pei SC, Tam IK. Effective color interpolation in CCD color filter arrays using signal correlation. IEEE Transactions on Circuits and Systems for Video Technology 2003;13: [9] Bayer BE. Color imaging array. US Patent , [10] Lukac R, Plataniotis KN. Data-adaptive filters for demosaicking: a framework. IEEE Transactions on Consumer Electronics, submitted for publication. [11] Hur BS, Kang MG. High definition color interpolation scheme for progressive scan CCD image sensor. IEEE Transactions on Consumer Electronics 2001;47:

12 150 ARTICLE IN PRESS R. Lukac, K.N. Plataniotis / Real-Time Imaging 11 (2005) [12] Ramanath R, Snyder WE, Bilbro GL, Sander WA. Demosaicking methods for the Bayer color array. Journal of Electronic Imaging 2002;11: [13] Freeman WT. Median filter for reconstructing missing color samples. US Patent , [14] Gunturk B, Altunbasak Y, Mersereau R. Color plane interpolation using alternating projections. IEEE Transactions on Image Processing 2002;11: [15] Lu W, Tang YP. Color filter array demosaicking: new method and performance measures. IEEE Transactions on Image Processing 2003;12: [16] Lukac R, Martin K, Plataniotis KN. Demosaicked image postprocessing using local color ratios. IEEE Transactions on Circuit and Systems for Video Technology 2004; 14: [17] Lukac R, Martin K, Plataniotis KN. Colour-difference based demosaicked image postprocessing. IEE Electronics Letters 2003; 39: [18] Lukac R, Martin K, Plataniotis KN, Smolka B. Robust correction stepfor CFA interpolation schemes. Proceedings of the 2004 International Conference on Image Processing ICIP , II: [19] Adams J. Design of practical color filter array interpolation algorithms for digital cameras. Proceedings of the SPIE 1997;3028: [20] Cok DR. Signal processing method and apparatus for producing interpolated chrominance values in a sampled color image signal. US Patent , [21] Kehtarnavaz N, Oh HJ, Yoo Y. Color filter array interpolation using color correlations and directional derivatives. Journal of Electronic Imaging 2003;12: [22] Hamilton JF, Adams JE. Adaptive color plane interpolation in single sensor color electronic camera. US Patent , [23] Kakarala R, Baharav Z. Adaptive demosaicing with the principle vector method. IEEE Transactions on Consumer Electronics 2002;48:932 7.