Content-based Grayscale Image Colorization

Transcription

1 Content-based Grayscale Image Colorization Dr. Bara'a Ali Attea Baghdad University, Iraq/ Baghdad Dr. Sarab Majeed Hameed Baghdad University, Iraq/ Baghdad Aminna Dahim Aboud Baghdad University, Iraq/ Baghdad ABSTRACT This paper presents a new, effective and simple approach for grayscale image colorization. Based on human vision theories and digital signal analysis, image layers of different spatial frequencies would have different perceptual and physical (i.e. surface roughness) properties. Associating this concept with the classical color transfer technique of Welsh et al simplifies the colorization process when the image has no distinct texture or confusing luminance distribution. The results demonstrate the effectiveness of the proposed colorization scheme when compared with the classical Welsh et al colorization method for coloring a wide range of images including homogeneous and heterogeneous ones. Key Words: color transfer, image processing, content-based decomposition, luminance distribution, color space. 1. Introduction One of the most common tasks in image processing and editing is to alter an image's color- the process of enhancing monochromatic image through the addition of color. The visual appeal of some image such as black-and-white photos, old movies, and scientific illustrations can be increased by adding editing colors to the achromatic information of these images. For some scientific images, their information content can be enhanced with color by exploiting variations in chromaticity and luminance information [1]. Additionally, colorization is shown to be useful in image compression, where with the luminance information and some samples of the color (much less than the ordinary sub-sampling in common compression schemes), the color components of the data can be faithfully recovered. This has implications also, in wireless image transmission, where lost image blocks can be recovered from the available channels [2]. Moreover, people can chat with live video using cheap monochromatic web-cams instead of color ones, limited bandwidth for transmitting monochromatic video, and inexpensive and fully automatic colorization software [3]. In general, colorization of grayscale images has several challenges including ambiguity, fuzzy boundary identification, and user expert. Since different colors may carry the same luminance in spite of differences in hue and/or saturation, the problem of colorizing gray-scaled images (i.e., the problem of inverting a gray palette to a color palette) has no inherently correct solution. Thus, this in general a severely under-constrained and ambiguous problem for which it makes no sense to try to find an optimum solution, and for which even the obtainment of reasonable solution requires some combination of strong prior knowledge about the scene depicted and decisive human intervention.

2 Even in the case of pseudo coloring, where the mapping of luminance values to color values is automatic, the choice of the color map is commonly determined by human decision [3][4]. In a broader context, colorization problem is related to a recent surge of interest on the general problem of transferring properties from one image (or movie) to another. Hertzmann in [5] developed an approach, image analogies, in which various types of styles (e.g., artistic styles like oil painting, water coloring, pen-and-ink) can be transferred between images. The style is captured from one source image and applied to the target image. Additionally, some techniques exist for transferring various types of textures, for instant natural texture and geometric texture [6] [7]. For color characteristics transfer, Reinhard et al [8] introduced a general form for borrowing one image's color characteristics from another image by imposing mean and standard deviation onto the data points of the input images in an attempt to provide a general form of color correction. In this paper, we combine contentbased image regions layering method with Welsh et al classical colorization [1] to improve the colorization power dramatically. Actually, one can relate the basic idea of our approach with the colorization methodology of Vieira et al [3]. Both approaches blend Welsh et al color transfer method with techniques from the very active area of content-based image decomposition and retrieval. In their work, Vieira et al utilized content-based image retrieval idea to automatically selecting the most suitable source color image from an image database to color the target image. In this paper, we exploit the idea of matching different areas of features in the source image with the corresponding features in the target image to transfer color between them. Our goal is to develop a colorization algorithm that is effective and general (i.e. it should produce pleasing results for a wide range of images), userfriendly (i.e. the amount of user intervention should be minimal), simplicity (the algorithm implementation could be as simple as possible), and Efficient (i.e. computational cost should be (if possible) comparable to times required by techniques provided by other authors). 2. Previous Work Traditionally, colorization is done by first segmenting the image into regions, and then proceeds to assign colors to each segment. Unfortunately, automatic segmentation algorithms often fail to correctly identify fuzzy or complex region boundaries. Moreover, colorization of movies requires, in addition, tracking regions across the frames of a shot. Existing tracking algorithms typically fail to robustly track non-rigid regions, a gain requiring massive user intervention in the process [9]. In [1] Welsh et al combine the idea of Reinhard et al [8] with a texture synthesis technique similar to image quilting of Efros and Freeman [10] to transfer color between images. Welsh et al presented two colorization methods. One is global fully automated, and requires no user intervention. In this version, the source color and target grayscale images have globally similar texture or luminance distributions. In the second method- a multi swatch color transfer- the user can select specific color moods in the color image to colorize specific regions of the grayscale image. Then, the rest regions of the target image are colorized by propagating colors from the colorized swatches in the target image. In general, Welsh et al colorization method works very impressively for natural and scientific illustration images. However, their fully automated colorization technique does not work very well with human faces and images that have delicate boundaries between regions or with very similar luminance distributions. It fails to classifying the differences between skin and lip or, more generally, between regions with similar or confusing luminance distributions. Later, Di Blasi and R. Recupero [4] presented a fully automated colorization scheme which avoids the full-

3 search sampling strategy of Welsh et al method using AntipoleTree Data Structure and a more refined searching strategy. As with Welsh et al method, Antipole colorization method work well for homogeneous images, and require less computation time but at the expense of increasing implementation complexity. Additionally, several researchers had described different colorization methods techniques adopted for still images and movies which proved to give well results for images consist of well visible outlines that emphasize the shape of image's homogeneous regions. Some of these are color-by-example technique of Sŷkora et al [11] for coloring black-and-white cartoons, video clip colorization of Pan et al [12], color inpainting of Sapiro [2], and color propagation based on pixel neighborhood tracking of Levin et al [9]. Likewise Welsh et al method, these colorization techniques may require additional user skill to select swatch pairs of source and target images to transfer color more accurately. However, these techniques have additional implementation complexities when compared with the classical full search strategy of Welsh et al method. 3. The Proposed Colorization Algorithm The context of content-based image decomposition has attracted several research interests in many fields related to computer science for over a decades. One such challenging issue is image indexing and retrieval. According to the frequency analysis theory of human visual system, there exist multiple frequency sensitive channels; each acts as band-pass filter responses only to a certain bandwidth of the visual stimulus. In other words, when we see the visual world, we perform some form of frequency analysis that will decompose the image into various frequency components (i.e. layers) each layer covers only certain areas of the spatial frequency spectrum. In the human color vision system, the spatial sharpness of a color image depends mainly on the sharpness of the light dark component of the image and very little on the structure of the opponent-color image components. The sharpness or roughness of an image region determines its perceptual significance of an image. Psychophysical studies of color vision suggest that there is a division of function between achromatic component (luminance) vision and chromatic components (color). Luminance channel is able to detect sharp edges and the fine details of patterns and textures in the image. On the other hand, color is used to fill in the color objects. A busy/ sharp area is associated with higher frequency distributions (e.g., object boundaries or textured surfaces) whereas a flat area may be associated with backgrounds or interior of objects. This may suggest that frequency analysis is mostly occurring in the luminance channel [13]. The idea of multiple spatial frequencies layering in luminance channel is applied in this paper to the development of an effective grayscale image colorization method. The classical colorization method of Welsh et al [1] is used to transfer color from the source layers to target layers that correspond to similar perceptual significance (similar objects or object parts). The main steps of the proposed colorization algorithms can be simply stated as: Preprocess both source color and target grayscale images by converting them from the correlated RGB color space to a de-correlated space (e.g., YIQ). Then perform global luminance distributions matching between both images to reach similar luminance histogram but with preserving the qualitative appearance of the original images. We follow luminance distributions matching of Hertzmann [5]. Given two color image A and B that have sufficiently different color histograms, luminance distributions matching can be addressed as: compute a linear mapping that matches the means and variances of the luminance distributions. More concretely, if Y (p) is the luminance of a pixel in image A, then we remap it as

4 σ γ(ρ) ' = B (γ(ρ) µ ) + µ σ A B (1) A Where µ and A luminance, and µ are the mean B σ and σ are B A the standard deviations of the luminance, both taken with respect to luminance distributions in A and B, respectively. De-compose both source color and target grayscale images into an equal number of spatial layers in which each layer has specific perceptual significance (or specific physical objects). Each source/target layer consists of a source/target image of the same size as the original one, but retains only pixels in areas within a certain spatial frequency range. This can be done by first applying a Gaussian low-pass filter to the image (we use 3 3 filter with coefficients 0.05, 0.25, 0.4, 0.25, and 0.05 used in [13]). Next, form a Laplacian image by taking the absolute value of subtracting the Gaussian smoothed image from the original image. This is followed by decomposing the original image into a number of layers (we usually use four layers). Each individual layer contains only those pixels in areas with specified Laplacian magnitude (sharpness/roughness magnitude). The frequency components of an individual target layer are colorized (using Welsh et al classical colorization method) from the corresponding source layer. In other words, the target features are colorized from source areas of similar spatial roughness (e.g., flat gray areas are colorized from the corresponding flat color areas). Each target layer is scanned in scan-line order so that, for every one of its pixels, the counterpart 4. Results source layer that belong to the same sharpness/roughness magnitude is only searched for best pixel matching in terms of luminance mean and standard deviation (within small pixel neighborhood, e.g., 3 3 or 5 5 ). Then, combine the chromatic components (both I andq ) of the selected source pixel with the achromatic component of the current target pixel. Continuing this process for all target pixels and for all layers will eventually convert the scalar luminance image to a vectorial color one. This section reports some results obtained by running the proposed colorization algorithm to a range of image domains with different sizes. The experiments include cases in which the grayscale image contains regions with similar or confusing luminance distributions, e.g., human faces. Our results (see figures 1and 2) are compared with results of Welsh et al classical full search method (both are implemented using un-optimized visual Basic code.) The neighborhood statistics required in both algorithms are precomputed over the source and target images. Processing time (in seconds) required for a single Pentium IV PC computer is also included for both colorization algorithms. By comparing our results with those of Welsh et al, we can easily demonstrate the effectiveness of the proposed easy to implement colorization scheme. Moreover, the computation time of our algorithm to obtain visually accepted results was better than the classical full search algorithm of Welsh et al. 5. Conclusion In this paper, we have introduced a new method for grayscale image colorization. It

5 is an extension to the classical Welsh et al colorization method. Based on human vision theories and digital signal analysis, image layers of different spatial frequencies would have different perceptual and physical properties. In this paper, the classical Welsh et al colorization is augmented to incorporate the image layering argument in an attempt to the development of content-based image colorization. This colorization method enhances the power of Welsh et al method, and provide effective results with better computation time, and with without any user intervention for swatch pairs selection. References: [1] T. Welsh, M. Ashikhmin, and K. Mueller, "Transferring color to grayscale images", ACM TOG, Vol. 20, No. 30, 2002, pp [2] G. Sapiro, "Inpainting the colors", IMA Preprint Series #1979, Institute for Mathematics and Its Applications, University of Minnesota, May [3] L. F. M. Vieira, R. D. Vilela, and E. R. do Nascimento, "Automatically choosing source color images for coloring grayscale images", In 16th Brazilian Symposium on Computer Graphics and Image Processing (SIBGRAPI 2003), Sao Carlos, Brazil. IEEE Computer Society, ISBN , 2003, pp [4] G. Di Blasi, and R. D. Reforgiato, "Fast colorization of gray images", In proceedings of Eurographics Italian Chapter [5] A. Hertzmann, "Algorithms for rendering in artistic styles", PhD thesis, New York University, [6] M. Ashikhmin, "Fast texture transfer", IEEE Computer Graphics and Applications, Vo. 23, No. 4, 2003, pp [7] A. Hertzmann, N. Oliver, B. Curless, and S. M. Seitz, 'Curve analogies", In Proc. 13 th Eurograpics Workshop on Rendering, [8] E. Reinhard, M. Ashikhmin, B. Gooch, and. P. Shirley, "Color transfer between images", IEEE Computer Graphics and Applications, Vol. 21, No. 5, 2001, pp [9] A. Levin, D. Lischinski, and Y. Weiss, "Colorization using Optimization", Proc. ACM SIGGRAPH 2004, Vol. 23, No. 3, pp [10] A. A. Efros, and W. T. Freeman, "Image quilting for texture synthesis and transfer", In Proceeding of ACM SIGGRAPH 2001, pp [11] D. Sŷkora, J. Buriánek, and J. Žára, "Unsupervised colorization of blackand-white cartoons", In Proceedings NPAR rd International Symposium on Non-Photorealistic Animation and Rendering. New York: ACM SIGGRAPH, 2004, ISBN , pp [12] Z. Pan, Z. Dong, and M. Zhang, "A new algorithm for adding color to video or clip animation", In the 12-th International Conference in Central Europe on Computer Graphics, Visualization and Computer Vision'2004, WSCG 2004, University of West Bohemia, Campus Bory, Plzen-Bory, Czech Republic, February 2-6, 2004, pp [13] G. Qiu, and K. Lam, "Frequency layered color indexing for contentbased image retrieval", IEEE Trans. Image Processing, Vol. 12, No. 1, 2003, pp

6 147 x x x x x x x x x x x x Figure1. Comparison results between the proposed colorization and Welsh et al classical colorization methods. 1 st column is the source color image with its size, 2 nd column is the target grayscale image with its size, 3 rd column is our results with computation time required, and 4 th column is Welsh et al results with computation time required.

7 147 x x x x x x x x x x 159 Figure2. Comparison results between the proposed colorization and Welsh et al classical colorization methods. 1 st column is the source color image with its size, 2 nd column is the target grayscale image with its size, 3 rd column is our results with computation time required, and 4 th column is Welsh et al results with computation time required