Te-Wei Chiang Department of Accounting Information Systems Chihlee Institute of Technology ctw@mail.chihlee.edu.tw Performance Analysis of s in Histogram-Based Image Retrieval Tienwei Tsai Department of Information Management Chihlee Institute of Technology twt@mail.chihlee.edu.tw Mann-Jung Hsiao Department of Information Management Kang-Ning Junior College hsiao@mis.knjc.edu.tw Abstract In this paper, we investigate the effectiveness of each component of the three well-known color spaces (RGB, YUV, and HSV) in histogram-based color image retrieval. An image retrieval demo system is built to make it easy to test the retrieval performance. In the demo system, each image is first transformed into the investigated color space. Then, the histogram for each color channel of the image is obtained, which is served as the color feature of the image. We have performed experiments on a database with 1000 images, using the Euclidean distance metric. The results show that the HSV and RGB color spaces perform well in fine matching; however, the YUV color space performs better in coarse classification. Moreover, even though the B and U components are less sensitive to human vision, they are quite important for the histogram-based retrieval. Key words:color space, content-based image retrieval, query-by-example, color histogram. 1. Introduction With the growth of multimedia application, the development of user-friendly image retrieval system that can efficiently and effectively retrieve the desired images from a large image database becomes more and more important. Generally, image retrieval procedures can be roughly divided into two approaches: query-by-text (QbT) and query-by-example (QbE). In QbT, queries are texts and targets are images; in QbE, queries are images and targets are images. For practicality, images in QbT retrieval are often annotated by words, such as time, place, or photographer. To access the desired image data, the seeker can construct queries using homogeneous descriptions, such as keywords, to match these annotations. Such retrieval is known as annotation-based image retrieval (ABIR). ABIR has the following drawbacks. First, manual image annotation is time-consuming and therefore costly. Second, human annotation is subjective. Furthermore, some images could not be annotated because it is difficult to describe their content with words. On the other hand, annotations are not necessary in a QbE setting, although they can be used. The retrieval is carried out according to the image contents. Such retrieval is known as contentbased image retrieval (CBIR) [3]. CBIR becomes popular for the purpose of retrieving the desired images automatically. Smeulder et al. [5] reviewed more than 200 references in this field. CBIR is a technology to search for similar images to a query based only on the image pixel representation. However, the query based on pixel information is quite time-consuming because it is necessary to devise a means of describing the location of each pixel and its intensity. Therefore, how to choose a suitable color space and reduce the data to be computed is a critical problem in image retrieval. In this paper, we investigate the effectiveness of each component of the three well-known color spaces, i.e., RGB, YUV, and HSV, for CBIR. In our demo system, each image is first transformed from the standard RGB color space into the YUV (or HSV) space. Then, the histogram for each component (e.g., luminance Y, blue chrominance U, and red chrominance V) of the image is obtained, which can be served as the color feature of the image. In the image database establishing phase, the features of each image are stored; in the image retrieving phase, the system compares the features of the query image with those of the images in the database, using the Euclidean distance metric and find out good matches. The rest of this paper is organized as follows. Section 2 and 3 introduce the color space and color histogram. Section 4 presents the distance measure. Section 5 shows the experimental results. Finally, conclusions are drawn in Section 6. 2. Color Space A color space is a model for representing color in terms of intensity values. It specifies how color information is represented. The common used color spaces [1], [2] includes: RGB, CMY(K), YUV, YCrCb, MTM, HSV, HSB, HLS, CIE L*a*b*, CIE L*u*v, etc. The RGB and CMY(K) color spaces are hardware-oriented. The RGB (Red, Green, and Blue) color model is perhaps the simplest color space, which is used in most color CRT monitors. The CMYK (cyan, magenta, yellow, and black) space is a color space that models the way ink builds up in printing. These two color spaces are not perceptually uniform. Alternative color spaces can be generated by transforming the RGB color space. For example, linear transformations of the RGB color spaces produce a number of important color spaces that include YUV (NTSC, PAL, and SECAM
color television standard), YCrCb (JPEG digital image coding standard and MPEG digital video coding standard). These linear transforms, each of which generates one luminance channel and two chrominance channels, were designed specifically to accommodate targeted display devices: YUV - color television and YCrCb - color computer display. Because these color spaces are not uniform, color distance does not correspond well to perceptual color dissimilarity. On the other hand, some user-oriented color models were developed according to human perception of colors, including MTM, HSV family, and CIE family. Humans feel colors through hue, saturation and brightness percepts. Each of the models is specified by a vector of values; each component of that vector being valid on a specified range. The details of RGB, YUV, and HSV color spaces, which are of most interest, are introduced as follows. 2.1 RGB Color Space A gray-level digital image can be defined to be a function of two variables, f(x, y), where x and y are spatial coordinates, and the amplitude f at a given pair of coordinates is called the intensity of the image at that point. Every digital image is composed of a finite number of elements, called pixels, each with a particular location and a finite value. Similarly, for a color image, each pixel (x, y) consists of three components: R(x, y), G(x, y), and B(x, y), each of which corresponds to the intensity of the red, green, and blue color in the pixel, respectively. 2.2 YUV Color Space Originally used for PAL (European "standard") analog video, YUV is based on the CIE Y primary, and also chrominance. The Y primary was specifically designed to follow the luminous efficiency function of human eyes. Chrominance is the difference between a color and a reference white at the same luminance. The following equations are used to convert from RGB to YUV color space: Y = 0.299R + 0.587G + 0.114B, (1) U = 0.492( B Y), and (2) V = 0.877( R Y). (3) Basically, the Y, U, and V components of an image can be regarded as the luminance, the blue chrominance, and the red chrominance, respectively. From Eq. (2), it can be found that the blue chrominance U is obtained from removing the luminance component Y from the blue component B; the red chrominance V is obtained similarly. The YUV model is based on the opponent color theory of human vision and intends to approximate color differences as perceived by humans. The opponent color theory was first proposed by Ewald Hering [4]. He thought that the colors red, yellow, green, and blue are special in that any other color can be described as a mix of them, and that they exist in opposite pairs. That is, either red or green is perceived and never greenish-red. 2.3 HSV Color Space The HSV stands for the Hue, Saturation, and Value based on the artists (Tint, Shade, and Tone). The Value represents intensity of a color, which is decoupled from the color information in the represented image. The hue and saturation components are intimately related to the way human eye perceives color resulting in image processing algorithms with physiological basis. As hue varies from 0 to 1.0, the corresponding colors vary from red, through yellow, green, cyan, blue, and magenta, back to red, so that there are actually red values both at 0 and 1.0. As saturation varies from 0 to 1.0, the corresponding colors (hues) vary from unsaturated (shades of gray) to fully saturated (no white component). As value, or brightness, varies from 0 to 1.0, the corresponding colors become increasingly brighter. The conversion formula from RGB to HSV color space is as follows: 1 [( R G) + ( R B)] / 2 H = cos (4) 2 ( R G) + ( R B)( G B) S = 1 3 [min( R, G, B)]/( R + G + B), and (5) V = ( R+ G+ B) / 3. (6) 3. Color Histogram The color histogram for an image is constructed by discretizing (or quantizing) the colors within the image and counting the number of pixels of each color. More formally, it is defined as h ( x, y, z) = N Prob( X = x, Y = y, Z ), (7) X, Y, Z = z where X, Y and Z represent the three color channels (R,G,B, or Y,U,V, or H,S,V) and N is the number of pixels in the image. The color histogram can be regarded as a set of vectors. For gray-scale images these are 2-D vectors. One dimension gives the value of the gray-level and the other the count of pixels at the gray-level. As for color images each color channel can be regarded as gray-scale images. More generally, we can set the number of bins in the color histograms to obtain the feature vector of desired size. Basically, there are three types of histogram distribution: 1) low key: the mean value of the histogram is significantly less than the center of the legal range; 2) midtone: the mean value of the histogram nears the center of the legal range; 3) high key: the mean value of the histogram is greater than the center of the legal range. Figure 1 illustrates the three types of histogram distribution. Note that the center value of range depends on the type of color component. For example, for R, G, B, and Y components, the center value is
(a) Figure 1. Illustration of three types of histogram distribution: (a) low key, midtone, and high key. 128; for U and V components of YUV space, the center value is 0; for H, S, and V components of HSV space, the center value is 0.5. To obtain the color feature of an image, some color components can easily be analyzed according their histogram type. For example, one can realize the brightness of an image by checking the histogram type of the color components such as G (for RGB), Y (for YUV), and V (for HSV); on the other hand, one can realize the dominant color of an image according to the following heuristic rules: 1) if the U component is high key, then the dominant color is blue; 2) if the U component is low key, then the dominant color is yellow; 3) if the V component of YUV space is high key, then the dominant color is red; 4) if the V component of YUV space is low key, then the dominant color is green; 5) if the H component is low key, then the dominant color is red; 6) if the H component is midtone, then the dominant color is green; 7) if the H component is high key, then the dominant color is blue, and so on. 4. Distance Measurement To decide which image in the image database is the most similar one with the query image, we have to define a measure to indicate the degree of similarity. Therefore, the distance (or dissimilarity) between a feature vector F m of the query image and that of an image in the database is based on the distance function. In our approach, the distance between two vectors is calculated on the basis of the sum of squared differences (SSD). Assume that F qm and F represent the mth feature of the query image Q and xm an image X in the database, respectively; each feature may come from the color histograms. Then, the distance between F and qm F can be defined as xm d m ( F qm, F xm K 1 ) = ( F i= 0 qm [ i] F xm [ i]) where i is the ith coefficient of the mth feature and F qm = F xm = K. 5. Experimental Results We evaluated performance on a test image database, which was downloaded from the WBIIS database [6]. It is a general-purpose database including 1,000 color images. 2, (8) The images are mostly photographic and have various contents, such as natural scenes, animals, insects, building, people, and so on. In the experiment, an image retrieval demo system is built to test the three color spaces and their components, as shown in Figure 2. 5.1 Color Channels and Histograms To gain deeper insight of each color space and their components, five sample images, i.e., white owl, pumpkins, red apples, green apples, and deer, are selected to illustrate each color component (or color channel) and the corresponding histograms (see Figures 3-7). It can be found that the shape of the histogram derived form the Y components are similar to that derived from the G components; it is because G is the dominant component that constitutes the Y component (see Eq. (1)). Intuitively, to verify whether an image s color tone is red or not, one may attempt to analyze the R component of the RGB image; however, R component is not as effective as the V component from this point of view. It makes vain attempt to analyze the R component of the RGB image because most of the energy of the R component contributes to the luminance of the image; on the other hand, it is more effective to analyze the V component of the image, which eliminates the component that constitutes the luminance of the image. For example, using the 3rd heuristic rule proposed in Section 3, the dominant color of the apples in Figure 5 can be detected from the histogram of the V component of the YUV space. Other heuristic rules can easily be verified from Figures 3-7. 5.2 Performance Evaluation To compare the three color spaces and their components in a quantitative manner, five classes of query images, referring to white owl (5 images), pumpkins (4 images), red apples (3 images), green apples (3 images), and deer (9 images), are served as the benchmark queries. To assess the ground-truth relevance score to each image for each benchmark query, each target image in the collection is assigned a relevance score as follows: 1 if it belonged to the same class as the query image, and 0 otherwise. The process was repeated for all the relevant images and an overall average retrieval effectiveness was computed for each of the color component and each of the query example. The overall average relevance score in top n was computed by averaging the individual values in each top n. The bin number used for each color component is 5. Then, the retrieval effectiveness for each color space and their components can be evaluated. Tables 1 to 5 show the retrieval results for each color space and their components using the five classes of benchmark query images. Table 6 shows the average results of the five classes of benchmark query images. To distinguish from the V component of the YUV space, the V component of the HSV space is renamed as V' in these tables. From the viewpoint of evaluating the
three color spaces, it can be seen that histogram-based retrieval in HSV and RGB color spaces showed better performance in terms of the top 5 and top 10 accuracy; however, the retrieval in YUV color space showed the best performance in terms of the top 20 and top 50 accuracy. From the viewpoint of evaluating the color components, it can be seen that histogram-based retrieval in B and U color components showed better performance in terms of the top 5, top 10, and top 20 accuracy; however, the retrieval in V component showed the best performance in terms of the top 50 accuracy. Thus, we can conclude that: The HSV and RGB color spaces perform well in fine matching; however, the YUV color space performs better in coarse classification. Even though the B and U components are less sensitive to human vision (see EQ. (1)-(2)), they are quite important in terms of the effectiveness of the histogram-based retrieval. The V component performs well in coarse classification. 6. Conclusions In this paper, we investigate the effectiveness of the three well-known color spaces, i.e., RGB, YUV, and HSV, and their components in histogram-based color image retrieval. An image retrieval demo system is built to make it easy to test the retrieval performance. We have performed experiments on a database with 1000 images, using the Euclidean distance metric. The results show that the HSV and RGB color spaces perform well in fine matching; however, the YUV color space performs better in coarse classification. Moreover, even though the B and U components are less sensitive to human vision, they are quite important for the histogram-based retrieval. References [1] Bimbo, A. D., Visual Information Retrieval, San Francisco: Morgan Kaufmann, 1999. [2] Castelli, V. and Bergman, L. D., Image Databases, New York: John Wiley & Sons, 2002. [3] Gudivada, V., and Raghavan, V., Content-Based Image Retrieval Systems, IEEE Computers, 28(9), pp.18-22, 1995. [4] Hering E., Outlines of a Theory of the Light Sense, Cambridge, Mass., Harvard Univ. Press, 1964. [5] Smeulders, A. W. M., Worring, M., Santini, S., Gupta, A., and Jain, R., Content-Based Image Retrieval at the End of the Early Years, IEEE Trans. on Pattern Analysis and Machine Intelligence, 22(12), pp. 1349-1380, 2000. [6] Wang, J. Z., Content Based Image Search Demo Page. http://bergman.stanford.edu/~zwang/project/imsearch/ WBIIS.html, 1996. Figure 2. The GUI of our demo system.
(a) Figure 3. (a) The query image (white owl); the R, G, B color channels and the corresponding color histograms; the Y, U, V color channels and the corresponding color histograms; the H, S, V color channels and the corresponding color histograms. (a) Figure 4. (a) The query image (pumpkins); the R, G, B color channels and the corresponding color histograms; the Y, U, V color channels and the corresponding color histograms; the H, S, V color channels and the corresponding color histograms.
(a) Figure 5. (a) The query image (red apples); the R, G, B color channels and the corresponding color histograms; the Y, U, V color channels and the corresponding color histograms; the H, S, V color channels and the corresponding color histograms. (a) Figure 6. (a) The query image (green apples); the R, G, B color channels and the corresponding color histograms; the Y, U, V color channels and the corresponding color histograms; the H, S, V color channels and the corresponding color histograms.
(a) Figure 7. (a) The query image (deer); the R, G, B color channels and the corresponding color histograms; the Y, U, V color channels and the corresponding color histograms; the H, S, V color channels and the corresponding color histograms. Table 1. Qurey 1: White Owl (5 relevant images) Comparison of Color Spaces and s # Relevant in Top 5 1.5 2 2 1 0.5 1.5 1 2 0.5 1.5 0.5 1 # Relevant in Top 10 2 2 2 1 1.5 2 1.5 2 0.5 1.5 2 2 # Relevant in Top 20 2.5 2.5 2.5 2 2.5 2 2 2 1.5 2 2 2.5 # Relevant in Top 50 3.5 3.5 3.5 3.5 3.5 2.5 3.5 2 2.5 2.5 2 3.5 Table 2. Qurey 2: Pumpkins (4 relevant images) Comparison of Color Spaces and s # Relevant in Top 5 2.5 1.5 2.5 0.5 0.5 1.0 0.0 1.0 1.0 0.0 1.0 0.0 # Relevant in Top 10 2.5 2.0 3.0 1.0 0.5 1.5 0.5 2.5 1.0 0.5 1.5 1.0 # Relevant in Top 20 3.0 3.0 3.0 1.0 1.0 3.0 1.0 2.5 1.0 1.5 2.5 1.0 # Relevant in Top 50 3.0 3.0 3.0 3.0 2.0 3.0 2.5 3.0 3.0 3.0 3.0 2.5
Table 3. Qurey 3: Red Apples (3 relevant images) Comparison of Color Spaces and s # Relevant in Top 5 2.0 2.0 2.0 0.0 1.0 2.0 1.0 1.0 1.0 1.0 0.0 2.0 # Relevant in Top 10 2.0 2.0 2.0 2.0 1.0 2.0 2.0 1.0 2.0 2.0 2.0 2.0 # Relevant in Top 20 2.0 2.0 2.0 2.0 2.0 2.0 2.0 1.0 2.0 2.0 2.0 2.0 # Relevant in Top 50 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 Table 4. Qurey 4: Green Apples (3 relevant images) Comparison of Color Spaces and s # Relevant in Top 5 2.0 2.0 2.0 2.0 1.0 1.0 1.0 2.0 0.0 1.0 1.0 1.0 # Relevant in Top 10 2.0 2.0 2.0 2.0 2.0 1.0 2.0 2.0 1.0 1.0 1.0 2.0 # Relevant in Top 20 2.0 2.0 2.0 2.0 2.0 1.0 2.0 2.0 1.0 2.0 1.0 2.0 # Relevant in Top 50 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 Table 5. Qurey 5: Deer (9 relevant images) Comparison of Color Spaces and s # Relevant in Top 5 2.5 2.5 2.5 2.0 1.5 1.5 1.0 2.0 0.5 0.5 2.0 2.0 # Relevant in Top 10 3.5 3.0 2.5 2.5 2.5 2.5 2.0 2.5 1.5 1.0 2.0 2.5 # Relevant in Top 20 3.5 4.5 3.5 3.0 3.0 3.0 2.5 2.5 4.5 3.0 2.5 3.5 # Relevant in Top 50 3.5 8.0 5.5 3.5 3.5 3.5 3.5 6.0 8.0 5.0 3.5 3.5 Table 6. The Overall Average Results Comparison of Color Spaces and s # Relevant in Top 5 2.1 2.0 2.2 1.1 0.9 1.4 0.8 1.6 0.6 0.8 0.9 1.2 # Relevant in Top 10 2.4 2.2 2.3 1.7 1.5 1.8 1.6 2.0 1.2 1.2 1.7 1.9 # Relevant in Top 20 2.6 2.8 2.6 2.0 2.1 2.2 1.9 2.0 2.0 2.1 2.0 2.2 # Relevant in Top 50 2.8 3.7 3.2 2.8 2.6 2.6 2.7 3.0 3.5 2.9 2.5 2.7