AMONG THE human senses, sight and color perception

Transcription

1 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 6, NO. 7, JULY Digital Color Imaging Gaurav Sharma, Member, IEEE, and H. Joel Trussell, Fellow, IEEE Abstract This paper surveys current technology and research in the area of digital color imaging. In order to establish the background and lay down terminology, fundamental concepts of color perception and measurement are first presented using vector-space notation and terminology. Present-day color recording and reproduction systems are reviewed along with the common mathematical models used for representing these devices. Algorithms for processing color images for display and communication are surveyed, and a forecast of research trends is attempted. An extensive bibliography is provided. I. INTRODUCTION AMONG THE human senses, sight and color perception are perhaps the most fascinating. There is, consequently, little wonder that color images pervade our daily life in television, photography, movies, books, and newspapers. With the digital revolution, color has become even more accessible. Color scanners, cathode ray tube (CRT) displays, and printers are now an integral part of the office environment. Extrapolating from current trends, homes will also have a plethora of digital color imaging products in the near future. The increased use of color has brought with it new challenges and problems. In order to meaningfully record and process color images, it is essential to understand the mechanisms of color vision and the capabilities and limitations of color imaging devices. It is also necessary to develop algorithms that minimize the impact of device limitations and preserve color information as images are exchanged between devices. The goal of this paper is to present a survey of the technology and research in these areas. The rest of this paper is broadly organized into four sections. Section II provides an introduction to color science for imaging applications. Commonly used color recording and reproduction devices are discussed in Section III. A survey of algorithms used for processing color images in desktop applications is presented in Section IV. Finally, research directions in color imaging are summarized in Section V. II. COLOR FUNDAMENTALS Prior to the time of Sir Isaac Newton, the nature of light and color was rather poorly understood [1], [2]. Newton s meticulous experiments [3], [4, Chap. 3] with sunlight and Manuscript received September 15, 1996; revised February 24, The associate editors coordinating the review of this manuscript and approving it for publication were Drs. Ping Wah Wong, Jan Allebach, Mark D. Fairchild, and Brian Funt. G. Sharma is with Xerox Corporation, Webster, NY USA ( sharma@wrc.xerox.com). H. J. Trussell is with the Electrical and Computer Engineering Department, North Carolina State University, Raleigh, NC USA ( hjt@eos.ncsu.edu). Publisher Item Identifier S (97) a prism helped dispel existing misconceptions and led to the realization that the color of light depended on its spectral composition. Even though Grimaldi preceded Newton in making these discoveries, his book [5], [2, pp ] on the subject received attention much later, and credit for the widespread dissemination of the new ideas goes to Newton. While Newton s experiments established a physical basis for color, they were still a long way from a system for colorimetry. Before a system to measure and specify color could be developed, it was necessary to understand the nature of the color sensing mechanisms in the human eye. While some progress in this direction was made in the late 18th century [6], the prevalent anthropocentric views contributed to a confusion between color vision and the nature of light [6], [7]. The wider acceptance of the wave theory of light paved the way for a better understanding of both light and color [8], [9]. Both Palmer [6] and Young [9] hypothesized that the human eye has three receptors, and the difference in their responses contributes to the sensation of color. However, Grassmann [10] and Maxwell [11] were the first to clearly state that color can be mathematically specified in terms of three independent variables. Grassmann also stated experimental laws of color matching that now bear his name [12, p. 118]. Maxwell [13], [14] demonstrated that any additive color mixture could be matched by proper amounts of three primary stimuli, a fact now referred to as trichromatic generalization or trichromacy. Around the same time, Helmholtz [15] explained the distinction between additive and subtractive color mixing and explained trichromacy in terms of spectral sensitivity curves of the three color sensing fibers in the eye. Trichromacy provided strong indirect evidence for the fact that the human eye has three color receptors. This fact was confirmed only much later by anatomical and physiological studies. The three receptors are known as the S, M, and L cones (short, medium, and long wavelength sensitive) and their spectral sensitivities have now been determined directly through microspectrophotometric measurements [16], [17]. Long before these measurements were possible, colormatching functions (CMF s) were determined through psychophysical experiments [12], [18] [21]. CMF s are sets of three functions related to the spectral sensitivities of the three cones by nonsingular linear transformations. The CMF s determined by Guild [19] and Wright [18] were used by the CIE (Commission Internationale de l Éclairage or the International Commission on Illumination) to establish a standard for a numerical specification of color in terms of three coordinates or tristimulus values. While the CMF s provide a basis for a linear model for color specification, it is clear that the human visual sensitivity /97$ IEEE

2 902 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 6, NO. 7, JULY 1997 to color changes is nonlinear. Since color differences between real world objects and images are evaluated by human viewers, it is desirable to determine uniform color spaces in which equal Euclidean distances correspond to roughly equal perceived color differences. Considerable research has focused on this problem since the establishment of colorimetry. Tristimulus values are useful for specifying colors and communicating color information precisely. Uniform color spaces are useful in evaluating color matching/mismatching of similar stimuli under identical adaptation conditions. Since the human visual system undergoes significant changes in response to its environment, tristimuli under different conditions of adaptation cannot be meaningfully compared. Since typical color reproduction problems involve different media or viewing conditions, it is necessary to consider descriptors of color appearance that transcend these adaptations. This is the goal of color appearance modeling. A. Trichromacy and Human Color Vision In the human eye, an image is formed by light focused onto the retina by the eye s lens. The three types of cones that govern color sensation are embedded in the retina, and contain photosensitive pigments with different spectral absorptances. If the spectral distribution of light incident on the retina is given by, where represents wavelength (we are ignoring any spatial variations in the light for the time being), the responses of the three cones can be modeled as a three vector with components given by where denotes the sensitivity of the th type of cones, and denotes the interval of wavelengths outside of which all these sensitivities are zero. Typically in air or vacuum, the visible region of the electromagnetic spectrum is specified by the wavelength region between nm and nm. Mathematically, the expressions in (1) correspond to inner product operations [22] in the Hilbert space of square integrable functions. Hence, the cone response mechanism corresponds to a projection of the spectrum onto the space spanned by three sensitivity functions. This space is called the human visual subspace (HVSS) [23] [26]. The perception of color depends on further nonlinear processing of the retinal responses. However, to a first order of approximation, under similar conditions of adaptation, the sensation of color may be specified by the responses of the cones. This is the basis of all colorimetry and will be implicitly assumed throughout this section. A discussion of perceptual uniformity and appearance will be postponed until Sections II-C and II-D. For computation, the spectral quantities in (1) may be replaced by their sampled counterparts to obtain summations as numerical approximations to the integrals. For most color spectra, a sampling rate of 10 nm provides sufficient accuracy, but in applications involving fluorescent lamps with sharp (1) spectral peaks, a higher sampling rate or alternative approaches may be required [27] [30]. If uniformly spaced samples are used over the visible range, (1) can be compactly written as where the superscript denotes the transpose, is an 3 matrix whose th column,, is the vector of samples of, and is the vector of samples of. The HVSS then corresponds to the column space of. In normal human observers, the spectral sensitivities of the three cones are linearly independent. Furthermore, the differences between the spectral sensitivities of color-normal observers are (relatively) small [18], [31], [12, p. 343] and arise primarily due to the difference in the spectral transmittance of the eye s lens and the optical medium ahead of the retina [18], [32] [34]. If a standardized set of cone responses is defined, color may be specified using the three-vector,, in (2), known as a tristimulus vector. Just as several different coordinate systems may be used for specifying position in three-dimensional (3- D) space, any nonsingular well-defined linear transformation of the tristimulus vector,, can also serve the purpose of color specification. Since the cone responses are difficult to measure directly, but nonsingular linear transformations of the cone responses are readily determined through color-matching experiments, such a transformed coordinate system is used for the measurement and specification of color. 1) Color Matching: Two spectra, represented by - vectors, and, produce the same cone responses and therefore represent the same color if To see how (2) encapsulates the principle of trichromacy and how CMF s are determined, consider three color primaries, i.e., three colorimetrically independent light sources. The term colorimetrically independent will be used in this paper to denote a collection of spectra such that the color of any one cannot be visually matched by any linear combination of the others. Mathematically, colorimetric independence of is equivalent to the linear independence of the three-vectors, and. Hence if, the matrix is nonsingular. For any visible spectrum,, the three-vector satisfies the relation which is the relation for a color match. Hence, for any visible spectrum,, there exists a combination of the primaries,, which matches the color of. This statement encapsulates the principle of trichromacy. From the standpoint of obtaining a physical match, the above mathematical argument requires some elaboration. It is possible that the obtained vector of primary strengths,, has negative components (in fact it can be readily shown that for any set of physical primaries there exist visible spectra for which this happens). (2) (3) (4)

3 SHARMA AND TRUSSELL: DIGITAL COLOR IMAGING 903 Fig. 1. Color-matching experiment. Since negative intensities of the primaries cannot be produced, the spectrum is not realizable using the primaries. A physical realization corresponding to the equations is, however, still possible by rearranging the terms in (4) and subtracting the primaries with negative strengths from. The double negation cancels out and corresponds to the addition of positive amounts of the appropriate primaries to. The setup for a typical color-matching experiment is shown schematically in Fig. 1. The observer views a small circular field that is split into two halves. The spectrum is displayed on one half of a visual field. On the other half of the visual field appears a linear combination of the primary sources. The observer attempts to visually match the input spectrum by adjusting the relative intensities of the primary sources. The vector,, denotes the relative intensities of the three primaries when a match is obtained. Physically, it may be impossible to match the input spectrum by adjusting the intensities of the primaries. When this happens, the observer is allowed to move one or two of the primaries so that they illuminate the same field as input spectrum, (see Fig. 2). As noted earlier, this procedure is mathematically equivalent to subtracting that amount of primary from the primary field, i.e., the strengths in corresponding to the primaries which were moved are negative. As demonstrated in the last paragraph, all visible spectra can be matched using this method. 2) Color-Matching Functions: The linearity of color matching expressed in (3) implies that if the color tristimulus values for a basis set of spectra are known, the color values for all linear combinations of those spectra can be readily deduced. The unit intensity monochromatic spectra, given by, where is an -vector having a one in the th position and zeros elsewhere, form a orthonormal basis in terms of which all spectra can be expressed. Hence, the color matching properties of all spectra (with respect to a given set of primaries) can be specified in terms of the color matching properties of these monochromatic spectra. Consider the color-matching experiment of the last section for the monochromatic spectra. Denoting the relative inten- Fig. 2. Color-matching experiment with negative value for primary p 1. sities of the three primaries required for matching by, the matches for all the monochromatic spectra can be written as Combining the results of all get (5) monochromatic spectra, we where is the identity matrix, and is the color-matching matrix corresponding to the primaries. The entries in the th column of correspond to the relative amount of the th primary required to match, respectively. The columns of are therefore referred to as the color-matching functions (CMF s) (associated with the primaries ). From (6), it can be readily seen that the color-matching matrix. Hence the CMF s are a nonsingular linear transformation of the sensitivities of the three cones in the eye. It also follows that the color of two spectra, and, matches if and only if. As mentioned earlier, color of a visible spectrum,, may be specified in terms of the tristimulus values,, instead of. The fact that the color-matching matrix is readily determinable using the procedure outlined above makes such a scheme for specifying color considerably attractive in comparison to one based on the actual cone sensitivities. Note also that the HVSS that was defined as the column space of can alternately be defined as the column space of. 3) Metamerism and Black Space: As stated in (3), two spectra represented by -vectors and match in color if (or ). Since (or equivalently )isan 3 matrix, with 3, it is clear that there are several different spectra that appear to be the same color to the observer. Two distinct spectra that appear the same are called metamers, and such a color match is said to be a metameric match (as opposed to a spectral match). Metamerism is both a boon and a curse in color applications. Most color output systems (such as CRT s and color photography) exploit metamerism to reproduce color. However, in the (6)

4 904 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 6, NO. 7, JULY 1997 Fig. 3. CIE r(); g(); b() color-matching functions. matching of reflective materials, a metameric match under one viewing illuminant is usually insufficient to establish a match under other viewing illuminants. A common manifestation of this phenomenon is the color match of (different) fabrics under one illumination and mismatch under another. The vector space view of color matching presented above was first given by Cohen and Kaupauf [35], [36], [24]. Tutorial descriptions using current notation and terminology appear in [23], [25], [37], and [38]. This approach allows us to deduce a number of interesting and useful properties of color vision. One such property is the decomposition of the dimensional spectral space into the 3-D HVSS and the -dimensional metameric black space, which was first hypothesized by Wyszecki [39]. Mathematically, this result states that any visible spectrum,, can be written as where is the orthogonal projector onto the column space of, i.e., the HVSS, and is the orthogonal projector onto the black space, which is the orthogonal complement of the HVSS. The projection,, is called the fundamental metamer of because all metamers of are given by. Another direct consequence of the above description of color matching is the fact that the primaries in any colormatching experiment are unique only up to metamers. Since metamers are visually identical, the CMF s are not changed if each of the three primaries are replaced by any of their metamers. The physical realization of metamers imposes additional constraints over and above those predicated by the equations above. In particular, any physically realizable spectrum needs to be nonnegative, and, hence, it is possible that the metamers (7) described by the above mathematics may not be realizable. In cases where a realizable metamer exists, set theoretic approaches may be used to incorporate nonnegativity and other constraints [37]. B. Colorimetry It was mentioned in Section II-A2 that the color of a visible spectrum,, can be specified in terms of the tristimulus values,, where is a matrix of CMF s. In order to have agreement between different measurements, it is necessary to define a standard set of CMF s with respect to which the tristimulus values are stated. A number of different standards have been defined for a variety of applications, and it is worth reviewing some of these standards and the historical reasons behind their development. 1) CIE Standards: The CIE is the primary organization responsible for standardization of color metrics and terminology. A colorimetry standard was first defined by the CIE in 1931 and continues to form the basis of modern colorimetry. The CIE 1931 recommendations define a standard colorimetric observer by providing two different but equivalent sets of CMF s. The first set of CMF s is known as the CIE Red Green Blue (RGB) CMF s,. These are associated with monochromatic primaries at wavelengths of 700.0, 546.1, and nm, respectively, with their radiant intensities adjusted so that the tristimulus values of the equienergy spectrum are all equal [40]. The equi-energy spectrum is the one whose spectral irradiance (as a function of wavelength) is constant. The CIE RGB CMF s are shown in Fig. 3. The second set of CMF s, known as the CIE XYZ CMF s, are and ; they are shown in Fig. 4. They were recommended for reasons of more convenient application in

5 SHARMA AND TRUSSELL: DIGITAL COLOR IMAGING 905 Fig. 4. CIE x(); y(); z() color matching functions. colorimetry and are defined in terms of a linear transformation of the CIE RGB CMF s [41]. When these CMF s were first defined, calculations were typically performed on desk calculators, and the repetitive summing and differencing due to the negative lobes of the CIE RGB CMF s was prone to errors. Hence, the transformation from the CIE RGB CMF s to CIE XYZ CMF s was determined so as to avoid negative values at all wavelengths [42]. Since an infinite number of transformations can be defined in order to meet this nonnegativity requirement, additional criteria were used in the choice of the CMF s [43], [44, p. 531]. Two of the important considerations were the choice of coincident with the luminous efficiency function [12] and the normalization of the three CMF s so as to yield equal tristimulus values for the equi-energy spectrum. The luminous efficiency function gives the relative sensitivity of the eye to the energy at each wavelength. From the discussion of Section II-A1, it is readily seen that CMF s that are nonnegative for all wavelengths cannot be obtained with any physically realizable primaries. Hence, any set of primaries corresponding to the CIE XYZ CMF s is not physically realizable. The tristimulus values obtained with the CIE RGB CMF s are called the CIE RGB tristimulus values, and those obtained with the CIE XYZ CMF s are called the CIE XYZ tristimulus values. The tristimulus value is usually called the luminance and correlates with the perceived brightness of the radiant spectrum. The two sets of CMF s described above are suitable for describing color matching when the angular subtense of the matching fields at the eye is between one and four degrees [12, p. 131], [40, p. 6]. When the inadequacy of these CMF s for matching fields with larger angular subtense became apparent, the CIE defined an alternate standard colorimetric observer in 1964 with different sets of CMF s [40]. Since imaging applications (unlike quality control applications in manufacturing) involve complex visual fields where the colorhomogeneous areas have small angular subtense, the CIE 1964 (10 observer) CMF s will not be discussed here. In addition to the CMF s, the CIE has defined a number of standard illuminants for use in colorimetry of nonluminous reflecting objects. The relative irradiance spectra of a number of these standard illuminants is shown in Fig. 5. To represent different phases of daylight, a continuum of daylight illuminants has been defined [40], which are uniquely specified in terms of their correlated color temperature. The correlated color temperature of an illuminant is defined as the temperature of a black body radiator whose color most closely resembles that of the illuminant [12]. D65 and D50 are two daylight illuminants commonly used in colorimetry, which correspond to correlated color temperatures of 6500 and 5000 K, respectively. The CIE illuminant A represents a black body radiator at a temperature of 2856 K and closely approximates the spectra of incandescent lamps. A nonluminous object is represented by the -vector,,of samples of its spectral reflectance, where. When the object is viewed under an illuminant with spectrum given by the vector,, the resulting spectral radiance at the eye is obtained as the product of the illuminant spectrum and the reflectance at each wavelength. Therefore, the CIE XYZ tristimulus values defining the color are given by where is the matrix of CIE XYZ CMF s, is the diagonal (8)

6 906 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 6, NO. 7, JULY 1997 Fig. 5. CIE standard illuminants. illuminant matrix with entries from along the diagonal, and. In analogy with the HVSS, the column space of is defined as the human visual illuminant subspace (HVISS) [26]. Note that (8) is based on an idealized model of illuminant object interaction that does not account for several geometry/surface effects such as the combination of specular and body reflectance components [30, pp ]. 2) Chromaticity Coordinates and Chromaticity Diagrams: Since color is specified by tristimuli, different colors may be visualized as vectors in 3-D space. However, such a visualization is difficult to reproduce on two-dimensional (2-D) media and therefore inconvenient. A useful 2-D representation of colors is obtained if tristimuli are normalized to lie in the unit plane, i.e., the plane over which the tristimulus values sum up to unity. Such a normalization is convenient as it destroys only information about the intensity of the stimulus and preserves complete information about the direction. The coordinates of the normalized tristimulus vector are called chromaticity coordinates, and a plot of colors on the unit plane using these coordinates is called a chromaticity diagram. Since the three chromaticity coordinates sum up to unity, typical diagrams plot only two chromaticity coordinates along mutually perpendicular axes. The most commonly used chromaticity diagram is the CIE xy chromaticity diagram. The CIE xyz chromaticity coordinates can be obtained from the tristimulus values in CIE XYZ space as (9) Fig. 6 shows a plot of the curve corresponding to visible monochromatic spectra on the CIE xy chromaticity diagram. This shark-fin-shaped curve, along which the wavelength (in nm) is indicated, is called the spectrum locus. From the linear relation between irradiance spectra and the tristimulus values, it can readily be seen that the chromaticity coordinates of any additive-combination of two spectra lie on the line segment joining their chromaticity coordinates [12]. From this observation, it follows that the region of chromaticities of all realizable spectral stimuli is the convex hull of the spectrum locus. In Fig. 6, this region of physically realizable chromaticities is the region inside the closed curve formed by the spectrum locus and the broken line joining its two extremes, which is known as the purple line. 3) Transformation of Primaries NTSC, SMPTE, and CCIR Primaries: If a different set of primary sources,, is used in the color matching experiment, a different set of CMF s,, are obtained. Since all CMF s are nonsingular linear transformations of the human cone responses, the CMF s are related by a linear transformation. The relation between the two color-matching matrices is given by [37] (10) Note that the columns of the 3 3 matrix are the tristimulus values of the primaries with respect to the primaries. Note also that the same transformation,, is useful for the conversion of tristimuli in the primary system to tristimuli in the primary system. Color television (TV) was one of the first consumer products exploiting the phenomenon of trichromacy. The three lightemitting color phosphors in the television CRT form the three primaries in this color-matching experiment. In the United States, the National Television Systems Committee (NTSC)

7 SHARMA AND TRUSSELL: DIGITAL COLOR IMAGING 907 Fig. 6. CIE xy chromaticity diagram. recommendations for a receiver primary system based on three phosphor primaries were adopted by the Federal Communications Commission (FCC) in 1953 for use as a standard in color TV. The FCC standard specified the CIE xy chromaticity coordinates for the phosphors [45] as (red), (green), (blue) [46]. In addition, the tristimulus values were assumed to correspond to a white color typically specified as the illuminant D65. The chromaticity coordinates along with the white balance condition define the CIE XYZ tristimuli of the NTSC primaries, which determine the relation of NTSC RGB tristimuli to CIE XYZ tristimuli as per (10). In the early color TV system, the signal-origination colorimetry was coupled with the colorimetry of displays, with the tacit assumption that the processing at the receiver involves only decoding and no color processing is performed. As display technology changed, manufacturers began using more efficient phosphors and incorporated some changes in the decoding as a compensation for the nonstandard phosphors [47]. Similar changes took place in the monitors used by broadcasters, but they were unaware of the compensating mechanisms in the consumer TV sets. As a result, there was considerable color variability in the broadcast TV system [45]. To overcome this problem, the chromaticities of a set of controlled phosphors was defined for use in broadcast monitors, which now forms the Society of Motion Picture and Television Engineers (SMPTE) C phosphor specification [48], [49]. Current commercial TV broadcasts in the United States are based on this specification. With the development of newer display technologies that are not based on CRT s (see Section III-A4), it is now recognized that signal-origination colorimetry needs to be decoupled from the receiver colorimetry and that color correction at the receiver should compensate for the difference. However, for compatibility reasons and to minimize noise in transformations, it is still desirable to keep the reference primaries for broadcast colorimetry close to the phosphor primaries. Toward this end, the International Radio Consultative Committee (CCIR) [50] has defined a set of phosphor primaries by the chromaticity coordinates (red), (green), and (blue) for use in high definition television (HDTV) systems. Prior to transmission, tristimuli in SMPTE RGB and CCIR RGB spaces are nonlinearly compressed (by raising them to a power of 0.45) and encoded for reducing transmission bandwidth [50], [51] (the reasons for these operations will be explained in Sections III-A1 and IV-C). Note however, that the encoding and nonlinear operations must be reversed before the signals can be converted to tristimuli spaces associated with other primaries. Transformations for the conversion of color tristimulus values between various systems can be found in [52, pp ], [53, p. 71], [54], and [55]. C. Uniform Color Spaces and Color Differences The standards for colorimetry defined in Section II-B provide a system for specifying color in terms of tristimulus values that can be used to represent colors unambiguously in a 3-D space. It is natural to consider the relation of the distance between colors in this 3-D space to the perceived difference between them. Before such a comparison can be made, it is necessary to have some means for quantifying perceived color differences. For widely different color stimuli, an observer s assessment of the magnitude of color difference is rather variable and subjective [12, p. 486]. At the same time, there is little practical value in quantifying large differences in color, and therefore most research has concentrated on quantifying small color differences. For this purpose, the notion of a just

8 908 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 6, NO. 7, JULY 1997 noticeable difference (JND) in stimuli has been extensively used as a unit by color scientists. An alternate empirically derived system, which has also been used often, is the Munsell color system [56], [57]. In the Munsell system, all possible colors are defined in terms of the perceptual attributes of lightness, hue, and chroma; and associated Munsell book(s) of color contain reflective samples, which (when viewed under daylight), are spaced apart in perceptually equal steps of these attributes [12]. Lightness, hue, chroma, and other terms of color perception will be used in this paper in accordance with common terminology, but a definition will not be attempted here because of their subjective nature. Definitions are, however, provided in [12, p. 487], [58], [59], and [60]. Several researchers have examined the distribution of JND colors in CIE xy chromaticity and CIE XYZ tristimuli spaces and have found that it varies widely over the color space [61] [65]. Hence, the CIE XYZ space is perceptually nonuniform in that equal perceptual differences between colors do not correspond to equal distances in the tristimulus space. Since perceptual uniformity is an extremely desirable feature for defining tolerances in color reproduction systems, considerable research has been directed toward the development of uniform color spaces. Traditionally, the problem has been decomposed into two sub-problems: i) one of determining a uniform lightness scale, and ii) the other of determining a uniform chromaticity diagram for equilightness color stimuli. The two are then combined with suitable scaling factors for the chromaticity scale and the lightness scale to make their units correspond to the same factor of a JND. The historical milestones in the search for uniform brightness and lightness scales are described in Wyszecki and Stiles [12, pp ]. Typical experiments determine these scales either by a process of repeated bisection of the scale extremes or by moving up in increments of a JND. A cube-root power law relation between brightness and luminance provides a satisfactory fit for most experimental data and, therefore, has the most widespread acceptance at present [12, p. 494]. The search for a uniform lightness scale was complemented by efforts toward determination of a uniform chromaticity scale for constant lightness. Two of these attempts are noteworthy. The first determined a linear transformation of the tristimulus space that yielded a chromaticity diagram with JND colors being roughly equispaced [66], [67]. This was the precursor of the CIE 1960 u, v diagram [12, p. 503]. The second was primarily motivated by the Munsell system and used a nonlinear transformation of the CIE XYZ tristimuli to obtain a chromatic-value diagram in which the distances of Munsell colors of equal lightness would be in proportion to their hue and chroma differences [68]. The form for the nonlinear transformation was based on a color vision model proposed earlier by Adams [69], and the diagram is therefore referred to as Adams chromatic-value diagram. Based on the aforementioned research, the CIE has recommended two uniform color spaces for practical applications: the CIE 1976 (CIELUV) space and the CIE 1976 (CIELAB) space [40]. These spaces are defined in terms of transformations from CIE XYZ tristimuli into these spaces. Both spaces employ a common lightness scale,, that depends only on the luminance value. The lightness scale is combined with different uniform chromaticity diagrams to obtain a 3-D uniform color space. For the CIELUV space, a later version of CIE 1960 u, v diagram is used whereas CIELAB uses a modification of Adams chromatic-value diagram [12, p. 503]. In either case, the transformations include a normalization involving the tristimuli of a white stimulus, which provides a crude approximation to the eye s adaptation ( see Section II-D1). Euclidean distances in either space provide a color-difference formula for evaluating color differences in perceptually relevant units. 1) The CIE 1976 Space: The values corresponding to a stimulus with CIE XYZ tristimulus values are given by [40] where (11) (12) (13) (14) (15) (16) (17) (18) and are the tristimuli of the white stimulus. The Euclidean distance between two color stimuli in CIELUV space is denoted by (delta E-uv), and is a measure of the total color difference between them. On an average, a value of around 2.9 corresponds to a JND [70]. As mentioned earlier, the value of serves as a correlate of lightness. In the plane, the radial distance and angular position serve as correlates of chroma and hue, respectively. 2) The CIE 1976 Space: The coordinate of the CIELAB space is identical to the coordinate for the CIELUV space, and the transformations for the and coordinates are given by (19) (20) where, and are as defined earlier. The Euclidean distance between two color stimuli in CIELAB space is denoted by (delta E-ab), and a value of around 2.3 corresponds to a JND [70]. Once again, in the plane, the radial distance and angular position serve as correlates of chroma and hue, respectively.

9 SHARMA AND TRUSSELL: DIGITAL COLOR IMAGING 909 2) Other Color Difference Formulae: As may be expected, the CIELUV and CIELAB color spaces are only approximately uniform and are often inadequate for specific applications. The uniformity of CIELAB and CIELUV is about the same, but the largest departures from uniformity occur in different regions of the color space [71] [73]. Several other uniform color spaces and color difference formulae have been proposed since the acceptance of the CIE standards. Since CIELAB has gained wide acceptance as a standard, most of the difference formulae attempt to use alternate (non-euclidean) distance measures 1 in the CIELAB space. Prominent among these are the CMC (l:c) distance function based on the CIELAB space [74] and the BFD (l:c) function [75], [76]. A comparison of these and other uniform color spaces using perceptibility and acceptability criteria appears in [70]. In image processing applications involving color, the CIELAB and CIELUV spaces have been used extensively, whereas in industrial color control applications the CMC formulae have found wider acceptance. Recently [77], the CIE issued a new recommendation for the computation of color differences in CIELAB space that incorporates several of the robust and attractive features of the CMC (l:c) distance function. D. Psychophysical Phenomena and Color Appearance Models The human visual system as a whole displays considerable adaptation. It is estimated that the total intensity range over which colors can be sensed is around 10 : 1. While the cones themselves respond only over a 1000 : 1 intensity range, the vast total operating range is achieved by adjustment of their sensitivity to light as a function of the incident photon flux [78]. This adjustment is believed to be largely achieved through a feedback from the neuronal layers that provide temporal lowpass filtering and adjust the cones output as a function of average illumination. A small fraction of the adaptation corresponding to a factor of around 8 : 1 is the result of a 4 : 1 change in the diameter of the pupil that acts as the aperture of the eye [60, p. 23]. Another fascinating aspect of human vision is the invariance of object colors under lights with widely varying intensity levels and spectral distributions. Thus objects are often recognized as having approximately the same color in phases of daylight having considerable difference in their spectral power distribution and also under artificial illumination. This phenomenon is called color constancy. The term chromatic adaptation is used to describe the changes in the visual system that relate to this and other psychophysical phenomena. While colorimetry provides a representation of colors in terms of three independent variables, it was realized early on that humans perceive color as having four distinct hues corresponding to the perceptually unique sensations of red, green, yellow, and blue. Thus, while yellow can be produced by the additive combination of red and green, it is clearly perceived as being qualitatively different from each of the two components. Hering [79] had considerable success in explaining color perception in terms of an opponent-colors theory, 1 Note several of these distance measures are asymmetric and as such do not satisfy the mathematical requirements for a metric [22, p. 91]. which assumed the existence of neural signals of opposite kinds with the red green hues forming one opponent pair and the yellow blue hues constituting the other. Such a theory also satisfactorily explains both the existence of some intermediate hues (such as red yellow, yellow green, green blue, and blue red) and the absence of other intermediate hues (such as reddish-greens and yellowish-blues). Initially, the trichromatic theory and the opponent-colors theory were considered competitors for explaining color vision. However, neither one by itself was capable of giving satisfactory explanations of several important color vision phenomena. In more recent years, these competing theories have been combined in the form of zone theories of color vision, which assume that there are two separate but sequential zones in which these theories apply. Thus, in these theories it is postulated that the retinal color sensing mechanism is trichromatic, but an opponent-color encoding is employed in the neural pathways carrying the retinal responses to the brain. These theories of color vision have formed the basis of a number of color appearance models that attempt to explain psychophysical phenomena. Typically in the interests of simplicity, these models follow the theories only approximately and involve empirically determined parameters. The simplicity, however, allows their practical use in color reproduction applications involving different media where a perceptual match is more desirable and relevant than a colorimetric match. A somewhat different but widely publicized color vision theory was the retinex (from retina and cortex) theory of Edwin Land [80], [81]. Through a series of experiments, Land demonstrated that integrated broadband reflectances in red, green, and blue channels show a much stronger correlation with perceived color than the actual spectral composition of radiant light incident at the eye. He further postulated that the human visual system is able to infer the broadband reflectances from a scene through a successive comparison of spatially neighboring areas. As a model of human color perception, the retinex theory has received only limited attention in recent literature, and has been largely superseded by other theories that explain a wider range of psychophysical effects. However, a computational version of the theory has recently been used, with moderate success, in the enhancement of color images [82], [83]. One may note here that some of the uniform color spaces include some aspects of color constancy and color appearance in their definitions. In particular, both the CIELAB and CIELUV spaces employ an opponent-color encoding and use white-point normalizations that partly explain color constancy. However, the notion of a color appearance model is distinct from that of a uniform color space. Typical uniform color spaces are useful only for comparing stimuli under similar conditions of adaptation and can yield incorrect results if used for comparing stimuli under different adaptation conditions. 1) Chromatic Adaptation and Color Constancy: Several mechanisms of chromatic adaptation have been proposed to explain the phenomenon of color constancy. Perhaps the most widely used of these in imaging applications is one proposed by Von Kries [84]. He hypothesized that the

10 910 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 6, NO. 7, JULY 1997 chromatic adaptation is achieved through individual adaptive gain control on each of the three cone responses. Thus, instead of (2), a more complete model represents the cone responses as (21) where is a diagonal matrix corresponding to the gains of the three channels, and the other terms are as before. The gains of the three channels depend on the state of adaptation of the eye, which is determined by preexposed stimuli and the surround, but independent of the test stimulus. This is known as the Von Kries coefficient rule. The term asymmetric-matching is used to describe matching of color stimuli under different adaptation conditions. Using the Von Kries coefficient rule, two radiant spectra, and, viewed under adaptation conditions specified by the diagonal matrices, and, respectively, will match if (22) Thus, under the Von Kries coefficient rule, chromatic adaptation can be modeled as a diagonal transformation for tristimuli specified in terms of the eye s cone responses. Usually, tristimulus values are specified not relative to the cone responses themselves, but to CMF s that are linear transformations of the cone responses. In this case, it can readily be seen [12, p. 432] that the tristimuli of color stimuli that are in an asymmetric color match are related by a similarity transformation [85] of the diagonal matrix. A Von Kries transformation is commonly used in color rendering applications because of its simplicity and is a part of several standards for device-independent color imaging [86], [87]. Typically, the diagonal matrix is determined by assuming that the cone responses on either side of (22) are identical for white stimuli (usually a perfect reflector illuminated by the illuminant under consideration). The whitepoint normalization in CIELAB space was primarily motivated by such a model. Since the CIE XYZ CMF s are not per se the cone responses of the eye, the diagonal transformation representing the normalization is not a Von Kries transformation and was chosen more for convenience than accuracy [88]. In actual practice, the Von Kries transformation can explain results obtained from psychophysical experiments only approximately [12, pp ]. At the same time, the constancy of metameric matches under different adaptation conditions provides strong evidence for the fact that the cone response curves vary only in scale while preserving the same shape [89, p. 15]. Therefore, it seems most likely that part of the adaptation lies in the nonlinear processing of the cone responses in the neural pathways leading to the brain. A number of alternatives to the Von Kries adaptation rule have been proposed to obtain better agreement with experimental observations. Most of these are nonlinear and use additional information that is often unavailable in imaging applications. A discussion of these is beyond the scope of this paper, and the reader is referred to [60, pp. 81, 217], [90] [92], and [88] for examples of such models. The phenomenon of color constancy suggests that the human visual system transforms recorded stimuli into representations of the scene reflectance that are (largely) independent of Fig. 7. Typical wiring diagram for human color vision models (adapted from [99]). the viewing illuminant. Several researchers have investigated algorithms for estimating illuminant-independent descriptors of reflectance spectra from recorded tristimuli, which have come to be known as computational color constancy algorithms [93] [97]. Several of these algorithms rely on lowdimensional linear models of object and illuminant spectra, which will be discussed briefly in Section III-B5. A discussion of how these algorithms relate to the Von Kries transformation rule and to human color vision can also be found in [98], [95], and [97]. 2) Opponent Processes Theory and Color Appearance Models: The modeling of chromatic adaptation is just one part of the overall goal of color appearance modeling. While color appearance models are empirically determined, they are usually based on physiological models of color vision. Most modern color vision models are based on wiring diagrams of the type shown in Fig. 7. The front end of the model consists of L, M, and S (long, medium, and short wavelength sensitive) cones. The cone responses undergo nonlinear transformations and are combined into two opponent color chromatic channels (R-G and Y-B), and one achromatic channel (A). A positive signal in the R-G channel is an indication of redness whereas a negative signal indicates greenness. Similarly, yellowness and blueness are opposed in the Y-B channel. The outputs of these channels combine to determine the perceptual attributes of hue, saturation, and brightness. It is obvious that the above color-vision model is an over simplification. Actual color appearance models are considerably more intricate and involve a much larger number of parameters, with mechanisms to account for spatial effects of surround and the adaptation of the cone responses, which was briefly discussed in the last section. Due to the immense

11 SHARMA AND TRUSSELL: DIGITAL COLOR IMAGING 911 practical importance of color appearance modeling to color reproduction systems, there has been considerable research in this area that cannot be readily summarized here. The interested reader is referred to [60, pp ] and [100] [112] for examples of some of the prominent color appearance models in current literature. A recent overview of the current understanding of human color vision can also be found in [113]. III. COLOR REPRODUCTION AND RECORDING SYSTEMS The basics of color discussed in the last section addressed the issue of specification of a single color stimulus. In practical systems, one is usually concerned with the processing of color images with a large number of colors. In the physical world, these images exist as spatially varying spectral radiance or reflectance distributions. Color information needs to be recorded from these distributions before any processing can be attempted. Conversely, the physical realization of color images from recorded information requires synthesis of spatially varying spectral radiance or reflectance distributions. In this section, some of the common color output and input systems are surveyed. Output systems are discussed first because color recording systems may also be used to record color reproductions and may exploit the characteristics of the reproduction device. A. Color Output Systems Nature provides a variety of mechanisms by which color may be produced. As many as fifteen distinct physical mechanisms have been identified that are responsible for color in nature [114]. While only a fraction of these mechanisms is suitable for technological exploitation, there is still considerable diversity in available technologies and devices for displaying and printing color images. Color output devices can broadly be classified into three types: additive, subtractive, and hybrid. Additive color systems produce color through the combination of differently colored lights, known as primaries. The qualifier additive is used to signify the fact that the final spectrum is the sum (or average) of the spectra of the individual lights, as was assumed in the discussion of color matching in Section II-A1. Examples of additive color systems include color CRT displays and projection video systems. Color in subtractive systems is produced through a process of removing (subtracting) unwanted spectral components from white light. Typically, such systems produce color on transparent or reflective media, which are illuminated by white light for viewing. Dye sublimation printers, color photographic prints, and color slides are representatives of the subtractive process. Hybrid systems use a combination of additive and subtractive processes to produce color. The main use of a hybrid system is in color halftone printing, which is commonly used for lithographic printing and in most desktop color printers. Any practical output system is capable of producing only a limited range of colors. The range of producible colors on a device is referred to as its gamut. The gamut of a device is a 3-D object and can be visualized using a 3-D representation of the color space, such as the colorimetry standards or uniform color spaces discussed earlier [115], [116]. Often, 2-D representations are more convenient for display, and chromaticity diagrams are used for this purpose. From the linearity of color matching, it can be readily seen that the gamut of additive systems in CIE XYZ space (or any of the other linear tristimulus spaces) is the convex polyhedron formed by linear combinations of the color tristimuli of the primaries over the realizable amplitude range. On the CIE xy chromaticity diagram, the gamut appears as a convex polygon with the primaries representing the vertices. For the usual case of three red, green, and blue primaries, the gamut appears as a triangle on the CIE xy chromaticity diagram. Since most subtractive and hybrid systems are nonlinear, their gamuts have irregular shape and are not characterized by such elegant geometric constructs. One may note here that in order to obtain the largest possible chromaticity gamut, most three-primary additive systems use red, green, and blue colored primaries. For the same reason, cyan, magenta, and yellow primaries are used in subtractive and hybrid systems. In order to discuss colorimetric reproduction on color output devices, it is useful to introduce some terminology. The term control values is used to denote signals that drive a device. The operation of the device can be represented as a multidimensional mapping from control values to colors specified in a device-independent color space. This mapping is referred to as the (device) characterization. Since specified colors in a device-independent color space need to be mapped to device control values to obtain colorimetric output, it is necessary to determine the inverse of the multidimensional devicecharacterization function. In this paper, the term calibration will be used for the entire procedure of characterizing a device and determining the inverse transformation. If the device s operation can be accurately represented by a parametric model, the characterization is readily done by determining the model parameters from a few measurements. If no useful model exists, a purely empirical approach is necessary, in which the characterization function is directly measured over a grid of device control values. The inversion may be performed in a closed form if the characterization uses a device model that allows this. If an empirical approach is employed in characterization or if the model used is noninvertible (often the case with nonlinear models), one has to resort to numerical methods in the inversion step. 1) Cathode Ray Tubes: The most widely used display device for television and computer monitors is the color CRT. The CRT produces visible light by bombardment of a thin layer of phosphor material by an energetic beam of electrons. The electron beam causes the phosphor to fluoresce and emit light whose spectral characteristics are governed by the chemical nature of the phosphor. The most commonly used color CRT tubes are the shadow-mask type, in which a mosaic of red, green, and blue light emitting phosphors on a screen is illuminated by three independent electron beams. The intensity of light emitted by the phosphors is governed by the velocity and number of electrons. The beam is scanned across the screen by electrostatic or electromagnetic deflection mechanisms. The number of electrons is modulated