Color Models and Color Image Processing. CS 663, Ajit Rajwade

Transcription

1 Color Models and Color Image Processing CS 663 Ajit Rajwade

2 Pouring in color Grascale image: D arra of size M N containing scalar intensit values gralevels. Color image: tpicall represented as a 3D arra of size M N 3 again containing scalar values. But each piel location now has three values called as RredGgreen Bblue intensit values. All file formats store color images based on this representation.

3 Questions What are RGB? How are red green blue determined? Are there other was of representing color? How do ou distinguish between different intensities of the same color shades? Between varing levels of whiteness in a color tints?

4

5 More questions How would ou define an edge in a color image? How do ou smooth color images? What do we know about human color perception? Wh do we consider onl 3 channels i.e. RGB? Are there images with more channels? Where are the used?

6 Color perception and phsics Human perception of color is not full understood but there are some well understood phsical principles. The color of light is defined b its constituent wavelengths inverse of frequenc. The visible part of the electromagnetic spectrum lies between 450 nm violet to 700 nm red.

7 Ultraviolet Infrared Color violet blue green ellow orange red Wavelength nm nm nm nm nm nm

8 Color phsics White light is a blend of several wavelengths of light which get separated b dispersive elements such as prisms. Objects which reflect light that is balanced in several visible wavelengths appear white. Objects which reflect light in a narrow range of wavelengths appear colored eample: green objects reflect light between 500 to 560 nm. No color starts or ends abruptl at a particular wavelength the transitions are smooth.

9 Human color perception The human retina has two tpes of receptor cells that respond to light the rods and the cones. The rods work in the low-light regime and are responsible for monochromatic vision. The cones respond to brighter light and are responsible for color perception. There are around 5-7 million cones in a single retina.

10 Human color perception There are 3 tpes of cones. Each tpe responds differentl to light of different wavelengths: L responsive to long wavelengths i.e. red M medium wavelengths i.e. green and S short wavelengths i.e. blue. Yellow color: L is stimulated a bit more than M and S is not stimulated Red: L is stimulated much more than M and S is not stimulated Violet: S is stimulated M and L are not Response sensitivit functions for LMS cells Color-blindness: absence of one or more of the three tpes of cones

11 Human color perception Consider a beam of light striking the retina. Let its spectral intensit as a function of wavelength λ be given as Iλ. The three tpes of cone cells re-weigh the spectral intensit and produce the following response: S S S M M M L L L c I d c I a c I d c I a c I d c I a CI c c c S M L N L L L I I I a a a.. 1 C is a matri of size 3 N λ and I is a vector of N λ elements Vectors of N λ

12 Human color perception The colors RGB are called primar colors their corresponding wavelengths are nm respectivel. These values were standardized b CIE International Commission on Illumination Commission Internationale de l Eclairage.

13 Displa sstems CRT/LCD The interior of a cathode ra tube CRT contains an arra of triangular dot patterns triads containing electron-sensitive phosphor. Each dot in the triad produces light in one of the three primar colors based on the intensit of that primar color. Thus the three primar colors get mied in different proportions b the color sensitive cones of the human ee to perceive different colors. Though the electronics of an LCD sstem is different from CRT the color displa follows similar principles.

14 Color Models Color Spaces A purpose of color model is to serve as a method of representing color. Some color models are oriented towards hardware eg: monitors printers others for applications involving color manipulation. Monitors: RGB Printers: CMY human perception: HSI efficient compression and transmission: YCbCr.

15 RGB color model Defines a cartesian coordinate sstem for colors in terms of RGB aes. Images in the RGB color model consist of three component images one for each primar color. When an RGB image is given as input to a displa sstem the three images combine to produce the composite image on screen. Tpicall an 8 bit integer is used to represent the intensit value in each channel giving rise to ^8^3 = ^7 colors.

16

17 CMYK color space The colors can magenta and ellow are opponents of red green and blue respectivel i.e. can and red lie on diagonall opposite corners of the RGB cube i.e. C = 55-R M = 55-G Y = 55-B Can magenta and ellow are called secondar colors of light or primar colors of pigments. A can colored surface illuminated with white light will not allow the reflection of red light. Likewise for magneta and green ellow and blue. CMY are the colors of ink pigments used in printing industr. Color printing is a subtractive process the ink subtracts certain color components from white light. For purposes of displa white is the full combination of RGB and black is the absence of light. For purposes of printing white is the absence of an printing and black is the full combination of CMY.

18 CMYK color space The printer puts down dots of different sizes shapes of CMY colors with tin spacing of different widths in between. The spacing is so tin that our ee perceives them as a single solid color optical illusion!. This process is called color half-toning. While black is a full combination of CMY it is printed on paper using a separate black-color ink to save costs. This is the K of the CMYK model.

19 Color half-toning Three eamples of color half-toning with CMYK separations. From left to right: The can separation the magenta separation the ellow separation the black separation the combined halftone pattern and finall how the human ee would observe the combined halftone pattern from a sufficient distance.

20 Digression: Gra-scale half-toning Left: Halftone dots. Right: How the human ee would see this sort of arrangement from a sufficient distance.

21 Digression: negative after-images!

22 HSI color space RGB CMY are not intuitive from the point of view of human perception/description. We don t think naturall of colors in the form of combinations of RGB. We tend of think of color as the following components: hue the inherent/pure color red orange purple etc. saturation the amount of white mied in the color i.e. pink versus magenta intensit the amount of black mied in the color i.e. dark red versus bright red.

23 Intensit increases as we move from black to white on the intensit line. Consider a plane perpendicular to the intensit line in 3D. Saturation of a color increases as we move on that plane awa from the point where the plane and the intensit line intersect. How to determine hue? Pick an point e.g. ellow in the RGB cube and draw a triangle connecting that point with the white point and black point. All points inside or on this triangle have the same hue. An such point would be a color corresponding to a conve combination of ellow black and white i.e. of the form a ellow + b black + c red where a b c are non-negative and sum to 1. B rotating this triangle about the intensit ais ou will get different hues.

24 B rotating the triangle about the intensit ais ou will get different hues. In fact hue is an ANGULAR quantit ranging from 0 to 360 degrees. B convention red is considered 0 degrees. HSI space Primar colors are separated b 10 degrees. The secondar colors of light are 60 degrees awa from the primar colors.

25 To be ver accurate this HSI spindle is actuall heagonal. But it is approimated as a circular spindle for convenience. This approimation does not alter the notion of hue or intensit and has an insignificant effect on the saturation.

26 RGB to HSI conversion Conversion formulae are obtained b making the preceding geometric intuition more precise: 3 3min 1 if if hue ] 0.5[ cos 1 B G R I B G R B G R S G B G B h B G B R G R B R G R Refer to tetbook for formulae to convert back from HSI to RGB

27 HSI and RGB

28 Practical use of hue G B G B h B G B R G R B R G R if if hue ] 0.5[ cos 1 Hue is invariant to: Scaling of RGB Constant offsets added to RGB What does this mean phsicall?

29 Practical use of hue To understand this we need to understand a model which tells ou the what color is observed at a particular point on a surface of an object illuminated b one or more light sources. This color is given b: I C I C ambient I C diffuse I C specular C { R G B} Ambient light sa due to sunlight: constant effect on all points of the object s surface Diffuse reflection of light from a directed source off a rough surface: varies from point to point on a surface Reflection from shin surface: varies from point to point on a surface

30 Diffuse reflection from an irregular surface Specular reflection

31 Diffuse reflection from a rough surface: diffuse means that incident light is reflected in all directions. Specular reflection: part of the surface acts like a mirror the incident light is reflected onl in particular directions Diffuse reflection Diffuse + specular reflection I C k a I a I C ambient k C d I C diffuse I L nˆ sˆ k C specular s C L rˆ vˆ { R G B} L=Strength of white light source k a k d k s : surface reflectivit fraction of incident light that is reflected off the surface Vector normal to the surface at a point Lighting direction For shin surfaces α is large. Viewing direction Direction of reflected light vˆ rˆ nˆ ŝ

32 Practical use of hue The ambient and specular components are assumed to be the same across RGB neutral reflection model. So the get subtracted out when computing R-GG-BB-R. Hence hue is invariant to specular reflection! Notice: hue is independent of strength of lighting wh? lighting direction wh? and viewing direction wh?. This makes hue useful in object detection and object recognition or in applications such as detection of faces/foliage in color images. Hue is thus said to be an illumination invariant feature.

33 Food for thought We ve heaped praises on hue all along. An ideas on its demerits? Suppose we define the following quantities rgb [the chromaticit vector] derived from RGB: r R G g b R G B R G B Is the chromaticit vector also an illumination invariant feature? How does it compare to hue? R B G B

34 Digression: Plaing with color: seeing is not! believing /a_0_p_vis/a_0_p_vis.html#

35 Operations on color images Color image histogram equalization Color image filtering Color edge detection

36 Histogram equalization Method 1: perform histogram equalization on RGB channels separatel. Method : Convert RGB to HSI histogram equalize the intensit convert back to RGB. Method 1 ma cause alterations in the hue which is undesirable. Method will change onl the intensit leaving hue and saturation unaltered. It is the preferred method.

37 Top row: original images Middle row: histogram equalization channel b channel Bottom row: histogram equalization on intensit of HSI and conversion back to RGB

38 Color image smoothing: bilateral filtering Remember the bilateral filter HW: an edgepreserving filter for grascale images. It smoothes the image based on local weighted combinations driven b difference between spatial coordinates and intensit values. N I j i I j i j w i j w i j i w j i I I I s N j i N j i neighborhood usuall square centeredat small ] [ ep

39 Bilateral filtering for color images You can filter each channel separatel i.e. } { neighborhood usuall square centeredat small ] [ ep B G R C N I j i I j i j i w j i w j i w j i I I I C C s C N j i C N j i C C C

40 Bilateral filtering for color images Or ou can filter the three channels in a coupled fashion i.e. the smoothing weights are same for all three channels and the are derived using information from all three channels. N I j i I j i j i w j i w j i w j i I I I B R G C C C s N j i N j i C C neighborhood usuall square centeredat small ] [ ep } {

41 What s wrong with separate channel bilateral filtering? Separate channel Channel b channel: Color artifacts around edges. RGB channels are highl inter-dependent ou shouldn t treat them as independent. Coupled

42

43

44 More eamples: see figure 4.3 of

45 Color Edges A color RGB image will have three gradient vectors one for each channel. We could compute edges separatel for each channel. Option: Combine add channel-per-channel edges together to get a composite edge image. Not a good one? Wh see net slide

46 Color Edges Problem: the two circled points have the same edge strength mathematicall though one appears to be a stronger edge. RGB = RGB = 000 R^ + G^ +B^ + R^+G^+B^ = 3*55^ RGB = RGB = 0055 R^ + G^ +B^ + R^+G^+B^ = 3*55^

47 Color Edge We want to ask the question: along which direction in XY space is the total magnitude of change in intensit the maimum? The squared change in intensit in a direction cos ϴ sin ϴ is given b square of the directional derivative of the intensit: We want to maimize this w.r.t ϴ. Take derivative with respect to ϴ and set it to zero. cos sin sin cos sin cos sin cos sin cos B B G G R R B G R B G R B B G G R R E

48 Color Edge This gives the color gradient direction which makes an angle ϴ w.r.t. the X ais given b: For a grascale image this turns out to be tan 1 1 B G R B G R B B G G R R I I I I I I I I I I tan 1 tan 1 tan 1

49 Color Edge Consider It turns out that the ϴ i.e. the color gradient we derived is given b the eigenvector of this matri corresponding to the larger eigenvalue. The direction perpendicular to it i.e. the eigenvector corresponding to the smaller eigenvalue is the color edge. sin cos sin cos cos sin sin cos B G R B B G G R R B B G G R R B G R B B G G R R B G R B G R E Local color gradient matri

50

51 The YCbCr color space

52 RGB and correlation coefficient The RGB space is inefficient from the point of view of image compression or transmission. This is because there is high correlation between the RGB values at corresponding piels. This is measured b the correlation coefficient which is given as follows: N i i i N i N i i i N i i i N r

53 RGB and correlation coefficient This is measured b the correlation coefficient which is given as follows: The values of r lie from -1 to 1. A high absolute value of r indicates high positive or negative correlation and a low value close to 0 indicates low correlation. N i i i N i N i i i N i i i N r

54 RGB and correlation coefficient For the following images the values of rrg rgb and rbr were all around 0.9.

55 PCA on RGB values Suppose ou take N color images and etract RGB values of each piel 3 1 vector at each location. Now suppose ou build an eigen-space out of this ou get 3 eigenvectors each corresponding to 3 different eigenvalues. The eigen-coefficients are said to be decorrelated!

56 PCA on RGB values The eigen-coefficients are said to be decorrelated! Wh? Because if the correlation matri of the RGB values if C then the correlation matri of the eigen-coefficients is V T CV which is a diagonal matri.

57 R channel B channel G channel

58 Image containing eigencoefficient value corresponding to 1st eigenvector with maimum eigenvalue

59 Image containing eigencoefficient value corresponding to nd eigenvector with second largest eigenvalue

60 Image containing eigencoefficient value corresponding to 3rd eigenvector with least eigenvalue The variances of the three eigen-coefficient values:

61 YCbCr color space The YCbCr color space is a similarl decorrelated color space with Y being the luminance channel similar to the V in HSV. And Cb Cr being the two chrominance channels. Y gives intensit information and the color information lies in Cb and Cr.

62 YCbCr color space The RGB to YCbCr conversion is given as follows: The YCbCr to RGB conversion is as follows:

63 YCbCr color space The luminance channel Y carries most information from the point of view of human perception and the human ee is less sensitive to changes in chrominance. This fact can be used to assign coarser quantization levels i.e. fewer bits for storing or transmitting Cb and Cr values as compared to the Y channel. This improves the compression rate without significant loss in perceptual qualit. The JPEG standard for color image compression uses the YCbCr format. For an image of size M N 3 it stores Y with full resolution i.e. as an M N image and Cb and Cr with 5% resolution i.e. as M/ N/ images.

64 Y channel Cr channel Cb channel The correlation coefficients between Y and Cr and between Y and Cr are around 0.1 for this image. The Cb and Cr correlation coefficient is around -0.4.

65 Where do the formulae for YCbCr emerge from? From another color space used earlier called as the YUV space given as follows: Y U V 0.3R 0.6G 0. 1B B Y R Y Here U and V are the chroma or chrominance components and Y is the luma component.

66 Where do the formulae for YCbCr emerge from? The formula for Y was obtained b pschovisual eperiments that estimated the amount of red green and blue that human users perceive this is proportional to the percentage of red green and blue cones in the retina which are around 33% 65% and % respectivel. But the blue cones are the most sensitive.

67 Where do the formulae for YCbCr emerge from? For RGB values in the [01] range the value of U lies in [ ] and the value of V lies in [ ]. In the YCbCr scheme the Cb and Cr values are scaled and shifted versions of U and V respectivel given as follows: Cb U Cr V / 0.5 /

68 Beond color: Hperspectral images Hperspectral images are images of the form M N L where L is the number of channels. L can range from 30 to or more. Finer division of wavelengths than possible in RGB! Can contain wavelengths in the infrared or ultraviolet regime.

69 Sources of confusion Hperspectral images are abbreviated as HSI! Hperspectral images are different from multispectral images. The latter contain few discrete and discontinuous wavelengths. The former contain man more wavelengths with continuit.

70 Beond color: Hperspectral images Widel used in remote sensing satellite images often different materials/geographical entities soil water vegetation concrete landmines mountains etc. can be detected/classified b spectral properties. Also used in chemistr pharmaceutical industr and patholog for classification of materials/tissues.

71 Eample multispectral image with 6 bands

72

73

74 Reference color image

75 Color Image Acquisition: Mosaicing and demosaicing In grascale image acquisition the image is stored on a CCD arra. One would imagine there would need to be three such CCD arras of equal size for RGB color images. But CCD arras are epensive! And there would be spatial alignment issues in the R G B channel images due to difference in location of the RGB sensors. So this is not followed in practice.

76 Color Image Acquisition: Mosaicing and demosaicing Instead at an piel onl one out of the RGB values is stored. This is accomplished in hardware b means of a color filter arra CFA.

77 Color Filter Arras A CFA is an arra of tin color filters placed before the image sensor arra of a camera. The resolution of this arra is the same as that of the image sensor arra. Each color filter ma allow a different wavelength of light to pass this is predetermined during the camera design.

78 Color Filter Arras The most common tpe of CFA is the Baer pattern which is shown below: The Baer pattern collects information at red green blue wavelengths onl in a repeated pattern shown above.

79 *The word mosaic or mosaiced is not to be confused with image panorama generation which is also called image mosaicing. Color Filter Arras The Baer pattern uses twice the number of green elements as compared to red or blue elements. This is because both the M and L cone cells of the retina are sensitive to green light. The raw uncompressed output of the Baer pattern is called as the Baer pattern image or the mosaiced * image. The mosaiced image needs to be converted to a normal RGB image b a process called color image demosaicing.

80 original scene /wiki/baer_filter Mosaiced image Mosaiced image just coded with the Baer filter colors Demosaiced image obtained b interpolating the missing color values at all the piels

81 A Demosaicing Algorithm There eist a plethora of demosaicing algorithms. We will stud one that uses the bilateral filter.

82 A Demosaicing Algorithm Wh not just use simple linear interpolation to fill in missing RGB values? It produces a color fringe artifact along edges. See figures and 11 of the article Demosaicing methods for Baer color arras b Ramanath et al 00 Journal of Electronic Imaging

83 A Demosaicing Algorithm: bilateral filter Assumption: The values in the R G B channels are highl correlated with one another. Make use of this to interpolate missing values in an of the channels Approach: To estimate green values at a red/blue piel compute gradient for the green channel in horizontal or vertical directions. Interpolate the green color using piels along the direction with the smaller gradient i.e. along the edge. To estimate red values at a green piel use a bilateral filter with spatial weights defined as usual and intensit weights using the differences between the interpolated green values red and blue values not used as the are not available.

84 A Demosaicing Algorithm: bilateral filter Assumption: The values in the R G B channels are highl correlated with one another. Make use of this to interpolate missing values in an of the channels For results and description of the approach see the following paper: Ramanath and Snder Adaptive demosaicking Journal of Electronic Imaging 003

85 Mosaicing a broader perspective Mosaicing is an eample of acquiring a signal in compressed format. Wh? Because the underling RGB image with N piels has some 3N values but onl N values are measured b the camera. A software routine demosaicing algorithm then interpolates the remaining values.

86 Mosaicing a broader perspective This can be written in the form: Φ R m R n ΦR m n m n Recovering from and Φ is an ill-posed problem as the number of knowns is less than the number of unknowns.

87 Mosaicing a broader perspective But there is some theor that can actuall be recovered without error if is a sparse vector and if Φ obes certain properties. This is called the theor of compressed sensing and is a ver active area of research in signal and image processing we will cover this theor at the end of the computer vision course. The Φ matri in case of the Baer pattern does not satisf the required properties however.