Color. Some slides are adopted from William T. Freeman

Color Some slides are adopted from William T. Freeman 1 1

Why Study Color Color is important to many visual tasks To find fruits in foliage To find people s skin (whether a person looks healthy) To group parts of one object together in a scene To distinguish material changes from shading changes Human eyes can discern thousands of colors but only about two dozens of shades of gray 2

3 Do you really know color?

4 Color is cyclic?

5

Why no purple in rainbow? Purple does not exist in nature. 6

Why no brown in rainbow? Brown does not exist in nature. 7

Spectrum of Visible Light Visible spectrum: approx. 400 ~ 700 nm Visible colors: ROYGBIV with IR and UV at the two extremes (excluding) Isaac Newton Fig: Color spectrum seen by passing white light through a prism (As of Gonzalez 2e Page 283) Fig: Wavelengths comprising the visible range of electromagnetic spectrum (As of Gonzalez 3e Page 284) 8

Spectrum of Reflected Light Color of objects depends on the spectrum of light reflected from it This spectrum depends on both the illumination and the surface spectral computation involves a wavelength-by-wavelength multiplication of relative energies.* = Foundations of Vision, by Brian Wandell, Sinauer Assoc., 1995 9

the spectra of light sources Blue sky Tungsten light bulb Foundations of Vision, by Brian Wandell, Sinauer Assoc., 1995 10

the spectral albedo of surfaces Spectral albedo for several different leaves, with color names attached. Notice that different colors typically have different spectral albedo, but that different spectral albedoes may result in the same perceived color (compare the two whites). 11 Computer Vision A Modern Approach, Forsyth and Ponce, 2002

How do We Perceive Color? Cones are the cells in our eyes to sense color. There are three types of cones that absorb red, green and blue. The sensation of a certain color is produced due to the mixed response of these three types of cones in a certain proportion. Each cone is sensitive to a wide range of wavelength Foundations of Vision, by Brian Wandell, Sinauer Assoc., 1995 12 Fig: Absorption of light by red, green and blue cones in the human eye as a function of wavelength (As of Gonzalez 2e Page 285)

Cone and Rod 13

primary colors primary color of lights primary color of pigments blue green red 14 400 500 600 700 nm 400 500 600 700 nm 400 500 600 700 nm cyan magenta yellow minus red 400 500 600 700 nm minus green 400 500 600 700 nm minus blue 400 500 600 700 nm

additive color mixing red green 400 500 600 700 nm When colors combine by adding the color spectra. Example color displays that follow this mixing rule: CRT phosphors, multiple projectors aimed at a screen 400 500 600 700 nm Red and green make yellow 400 500 600 700 nm 15 Fig: Primary & Secondary colors of light & pigments (As of Gonzalez 2e Page 286)

16

subtractive color mixing cyan 400 500 600 700 nm When colors combine by multiplying the color spectra. Examples that follow this mixing rule: most photographic films, paint, cascaded optical filters. yellow 400 500 600 700 nm Cyan and yellow make green 17 400 500 600 700 nm Fig: Primary & Secondary colors of light & pigments (As of Gonzalez 2e Page 286)

How do Cameras Capture Color? Each pixel captures red, green and blue intensities A color is represented as a [R,G,B] three vector Example sensitivity of real cameras 18 Grasshopper camera GRAS-14S5C from PointGray

Summary of Color Imaging spectra camera sensitivity reflected signal 19

Questions? 20

Multispectral Imaging reflected signal Record the raw reflected signal (record the intensity at all wavelength) 21

Multispectral Imaging ICCV 2009 22

23 slides from Stephen Lin, ICCV 2009

24 slides from Stephen Lin, ICCV 2009

25 slides from Stephen Lin, ICCV 2009

26 slides from Stephen Lin, ICCV 2009

27 slides from Stephen Lin, ICCV 2009

28 slides from Stephen Lin, ICCV 2009

29 slides from Stephen Lin, ICCV 2009

Why Multispectral Imaging? Additional information to solve difficult vision problems (a good example of computational photography) Examples 1: distinct real and printed skin Example 2: see throw inks 610nm 30 645nm

Questions? 31

Why & How to Specify Color Numerically? Accurate color is commercially valuable Many products are identified by color ( golden arches of McDonald's, the orange NUS logo); Few color names in natural languages About 10 English words; other languages have fewer/ more, but not many more. It s common to disagree on appropriate color names. Color reproduction problems increased by prevalence of digital imaging and printing How to ensure a color look the same on different media (e.g. monitors, phones and printers)? How to specify color numerically? Mix primary colors to represent other colors How to decide the mixture coefficients?? 32

Color Matching Color matching: use three primary colors to represent other colors 33 Foundations of Vision, by Brian Wandell, Sinauer Assoc., 1995

34 color matching experiment 1

color matching experiment 1 35 p 1 p 2 p 3

color matching experiment 1 36 p 1 p 2 p 3

color matching experiment 1 The primary color amounts needed for a match 37 p 1 p 2 p 3

38 color matching experiment 2

color matching experiment 2 39 p 1 p 2 p 3

color matching experiment 2 40 p 1 p 2 p 3

color matching experiment 2 We say a negative amount of p 2 was needed to make the match, because we added it to the test color s side. The primary color amounts needed for a match: p 1 p 2 p 3 p 1 p 2 p 3 41 p 1 p 2 p 3

Questions? 42

Color Computation Problems Compute the color match (mixture weight) for any light signal for any specified primary colors Input: a light spectrum, three primary colors Output: mixing weights of your primaries that would give perceptual match to that light spectrum. Possible solution color matching for all visible light signals. Can we reduce the workload?? 43

color A color B Once we know the mixture coefficients of A and B, we can determine that of any color on the line segment AB 44 Foundations of Vision, by Brian Wandell, Sinauer Assoc., 1995

Grassman s Laws = means two colors are perceived the same by the color matching experiment, + means mixing two colors First, if : T a = w a1 P 1 + w a2 P 2 + w a3 P 3 T b = w b1 P 1 + w b2 P 2 + w b3 P 3 then T a + T b = (w a1 +w b1 ) P 1 + (w a2 +w b2 ) P 2 + (w a3 +w b3 ) P 3 Second, Finally, if : T a = w 1 P 1 + w 2 P 2 + w 3 P 3 T b = w 1 P 1 + w 2 P 2 + w 3 P 3 then T a = T b if : T a = w 1 P 1 + w 2 P 2 + w 3 P 3 then k T a = (kw 1 ) P 1 + (kw 2 ) P 2 + (kw 3 ) P 3 These statements are as true as any biological law. They mean that additive color matching is linear. 45

compute the color match for any color signal for any set of primary colors p 1 p 2 p 3 Examples of why you d want to do that: Want to match skin color of a person in a photograph printed on an ink jet printer to their true skin color. 46

measuring color matching functions Pick a set of primaries, p1( λ ), p2( λ ), p3( λ ) Measure the mixture coefficients c1 ( λ0), c2( λ0), c3( λ0) needed to match a monochromatic light, t( λ at each spectral wavelength 0 ) λ 0 c1 ( λ0), c2( λ0), c3( λ0) are called color matching functions they are functions of λ 0 47 λ 0 p 1 p 2 p 3

48 using color matching functions A monochromatic light of wavelength will be matched by mixing primaries according to λ i ) ( ), ( ), ( 3 2 1 i i i c c c λ λ λ + + = = 1 0 ) ( 0 1 ) ( ) ( ) ( 1 1 N N t t t t t λ λ λ λ Any spectral signal can be thought of as a linear combination of many monochromatic lights, with the linear coefficient given by the spectral power at each wavelength. According to Grassman s law, the weights of this signal is a combination of the weights of monochromatic lights.

Metamerism https://www.youtube.com/watch?v=5wpehiyuuzy 49

Internal Review So, for any set of primary colors, if we are given the spectral color matching functions for a set of primary lights, we can calculate the amounts of each primary needed to give a perceptual match to any spectral signal. 50

Common color representation 51

Color spaces: RGB Default color space 0,1,0 R (G=0,B=0) 1,0,0 0,0,1 RGB cube Easy for devices But not perceptual Where do the grays live? Where is hue and saturation? G (R=0,B=0) B (R=0,G=0) Image from: http://en.wikipedia.org/wiki/file:rgb_color_solid_cube.png

HSV Hue, Saturation, Value (Intensity) RGB cube on its vertex Decouples the three components (a bit) Use rgb2hsv() and hsv2rgb() in Matlab Slide by Steve Seitz

Color spaces: HSV Intuitive color space H (S=1,V=1) S (H=1,V=1) V (H=1,S=0)

55 Physical vs Biological

HSV Color Space Characteristics used to distinguish one color from another are: Brightness, Hue, and Saturation. Brightness embodies the chromatic notion of intensity. Hue is associated with the dominant wavelength in a mixture of light waves. It represents the dominant color as perceived by an observer. Saturation refers to the relative purity or the amount of white light mixed with hue. The pure spectrum colors are fully saturated. Color such as pink (red and white) is less saturated. The degree of saturation is inversely proportional to the amount of white light added. 56

conversion from RGB to HSV The following formulas show how to convert from RGB space to HSI: To convert from HSI to RGB, the process depends on which color sector H lies in. B = I(1 S) For the RG sector: o o S cos 120 H 240 R = I 1 + For the GB sector: cos 60 o o For the BR sector: 0 H 120 G = 3I ( R + B) 57 S = 1 H θ = 360 3 ( R + G + B) [ ] [( R G) + ( R B) ] o θ if if min( R, G, B) B G B > G o 240 V θ H 360 o = 1 3 = cos ( R + G + B) 1 B = I 1 + R = 3I H = H 240 G = I(1 S) 1 2 2 2 [ ] ( R G) + ( R B)( G B) 1 S cos H H cos( 60 ) ( G + B) H = H 120 R = I(1 S) G = I 1 + B = 3I H H ( ) S cos H H cos( 60 ) ( R + G)

CIE XYZ Color Space Since we can define colors using almost any set of primary colors, let s agree on a set of primaries and color matching functions for the world to use Commission Internationale d Eclairage, 1931 The color matching functions from RGB has negative coefficients p 1 = 645.2 nm p 2 = 525.3 nm p 3 = 444.4 nm 58

CIE XYZ Color Space Commission Internationale d Eclairage, 1931 as with any standards decision, there are some irratating aspects of the XYZ colormatching functions as well no set of physically realizable primary lights that by direct measurement will yield the color matching functions. Foundations of Vision, by Brian Wandell, Sinauer Assoc., 1995 CIE XYZ: Usually compute x, y, where x=x/(x+y+z) y=y/(x+y+z) Color matching functions are positive everywhere, but primary colors are imaginary. 59

CIE XYZ Color Space A qualitative rendering of the CIE (x,y) space. The blobby region represents visible colors. x=x/(x+y+z) y=y/(x+y+z) There are sets of (x, y) coordinates that don t represent real colors, because the primaries are not real lights (so that the color matching functions could be positive everywhere). 60 Fig: Chromaticity Diagram (As of Gonzalez 2e Page 288)

pure wavelength in chromaticity diagram The colors of mono-chromatic lights form the boundary of the visible light blob 61

CIE Color Space Think of X, Y, Z as coordinates Linear transform from srgb Generate all visible colors by positive coefficients Many points in XYZ do not correspond to visible colors! XYZ vs. RGB XYZ is rarely used for storage There are tons of flavors of RGB (srgb, Adobe RGB), all different matrices! XYZ is more standardized XYZ can reproduce all colors with positive values 62

Questions? 63

YUV Color Space The YUV color model defines a color space in terms of one luminance and two chrominance components. The YUV color model is used in the PAL, NTSC, and SECAM composite color video standards. 64

YUV Color Space The YUV color model defines a color space in terms of one luminance and two chrominance components. The YUV color model is used in the PAL, NTSC, and SECAM composite color video standards. 65

Color Spaces R G B V H S 66 Y U V

Questions? 67

Spatial Resolution and Color R G original B 68

blurring the G component original R G processed B 69

blurring the R component original R G processed B 70

blurring the B component original R G processed B 71

YUV Color Components Y U original V 72

blurring the Y Lab component original Y U processed V 73

blurring the U Lab component original Y U processed V 74

blurring the V Lab component original Y U processed V 75

Questions? 76

Capture Color Images Sergei Prokudin-Gorskii (1863-1944), took three monochrome pictures in sequence, each through a different colored filter. The original color scene can be reconstructed by projecting these three pictures using correctly colored light. 77 3-CCD cameras capture each channel of red, green and blue light by a separate CCD Light coming into lens is split by a prism High image quality, but complicated and expensive system

Capture Color Images Bayer Pattern Most modern digital cameras have a single CCD sensor with color filter array (CFA) applied. Each pixel record the intensity of either red, green or blue. The most popular filter array is the Bayer Pattern filter. Fig: RGB Bayer filter (named after its inventor, Dr. Bryce E. Bayer from Kodak), picture from wiki the captured picture the reference real picture 78 There are more green pixels than red and blue The missing RGB values will be interpolated from the captured ones. This process is known as Demosaicing.

79 Siggraph 2009

Use IR and UV Flashlight 80 Invisible flashlight will not disturb the captured subjects. Explore the correlation between the flash and non-flash image to denoise the ambient image. F: dark flash image; A: ambient image; R: denoise ambient image; L: long exposure image under ambient light as a reference.

Color Lines 81

Color Distribution of Real Images Pixel value varies because of surface color, illumination, noise, etc A good color representation should separate surface color changes from other sources so that we can tell if two points have similar color or not Popular models (YUV, Lab, etc) are linear transformations of the RGB space But pixels in real images form curved clusters 82

Color Distribution of Real Images Primary causes for non-linear color distribution Scene changes such as highlight Sensor saturation 83

Color Distribution of Real Images Primary causes for non-linear color distribution Scene changes such as highlight Sensor saturation Camera built-in processing Color blending 84

Color Lines Represent the color distribution by a set of lines A color line consists of: A set of skeleton points of the line A 2D Gaussian for each point in the plane perpendicular to the line Typically, an image needs about ~ 40 lines, each with ~ 6 points (and their 2D Gaussian) 85

Estimating Color Lines Slice the RGB space by hemispheres of equal distance Build 2D histgram for pixels distributed within two hemispheres Find local maximums Merge nearby maximums Fit a 2D Gaussian for each maximum Connect nearby maximum points in consecutive slices 86

Color Lines Applications Reasonable segmentation by grouping pixels of the same line together 87

Color Lines Applications Recolor objects by applying transformations to lines 88

Questions? 89

Recent Papers on Color Color Transfer (transfer the color of one image to another) Transferring Color to Greyscale Images, T. Welsh, M. Ashikhmin, K. Mueller, SIGGRAPH 2002 Colorization using Optimization, A. Levin, D. Lischinski, Y. Weiss, SIGGRAPH 2004 Semantic Colorization with Internet images, Y. Chia, S. Zhuo, R. Gupta, Y. Tai, S. Cho, P. Tan, S. Lin, Siggraph Asia 2011 Color2Gray: Salience-Preserving Color Removal, A. Gooch, S. Olsen, J. Tumblin, B. Gooch, SIGGRAPH 2005 Example-Based Image Color and Tone Style Enhancement, B. Wang, Y. Yu, Y.Q. Xu, SIGGRAPH 2011 Color Compatibility (which color combination is pretty) Color Harmonization, D. Cohen-Or, O. Sorkine, R. Gal, T. Leyvand Y.Q. Xu, SIGGRAPH 2006 Color Compatibility From Large Datasets, P. O Donovan, A. Agarwala, A. Hertzmann, SIGGRAPH 2011 90