CS 1699: Intro to Computer Vision. Color. Prof. Adriana Kovashka University of Pittsburgh September 22, 2015

Transcription

1 CS 1699: Intro to Computer Vision Color Prof. Adriana Kovashka University of Pittsburgh September 22, 2015

2 Today Review: SIFT features Physics and perception of color Color matching Color spaces Uses of color in computer vision

3 Announcement Homework 2 released 9/17 Small changes made 9/18 Homework 1 due tonight at 11:59pm Review late policy Reminder: Do not look for or use existing implementations

4 Harris Detector: Summary Compute image gradients Ix and Iy for all pixels For each pixel Compute by looping over neighbors x, y compute (k :empirical constant, k = ) Find points with large corner response function R (R > threshold) Take the points of locally maximum R as the detected feature points (i.e., pixels where R is bigger than for all the 4 or 8 neighbors). D. Frolova, D. Simakov 4

5 K. Grauman Example of Harris application

6 Local Descriptors: SIFT Descriptor [Lowe, ICCV 1999] K. Grauman, B. Leibe Histogram of oriented gradients Captures important texture information Robust to small translations / affine deformations

7 Computing gradients tan(α)= opposite side adjacent side

8 Gradients m(x, y) = sqrt(1 + 0) = 1 Θ(x, y) = atan(0/1) = 0

9 Gradients m(x, y) = sqrt(0 + 1) = 1 Θ(x, y) = atan(1/0) = 90

10 Gradients m(x, y) = sqrt(1 + 1) = 1.41 Θ(x, y) = atan(1/1) = 45

11 Scale Invariant Feature Transform Basic idea: Take 16x16 square window around detected feature Compute gradient orientation for each pixel Create histogram over edge orientations weighted by magnitude 0 2 angle histogram L. Zitnick, adapted from D. Lowe

12 SIFT descriptor Full version Divide the 16x16 window into a 4x4 grid of cells (2x2 case shown below) Compute an orientation histogram for each cell 16 cells * 8 orientations = 128 dimensional descriptor L. Zitnick, adapted from D. Lowe

13 SIFT descriptor Full version Divide the 16x16 window into a 4x4 grid of cells (2x2 case shown below) Compute an orientation histogram for each cell 16 cells * 8 orientations = 128 dimensional descriptor Threshold normalize the descriptor: such that: 0.2 L. Zitnick, adapted from D. Lowe

14 Making descriptor rotation invariant CSE 576: Computer Vision Rotate patch according to its dominant gradient orientation This puts the patches into a canonical orientation. K. Grauman Image from Matthew Brown

15 Examples of Using SIFT

16 Examples of Using SIFT

17 Today Review: SIFT features Physics and perception of color Color matching Color spaces Uses of color in computer vision

18 Color and light Color of light arriving at camera depends on Spectral reflectance of the surface light is leaving Spectral radiance of light falling on that patch Color perceived depends on Physics of light Visual system receptors Brain processing, environment K. Grauman

19 The Eye The human eye is a camera! Lens - changes shape by using ciliary muscles (to focus on objects at different distances) Pupil - the hole (aperture) whose size is controlled by the iris Iris - colored annulus with radial muscles Retina - photoreceptor cells Slide by Steve Seitz

20 Retina up-close Light D. Hoiem

21 Color Sensing in Cameras: Bayer Grid Estimate RGB at each cell from neighboring values Slide by Steve Seitz

22 Two types of light-sensitive receptors Cones cone-shaped less sensitive operate in high light color vision Rods rod-shaped highly sensitive operate at night gray-scale vision slower to respond Stephen E. Palmer, 2002 Slide Credit: Efros

23 Rod / Cone Sensitivity Slide Credit: Efros

24 . Distribution of Rods and Cones # Receptors/mm2 150, ,000 50, Rods 60 Cones 40 Fovea 20 0 Blind Spot Rods Cones Visual Angle (degrees from fovea) - Rods responsible for intensity - Cones responsible for color - Fovea: small region (1 or 2 ) at the center of the visual field containing the highest density of cones (and no rods). Less visual acuity in the periphery Night Sky: why are there more stars off-center? Adapted from A. Efros, K. Grauman, S. Seitz, P. Duygulu Stephen E. Palmer, 2002

25 Electromagnetic spectrum Human Luminance Sensitivity Function K. Grauman Image credit: nasa.gov

26 The Physics of Light Any patch of light can be completely described physically by its spectrum: the number of photons (per time unit) at each wavelength nm. # Photons (per ms.) Wavelength (nm.) Stephen E. Palmer, 2002

27 . The Physics of Light Some examples of the spectra of light sources A. Ruby Laser B. Gallium Phosphide Crystal Wavelength (nm.) D. Normal Daylight # Photons # Photons Wavelength (nm.) C. Tungsten Lightbulb # Photons # Photons Stephen E. Palmer, 2002

28 % Photons Reflected The Physics of Light Some examples of the reflectance spectra of surfaces Red Yellow Blue Purple Wavelength (nm) Stephen E. Palmer, 2002

29 . Physiology of Color Vision Three kinds of cones: nm. RELATIVE ABSORBANCE (%) 100 S M L WAVELENGTH (nm.) Why are M and L cones so close? Stephen E. Palmer, 2002

30 3 is better than 2 M and L on the X-chromosome Why men are more likely to be color blind (see what it s like: L has high variation, so some women are tetrachromatic Some animals have 1 (night animals), 2 (e.g., dogs), 4 (fish, birds), 5 (pigeons, some reptiles/amphibians), or even 12 (mantis shrimp) D. Hoiem

31 Human photoreceptors Possible evolutionary pressure for developing receptors for different wavelengths in primates Osorio & Vorobyev, 1996 K. Grauman

32 Measuring spectra Spectroradiometer: separate input light into its different wavelengths, and measure the energy at each. K. Grauman Foundations of Vision, B. Wandell

33 Metamers Spectral reflectances for some natural objects: how much of each wavelength is reflected for that surface K. Grauman Forsyth & Ponce, measurements by E. Koivisto

34 We don t perceive a spectrum (or even RGB) D. Hoiem We perceive Hue: mean wavelength, color Saturation: variance, vividness Intensity: total amount of light Same perceived color can be recreated with combinations of three primary colors ( trichromacy )

35 Color mixing Cartoon spectra for color names: Source: W. Freeman

36 Additive color mixing Colors combine by adding color spectra Light adds to black. Source: W. Freeman

37 Examples of additive color systems CRT phosphors multiple projectors K. Grauman

38 Subtractive color mixing Colors combine by multiplying color spectra. Pigments remove color from incident light (white). Source: W. Freeman

39 Examples of subtractive color systems Printing on paper Crayons Most photographic film K. Grauman

40 Fun with color!

41 Chromatic adaptation K. Grauman

42 Chromatic adaptation K. Grauman

43 Brightness perception Edward Adelson K. Grauman

44 Edward Adelson K. Grauman

45 Edward Adelson K. Grauman

46 Color constancy Interpret surface in terms of true color, rather than observed intensity Humans are good at it Computers are not nearly as good D. Hoiem

47 Look at blue squares Look at yellow squares Content 2008 R.Beau Lotto K. Grauman

48 Content 2008 R.Beau Lotto K. Grauman

49 Content 2008 R.Beau Lotto K. Grauman

50 Content 2008 R.Beau Lotto K. Grauman

51 Content 2008 R.Beau Lotto K. Grauman

52 Content 2008 R.Beau Lotto K. Grauman

53 Name that color High level interactions affect perception and processing. K. Grauman

54 Reasons for illusions Chromatic adaptation: we adapt to a particular illuminant Assimilation, contrast effects, chromatic induction: nearby colors affect what is perceived; receptor excitations interact across image and time Afterimages: tired receptors produce negative response Color matching ~= color appearance Physics of light ~= perception of light K. Grauman

55 Chromatic adaptation The visual system changes its sensitivity depending on the luminances prevailing in the visual field The exact mechanism is poorly understood Adapting to different brightness levels Changing the size of the iris opening (i.e., the aperture) changes the amount of light that can enter the eye Think of walking into a building from full sunshine Adapting to different color temperature The receptive cells on the retina change their sensitivity For example: if there is an increased amount of red light, the cells receptive to red decrease their sensitivity until the scene looks white again We actually adapt better in brighter scenes: This is why candlelit scenes still look yellow K. Grauman

56 Today Review: SIFT features Physics and perception of color Color matching Color spaces Uses of color in computer vision

57 Color matching experiments Goal: find out what spectral radiances produce same response in human observers K. Grauman

58 Color matching experiments Observer adjusts weight (intensity) for primary lights (fixed SPD s) to match appearance of test light. Foundations of Vision, by Brian Wandell, Sinauer Assoc., 1995 After Judd & Wyszecki.

59 Color matching experiments Goal: find out what spectral radiances produce same response in human observers Assumption: simple viewing conditions, where we say test light alone affects perception Ignoring additional factors for now like adaptation, complex surrounding scenes, etc. K. Grauman

60 Slide credit: W. Freeman Color matching experiment 1

61 Color matching experiment 1 Slide credit: W. Freeman p 1 p 2 p 3

62 Color matching experiment 1 Slide credit: W. Freeman p 1 p 2 p 3

63 Color matching experiment 1 The primary color amounts needed for a match Slide credit: W. Freeman p 1 p 2 p 3

64 Slide credit: W. Freeman Color matching experiment 2

65 Color matching experiment 2 Slide credit: W. Freeman p 1 p 2 p 3

66 Color matching experiment 2 Slide credit: W. Freeman p 1 p 2 p 3

67 Color matching experiment 2 We say a negative amount of p 2 was needed to make the match, because we added it to the test color s side. The primary color amounts needed for a match: p 1 p 2 p 3 p 1 p 2 p 3 p 1 p 2 p 3

68 Color matching What must we require of the primary lights chosen? How are three numbers enough to represent entire spectrum? K. Grauman

69 Trichromacy In color matching experiments, most people can match any given light with three primaries Primaries must be independent For the same light and same primaries, most people select the same weights Exception: color blindness Trichromatic color theory Three numbers seem to be sufficient for encoding color Dates back to 18 th century (Thomas Young) L. Lazebnik

70 K. Grauman Grassman s laws If two test lights can be matched with the same set of weights, then they match each other: Suppose A = u 1 P 1 + u 2 P 2 + u 3 P 3 and B = u 1 P 1 + u 2 P 2 + u 3 P 3. Then A = B. If we scale the test light, then the matches get scaled by the same amount: Suppose A = u 1 P 1 + u 2 P 2 + u 3 P 3. Then ka = (ku 1 ) P 1 + (ku 2 ) P 2 + (ku 3 ) P 3. If we mix two test lights, then mixing the matches will match the result (superposition): Suppose A = u 1 P 1 + u 2 P 2 + u 3 P 3 and B = v 1 P 1 + v 2 P 2 + v 3 P 3. Then A+B = (u 1 +v 1 ) P 1 + (u 2 +v 2 ) P 2 + (u 3 +v 3 ) P 3. Here = means matches.

71 Computing color matches How do we compute the weights that will yield a perceptual match for any test light using a given set of primaries? 1. Select primaries 2. Estimate their color matching functions: observer matches series of monochromatic lights, one at each wavelength ) ( ) ( ) ( ) ( ) ( ) ( N N N c c c c c c C K. Grauman

72 Computing color matches Color matching functions for a particular set of primaries p 1 = nm p 2 = nm p 3 = nm Rows of matrix C Foundations of Vision, by Brian Wandell, Sinauer Assoc., 1995 Slide credit: W. Freeman

73 Computing color matches i matches c ( i ), c2( i ), c3( 1 i ) K. Grauman Now have matching functions for all monochromatic light sources, so we know how to match a unit of each wavelength. Arbitrary new spectral signal is a linear combination of the monochromatic sources. t t t( 1 ) t( N )

74 Computing color matches So, given any set of primaries and their associated matching functions (C), we can compute weights (e) needed on each primary to give a perceptual match to any test light t (spectral signal). K. Grauman Fig from B. Wandell, 1996

75 Computing color matches Why is computing the color match for any color signal for a given set of primaries useful? Want to paint a carton of Kodak film with the Kodak yellow color. Want to match skin color of a person in a photograph printed on an ink jet printer to their true skin color. Want the colors in the world, on a monitor, and in a print format to all look the same. Adapted from W. Freeman Image credit: pbs.org

76 Today Review: SIFT features Physics and perception of color Color matching Color spaces Uses of color in computer vision

77 Why specify color numerically? Accurate color reproduction is commercially valuable Many products are identified by color ( golden arches) Few color names are widely recognized by English speakers 11: black, blue, brown, grey, green, orange, pink, purple, red, white, and yellow. Other languages have fewer/more. Common to disagree on appropriate color names. Color reproduction problems increased by prevalence of digital imaging e.g. digital libraries of art. How to ensure that everyone perceives the same color? Forsyth & Ponce

78 Standard color spaces Use a common set of primaries/color matching functions Linear color space examples RGB CIE XYZ Non-linear color space HSV K. Grauman

79 Linear color spaces Defined by a choice of three primaries The coordinates of a color are given by the weights of the primaries used to match it mixing two lights produces colors that lie along a straight line in color space mixing three lights produces colors that lie within the triangle they define in color space L. Lazebnik

80 Color spaces: RGB Default color space 0,1,0 R (G=0,B=0) 1,0,0 G (R=0,B=0) 0,0,1 Some drawbacks Strongly correlated channels Non-perceptual D. Hoiem B (R=0,G=0) Image from:

81 Trichromacy and CIE-XYZ Perceptual equivalents with RGB Perceptual equivalents with CIE-XYZ D. Hoiem

82 Color Space: CIE-XYZ D. Hoiem RGB portion is in triangle

83 Color Space: CIE-XYZ

84 D. Hoiem Perceptual uniformity

85 Distances in color space Are distances between points in a color space perceptually meaningful? K. Grauman

86 Distances in color space Not necessarily: CIE XYZ is not a uniform color space, so magnitude of differences in coordinates are poor indicator of color distance. K. Grauman McAdam ellipses: Just noticeable differences in color

87 Uniform color spaces Attempt to correct this limitation by remapping color space so that justnoticeable differences are contained by circles distances more perceptually meaningful. CIE XYZ Examples: CIE u v CIE Lab CIE u v K. Grauman

88 Color spaces: CIE L*a*b* Perceptually uniform color space L (a=0,b=0) a (L=65,b=0) Luminance = brightness Chrominance = color D. Hoiem b (L=65,a=0)

89 Color spaces: HSV Intuitive color space H (S=1,V=1) S (H=1,V=1) V (H=1,S=0) D. Hoiem

90 HSV color space Hue, Saturation, Value (Brightness) Nonlinear reflects topology of colors by coding hue as an angle Matlab: hsv2rgb, rgb2hsv. K. Grauman Image from mathworks.com

91 Today Review: SIFT features Physics and perception of color Color matching Color spaces Uses of color in computer vision

92 Pixel counts Color as a low-level cue for CBIR R G B Color intensity Color histograms: Use distribution of colors to describe image No spatial info invariant to translation, rotation, scale K. Grauman

93 Color as a low-level cue for CBIR R G B Given two histogram vectors, sum the minimum counts per bin: I( x, y) n i 1 min, xi yi = [1, 3, 5] = [2, 0, 3] [ 1, 0, 3 ] K. Grauman

94 Color-based image retrieval Given collection (database) of images: Extract and store one color histogram per image Given new query image: Extract its color histogram For each database image: Compute intersection between query histogram and database histogram Sort intersection values (highest score = most similar) Rank database items relative to query based on this sorted order K. Grauman

95 Color-based image retrieval Example database K. Grauman

96 Color-based image retrieval K. Grauman Example retrievals

97 Color-based image retrieval K. Grauman Example retrievals

98 Color-based image retrieval Color histograms for image matching K. Grauman Source: L. Lazebnik

99 Color-based skin detection K. Grauman M. Jones and J. Rehg, Statistical Color Models with Application to Skin Detection, IJCV 2002.

100 Color-based segmentation for robot soccer Towards Eliminating Manual Color Calibration at RoboCup. Mohan Sridharan and Peter Stone. RoboCup-2005: Robot Soccer World Cup IX, Springer Verlag, K. Grauman

101 Summary Color perception differs from the physics of color Various color spaces exist, with different strengths and weaknesses Color has limited application in computer vision

102 Next time Segmentation Clustering [Figure by J. Shi]