Capturing light and color

Transcription

1 Capturing light and color Friday, 10/02/2017 Antonis Argyros Szeliski 2.2, 2.3, 3.1 1

2 Recap from last lecture Pinhole camera model Perspective projection Focal length and depth/field of view Lenses

3 Recap - Projection matrix Slide Credit: Saverese R,T j w X k w x i w O w x= K[ R t] X x: Image Coordinates: (u,v,1) K: Intrinsic Matrix (3x3) R: Rotation (3x3) t: Translation (3x1) X: World Coordinates: (X,Y,Z,1)

4 Adding a lens circle of confusion A lens focuses light onto the film There is a specific distance at which objects are in focus other points project to a circle of confusion in the image Changing the shape of the lens changes this distance

5 The long trip of light Light source properties Sensor characteristics Exposure Surface shape Optics Surface reflectance properties 5 Slide by L. Fei-Fei

6 What we will discuss today Geometry Relations between 3D world points and their 2D projections on the image Radiometry The relation of emitted light to the light that reaches the camera Photometry How to measure the light that reaches the camera Digitization The transformation of the spatio-temporally continuous signal to digital representations 6

7 A photon s life choices Absorption Diffusion Reflection Transparency Refraction Fluorescence Subsurface scattering? λ light source

8 A photon s life choices Absorption Diffusion Reflection Transparency Refraction Fluorescence Subsurface scattering λ light source

9 A photon s life choices Absorption Diffuse Reflection Reflection Transparency Refraction Fluorescence Subsurface scattering λ light source

10 A photon s life choices Absorption Diffusion Specular Reflection Transparency Refraction Fluorescence Subsurface scattering λ light source

11 A photon s life choices Absorption Diffusion Reflection Transparency Refraction Fluorescence Subsurface scattering λ light source

12 A photon s life choices Absorption Diffusion Reflection Transparency Refraction Fluorescence Subsurface scattering λ light source

13 A photon s life choices Absorption Diffusion Reflection Transparency Refraction Fluorescence Subsurface scattering λ 2 λ 1 light source

14 A photon s life choices Absorption Diffusion Reflection Transparency Refraction Fluorescence Subsurface scattering λ light source

15 Radiometry What determines the value of an image pixel? Radiance (L): energy carried by a ray Irradiance (E): energy arriving at the sensor Sensor characteristics 15

16 The journey of the light ray X = E t E d = π 2 cos 4 α 4 z' L Z = f ( E t) Camera response function: the mapping f from irradiance to pixel values 16 Source: S. Seitz, P. Debevec

17 Lambertian Reflectance In computer vision, surfaces are often assumed to be ideal diffuse reflectors with known dependence on viewing direction. Lambert s law: Viewed brightness does not depend on viewing direction, but it does depend on direction of illumination

18 What is color? Color is the result of interaction between physical light in the environment and our visual system Color is a psychological property of our visual experiences when we look at objects and lights, not a physical property of those objects or lights (S. Palmer, Vision Science: Photons to Phenomenology) Wassily Kandinsky ( ), Murnau Street with Women,

19 Electromagnetic spectrum Human Luminance Sensitivity Function 19

20 Beyond the visible spectrum γ - rays Χ - rays UV IR Long radiowaves 20

21 The Physics of Light: light sources Any source of light can be completely described physically by its spectrum: the amount of energy emitted (per time unit) at each wavelength nm. Relative spectral power # Photons (per ms.) Wavelength (nm.) 21 Stephen E. Palmer, 2002

22 The Physics of Light: light sources Some examples of the spectra of light sources A. Ruby Laser B. Gallium Phosphide Crystal Wavelength (nm.) D. Normal Daylight # Rel. Photons power Rel. # Photons power Wavelength (nm.) C. Tungsten Lightbulb Rel. # Photons power Rel. # Photons power Stephen E. Palmer, 2002

23 The Physics of Light: reflectance spectra of surfaces Some examples of the reflectance spectra of surfaces % Light Reflected Red Yellow Blue Purple Wavelength (nm) Stephen E. Palmer, 2002

24 Interaction of light and surfaces Observed color is the result of interaction of light source spectrum with surface reflectance 24

25 The Eye The human eye is a camera (the best)! Iris - colored annulus with radial muscles Pupil - the hole (aperture) whose size is controlled by the iris Lens - changes shape by using ciliary muscles (to focus on objects at different distances) What s the film? photoreceptor cells (rods and cones) in the retina 25 Slide by Steve Seitz

26 Two types of light-sensitive receptors Cones cone-shaped less sensitive operate in high light color vision Rods rod-shaped highly sensitive operate at night gray-scale vision cone rod Stephen E. Palmer, 2002

27 Rod / Cone sensitivity

28 Density of rods and cones cone pigment molecules Rods and cones are non-uniformly distributed on the retina Rods responsible for intensity, cones responsible for color Fovea - Small region (1 or 2 ) at the center of the visual field containing the highest density of cones (and no rods). Less visual acuity in the periphery rod 28 Slide by Steve Seitz

29 Standardizing color experience We would like to understand which spectra produce the same color sensation from people under similar viewing conditions Color matching experiments Foundations of Vision, by Brian Wandell, Sinauer Assoc.,

30 Color matching experiment 30 Source: W. Freeman

31 Color matching experiment p 1 p 2 p 3 31 Source: W. Freeman

32 Color matching experiment p 1 p 2 p 3 32 Source: W. Freeman

33 Color matching experiment The primary color amounts needed for a match p 1 p 2 p 3 33 Source: W. Freeman

34 Trichromacy Three numbers seem to be sufficient for encoding color In color matching experiments, most people can match any given light with three primaries Exception: color blindness For the same light and same primaries, most people select the same weights 34

35 Grassman s Laws If two test lights can be matched with the same set of weights, then they are the same If we mix two test lights, then mixing the matches will match the result: Suppose A = u 1 P 1 + u 2 P 2 + u 3 P 3 and B = v 1 P 1 + v 2 P 2 + v 3 P 3. Then A+B = (u 1 +v 1 ) P 1 + (u 2 +v 2 ) P 2 + (u 3 +v 3 ) P 3. If we scale the test light, then the matches get scaled by the same amount: Suppose A = u 1 P 1 + u 2 P 2 + u 3 P 3. Then ka = (ku 1 ) P 1 + (ku 2 ) P 2 + (ku 3 ) P 3. 35

36 Color Sensing in Camera (RGB) 3-chip vs. 1-chip: quality vs. cost Slide by Steve Seitz

37 Practical Color Sensing: Bayer Grid Estimate RGB at G cells from neighboring values Slide by Steve Seitz

38 Color Image R G B

39 Images in Matlab Images represented as a matrix Suppose we have a NxM RGB image called im im(1,1,1) = top-left pixel value in R-channel im(y, x, b) = y pixels down, x pixels to right in the b th channel im(n, M, 3) = bottom-right pixel in B-channel imread(filename) returns a uint8 image (values 0 to 255) Convert to double format (values 0 to 1) with im2double row column G R B

40 Color spaces How can we represent color?

41 Color spaces: RGB Default color space 0,1,0 R (G=0,B=0) 1,0,0 0,0,1 G (R=0,B=0) Drawback: Non-perceptual: a change of the same amount in a color value, does not produce a change of about the same visual importance. B (R=0,G=0) Image from:

42 Color spaces: HSV Intuitive color space H (S=1,V=1) S (H=1,V=1) V (H=1,S=0)

43 Color spaces: YCbCr Y=0 Y=0.5 Y (Cb=0.5,Cr=0.5) Cr Cb Y=1 Cb (Y=0.5,Cr=0.5) Cr (Y=0.5,Cb=05)

44 Color spaces: L*a*b* L (a=0,b=0) a (L=65,b=0) b (L=65,a=0)

45 If you had to choose, would you rather go without luminance or chrominance?

46 If you had to choose, would you rather go without luminance or chrominance?

47 Most information in intensity Only color shown constant intensity

48 Most information in intensity Only intensity shown constant color

49 Most information in intensity Original image

50 Lightness constancy 50

51 Lightness constancy 51

52 Chromatic adaptation The visual system changes its sensitivity depending on the luminances prevailing in the visual field Adapting to different brightness levels Changing the size of the iris opening (i.e., the aperture) changes the amount of light that can enter the eye Think of walking into a building from full sunshine Adapting to different color temperature The receptive cells on the retina change their sensitivity For example: if there is an increased amount of red light, the cells receptive to red decrease their sensitivity until the scene looks white again 52

53 Name that color Πράσινο Κόκκινο Μπλέ Κίτρινο Πράσινο Κόκκινο Πορτοκαλί Μώβ Καφέ Κίτρινο Μαύρο 53

54 Digital camera A digital camera replaces film with a sensor array Each cell in the array is light-sensitive diode that converts photons to electrons Two common types Charge Coupled Device (CCD) Complementary metal oxide semiconductor (CMOS) 54 Slide by Steve Seitz

55 Digitization Let s digitize this image We will assume a square sampling grid We will be varying the sampling density 55

56 Digitization A sample per red point Sampling interval Coarse sampling: 14 lines, 20 points / line Denser sampling: 68 lines, 100 points/line 56

57 Consequences of the sampling interval/rate Look at the fence Sampling interval White image! Grey image!

58 Consequences of the sampling interval/rate Look at the fence Sampling interval Now the fence is visible!

59 Consequences of the sampling interval/rate Look at the repetitive pattern on the fence: Sampling intervals s' d s Case 1: s' = d Sampling interval equal to the period of the repetitive pattern Invisible fence Case 2: s = d/2 Sampling interval equal to half the period of the repetitive pattern Visible fence 59

60 Sampling theorem If the width of the thinnest structure is d, then the sampling interval should be at smaller than d/2 The repetitive pattern has a certain frequency. To maintain this frequency, sampling must be done with at least double this frequency (Nyquist rate) 60

61 Sampling Basic idea, in theory 23 Digitized image Continuous signal Dirac Delta Function 2D "Comb" δ(x,y) = 0 for x ǂ 0, y ǂ 0 s δ(x,y) dx dy = 1 f(x,y)δ(x-a,y-b) dx dy = f(a,b) 61

62 Sampling Basic idea, in practice... The ideal situation is not feasible in practice A sample corresponds to a region that has a certain area The value of a sample depends on the value of the continuous function over this area 23 s 62

63 as a result. The mixed pixel problem 63

64 Remember: Color sensing in camera: Color filter array Bayer grid Estimate missing components from neighboring values (demosaicing) 64 Source: Steve Seitz

65 Problem with demosaicing: color moire 65

66 Signal quantization Goal: Ok, we saw where to sample, what about the actual value measured at each sample? I(x,y) =.1583 volts =???? Digital value? 66

67 Quantization I(x,y) = continuous signal 0 I(x,y) M We need to quantize it in K levels/values 0,1,...K-1 K is usually a power of 2: K #Bits We need to define the mapping between the (analog) input signal to the (digital) output signal. Several such mappings (linear, logarithmic, etc) 67

68 E.g., linear mapping uantization Level K Signal Value M 68

69 Selection of K Continuous signal K=2 K=4 K=16 K=32 69

70 Selection of K K=2 (for each basic color) K=4 (for each basic color) 70

71 Loss during spatial sampling

72 Loss during quantization

73 Image histogram H The histogram of an image represents the number of pixels having a particular intensity / gray value 8 Pixel Count Image M x N I max i= I min H ( i) = MN Gray Value 73

74 Image histogram & image content

75 Image histogram vs image properties Dark image Bright image Low contrast High contrast 75

76 Point processing An image processing operator defines an operator that, when applied at an input image f results in a new image g. The simplest of such operators are those that processing is independent of position in the image Such operators are called point processing operators 76

77 Point processing O = T(I) T 255 Output O Input I 77

78 Point processing

79 Point processing 255 Output 0 Input

80 Contrast stretching 80

81 Point processing O = I γ γ < 1 : increase contrast in dark regions γ > 1 : increase contrast in bright regions 1 γ<1 0.5 γ=1 γ>

82 γ=

83 γ=

84 Some basic types 84

85 Image negative 85