Capturing light and color

Capturing light and color Friday, 10/02/2017 Antonis Argyros e-mail: argyros@csd.uoc.gr Szeliski 2.2, 2.3, 3.1 1

Recap from last lecture Pinhole camera model Perspective projection Focal length and depth/field of view Lenses

Recap - Projection matrix Slide Credit: Saverese R,T j w X k w x i w O w x= K[ R t] X x: Image Coordinates: (u,v,1) K: Intrinsic Matrix (3x3) R: Rotation (3x3) t: Translation (3x1) X: World Coordinates: (X,Y,Z,1)

Adding a lens circle of confusion A lens focuses light onto the film There is a specific distance at which objects are in focus other points project to a circle of confusion in the image Changing the shape of the lens changes this distance

The long trip of light Light source properties Sensor characteristics Exposure Surface shape Optics Surface reflectance properties 5 Slide by L. Fei-Fei

What we will discuss today Geometry Relations between 3D world points and their 2D projections on the image Radiometry The relation of emitted light to the light that reaches the camera Photometry How to measure the light that reaches the camera Digitization The transformation of the spatio-temporally continuous signal to digital representations 6

A photon s life choices Absorption Diffusion Reflection Transparency Refraction Fluorescence Subsurface scattering? λ light source

A photon s life choices Absorption Diffusion Reflection Transparency Refraction Fluorescence Subsurface scattering λ light source

A photon s life choices Absorption Diffuse Reflection Reflection Transparency Refraction Fluorescence Subsurface scattering λ light source

A photon s life choices Absorption Diffusion Specular Reflection Transparency Refraction Fluorescence Subsurface scattering λ light source

A photon s life choices Absorption Diffusion Reflection Transparency Refraction Fluorescence Subsurface scattering λ light source

A photon s life choices Absorption Diffusion Reflection Transparency Refraction Fluorescence Subsurface scattering λ light source

A photon s life choices Absorption Diffusion Reflection Transparency Refraction Fluorescence Subsurface scattering λ 2 λ 1 light source

A photon s life choices Absorption Diffusion Reflection Transparency Refraction Fluorescence Subsurface scattering λ light source

Radiometry What determines the value of an image pixel? Radiance (L): energy carried by a ray Irradiance (E): energy arriving at the sensor Sensor characteristics 15

The journey of the light ray X = E t E d = π 2 cos 4 α 4 z' L Z = f ( E t) Camera response function: the mapping f from irradiance to pixel values 16 Source: S. Seitz, P. Debevec

Lambertian Reflectance In computer vision, surfaces are often assumed to be ideal diffuse reflectors with known dependence on viewing direction. Lambert s law: Viewed brightness does not depend on viewing direction, but it does depend on direction of illumination

What is color? Color is the result of interaction between physical light in the environment and our visual system Color is a psychological property of our visual experiences when we look at objects and lights, not a physical property of those objects or lights (S. Palmer, Vision Science: Photons to Phenomenology) Wassily Kandinsky (1866-1944), Murnau Street with Women, 1908 18

Electromagnetic spectrum Human Luminance Sensitivity Function 19

Beyond the visible spectrum γ - rays Χ - rays UV IR Long radiowaves 20

The Physics of Light: light sources Any source of light can be completely described physically by its spectrum: the amount of energy emitted (per time unit) at each wavelength 400-700 nm. Relative spectral power # Photons (per ms.) 400 500 600 700 Wavelength (nm.) 21 Stephen E. Palmer, 2002

The Physics of Light: light sources Some examples of the spectra of light sources A. Ruby Laser B. Gallium Phosphide Crystal 400 500 600 700 Wavelength (nm.) D. Normal Daylight # Rel. Photons power Rel. # Photons power Wavelength (nm.) 400 500 600 700 C. Tungsten Lightbulb Rel. # Photons power Rel. # Photons power 400 500 600 700 400 500 600 700 22 Stephen E. Palmer, 2002

The Physics of Light: reflectance spectra of surfaces Some examples of the reflectance spectra of surfaces % Light Reflected Red Yellow Blue Purple 400 700 400 700 400 700 Wavelength (nm) 400 700 23 Stephen E. Palmer, 2002

Interaction of light and surfaces Observed color is the result of interaction of light source spectrum with surface reflectance 24

The Eye The human eye is a camera (the best)! Iris - colored annulus with radial muscles Pupil - the hole (aperture) whose size is controlled by the iris Lens - changes shape by using ciliary muscles (to focus on objects at different distances) What s the film? photoreceptor cells (rods and cones) in the retina 25 Slide by Steve Seitz

Two types of light-sensitive receptors Cones cone-shaped less sensitive operate in high light color vision Rods rod-shaped highly sensitive operate at night gray-scale vision cone rod Stephen E. Palmer, 2002

Rod / Cone sensitivity

Density of rods and cones cone pigment molecules Rods and cones are non-uniformly distributed on the retina Rods responsible for intensity, cones responsible for color Fovea - Small region (1 or 2 ) at the center of the visual field containing the highest density of cones (and no rods). Less visual acuity in the periphery rod 28 Slide by Steve Seitz

Standardizing color experience We would like to understand which spectra produce the same color sensation from people under similar viewing conditions Color matching experiments Foundations of Vision, by Brian Wandell, Sinauer Assoc., 1995 29

Color matching experiment 30 Source: W. Freeman

Color matching experiment p 1 p 2 p 3 31 Source: W. Freeman

Color matching experiment p 1 p 2 p 3 32 Source: W. Freeman

Color matching experiment The primary color amounts needed for a match p 1 p 2 p 3 33 Source: W. Freeman

Trichromacy Three numbers seem to be sufficient for encoding color In color matching experiments, most people can match any given light with three primaries Exception: color blindness For the same light and same primaries, most people select the same weights 34

Grassman s Laws If two test lights can be matched with the same set of weights, then they are the same If we mix two test lights, then mixing the matches will match the result: Suppose A = u 1 P 1 + u 2 P 2 + u 3 P 3 and B = v 1 P 1 + v 2 P 2 + v 3 P 3. Then A+B = (u 1 +v 1 ) P 1 + (u 2 +v 2 ) P 2 + (u 3 +v 3 ) P 3. If we scale the test light, then the matches get scaled by the same amount: Suppose A = u 1 P 1 + u 2 P 2 + u 3 P 3. Then ka = (ku 1 ) P 1 + (ku 2 ) P 2 + (ku 3 ) P 3. 35

Color Sensing in Camera (RGB) 3-chip vs. 1-chip: quality vs. cost http://www.cooldictionary.com/words/bayer-filter.wikipedia Slide by Steve Seitz

Practical Color Sensing: Bayer Grid Estimate RGB at G cells from neighboring values Slide by Steve Seitz

Color Image R G B

Images in Matlab Images represented as a matrix Suppose we have a NxM RGB image called im im(1,1,1) = top-left pixel value in R-channel im(y, x, b) = y pixels down, x pixels to right in the b th channel im(n, M, 3) = bottom-right pixel in B-channel imread(filename) returns a uint8 image (values 0 to 255) Convert to double format (values 0 to 1) with im2double row column 0.92 0.93 0.94 0.97 0.62 0.37 0.85 0.97 0.93 0.92 0.99 0.95 0.89 0.82 0.89 0.56 0.31 0.75 0.92 0.81 0.95 0.91 0.89 0.72 0.51 0.55 0.51 0.42 0.57 0.41 0.49 0.91 0.92 0.92 0.93 0.94 0.97 0.62 0.37 0.85 0.97 0.93 0.92 0.99 G 0.96 0.95 0.88 0.94 0.56 0.46 0.91 0.87 0.90 0.97 0.95 0.95 0.89 0.82 0.89 0.56 0.31 0.75 0.92 0.81 0.95 0.91 0.71 0.81 0.81 0.87 0.57 0.37 0.80 0.88 0.89 0.79 0.85 0.89 0.72 0.51 0.55 0.51 0.42 0.57 0.41 0.49 0.91 0.92 0.49 0.62 0.60 0.58 0.92 0.50 0.93 0.60 0.94 0.58 0.97 0.50 0.62 0.61 0.37 0.45 0.85 0.33 0.97 0.93 0.92 0.99 0.96 0.95 0.88 0.94 0.56 0.46 0.91 0.87 0.90 0.97 0.95 0.86 0.84 0.74 0.58 0.95 0.51 0.89 0.39 0.82 0.73 0.89 0.92 0.56 0.91 0.31 0.49 0.75 0.74 0.92 0.81 0.95 0.91 0.71 0.81 0.81 0.87 0.57 0.37 0.80 0.88 0.89 0.79 0.85 0.96 0.67 0.54 0.85 0.89 0.48 0.72 0.37 0.51 0.88 0.55 0.90 0.51 0.94 0.42 0.82 0.57 0.93 0.41 0.49 0.91 0.92 0.49 0.62 0.60 0.58 0.50 0.60 0.58 0.50 0.61 0.45 0.33 0.69 0.49 0.56 0.66 0.96 0.43 0.95 0.42 0.88 0.77 0.94 0.73 0.56 0.71 0.46 0.90 0.91 0.99 0.87 0.90 0.97 0.95 0.86 0.84 0.74 0.58 0.51 0.39 0.73 0.92 0.91 0.49 0.74 0.79 0.73 0.90 0.67 0.71 0.33 0.81 0.61 0.81 0.69 0.87 0.79 0.57 0.73 0.37 0.93 0.80 0.97 0.88 0.89 0.79 0.85 0.96 0.67 0.54 0.85 0.48 0.37 0.88 0.90 0.94 0.82 0.93 0.91 0.94 0.89 0.49 0.49 0.41 0.62 0.78 0.60 0.78 0.58 0.77 0.50 0.89 0.60 0.99 0.58 0.93 0.50 0.61 0.45 0.33 0.69 0.49 0.56 0.66 0.43 0.42 0.77 0.73 0.71 0.90 0.99 0.86 0.84 0.74 0.58 0.51 0.39 0.73 0.92 0.91 0.49 0.74 0.79 0.73 0.90 0.67 0.33 0.61 0.69 0.79 0.73 0.93 0.97 0.96 0.67 0.54 0.85 0.48 0.37 0.88 0.90 0.94 0.82 0.93 0.91 0.94 0.89 0.49 0.41 0.78 0.78 0.77 0.89 0.99 0.93 0.69 0.49 0.56 0.66 0.43 0.42 0.77 0.73 0.71 0.90 0.99 0.79 0.73 0.90 0.67 0.33 0.61 0.69 0.79 0.73 0.93 0.97 0.91 0.94 0.89 0.49 0.41 0.78 0.78 0.77 0.89 0.99 0.93 R B

Color spaces How can we represent color? http://en.wikipedia.org/wiki/file:rgb_illumination.jpg

Color spaces: RGB Default color space 0,1,0 R (G=0,B=0) 1,0,0 0,0,1 G (R=0,B=0) Drawback: Non-perceptual: a change of the same amount in a color value, does not produce a change of about the same visual importance. B (R=0,G=0) Image from: http://en.wikipedia.org/wiki/file:rgb_color_solid_cube.png

Color spaces: HSV Intuitive color space H (S=1,V=1) S (H=1,V=1) V (H=1,S=0)

Color spaces: YCbCr Y=0 Y=0.5 Y (Cb=0.5,Cr=0.5) Cr Cb Y=1 Cb (Y=0.5,Cr=0.5) Cr (Y=0.5,Cb=05)

Color spaces: L*a*b* L (a=0,b=0) a (L=65,b=0) b (L=65,a=0)

If you had to choose, would you rather go without luminance or chrominance?

If you had to choose, would you rather go without luminance or chrominance?

Most information in intensity Only color shown constant intensity

Most information in intensity Only intensity shown constant color

Most information in intensity Original image

Lightness constancy http://web.mit.edu/persci/people/adelson/checkershadow_illusion.html 50

Lightness constancy http://web.mit.edu/persci/people/adelson/checkershadow_illusion.html 51

Chromatic adaptation The visual system changes its sensitivity depending on the luminances prevailing in the visual field Adapting to different brightness levels Changing the size of the iris opening (i.e., the aperture) changes the amount of light that can enter the eye Think of walking into a building from full sunshine Adapting to different color temperature The receptive cells on the retina change their sensitivity For example: if there is an increased amount of red light, the cells receptive to red decrease their sensitivity until the scene looks white again http://www.schorsch.com/kbase/glossary/adaptation.html 52

Name that color Πράσινο Κόκκινο Μπλέ Κίτρινο Πράσινο Κόκκινο Πορτοκαλί Μώβ Καφέ Κίτρινο Μαύρο 53

Digital camera A digital camera replaces film with a sensor array Each cell in the array is light-sensitive diode that converts photons to electrons Two common types Charge Coupled Device (CCD) Complementary metal oxide semiconductor (CMOS) http://electronics.howstuffworks.com/digital-camera.htm 54 Slide by Steve Seitz

Digitization Let s digitize this image We will assume a square sampling grid We will be varying the sampling density 55

Digitization A sample per red point Sampling interval Coarse sampling: 14 lines, 20 points / line Denser sampling: 68 lines, 100 points/line 56

Consequences of the sampling interval/rate Look at the fence Sampling interval White image! Grey image! 100 100 100 100 100 100 40 40 40 40 40 40 100 100 100 100 100 100 40 40 40 40 40 40 100 100 100 100 100 100 40 40 40 40 40 40 100 100 100 100 100 100 40 40 40 40 40 40 57

Consequences of the sampling interval/rate Look at the fence Sampling interval 40 100 40 100 40 40 100 40 100 40 40 100 40 100 40 Now the fence is visible! 40 100 40 100 40 58

Consequences of the sampling interval/rate Look at the repetitive pattern on the fence: Sampling intervals s' d s Case 1: s' = d Sampling interval equal to the period of the repetitive pattern Invisible fence Case 2: s = d/2 Sampling interval equal to half the period of the repetitive pattern Visible fence 59

Sampling theorem If the width of the thinnest structure is d, then the sampling interval should be at smaller than d/2 The repetitive pattern has a certain frequency. To maintain this frequency, sampling must be done with at least double this frequency (Nyquist rate) 60

Sampling Basic idea, in theory 23 Digitized image Continuous signal Dirac Delta Function 2D "Comb" δ(x,y) = 0 for x ǂ 0, y ǂ 0 s δ(x,y) dx dy = 1 f(x,y)δ(x-a,y-b) dx dy = f(a,b) 61

Sampling Basic idea, in practice... The ideal situation is not feasible in practice A sample corresponds to a region that has a certain area The value of a sample depends on the value of the continuous function over this area 23 s 62

as a result. The mixed pixel problem 63

Remember: Color sensing in camera: Color filter array Bayer grid Estimate missing components from neighboring values (demosaicing) 64 Source: Steve Seitz

Problem with demosaicing: color moire 65

Signal quantization Goal: Ok, we saw where to sample, what about the actual value measured at each sample? I(x,y) =.1583 volts =???? Digital value? 66

Quantization I(x,y) = continuous signal 0 I(x,y) M We need to quantize it in K levels/values 0,1,...K-1 K is usually a power of 2: K #Bits 2 1 4 2 8 3 16 4 32 5 64 6 128 7 256 8 We need to define the mapping between the (analog) input signal to the (digital) output signal. Several such mappings (linear, logarithmic, etc) 67

E.g., linear mapping uantization Level K-1 3 2 1 0 0 Signal Value M 68

Selection of K Continuous signal K=2 K=4 K=16 K=32 69

Selection of K K=2 (for each basic color) K=4 (for each basic color) 70

Loss during spatial sampling

Loss during quantization

Image histogram H The histogram of an image represents the number of pixels having a particular intensity / gray value 8 Pixel Count 7 6 5 4 3 2 1 0 0 1 2 3 4 5 6 7 Image M x N I max i= I min H ( i) = MN Gray Value 73

Image histogram & image content 1241 0 256 1693 0 256 74

Image histogram vs image properties Dark image Bright image Low contrast High contrast 75

Point processing An image processing operator defines an operator that, when applied at an input image f results in a new image g. The simplest of such operators are those that processing is independent of position in the image Such operators are called point processing operators 76

Point processing O = T(I) T 255 Output O 0 255 Input I 77

Point processing 1 1 1 0.5 0.5 0.5 0 0 0.5 1 4000 4000 0 0 0.5 1 4000 0 0 0.5 1 2000 2000 2000 0 0 100 200 0 0 0.5 1 0 0 0.5 1 78

Point processing 255 Output 0 Input 255 79

Contrast stretching 80

Point processing O = I γ γ < 1 : increase contrast in dark regions γ > 1 : increase contrast in bright regions 1 γ<1 0.5 γ=1 γ>1 0 0 0.5 1 81

γ=0.5 1 0.5 0 0 0.5 1 82

γ=3.0 1 0.5 4000 3500 3000 2500 2000 1500 1000 500 0 0 50 100 150 200 250 0 0 0.5 1 4000 3500 3000 2500 2000 1500 1000 500 0 83 0 50 100 150 200 250

Some basic types 84

Image negative 85