Computational Photography. Computational Photography. The Camera. Camera Developments

Transcription

1 Computational Photography Computer Science Semester B 29-2 Lecture: Sunday 2:-5: Room: 33 The Camera A camera is a device that takes photos of images Camera Obscura (Latin = "dark chamber") Dr. Hagit Hel-Or hagit@cs.haifa.ac.il Office: 45 Office Hours: by appointment Course Internet Site: 9 th century camera Sony s smile recognition camera 2 Camera Developments permanent capturing (wet plates) exposure time and motion capture (dry plates) portability (film-kodak) quality and size (35 mm) optics (SLR) instancy (polaroid) digital sensors 2 computational photography Computational Photography Wikipedia: Computational photography refers broadly to computational imaging techniques that enhance or extend the capabilities of digital photography. The output of these techniques is an ordinary photograph, but one that could not have been taken by a traditional camera Earliest surviving photograph. This image required an eight-hour exposure. 4 Steve Mann: The Cyberman

2 Goal: Record a richer, multi-layered layered visual experience. Overcome limitations of today s s cameras 2. Support better post-capture processing 3. Enables new classes of recording the visual signal 4. Synthesize impossible photos Topics Intro and image formation Acquisition and camera model Single exposure enhancement Panoramas and feature-based registration Blending and Composition Appearance-based registration Multi exposure enhancement Multi exposure enhancement Data Driven Synthesis Segmentation and Matting Single View Modeling Light Field Project presentation 5 6 Administration Pre-requisites / prior knowledge Course Home Page: Messages Lecture slides and handouts Matlab guides Homework, Grades Exercises: Programming in Matlab, ~3 Assignments Final project 7 Administration (Cont.) Matlab software: Available in PC labs Student version Grading policy: Final Grade will be based on: Exercises (4%), Final project (6%) Exercises will be weighted Exercises can be submitted in pairs Office Hours: by appointment 8 2

3 Further Readings Related papers New book: Computational Photography by R. Raskar and J. Tumblin 9 Syllabus Image Formation Image formation HVS pathway Color models Acquisition and camera model Camera model + perspective projections Sensors Noise models & Distortions Sampling (spatial+temporal) and quantization Camera parameters Camera Parameters trade-offs. Single exposure enhancement White Balancing De-mosaicing De-noising De-blurring Geometrical distortion correction Panoramas and feature based registration Image features SIFT Panoramas Feature based registration Panoramas Homography RANSAC Image stitching Syllabus cont. Syllabus cont. Blending and Composition Pyramid blending Optimal cut Seam Carving Graph-cut Gradient domain editing Appearance based registration Similarity measures Lucas Kanade optical flow Multi-modal registration Applications Multi exposure enhancement (2 weeks) HDR Super-resolution multi-exposure fusion Data Driven Synthesis Texture synthesis Video texture Quilting Image analogies Super-Resolution Image Completion Segmentation and Matting Segmentation using Graph cut. mean-shift Spectral clustering Interactive and semi-automatic Matting Single View Modeling Camera Calibration Measurements in affine camera 3D reconstruction Light Field Plenoptic function and the Lumiograph Re-sampling the plenoptic function 2 3

4 . Image Formation 2. Camera Model and Acquisition Taking a picture HVS pathway Color models Cornea Pupil Lens Fovea Vitreous Humor Optic Nerve Perspective projections Camera pipeline and parameters Sensors and optics Sampling and quantization Noise models & Distortions Camera Parameters trade-offs. Iris Optic Disc Retina Ocular Muscle Single Exposure Enhancement White Balancing De-mosaicing De-noising De-blurring Geometrical distortion correction 4. Panoramas and Feature Based Registration Image features SIFT Feature based registration Panoramas Homography RANSAC Image stitching Difference in white point 5 6 4

5 5. Blending and Composition 6. Appearance Based Registration (warping?) Pyramid blending Gradient domain editing Optimal cut Graph-cut Similarity measures Lucas Kanade optical flow Multi-modal registration Applications Multi Exposure Enhancement HDR Super-resolution Different-exposures fusion 8. Data Driven Synthesis Texture synthesis Video texture Quilting Image analogies Super-Resolution Image Completion 9 2 5

6 9. Segmentation and Matting Segmentation using Graph cut. Mean-shift Spectral clustering Interactive and semi-automatic Matting. Single View Modeling Camera Calibration 3D reconstruction Metrology 2 Flagellation by Pietro della Francesca (46-92, Italian Renaissance period) Animation by Criminisi et al., ICCV Light Field Plenoptic function and the Lumiograph Re-sampling the plenoptic function Computational Photography Today s s Topic - Image Formation What is an image? What is color?

7 The Visual Sciences Image/video Processing 2D Images What is an Image? An image is a projection of a 3D scene into a 2D projection plane. An image can be defined as a 2 variable function I(x,y), where for each position (x,y) in the projection plane, I(x,y) defines the light intensity at this point. Computer Vision Rendering 3D Object Geometric Modeling Model The Pinhole Camera Model Projection Model (where) Image plane COP Focal length Pinhole model: Captures pencil of rays all rays through a single point The point is called Center of Projection (COP) The image is formed on the Image Plane Effective focal length f - distance from COP to Image Plane The coordinate system We will use the pin-hole model as an approximation Put the optical center (Center Of Projection) at the origin Put the image plane (Projection Plane) in front of the COP The camera looks down the negative z axis 27 Slide by Steve Seitz 28 Slide by Steve Seitz 7

8 Funny things happen Parallel lines aren t 29 3 Lengths can t t be trusted... The Shading Model (what) Shading Model: Given the illumination incident at a point on a surface, what is reflected?

9 Shading Model Parameters Light and the Visible Spectrum The factors determining the shading effects are: The light source properties: Positions, Electromagnetic Spectrum, Shape. The surface properties: Position, orientation, Reflectance properties. The eye (camera) properties: Position, orientation, Sensor spectrum sensitivities. Newton s Experiment, 665 Cambridge. Discovering the fundamental spectral components of light The light Spectrum Electromagnetic Radiation - Spectrum Ultraviolet Short- Gamma X rays Infrared Radar FM TV wave AM AC electricity Wavelength in meters (m) Visible light 4 nm 5 nm 6 nm 7 nm Wavelength in nanometers (nm)

10 Monochromators Monochromators measure the power or energy at different wavelengths Light Parameters The Spectral Power Distribution (SPD) of a light is a function e(λ) which defines the energy at each wavelength. Relative Power.5 Wavelength (λ) Examples of Spectral Power Distributions Light Spatial Distribution A B C D.5.5 Blue Skylight.5 Red monitor phosphor 39 Tungsten bulb.5 Monochromatic light Point Source (A): All light rays originate at a point and radially diverge. A reasonable approximation for sources whose dimensions are small compared to the object size. Parallel source (B): Light rays are all parallel. May be modeled as a point source at infinity (the sun). Distributed source (C): All light rays originate at a finite area in space. A nearby source such as a fluorescent light. Ambient source (D) homogeneous non-directed, background light. 4

11 Surface Parameters Incident light normal Specular reflection Diffuse reflection Specular reflection mirror like reflection at the surface Diffuse (lambertian) reflection reflected randomly between color particles reflection is equal in all directions Different Types of Surfaces 4 42 Spectral Property of Diffuse Surfaces Yellow Blue Red Gray Wavelength (nm) Surface Body Reflectances (albedo) R V Ambient reflection: I amb = K(λ) e a (λ) Diffuse reflection: N 44 θ L Surface properties Light properties I diff = K(λ) e p (λ) (N L) Specular reflection: I spec = K s (λ)e p (λ) (R V) n e a e p - the ambient and point light intensities. K, K s [,] - the surface ambient / diffuse / specular reflectivity. N - the surface normal, L - the light direction, V viewing direction geometry

12 R V N θ L Ambient reflection: I amb = K(λ) e a (λ) Diffuse reflection: I diff = K(λ) e p (λ) (N L) Specular reflection: I spec = K s (λ)e p (λ) (R V) n e p e a - the ambient and point light intensities. K, K s [,] - the surface ambient / diffuse / specular reflectivity. N - the surface normal, L - the light direction, V viewing direction Ambient surface Diffuse surface Diffuse + Specular The final illumination equation: Composition of Light Sources I(λ) = I amb +I diff +I spec If several light sources are placed in the scene: I(λ)= I amb +Σ k (I k diff +Ik spec )

13 The Human Visual System The Human Eye Lens Retina Optic Nerve Optic Chiasm Cornea Pupil Fovea Vitreous Humor Optic Nerve The Visual Pathway Lateral Geniculate Nucleus (LGN) Visual Cortex קרנית - Cornea אישו ן - Pupil קשתית - Iris רשתית - Retina Iris Ocular Muscle Optic Disc Retina 49 5 The Human Eye Imaging System

14 Eye v.s.. Camera The Human Retina rods cones bipolar ganglion horizontal amacrine light 53 Yaho Wang s slides 54 Retina contains 2 types of photo-receptors Cones: Day vision, can perceive color tone Rods: Night vision, perceive brightness only Cones: High illumination levels (Photopic vision) Sensitive to color (there are three cone types: L,M,S) Produces high-resolution vision 6-7 million cone receptors, located primarily in the central portion of the retina Cone Spectral Sensitivity 55 Relative sensitivity Wavelength (nm) 56 L M S A side note: Humans and some monkeys have three types of cones (trichromatic vision); most other mammals have two types of cones (dichromatic vision). Marine mammals have one type of cone. Most birds and fish have four types. Lacking one or more type of cones result in color blindness. 4

15 Rods: Low illumination levels (Scotopic vision). Highly sensitive (respond to a single photon). Produces lower-resolution vision million rods in each eye. No rods in fovea. Cone Receptor Mosaic (Roorda and Williams, 999) Rod Spectral Sensitivity Relative sensitivity Wavelength (nm) 57 L-cones M-cones S-cones 58 Photoreceptor Distribution Foveal Periphery photoreceptors Cone s Distribution: L-cones (Red) occur at about ~65% of the cones throughout the retina. M-cones (green) occur at about ~3% of the cones. S-cones (blue) occur at about ~2-5% of the cones (Why so few?). rods S - Cones 59 L/M - Cones Receptors per square mm 8 x 4 rods cones fovea 6 Distribution of rod and cone photoreceptors Degrees of Visual Angle 5

16 The Cone Responses Output Sensors I(λ) Fixed, point source illuminant l(λ),m(λ),s(λ) Cone responsivities Illuminant L = l( λ) I( λ) M = m( λ) I( λ) S = s( λ) I( λ) 6 Metamer - two lights that appear the same. They might have different SPDs (spectral power distributions). Power 2 Tungsten light 8 4 Wavelength (nm) 62 Monitor emission The phosphors of the monitor were set to match the tungsten light. The Trichromatic Color Theory Trichromatic: tri =three chroma =color color vision is based on three primaries (i.e., it is 3D). Thomas Young ( ) - A few different retinal receptors operating with different wavelength sensitivities will allow humans to perceive the number of colors that they do. Suggested 3 receptors. Helmholtz & Maxwell (85) - Color matching with 3 primaries. 63 Color Matching Experiment Given a set of 3 primaries, one can determine for every spectral distribution, the intensity of the guns required to match the color of that spectral distribution. The 3 numbers can serve as a color representation. T(λ) test match 64 Primaries ( λ ) rr ( λ ) + gg ( λ ) bb ( λ ) T + R(λ) G(λ) B(λ) 6

17 Color matching experiment for Monochromatic lights Primary Intensities Primary Intensity 3 r(λ) 2 b(λ) g(λ) Wavelength (nm) Observation - Color matching is linear: if (S P) then (S+N P+N) if (S P) then (α S αp) Outcome : Any T(λ) can be matched: r = T λ r λ dλ ; g = T λ g λ dλ ; b = ( ) ( ) ( ) ( ) T ( λ) b ( λ) dλ Outcome 2: CMF can be calculated for any chosen primaries U(λ), V(λ), W(λ): Stiles & Burch (959) Color matching functions. Primaries are: and Problems: Some perceived colors cannot be generated. This is true for any choice of visible primaries. u c v = c w c ru rv rw c c c gu gv gw c c c bu bv bw r g b

18 The CIE Color Standard The CIE (Commission Internationale d Eclairage) defined in 93 three hypothetical lights X, Y, and Z whose matching functions are positive everywhere: Tristimulus Let X, Y, and Z be the tristimulus values. A color can be specified by its trichromatic coefficients, defined as X x = X + Y + Z Y y = X + Y + Z Z z = X + Y + Z X ratio Y ratio Z ratio Two trichromatic coefficients are enough to specify a color. (x + y + z = ) 69 From: Bahadir Gunturk 7 CIE Chromaticity Diagram Input light spectrum CIE Chromaticity Diagram Input light spectrum y y x x From: Bahadir Gunturk 7 From: Bahadir Gunturk 72 8

19 CIE Chromaticity Diagram Input light spectrum CIE Chromaticity Diagram Input light spectrum y y 7nm Boundary 38nm x x From: Bahadir Gunturk 73 From: Bahadir Gunturk 74 CIE Chromaticity Diagram Input light spectrum CIE Chromaticity Diagram Light composition y 7nm Boundary 38nm x From: Bahadir Gunturk 75 From: Bahadir Gunturk 76 9

20 CIE Chromaticity Diagram Light composition Pure wavelength in chromaticity diagram Blue: big value of Z, therefore x and y small Light composition From: Bahadir Gunturk Pure wavelength in chromaticity diagram Then y increases Pure wavelength in chromaticity diagram Green: y is big

21 Pure wavelength in chromaticity diagram Yellow: x & y are equal Pure wavelength in chromaticity diagram Red: big x, but y is not null 8 82 CIE Chromaticity Diagram The CIE chromaticity diagram is helpful to determine the range of colors that can be obtained from any given colors in the diagram. The srgb Color Standard The srgb is a device-independent color space. It was created in 996 by HP and Microsoft for use on monitors and printers. It is the most commonly used color space. It is defined by a transformation from the xyz color space. Gamut: The range of colors that can be produced by the given primaries. Source:

22 Color Spaces RGB Color Space (additive) Define colors with (r, g, b) ; amounts of red, green, and blue CMY Color Space (subtractive) Cyan, magenta, and yellow are the complements of red, green, and blue We can use them as filters to subtract from white The space is the same as RGB except the origin is white instead of black HSV color space Hue - the color tone (red, green, purple). Saturation - purity (distance of color from gray ). Value - brightness of the color

23 HSV - a more intuitive color space Saturation Value Opponent Colors Ganglion cells / LGN cells B- Y+ Y- B+ Cortical cells R- G+ B- Opponent process - possible neural connections: S M L R- G+ G- R+ Y+ Hue Color Contrast detectors Color edge detectors L+M-S L-M L+M+S Blue-Yellow Red-Green Black-White 89 9 The Statistics of Color Images A joint Histogram of r x v.s. g x Observation: Color bands are highly correlated in high spatial frequencies H Green derivative Red derivative

24 A joint Histogram of g x v.s. b x A joint Histogram of r x v.s. b x Blue derivative Blue derivative Green derivative Red derivative Joint histograms of R v.s. G for a low pass images

25 Define a new color basis (l,c,c 2 ): l R c = T G where T = n c2 B Green derivative A joint Histogram of r x v.s. g x L c 2 l luminance C - red/green C 2 blue/yellow l luminance value C Red-Green C 2 Blue-Yellow Observations: The l channel encodes the color luminance. C and C 2 channels encode the chrominance. In the chrominance channels high freq. are attenuated. In the luminance channel high freq. are maintained. The 3 opponent channels are uncorrelated in the high freq. Efficient for color image encoding Red derivative Original Image High freq. details Low freq. details Low freq. details Claim: The HVS high spatial sensitivity in the luminance domain and low spatial sensitivity in the chrominance domains is a direct outcome of the statistical properties of color images! 99 25

26 After blurring C and C 2 bands After blurring l band as well 2 Opponent Color Spaces The standard representation used in TV broadcasting Backwards compatibility with B/W TV Low bit rate is needed in the chrominance channels There are various opponent representations: YIQ - used for NTSC color TV YUV (also called YC b C r ) - used for PAL TV and video Question: why are S cones sparse in retina? Opponent Color Spaces R = R H + R L G = G H + G L G Dense Sampling R = RH + RL = GH Sparse sampling

27 Color matching predicts matches, not appearance Color Appearance 5 6 Color Appearance Color Appearance

28 Color Appearance Summary Image Formation Projection Model defines projection of scene to image plane (where). Shading Model defines color projected onto the image plane (what). Albers (975) 9 Summary Image Formation Summary Image Formation Illumination Color Signal Shading Model involves: 2 3 Light source properties Surface reflectance properties Sensor properties Reflectance Cone Absorptions Cone Sensitivities L M S 2 28