Acquisition Some slides from: Yung-Yu Chuang (DigiVfx) Jan Neumann, Pat Hanrahan, Alexei Efros
Image Acquisition Digital Camera Film
Outline Pinhole camera Lens Lens aberrations Exposure Sensors Noise Sensing color
Camera trial #1 scene film Put a piece of film in front of an object. source: Yung-Yu Chuang
Pinhole camera pinhole camera scene barrier film Add a barrier to block off most of the rays It reduces blurring The pinhole is known as the aperture The image is inverted
Modeling projection The coordinate system Put the optical center (Center Of Projection) at the origin Put the image plane (Projection Plane) in front of the COP The camera looks down the negative z axis we need this if we want right-handed-coordinates
Modeling projection Projection equations Compute intersection with PP of ray from (x,y,z) to COP Derived using similar triangles (on board) We get the projection by throwing out the last coordinate:
In Homogenous Coordinates Projection is a matrix multiply using homogeneous coordinates:
Projection
Projection http://users.skynet.be/j.beever/pave.htm
Pinhole camera pinhole camera scene barrier film Add a barrier to block off most of the rays It reduces blurring The pinhole is known as the aperture The image is inverted
Shrinking the Pinhole Aperture Why make the aperture as small as possible? Less light gets through Diffraction effect
Shrinking the Pinhole Aperture Sharpest image is obtained when: pinhole diameter d = 2 f λ Example: If f = 50mm, λ = 600nm (red), d = 0.36mm
High-end commercial pinhole cameras ~$200
Pinhole Images Exposure 4 seconds Exposure 96 minutes Images copyright 2000 Zero Image Co.
Outline Pinholecamera Lens Lens Aberrations Exposure Sensors Noise Sensing color
Adding a lens scene film
Adding a lens circle of confusion scene lens film A lens focuses light onto the film There is a specific distance at which objects are in focus Other points project to a circle of confusion in the image
(Thin) Lens Thin lens equation: Any object point satisfying this equation is in focus
Circle of Confusion aperture Blur Circle, b aperture diameter d o i o' i' Blur Circle Diameter b : Derive using similar triangles b = d i' ( i ' i)
Aperture controls Depth of Field f / 5.6 Changing the aperture affects depth of field Smaller aperture: better DOF increased exposure f / 32
Depth of Field http://www.cambridgeincolour.com/tutorials/depth-of-field.htm
Thick Lens Corrects aberrations Change zoom
Field of View (Zoom)
Field of View (Zoom)
FOV depends on Focal Length ϕ/2 d/2 f f d = image size 2
FOV depends on Focal Length d = image size 2
FOV depends on Focal Length d = image size 2 Smaller FOV larger Focal Length
FOV depends on Focal Length focal point image size For closer objects: if focal point is larger but image distance and size remain unchanged the objects in focus are more distant.
Simplified Zoom Lens in Operation From wikipedia
Outline Pinholecamera Lens Lens aberrations Exposure Sensors Noise Sensing color
Radial Distortion No distortion Pin cushion Barrel Radial distortion of the image Caused by imperfect lenses Deviations are most noticeable for rays that pass through the edge of the lens
Correcting radial distortion from Helmut Dersch
Radial Distortions No Distortion Barrel Distortion Pincushion Distortion Radial distance from Image Center: r u = r d + k 1 r d 3 r u = undistorted radius r d = distorted radius
Correcting Radial Distortions Before After http://www.grasshopperonline.com/barrel_distortion_correction_software.html
Vignetting photo by Robert Johnes
Vignetting L3 L2 L1 B A More light passes through lens L3 for scene point A than scene point B Results in spatially non-uniform brightness (in the periphery of the image)
Chromatic Aberration longitudinal chromatic aberration (axial) transverse chromatic aberration (lateral)
Chromatic Aberration longitudinal chromatic aberration (axial) Canon EF 85/1.2 L USM transverse chromatic aberration (lateral) Cosina 3.5-4.5/19-35 @ 20 mm Good lens Carl Zeiss Distagon 2.8/21 http://www.vanwalree.com/optics/chromatic.html
Chromatic Aberration Near Lens Center Near Lens Outer Edge
Spherical aberration Rays parallel to the axis do not converge Outer portions of the lens yield smaller focal lengths
Spherical aberration Spherical lens are free of chromatic aberration but do not focus well. Parabolic lens does.
Spherical aberration
Astigmatism Different focal length for inclined rays
Astigmatism Change in size and shape of blur patches
Coma point off the axis depicted as comet shaped blob
Lens Glare Stray inter-reflections of light within the optical lens system Happens when very bright sources are present in the scene Reading: http://www.dpreview.com
Outline Pinholecamera Lens Lens aberrations Exposure Sensors Noise Sensing color
Exposure Two main parameters: Aperture (in f stop) Shutter speed (in fraction of a second)
Shutter
Leaf Shutter Advantages Uniform illumination Entire frame illuminated at once Disadvantages Illumination not constant over time Limitations on shutter speed
Focal Plane Shutter Advantages Cost effective (one shutter needed for all lenses Can achieve very fast shutter speeds (~1/10000 sec) Disadvantages May cause time distortion
Aperture Aperture is the diameter of the lens opening, usually specified by f-stop, f/d, a fraction of the focal length. f/2.0 on a 50mm means that the aperture is 25mm f/2.0 on a 100mm means that the aperture is 50mm When a change in f-stop occurs, the light is either doubled or cut in half. Lower f-stop, more light (larger lens opening) Higher f-stop, less light (smaller lens opening)
Aperture Constant speed.
Shutter speed Rule of Thumb 1 step in the shutter speed scale corresponds to 1 stop in the aperture scale. Handheld camera: shutter speed = 1 / f Stabilized gear: 2-3 shutter speeds slower Typical speeds: 1/1000 s, 1/500 s, 1/250 s, 1/125 s, 1/60 s, 1/30 s, 1/15 s, 1/8 s, 1/4 s, 1/2 s 1 s
Aperture vs. Shutter Depth of Field f/22 f/4 Small Aperture Large Aperture (Low speed) (High speed)
Aperture vs. Shutter Motion Blur 1/30 sec. @ f/22 1/6400 sec. @ f/2.5 Small Aperture (Low speed) Large Aperture (High speed)
Dynamic Range
Short exposure Real world radiance Picture intensity 10-6 10 6 dynamic range 10-6 10 6 Pixel value 0 to 255
Long exposure Real world radiance Picture intensity 10-6 10 6 dynamic range 10-6 10 6 Pixel value 0 to 255
Varying shutter speeds
HDR High Dynamic Range
Outline Pinholecamera Lens Lens aberrations Exposure Sensors Noise Sensing color
Spatial Sampling When a continuous scene is imaged on the sensor, the continuous image is divided into discrete elements - picture elements (pixels)
Spatial Sampling
Sampling The density of the sampling denotes the separation capability of the resulting image Image resolution defines the finest details that are still visible by the image We use a cyclic pattern to test the separation capability of an image 0 x
Sampling Frequency
Sampling Frequency
Nyquist Frequency Nyquist Rule: To observe details at frequency f (wavelength d) one must sample at frequency > 2f (sampling intervals < d/2) The Frequency 2f is the NYQUIST frequency. Aliasing: If the pattern wavelength is less than 2d erroneous patterns may be produced. 1D Example: 0
Aliasing - Moiré Patterns
Quantization
Digitizers (Quantization)
Image Sensors CCD Charge Coupled Device CMOS Complementary Metal Oxide Semiconductor
MOS (Metal Oxide Semiconductor) Photosensitive element Charge acquired depends on the number of photons which reach the element CCD devices are arrays of this basic element
Photoelectric Effect photon photon Increasing energy Conduction Band Valence Band 1.26eV Hole Electron Thermally generated electrons are indistinguishable from photo-generated electrons Dark Current.
Quantum Efficiency Not every photon hitting a pixel creates a free electron Quantum Efficiency (QE) = electrons collected / photons hitting the pixel QE heavily depends on the wavelength QE [%] blue green red QE < 100% degrades the SNR of a camera lambda [nm] SNR e = QE SNR p Typical max QE values : 25% (CMOS) 60% (CCD)
CCD (Charge Coupled Device) Boyle and Smith, 1969, Bell Labs Converts light into electrical signal (pixels)
CCD Readout Bucket Brigade Integration Charge Shift and Read-out Charge Amplifier
CMOS (Complementary Metal-Oxide Semiconductor) Each pixel owns its own charge-voltage conversion No need for external shutter (electronic shutter) The chip outputs digital bits Much faster than CCD devices
CCD vs. CMOS Mature technology Specific technology High production cost High power consumption Higher fill factor Blooming Sequential readout Recent technology Standard IC technology Cheap Low power Less sensitive Per pixel amplification Random pixel access Smart pixels On chip integration with other components
Sensor Parameters Fill factor The area in the sensor that is truly sensitive to light Shift registers and others can reduce it up to a 30% Well capacity The quantity of charge that can be stored in each pixel Close relation with pixel dimensions Integration time: Exposure time that is required to excite the CCD elements Depends on the scene brightness Acquisition time: Time needed to transfer the information gathered by the CCD Depends on the number of pixels in the sensor
Fill Factor The ratio between the light sensitive pixel area and the total pixel area. Total pixel area: 5µm x 5µm Photo Sensing Area Fill factor 40%
Outline Pinholecamera Lens Lens aberrations Exposure Sensors Noise Sensing color
Sensor noise Noise Sources Photon noise / Shot Noise (Poisson) Dark Noise (Constant) Thermal noise (Poisson) Resetting (fixed) Read-out noise Blooming (After T. Lomheim, The Aerospace Corporation) http://www.stw.tu-ilmenau.de/~ff/beruf_cc/cmos/cmos_noise.pdf
Photon Shot Noise Light is quantum in nature Noise due to statistics of the detected photons themselves The probability distribution for N photons to be counted in an observation time T is Poisson P ( N F, T ) = ( FT ) N FT e N! F = fixed average flux (photons/sec)
Poisson Distribution, FT = 5 0.2 0.15 0.1 0.05 0 0 20 40 60 80 100 N Poisson Distribution : std equals sqrt of Mean. σ = FT = 2 shot N photons
Poisson Distribution, FT = 10 0.14 0.12 0.1 0.08 0.06 0.04 0.02 0 0 20 40 60 80 100 N
Poisson Distribution, FT = 20 0.1 0.08 0.06 0.04 0.02 0 0 20 40 60 80 100 N
Poisson Distribution, FT = 50 0.06 0.05 0.04 0.03 0.02 0.01 0 0 20 40 60 80 100 As FT grows, Poisson distribution approaches Gaussian distribution. Signal To Noise (SNR) Increases with Mean. SNR = N σ 2 2 N = N N 2 = N
Photon Noise More noise in bright parts of the image You can identify the white and black regions from the noise image
Photon Noise Photon Noise more noticeable in dark images.
Dark Current Noise Electron emission when no light Dark current noise is high for long exposures To remove (some) of it Calibrate the camera (make response linear) Capture the image of the scene as usual Cover the lens with the lens cap and take another picture Subtract the second image from the first image
Dark Current Noise Original image + Dark Current Noise Image with lens cap on Result of subtraction Copyright Timo Autiokari,
Sensor noise ideal relationship between electrons and impinging photons CCD capacity limit Light Signal (QE = 50) Photon Noise (QE = 50)
Outline Pinholecamera Lens Lens aberrations Exposure Sensors Noise Sensing color
Sensing Color light beam splitter 3 CCD Bayer pattern Foveon X3 TM
Multi-Chip wavelength dependent
Field Sequential
Field Sequential
Field Sequential
Color Filter Array (CFA) Fuji Corporation
Color filter array Bayer pattern Color filter arrays (CFAs)/color filter mosaics
Bayer s pattern
Demosaicking CFA s
Color filter array red green blue output
X3 technology red green blue output
Foveon X3 sensor Bayer CFA X3 sensor
Cameras with X3 Sigma SD14 Polaroid X530 Hanvision HVDUO 5M/10M Out of production
Color processing After color values are recorded, more color processing usually happens: White balance Non-linearity to approximate film response or match TV monitor gamma
White Balance warmer +3 automatic white balance
Gamma Correction Gamma correction applied by the converter redistributes the pixel luminance values so that limited brightness range captured by the sensor is mapped to match our eye s sensitivity. Gamma = 2.2 is a good match to distribute relative brightness in a print or in a video display.
Gamma =1 vs. Gamma = 2.2
Space of response curves
Outline Pinholecamera Lens Lens aberrations Exposure Sensors Noise Sensing color Summary
Camera pipeline
Sensor Response The response of a sensor is proportional to the radiance and the throughput
Measurement Equation Scene Radiance Optics Pixel Response Shutter L(x,ω,t,λ) (x,ω ) = T(x,ω,λ) P(x,λ) S(x,ω,t)
Degradation Due To Sampling Sampling in space Pixels Sampling in intensity Quantization Sampling in color Color Filter Array (CFA) Sampling in time Exposure Sampling in frequency Lens and pixel PSF (point-spread-function)