Acquisition. Some slides from: Yung-Yu Chuang (DigiVfx) Jan Neumann, Pat Hanrahan, Alexei Efros

Transcription

1 Acquisition Some slides from: Yung-Yu Chuang (DigiVfx) Jan Neumann, Pat Hanrahan, Alexei Efros

2 Image Acquisition Digital Camera Film

3 Outline Pinhole camera Lens Lens aberrations Exposure Sensors Noise Sensing color

4 Camera trial #1 scene film Put a piece of film in front of an object. source: Yung-Yu Chuang

5 Pinhole camera pinhole camera scene barrier film Add a barrier to block off most of the rays It reduces blurring The pinhole is known as the aperture The image is inverted

6

7 Modeling projection The coordinate system Put the optical center (Center Of Projection) at the origin Put the image plane (Projection Plane) in front of the COP The camera looks down the negative z axis we need this if we want right-handed-coordinates

8 Modeling projection Projection equations Compute intersection with PP of ray from (x,y,z) to COP Derived using similar triangles (on board) We get the projection by throwing out the last coordinate:

9 In Homogenous Coordinates Projection is a matrix multiply using homogeneous coordinates:

10 Projection

11 Projection

12 Pinhole camera pinhole camera scene barrier film Add a barrier to block off most of the rays It reduces blurring The pinhole is known as the aperture The image is inverted

13 Shrinking the Pinhole Aperture Why make the aperture as small as possible? Less light gets through Diffraction effect

14 Shrinking the Pinhole Aperture Sharpest image is obtained when: pinhole diameter d = 2 f λ Example: If f = 50mm, λ = 600nm (red), d = 0.36mm

15 High-end commercial pinhole cameras ~$200

16 Pinhole Images Exposure 4 seconds Exposure 96 minutes Images copyright 2000 Zero Image Co.

17 Outline Pinholecamera Lens Lens Aberrations Exposure Sensors Noise Sensing color

18 Adding a lens scene film

19 Adding a lens circle of confusion scene lens film A lens focuses light onto the film There is a specific distance at which objects are in focus Other points project to a circle of confusion in the image

20 (Thin) Lens Thin lens equation: Any object point satisfying this equation is in focus

21 Circle of Confusion aperture Blur Circle, b aperture diameter d o i o' i' Blur Circle Diameter b : Derive using similar triangles b = d i' ( i ' i)

22 Aperture controls Depth of Field f / 5.6 Changing the aperture affects depth of field Smaller aperture: better DOF increased exposure f / 32

23 Depth of Field

24 Thick Lens Corrects aberrations Change zoom

25 Field of View (Zoom)

26 Field of View (Zoom)

27 FOV depends on Focal Length ϕ/2 d/2 f f d = image size 2

28 FOV depends on Focal Length d = image size 2

29 FOV depends on Focal Length d = image size 2 Smaller FOV larger Focal Length

30 FOV depends on Focal Length focal point image size For closer objects: if focal point is larger but image distance and size remain unchanged the objects in focus are more distant.

31 Simplified Zoom Lens in Operation From wikipedia

32 Outline Pinholecamera Lens Lens aberrations Exposure Sensors Noise Sensing color

33 Radial Distortion No distortion Pin cushion Barrel Radial distortion of the image Caused by imperfect lenses Deviations are most noticeable for rays that pass through the edge of the lens

34 Correcting radial distortion from Helmut Dersch

35 Radial Distortions No Distortion Barrel Distortion Pincushion Distortion Radial distance from Image Center: r u = r d + k 1 r d 3 r u = undistorted radius r d = distorted radius

36 Correcting Radial Distortions Before After

37 Vignetting photo by Robert Johnes

38 Vignetting L3 L2 L1 B A More light passes through lens L3 for scene point A than scene point B Results in spatially non-uniform brightness (in the periphery of the image)

39 Chromatic Aberration longitudinal chromatic aberration (axial) transverse chromatic aberration (lateral)

40 Chromatic Aberration longitudinal chromatic aberration (axial) Canon EF 85/1.2 L USM transverse chromatic aberration (lateral) Cosina 20 mm Good lens Carl Zeiss Distagon 2.8/21

41 Chromatic Aberration Near Lens Center Near Lens Outer Edge

42 Spherical aberration Rays parallel to the axis do not converge Outer portions of the lens yield smaller focal lengths

43 Spherical aberration Spherical lens are free of chromatic aberration but do not focus well. Parabolic lens does.

44 Spherical aberration

45 Astigmatism Different focal length for inclined rays

46 Astigmatism Change in size and shape of blur patches

47 Coma point off the axis depicted as comet shaped blob

48 Lens Glare Stray inter-reflections of light within the optical lens system Happens when very bright sources are present in the scene Reading:

49 Outline Pinholecamera Lens Lens aberrations Exposure Sensors Noise Sensing color

50 Exposure Two main parameters: Aperture (in f stop) Shutter speed (in fraction of a second)

51 Shutter

52 Leaf Shutter Advantages Uniform illumination Entire frame illuminated at once Disadvantages Illumination not constant over time Limitations on shutter speed

53 Focal Plane Shutter Advantages Cost effective (one shutter needed for all lenses Can achieve very fast shutter speeds (~1/10000 sec) Disadvantages May cause time distortion

54 Aperture Aperture is the diameter of the lens opening, usually specified by f-stop, f/d, a fraction of the focal length. f/2.0 on a 50mm means that the aperture is 25mm f/2.0 on a 100mm means that the aperture is 50mm When a change in f-stop occurs, the light is either doubled or cut in half. Lower f-stop, more light (larger lens opening) Higher f-stop, less light (smaller lens opening)

55 Aperture Constant speed.

56 Shutter speed Rule of Thumb 1 step in the shutter speed scale corresponds to 1 stop in the aperture scale. Handheld camera: shutter speed = 1 / f Stabilized gear: 2-3 shutter speeds slower Typical speeds: 1/1000 s, 1/500 s, 1/250 s, 1/125 s, 1/60 s, 1/30 s, 1/15 s, 1/8 s, 1/4 s, 1/2 s 1 s

57 Aperture vs. Shutter Depth of Field f/22 f/4 Small Aperture Large Aperture (Low speed) (High speed)

58 Aperture vs. Shutter Motion Blur 1/30 f/22 1/6400 f/2.5 Small Aperture (Low speed) Large Aperture (High speed)

59 Dynamic Range

60 Short exposure Real world radiance Picture intensity dynamic range Pixel value 0 to 255

61 Long exposure Real world radiance Picture intensity dynamic range Pixel value 0 to 255

62 Varying shutter speeds

63 HDR High Dynamic Range

64 Outline Pinholecamera Lens Lens aberrations Exposure Sensors Noise Sensing color

65 Spatial Sampling When a continuous scene is imaged on the sensor, the continuous image is divided into discrete elements - picture elements (pixels)

66 Spatial Sampling

67 Sampling The density of the sampling denotes the separation capability of the resulting image Image resolution defines the finest details that are still visible by the image We use a cyclic pattern to test the separation capability of an image 0 x

68 Sampling Frequency

69 Sampling Frequency

70 Nyquist Frequency Nyquist Rule: To observe details at frequency f (wavelength d) one must sample at frequency > 2f (sampling intervals < d/2) The Frequency 2f is the NYQUIST frequency. Aliasing: If the pattern wavelength is less than 2d erroneous patterns may be produced. 1D Example: 0

71 Aliasing - Moiré Patterns

72 Quantization

73 Digitizers (Quantization)

74 Image Sensors CCD Charge Coupled Device CMOS Complementary Metal Oxide Semiconductor

75 MOS (Metal Oxide Semiconductor) Photosensitive element Charge acquired depends on the number of photons which reach the element CCD devices are arrays of this basic element

76 Photoelectric Effect photon photon Increasing energy Conduction Band Valence Band 1.26eV Hole Electron Thermally generated electrons are indistinguishable from photo-generated electrons Dark Current.

77 Quantum Efficiency Not every photon hitting a pixel creates a free electron Quantum Efficiency (QE) = electrons collected / photons hitting the pixel QE heavily depends on the wavelength QE [%] blue green red QE < 100% degrades the SNR of a camera lambda [nm] SNR e = QE SNR p Typical max QE values : 25% (CMOS) 60% (CCD)

78 CCD (Charge Coupled Device) Boyle and Smith, 1969, Bell Labs Converts light into electrical signal (pixels)

79 CCD Readout Bucket Brigade Integration Charge Shift and Read-out Charge Amplifier

80 CMOS (Complementary Metal-Oxide Semiconductor) Each pixel owns its own charge-voltage conversion No need for external shutter (electronic shutter) The chip outputs digital bits Much faster than CCD devices

81 CCD vs. CMOS Mature technology Specific technology High production cost High power consumption Higher fill factor Blooming Sequential readout Recent technology Standard IC technology Cheap Low power Less sensitive Per pixel amplification Random pixel access Smart pixels On chip integration with other components

82 Sensor Parameters Fill factor The area in the sensor that is truly sensitive to light Shift registers and others can reduce it up to a 30% Well capacity The quantity of charge that can be stored in each pixel Close relation with pixel dimensions Integration time: Exposure time that is required to excite the CCD elements Depends on the scene brightness Acquisition time: Time needed to transfer the information gathered by the CCD Depends on the number of pixels in the sensor

83 Fill Factor The ratio between the light sensitive pixel area and the total pixel area. Total pixel area: 5µm x 5µm Photo Sensing Area Fill factor 40%

84 Outline Pinholecamera Lens Lens aberrations Exposure Sensors Noise Sensing color

85 Sensor noise Noise Sources Photon noise / Shot Noise (Poisson) Dark Noise (Constant) Thermal noise (Poisson) Resetting (fixed) Read-out noise Blooming (After T. Lomheim, The Aerospace Corporation)

86 Photon Shot Noise Light is quantum in nature Noise due to statistics of the detected photons themselves The probability distribution for N photons to be counted in an observation time T is Poisson P ( N F, T ) = ( FT ) N FT e N! F = fixed average flux (photons/sec)

87 Poisson Distribution, FT = N Poisson Distribution : std equals sqrt of Mean. σ = FT = 2 shot N photons

88 Poisson Distribution, FT = N

89 Poisson Distribution, FT = N

90 Poisson Distribution, FT = As FT grows, Poisson distribution approaches Gaussian distribution. Signal To Noise (SNR) Increases with Mean. SNR = N σ 2 2 N = N N 2 = N

91 Photon Noise More noise in bright parts of the image You can identify the white and black regions from the noise image

92 Photon Noise Photon Noise more noticeable in dark images.

93 Dark Current Noise Electron emission when no light Dark current noise is high for long exposures To remove (some) of it Calibrate the camera (make response linear) Capture the image of the scene as usual Cover the lens with the lens cap and take another picture Subtract the second image from the first image

94 Dark Current Noise Original image + Dark Current Noise Image with lens cap on Result of subtraction Copyright Timo Autiokari,

95 Sensor noise ideal relationship between electrons and impinging photons CCD capacity limit Light Signal (QE = 50) Photon Noise (QE = 50)

96 Outline Pinholecamera Lens Lens aberrations Exposure Sensors Noise Sensing color

97 Sensing Color light beam splitter 3 CCD Bayer pattern Foveon X3 TM

98 Multi-Chip wavelength dependent

99 Field Sequential

100 Field Sequential

101 Field Sequential

102 Color Filter Array (CFA) Fuji Corporation

103 Color filter array Bayer pattern Color filter arrays (CFAs)/color filter mosaics

104 Bayer s pattern

105 Demosaicking CFA s

106 Color filter array red green blue output

107 X3 technology red green blue output

108 Foveon X3 sensor Bayer CFA X3 sensor

109 Cameras with X3 Sigma SD14 Polaroid X530 Hanvision HVDUO 5M/10M Out of production

110 Color processing After color values are recorded, more color processing usually happens: White balance Non-linearity to approximate film response or match TV monitor gamma

111 White Balance warmer +3 automatic white balance

112 Gamma Correction Gamma correction applied by the converter redistributes the pixel luminance values so that limited brightness range captured by the sensor is mapped to match our eye s sensitivity. Gamma = 2.2 is a good match to distribute relative brightness in a print or in a video display.

113 Gamma =1 vs. Gamma = 2.2

114 Space of response curves

115 Outline Pinholecamera Lens Lens aberrations Exposure Sensors Noise Sensing color Summary

116 Camera pipeline

117 Sensor Response The response of a sensor is proportional to the radiance and the throughput

118 Measurement Equation Scene Radiance Optics Pixel Response Shutter L(x,ω,t,λ) (x,ω ) = T(x,ω,λ) P(x,λ) S(x,ω,t)

119 Degradation Due To Sampling Sampling in space Pixels Sampling in intensity Quantization Sampling in color Color Filter Array (CFA) Sampling in time Exposure Sampling in frequency Lens and pixel PSF (point-spread-function)