Imaging Overview. For understanding work in computational photography and computational illumination

Transcription

1 Imaging Overview For understanding work in computational photography and computational illumination

2 Light and Optics Optics The branch of physics that deals with light Ray optics Wave optics Photon optics Light The visible portion of the electromagnetic spectrum; visible radiant energy; EM radiation We re also interested in other parts of the EM spectrum (e.g., X-rays, UV rays, infrared) Visible the portion of the EM spectrum that the human eye responds to Not the same for all animals For humans, ~ nm wavelength 2

3 Light Light has wave and particle characteristics Travels in a straight line within a medium, at a constant velocity Travels at 300,000 kps in a vacuum (c) Absolute maximum speed Slower in materials Sources emit (radiate) light Surfaces reflect, refract, transmit, scatter, and absorb light 3

4 EM Spectrum Short wavelength High frequency High energy Long wavelength Low frequency Low energy Energy, frequency, wavelength Color f (wavelengths) Intensity f (# of photons/sec) White light? Black? Visible light ~ nm 4

5 Light sources: the sun The light emitted by the sun falls within the visible region and extends beyond the red (into the infrared) and the ultraviolet (UV) Most of the UV and IR is absorbed by the earth s atmosphere Glass absorbs some UV Water absorbs some light 5

6 6

7 Other typical light sources Incandescent light source Produced by the temperature of an object. Fluorescent light source Light is generated as a result of electrical current traveling through a charged gas A glass tube is filled with mercury vapor (and others such as sodium) and the electrodes are connected to an alternating current (AC) source The electric source ionizes the atoms in the tube which emit light primarily in the UV range The inside of the tube is coated with phosphors which absorb UV light and produce visible light. 7

8 Power spectrum of common light sources Incandescent sources 8

9 The Photoelectric Effect Photoelectric effect: the interaction of a photon with an electron in a solid Described by Albert Einstein in 1905 Quantized energy levels; introduced the term photon Earned him the Nobel Prize The photon transfers all its energy to an electron which leaves the surface of the material 9

10 Polarization How light waves travel Direction Frequency Phase Orientation Light waves are transverse i.e., the wave motion is perpendicular to the direction of travel Sounds waves are longitudinal (parallel to the direction of travel) Some materials selectively transmit or reflect light based on the wave orientation Polarization Human vision does not distinguish between polarized and unpolarized light But machine vision systems can and sunglasses do! 10

11 Polarization (cont.) 11

12 Nonpolarized sunglasses (filters) decrease the intensity of everything by the same amount. Polarized sunglasses (filters) can selectively eliminate the reflection from light coming from above the water surface. 12

13 Reflection and Refraction At the interface between media, there is reflection and refraction (and absorption) Reflection light is reflected from the surface Refraction light proceeds through the material in a different direction, depending on the index of refraction 13

14 Reflection The angle of incidence equals the angle of reflection Rays and normal are coplanar 14

15 Reflection (cont.) Specular and diffuse reflection Specular Diffuse 15

16 Refraction Refraction occurs toward the normal when light enters a more dense medium Incident ray Normal Reflected ray Snell s Law for refraction n 1 1 n 1 sin 1 = n 2 sin 2 n 2 2 Index of refraction for a medium: n = c / v m Refracted ray 16

17 Refraction (cont.) Refractive index the ratio of the velocity of light in vacuum to the velocity of light in a medium is referred to as the index of refraction (n) Medium n Vacuum 1.0 Air Water 1.33 Glass 1.5 Diamond

18 Why is a beer mug so thick? Top View 18

19 What is really there What you appear to have (from a side view) 19

20 Half-silvered mirrors and reflection One way mirror, beam splitter What s wrong with this? 20

21 Catadioptric cameras Dioptric refractive (lenses) Catoptric reflective (mirrors) Catadioptric imaging Image systems (cameras) constructed with dioptric and catoptric elements The mixing of mirrors and lenses in an optical system Using both reflection and refraction in an optical system 21

22 22

23 Lenses Two refractions (air-to-glass, glass-to-air) Rays from infinity 23

24 Focus Principle of reversibility direction doesn t matter when considering the geometry 24

25 Lens Images Lenses form real images and virtual images A real image is formed behind the optical system (opposite side from the source) A virtual image is formed in front of the optical system (same side as the source) e.g., camera e.g., magnifying lens 25

26 Lens Types Converging lens Positive refractive power Convex Converging Diverging lens Negative refractive power Concave Diverging 26

27 Focal Point of a Lens (Assumes lens in air) 1 f ( n 1) 1 r 1 1 r 2 Focal distance is a function of the index of refraction (n) and the surface radii for the two sides of the lens So which has a larger f : Which is more powerful? 27

28 Thin Lens Model The thin lens model is an ideal approximation Disregard the lens thickness Disregard blurring (rays don t really converge perfectly!) All light bending takes place at the principal plane, where the incoming rays intersect with the outgoing rays Real lens Ideal lens Principal plane 28

29 Thin Lens Equation object image z 0 z z 1 z 1 z 1 f z distance of point/object from lens origin z distance from lens origin where point/object is in focus f focal distance (constant property of the lens) 29

30 Focus Is this drawn right? object image z 0 z z 1 z 1 z 1 f Be careful about signs z can be negative! virtual image So everything in the plane Z=z will be imaged (focused) at Z=z Everything else will be out of focus (sort of) *Technically, both these statements are false, but they re close enough. 30

31 Focusing z' z We want to put the image plane here (at Z = z ) in order to get a perfectly focused image of the object 1 z 1 z 1 f 31

32 Misfocus Blur /Disk If the image plane is not exactly at Z=z, then there will be some misfocus blur 32

33 Depth of Focus/Field Blur is caused (primarily) by imaging points away from the focal plane Depending on sensor resolution, small amounts of blur may not matter All in focus Partly in focus 33

34 Depth of Focus/Field Depending on sensor resolution, small amounts of misfocus blur may not matter sensor placement resulting images 34

35 Depth of Focus/Field Depth of focus the distance of the imager along the axis where a single object point produces a focused image point Depth of field the distance of the object point along the axis Sensor spacing d 35

36 Depth of Focus Aperture size affects depth of focus 36

37 Depth of field/focus 37

38 Aperture An aperture limits the area that light can pass through object image What does a small aperture do to the misfocus blur? 38

39 Aperture effect on misfocus blur A smaller aperture reduces misfocus blur object image plane This produces a larger depth of focus (depth of field) 39

40 Lens Aberrations Aberrations: Systematic deviations from the ideal path of the image-forming rays Causes blur and other problems in the image Good optical design minimizes aberrations (but doesn t eliminate them!) Typically small effect for paraxial rays Types: Spherical aberration Coma Astigmatism Radial distortion Chromatic aberration Radial distortion is typically significant and must be accounted for 40

41 Lens Aberrations Longitudinal and lateral (transverse) effects Ideal Real 41

42 Chromatic aberration examples High quality lens Low quality lens Color artifacts are visible 42

43 Chromatic aberration examples Near Lens Center Near Lens Outer Edge

44 Radial Distortion Variation in the magnification for object points at different distances from the optical axis Effect increases with distance from the optical axis Straight lines become bent! Independent of aperture size Two main descriptions: barrel distortion and pincushion distortion Can be modeled and corrected for Correct Barrel Pincushion 44

45 45

46 Correcting for radial distortion Original Corrected 46

47 Correcting for radial distortion 47

48 Modeling radial distortion Radial distortion can be modeled and corrected for Once per lens: doesn t depend on specific imaging conditions Estimate distortion parameters using a standard pattern Then warp pixels (x n ', y n ') to (x d ', y d ') κ 1 and κ 2 are estimated experimentally

49 Vignetting 49

50 Vignetting Vignetting is the darkening of the corners of an image relative to its center, due to several possible causes Results in non-uniform brightness away from the image center Geometry of multiple lenses and/or apertures Cos 4 falloff E d 4 f 2 4 cos L Lost rays Angle from the optical axis 50

51 Lens lessons No lens is perfect Even a perfectly-shaped lens does not focus perfectly! Monochromatic light images best Good optical design (multiple lenses) and good craftsmanship (careful and precise lens grinding) can reduce aberration effects Paraxial rays are your best bet Some effects are only noticeable at high resolution, some only with a wide-angle lens The lens system determines where the image plane will be placed and the field of view (horizontal and vertical) of the image but after that, we can ignore it with respect to imaging geometry 51

52 Julian Beever sidewalk art 52

53 53

54 54

55 Sensor response model Sensor responses to a given irradiance: 55 d R E c d R E c d R E c ) ( ) ( ) ( ) ( ) ( ) ( Sensor responses Spectral power distribution of light energy (irradiance * area) Sensor sensitivities

56 Responses to a source λ E(λ) c 1 R 1 (λ) c 2 R 2 (λ) c 3 R 3 (λ) Sensor responses d R E c d R E c d R E c ) ( ) ( ) ( ) ( ) ( ) (

57 The Light Source E(λ) E() is the spectral power distribution Power at each wavelength Multiple sources are additive λ Spectral power distribution can be measured by a spectroradiometer Typically smooth enough to approximate by a series of samples every 5 or 10 nm (from 380 to 730 nm) Vector of dimension 71 or 36 Each element of the vector is a sample at a single wavelength (ideally) 57

58 Spectroradiometer Movable slit Source Sensor Spectral range: nm Wavelength accuracy: ±0.5 nm Repeatability: ±0.2 nm Resolution: 0.2 nm 58

59 Is this what human vision or cameras do? Are they spectroradiometers (or spectrophotometers )? No, color vision doesn t do this Rather than 36 or 71 or 1751 spectral samples at a given point, we have three broad samples (in normal color vision) d E c d E c d E c 730) ( ) ( ) ( ) ( 380) ( ) ( Spectroradiometer d R E c d R E c d R E c ) ( ) ( ) ( ) ( ) ( ) ( Cones/Sensors 59

60 Digital representation Monochromatic light? E(λ) and R i (λ) can be represented by discrete samples E(λ) Vector e (or s) R i (λ) λ Vector r i λ 60

61 Color at a point? Cones/sensors are not co-located their responses are at different locations. (Same thing for color displays.) How can we define color at an (x, y) location? Spatial integration tradeoff of spatial resolution and color fidelity (like halftoning and dithering for display) 61

62 Sensor, Imager, Pixel Not necessarily the resulting image size! An imager (sensor array) typically comprises n x m sensors 320x240 to 7000x9000 or more (high end astronomy) Sensor sizes range from 15x15m down to 3x3 m or smaller Each sensor contains a photodetector and devices for readout Technically: Imager a rectangular array of sensors upon which the scene is focused (photosensor array) Sensor (photosensor) a single photosensitive element that generates and stores an electric charge when illuminated. Usually includes the circuitry that stores and transfers it charge to a shift register Pixel (picture element) atomic component of the image (technically not the sensor, but ) However, these are often intermingled 62

63 Imagers What do they do? Charge generation Intercepting photons and generating signal electrons through the photoelectric effect Quantum efficiency (QE) ideally 100% at all wavelengths However: absorption, reflection, transmission losses Charge collection Accurately represent an image from generated electrons at each sensor Number of sensors on the chip, well capacity, variation in sensitivity across pixels, loss to neighbors (thermal diffusion) Charge transfer Getting the charge to the output amplifier Electron traps result from flaws in design, processing, or the silicon CMOS avoids many transfer issues pixel addressable Charge measurement Conversion of signal charge to voltage (capacitor connected to output amplifier) 63

64 Fill Factor The fill factor of a sensor is the fraction of the sensor area occupied by the photodetector Ranges typically from 20% to 90% Larger is better why? Because the sensitive area of a CCD sensor is only a fraction of its total area, on-chip lenses are used to concentrate the light of the optical image into the sensor area of the pixel and thus increase sensitivity (and effective fill factor) 64

65 Microlens: Sony example 65

66 Dynamic Range What range of incident light levels is represented correctly by the camera? Consider the ratio of sensor well depth to readout noise E.g., well depth of 85,000 electrons, readout noise of 12 electrons gives a dynamic range of 20 log (85,000/12) = 77 db How many digitization levels are appropriate for this sensor? 85,000/12 = bits (8192 levels) is sufficient How many bits for a camera with dynamic range of 48 db? 48 = 20 log x x = log -1 (48/20) = 251 levels 8 bits 66

67 Color Sensors CCD and CMOS chips do not have any inherent ability to discriminate color (i.e., photon wavelength/energy) They sense number of photons, not wavelengths Essentially grayscale sensors need filters to discriminate colors! Two approaches to sensing color 3-chip color: Split the incident light into its primary colors (usually red, green and blue) by filters and prisms Three separate imagers Single-chip color: Use filters on the imager, then reconstruct color in the camera electronics Filters absorb light (2/3 or more), so sensitivity is lower 67

68 3-Chip Color The R, G, and B sensors are identical This isn t how the eye works! Lens To R sensor array Color filters Incident light To G sensor array Neutral density filter Infrared filter Low-pass filter To B sensor array Prisms + half-silvered mirrors How much light energy reaches each sensor? 68

69 Single-Chip Color Uses a mosaic color filter Incident light Each photosensor is covered by a single filter Must reconstruct (R, G, B) values via interpolation To sensor array Color filter R( x, y) G( x, y) B( x, y) f f f R G B ( I( x dx, y dy)) ( I( x dx, y dy)) ( I( x dx, y dy)) 69

70 Rolling shutter A drawback for most CMOS sensors is the rolling shutter 70

71 Resolution Kinds of resolution How many picture elements (pixels)? (Spatial resolution) How many sensors? (Color resolution) # of bits (related to SNR) (Depth resolution) Sensitivity (Signal resolution) Noise (Limits resolution) 71

72 Very high resolution CMOS sensors UDTV (super hi-vision) sensor: 7680 x Mpixels 60 fps 2 Gp/s Sony 120MP sensor 72

73 Non-standard imaging sensors In a typical RGB camera, the sensitivity curves of three sensor types are similar to the human cone sensitivities But they don t have to be! Cameras can have: Fewer or more than three sensor types Different sensitivity curves (e.g., non-overlapping) Sensors that respond to non-visible light (UV and IR) 73

74 Imaging the invisible Many useful non-visible light images can be formed: Radio images Used in astronomy Used to discover the cosmic microwave background radiation Terahertz imaging 100 micron to 1 mm wavelength Used in non-destructive testing and evaluation in many industries Non-ionizing, possible for airport security scanning Various medical imaging technologies X-rays, CT, MRI, fmri, PET, SPECT, ultrasound, etc. A radio telescope T-rays screening 74

75 Infrared imaging Three primary IR wavebands (one version) Near-infrared (shortwave IR) 700 nm to 3000 nm Indium gallium arsenide (InGaAs) detector technology Midwave IR 3000 nm to 6000 nm Indium antimonide (InSb) detectors Longwave IR 6000 nm to nm Quantum-well infrared photodetector (QWIP) array cameras Thermal energy (heat) is detectable as electromagnetic radiation IR cameras (also called thermal imagers) create thermal images 75

76 Near IR example Visible light image Near-IR image 76

77 Infrared (IR) sensors and cameras Used quite a bit in industrial, military, and law enforcement applications, e.g., by detecting A gun under clothing A suspect hiding behind bushes or in a dark alley A recently driven car (warmth under hood) Objects hidden in interior walls, hidden compartments in cars People through smoke-filled areas Hot spots in welding, process control People, buildings, etc. from missiles And for night vision 77

78 Sensing emitted or reflected energy? Thermal imaging vs. Image intensification Is the source emitting or reflecting EM energy? Midwave and longwave IR emitting, thermal imaging Near-IR could be either (usually reflecting) Intensified, reflected energy Thermal IR 78

79 Infrared images All images pseudocolored 79

80 All images pseudocolored 80

81 RGB Infrared 81