Machine Vision: Image Formation

Transcription

1 Machine Vision: Image Formation version 1.1 MediaRobotics Lab, March 2008 References: Forsyth / Ponce: Computer Vision Horn: Robot Vision Kodak CCD Primer, #KCP Adaptive Fuzzy Color Interpolation, Journal of Electronic Imaging Vol. 11(3), July Trade Association: 1394 Standards and Specifications Summary

2 A metal ring is heated and observed with a CCD camera. The ring will appear bright on the screen before a red glow can be seen by the unaided eye.

3 Stefan-Boltzman Law : total amount of radiation (F) emitted per unit time from a unit area of a black body only depends on its temperature (T) according to the following formula F = b *T^4

4 Up until the late 1800's the wave picture of light was the prevalent theory, as it could explain most of the experiments done on light. An exception was associated with blackbody radiation, which is the characteristic radiation that a body emits when heated. It was known that this radiation changes in nature as the temperature changes, and experiments on ``blackbodies'' (perfect absorbers and emitters) I theory: I ~ 1/lambda^4 experiment lambda In 1900 Planck devised a theory of blackbody radiation which gave good agreement for all wavelengths. In this theory the molecules of a body cannot have arbitrary energies but instead are quantized - the energies can only have discrete values. The magnitude of these energies is given by the formula E = n*h*f

5 e E1 e frequency = de / h h: Planck s constant m2 kg / s E2

6

7 .. Maxwell >> Hertz/Planck >> Einstein >> Feynman 4hf de = h*f 3hf 2hf 1hf 0 E = h*f = h*c / lambda h*f = emission energy ( "Work Function W" ) + 1/2 mv^2 h = Planck's Constant = 6.63 x Js v = max speed

8

9

10 Rods Cones Very numerous, about 120 million 6-7 million on the retna sensitive receptor, low-light color sensitive cannot discriminate color three different cone receptors concentrated near optic center known as the macula

11

12 Visual Cortex Visual cortex is the term applied to both the primary visual cortex (also known as striate cortex or "V1" and upstream visual cortical areas also known as cortical areas V2, V3, V4, V5. The visual cortex occupies about one third of the surface of the cerebral cortex in humans. It is thought to be divided into as many as thirty interconnected visual areas, but at the present time there is good evidence for only 4 of these areas, V1, V2, V3 and V5. The first cortical visual area, the one that receives information directly from the lateral geniculate nucleus, is the Primary Visual Cortex, or V1.

13

14 V1 The correspondence between a given location in V1 and in the subjective visual field is very precise: even the blind spots are mapped into V1. In human and animals with a fovea in the retina, a large portion of V1 is mapped to the small, central portion of visual field, a phenomenon known as cortical magnification. The neuronal V1 responses can discriminate small changes in visual orientations, spatial frequencies and colors. Furthermore, individual V1 neurons in human and animals with binocular vision have ocular dominance, namely tuning to one of the two eyes. In the spatial domain, the functioning of V1 can be thought of as similar to many spatially local, complex Fourier transforms. Theoretically, these filters together can carry out neuronal processing of spatial frequency, orientation, motion, direction, and speed. V2 V2 is the second major area in the visual cortex, and first region within the visual association area. It receives strong feedforward connections from V1 and sends strong connections to V3, V4, and V5. It also sends strong feedback connections to the V1. Cells in V2 are tuned to simple properties such as orientation, spatial frequency, and color. The responses of many V2 neurons are also modulated by more complex properties, such as the orientation of illusory contours and whether the stimulus is part of the figure or the ground.

15 V4 V4 is the first area in the ventral stream to show strong attentional modulation. Like V1, V4 is tuned for orientation, spatial frequency, and color. Unlike V1, it is tuned for object features of intermediate complexity, like simple geometric shapes (form recognition). Visual area V4 is not tuned for complex objects such as faces. Recent work has shown that V4 exhibits long-term plasticity, is gated by signals coming from the frontal eye fields, shows changes in the spatial profile of its receptive fields with attention, and encodes hazard functions. V5 Visual area V5 appears to process complex visual motion stimuli. It contains many neurons selective for the motion of complex visual features such as line ends, corners (complex objects, such as faces).

16

17

18 Human Physiology in Space" by Barbara F. Lujan and Ronald J. White, 1994

19 Additive color involves the use of colored lights. When combined, the additive primary colors produce the appearance of white. The importance of RGB as a color model is that it relates very closely to the way we perceive color with the r g b receptors in our retinas. Television and computer monitors create color using the primary colors of light. Each pixel on a monitor screen starts out as black. When the red, green and blue phosphors of a pixel are illuminated simultaneously, that pixel becomes white.

20 The CIE 1931 "Standard Observer" is for a 2-degree field of observation based on tables of Guild and Wright. The CIE 1964 "Standard Observer" is for a 10-degree field from the work of Stiles and Burch, and Speranskaya. The experiments leading to the 1931 standard observer were performed using only the fovea, which covers about a 2-degree angle of vision. The 1964 supplementary standard observer was based on color-matching experiments using a 10-degree area on the retina. CIE standard observers are averages based on experiments with small numbers (~15-20) of people with normal color vision. No real observer is probably exactly like the CIE standard observer. The 1964 work included a few foreign post-doctoral fellows but the early work included only Englishmen from the region near to London."

21 Conversion of 1931 xy coordinates to 1960 uv coordinates: u = 4x / (-2x + 12y + 3) v = 6y / (-2x + 12y + 3) In terms of the tristimulus values X, Y and Z: u = 4X / (X + 15Y +3Z) v = 6Y / (X + 15Y + 3Z)

22 Yellow Color Name RGB CODE HEX # Pale Goldenrod eee8aa Light Goldenrod Yellow fafad2 Light Yellow ffffe0 Yellow ffff00 Gold ffd700 Light Goldenrod eedd82 Goldenrod daa520 Dark Goldenrod b8860b Sample the RGB color model could not reproduce all spectral light without introducing the effect of negative RGB values

23 99CC 66CC 33CC CC99 CC66 CC33 00CC CC00 FFFF 99FF 66FF 33FF FF99 FF66 FF33 00FF FF CCC CCF FFC C00 F00 CC FF HSL Hue, Saturation, Luminescence

24

25

26 f

27 1 1 1 = image object focal length Fstop = focal length / diameter of lens Fstop increments with a doubling of light capture capacity (which is a function of area): n 2 = 1, 1.4, , 4, 5.6, 8, 11, 16, 22, 32 Fstop = 2 aperture for n = 0,1,2,3,4..

28 Typically lenses consist of a number of lens elements with different thicknesses and curvatures, designed so that the combination corrects the imaging defects (aberrations). The focal length of a thick lens is measured from two planes called the principal planes. The thick lens acts as though it were a thin lens placed at the entrance of the lens when considered from the object side, and at the exit of the lens when considered from the image side. The principal planes serve as the reference for the location of the front focal point, back focal point, object and image positions. The lens equation, for the above simple example, operates as though the space between the planes does not exist. In reality, the planes can be crossed inside of the lens or lay entirely outside the physical boundaries of the lens.

29 Field of View (FOV) Is the object area that is focused by the lens onto the image sensor. Typically, the FOV (h x v) should be slightly larger than an area containing all desired features. The FOV can be adjusted by adjusting the camera s distance from the object (working distance) the greater the distance, the larger the FOV. It can also be adjusted by changing the focal length of the lens the longer the lens focal length, the smaller the FOV. The FOV can be calculated using following equations: H = (d0*h/f)-h -> h = - H / (1- d0 / f) V = (d0*v/f)-v -> v = - V / (1- d0 / f) The magnification factor is: H / h or V / v

30 Resolution is the ability of a lens to distinguish two features that are close together. Also, a lens with high resolution will show an edge transition in fewer pixels than a lens with low resolution.

31 Panoromic Imaging:

32 Image carriers Niepce, 1816? paper soaked in silver chloride in camera obscura Daguerre, 1826 mercury fumes on silver platting on copper Legray/Archer, 1850 wet-plate negative/positive process Maddox, 1970 gelatin process (no immediate development req) Eastman, 1889 photographic film Lumiere, 1908 color photographic film CCD, 1970s charge coupled devices (originally designed for low cost memory devices)

33 NTSC (National Television Standards Committee) video format: signal is an interlaced composite video signal of 525 lines and 60 fields per second (30 frames per second), with a bandwidth limited to 4 MHz to fit into a 6 MHz broadcast television channel without interfering with adjacent channels. Starting with rgb data three signals are generated: Y = 0.30R G B (intensity) I = 0.60R G -0.32B Q = 0.21R G B I and Q are 90 degrees out of phase; When viewed in time, one line of a video signal will have 3 characteristics: An average level which fixes the luminance(b/w) signal, an oscillation at 3.58MHz whose magnitude fixes the saturation and whose phase fixes the hue of the color Lines are separated by horizontal sync signal, pages by vertical sync signal; both used to synchronize camera to framegrabber/tvset

34 Time domain representation of ntsc video signal (one horizontal line) V(max) falling edge Time [us] a b c active video information T (53.5 us) a) horizontal sync b) horizontal back porch (no signal) c) horizontal front porch (no signal) T(one line of frame) For NTSC, the line interval is Tl = 1/(30 *525) = 63.5µs. But the horizontal retrace takes Th = 10µs; actual time for scanning each line is T = 53.5µs. The vertical retrace between adjacent fields takes Tv = 1333µs, which is equivalent to the time for 21 scan lines per field * the number of active lines is = 483/frame. The actual vertical retrace only takes the time to scan nine horizontal lines. The remaining time (twelve scan lines) is for broadcasters wishing to transmit additional data in the TV signal (e.g., closed caption, teletext, etc.).

35

36 TABLE 1.1 PARAMETERS OF ANALOG COLOR TV SYSTEMS Parameters NTSC PAL SECAM Field rate Line number/frame Line rate (line/s) 15,750 15,625 15,625 Image aspect ratio 4:3 4:3 4:3 Color coordinate YIQ YUV YDbDr Luminance bandwidth (MHz) , Chrominance bandwidth (MHz) 1.5 (I), 0.5 (Q) 1.3 (U, V) 1.0 (U, V) Color subcarrier (MHz) (Db), 4.41 (Dr) Color modulation QAM QAM FM Audio subcarrier (MHz) , Composite signal bandwidth (MHz) ,

37 CCD: Charge Coupled Device Phillips Research Labs (Sangster/Teer) invent the BucketBrigade Device (transfers packets from one transistor to another) Bell Labs (Boyle/Smith) extend concept by inventing transport mechanism from one capacitor to a second one >> charge coupled device >> a memory device that happens to be sensitive to light JPL initiates Scientific Grade large array CCD program >>first used as image sensors in astronomy

38

39 CCD: Charge Coupled Device per 1/3 inch chip,distributed according to human color sensitivity (more green than red, blue) - light hits array of detector cells with de = h*f - light sensitive diodes, sensitive to R, or G or B band, translate the flux of light energy into electric charge. - electrons freed and stored in potential wells as charge - sequential reading of this charge and conversion to voltage - linear mapping of this voltage to light intensity - filtering and spatial interpolation to derive 3 base color bands - combine the 3 base color bands to represent any color... - scanning the chip line by line results in a video signal

40 CCD performance a function of: - wavelength sensitivity of cells - number of cells/pixels (row x column) - physical size of each pixel (6-20 um) - depth of cell: # of bits to code brightness - noise cancellation techniques

41 Noise Thermal noise: photons and thermal energy can free electrons (>> dark current) Photodiode noise: CCD is impure, imperfect, Q/E not 100% Photon noise: not time constant occurrence Electronics noise: stray capacitance vary effective voltage

42 Source: Kodak CCD Primer, #KCP-001

43 Source: Kodak CCD Primer, #KCP-001

44 Source: Kodak CCD Primer, #KCP-001

45 Charge versus time graphs for an RGB pixel inside a CCD cell

46

47 Single CCD >> spatial color interpolation - columns of alternating colors - 25% more green >>human vision more sensitive to G - each cell has an intensity value of merge data from N (4) cells to map to a colored pixel value Bayer filter

48 Fuji Film, Super CCD

49 Source: Kodak CCD Primer, #KCP-001

50 On-chip micro lens Color filter Photo shield Poly Silicon Register Sensor SONY HAD sensor, schematic

51 AMTEL TH7887A Area Array CCD Image Sensor 1024x1024

52 Source: Kodak CCD Primer, #KCP-001

53

54 Frame grabber: The frame grabber digitizes the periodic signal into a 2D, rectangular array N x M of integer values, stored in the frame buffer. (conversion from ccd cell values to pixel values)

55 Camera: Image formation (lens), image capture (CCD) Reprinted from the January 2001 issue of P HO TONIC S S P E C TR A

56 CMOS: Complementary Metal Oxide Semiconductor advantages: - easier to fabricate (higher volume, higher yield, less demanding A/D and clock circuitry.) -> cheap - lower voltage, lower power consumption -> good for battery operated devices - can be integrated to (other) on board circuitry -> microprocessor control disadvantages: - higher (stationary noise) -> lower image quality - smaller effective sensor area -> lower sensitivity

57 IEEE 1394 (also known as Fire-Wire and i.link ) is an established standard for high-speed digital data transmission. IEEE 1394 combines several advantages if compared with other bus solutions, like high bandwidth (currently up to 800 MBit/s), ease of use without assignment of node IDs or elaborate set up by the enduser, hot-plug capability, and a flexible network topology. Like any other communication standard, IEEE 1394 consists of several protocol layers. The two lower layers (Physical Layer and Link Layer) are realized by application-specific Integrated Circuits, whereas higher layers are implemented in software. This comprises the basic IEEE 1394 Protocol Stack as well as application dependent protocols that use the basic IEEE 1394 stack.

58 IEEE1394 DCAM vs other formats

59

60

61 Device type Compliance Interface Speed Sensor type Scanning Effective pixels Pixel shape Color Resolution Optics Focus Iris Horizontal view angle Vertical view angle Coating Picture Modes modes Resolutions Codings IIDC FireWire Digital Camera v 1.04 FireWire, 2 ports ( 6 pins ) 400 Mbps SONY Wfine* ¼" CCD progressive (H x V) 659 x 494 (H x V) square, 5.6 x 5.6 μm Yes, RGB filtering, Bayer TV-lines (H x V) 480 x 480 f 4.65 mm built-in Manual, from 5 mm to infinite Fixed anti-reflective progressive VGA uncompressed, selection by FireWire link 640x480, 320x240, 160x120 YUV, RGB, Monochrome

62

63 CMUcam (Robotics Institute at Carnegie Mellon University) : Scenix (ubicom) SX28AX clocked at 75MHz to retrieve pixel data from an Onmivision OV6620 sensor (352x488 pixels) contained on a C3088 module. With a maximum speed of 16.7fps the camera can track the position and size of a colourful or bright object, measure the RGB or YUV statistics of an image region or automatically acquire and track the first object it senses. (Rs232 or TTL serial port communication protocol; = 1.4Watt)