Outline. Image formation: the pinhole camera model Images as functions Digital images Color, light and shading. Reading: textbook: 2.1, 2.2, 2.

Transcription

1 Image Basics 1

2 Outline Image formation: the pinhole camera model Images as functions Digital images Color, light and shading Reading: textbook: 2.1, 2.2, 2.4 2

3 Image formation Images are acquired through some device from some real-world scene, anatomy, etc 3

4 Images acquired with cameras Basic abstraction: pinhole camera Abstract camera model - box with a small hole in it Pinhole cameras work in practice The pinhole perspective projection equations were discovered by Brunelleschi, in the 15 th century. First pinhole camera: 16 th century. still represent the most used theoretical camera model 4

5 Camera obscura : each point on the image plane sees light from only one direction, the one that passes through the pinhole. The pinhole is the center of projection through which all light passes.! Perspective projection creates inverted images.! It is sometimes convenient to consider a virtual image in a plane lying in front of a pinhole and symmetric to the image plane with respect to the pinhole point. 5

6 Pinhole optics Using ray-tracing, we see that only a narrow light beam passes through a pinhole A) In a wide pinhole, light from the source spreads across the image, making it blurry. B) In a narrow pinhole, only a small amount of light is let in. The image is sharper. Small apertures require longer exposure times. The sharpness is limited by diffraction. 6

7 Pinhole too big - many directions are averaged, blurring the image Pinhole too smalldiffraction effects blur the image. Generally, images from pinhole cameras are dark, because a very small set of rays from a particular point hits the screen. 7

8 Pinhole optics summary Pinhole optics focuses images: - without lenses - with an infinite depth of field - Depth of field = The distance between the nearest and farthest objects that appear in acceptably sharp focus in the image. Small pinhole: - Better focus - Less light energy available from any scene point - The sharpness is limited by difraction 8

9 The perspective projection equation 9

10 Perspective effect: Far objects appear smaller than close ones! The image plane is behind the pinhole (inverted images). 10

11 Pinhole optics: Horizon and vanishing points (virtual image plane). H is the horizon line. Considering all the possible sets of parallel lines in plane Π, their intersection (vanishing points) lie on the horizon line. 11

12 Parallel lines meet at vanishing points 12

13 Image functions The image can be modeled by a function of two or three variables; f(x,y) f(x,y,z) f(x,y,t) Values in an image can be of many types: Scalars: monochromatic images; Physical significance: X-Ray, MRI, Range images Vectors: color images (R,G,B); LANDSAT images ( 7 distinct channels) 13

14 14

15 Digital images Sampling=spacing of discrete values in the domain of an image sampling rate how many samples are taken per unit of each dimension. dots per inch, etc. Quantization= spacing of discrete values in the range of an image number of bits per pixel. black and white images (1 bit per pixel), 24-bit color images, etc. Sampling and quantization are independent Shannon s sampling theorem: must sample at at least twice the highest spatial frequency in the image. Resolution: ability to discern fine detail in the image 15

16 Sensing illuminated objects Light reaches surfaces in 3D Surfaces reflect Sensor element receives light energy What counts: Light intensity Angles Surface material 16

17 The perception of an object s brightness and colour depends on: The amount of energy and its spectral distribution (various wavelengths) illuminating the object surface The spectral reflectance of the object surface (i.e. how the surface changes the received spectrum into the radiated spectrum) The spectral sensitivity of the sensor (human eye, CCD array etc) irradiated by the light energy from the object s surface 17

18 Non-white light sources (Left) a warm light source: it enhances reds and oranges while dulling blues and greens; (Middle) a neutral light source; (Right) a cool source: it enhances blues and greens while dulling reds and oranges. 18

19 Light Sources General light sources are difficult to work with. We must integrate light coming from all points on the source. 19

20 Light source models Ambient light Directional light Point light Etc. Important for computer vision but also for computer graphics; Computer Vision: known light sources allows for shadow detection and removal Computer Graphics: source light for realistic scene rendering (shadow and specularity modeling) and for facilitating 3D data visualization 20

21 Ambient light Objects not directly lit are typically still visible ceilings, undersides of desks etc. This is the result of indirect illumination from emitters, undergoing multiple reflections from intermediate surfaces The ambient light source model illuminates all surfaces equally amount reflected depends only on surface properties is the preferred model in Computer Vision 21

22 Ambient light (cont d) The ambient light reflected from a surface depends on: -The surface properties, k ambient -The intensity of the ambient light source (constant for all points on all surfaces) I reflected = k ambient I ambient The ambient light is not necessarily white! Adapted from David Luebke, Lecture notes, Introduction to Computer Graphics All object points have the same intensity in the image! 22

23 Directional Light Sources Simplifying assumption: all rays of light from the source are parallel As if the source were infinitely far away from the surfaces in the scene A good approximation to sunlight The direction from a surface to the light source is important in lighting the surface Adapted from David Luebke, Lecture notes, Introduction to Computer Graphics 23

24 Directional Light Sources The same scene lit with a directional and an ambient light source Adapted from David Luebke, Lecture notes, Introduction to Computer Graphics 24

25 Point Light Sources A point light source emits light equally in all directions from a single point The direction to the light from a point on a surface thus differs for different points: we need to calculate a normalized vector to the light source for every point we light: Adapted from David Luebke, Lecture notes, Introduction to Computer Graphics p 25

26 What happens to light when it arrives at surfaces? Consider reflection from the object surface The fraction of the incident radiation that a surface element reflects is called its albedo. Usually, the albedo is considered an intrinsic property. Albedo is low for dark surfaces and high for light surfaces 26

27 Ideal diffuse (Lambertian) reflection An ideal diffuse reflector, at the microscopic level, is a very rough surface (example: chalk) Because of these microscopic variations, an incoming ray of light is equally likely to be reflected in any direction over the hemisphere: 27

28 Lambert s Law The reflected intensity depends on the angle between the normal of the surface and the direction of the incident ray of light. 28

29 Diffuse reflection (cont d) Diffuse or Lambertian reflection distributes energy uniformly in all directions of the hemisphere centered at a surface element. Thus, it reflects light uniformly in all directions. A planar patch will have uniform brightness for all viewpoints from which that patch is visible. A surface element is not visible when n v < 0 where v is the direction towards the viewpoint From Shapiro and Stockman 29

30 Diffuse Lighting Example From Shapiro and Stockman Lambertian objects: - Vase and egg A plot of intensities along one row Shape of the plot shows that the shape of the object surface is closely related to the amount of reflected light. 30

31 Another Diffuse Lighting Example If we assume a diffuse reflection model, what can we say about the position of the lighting source? 31

32 Specular reflection Specular reflecting surfaces are very smooth at a microscopic level. Ex: polished metal, glossy car finish. They behave like a mirror, reflecting almost all the received illumination along the ray of reflection. The reflected energy will have the same spectral composition as the received light, regardless of the surface colour. Specular highlights are white when a white light source is used. They are also view dependent, since reflection obeys Snell s law. 32

33 Example of specular reflection Veggie Vision, IBM 33

34 Attenuation (darkening) with distance The energy per unit area 1 2 r Similarly, light energy reflecting off a surface element decreases in intensity with the square of the distance from which the surface is viewed. Conservation of energy: the total energy radiating from a point source of constant energy flux per unit of time through any enclosing spherical surface is the same. A S = 4πr 2 34

35 Shading Image intensities are non-uniform due to the non-uniform distances along the imaging rays. Cast shadows are caused by the fact that the objects block a certain amount of light energy (opaque assumption) Computer Vision: how to accurately detect shaded objects based on colour information Computer Graphics: how to model shading in order to create a realistic image of the object 35

36 Colour perception The perception of an object s color depends on 3 factors: - the spectrum of energy in various wavelengths illuminating the object surface - the spectral reflectance of the object surface - the spectral sensitivity of the sensor irradiated by the light energy from the object s surface. 36

37 Colour perception: example We usually think of colour as an intrinsic property of objects. However: A blue object has a surface material that appears blue when illuminated with white light. The same object will appear violet if illuminated only with red light. A blue car under intense sunlight (white) will become hot and radiate energy in the IR spectrum (which can be imaged with an IR camera) 37

38 Spectral sensitivity of the sensor A pair of visible (left) and IR (right) images from the same scene. The IR and visible cameras share approximately the same field of view 38

39 Sensors for colour perception in the human eye Three types of cones having the ability to sense three different (but overlapping) spectral regions. 39

40 Sensitivity curves for cones - blue cone is mildly sensitive to blue light nm (peak at 430 nm) - green cone is very sensitive to green light, but also sensitive to blue and red (peak at 560) - red cone (peak at 610) 40

41 The three-cone theory suppose that light in the yellow range of wavelengths strikes the retina. it activates both the green and the red cones of the retina. electrical messages are sent by both the red and the green cones to the brain. The brain recognizes that the light has activated both the red and the green cones and somehow interprets this to mean that the object is yellow. In this sense, the yellow appearance of objects is simply the result of yellow light that stimulating the red and the green cones simultaneously. 41

42 The three cone theory (cont d) 42

43 Colour representation in the RGB basis The primary colors of light (red, green and blue) are those that stimulate one type of cone most predominantly. An arbitrary colour is created by mixing primary colours in appropriate amounts. The RGB system is additive because it involves simultaneous light emission. a yellow sensation can be obtained by combining red light and green light 43

44 Colour representation in RGB basis Normalized (r,g,b) coordinates: Useful when object surface is not uniformly illuminated (shadows) R + G + B I = ; 3 R normalized red r = R + G + B G normalized green g = R + G + B B normalized blue b = R + G + B 44

45 Colour representation in CMY basis Printers produce color by reflective light The process describes what kind of inks need to be applied so that the light reflected from the white substrate (paper) and passing through the inks produces a given colour. This is a subtractive process and uses a model based on the colors: Cyan, Magenta, Yellow. cyan = green + blue, so light reflected from a cyan pigment has no red component, i.e., the red is absorbed by cyan. Similarly magenta subtracts green and yellow subtracts blue. The conversion between the RGB and CMY is easily computed as below: C=1-R M=1-G Y=1-B 45

46 The HSV colour space HSV=[Hue Saturation Value] Same thing as HSI=[Hue Saturation Intensity] This representation allows for an efficient separation of brightness (V) and chromaticity (H,S) information. It is useful for shadow removal 46

47 The HSV colour space We obtain the hexagon by looking at the RGB cube along the gray diagonal 47

48 The HSV colour space HSV color model is more intuitive than RGB The user specifies a color (hue) and then adds white or black. There are 3 color parameters: Hue, Saturation, and Value. Changing the S parameter corresponds to adding or subtracting white (creating tints) Changing the V parameter corresponds to adding or subtracting black (creating shades) For example: pure blue H = 240, S = V = 1 dark blue H = 240, S = 1, V = 0.40 light blue H = 240, S =.3, V = 1.0 Adapted from 48

49 Shadow removal Cast shadows are most difficult to deal with. Motion information has little relevance: shadow motion is consistent with object motion. Colour information is useful. The shadow cast by a moving object exhibits similar chromaticity but lower brightness than the corresponding background region. 49

50 Comparing color codes 50

51 Exploiting colour information in computer vision 51

52 Colors can be used for image segmentation into regions Can cluster on color values and pixel locations Can use connected components and an approximate color criteria to find regions Can train an algorithm to look for certain colored regions for example, skin color 52

53 Finding a face in video frame (left) input video frame (center) pixels classified according to RGB space (right) largest connected component with aspect similar to a face (all work contributed by Vera Bakic) 53

54 Example A computer vision-based system that deals with image formation issues and exploits colour information 54

55 Veggie vision: system outline Task description: the system recognizes fruits and vegetables in a supermarket Main steps: Imaging the produce Foreground segmentation Feature selection (colour, shape, texture, size) Classification and recognition The system is strongly task-oriented, thus customized solutions are preferred over generic paradigms of Computer Vision. 55

56 Veggie Vision publication and patents VeggieVision: A Produce Recognition System R. M. Bolle, J. Connell, N. Haas, R. Mohan, G. Taubin Proc. of the Third IEEE Workshop on Applications of Computer Vision (WACV-96), December Related patents: US Veggie Vision Concept US Veggie Segmentation Box US Veggie Learning US Veggie Size Measure US Veggie Size Measure 2 56

57 Veggie vision : imaging setup Upward looking imaging system with a transparent top surface (glass window) integrated with a scale. Quasi-uniform illumination: two circular fluorescent bulbs Linear polarizing filter covering the internal light sources; second polarizer on the camera, orthogonal to the first. reason: if polarized light is normally incident on a surface - specular reflections will have the same polarization - diffuse reflection will be unpolarized - the orthogonal polarizer eliminates the specular reflection 57

58 Veggie vision: imaging the produce Issue Specular reflection introduces errors in colourbased segmentation - it may be due to the wrapping bag Solution - filtering out highlights 58

59 Parallel configuration of polarizers Orthogonal configuration of polarizers 59

60 Veggie vision: figure-ground segmentation Scene illumination=ambient illumination + illumination of the system light source Issue: ambient illumination is not controlled. Solution: data fusion between two images Image 1: lights off. Image 2: lights on. Foreground detection through image differencing and thresholding 60

61 Veggie vision: feature selection Colour Texture Shape Descriptors must be invariant to: -Translation -Rotation -Number of produce items Size 61

62 Veggie vision: feature selection All descriptors will be extracted from histograms of colour a histogram = a global, compact representation of an object (see section 2.3.2) Example: the brightness histogram of an image provides information about the frequency of all brightness levels in the image If the image has 256 brightness levels, then its brightness histogram is a one dimensional array with 256 elements. Useful function in Matlab: imhist 62

63 Veggie vision: feature selection colour histograms a) apples b) oranges hue saturation intensity 63

64 Veggie vision media coverage BBC News Online about the "Future Store" in Rheinberg, Germany, part of the METRO retail group The intelligent scale doesn't like our bunch of bananas. When we had put tomatoes on the scale, its digital camera took just a split second to recognize the produce, weigh it and print a bar-coded price tag. [..] No need to find "tomatoes" on a 50- button display. But now the scale is baffled, and offers four choices: Are we weighing bananas, chicory salad, long beans or avocados? Touching the banana logo on the screen solves the slip-up. Story from BBC NEWS: Published: 2004/05/18 BBC MMV 64