CSE 473/573 Computer Vision and Image Processing (CVIP)

CSE 473/573 Computer Vision and Image Processing (CVIP) Ifeoma Nwogu inwogu@buffalo.edu Lecture 4 Image formation(part I)

Schedule Last class linear algebra overview Today Image formation and camera properties Readings for today: Forsyth and Ponce 1.1, 1.4, Szeliski 2.1 and 2.3.1 (optional).

Physical parameters of image formation Optical Sensor s lens type focal length, field of view, aperture Geometric Type of projection Camera pose Photometric Type, direction, intensity of light reaching sensor Surfaces reflectance properties

What is an image? Till now: a function a 2D pattern of intensity values Today: a 2D projection of 3D points What is a camera? Some device that allows the projection of light from 3D points to some medium that will record the light pattern.

1st known photograph View from the Window at le Gras, Joseph Nicéphore Niépce 1826 Reproduction, 1952 Heliograph- a pewter plate coated with bitumen of Judea (an asphalt derivative of petroleum); after at least a day-long exposure of eight hours, the plate was removed and the latent image of the view from the window was rendered visible by washing it with a mixture of oil of lavender and white petroleum which dissolved away the parts of the bitumen which had not been hardened by light. Harry Ransom Center UT Austin

Image formation Let s design a camera: Put a film in front of an object Will we get a reasonable image? Why? Why not?

Turning a room into a camera obscura Hotel room, contrast enhanced View from hotel window Accidental pinholes produce images that are unnoticed or misinterpreted as shadows A. Torralba and W. Freeman, Accidental Pinhole and Pinspeck Cameras, CVPR 2012

Image formation Let s design a camera: Put a film in front of an object Add a barrier with an opening to block off most of the rays (reduce blurring) Opening is called aperture

Ist known camera Known to Aristotle (384-322 B.C.) According to DaVinci When images of illuminated objects... penetrate through a small hole into a very dark room... you will see [on the opposite wall] these objects in their proper form and color, reduced in size, in a reversed position, owing to the intersection of the rays". Depth of the room is the focal length How does the aperture size affect the image?

Shrinking the aperture Pinhole too big - many directions are averaged, blurring the image Pinhole too smalldiffraction effects blur the image Generally, pinhole cameras are dark, because a very small set of rays from a particular point hits the screen. Slide by Steve Seitz

Shrinking the aperture Pinhole too big - many directions are averaged, blurring the image Pinhole too smalldiffraction effects blur the image Generally, pinhole cameras are dark, because a very small set of rays from a particular point hits the screen.

Adding a lens - concept of focus A lens focuses light onto the film There is a specific distance at which objects are in focus other points project to a circle of confusion in the image Changing the shape or relative locations of the lens elements changes this distance

The thin lens

The thin lens Sign is +ve when incident lens surface is convex, and ve when concave

Depth of field http://www.cambridgeincolour.com/tutorials/depth-of-field.htm Depth of field is the range of distance within the subject that is acceptably sharp. Slide by A. Efros

How can we control the depth of field? Changing the aperture size affects depth of field A smaller aperture increases the range in which the object is approximately in focus But small aperture reduces amount of light need to increase exposure Slide by A. Efros

Field of View (FOV) FOV is the extent of the observable world that is seen at any given moment. For cameras, it is a solid angle through which a detector is sensitive to light the area of the inspection captured on the camera s imager.

Zooming and Moving are not the same Large FOV, small f Camera close to car Small FOV, large f Camera far from the car

Real lens systems

Lens flaws: chromatic aberration A lens can have different refractive indices for different wavelengths: causes color fringing Near Lens Center Near Lens Outer Edge

Lens flaws: Spherical aberration Spherical lenses don t focus light perfectly Rays farther from the optical axis focus closer

Lens flaws: Spherical aberration Left: image showing low level of spherical aberration and right: image showing high level of spherical aberration http://www.mto-ophtalmo.ch/intraocular-lenses/neutralasphericity/

Caused by imperfect lenses Radial distortion Deviations are most noticeable near the edge of the lens No distortion Pin cushion Barrel

Lens flaws: Vignetting

Digital camera A digital camera replaces film with a sensor array Each cell in the array is light-sensitive diode that converts photons to electrons Two common types Charge Coupled Device (CCD) Complementary metal oxide semiconductor (CMOS) http://electronics.howstuffworks.com/digital-camera.htm Slide by Steve Seitz

CCD vs. CMOS CCD: transports the charge across the chip and reads it at one corner of the array. An analog-to-digital converter (ADC) then turns each pixel's value into a digital value by measuring the amount of charge at each photosite and converting that measurement to binary form CMOS: uses several transistors at each pixel to amplify and move the charge using more traditional wires. The CMOS signal is digital, so it needs no ADC. http://electronics.howstuffworks.com/digital-camera.htm http://www.dalsa.com/shared/content/pdfs/ccd_vs_cmos_litwiller_2005.pdf

Geometric projections

Types of 3D projections 3D projection is any method of mapping threedimensional points to a two-dimensional plane. Perspective projections objects in the distance appear smaller than those close by Parallel lines converge at an image point in infinity, on the horizon Weak perspective projections perspective effects, not over the scale of individual objects Orthographic projections objects in the distance appear same size as those close by parallel lengths at all points are of the same scale regardless of distance from the camera

Distant objects are smaller Effects of perspective projection: Apparent size of object depends on their distance e.g. B and C have the same height but in reality A and C are half the size of B Distance d from pinhole O to the plane of C is half the distance from O to plane of A and B.

It is common to draw the image plane (or film) in front of the focal point. Moving the film plane merely scales the image. Parallel lines meet Projection of 2 parallel lines lying in the same plane: The projections of 2 parallel lines in the same plane Φ appear to converge on h h is a horizontal line formed by the intersection of image plane Π and a plane parallel to Φ passing through the aperture O. The line L in plane Φ and parallel to image plane Π has no image

Vanishing points Each set of parallel lines (=direction) meets at a different point The vanishing point for this direction Sets of parallel lines on the same plane lead to collinear vanishing points. The line is called the horizon for that plane Good ways to spot faked images scale and perspective don t work vanishing points behave badly supermarket tabloids are a great source.

Example of a scene vanishing point

Perspective projection Consider a coordinate system (O, i, j, k) attached to the camera whose origin O coincides with the camera aperture. O is located at a distance d along the vector k. The line passing through the aperture and perpendicular to Π is the optical axis The point c where this line intersects with the plane Π is the image center. This is often the origin of the image plane coordinate frame.

Perspective projection equations In image space, z = d Since P, O, and p are collinear, Op = λop for some λ, x = λx, y = λy, d = λz OR λ = x X = y Y = d Z Therefore, x = d X Z and y = d Y Z

Weak perspective An even coarser approximation of image formation Consider front-parallel plane Π o defined by Z = Z o For any point P in Π o x = -mx, y = -my, where m = - d ZZ m is the positive magnification associated with plane Π o

Issue Weak perspective perspective effects, but not over the scale of individual objects collect points into a group at about the same depth, then divide each point by the depth of its group Advantage: easy Disadvantage: wrong

Orthographic projection No reversal of image features m = -1 (unnatural negative magnification) All light rays are parallel to the k-axis and orthogonal to Π x = X, y = Y Useful for creating to-scale drawings for construction and engineering (showing details)

Modeling projection d y z x Projection equation: ( x, y, z) ( d x z, d y z ) Source: J. Ponce, S. Seitz

Homogeneous coordinates ( x, y, z) ( f x, f z Is this a linear transformation? no division by z is nonlinear y ) z Trick: add one more coordinate: homogeneous image coordinates homogeneous scene coordinates Converting from homogeneous coordinates Slide by Steve Seitz

Perspective Projection Matrix Projection is a matrix multiplication using homogeneous coordinates f 0 0 0 f 0 0 0 1 0 0 0 x y z 1 f x = f y z In practice: lots of coordinate transformations ( f x z, f y z ) divide by the third coordinate 2D point (3x1) = Camera to pixel coord. trans. matrix (3x3) Perspective projection matrix (3x4) World to camera coord. trans. matrix (4x4) 3D point (4x1)

Orthographic projection (sort of ) M.C. Escher's waterfall http://glasnost.itcarlow.ie/~powerk/generalgraphicsnotes/projection/orthographicprojection.html

Orthographic Projection Special case of perspective projection Distance from center of projection to image plane is infinite Also called parallel projection What s the projection matrix? Slide by Steve Seitz

Physical parameters of image formation Optical Sensor s lens type focal length, field of view, aperture Geometric Type of projection Camera pose Photometric Type, direction, intensity of light reaching sensor Surfaces reflectance properties

Slide Credits David Forsyth UIUC, slides accompanying Forsyth and Ponce Computer Vision book, 2/e Rob Fergus NYU AaronBobick GA Tech Svetlana Lazebnik - UIUC

Next class More on image formation (photometric) Readings for next lecture: Forsyth and Ponce 2.1, 2.2.4; Szeliski 2.2 (optional) Readings for today: Forsyth and Ponce 1.1, 1.4; Szeliski 2.1 and 2.3.1, (optional)

Questions