Computational Photography: Interactive Imaging and Graphics

Computational Photography: Interactive Imaging and Graphics Jesus J Caban, PhD Outline 1. Finish talking about the class 2. Image Formation 3. Assignment #1 $&

Computational Photography! ()*+,-./)0.1&+2)-)34.+25&67&.0&8*843603&4878.492&.48.&.-& -28&60-84789/)0&):&9)*+,-84&34.+2697;&6*.38&+4)9877603;&.0<& 9)*+,-84&=676)0>&! Computer Vision Image Processing Graphics Computational Photography CMSC 691 will cover topics / concepts not necessarily inside the CP area. Syllabus '&

Other Topics! Image Warping! Feature extraction! Merge images! Texture Analysis! MRF! Segmentation (Grabcut)! Object recognition! Video compression! Image retrieval! Animation! Medical Image Processing! Volume Rendering! Image Features (SIFT)! Edge detection! Image Quality! 3D reconstruction! Gesture Recognition Final Project! Draft Proposal (10%)! Revised Proposal (10%)! Literature survey (20%)! Final paper (40%)! Final presentation (20%)?&

How to select your project?! Find something you like! Try to make something related to your thesis / dissertation! Read some of the papers / topics in advance Image Formation CMSC 491/691 @&

Digital Images! A digital image is a 2D Intensity function f(x,y)! x and y are spatial coordinates! the amplitude of function f is called the intensity! Each discrete sample of the intensity function is referred to as a picture element or pixel! Can view intensity values as a height map using intensity values as the distance along Z-axis Digital Image Representation Issues! Typical grayscale images has 256 Grayscale values! 512x512 pixels requires over a! million bytes of storage! Image formation is inherently a noisy process that maps a continuous function to a discrete set of samples! Images and course terminology! f(x,y) the intensity at a point in the world (radiometry)! f(u,v) measured intensity on the real image plane (non discrete)! f(i,j) intensity value at a pixel location i,j (digitized) (x,y) (u,v) -f (i,j) A&

Formation of a Digital Image! Steps: 1. World: reality / geometry 2. Optics: focus light from world onto sensor 3. Sensor: convert light to electrical energy 4. Signal: representation of incident light as continuous electrical energy 5. Digitizer: converts continuous signal to discrete signals 6. Digital Representation: final representation of reality in computer memory 1 2 3 4 5 (i,j) 6 The Geometry of Image Formation! Describes the projection of 3D to 2D! Typical Assumptions! Ideal pinhole lens! Light travels in straight lines! Various types of projections! Orthographic! Perspective! Spherical! Oblique! isometric Pinhole Camera B&

Orthographic Projection P P O Q Q! Used for engineering drawings, architectural diagrams! Preserves measurement accuracy! All views are the same scale Perspective Projection Q f O P P Q! More realistic! Has some issues C&

Equivalent Image Geometries! Consider case with object on optical axis! More convenient geometry, with an upright image! Both are equivalent mathematically Perspective Projection Equations (X-Z Plane) P(x,y,z) -f (u,v) By similar triangles: #&

Perspective Projection Equations (Y-Z Plane) By similar triangles: The perspective Transform Equations! Given a point in the 3D world! The two equations transform a world coordinate (x,y,z) into image coordinate (u,v)! Notice:! Position on the image plane is related to depth.! (u,v) are scaled equally based on focal length and depth!&

Perspective Projection! Viewing rays are not parallel, but converge to the image center! Each point in the image corresponds to a particular direction defined by a ray from the point through the pinhole.! Vanishing points are a property of the parallel foreshortening of image structure under perspective projection Formation of a Digital Image! Steps: 1. World: reality / geometry 2. Optics: focus light from world onto sensor 3. Sensor: convert light to electrical energy 4. Signal: representation of incident light as continuous electrical energy 5. Digitizer: converts continuous signal to discrete signals 6. Digital Representation: final representation of reality in computer memory 1 2 3 4 5 (i,j) 6 $%&

Image Formation In The Eye! Muscles within the eye can be used to change the shape of the lens allowing us focus on objects that are near or far away! An image is focused onto the retina causing rods and cones to become excited which ultimately send signals to the brain Why lens?! Assume a continuous series of properly arranged glass pieces that focus light to a single point. This is know as a lens. The lens takes multiple paths of light from a single source and focuses them only a single point on the image plane (retina)! The pinhole structures allows the world to be focused onto photoreceptive plane! Incoming point light source illuminates retina at only one point $$&

Brightness Adaptation & Discrimination! The human visual system can perceive approximately 10 10 different light intensity levels! However, at any one time we can only discriminate between a much smaller number brightness adaptation! Similarly, the perceived intensity of a region is related to the light intensities of the regions surrounding it Brightness Adaptation & Discrimination (cont ) For more great illusion examples take a look at: http://web.mit.edu/persci/gaz/ $'&

Optical Illusions! Our visual systems play lots of interesting tricks on us Reflected Light! The colours that we perceive are determined by the nature of the light reflected from an object! For example, if white light is shone onto a green object most wavelengths are absorbed, while green light is reflected from the object White Light Colours Absorbed Green Light $?&

Introduction to Radiometry! Radiometry: measurement of electromagnetic radiation! what the brightness of the point will be! Brightness: informal notion used to describe both scene and image brightness! Scene brightness: related to energy flux emitted (radiated) from a surface! Irradiance: image brightness: related to energy flux incident on the image plane! Image intensity is an under-constrained problem Image Intensities source sensor normal Need to consider light propagation in a cone surface element Image intensities = f ( normal, surface reflectance, illumination ) $@&

Radiance source sensor normal surface element Surface Radiance: 2 (watts / m steradian ) Radiance: power per unit foreshortened area emitted into a unit solid angle. L depends on direction Surface can radiate into whole hemisphere. L depends on reflectance properties of surface. Some Observations about Radiance! Function of several properties! Total irradiance (Light Flux (power) incident per unit surface area.)! Surface properties (reflectance as a function of wavelength)! Radiance measures power per unit foreshorten area into a solid angle (watts/square meter)! Total watts (on a light bulb for example) are emitted through 4 PI radians! Lumens (another measure of illumination strength) are related to radiance What about Bi-Directional Reflectance Distribution (BRDF)? $A&

Different Scatter Types Backscattering (sun behind observer) of a soybean field. By Don Deering Forwardscattering (sun opposite observer) of a soybean field. Note specular reflection. Left: backscattering (sun behind observer). Right: forwardscattering (sun opposite observer) $B&

Formation of a Digital Image! Steps: 1. World: reality / geometry 2. Optics: focus light from world onto sensor 3. Sensor: convert light to electrical energy 4. Signal: representation of incident light as continuous electrical energy 5. Digitizer: converts continuous signal to discrete signals 6. Digital Representation: final representation of reality in computer memory 1 2 3 4 5 (i,j) 6 Image Sensing! Incoming energy lands on a sensor material responsive to that type of energy and this generates a voltage! Collections of sensors are arranged to capture images Imaging Sensor Line of Image Sensors Array of Image Sensors $C&

Charge-coupled device (CCD)! Invented at AT&T Bell Labs back in 1969.! Boyle and Smith received the 2009 Nobel Prize for Physics for their work on CCDs.! Basic operation: 1. Image projected through a lens onto the capacitor array 2. Each capacitor accumulates an electric charge proportional to the light intensity at that location 3. A control circuit causes each capacitor to transfer its contents to its neighbor 4. The last capacitor dumps its charge into a charge amplifier which converts the charge into a voltage 5. The entire array is converted to a sequence of voltages. CCD Filter Each pixel contains a Bayer filter (50% green, 25% red, 25% blue) Bayer pattern mimics the human eye (more rod cells which are sensitive to green) Some cameras can output the Bayer pattern image. Color resolution is lower than the luminance resolution $#&

CCDs 3CCDs and Filters! Three-CCDs is another (better) color separation method. Each of the CCDs respond to a particular color.! Disadvantages:! Distortion! intensity of ray, etc. Complementary metal-oxide semiconductor (CMOS)! CMOS! Array of pixel sensors! Each pixel sensor contains a photodetector and an active amplifier! Solves the speed and scalability issues! Consumes less power than a CCD! Less image lag! Can be fabricated much cheaper! Can support image processing functions within the same circuit! Mostly used for cellphones, webcams, security cameras, etc. $!&

Formation of a Digital Image! Steps: 1. World: reality / geometry 2. Optics: focus light from world onto sensor 3. Sensor: convert light to electrical energy 4. Signal: representation of incident light as continuous electrical energy 5. Digitizer: converts continuous signal to discrete signals 6. Digital Representation: final representation of reality in computer memory 1 2 3 4 5 (i,j) 6 Image Acquisition! Images are typically generated by illuminating a scene and absorbing the energy reflected by the objects in that scene Similar concept for CT, X-ray, ultrasound, etc '%&

Image Sampling And Quantization! A digital sensor can only measure a limited number of samples at a discrete set of energy levels! Quantization is the process of converting a continuous analogue signal into a digital representation of this signal Sampling: Spatial Domain! = '$&

Image Sampling And Quantization Image Sampling And Quantization (cont )! Remember that a digital image is always only an approximation of a real world scene ''&

Image Sampling Example original image sampled by a factor of 2 sampled by a factor of 4 sampled by a factor of 8 Sampling and Reconstruction! = '?&

Sampling Theorem! This result if known as the Sampling Theorem and is due to Claude Shannon who first discovered it in 1949 A signal can be reconstructed from its samples without loss of information, if the original signal has no frequencies above 1/2 the Sampling frequency! For a given bandlimited function, the rate at which it must be sampled is called the Nyquist Frequency Quantization! The process of constraining / mapping a continuous function to a discrete set of values. 255 0 '@&

Image Quantization Example! 256 gray levels (8bits/pixel) 32 gray levels (5 bits/pixel) 16 gray levels (4 bits/pixel)! 8 gray levels (3 bits/pixel) 4 gray levels (2 bits/pixel) 2 gray levels (1 bit/pixel) Formation of a Digital Image! Steps: 1. World: reality / geometry 2. Optics: focus light from world onto sensor 3. Sensor: convert light to electrical energy 4. Signal: representation of incident light as continuous electrical energy 5. Digitizer: converts continuous signal to discrete signals 6. Digital Representation: final representation of reality in computer memory 1 2 3 4 5 (i,j) 6 'A&

PGM format! A popular format for grayscale images (8 bits/pixel)! Closely-related formats are:! PBM (Portable Bitmap), for binary images (1 bit/pixel)! PPM (Portable Pixelmap), for color images (24 bits/pixel) o ASCII or binary (raw) storage ASCI Binary Compression! A 5MP image (e.g 2580x2048) can use 5.3MB+! RAW12: 7.9MB! RAW10: 6.6MB! RAW8: 5.3MB! JPEG: 1.5MB! Full-motion video! 1min at 640x480 => 1.66GB 'B&

Approaches to Compression! Redundancy reduction! Remove redundancies in signal to reconstruct a signal of higher information content per bit! Irrelevancy reduction! Omit part of the signal not required by observer! The human visual system may not register small changes in gray values Two main schools of image compression! Lossless! After decompression, reconstructed image is numerically identical to original image! Can only achieve a small amount of compression! Lossy! Discards components of the signal that are known to be redundant! Signal is therefore changed from input! Capable of higher compression rates 'C&

Typical Image Coder Source Encoder Quantizer Entropy Encoder Source Encoder Discrete Fourier Transform (DFT) Discrete Cosine Transform (DCT) Discrete Wavelet Transform (DWT) Output Signal Quantizer: Reduce the number of bits needed to store pre-encoded signal Many-to-one: lossy Scalar quantization (SQ): quantize each coefficient Vector quantization (VQ): quantize group of coefficients Entropy Encoder: Further compress quantized values in a lossless manner Build a probability model of each quantized value Re-encode signal based on these probabilities Most commonly used encoders: Huffman, RLE JPEG Block Diagram 8x8 blocks Source Image B G R DCT-based encoding FDCT Quantizer Entropy Encoder Compressed image data Table Table '#&

Example Difference BMP (Bitmap)! Use 3 bytes per pixel, one each for R, G, and B! Can represent up to 2 24 = 16.7 million colors! No entropy coding! File size in bytes = 3*length*height, which can be very large! Can use fewer than 8 bits per color, but you need to store the color palette! Performs well with ZIP, RAR, etc. '!&

GIF (Graphics Interchange Format)! Can use up to 256 colors from 24-bit RGB color space! If source image contains more than 256 colors, need to reprocess image to fewer colors! Suitable for simpler images such as logos and textual graphics, not so much for photographs! Uses LZW lossless data compression Formation of a Digital Image! Steps: 1. World: reality / geometry 2. Optics: focus light from world onto sensor 3. Sensor: convert light to electrical energy 4. Signal: representation of incident light as continuous electrical energy 5. Digitizer: converts continuous signal to discrete signals 6. Digital Representation: final representation of reality in computer memory 1 2 3 4 5 (i,j) 6?%&

Summary: Image Formation Single-lens Reflex (SLR) Camera! Light comes through lens! Reflected by the mirror! Projected on a focusing screen! Goes through a condensing lends! Reflected by a pentaprism! Shows in the viewfinder! When click:! Mirror moves upwards! Shutter opens! Image projected to sensor?$&

Point-and-shoot camera Camera settings 1. Shutter speed 2. Aperture 3. ISO?'&

Shutter speed! Term used to discuss exposure time! Length of time a camera s shutter is open! The exposure is proportional to the duration of light reaching the image sensor. Shutter speed??&

Aperture! Before light reaches film, it must pass through an opening called an aperture.! The aperture is like a pupil! You can control the aperture by setting the F-Stop Balancing Shutter and Aperture?@&

Depth of Field ISO! Indication of how sensitive a film was to light! Lower the number the less sensitivity the camera is to light and the finer the grain in the shots you are taking! Higher ISO setting are generally used in darker situations to get faster shutter speeds, however the cost is noisier shots! Rule of thumb:! ISO 100-320: good for very bright outdoor conditions! ISO 400-800: is best used in indoors where light is not as bright as outdoors! ISO >800: very dark conditions and often causes a lot of noise?a&

ISO Conclusion Image Formation 1 2 3 4 5 (i,j)?b&

Homework #1 What about an image-base coin counter??c&

1 Dime 1 Penny 2 Nickels 2 Quarters Total: 71 cents 5 pennys 2 nickels 1 dime 6 quarters Total: $1.75?#&

Assumptions and Requirements! Distance between the camera and coins constant! Size (e.g radius) of each individual penny, nickel, dime, quarter very similar (± 3-4 pixels)! Use OpenCV to complete the project! Can use C, C++, Java, or Python! Only you have to test your program in 10/23 images?!&

Acknowledgements! Some of the images and diagrams have been taken from the Gonzalez et al, Digital Image Processing book. @%&