Mech 296: Vision for Robotic Applications. Vision for Robotic Applications

Transcription

1 Mech 296: Vision for Robotic Applications Lecture 1: Monochrome Images 1.1 Vision for Robotic Applications Instructors, Jeff Ota, Class Goal Design and implement a vision-based, closed-loop control system for a mobile robot 1.2 1

2 Syllabus Topics 1. Monochrome images 2. Color Imaging 3. Visual Sensing 4. Eigenvalue Applications in Vision and Control 5. Visual Servoing 6. Embedded Programming for Lab Assignments 7. Filtering 8. Obstacle Avoidance and Tracking 9. Correlation 10. Project Demonstrations 1.3 Theme Combining Vision with Control Control Ref. Control Law Robot Vision Sensor 1.4 2

3 Course Pre-requisites Computer Vision Build on linear algebra, Matlab experience Feedback control Build on dynamics and controls experience Robot systems Build on programming experience 1.5 Other Info Homework Labs Grading (and Late) Policy Final Project Textbooks Class Monday before Thanksgiving 1.6 3

4 Today Computer Vision: What is an Image? Digital photography Image formats and Hardware Grayscale and B & W images Segmentation Control: Manufacturing Applications Programmable logic Planar positioning 1.7 What is an image? 1.8 4

5 Image Examples Digital Photograph Contour Map Thermal image (infrared) Pencil sketch Matlab plot Building floorplan Visual representations of spatially related information Source: IR_camera.html 1.9 Digital Image Monochrome Digital Image: 2D Array of luminance values (i.e ) organized by row (y) and column (x)

6 Color Images Digital Color Image: 3D Array of luminance values organized by row (y), column (x) and color axis 1.11 Hardware Quadrant Photodetector Photodetector Array Silicon sensitive to visible light and IR Photoelectric effect: Individual photons loose electrons; these electrons form a current, which is integrated over time to measure irradiance. Materials: Exotic materials are available for specialized applications, but silicon devices are inexpensive to manufacture. Silicon Devices 1. Photodiode, Phototransistor Single pixel devices, industrial applications (proximity, daylight) 2. Quad Photodetector Four pixel devices, industrial applications (alignment) 3. CCD (Charge-Coupled Device) Array Most high quality cameras, videocameras 4. CMOS (Complementary Metal Oxide Semiconductor) Array Webcams, phone cameras, compact cameras

7 CCD and CMOS Devices CCD Advantage: Low noise Disadvantage: Higher power, integration challenges, serial readout CMOS Advantage: Low power, easy integration, high frame rate possible Disadvantage: High amplification noise, nonoptimized photodetectors 1.13 Sample Images: CMOS and CCD

8 Image Formats Uncompressed Binary file list of characters, no header Bitmap (*.bmp) generally bmp s have no compression/lossless compression Compressed JPEG good for photos (millions of colors), dramatic compression possible using cosine transform (similar to FFT) GIF web optimized compression for graphics (256 colors, interlaced images appear in stages) PNG web optimized compression for photos and graphics, interlaced, lossless compression (details resolved better than JPEG, but larger file size) 1.15 Images in Matlab Matlab provides array of imaging functions imread: read standard image format as array Ex: Monochrome image: Builds 2D array with pixel values and convert from uint8 format into floating point format newim8 = imread( bw.bmp ); %read image newim = double(newim8); %convert to float Ex: Color image: Build 3D array with pixels range of (floating point) and extract the blue values colorim = imread( car.jpg ); %read image blueim = colorim(:,:,3); %extract blue

9 Write & Display Images imwrite: write image array to file Ex: Brighten image and store as JPEG newim = newim * 1.2; %brighten imwrite(newim, bright.jpg, jpg ); %write file image: display color image (pixels from 0 1) Ex: Display color image with pixel coordinates labels image(colorim); %display image axis image %count pixels from upper left imshow: display monochrome images Ex: Display grayscale image with pixels imshow(grayim,[0 255]); %display image 1.17 Binary Image Files binary I/O: read binary file Ex: Read binary data file as list and convert to array fid = fopen( img.txt, r ); %open file for reading binlist = fread(fid); %read binary data nrow = 100; ncol = 100; %define array shape binim = reshape(binlist,nrow,ncol); %convert list to array fclose(fid); %close file Note: This technique assumes each pixel is stored as a byte (with a value of 0 255). Each byte looks like an ASCII character if the file img.txt is examined with a text viewer

10 Monochrome Image Applications Quality Testing Laser Alignment 1.19 Segmentation Dividing image into regions Manual Segmentation statistical analysis Automated Segmentation feedback control

11 Segmentation Methods Manual Segmentation roipoly: prompt human to define a closed-contour segment based on an input image Ex: Create a black & white mask defining bottle bwmask_manual = roipoly(graybottle); Automated Segmentation Global threshold: simplest automated technique Ex: Create a black & white mask defining bottle bwmask_auto = graybottle > 140; Many automated techniques exist; segmentation is still a research topic Automated Threshold Segmentation Two notions important in automated segmentation 1. Pixel Values 2. Object shape In threshold segmentation 1. Threshold acts on pixel values 2. Shape is defined by pixel connections among neighboring pixels

12 Statistical Classification Why don t the black & white masks that result from automated & manual segmentation match? We can evaluate quality of automated segmentation (compared to human perception) using statistical analysis Define 4 pixel groups 1. Pixels correctly classified (in object segment) 2. Pixels correctly classified (in background segment) 3. Missed Detects (incorrectly classified as background pixel) 4. False Detects (incorrectly classified as object pixel) Region 1 Pixel PDF Region 2 Pixel PDF False Detects Missed Detects Threshold Statistical Classification Model using Gaussian probability distribution functions Consider pixel distributions in two regions Intensity probability for object: p o (x) Intensity probability for background: p b (x) Clearly, pixel distribution cannot be Gaussian, since distribution range is finite: [0 255], but model approximately as Gaussian (Normal) distribution with form N(μ,σ) p o N(160,10) p b N(120,20) Apply threshold, T, classifying all pixel values with x > T as object pixels, and all values with x < T as background pixels Missed Detection Probability: P MD = p o (x<t) False Detection Probability: P FD = p b (x>t)

13 Statistical Classification, cont. P = p ( x< T) MD = o T x= p ( x) dx o T 2 1 ( x μo) = exp dx 2 2σ x= πσo o T μ = erf + 1 o σ o P = p ( x> T) FD b = 1 p ( x< T) = 1 p ( x) dx b T 2 1 ( x μ ) b = 1 exp 2 dx 2σ x= πσb b T μ 1 1 b = 2 2erf 2σ b T x= b For T = 145 and N(160,10) P MD = For T = 145 and N(120,20) P MD = Receiver Operating Characteristic (ROC) Curve Choose the threshold to balance P MD and P FD This tradeoff is illustrated by the receiver operating characteristic (ROC) The ROC curve plots 1-P MD as a function of P FD The ideal ROC curve is a right angle Probability Distributions ROC Curve 0.06 Region 1 Pixel PDF 0.05 Region 2 Pixel PDF False Detects Missed Detects 0.04 Threshold Correctly Classified as "o": 1-P MD Falsely Classified as "o": P FD 13

14 Object Shape Classification theory describes intensity values with no notion of the spatial relationship among pixels For thresholding approach, shape is a result (output) of classification rather than an input Other segmentation strategies may achieve higher accuracy (lower P MD, P FD ) in exchange for more computational processing by using shape as an input to the segmentation process Thresholding performs remarkably well, however, for many robotic applications Usually, thresholding will identify more than one pixel cluster ( blob ) in an image 1.27 Binary Connectivity Usually, thresholding will identify more than one pixel cluster ( blob ) in an image It is natural to distinguish these clusters based on a binary connectivity rule 4-Connectivity: A pixel is a member of a cluster if any of the vertically or horizontally neighboring pixels are members of that cluster 8-Connectivity: A pixel is a member of a cluster if any of the vertically, horizontally, or diagonally neighboring pixels are members of that cluster Connectivity algorithm In C: Recursive algorithm on binary image In Matlab: Use built in function bwlabel: Assign an integer label to each connected cluster of pixels in a binary (black & white) image using 4 or 8 connectivity %Assign labels using 4-Connectivity labeledmasks=bwlabel(binaryimage,4); 4-Connectivity 8-Connectivity

15 Connectivity Example How many pixel clusters present? 1.29 Overview of Threshold Segmentation Real Data Obtain Distributions: Real data are not Gaussian, so obtain pixel distribution directly from image Manually segment object (roipoly) Obtain indices for segmented pixels (find) Construct histogram (hist) Normalize bins by total # of points in histogram to obtain discrete PDF Compute P MD and P FD vs. threshold Plot ROC curve Select an operating point on ROC, and set threshold accordingly A reasonable starting point: choose threshold that gives P MD = P FD Apply threshold Create binary mask for all pixels above threshold Label pixel clusters (bwlabel) Result: A mask describing multiple image segments (including background and one or more objects)

16 Extracting Data from a Segmented Image in Matlab Matlab can automatically compute statistics for each segment regionprops: operates on bwlabel output; gets statistics for each labeled segment propertylist=regionprops(labeledimage); The output, propertylist is a structure with the pixel area, pixel centroid (x,y) and bounding box for each labeled segment. %Area of the first segment in the list firstsegmentarea = propertylist(1).area; It is often useful to identify the segment with the largest area in many cases, this is the object of interest 1.31 Application 1: Industrial Control Quality Assurance Example Plant: Filtration equipment Sensor: Use vision to scan bottled liquid for impurities Control: If impurities detected, then signal for replacement filter (Programmable Logic Control) Threshold Control Law Filter Image Sensor

17 Application 2: Laser Alignment Assume vision signal at 30 Hz Stepper motors controlling laser can make 30 steps/second Define control law Threshold Control Law Laser Position (x,y) Quad Photo Detector 1.33 Quadrant Photodiode Solution Balance votage on all four diodes, simultaneously y x u x = -k * (I 1 + I 4 I 2 I 3 ) u y = -k * (I 1 + I 2 I 3 I 4 ) Example: Laser pointing

18 Centroid for Image Array Quadrant photodiode control is based on relative intensity More precise positioning control is possible with a full image array To locate a segment, compute its centroid The centroid (column, row) is another of the Matlab region properties that can be accessed with regionprops 1.35 Centroid Under the Hood The centroid is a density -weighted average Centroid is first moment of a density function, ρ i,j, for all pixels in the image For a binaray image, ρ i,j is either 0 or 1 Centroid is a 2D representation of a 3D object Errors are possible if object rotates out of plane x centroid ρ x ρ i, j i, j i, j i, j i, j i, j ycentroid ρi, j ρi, j i, j i, j = = y

19 Today s Summary Image Hardware and Formats Threshold Segmentation Statistical classification analysis Connectivity Planar positioning Centroid (binary, intensity-based) 1.37 Matlab Index % Standard Matlab Functions erf (24): Integrate Gaussian distribution find (29): Obtain indices for all matrix entries that are (logically) true hist(29): Generate histogram data % File input/output and display fopen (17): Open binary formatted file fread (17): Read binary formatted data fclose (17): Close binary file after read complete imread (15): Read arbitrary image formats into Matlab imwrite (16): Write array in an image format image (16): Display color image imshow (16): Display black & white image % Segmentation bwlabel (26): Establish connectivity for regions in a binary image regionprops (30): Obtain descriptive statistics for labeled segments roipoly (20): Define a segment manually