Machine Vision: Image Representation

Transcription

1 Machine Vision: Image Representation MediaRobotics Lab, Feb 21 Source: January 21 issue of PHOTONICS SPECTRA

2 Camera A/D Memory: analogue video Camera Memory: IEEE1394 CCD (Charged Coupled Device): Irradiance to video Frame Grabber: Video signal to digitized array Host Computer: Image processing l e n s C C D video signal Frame buffer: digital data Display buffer Machine vision algorithms

3 CCD charge coupled device: Light sensitive diodes, sensitive to R, or G or B band, translate flux of light energy into electric charge. 5 per 1/3 inch chip distributed according to human color sensitivity (more green than red, blue) - Interpolate the r-g-b data to get all the colors Scanning the chip line by line results in a video signal

4 Image Constraints Digital pictures are drawn on a two-dimensional screen with fixed dimensions and are subject to the following constraints: Aspect ratio o ratio of image width to image height o usually 4:3 (horizontal picture) Resolution o amount of detail that may be reproduced o for raster displays, this is the number of pixel columns x rows Coordinate system o maps pixels and points to screen locations

5 Raster images versus vector graphics Raster images o Each sample is assigned a quantized color value o Each sample corresponds to one pixel on the screen o Raster display redraws every pixel o Advantages: flexibility (requires no underlying structure) supports subtle color changes, simple to display o Disadvantages: resolution- and orientation-dependent typically large files Vector graphics o Pictures are represented by shapes composed of connected endpoints o Vector formats may also include faces (filled or unfilled) and curves o Endpoints may be multidimensional o Vector display redraws the image at regular time intervals o Advantages: compact representation of underlying structure, transforms well o Disadvantages: display processing times, subtle coloring changes

6 Digital File Formats Digital file formats contain different kinds of information Header - contains information about the picture o Picture format (what fields occur where, and what they mean) o Resolution and quantization o Title, date, creation software, other annotations o Color lookup table Data - the picture itself o ASCII data - large, but moves readily between systems o Binary data Footer - supplements the header

7 JPEG Header Information Offset size description 2 JPEG SOI marker (FFD8 hex) 2 2 image width in pixels 4 2 image height in pixels 6 1 number of components (1 = grayscale, 3 = RGB) 7 1 horizontal/vertical sampling factors for component sampling factors for component 2 (if RGB) 9 1 sampling factors for component 3 (if RGB)

8 Raster Image Formats JPEG - The Joint Photographic Experts Group File Interchange Format. JPEG can store up to 24 bit color, and it can do interlaced display like GIF. Like GIF, JPEG is a data-stream format - you can view images before you've got all of the data. Also like GIF, JPEG supports interlacing. The JPEG format supports CMYK (process color - Cyan, Magenta, Yellow and black in a subtractive color model) images, which makes it suitable for use in publishing applications. JPEG compression economizes on the way data is stored and also identifies and discards non-visual data. Because it discards data, the JPEG algorithm is referred to as lossy. JPEG does not support transparency (as opposed to GIF). PNG - The Portable Network Graphics format. Supports alpha channels for masking, gamma for color correction, and 48-bit color representation. It is platform independent. TIF - Tagged-Image File Format (TIFF) is used mainly for exchanging documents between different applications and different computer platforms. TIFF was primarily designed to become the standard format. In order to function as standard, the format was designed to handle just about any possibility.

9

10

11 .. linear memory space

12

13 ... linear memory space, r, then b, then g data

14 MATRICES Matrices, ordered sets of numbers in multiple dimensions, are described by the field of matrix theory. They can be added, multiplied, and decomposed in various ways, and form a key concept of linear algebra. examples is a 2 (row) by 3 (column) matrix is a 3 by 5 matrix

15 MATRICES General format of an m x n matrix (wikipedia)

16 MATRICES Matrix equality: if A = B, then A(i,j) = B(i,j) Matrix addition if C = A + B, then C(i,j) = A(i,j) + B(i,j) Matrix subtraction if C = A - B, then C(i,j) = A(i,j) - B(i,j) Scalar multiplication if B = aa, then B(i,j) = aa(i,j) Matrix multiplication If A(m,n) and B(n,p), then (AB) ij = Σ (A ir B jr ) from r=1 to n

17 = = = = X 5 =

18 Matrix Multiplication 2 1 X 2 = 3 2x2 + 1x3 = 7 2x2 + 1x 2x + 1x X 2 1 = 3x2 + 4x 3x + 4x1 = AB 1,1 = A 1,1 B 1,1 + A 1,2 B 2,1 AB 1,2 = A 1,1 B 1,2 + A 1,2 B 2,2

19 OpenCV Matrices CvMat* cvcreatemat (int rows, int cols, int type); Matrices are stored row by row where each row has a 4 byte alignment CV_8UC1: 8-bit unsigned single-channel matrix CV_32SC2 :32-bit signed matrix with two channels Create a matrix CvMat* M = cvcreatemat (4,4,CV_32FC1); Release the created matrix: CvReleaseMat (&M); Clone/Copy a matrix: Initialize a matrix: CvMat* M2=cvCloneMat (M); double a [ ] = { 1, 2, 3, 4, 5, 6, 7, 8, 9, 1, 11, 12 }; CvMat M3=cvMat (3, 4, CV_64FC1, a);

20 OpenCV Matrices Direct matrix element access of an initialized matrix: double a[12]; CvMat Ma = cvmat(3, 4, CV_8UC1, a); a[i*4+j] = 2.; OR Ma(i,j)=2.; Matrix-matrix operations: CvMat *Ma, *Mb, *Mc; cvadd(ma, Mb, Mc); cvsub(ma, Mb, Mc); cvmatmul(ma, Mb, Mc); // Ma+Mb -> Mc // Ma-Mb -> Mc // Ma*Mb -> Mc Elementwise matrix operations: CvMat *Ma, *Mb, *Mc; cvmul(ma, Mb, Mc); cvdiv(ma, Mb, Mc); cvadds(ma, cvscalar(-1.), Mc); // Ma.*Mb -> Mc // Ma./Mb -> Mc // Ma.-1 -> Mc

21 OpenCV Matrices Vector products: double va[] = {1, 2, 3}; double vb[] = {,, 1}; double vc[3]; CvMat Va=cvMat(3, 1, CV_64FC1, va); CvMat Vb=cvMat(3, 1, CV_64FC1, vb); CvMat Vc=cvMat(3, 1, CV_64FC1, vc); double res=cvdotproduct(&va,&vb); cvcrossproduct(&va, &Vb, &Vc); // dot product: Va. Vb -> res // cross product: Va x Vb -> Vc Va, Vb, Vc, must be 3 element vectors in a cross product

22 IPL / PIL image IplImage int nchannels; // Number of color channels (1,2,3,4) int depth; // Pixel depth in bits: // IPL_DEPTH_8U, IPL_DEPTH_8S, // IPL_DEPTH_16U,IPL_DEPTH_16S, // IPL_DEPTH_32S,IPL_DEPTH_32F, // IPL_DEPTH_64F int width; // image width in pixels int height; // image height in pixels char* imagedata; // pointer to aligned image data // Note that color images are stored in BGR order int dataorder; // - interleaved color channels, // 1 - separate color channels // cvcreateimage can only create interleaved images int origin; // - top-left origin, // 1 - bottom-left origin (Windows bitmaps style) int widthstep; // size of aligned image row in bytes int imagesize; // image data size in bytes = height*widthstep

23 Basic steps in PIL Allocate a 3-channel float image Image im Clone an image: Image im1, im2 im2 = im1 Reading an image from a file: im = Image.open("test.jpg")

24 Basic steps in OpenCv Allocate a 1-channel byte image IplImage* img1=cvcreateimage(...ipl_depth_8u,1); Allocate a 3-channel float image IplImage* img2=cvcreateimage(...ipl_depth_32f,3); Release an image: IplImage* img=cvcreateimage(...); cvreleaseimage(&img); Clone an image: IplImage* img1=cvcreateimage(...); IplImage* img2; img2=cvcloneimage(img1); Reading an image from a file: IplImage* img=; img=cvloadimage(filename);

25 Basics operations with OpenCV from opencv.cv import * from opencv.highgui import * # #create windows for viewing cvnamedwindow ('input', CV_WINDOW_AUTOSIZE) cvnamedwindow ('output', CV_WINDOW_AUTOSIZE) #load an image img = cvloadimage("images/purple_sm.jpg", 1) #create a few placeholder images of the same size as the original and initialize them color = cvcreateimage (cvsize (img.width, img.height), 8, 1) #8bit, single channel bw = cvcreateimage (cvsize (img.width, img.height), 8, 1) #8bit, single channel #binarize and threshold the image cvconvertimage(img, bw, ) th_low= 1 th_high = cvthreshold(bw, bw, th_low, th_high, CV_THRESH_BINARY) cvshowimage ('output', bw)

26 Low level access of image data with python-opencv Import the adaptors library pixel level access all bands of the first pixel from opencv.adaptors import * data_3cv = Ipl2NumPy(img_cv) first_pixel = data_3cv[,] red component of the first pixel p1red = data_3cv[,][] alternate approach data_red = Ipl2NumPy(red) print "RED: ", data_red[,]

27 Getting statistics from a gray scale image with python-opencv cvconvertimage(img_cv, bw, ) data_1cv = Ipl2NumPy(bw) data_1cv = reshape(data_1cv, (bw.height*bw.width,1)) m = mean(data_1cv) med = median(data_1cv) stdev = std(data_1cv) variance = cov(data_1cv) print "mean: ", m print "median: ", med print "standard devation: ", stdev print "variance: ", variance

28 Alternative conversion to NumPy array def PIL2NumPy(input): if not (isinstance(input, PIL.Image.Image) or isinstance(input, Image.Image)): raise TypeError, 'Must be called with PIL.Image.Image or Image.Image!' modes_map = { "RGBA" : numpy.uint8, "RGB" : numpy.uint8, "L" : numpy.uint8, "F" : numpy.float32 } if not modes_map.has_key(input.mode): raise ValueError, 'Unknown or unsupported input mode!. Supported modes are RGB, L and F.' result_ro = numpy.asarray(input, dtype=modes_map[input.mode]) # Read-only array return result_ro.copy() # Return a writeable array #call this function like this: numpy_data = PIL2NumPy(input).astype(float)