a) Which points will be assigned to each center in the first iteration? b) What will be the values of the k new centers (means)?

Transcription

1 CS 378 Cmputer Visin Prblem set 2 Out: Tuesday Sept 22 Due: Mnday Oct 5, by 11:59 PM See the end f this dcument fr submissin instructins. I. Shrt answer prblems [30 pints] 1. Suppse we are using k-means clustering t grup pixels in a (tiny) image based n their intensity. The image s intensities are: 5, 10, 3, 20, 9, 0. We pick the initial centers randmly t be 0 and 9, and set the number f clusters k=2. a) Which pints will be assigned t each center in the first iteratin? b) What will be the values f the k new centers (means)? 2. In rder t segment an image int regins with cnsistent texture, we culd use clustering t grup pixels that prduce similar respnses t a bank f linear filters. Say the filter bank cnsists f eight n x n kernels, each f which is designed t get maximal respnses fr cntrast at a particular rientatin. After filtering an image with all eight filters, each pixel gets an 8- dimensinal feature vectr, crrespnding t all f the filter respnses at that psitin. Fr any tw such vectrs p i and p j, their affinity is defined as: 1 2 Affinity( pi, p j ) = exp p 2 i p j 2σ. This prcedure will have sme prblems arund pixels that are psitined near a true regin bundary in the image. Why? 3. The shapes belw exhibit reflectinal symmetry abut sme vertical axis. Suggest a vtingbased (Hugh transfrm-like) methd that culd detect such symmetry and estimate the axis placement, given sme edge pints n the bject s cntur. Yu may assume we nly care abut bjects that are symmetric abut an axis parallel t the image s y-axis.

2 II. Prgramming prblem: texture-based image matching and segmentatin [70 pints] The gal is t use texture features t segment r cmpare images. We ll experiment with images in which texture is a defining feature. In bth cases, we ll cmpare against a clr-nly baseline. Write prgrams that can d the fllwing: A. Glbal image cmparisns using texture: Use texture t cmpute a glbal cmparisn between tw images, and then classify the image using nearest-neighbr classificatin. First, cmpute a textn cdebk frm a large (~100K) randm sample f filter respnse vectrs extracted frm the prvided images. Secnd, summarize each image by a single glbal (imagewide) histgram f its textn ccurrences. Finally, given any image as a new query, srt the remaining images based n their similarity accrding t the χ 2 (chi-squared) distance between their histgrams. Classify each image in turn as belnging t ne f the tw candidate categries (dalmatian r zebra), depending n which class its nearest neighbr belngs t. Cmpare the verall accuracies when using either glbal texture histgrams r glbal clr histgrams cmputed with the hue channel.

3 B. Texture-based image segmentatin: Given an image, segment it int R regins using lcal textn histgrams cmputed within windws. First, cmpute a textn cdebk (dictinary) frm the input image using k-means n its filter respnse vectrs, where k equals the number f desired textns. Secnd, cnstruct a textn histgram fr each pixel based n the textn cunts within its neighbrhd (as defined by a lcal windw f fixed scale that yu chse). Finally, use k-means again t cluster thse textn histgrams, gruping the image s pixels int R regins. Chse parameter values (k, R, and windw size) that yield a reasnable lking segmentatin (f curse, it wn t perfectly agree with the bject bundaries). Cmpare the utput t what yu get when using k-means t cluster the pixels RGB values int regins. See the end f this dcument fr prvided cde and data, and tips n where t get started cding. Answer each f the fllwing, and include image displays where apprpriate. 1. [10 pts] Lk at the raw filter respnses (as images) fr sme examples frm each class. Chse an image frm each class (ne zebra, ne dalmatian) and, by visual inspectin, select tw f the 38 filters fr which these images respnses illustrate the texture differences well. Display the images and the crrespnding respnses, and explain what we are seeing. 2. [10 pts] Cmpute a universal textn dictinary using filter respnses sampled frm the 26 prvided images. Fr the same tw selected images as abve, display the textn map, where every pixel is shaded r clred accrding t which textn it was assigned. (i.e., if there are 100 ttal textns in the dictinary, yu will have at mst 100 unique values in the textn map fr an image). Briefly explain. 3. [10 pts] Reprt the 1-nearest-neighbr classificatin accuracy using the texture representatin (as described in part A). Classify each f the 26 examples based n hw the ther 25 are ranked. If the first-ranked neighbr has the same label as the input image, it is crrect. The accuracy is the percentage crrectly classified ut f all 26. Briefly explain the results, including any interesting errr cases. 4. [10 pts] Reprt the 1-nearest neighbr classificatin accuracy using clr histgrams frmed with the hue channel f the HSV image (als with χ 2 distance). Hw des it cmpare t the texturebased representatin? 5. [10 pts] Chse tw image examples frm each class (2 zebras, 2 dalmatins), and fr each, display their five mst similar examples accrding t (a) the texture representatin, and (b) the clr representatin. Label the subplts clearly as t what each is shwing. 6. [20 pts] Select ne image and use its filter respnses t cmpute an image-specific textn dictinary. Segment the image using texture features cmputed within windws (as described in part B), and display the riginal image and segmentatin result. Then segment the same image by clustering pixels by their RGB values; display the result. Briefly explain. III. [OPTIONAL] Extra credit [up t 10 pints each, max 20 pints extra credit] 1. Implement basic backgrund subtractin. Use the squared difference at each pixel between the current frame and the backgrund frame t determine where the largest changes ccur. Create the backgrund frame itself by taking the median f all intensities alng the prvided sequence (pset2_extra_credit_images.tar.gz). Let the fregrund pixels be thse that survive sme selected threshld n the squared differences. Extract the cnnected cmpnents, and

4 clean up the fregrund regins with mrphlgical peratrs. (useful Matlab functins: median, bwlabel, imdilate, imclse, imerde, impen). Shw the results. 2. Using a universal textn dictinary, first segment tw images frm the same class (e.g., tw zebras) int regins based n their textn histgrams cmputed frm lcal windws. Then, use the mean textn histgram assciated with each resulting regin t describe it. Let this be the regin-level descriptr. Fr each regin in the first image, find the regin in the secnd image whse descriptr is clsest (using L2). Display the regin-t-regin matching results. Then add the regin s centrid psitin t the descriptrs, and cmpare the regin-t-regin matching results. Briefly explain. 3. Expand n #6 in Sectin II abve: examine and shw the impact f the neighbrhd windw scale parameter n the resulting segmentatins, fr ne f the images. Prvided cde and data: dist2.m: This functin des fast cmputatin f the squared Euclidean distances between tw lists f vectrs. This will be helpful when mapping the per-pixel vectrs f filter respnses t cluster centers t assign a textn t each pixel using the universal textn dictinary (part A). See the specificatins at the tp f the file. filterbank.mat: A.mat data file cntaining the filter bank as a single variable, F. The filter bank is stred as a 49 x 49 x 38 matrix. It cntains 38 ttal filters, where each filter is 49 x 49. (Lad int memry with lad ). makerfsfilters.m: Fr yur reference, this is the Matlab cde used t generate the prvided filter bank. (F = makerfsfilters;) (Cde by Manik Varma et al., displayfilterbank.m : Simple functin t display the individual filters in the filter bank. pset2images.tar.gz: Images fr the experiments. There are tw classes, with 13 clr images in each class. Tips: Where t start? Yur cde will need t be able t perfrm the fllwing tasks: Apply the filter bank t an image t prduce a vectr f respnses at each pixel. Fr N filters, yu will have N values cmputed fr each pixel. Cllect a randm subsample f filter respnse vectrs frm the images. Given a set f filter respnses and the desired number f textns, frm a textn dictinary (cdebk) using kmeans clustering. Given an image s filter respnses and a textn dictinary, map each pixel t its assciated textn (e.g., an index, r cluster ID). Given an image, windw scale, and textn dictinary, at each pixel cmpute the lcal textn histgram using the pixels falling within its lcal neighbrhd windw. (B) Cluster a single image s textn histgrams with k-means t frm a texture-based regin segmentatin. (B) Given an image and textn dictinary, cmpute the image s single glbal textn histgram. (A) Given an image, segment its pixels with k-means n the RGB (3-dimensinal) values. (B)

5 Cmpute an image s clr histgram using the hue channel (see rgb2hsv). The number f bins is a parameter; a value arund 32 shuld be fine. (A) Given tw histgrams, cmpute the χ 2 distance between them. Fr histgrams h i and h j, each with K ttal bins, the distance is: where h i (k) dentes the cunt in the k-th bin f histgram h i. Given a query image, srt all ther images relative t it using the χ 2 distance. Cmpute the accuracy f the 1-nearest neighbr classifier (when using either texture r clr features). Useful Matlab functins: cnv2, fspecial, imread, rgb2gray, imagesc, imshw, subplt, rgb2hsv, kmeans, reshape, dir, min, randperm, srt, lad, im2duble, axis equal;, label2rgb, histc Filtering shuld be dne with dubles, and n grayscale images. Relevant papers fr additinal backgrund reading (see class website fr links): A Statistical Apprach t Texture Classificatin frm Single Images, by Manik Varma and Andrew Zisserman, Internatinal Jurnal f Cmputer Visin, Vl 62, N 1-2, pages , When is Scene Identificatin Just Texture Recgnitin?, by Laura Walker Renninger and Jitendra Malik, Visin Research, Vl 44, pages , Cntur and Texture Analysis fr Image Segmentatin, by J. Malik, S. Belngie, T. Leung, and J. Shi, Internatinal Jurnal f Cmputer Visin, Vl 43, N. 1, pages 7-27, Submissin instructins: what t hand in Electrnically: Yur well-dcumented Matlab cde.m files. A pdf file cntaining the fllwing: Yur name and CS lgin ID at the tp. Yur answers t Sectin I, numbered. Yur respnses and image results Sectin II, numbered. Insert image figures in the apprpriate places fr these questins, and label clearly. (ptinal): any results and descriptins fr extra credit prtins in Sectin III. Submit all the abve with ne call t turnin: >> turnin --submit jaechul pset2 pset2.pdf cdefilexyz.m cdefileabc.m datafile.mat etc. Hardcpy: Print ut the pdf file, and bring it t class n Tuesday 10/6/09. D nt print ut cde. The hardcpy must be identical t what is submitted electrnically by Mnday night.