CSE 527: Introduction to Computer Vision

Transcription

1 CSE 527: Introduction to Computer Vision Week 7 - Class 2: Segmentation 2 October 12th, 2017

2 Today Segmentation, continued: - Superpixels Graph-cut methods Mid-term: - Practice questions Administrations

3 Segmentation Another simple idea: Segment-by-Clustering (colors) k-means: - Pick k cluster centers (random, or some good heuristic) For each pixel color value - assign the closest cluster Recompute cluster center (mean) from assigned pixels. Repeat until converged. K = 15

4 Superpixels Group together similar looking pixels. - Speed up pixel-algorithms. Can be used for bottom-up segmentation. Unsupervised (mostly). How do we get superpixels? Hint: think about k-means, what can we add?

5 Superpixels By Clustering Use regular k-means, but consider color and position as features. Problem: k-means considers all the pixels in the image. - It s a waste of time. Produces oddly shaped superpixels. Can be spatially incoherent (non-contiguous) What can we do about that?

6 Superpixels By Clustering SLIC: Simple Linear Iterative Clustering From 2011, and already has >3,000 citations(!!) Key innovation over color-spatial k-means: - Limiting the search space to a region proportional to the superpixel size. A weighted distance measure to combine color and spatial proximity Considered state-of-the-art in speed and quality. [Achanta 2011]

7 Superpixels by Clustering k-means vs. SLIC n = 100

8 Questions?

9 Segmentation via Graph-Cuts

10 Image As Graph Fully connected graph: i Cij j j i Edges have a weight Cij proportional to image similarity (e.g. color, intensity, texture, etc.)

11 Image As Graph Graph cuts: i Cij j j i Break the graph to connected components. Delete weak links: - Similar pixels should remain in the same cluster.

12 Graph Cuts Cost of cut: sum of edge weights. A minimal graph cut should give a good segmentation. - It would cut the weakest (dissimilar) links, keep similar nodes connected. Problem: Truly minimal cuts are bad, e.g. a single pixel. Why is that?

13 Normalized Cuts Take a Normalized Cut. Where Is the sum of weights within cluster A. And Is the sum of all A weights (including outside). Insight: Encourage large segments - Find equilibrium between size of A and size of B (no one can be too small / large) - Look for edges that are weak relative to edges both inside and emanating from a particular region

14 Normalized Cuts

15 Questions?

16 Binary Segmentation Binary Segmentation: - Goal: Model the appearance (e.g. color) of FG and BG, and create segments s.t. they don t mix. Separate object from background. For each pixel assign a label: FG, BG (0 or 1) A labeling L over the image. How do we make sure they re coherent and don t mix? Hint: think about graph-cuts, what can we add? we need two things.. [from Snavely]

17 Energy-Based Graph Cut Find a labeling L that minimizes the energy (cost): Where: Similarity within label. And: Similarity in color / intensity between neighbors. [from Snavely]

18 Energy-Based Cuts Match Term [from Snavely]

19 Energy-Based Cuts Smoothness Term [from Snavely]

20 Energy-Based Cuts Solving - We are still looking to find the minimal cut in the graph... We assume just 2 labels (FG, BG), so pixels should be either in BG or FG. Nodes are only connected to immediate neighbors (not fully connected). We still have a cost / weight for each edge. How do we solve such a problem? Hint: A standard graph-cut algorithm in CS / graph theory i E j

21 Energy-Based Cuts Solving - s We are still looking to find the minimal cut in the graph... We assume just 2 labels (FG, BG), so pixels should be either in BG or FG. Nodes are only connected to immediate (4, 8) neighbors (not fully connected). We still have a cost / weight for each edge. i j Min Cut - Max Flow! Add source and target (sink) nodes, and connect each to all graph nodes. t

22

23

24

25

26

27

28

29

30

31

32 Energy-Based Cuts Marking, Initialize Mark some pixels on the image. In the graph set their edge to S,T with : To force them to be BG or FG. [from Snavely]

33 Energy-Based Cuts Solving [from Snavely]

34 Examples [from Snavely]

35 Examples [from Szeliski]

36 Examples GrabCut [from Szeliski]

37 Examples [from Prince]

38 Examples [from Prince]

39 Questions?

40 Mid-Term Administrations 25 Multiple-choice questions, 4 pts/ea. In rooms: 2120, 2129 (old CS bldg) Computerized - via Blackboard. 80 minutes. Do not bring your own computer - use the classroom computers. No external materials allowed. Internet access will be blocked.

41 Mid-Term Material The test will cover: - Optical flow Mean shift, CamShift Kalman Filter Particle Filter Image stitching Dimensionality reduction Object detection - Viola-Jones Bag of Visual Words Image Features - Tracking FAST MOPS SIFT Feature Matching - Strategies ROC curve

42 Example Question 1 What is a Bayer pattern and what is it used for? (1) It is a square arrangement of R,G,B pixels/photosensors on a camera sensor or screen, and it corresponds with the human arrangement of sensors on the retina. (3) A Bayer pattern is a type of dynamic programming pattern in image processing, which makes it very efficient to find edges. (2) It is a pattern of edge detecting kernels that are used in a convolution to find the 2nd-order gradients in an image. (4) A bank of several 2D high-frequency patterns used to decompose an image to filter-response frequencies (similar to Fourier Transform)

43 Example Question 2 When does aliasing happen? (most correct answer) (1) When sampling too fast. Sampling frequency is much higher than signal and unwanted high frequencies are caught. (3) When sampling a noisy signal. If high-frequency noise in the signal is present, the sampling catches unwanted noise. (2) When not sampling fast enough. Sampling frequency is less than twice the highest frequency in the signal. (4) When high-pass filtering is not applied. The sampling frequency must match the signal frequency, so a high-pass filter will align the frequencies.

44 Example Question 3 What is the result of the following operation? * -1/9-1/9-1/9-1/9 2-1/9-1/9-1/9-1/9-1/9 = (1) (2) (3) (4)

45 Wrap Up Next: - Mid-Term Thu 10/19: Multiple-view geometry