Vision for Robotics Lab session 8 CAMSHIFT

Transcription

1 Vision for Robotics Lab session 8 CAMSHIFT The OpenCV implementation of the CAMSHIFT algorithm relies on a number of auxiliary functions. The first of these is cv2.calchist, which calculates the histogram of the values in an array. For CAMSHIFT we need to calculate the histogram of an image that contains the object of interest in function of characteristics such as intensity (for grayscale images) and hue (for color images). This function takes the following arguments: 1. An image (this can actually be more than one image, as one can obtain the histogram of a set of images. In this session, though, we will work with only one) 2. The channels of which the histogram will be a function. For example, in a grayscale image there is only one channel (pixel intensity), so this function will give as a result the (hopefully bimodal) histogram with which we are familiar. For color images, one can obtain individual histograms for the R, G and B values. In our case, as we are using HSV images we will only use the H channel. Since this is the first channel, we pass to the function a 0 for channel index. 3. A mask as was explained previously, a mask consists simply of a binary image where the object of interest has been isolated. It is used so that only the histogram of the object (and no other part of the image) is calculated. 4. Histogram size This is the amount of bins of which each dimension of the histogram will consist (one dimension per channel). In our case, we are using only one channel (hue), so we have only one dimension. Under the OpenCV implementation, hue ranges from 0 to 180. We will define a histogram of size 180, so there is a bin for every hue degree (this decision is arbitrary: we could have as many or as few bins as we need). 5. Finally, the histogram range. We have determined that we want 180 bins.this means that if we plotted the histogram, it would consist of 180 vertical bars. To specify that we want one bin for every hue degree, we will make the range go from 0 to 180 (if we wanted, though, we could have any number of bins, from any segment of the hue spectrum, simply by modifying the size and range of the histogram.

2 Figure 1. Bin visualization. Each bin represents a range of values. Pixels whose value is within one of these ranges will fall into the corresponding bin. Figure 2. Example of a histogram. Note that the number of bars is much smaller than the number of colors in a color image. For this histogram, the number of colors is decreased by grouping them into bins. Each bar on the histogram represents the amount of pixels that fell into that bin. The next auxiliary function is called cv2.normalize(), and it will be applied to the histogram obtained in the previous step. We will use this function to adjust the values of the histogram so they occupy a specific range. The reason why we need to normalize the histogram is because in the next step calculating the back projection, no histogram value should exceed 255 (we will see why further below).

3 The arguments this function takes are: 1. A source array in our case, it will be the histogram, even though the normalize() function can normalize any array. 2. A destiny array (it can be the source array, which will simply be overwritten). 3. The lower bound of the range we want. 4. The upper bound of the range we want. 5. The normalization type for our purposes, we will use cv2.norm_minmax, which adjusts the range of values in the array to the range we specified. Figure 3. Original image Figure 4. Image after conversion to HSV Figure 5. Histogram of hue channel of the HSV image. The height of the bars extend beyond the image shown. This is becaus ethe histogram is not normalized. Figure 6. Normalized histogram. The range shown is

4 Figure 8. Normalized histograma of the object of interest, calculated using the mask. Figure 7. Mask with the object of interest. The next function is cv2.calcbackproject(), and it's used to calculate something called a back projection. A back projection is a representation of how well the value of each pixel in an image fits the value distribution in a histogram. It works in the following manner: for each pixel in the image, it checks into which histogram bin it would fall. In a new image (the back projection), it copies that pixel, but with a value (intensity) that depends on the bin in whic it fell. The histogram was normalized to 255 so that the maximum pixel value in the the back projectin is white. Statistically speaking, the values in this new image represent the probability for each pixel that it belongs to the object that generated the histogram. Figure 9. Back projection of the HSV image. This image is in grayscale, where the value of each pixel depends on the probability that that pixel belongs to the object of interest.

5 The arguments that CalcBackProject takes are: 1. Source image given that we are using HSV to generate the histogram, the image for this argument must also be HSV, otherwise the function would be trying to compare RGB to HSV values. 2. The channels used. These channels must coincide with the ones that were used to build the histogram. In our case we only used hue, which is channel The histogram of teh object of interest. 4. The range of values to use from the histogram. We will use the full range of the histogram (0 180), since we calculated it specifically for the object we want, so it only contains the colors of interest. 5. A scaling factor. Since we already normalized the histogram to the values we will use, we don't need to scale them,so the factor will be 1. We have mostly covered CAMSHIFT in class already. However, it is useful to go over the equations that the algorithm uses to determine the size and orientation of the tracking window. CAMSHIFT takes as an argument the back projection, and uses moments to determine the centroid of the pixels in the window. Let us recall that, even though we usually calculate moments only on binary images, we can also do so with grayscale images. In this case, darker pixels will have a smaller weighting (i.e., in a region where there are pixels of different intensities, the centroid of all these pixels will be shifted towards the lighter pixels). Since the back projection is a representatin of the probability that each pixel belongs to the tracked object, the weighted centroid of the pixels has a greater chance of corresponding to the object than a nonweighted centroid (which would be calculated from a binary image). The steps CAMSHIFT follows are: 1. Calculate the centroid of all the pixels in the image, using 2. Place a window centered in that centroid, whose initial size will be arbitrary (but small). 3. Calculate the centroid of the pixels in the back projection 4. Calculate the back projection of the subimage in that window, and move the window to the new centroid. Repeat this until convergence (or for a set number of iterations). Obtain and save moment M 00 of the pixels in the window, as well as their centroid. 5. In the next frame, place the window on the saved centroid and change the window size using

6 Where s is the length of the side of the (square) window. The 256 factor is used to normalize (since the pixels with the highest value in the back projection are white, meaning 255). The 2 coefficient increases the size of the window. This is done to ensure that the window covers a larger portin of the object (which would not happen without this coefficient, because lowprobability pixels would not contribute as much to M 00 ). 6. Finally, calculate the orientation and the height and width of the object. For this we will use the eigenvalues of the pixels in the window. The first two eigenvalues correspond to height and width, and we can calculate them using We also calculate the orientaion of the object with The CAMSHIFT algorithm is implemented in OpenCv as cv2.camshift(), and takes the following arguments: 1. The back projection

7 2. An initial search window, defined by (origin in y, origin in x, height, width) 3. The algorithm termination criterion. This is defined by the user as (criterion type, iterations, epsilon), where the criterion type can be cv2.term_criteria_eps, cv2.term_criteria_count or a combination of both: cv2.term_criteria_eps cv2.term_criteria_count). The criterion type especifies if the algorithm will stop when the window center is at distance < epsilon to the centroid of the pixels in the window, or if it will stop after a set number of iterations, or whichever happens first. CAMSHIFT returns two variables. The first one is an array that defines a rectangle (of type RotatedRect) that contains the object, and which contains three elements: the coordinates of its center, the length of its sides and its orientation, measured in a clockwise fashion. The second variable is the new window, in the format described above. To be able to draw the rectangle we will use cv.boxpoints() 1, which finds the points of the corners of the rectangle. To draw it on the image we can use cv2.polylines(), which tales as arguments: 1. The image where it will be drawn. 2. The array of points obtained by cv.boxpoints(). 3. A flag indicating whether the figure that the points describe is closed (if it is, besides generating lines between the points sequentially, it generates a line between the last point and the first). In our case, the rectangle is closed, so this flag should be True. 4. Color, given as a BGR tuple. 5. Opcionally, an integer specifying the line thickness. As homework for this session, you must: 1. Make a video that shows your tracking results, using the method shown in lab session 6. The frames should show the coordinates and heading (orientationt) of the object at all times. To obtain the orientation, the object can have markers of different colors at its ends. You will have to segment these markers separately, calculate their centroids and use a bit of trigonometry to figure out the angle of the vector between them. 2. Do this again, but this time using CAMSHIFT (color markers not required). In addition to the normal report sections such as theoretical framework, conclusions, etc. There will be 5 extra points to whoever can explain why the image looks like that when it is displayed after conversion to HSV. 1 Note that this is a cv function, not cv2. This function also exists in cv2, but is not included in the installation we made.