Figure 1. Mr Bean cartoon

Transcription

1 Dan Diggins MSc Computer Animation 2005 Major Animation Assignment Live Footage Tooning using FilterMan 1 Introduction This report discusses the processes and techniques used to convert live action footage into a toon style effect. The inspiration for the final effect was based on the Mr Bean animation series. In the Mr Bean cartoons, all the active elements in the scene have a thick black outline. Figure 1. Mr Bean cartoon To assist in the production of the technique for creating the toon effect used in this project, a new application was developed called FilterMan. The application provides an editing framework in which a series of motion detecting, noise reducing and other filters can be applied to realtime video footage using the application s embedded toolbars. The user can then create a toon effect, as shown in this report, or any other effect of their choice to their own video footage. For this project, the use of FilterMan was restricted to creating a toon effect. In this report, the stages of developing the application are discussed, starting with a look into the issues that surround motion detection between frames in realtime video, and how these were resolved. The report also examines the problems of noise reduction in consecutive frames and what techniques exist to 1 FilterMan is a stand alone application created by Dan Diggins which enables video editing in order for the user to apply various effects, including tooning. Page 1

2 distinguish the noise from the image. Once these key areas have been addressed, the report then looks at the development of the toon effect itself and how it was applied to the realtime footage. Both the FilterMan application and examples of how it can be used to develop a toon effect on realtime video are shown on the attached CD. Motion Detection The key factor of the live footage to toon effect is the ability to detect movement within a sequence of frames. The original design for determining the movement in the frames was to compare each pixel from the current frame with the previous frames. This technique did produce the differences between frames, however the results had major flaws. In figure 2, two consecutive images are shown from a sequence. Comparing each pixel from each image produces the result in figure 3. The differences between the images can clearly been seen as white. The first major problem with this comparative technique is that it does not highlight an entire object as moving, but only the parts of the object that move. For example, in Figure 2, the arm of the girl is moving, but in order to get the desired results, the entire girl should be highlighted. The second major problem with the consecutive frame comparative technique is double differencing. This occurs because an object appears to have moved in both the previous and current frames and so two changes are recorded, eg in Figure 3, the girl appears to have two arms. Figure 2. Two consecutive frames Page 2

3 Figure 3. Difference of images in figure 2 The consecutive frame technique for detecting motion was discarded due to its major flaw of recognising differences between frames. A common approach for detecting motion is to use a clean image. A clean image is used as the background for the sequence and, by comparing sequences of images with the background, will produce all the different pixels regardless of whether there is movement or not. Figure 4 shows an example of the results of differencing using a clean background. Figure 4. Differencing using a clean background The restriction with using the clean background differencing approach is that the camera must be stationary; the slightest movement of the camera will show up as a difference. Page 3

4 Noise Although two consecutive frames may look identical to the human eye, there is always some degree of noise when making a pixel by pixel comparison. The noise is usually caused by fluctuations in the light. In Figure 4, the actual differences can clearly be seen, however there are a lot of false differences caused by noise. A test sequence was filmed to be used for determining the best approach for creating the most efficient differencing technique. Figure 5 shows the results from a simple rgb pixel comparison. As can be clearly seen there is a substantial amount of noise that needs to be removed. Figure 5. Test sequence frame different The first step to reduce the amount of noise was to look at how each pixel should be compared. Three different comparative techniques were used: 1. Luminance The luminance technique takes the red, green and blue elements and creates a single grey value. To calculate the grey value, a standard formula is used: Y = 0.299R G B This formula comes from the YIQ colour model used in colour television broadcasting. The YIQ is simply RGB recoded for transmission efficiency. The Y element is the only value displayed on a black and white television. 2. HSV The HSV technique was used as an attempt to exclude the changes in luminance from image pixel comparison. By comparing only the hue value and not the saturation (brightness) it was assumed that two pixels that only differed in brightness would not be treated as different. Page 4

5 3. RGB The RGB technique simply made a comparison of each of the pixel s red, green and blue elements. All of the three discussed techniques produced varying degrees of success. None of the techniques, however, could totally overcome the problem of shadows and changes in brightness. It wasn t until further examination of the film sequence and the camera that took the shot, that a major problem was highlighted. The camera s auto exposure was not switched off for the test shot; therefore the camera was constantly adjusting the amount of light. With the amount of light changing almost at every frame, it prevented a suitable single technique for comparing an entire sequence from being successful. Switching the auto exposure off produced much better test sequences for doing image comparisons and the brightness problem was reduced significantly. The RGB pixel differencing technique produced the most efficient results for comparing pixels. The luminance was the worst mainly because there were only 255 shades to compare against. The HSV did perform well as expected when processing varying degrees of brightness, ie shadows, however, it failed in areas where the RGB succeeded. After correcting the auto exposure problem, the brightness problem was no longer an issue and so the HSV technique was not required. The final formula for calculating where two pixels are different is: Different x,y = (Diff x,y,red <= threshold) & (Diff x,y,green <= threshold) & (Diff x,y,blue <= threshold) Where: Diff x,y,color = CurrentImage x,y,color - BackgroundImage x,y,color Applying a suitable threshold to the pixel comparison produced the result in Figure 6. Low Medium High Figure 6. Varying degrees of thresholds Page 5

6 More Noise Using a suitable threshold in the differencing process produced almost perfect results with only a few traces of noise remaining. In most circumstances the remaining noise would be acceptable but for this particular application, it would cause a detrimental effect, particularly in the outlining stage. An analysis of the remaining noise revealed that most were either single pixels or small groups of pixels. A common approach for removing or reducing this type of noise is to use an averaging operator. Two of the more common averaging operators are the standard averaging operator and the Gaussian averaging operator. Both use a template of pre-defined coefficients; usually a square matrix consisting of an odd number of rows and columns. Figure 7 shows a 3x3 template and the formula for calculating the new pixel value. W 0 W 1 W 2 W 3 W 4 W 5 W 6 W 7 W 8 N x,y = S x-1,y-1 * W 0 + S x,y-1 * W 1 + S x+1,y-1 * W 2 + S x-1,y * W 0 + S x,y * W 1 + S x+1,y * W 2 + S x-1,y+1 * W 0 + S x,y+1 * W 1 + S x+1,y+1 * W 2 Where : N = New Image S = Source Image Figure 7. 3x3 template and new pixel value calculation Standard Averaging Template The averaging template is, as its name describes, an average of the neighbouring pixels. The template coefficients for a 3x3 matrix are shown in Figure 8. 3x3 Averaging Coefficients 1/9 1/9 1/9 1/9 1/9 1/9 1/9 1/9 1/9 Pixel value before averaging Pixel value after averaging Figure 8. 3x3 averaging coefficients and sample pixel value Page 6

7 In the example in Figure 8, it can be seen that the centre pixel was at full intensity, but after applying the filter the value is a third of what it was. In subsequent filters, the centre pixel can be excluded from further operations by using thresholding. Alternatively, the pixel could be set to zero if it falls below a defined threshold. For performance reasons, further operations will simply use a threshold to ignore unwanted pixels. Gaussian Averaging Template The Gaussian averaging template offers better smoothing capabilities. It uses radial weightings so that, the further a neighbouring pixel is from the centre pixel, the less influence it will have over the new pixel value. The coefficients for the template are based on the size of the matrix. The formula for calculating the coefficients is: W (x,y) = e x 2 + y -( 2 2σ ) 2 Where σ is a controlling threshold that removes the influence of pixels more than 3σ radial units from the centre. An example 5x5 template is shown in Figure Figure 9. 5x5 Gaussian averaging coefficients ( σ = 1.0 ) The advantage of the Gaussian template over the standard averaging template is that more features are maintained. However, in this project, the noise reduction is applied to a featureless alpha channel and so there are no features to retain. Therefore the standard averaging template was used, as it performed slightly better on a black and white image. A side effect of using the standard averaging template on an image was that, as well as removing noise, it also smoothed out jagged edges. Figure 10 shows the effect of applying the averaging template. Page 7

8 Figure 10. Averaging operator applied to difference image The Toon Effect The results from the motion detection and noise reduction filters produced an alpha channel mask that represented the elements different from the clean background. Using this alpha channel it was possible to apply pixel operations on only the different and moving elements. In order to design the toon effect, it was necessary to reduce the number of colours. To reduce the number of colours, each pixel s red, green and blue elements were assigned a new value based on defined intervals in the 0 to 255 colour range. A range value defined how many intervals each colour should have. For example, if the range was set at 10, then each interval would be set at 0, 25, 50, 75, etc. Each red, green and blue value would be reassigned to the closest lowest interval. Figure 11 shows the effect on rgb colours for different range values. R G B R G B R G B R G B Range = 10 Intervals = 0, 25, 50, 75, 100, etc Range = 5 Intervals = 0, 51, 102, 153, 204 Figure 11. Reducing rgb values to closest lowest interval Page 8

9 Because all the red, green and blue elements are reduced in value, the overall effect, as well as reducing the colours, is to reduce the brightness of each pixel, as shown in Figure 12. Figure 12. Applying the toon effect The Outliner The final operation for the desired effect was to add an outline around the moving objects in the scene. To determine where the outline should be drawn, the alpha channel created by the differencing and averaging operators was used. When a black pixel was adjacent to a white pixel then the outline was drawn at that point. Figure 13 shows the effect of applying the outliner. Original Alpha Result Figure 13. Outliner applied to original image Page 9

10 Conclusion The original idea for this assignment was to create an application for processing footage in realtime. However, it became apparent quite early on that the number of steps required for the visual effect could not be done in realtime, because the system cannot process them fast enough. Currently, the system can process 720x576 frames at 6 frames per second. Some steps are more processor intensive than others. The averaging operator is particularly slow as it has to process at least 9 pixels for every pixel in the image. Further investigation into other efficient techniques for producing the difference alpha channel may mean the averaging step is no longer required. Page 10

11 References BASSMANN, H., Ad Oculos : digital image processing. Student version 2.0. London : International Thomson Publishing CRANE, R., A simplified approach to image processing : classical and modern techniques. Upper Saddle River, N.J. : Prentice Hall FOLEY. J.D., Computer Graphics - Principles and Practice. 2 nd ed. Wokingham, England: Addison-Wesley Publishing Company GOMES, J., Image processing for computer graphics. New York : Springer NIXON, MARK., Feature Extraction & Image Processing. Cornwall: Newnes PARKER, J. R., Algorithms for image processing and computer vision. Chichester : Wiley SEUL, M., Practical algorithms for image analysis : description, examples, and code. Cambridge : Cambridge University Press UMBAUGH, S.E., Computer vision and image processing : a practical approach using CVIP tools. London : Prentice Hall International Page 11