Fake Impressionist Paintings for Images and Video

Transcription

1 Fake Impressionist Paintings for Images and Video Patrick Gregory Callahan Department of Materials Science and Engineering Carnegie Mellon University May 7, Abstract A technique for processing images and video to create an impressionist effect has been implemented in the Interactive Data Language (IDL). It is based on the technique described in Litwinowicz [1]. The technique uses brush strokes placed randomly across an image. The brush strokes have a user-defined length and width, and color which is the mean color of a neighborhood of pixels around its center. The orientation of the brush stroke can be a constant defined by the user, or the orientation can be determined by image gradient. Frame-to-frame (temporal) coherence between consecutive frames in a video can be achieved using the optical flow between each frame. Instead of generating new brush strokes each frame, the centers of the brush strokes are moved according to the optical flow. Automated image segmentation has also been used to separate images into regions which can then be rendered differently as the user prefers, for example background and foreground can have different brush stroke sizes. Index Terms Non-Photorealistic Rendering, Optical Flow, Impressionist Painting, Abstract Images I. INTRODUCTION T HE impressionist style of painting was developed in the nineteenth century. It was a new and abstract representation of reality that sought subjects in everyday life and in the fleeting impressions of a scene [2]. The work presented here attempts to use real images to produce images that are similar to impressionist paintings. This is accomplished by applying brush strokes in the image in a controlled way based on the data from the original image. This can remove some of the higher frequency information producing an abstract effect. The inspiration for this work is the paper by Litwinowicz [1]. The impressionist effect can also be applied to videos with frame-to-frame coherence by computing the optical flow between frames in a video. The optical flow can then be used to move brush strokes from the first frame producing the second frame. Individual images can be segmented into different regions. These different regions can be rendered differently according to how the user defines them. Different colors, intensities, or brush strokes can be assigned to different regions to alter the effect on the rendered image. This is beneficial since real paintings use various brush strokes in different regions of the painting. II. BACKGROUND In 1990, Haberli [3] detailed several methods for producing abstract, impressionistic paintings from computer generated or photographic images. The primary goal of the work of Litwinowicz was to produce temporally coherent videos with an impressionist look by modifying the approach of Haberli [1]. For a more general discussion on producing paintings from real images and other nonphotorealistic rendering techniques, the reader is directed to [4]. III. METHODS AND DISCUSSION A. Brush Stroke Generation The paintings generated by the algorithm are composed of a number of brush strokes. The brush strokes have a length L, a radius R, and an orientation given by the angle θ. The center of the brush stroke, (c x,c y ), is determined, then the mean color in a neighborhood around the center is found. The work discussed here uses a neighborhood size of 4 pixels. A random list of pixel indices in the image is generated, the number depending on L, R, and the size of the image. This random list is used to choose the brush stroke centers and the order in which they are rendered. The pixel indices of the entire brush stroke are found by generating a rectangle at (c x,c y ) with L, R, and θ. Then the mean color in the neighborhood surrounding (c x,c y ) is assigned to each pixel inside the brush stroke. B. Random Perturbations Random perturbations are added to brush strokes giving the images a more natural look after the technique of Litwinowicz [1]. The red, blue, and green channel of each brush stroke had a random value, r, g, and b, in the range [-8, 8] (of image intensity [0, 255]) added to it. Litwinowicz used values in the range [-15, 15], but I believe this relatively large magnitude leads to an overprocessed look in the images so I used the smaller magnitude. The brush stroke is scaled by an intensity factor

2 2 Fig. 1. An original image which is taken from the Prokudin- Gorskii collection[5]. intensity in the range [0.92, 1.08], which is again less than Litwinowicz used for the same reason. After these random perturbations are added, the image intnsity values are kept within the bounds of the byte scale, [0,255]. The orientation is also perturbed by θ. Figure 1 shows an original image taken from the Prokudin-Gorskii collection [5]. Figure 2 shows an impressionist effect applied ot the image, using brush strokes having L = 10, R = 4, θ= 45, and including r, g, b, θ, and intensity. While random perturbations can create a nice effect, their magnitudes should be controlled carefully to produce the desired effect. A more photo-realistic looking painting would require smaller magnitudes than I have used here. In my testing, it seemed that outdoor images look better with smaller perturbations while indoor pictures look alright with larger magnitudes, however, other users may disagree. C. Brush Stroke Clipping Comparing Figure 1 to Figure 2, it is apparent that the painting algorithm does not preserve edges. The horizon line, the river bank, and the road, among other things have become quite jagged with the application of the algorithm, and this becomes more pronounced with larger brush strokes. In order to maintain relatively smooth lines, edges can be used to clip brush strokes. An edge map is generated from using the Canny Operator [6]. Prior to application of the brush strokes, the modified algorithm checks whether or not it intersects any edges. When a Fig. 2. The impressionist version of the original image with brush strokes having L = 10, R = 4, θ= 45, and including r, g, b, θ, and intensity. brush stroke intersects with an edge, it is divided into separate regions. The region which includes (c x,c y ) is then the only region of that brush stroke which is drawn. The method is illustrated in Figure 3. Figure 4 shows the effect of this method on a processed image. The issue with the brush clipping technique is that it requires the correct edge map to produce good results. The edge map depends on the type of edge detector used, thresholding of the edge map, and whether the image is smoothed before the edge map is generated. These issues should be taken into consideration for each set of images or video that is going to be processed to ensure that the edges are not too smooth or jagged, or too few or many. D. Brush Stroke Orientation Until this point in the discussion, θ was kept constant besides random perturbations. Although the brush stroke clipping discussed in the previous section reduces the amount of aliasing along edges, it makes sense to align brush strokes along the direction of edges, or the gradient normal direction. So we make the orientation equal to the edge direction, which is given by θ = arctan ( gx g y ) + 90 (1) where g i = I s / i, the gradient of the gaussian blurred image I s in the ith direction. The orientation at each point in the image is calculated, so the orientation at the center

3 FAKE IMPRESSIONIST PAINTINGS FOR IMAGES AND VIDEO 3 Fig. 3. The edge map of the original image generated using a Canny edge detector. A brush stroke that intersects an edge is drawn and separated into different regions, and only the region which is at (c x,c y ) is drawn. This demonstrates the brush stroke clipping method. Fig. 5. The effect of using the gradient normal to determine the orientation. of brush strokes, (c x,c y ), is used to generate the impressionist images. Figure 5 shows an image produced using this method to determine the orientation θ of the brush strokes. Using the gradient normal direction to determine the brush stroke orientation produces a better image in this author s opinion. Comparing Figure 4 and Figure 5 it is apparent that this automated orientation determination method has allowed the brush strokes to be oriented in the direction of the edges, especially at the horizon and the riverbank. Obviously a painter would not always follow this convention, and may prefer to sometimes paint brush strokes in directions normal to the direction, so there may be some additional logic that could be added to generate images that look more hand-crafted. E. Frame-to-Frame Coherence Fig. 4. The effect of the brush stroke clipping method on a processed image is shown. It is possible to apply an impressionist effect to a video by generating new brush strokes for every frame. However, there will be no frame-to-frame coherence if this approach is taken. Litwinowicz [1] proposed to use the optical flow between images to move the center of brush strokes instead of generating new brush strokes for each frame. So in the proposed method, the first frame is generated using the methods described above. The next frame uses the same brush strokes but moves them based on the optical flow. To calculate the optical flow, the equation

4 4 0 = ( ) I V x + x ( ) I V y + y ( ) I t must be solved for V x and V y the velocities in the x and y directions, respectively. First, the images are blurred using a gaussian filter, then the Lucas-Kanade [7] method is used to solve Equation 2 for V x and V y. The Lucas-Kanade method involves finding the optical flow at each point by finding the gradients in a neighborhood surrounding the point, thenthe over-constrained system can be solved using a method such as least squares. I used a neighborhood size of 4 pixels. Two frames from a video are shown in Figure 6(a)-(b). The optical flow between the two images are shown in Figure 6(c)-(d). The impressionist versions of the two frames are shown in Figure 6(e)-(f). The second frame was generated with the same brush strokes as the first frame with the centers of the brush strokes moved by the optical flow. There are problems associated with this method. Since new brush strokes are not being generated, if brush strokes are moved to far, there will be blank regions in the image. Brush strokes can be moved past the dimensions of the image, or many brush strokes can be moved to a small area, making the image cluttered and causing many of the strokes to be covered. There are solutions to these problems, but they may create other issues. Litwinowicz added and subtracted brush strokes depending on how close or far apart they were. The method works to keep the images from getting blank spaces, but it also adds or subtracts new brush strokes, which can spoil the frame to frame coherence. The method also does not do well with changing backgrounds, since new brush strokes must be added as the background changes in order for the images to look right. F. Region Extraction In order to differentiate between the regions of interest in an image, some sort of segmentation must be performed. I decided to use Expectation- Maximization/Maximization of the Posterior Marginals (EM/MPM) algorithm described in [8], which segments an image by attempting to minimize the number of misclassified pixels. This algorithm was chosen because it is a fully-automated segmentation algorithm which requires little user input. The problem with the algorithm that the number of classes is user-defined and assumed to be known prior to segmentation. Increasing the number of classes increases the processing time of the algorithm. To overcome this, I use a small number of region labels (around 4). Although there may be many separate regions in the image, as long as they are not contiguous they can (2) (a) The first frame of a video (b) The second frame of a video (c) Optical flow in the x direction (d) Optical flow in the y direction (e) The impressionist version of the first frame. (f) The second frame of the video produced using the same brush strokes after moving them based on the optical flow. Fig. 6. (a)-(b) Two frames from a video, (c)-(d) the optical flow in the x and y directions, and (e)-(f) the impressionist paintings generated from them with frame-to-frame coherence.

5 FAKE IMPRESSIONIST PAINTINGS FOR IMAGES AND VIDEO 5 be separated. After segmentation by the EM/MPM algorithm, the regions can be separated by using a Sobel edge detector to find edges between different regions. Then all non-contiguous regions are relabeled in order to increase the number of region labels. Figure 7(a) shows an image which has been segmented and labeled with this method. Using this method increased the number of different region labels from 4 for the EM/MPM segmented image to 152 in the example image shown in Figure 7(a). Once the image has been segmented, the user needs to define the regions of interest. This is done using a user interface in IDL by clicking anywhere in each region they want to choose. A variety of things can be done once the regions of interest have been chosen. One can render the regions of interest in color, and the other regions in gray scale, or render the foreground more brightly than the background. An example showing the regions of interest (in this case the sky and the river were chosen) in color and the other regions in gray scale is shown in 7(b). An example showing the regions of interest rendered more brightly than the other regions is shown in 7(c). This is a good way to produce effects that significantly change the appearance of the images with little user input. Painters often use different styles of brush strokes in different regions of a painting [9]. For example, they may use larger brush strokes for the background of an images and finer, more detailed brush strokes in the foreground. Region extraction was also used to separate the foreground from the background in order to test the effect of rendering brush strokes of different sizes in different regions. After the background and foreground were chosen, the impressionist algorithm was run with larger brush sizes for the background, and smaller sizes for the foreground. The background was then gaussian blurred, and the two were combined. The result is shown in Figure 8. There are other methods possible for changing the size of brush strokes, but this is a promising method for controlling brush strokes depending on the region. IV. FUTURE WORK AND CONCLUSIONS There are many possible additions or changes that could be made to the methods described here. The shape of the brush stroke shape could be modified, although it could make brush stroke orientation unnecessary depending on the brush stroke shape. Brush textures would also be a nice addition. Brush strokes in real paintings are typically not a uniform shape or color, so variations in colors or shape depending on the location in the stroke, length of the stroke, or other variables could be added for a nice effect. As discussed, there are problems with the method (a) The result of the segmentation and region labeling method. (b) The segmented image has been used to generate a colored sky and river with other regions in gray scale. (c) The segmented image has been used to render the sky and river with normal intensity and the rest of the image with lower intensity. Fig. 7. (a) An image that has been segmented, but foreground and background have not been chosen, and (b) the regions of the sky and river have been chosen.

6 6 Fig. 8. The effect of using large brush strokes in the background and smaller brush strokes in the background. The sky and river were chosen as the background. used to generate frame-to-frame coherence. While the workarounds can help, they introduce problems as well. For a video with a static background I would propose calculating the mean background and using it to generate a base frame, to which brush strokes could be added or subtracted where there are deviations from the mean background. Also, for a video with changing backgrounds, an algorithm detecting scene changes could be used to generate frames with new sets of brushstrokes. The segmented images might be able to be used to generate other kinds of non-photorealistic rendering, such as cel shaded images. By leaving the edges in the image and using the average color in each region, a nice effect might be produced. There are many possible ways to produce images that look like paintings. The work discussed here was used to generate paintings from real images and videos with frame-to-frame coherence. The method for frame-toframe coherence needs additional work to behave correctly with videos that have changing backgrounds. Images have also been segmented in order to render different regions with different color, intensity, or brush strokes. REFERENCES [1] Peter Litwinowicz. Processing images and video for an impressionist effect. SIGGRAPH Conference Proceedings, [2] Diana Newell. The Impressionists. Ivy Press, [3] Paul Haberli. Paint by numbers: Abstract image representations. SIGGRAPH Conference Proceedings, [4] Thomas Strothotte and Stefan Schlechtweg. Non-Photorealistric Computer Graphics: Modeling, Rendering, and Animation. Morgan Kauffmann Publishers, [5] Library of Congress. The Empire That Was Russia: The Prokudin-Gorskii Photographic Record Recreated. February [6] Wesley E. Snyder and Hairong Qi. Machine Vision. Cambridge University Press, [7] Bruce D. Lucas and Takeo Kanade. An iterative image registration technique with an application to stereo vision. Proceedings of Imaging Understanding Workshop, [8] Mary L. Comer and Edward J. Delp. The em/mpm algorithm for segmentation of textured images: Analysis and further experimental results. IEEE Transactions On Image Processing, 9(10):1731, [9] Michio Shiraishi and Yasushi Yamaguchi. An algorithm for automatic painterly rendering based on local source image approximation. NPAR 2000: First International Symposium on Non Photorealistic Animation and Rendering, ACKNOWLEDGEMENTS Thanks to Professor Efros for doing a great job of combining art and science in Computational Photography and making the course interesting and fun. Also, thanks to Laura Trutoiu and Alvaro Collet for their help throughout the course.