Improvements of Bayesian Matting

Transcription

1 Imrovements of Bayesian Matting Mikhail Sindeyev, Vadim Konushin, Vladimir Vezhnevets Deartment of omutational Mathematics and ybernetics, Grahics and Media Lab Moscow State Lomonosov University, Moscow, Russia {msindeev, vadim, Abstract Digital image matting is a rocess of extracting a foreground object from an arbitrary natural image. Unlike the image segmentation task it is required to rocess fuzzy objects (like hair, feathers, etc.) and roduce correct oacity channel for them. The result can then be comosited onto a new background or edited by rocessing foreground and background layers searately. Digital image matting has become a comulsory ste in many hoto-editing and video-comositing tasks. urrently rofessional digital artists have to accurately trace objects contours and aint the details to achieve maximum quality. Our aim is to create a convenient workflow for automating this rocess and make it ossible to effectively handle high-resolution images. In this aer we show how a smoothness constraint can be incororated into Bayesian matting algorithm framework as additional regularization to imrove the result quality without affecting the comutation seed. We also demonstrate the hierarchical aroach that significantly increases rocessing seed without noticeable loss of quality. This allows us to create convenient digital image matting system. Keywords: Bayesian matting, image editing, digital comositing, foreground extraction, alha estimation. 1. INTRODUTION In the matting roblem it is assumed that the source image is a comosite of two images F and B (foreground and background) with oacity channel. These values should satisfy the RGBsace comositing equation in each ixel: = F + (1 )B, (1) where, F and B are 3D vectors of RGB values, 1. The task is to reconstruct the, F and sometimes B images from the source image using some additional user inut. Tyically matting algorithms takes a source image and a trima image as inut. Trima image is a user-secified segmentation of the image into three regions: foreground, background and unknown. While the former two rovide the knowledge about the object to be extracted, the latter denotes the area to which the algorithm should be alied. The result of the algorithm is a foreground image layer with color and oacity information available for each ixel, and a background layer. When comosited together, these two layers should roduce exactly the source image. The roblem is to reconstruct F, B and values at each ixel from single observation from a limited user inut. The trima secifies the areas with = (B = ), = 1 (F = ) and unknown F, B,. Note that if two of these three values are known, the fourth one can be easily calculated. The roblem is heavily under-constrained, since for each color there is an infinite number of combinations of foreground and background colors. In order to constrain the roblem and make it formally solvable some regularization is required. In the next section we make an overview of most notable algorithms, that roose different regularizations of the roblem. In the third section we describe Bayesian matting algorithm in more details and roose our imrovements. We show how a smoothness constraint can be incororated into Bayesian matting algorithm. We also demonstrate the hierarchical aroach and discuss its ossible integration with smoothness constraint. After it we show the results and comarisons with other algorithms and outline the future work.. PREVIOUS WORK In this section we briefly overview several state-of-the-art matting algorithms and outline their main ideas. Knockout algorithm requires a recise trima, ideally with unknown region containing only ixels with < < 1. When rocessing an unknown region ixel, F and B values are estimated by averaging color along the foreground/background region border in the neighborhood of the ixel being rocessed. value is then calculated for each color comonent indeendently and weighted average is used as final value. While being very fast, this algorithm roduces oor results when F and/or B values in the ixel are inconsistent with the color along the corresonding region boundary. This haens in many images and the algorithm roduces incorrect and noisy results. Ruzon-Tomasi method [6] is a color statistics based algorithm. The distributions for foreground and background colors are modeled as the mixtures of unoriented Gaussians. olor statistics is calculated for rather big image fragments. Then value is calculated under assumtion that color comes from in-between distribution which is an interolation of foreground and background distributions. This algorithm maximizes robability density of this distribution in oint. The disadvantage of the algorithm is its relying on the color statistics over large image sub-regions which usually contains many overlas and cannot be handled correctly. Bayesian matting [] also uses color statistics, but erforms erixel color distribution estimation. Pixels are rocessed starting from foreground and background region borders contracting unknown region ste by ste. Pixels rocessed on earlier stes rovide new foreground and background samles in addition to ixels from known regions. Used color model is a set of oriented Gaussians. Algorithm involves Bayesian framework to maximize the likelihood of F, B and values. onditional robability for F, B and given observed color can be written using Bayes s rule as: Grahion'7 Russia, Moscow, June 3-7, 7

2 P( F,B, )P(F)P(B)P( ) P(F, B, ) =, P() where P( F,B,) is estimated using the distance between and the mix of F and B (i.e. by the norm of the difference of the left hand side and right hand side of equation (1)), P(F) and P(B) are estimated via robability density of foreground and background Gaussians, P() is ignored (assuming all values to be equirobable), P() is constant relatively to maximization arameters. There is an extension of Bayesian matting algorithm roosed in [1]. It introduces P() term (which is ignored in the original algorithm) based on learnt riors (joint distribution of image and gradients) and some additional riors, e.g. image edge magnitude. However, it uses global non-linear minimization of the energy function which is robably very slow (there is no time comarison in [1]). Their edge rior is more suitable for matting hard edges and robably oversharens smooth objects, e.g. hair. Poisson matting algorithm [7] assumes that F and B images are smooth in the unknown region. F and B values are estimated at each ixel by roagating color values from boundary and blurring the result. Poisson s artial differential equation constructed by taking the gradient of equation (1) is used for finding image. Then the unknown region is reduced by fixing ixels that are close to being ure foreground or ure background, and the rocedure is iteratively reeated until convergence. Poisson matting roduces oor results when foreground/background image is not smooth (i.e. contains edges) or contains colors that are much different from those on the unknown region border. Belief roagation algorithm [9] roduces good result with very sarse trimas i.e. containing small foreground and background regions reresented with a few strokes with the rest of the image being the unknown region. Discrete set of alha values is used. The roblem is formulated as energy minimization with the exression for energy consisting of data term, which forces F and B values to conform to the local statistics, and smoothness term. Markov Random Field (MRF) is constructed for the image ixels and discrete values and solved using Belief Proagation method. Then the color statistics is refined and the algorithm is alied iteratively until convergence. However, this rocess is rather slow even for a single iteration and takes a while to converge. losed Form Solution to image matting algorithm [4] deals with a quadratic cost function. The main assumtion is that colors in F and B images are locally linear i.e. for each of those images they are aroximately linear combinations of two colors. In this case is linearly deendent on color in small image windows: a + b, where is a ixel in a small image window (e.g. 3x3), a and b are coefficients fixed inside this window. For grayscale images a and b are related with F and B by the following equations: a = 1 / (F B), b = B / (F B). () For color images they can also be exressed in terms of F and B (with a being a 3D vector): k k a + b, (3) k where k is a color comonent index. The cost function is constructed which enalizes the difference between the ixel value and the one comuted using (3). a and b coefficients are eliminated by exressing them in terms of known and to-be-found values. Least squares method is used to exress a and b in (3) using values over the neighborhoods of nearby ixels. It is shown in [4] that this cost function is quadratic with resect to. The cost function is then minimized by solving a sarse system of linear equations with matrix of size N by N where N is number of ixels in unknown region. This gives the image directly from the source image. F and B are calculated later using another quadratic cost function. Disadvantages of this algorithm include low comutation seed and the lack of color statistics. The latter does not affect many images but usually roduces glows in alha channel in small holes and thin grooves (because large number of nearby oaque ixels imedes the roagation of background color information). Also the assumtion of local foreground/background color linearity may not hold for noisy images. 3. OUR IMPROVEMENTS OF BAYESIAN MATTING We have chosen Bayesian Matting algorithm [] because at the moment it is the best color statistics based non-iterative algorithm and demonstrates otimal seed/quality balance. It doesn t rely on any strong assumtions about alha and color channels like Poisson matting does. It works with comlex distribution of foreground/background colors and its rocessing time is linear of number of ixels. Usage of statistics-based algorithm allows us to erform recalculation of the result in a small region without need to recalculate the whole image. In the next several aragrahs we are going to describe Bayesian matting algorithm in more detail to form the base for our imrovements. Taking a logarithm of () and omitting terms not affecting the arameters to be calculated, we get L(F, B, ) = L( F,B, ) + L(F) + L(B), (4) where L( ) = log P( ). The authors of [] use the following estimations of L( F,B,), L(F), L(B): L( (with user-secified F,B, ) = F (1 )B / σ σ ), T L(F) = (F F) Σ 1 F (F F) /, where F and Σ F are mean and covariance matrix of foreground Gaussian, L(B) similarly to L(F). Grahion'7 Russia, Moscow, June 3-7, 7

3 The authors of Bayesian Matting left deriving P() from groundtruth alha mattes for future work, but as far as we know did not ublish any aers or results on this. If there are several airs of foreground/background clusters, otimal F, B and are calculated for each air, then the air with the greatest likelihood value L(F, B, ) is selected. In order to maximize the non-quadratic function (4) the authors use an iterative rocedure by alternately assuming and F, B to be constant, what gives them two quadratic sub-roblems. They use the mean value over the neighborhood of the ixel being rocessed as the initial guess. For constant the following 6x6 system of linear equations for F and B is derived: 1 Σ + F F I / σ I (1 ) / σ = 1 I (1 ) / σ Σ + I B B (1 ) / σ (5) 1 Σ F F + / σ = 1 Σ B B + (1 ) / σ For constant F and B the solution for is simly the rojection of onto line segment FB: = ( B) ( F B) F B alculation of F, B and by alternating formulas (5) and (6) is reeated until convergence. 3.1 Smoothness constraint Bayesian matting is sensitive to overlaing of foreground and background Gaussians. In the original algorithm such makes estimation unstable and usually roduces imulse noise in generated oacity channel. Simle blur and median filters can imrove alha channel quality, but small details in the matte can be lost. Instead, we roose to add smoothness term into the Bayesian framework to regularize the estimation rocess in such cases. We model smoothness as 1D Gaussian with mean value being the average among already rocessed ixels, i.e. the same value that is used as initial guess for when solving the system (5). Our smoothness term is introduced into () as P(). We use the following term for L(): / L( ) = σ Thus we are maximizing the following log-likelihood: L(F, B, ) = L( F,B, ) + L(F) + L(B) + L( ), (7) Forcing artial derivative of (7) with resect to to equal zero gives us the following solution for : / σ + ( B) ( F B)/ σ (6) = (8) 1/ σ + F B / σ Formula (8) relaces formula (6) in the otimization rocedure. We can define σ in several ways. First, we can use fixed useradjustable value. Second, we can base it on the distance between foreground and background Gaussian to revent oversmoothing while keeing regions of high uncertainty (caused by overlaing of these Gaussians) consistent with nearby ixels: where σ = σ + λ dist( P( F), P( B)), σ and λ are user-secified values (in our exeriments σ = λ =.1 and dist (, ) is the distance metrics we set ) between two distributions (we use the distance between Gaussians centers). Third, we can use two-ass Bayesian matting using likelihood values calculated on the first ass for estimating σ on the second ass (it is similar to uncertainty ma used in [9]). To make smoothing less uniform and force it to conform to the color changes, we use weighted average for in ixel q: 1 w = W where the sum is taken over the already rocessed (or known) ixels in the neighborhood of ixel q with the following weight (W is a sum of all weights w for the ixel q): w ex σ w q = (we have emirically chosen value of. for σ w ). This value is also used as initial guess for in fixed-alha equation (5). Similar weighted-averaging method is used in many segmentation-related ublications. The usage of smoothness term ractically does not affect comutation time. The examle results are shown in figures 1, and 3. omutation times are comared in section Hierarchical aroach Another imrovement is the hierarchical Bayesian matting. It aims to reduce rocessing time without losing the matte quality. A straightforward way to do this is to rocess small-scale image first, then revert to the source size and erform Bayesian matting again with much smaller samling radius. But there is a more effective way to do this: by alying losed Form Solution [4] hierarchical aroach to Bayesian matting result we calculate a and b coefficient images using equation (3) from smaller image and use them to usamle image back to original size. This allows us to comletely eliminate second ass of Bayesian algorithm since we usually get accurate alha channel. To restore F and B channels we can also assume that their RGB channels are linear combinations of channels of (though we can also erform second ass of Bayesian matting for constant ). We generate a smaller image using bilinear downsamling. The trima is downscaled using the resamling filter that marks the ixel of smaller image as foreground/background only if all corresonding ixels of the source trima are foreground/background, otherwise it is marked as unknown. Bayesian matting arameters such as sigma value used for satial weight of the samle are also downscaled. Bayesian matting is erformed on a smaller image/trima air and roduces, F and B images. These images are required to be usamled back to the original size. We aly the following rocedure to image and each channel of F and B images (but we will refer to the channel being rocessed as ): Grahion'7 Russia, Moscow, June 3-7, 7

4 (a) (b) (c) (d) (e) (f) Figure 1 Examle of Bayesian matting with smoothness term. (a) himunk image from Berkeley data set [5]. (b) Our trima. (c) Alha obtained by standard Bayesian matting (our imlementation). (d) Alha obtained by losed Form Solution [4]. (e) Alha obtained by Bayesian matting with our smoothness term. (f) omosite on a constant-color background using the result of (e). (a) (b) (c) (d) Figure Examle of Bayesian matting with smoothness term. (a) Image from [] website. (b) Trima from [9]. (c) omosite obtained by standard Bayesian matting. Artifacts and roblem areas denoted by red arrows. (d) omosite obtained by our imroved Bayesian matting algorithm. Grahion'7 Russia, Moscow, June 3-7, 7

5 1. alculate a and b coefficients at each ixel of the downscaled image that give the least squares aroximation over a 3x3 window as in equation (3).. Resize the coefficient images using bilinear usamling filter. 3. Produce uscaled image (of original size) by alying equation (3) to the source image using a and b coefficients from uscaled coefficient images obtained on ste. Aarently we can use scale factors which may be non-ower-oftwo and even non-integer (however, the usamling rocedure otimized for ower-of-two factors works a little faster). We use 3x3 windows around the ixel. It can be noticed that alying equation (3) to a downscaled image blurs the alha channel. To revent this on ste 1 we constrain alha value at the ixel being rocessed (i.e. at the central ixel of 3x3 window) to equal the right-hand side of (3) exactly. Using Bayesian matting with smoothing on a downsamled image instead of losed Form Solution, as in [4], increases total seed of the algorithm, and gives better results on some images, like in Figure 1. Performing Bayesian matting on the image of smaller size gives us a non-linear seed u. For reasonable downscale factors ( 4) the matte quality does not decrease in any noticeable way. 4. RESULTS AND OMPARISONS Here we rovide some results obtained with our imroved algorithm comared to the original algorithm (our own imlementation is used) both in quality and rocessing time. We also comare the result to losed Form Solution matting algorithm [4] using the MATLAB code rovided by the authors. Unfortunately, we could not do time comarisons in this case because the MATLAB code is very slow. In Figure 1 we show how the introduced smoothness constraint hels to matte a rather comlicated image with many color similarities between foreground and background. Processing time for both standard and smoothing Bayesian algorithms is.8 seconds on AMD Semron 31+ (18 MHz) rocessor (image size is 99x19 ixels). Note that in case when foreground and background color statistics are hardly distinguishable, object boundary is attracted to the center line of the unknown region (a set of ixels that are equidistant from the foreground and background regions in L 1 -distance). This haens when maximization of L(F)+L(B)+L() terms fails to achieve high likelihood value and the smoothness term L() overowers the (a) (b) (c) (d) Figure 3 Examle of Bayesian matting with smoothness term. (a) Image from Berkeley data set [5]. (b) Our trima. (c) omosite obtained by standard Bayesian matting (with several close-us). (d) omosite obtained by our imroved Bayesian matting algorithm (with several close-us). Grahion'7 Russia, Moscow, June 3-7, 7

6 (a) (b) (c) (d) (e) (f) (g) (h) (i) Figure 4 Examle of hierarchical Bayesian matting. (a) An image from [3] (also used in [7]). (b) orresonding trima. (c), (d) Alha and foreground images obtained by standard Bayesian matting. (e) The result of (c), (d) comosited onto constant-color background. (f), (g) Alha and foreground images obtained by hierarchical Bayesian matting with downscale factor of 8. (h) The result of (f), (g) comosited onto a constant-color background. (i) A comosite of (f) using the foreground obtained by simle bilinear uscaling. Shown to emhasize the sharness of details reconstructed in (h). other terms. This usually gives a good satial-based rather than color-statistics based guess of object contour. However, many small details, absent in the unknown region shae but found using color statistics, are reconstructed. In Figure we show how smoothness term imroves foreground color and small hair details. In this image small inaccuracies in the estimated alha matte lead to incorrect estimation of foreground color. Smoothness term imroves alha matte insignificantly but this is enough to get good color estimation. Areas of similar foreground and background colors (esecially on the lower left of the image) are also imroved. The same can be seen in Figure 3. Small black sots and small holes along the edge are effectively handled by our algorithm. The result image can be used for creating a comosite without the need for manual clean-u. In Figure 4 we demonstrate our hierarchical aroach. Downscaling factor of 8 was used. Standard Bayesian algorithm took 5. seconds to comute on the rocessor mentioned above (image size is 64x48). The hierarchical aroach reduced this time to.16 seconds while reserving the quality of the result. In Figure 4 (h) we show a comosite obtained using the alha matte from Figure 4 (e) with bilinearly uscaled foreground. Hair details, though reserved in alha matte, become blurred with loose foreground color image. This demonstrates the imortance of alying the usamling algorithm to F and B images. We can also aroximately comare the rocessing time with that of Poisson matting [7] assuming that their and our imlementation of standard Bayesian matting runs the same time. This gives us a rough estimate of.3 seconds rocessing time for Poisson matting on our rocessor. Table 1 shows the time comarison between the standard and the hierarchical Bayesian Matting algorithms on several images: Grahion'7 Russia, Moscow, June 3-7, 7

7 woman image from Figure 4, car, flower and lighthouse images from Figure 5. For each image we chose maximal downscaling factor which roduced insignificant deviation of the result from the standard Bayesian Matting result. 5. FUTURE WORK 5.1 Selecting otimal arameters for smoothness constraint One of the natural imrovements is selecting otimal exression for σ used in (8) to roduce the best results. In addition, we can evaluate several workflows involving smoothness and hierarchical asses to find the best seed/quality comromise. Image Resolution TS (sec) D TH (sec) Woman image 64x ar image 73x Flower image 6x Lighthouse image 3x Table 1. Processing time comarison of the standard and hierarchical Bayesian matting algorithms. TS time for the standard algorithm, D downscaling factor, TH time for the hierarchical algorithm. We also consider adding other user-controlled arameters besides trima which could imrove the result. For examle, we can add smoothness brush which would allow the user to roughly secify areas where σ should be increased. We also want to make recalculation of the result as quick as ossible (based on revious refinement ste result) to save the user from waiting full image rocessing time after adding only several strokes to the trima. 5. Trima generation from user strokes Algorithms roducing good results from rough strokes could be more referable than those which require recise trima, if they were fast. At the moment there are no algorithms that can quickly erform full rocessing from several user strokes. So the following aroach can be a good comromise: using a fast algorithm to roduce more-or-less recise trima from several user strokes can recede full rocessing algorithm. We consider using Growut algorithm [8] for this ste. The roduced trima can also be made user-editable. 5.3 Video One of the challenging fields of matting algorithms alication is video comositing. In site of recent achievements in natural image matting, video/film-editing studios continue to use chromakeying and rotoscoing for foreground object extraction from video sequence. Used chroma-keying algorithms require the object to be filmed on accurately lit constant color background and roduce accetable yet far from ideal results since they use simle heuristics to comute F and from. For footage without chroma-key background rotoscoing is used, which means that object contours should be accurately traced in each frame by human oerator. After the object is recisely traced boundary feathering can roduce very good channel, but F channel is again comuted either by using simle heuristics or assumed to equal source image. This roduces halos around objects which are usually removed by contracting alha matte. Usage of matting algorithms can significantly imrove the result and increase working seed even if the trima has to be handdrawn for each frame. 6. ONLUSION Automatic user-guided matting is imortant for many image- and video-comositing tasks. In this aer we roosed two imrovements to Bayesian matting algorithm. We comared the results of imroved algorithm with the original one demonstrating that our algorithm roduces smooth alha image and handles color ambiguities by roagating alha values from nearby ixels. Our smoothness term significantly imroves the result of Bayesian matting without increasing the comutation time while the hierarchical aroach effectively decreases this time still roducing the accetable result. This allows us to create userfriendly matting environment for use in image and (in future) video rocessing. 7. REFERENES [1] Aostoloff, N. and Fitzgibbon, A. Bayesian video matting using learnt image riors, Proc. of IEEE VPR, , 4. [] huang, Y., urless, B., Salesin, D. and Szeliski, R. A Bayesian Aroach to Digital Matting, Proc. of IEEE VPR, , 1. [3] huang, Y., Agarwala, A., urless, B., Salesin, D. and Szeliski, R. Video matting of comlex scenes, AM Trans. Grah., 1(3):43 48,. [4] Levin, A., Lischinski, D., Weiss, Y. A losed Form Solution to Natural Image Matting, Proc. of IEEE VPR, , 6 [5] Martin, D., Fowlkes,., Tal, D., Malik, J. A Database of Human Segmented Natural Images and its Alication to Evaluating Segmentation Algorithms and Measuring Ecological Statistics, Proc. of IV, vol., , 1. [6] Ruzon, M. and Tomasi,. Alha estimation in natural images, Proc. of IEEE VPR, 18 5,. [7] Sun, J., Jia, J., Tang,.-K., and Shum, H.-Y. Poisson matting, AM Trans. Grah., 3(3):315 31, 4. [8] Vezhnevets, V. and Konouchine, V. Growut: Interactive multi-label N-D image segmentation by cellular automata, Proc. of Grahicon, , 5. [9] Wang, J. and ohen, M. F. An iterative otimization aroach for unified image segmentation and matting, Proc. of IV, vol., , 5. Grahion'7 Russia, Moscow, June 3-7, 7

8 (a) (b) (c) (d) (e) (f) (g) (h) (i) (j) (k) (l) Figure 5 Examles of hierarchical Bayesian Matting. (a), (e), (i) Source Images. ar image is taken from the website of [], flower image is taken from the dataset [5], and the lighthouse image is taken from []. (b), (f), (j) Trimas. (c), (g), (k) Standard Bayesian Matting results. (d), (h), (l) Hierarchical Bayesian Matting results (comosites). See Table 1 for rocessing time comarison. About the authors Mikhail Sindeyev is a 3rd year student at Grahics and Media Laboratory of Moscow State Lomonosov University, Deartment of omutational Mathematics and ybernetics. His research interests include image and video rocessing, 3D reconstruction, comuter vision and adjacent fields. His address is msindeev@grahics.cs.msu.ru. Vadim Konushin is a 5th year student at Grahics and Media Laboratory of Moscow State Lomonosov University, Deartment of omutational Mathematics and ybernetics. His research interests include image and video rocessing, attern recognition, comuter vision and adjacent fields. His address is vadim@grahics.cs.msu.ru. Vladimir Vezhnevets is a vice head of Grahics and Media Laboratory of Moscow State Lomonosov University, Deartment of omutational Mathematics and ybernetics. He graduated with honors from Faculty of omutational Mathematics and ybernetics of Moscow State University in He received his PhD in in Moscow State University also. His research interests include image rocessing, attern recognition, comuter vision and adjacent fields. His address is vv@grahics.cs.msu.ru. Grahion'7 Russia, Moscow, June 3-7, 7