THE INCREASING demand for video signal communication

720 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 7, NO. 5, MAY 1998 A Bayes Decision Test for Detecting Uncovered- Background and Moving Pixels in Image Sequences Kristine E. Matthews, Member, IEEE, and Nader M. Namazi, Senior Member, IEEE Abstract We present a ternary hypothesis test for the detection of stationary, moving, and uncovered-background pixels between two image frames in a noisy image sequence using the Bayes decision criterion. Unlike many uncovered-background detection schemes, our scheme does not require motion estimation for the differentiation between moving pixels and uncovered-background pixels. We formulate the Bayes decision rule using a single intensity-difference measurement at each pixel and using multiple intensity-difference measurements in the neighborhood of each pixel. We quantitatively evaluate our detection algorithm on an image sequence which we have generated and qualitatively on the Trevor White image sequence. Index Terms Background prediction, hypothesis testing, image segmentation, sequence coding, uncovered-background detection. I. INTRODUCTION THE INCREASING demand for video signal communication services and the tremendous quantity of digital data such transmissions produce have been the impetus behind a vast amount of research in the area of redundancy reduction coding. Motion-compensated interframe coding has emerged as one of the most effective methods for reducing the quantity of transmitted information. Motion-compensated interframe coding predicts a future frame in an image sequence from a previous (or reference) frame by estimating the motion present in the scene and performing motion compensation on the previous frame. The existence of newly introduced pixels, such as uncovered-background pixels, in an image frame degrade the quality of the motion-compensated reconstructed image frames. This degradation arises from the fact that motioncompensated interframe coding schemes do not, in general, provide for uncovered-background pixels. Therefore, a good coding scheme will use background prediction in addition to motion compensation for interframe coding. Segmentation of an image frame in a sequence of images into regions of uncovered-background, moving, and stationary pixels is an essential part of uncovered-background prediction and motion compensation for image sequence coding Manuscript received September 17, 1995; revised May 28, 1997. This work was supported in part by a Catholic University Research Grant-in-Aid. The associate editor coordinating the review of this manuscript and approving it for publication was Prof. Patrick A. Kelly. K. E. Matthews was with the Department of Electrical Engineering, The Catholic University of America, Washington, DC 20064 USA. She is currently with Sharp Laboratories of America, Inc., Camas, WA 98607 USA (e-mail: matthews@sharplabs.com). N. M. Namazi is with the Department of Electrical Engineering, The Catholic University of America, Washington, DC 20064 USA. Publisher Item Identifier S 1057-7149(98)03075-9. [5] [9]. Some authors have suggested methods for uncoveredbackground detection based on change detection [1], but such methods are sensitive to noise. While other authors detect moving pixels in the presence of noise, they do not treat uncovered-background pixels [2]. In addition, most of the uncovered-background detection methods described in the literature use change detection for initial image segmentation and then use information provided by motion estimation to further segment the changed pixels into moving and uncoveredbackground pixels [1], [4]. In this paper, we present a method for segmenting an image into regions of pixels that correspond to those pixels which are displaced from the previous frame (i.e., moving pixels), those pixels which are the same as in the previous frame (i.e., stationary pixels), and those pixels which are newly introduced uncovered-background pixels (i.e., uncovered-background pixels). The basis of our method is ternary hypothesis testing using the Bayes decision criterion. Our method directly considers image noise and is thus more robust than change detection. In addition, our method does not require information provided by motion estimation. This is an advantage in that computationally expensive motion estimation is not needed for segmentation, and for coding purposes, the motion of only those pixel determined as moving needs to be estimated. The paper is organized into four additional sections. In Section II, we present the mathematical formulation of the problem. Section III describes the test experiments and Section IV shows the results of the experiments. We summarize and conclude in Section V. II. MATHEMATICAL FORMULATION A. Formulation of the Likelihood Ratio Test Consider two consecutive image frames in a sequence of image frames, and write the noisy intensity of the first (or previous) image frame as where denotes the spatial location,, and integer, of a pixel in the image frame, and are the noisy and noise-free intensities of the pixel, respectively, and is zero-mean, additive, white, Gaussian noise (AWGN). Assuming no illumination changes, no camera motion, and no changes in image acquisition parameters (i.e., camera focus etc.), the noise-free intensity at each pixel in the second (or current) frame is either a displaced value from the previous (1) 1057 7149/98$10.00 1998 IEEE

MATTHEWS AND NAMAZI: BAYES DECISION TEST 721 frame (i.e., a moving pixel), the same value as in the previous frame (i.e., a stationary pixel), or an uncovered-background value (i.e., an uncovered-background pixel). Therefore, we write the following expression for the noisy intensity of the present frame in terms of the intensity of the previous frame and the background intensity: where is a nonuniform-displacement vector with components and, is the noise-free intensity of the scene background, and,, and are regions of moving, stationary, and uncovered-background pixels, respectively. Defining and, and using the first-order approximation of, a valid approximation assuming the components of sufficiently small, we obtain expressions for the noisy intensity-difference image and where is the intensity-gradient vector of the previous frame at k, and denotes vector transposition. Given the measurement in (3) and (4), we formulate the following three hypotheses: Under the hypotheses assignments of (5), we form the likelihood ratios and (2) (3) (4) (5) (6) (7) B. Evaluating the Likelihood Ratios Namazi et al., developed an expression for in [2]. We repeat their results here, and a detailed derivation and treatment can be found in the reference. They showed that where (9) (10) (11) (12) is the support of, is the noise variance, denotes determinant, and is the covariance matrix of the motion vector for any, and is of the form (13) where and are the variances of and, respectively, and is the cross-variance between and. Assuming that the motion variance is the same in both the - and -directions and that the motion in the - and -directions are uncorrelated with each other and with, then (14) where denotes the Euclidean norm, and is the variance of the intensity-difference image. Namazi et al. approximate by. It remains to evaluate the likelihood ratio of (7). Begin by assuming the background and the object intensities are statistically independent. Define, and proceed as follows: (15) where denotes the conditional probability density function (pdf). The ternary hypothesis test using (6) and (7) is shown in (8), at the bottom of the page, where is the a priori probability of a pixel belonging to the region corresponding to, and is the cost of deciding that a pixel belongs to the region corresponding to when the pixel actually belongs to the region corresponding to where is the support of, and (16) (8)

722 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 7, NO. 5, MAY 1998 Therefore, as shown in (17), at the bottom of the page, and from [2], we have (18) Thus, the likelihood ratio in (7) is given in (19), shown at the bottom of the page. From (19) it is clear that in order to evaluate, knowledge of is necessary. 1) Evaluation of : Recall,, where is the intensity of the uncovered background, and refers to the region of uncovered background; thus, refers to the intensity of the scene pixels which have moved from one frame to the next uncovering the background. Therefore, is the intensity of the portions of the moving object, in particular, the pixels on the boundaries of the moving object, which uncover background as the object moves. We assume the statistics of the intensity of the scene background will sufficiently describe the statistics of the intensity of the uncovered background. Similarly, it is logical that the statistics of the intensity of the primary object in the scene will sufficiently describe the statistics of the intensity of the moving-object pixels. We use the intensity histogram of the moving object and the intensity histogram of the background to estimate.in order to determine these histograms, we use an active contour model, under the assumption that a likely application for this type of detection is video-teleconferencing or videophone, to extract the contour of the moving object. From the contour, we form an image mask of the boundary pixels of the moving object for the histogram calculation. Given the videoteleconferencing or videophone assumption, it is reasonable that the moving object is a person and that the person is likely to be nearly centered in the video frame. Active contour models (i.e., snakes) describe the boundary detection problem in terms of energy minimization. Internal energy functionals impose constraints on the flexibility and rigidity of the contour or snake, and external energy derived from the image attracts the contour to the salient image features. The literature contains many examples of applications and formulations of snakes [10] [15]. Waite and Welsh use snakes to determine the head boundary in an image using a variational calculus approach [10]. For the extraction of the boundary of the person in the image frame, we implemented Lai and Chin s snake formulation as described in detail in [12]. They use the minimax criterion to avoid explicitly selecting the local regularization parameters, and they formulate the energy functionals for robust, rotation Fig. 1. Snake initialization. Right line: from Hough transform. Left line: from reflection. invariant, boundary detection. In addition, they propose and show that the Hough transform is an appropriate initialization for the contours, therefore allowing automated initialization. In order to speed up the contour extraction, we use a subsampled, slightly blurred version of the first frame in the image sequence for the boundary detection. This reduces the computations in both the Hough transform and each iteration of the snake algorithm. We use the Hough transform to determine the initial position of the snake elements. We use the highest peak in the parameter space to find the dominant line in the image. To obtain the initial snake position, we reflect the line from the Hough transform about the vertical line through the top endpoint. We extend the reflected line to the image boundary and close the contour formed by the line generated by the Hough transform and the reflection of this line. Fig. 1 shows the initial location of the snake superimposed on the image. In Fig. 2 we show the location of the snake at convergence. We use the converged snake to form a mask to separate the object from the background, and we calculate the intensity histogram for each region. For, we use a strip of the boundary pixels from inside the contour (see Fig. 3), as these are the moving object pixels most likely to uncover the (17) (19)

MATTHEWS AND NAMAZI: BAYES DECISION TEST 723 statistical independence of the noise samples, and assuming independence of image intensity at different pixels, the joint pdf in the multiple measurements classification is the product of the individual pdf s; that is, (20) Fig. 2. Snake at convergence. III. PERFORMANCE EVALUATION In order to evaluate the segmentation algorithm, we performed several sets of experiments. The experiments fall into two categories: i) experiments to quantitatively determine the performance of the ternary hypothesis test and ii) experiments to qualitatively determine the performance of the ternary hypothesis test. The results for each quantitative experiment are in the form of a confusion matrix which indicates the percentage of pixels of a given region that are classified in each of the three different regions. We qualitatively evaluated the method using a standard image sequence. We used several frames of the Trevor White sequence, and we show the resulting segmentations for visual inspection. The gradient values used in the hypothesis test are estimated from the noisy reference frame, and AWGN is independently added to each frame. The noise variance of the difference image,, is therefore the sum of the noise variances of each frame. The noise variance of each frame is assumed known, and as in [2], we approximate by. A. Quantitative Evaluation To quantitatively evaluate the ternary hypothesis test, we generated several test image sequences consisting of moving objects and different background scenes. For a range of signal-to-noise ratios (SNR s), we segmented the image frame using the ternary hypothesis test with single measurements for classifying each pixel and multiple measurements for classifying each pixel. We define SNR in db as SNR (21) Fig. 3. Mask for histogram calculation. background pixels. We approximate the pdf of by the convolution of the normalized histogram of the background with the flipped normalized histogram of the object. C. Single Measurements and Multiple Measurements We perform the ternary hypothesis test using a single sample, the intensity-difference at the spatial location of the pixel we are classifying, or a vector of samples, the intensitydifferences in an ( and odd) neighborhood of pixels centered on the pixel we are classifying. Due to the where is the average image variance. From the resulting segmentations, we calculate a confusion matrix. The experiments to quantitatively evaluate the decision test were designed to measure the performance of the hypothesis testing in a controlled environment. For these tests, we assumed knowledge of the pdf s of and. B. Qualitative Evaluation We use two frames of the Trevor White image sequence to qualitatively test the performance of the detection algorithm. We test on two frames of the sequence that are sufficiently far apart to ensure regions of uncovered-background and moving pixels. We present the resulting segmentations for visual inspection. For these tests, we incorporate the application assumption of a person approximately centered in the image

724 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 7, NO. 5, MAY 1998 Fig. 4. Generated test image sequence. Noise-free previous frame. Noise-free present frame. where, for example, corresponds to the cost of classifying an uncovered-background pixel as a stationary pixel. We performed simulations using several different a priori probabilities and costs. The cost values are application dependent and should reflect the tolerance for misclassification in the various regions. The a priori probabilities are image sequence dependent and should reflect a priori knowledge about the expected amount of motion in the image sequence. The results of the decision test will be influenced by the accuracy of this a priori information. IV. RESULTS Fig. 5. Generated test image sequence: true segmentation. Black: stationary pixels. White: moving pixels. Gray: uncovered-background pixels. and use the snake algorithm as described in Section II-B1 to experimentally determine the pdf s of and. C. A Priori Knowledge In order to formulate the ternary hypothesis test, we must provide the a priori probabilities,, and the costs,. We represent this information in the form (22) where is the a priori probability that a pixel is moving between image frames, is the a priori probability that a pixel is stationary between image frames, and is the a priori probability that a pixel is a background pixel which is uncovered between image frames. We represent the a priori costs as a matrix of the form (23) A. Quantitative Performance Evaluation We used the two image frames shown in Fig. 4 to quantitatively evaluate the ternary hypothesis test. Fig. 5 shows the true segmentation of the image frame with white representing moving pixels, black representing stationary pixels, and gray representing uncovered-background pixels. In Fig. 6 we show segmentations resulting from hypothesis testing for image frames with an SNR of 20 db and for the two different cost matrices and (24) (25) and the probability matrix. The corresponding confusion matrices are shown in Tables I and II, respectively. In Fig. 7 we show the segmentations of the 20 db SNR test sequence that result from using the true a priori probabilities and the two cost matrices and. The confusion matrices for these test cases are shown in Tables III and IV.

MATTHEWS AND NAMAZI: BAYES DECISION TEST 725 Fig. 6. pixels. Gray: uncovered-background pixels. Resulting segmentations using single measurements at SNR of 20 db. C (1) and P. C (2) and P. Black: stationary pixels. White: moving Fig. 7. White: moving pixels. Gray: uncovered-background pixels. Resulting segmentations using single measurements at SNR of 20 db. C (1) and P (true). C (2) and P (true). Black: stationary pixels. TABLE I CONFUSION MATRIX (IN PERCENTAGE) AT SNR OF 20 db USING SINGLE MEASUREMENTS, P, AND C (1) TABLE II CONFUSION MATRIX (IN PERCENTAGE) AT SNR OF 20 db USING SINGLE MEASUREMENTS, P, AND C (2) Using the cost matrix assigns no cost to a correct decision and the same cost to all incorrect decisions. Using however, makes it more costly to make an incorrect assignment of a pixel to the region of stationary pixels. The differences between Fig. 6 and clearly demonstrate the effect of the cost matrix on the decision test. We see in Fig. 6 the probability of detecting moving and uncovered- background pixels is increased at the expense of many more false alarms in the stationary region. Fig. 8 shows segmentation results using single measurements and multiple measurements (3 3 neighborhood) and the cost matrix for image frames with an SNR of 5 db. In this high noise situation, the use of measurements in the neighborhood of the pixel enhances the decision procedure.

726 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 7, NO. 5, MAY 1998 Fig. 8. Resulting segmentations using C (1) and P at SNR of 5 db. Single measurements. Multiple measurements. Black: stationary pixels. White: moving pixels. Gray: uncovered-background pixels. Fig. 9. Trevor White image sequence. Noise-free previous frame. Noise-free present frame. TABLE III CONFUSION MATRIX (IN PERCENTAGE) AT SNR OF 20 db USING SINGLE MEASUREMENTS, P (true), AND C (1) TABLE IV CONFUSION MATRIX (IN PERCENTAGE) AT SNR OF 20 db USING SINGLE MEASUREMENTS, P (true), AND C (2) B. Qualitative Performance Evaluation We used two frames of the Trevor White sequence to further examine the performance of the Bayes test. The two frames of the sequence that we used are shown in Fig. 9. Between the two frames, the motion of Trevor White is up and to the right in the image. We therefore expect to see uncovered-background pixels at his right arm and moving pixels at his left arm. In Fig. 10 and we show the segmentation which results from the ternary hypothesis test using the cost matrix, the a priori probabilities, and single and multiple measurements (3 3 neighborhood), respectively, for the 20 db SNR sequence. The segmentations using the

MATTHEWS AND NAMAZI: BAYES DECISION TEST 727 Fig. 10. Resulting segmentations using C (1) and P at SNR of 20 db. Single measurements. Multiple measurements. Black: stationary pixels. White: moving pixels. Gray: uncovered-background pixels. Fig. 11. Resulting segmentations using C (1) and P at SNR of 5 db. Single measurements. Multiple measurements. Black: stationary pixels. White: moving pixels. Gray: uncovered-background pixels. same parameters and the 5 db SNR sequence are shown in Fig. 11 and. Visual inspection of these segmentations shows that the background pixels which are uncovered by the motion of the right arm are indeed detected even at an SNR of 5 db, as are the moving pixels of the left arm. The uncovered-background pixels under the left arm are also detected, especially in the segmentations resulting from multiple measurements. Problems are evident however, in the region of Trevor White s tie. This misclassification is due to the similarity of the histograms of the tie and the background scene. V. SUMMARY AND CONCLUSIONS We have presented a ternary hypothesis test, based on Bayes decision criteria, for the detection of moving, stationary, and uncovered-background pixels in image sequences. The anticipated use of such a test is the improvement of motioncompensated, background-predictive video coding with application to video-teleconferencing and videophone. The ternary hypothesis test does not require prior motion estimation, and the test is robust to noise. We have shown that the use of snakes for extracting the contour of a person in an image frame allows automated calculation of the pdf, and hence automation of the ternary hypothesis test for the application of video-teleconferencing and videophone. In this paper, we have formulated and tested the performance of the ternary hypothesis test. We have presented segmentation results based only on the strength of the hypothesis test. In many uncovered-background detection schemes, other authors suggest further processing of the segmentation

728 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 7, NO. 5, MAY 1998 results to eliminate small isolated regions and to smooth region boundaries [1]. We do not consider further processing of the segmentation results from the ternary hypothesis test in this paper so as not to obscure the evaluation of the hypothesis test. [13] D. Geiger, A. Gupta, L. A. Costa, and J. Vlontzos, Dynamic programming for detecting, tracking, and matching deformable contours, IEEE Trans. Pattern Anal. Machine Intell., vol. 3, pp. 294 302, Mar. 1995. [14] A. Blake and A. Yuille, Eds., Active Vision. Cambridge, MA: MIT Press, 1992. [15] S. Ranganath, Contour extraction from cardiac MRI studies using snakes, IEEE Trans. Med. Imag., vol. 14, pp. 328 338, June 1995. REFERENCES [1] R. Thoma and M. Bierling, Motion compensating interpolation considering covered and uncovered background, Signal Process.: Image Commun. 1, pp. 191 212, 1989. [2] N. M. Namazi, P. Penafiel, and C. M. Fan, Nonuniform image motion estimation using Kalman filtering, IEEE Trans. Image Processing, vol. 3, pp. 678 683, Sept. 1994. [3] H. L. Van Trees, Detection, Estimation, and Modulation Theory, Part I. New York: Wiley, 1968. [4] C. Lettera and L. Masera, Foreground/background segmentation in videotelephony, Signal Process.: Image Commun. 1, pp. 181 189, 1989. [5] N. Diehl, Object-oriented motion estimation and segmentation in image sequences, Signal Process.: Image Commun. 3, pp. 23 56, 1991. [6] D. Hepper, Efficiency analysis and application of uncovered background prediction in a low bit rate image coder, IEEE Trans. Commun., vol. 38, pp. 1578 1584, Sept. 1990. [7] N. Mukawa and H. Kuroda, Uncovered background prediction in interframe coding, IEEE Trans. Commun., vol. COMM-33, pp. 1227 1231, Nov. 1985. [8] S. C. Brofferio, An object-background image model for predictive video coding, IEEE Trans. Commun., vol. 37, pp. 1391 1394, Dec. 1989. [9] S. Brofferio and F. Rocca, Interframe redundancy reduction of video signals generated by translating objects, IEEE Trans. Commun., pp. 448 455, Apr. 1977. [10] J. B. Waite and W. J. Welsh, Head boundary location using snakes, in Audiovisual Telecommunications, N. D. Kenyon and C. Nightingale, Eds. London, U.K.: Chapman & Hall, 1992. [11] L. D. Cohen and I. Cohen, Finite-element methods for active contour models and balloons for 2-D and 3-D images, IEEE Trans. Pattern Anal. Machine Intell., vol. 15, pp. 1131 1147, Nov. 1993. [12] K. F. Lai and R. T. Chin, On regularization, formulation and initialization of the active contour models (snakes), in Asian Conf. Computer Vision, Osaka, Japan, Nov. 1993, pp. 542 545. Kristine E. Matthews (S 85 M 98) received the B.S. degree in 1985 from the University of Colorado, Boulder, the M.S. degree in 1988 from the University of Wisconsin, Madison, and the Ph.D. degree in 1997 from The Catholic University of America, Washington, DC, all in electrical engineering. From September 1995 to January 1997, she was an NIH Pre-Doctoral IRTA Research Fellow. From 1989 to 1992, she was a Technical Staff Member at Los Alamos National Laboratory, Los Alamos, NM, where she also worked as a Graduate Research Assistant during summers as a student. From 1988 to 1989, she was a Member of the Associate Professional Staff at the Applied Physics Laboratory, Johns Hopkins University, Laurel, MD. She is currently a Member of the Technical Staff, Digital Video Department, Sharp Laboratories of America, Inc., Camas, WA. Dr. Matthews is a member of Tau Beta Pi, Eta Kappa Nu, and SWE. Nader M. Namazi (S 84 M 85 SM 92) received the bachelor s degree in electrical engineering from the Iran College of Science and Technology, Tehran, in 1977, and the M.S. and Ph.D. degrees in electrical engineering from the University of Missouri Rolla in 1981 and 1985, respectively. From 1985 to 1992, he was with the Michigan Technological University, Houghton, where he was promoted to the rank of Associate Professor. Since 1992, he has been with The Catholic University of America, Washington, D.C., where he is currently an Associate Professor of Electrical Engineering. He was the Editor-in-Chief of the International Journal of Modeling and Simulation from 1994 to 1996. He is the author of New Algorithms for Variable Time Delay and Nonuniform Image Motion Estimation (Ablex, 1994). Dr. Namazi is a member of Eta Kappa Nu.