Upper-Body Contour Extraction Using Face and Body Shape Variance Information

Transcription

1 Uer-ody Contour Etraction Using ace and ody Shae Variance nformation Kazuki Hoshiai 1, Shinya ujie 2, and Tetsunori Kobayashi 1 1 Deartment of Comuter Science and Engineering, Waseda University, Okubo 3 4 1, Shinjuku ku, Tokyo , Jaan hoshiai@cl.cs.waseda.j, koba@waseda.j 2 Waseda nstitute for Advanced Study, Waseda University, Nishiwaseda 1 6 1, Shinjuku-ku, Tokyo , Jaan fujie@cl.cs.waseda.j Abstract. We roose a fitting method using a model that integrates face and body shae variance information for uer-body contour etraction. Accurate body-contour etraction is necessary for various alications, such as ose estimation, gesture recognition, and so on. n this study, we regard it as the shae model fitting roblem. A model including shae variance information can fit to the contour robustly even in the noisy case. AAMs are one of these models and can fit to a face successfully. t needs aearance information for effective fitting, but it can not be used in our case because aearance of uer-body easily changes by clothes. nstead of intensity image, roosed method uses edge image as aearance information. However, discrimination between a true contour edge of uer-body and other edges is difficult. To solve this roblem, we integrate shaes of uer-body and face. t is eected that this integrated model is more robust to edges in clutter background and various locations of the body than a body shae model using only body shae information. We conduct eeriments and confirm imrovement in accuracy by integration of face and body variance information. Keywords: Contour etraction, Active Aearance Models, Active ody Shae Models, Active ntegrated Shae Models. 1 ntroduction We roose a contour shae model integrating body shae variance information and face model of Active Aearance Models AAMs [1,2], and achieve accuracy imrovement of uer-body contour etraction. Etracting human contour with high accuracy is imortant to estimate ositions of hysical arts such as arm, head and so on. The most common aroach to etract human contour is background subtraction [3]. A system such as a robot with active cameras can not utilize this aroach which assumes fied camera. There is Snakes [4] as another contour etraction method. t does not use rior knowledge about the contour shae. Owing to this, the model instance tends to converge on a wrong shae. n this study, we regard body etraction roblem as shae model fitting to a contour. We roose a new model and a novel fitting method using nverse Comositional mage Alignment CA algorithm [2,5] of AAMs. T. Wada,. Huang, and S. Lin Eds.: PSVT 2009, LNCS 5414, , c Sringer-Verlag erlin Heidelberg 2009

2 Uer-ody Contour Etraction Using ace and ody Shae Variance nformation ntegration of ace and ody Shae Model We regard human contour etraction as shae model fitting to a body contour. n this study, we try to imrove a model fitting accuracy, using the integrated model of face and body shae variance information. This section describes fitting systems of face and body, and later describes a fitting system of integrated their systems. 2.1 ace Model Active Shae Models ASMs [6] are a method of searching object contour using shae model. t has a habit of robustness to noise for learning the average shae and nonrigid deformation attern using a set of training images reliminarily given the coordinates of feature oints. or enormous amount of calculation, ASMs are etended to aly hierarchical aroach using low-resolution images and motion rediction using Kalman filter for reducing iteration count of fitting [7]. One of the successful object detection and searching methods is AAMs. t has models constructed from both shae and aearance, and erforms matching a normalized inut image and a temlate image. AAMs allow real time tracking using CA algorithm as a model fitting algorithm. ASMs and AAMs constrain deformation of the object shae which consists of a number of feature oints by Princial Comonent Analysis PCA. Their fitting algorithms have considerably difference between their models. After transferring each feature oint, ASMs arrange the all oints in accordance with PCA. On the other hand, AAMs transfer the general shae while maintaining consistency of the relation between the oints, for solving the otimization roblem of shae arameter based on eigen vector for fitting the object. As a result, AAMs can fit more accurately than ASMs. urthermore, AAMs outerform ASMs about the convergence seed of fitting by CA algorithm. The fitting rocess of AAMs is as follows. irstly inut images are transformed to temlate sace using shae arameters. Secondly, the error of the translated image and the temlate image is calculated. inally, the shae arameters, minimizing the error, are calculated using a face model. Model fitting to the face is erformed using the shae arameters based on the calculation result. The fitting system of AAMs to face region is illustrated in ig. 1 a. 2.2 ody Model We aly AAMs to uer-body contour for fitting. When we use AAMs to face, a temlate image is a face image. A face includes universal information that does not deend on individual, so it can use intensity values as aearance information directly. On the other hand, it is difficult for uer-body model to use only intensity values inside its contour, because uer-body vision changes by clothes easily. Using the edge detected in boundary of human region and background region, we regard an edge image as aearance information to solve this roblem. The flow of fitting system is similar to a system of face ecet for regarding an edge image as an inut image. The fitting system to uer-body contour is illustrated in ig. 1 b.

3 864 K. Hoshiai, S. ujie, and T. Kobayashi Shae Parameter nut mage Transform E = T Transform mage W Error E W Temlate mage T Shae Parameter nut mage Edge Detection Edge mage s Transform s E = T Transform mage s W Error E W S Temlate mage T Udate Parameter ace Shae Model Udate Parameter ody Shae Model a b ig. 1. itting algorithms for face shae model and uer-body shae model. a itting algorithm for face shae model, b itting algorithm for uer-body shae model. 2.3 ntegrated Model An edge image includes not only an uer-body contour edge of target erson but also other edges. To distinguish an uer-body contour edge from other edges is difficult. Even in noisy environment, face model can be fitted because it does not deend on background. n this study, we integrate two models described in revious sections and Shae Parameter nut mage Edge Detection Edge mage S Transform Transform mage W Transform S Transform mage W S E = T W Temlate mage T E = T W S Temlate mage T Error E Error E Udate Parameter ntegrated Shae Model ig. 2. itting algorithm for integrated model

4 Uer-ody Contour Etraction Using ace and ody Shae Variance nformation 865 limit the search sace for uer-body contour based on the osition, direction and scale of face. Procedure of fitting the model which is constructed from both face and body shae variance information is as follows. irstly we erform transforming inut image in face region, and edge image in uer-body contour, using integrated shae arameters of face and body shae variance. Secondly, the errors are calculated for face and body, searately. inally, the shae arameters minimizing that errors are calculated by the integrated model. Model fitting to a face and a body contour is erformed based on the shae arameters obtained by calculation result. The fitting system of integrated model is illustrated in ig Active Aearance Models We briefly overview the AAMs and their efficient fitting algorithm [1,2]. Then, we summarize the conditions which a model must satisfy, in order to aly that algorithm. 3.1 Shae Model AAMs consist of two active models. irst one is the shae model. A shae is defined as a set of v vertices and lines connecting them, as shown in ig. 3 a. The shae vector s is reresented as s =[ 1,y 1, 2,y 2,, v,y v ] T 1 where i and y i reresent and y coordinate of ith verte resectively. PCA is alied to training data which are face images with hand-labeled feature oints, then a shae is reresented by linear sum of the average shae and the difference shaes, s = s 0 + n i s i 2 where s 0 is the average shae and s i is ith rincial comonent as a difference shae. Shae arameters are reresented as the vector =[ 1, 2,, n ] T.Theshaeof model instance is determined by the. 3.2 Aearance Model Second model is an aearance model. Aearance model reresents the variance of grayscale image in temlate sace. Temlate sace is defined as 2-D sace constructed by the average shae. Alying PCA to the face images of training data reresented in temlate sace, an average aearance A 0 and m rincial comonents A i are calculated. then describes iels in temlate sace. We regard the average aearance as the averaged face image, and the rincial comonents as difference face image. A =A 0 + i=1 m λ i A i 3 i=1

5 866 K. Hoshiai, S. ujie, and T. Kobayashi a b c d ig. 3. Active Aearance Models and Active ody Shae Models. a asic shae of AAMs s 0, b Temlate image of AAMs T, c asic shae of ASMs s 0, d Temlate image of ASMs T. Aearance arameters are reresented as the vector λ =[λ 1,λ 2,,λ m ] T.The aearance of the model instance with resect to the face region is determined by the λ. The averaged face image A 0 is regarded as the temlate image T shown in ig. 3 b. 3.3 itting Algorithm itting AAMs is regarded as minimizing the error between an inut image and model instance. The error function is defined as m E =A 0 + λ i A i W 4 i=1 where is an inut image and W is a war function waring oints in temlate sace to relative oints in inut image sace. Note this waring function is determined by a shae arameter. or simlicity, we ignore aearance variation. Then, the target is to find a arameter that minimizes the sum of square error function, that is argmin s 0 [T W ] 2. 5 ecause, is usually a non-linear function, it is difficult to calculate directly. Then, AAMs use fitting algorithm named nverse Comositional mage Alignment CA. Small war reresented by Δ is introduced to temlate sace, argmin Δ s 0 [T W Δ W ] 2. 6 Δ that minimizes the error function in contet of given is calculated. 4 Active ody Shae Models We roose a fitting method of shae model using rior knowledge of contour variance information. We aly the framework of AAMs to body contour model in order to use rior knowledge. We call this model Active ody Shae Models ASMs.

6 Uer-ody Contour Etraction Using ace and ody Shae Variance nformation ody Shae Model A body shae model is reresented by the vector of arranged,y coordinate value of u feature oints. s =[ 1,y 1, 2,y 2,, u,y u ] T 7 An average shae s and l rincial comonents s i training data, which are similar to Eq. 2. are calculated by PCA for all l s = s 0 + i s i 8 Here, i describes the size of shae variance in terms of uer-body contour. n addition, shae arameters are reresented as the vector =[ 1, 2,, l ] T.The shae of the model instance with resect to the uer-body contour is determined by. i=1 4.2 Alying CA Algorithm to ASMs The differences of ASMs from normal AAMs are 1 inut and temlate images are edge images, and 2 shae is just a line not closed olygons. ormer means that the gradient of temlate image cannot be calculated because edge image is a set of shae lines. t also causes a roblem for the error function calculation, because edge image is very sarse, so the error value has low reliability. Latter means the iece-wise Affine transform cannot be used as the war function directly. Against these roblems, we redefine temlate image, inut image, and war function. 1 Temlate image: T A line is constructed by connecting vertices in basic shae sequentially. We smooth this line to a constant distance and treat it as a temlate image. We show eamles of a basic shae and a temlate image in ig. 3 c,d. We fied the etraction range above elbows, and defined the number of feature oints was 35, in this study. 2 nut image: s The edge image is etracted by alying canny edge detector to original inut image. Then this image is smoothed and the iel values are normalized from 0 to 255 for the matching. We regard this image as new inut image for ASMs. An eamle of generation of an edge image is shown in ig War function: W Temlate sace is regarded as belt-like sace around a line linking the feature oints. This belt-like sace is constructed by the set of quadrangular area. We can transform each quadrangular area in temlate sace to inut image sace. The dividing boundaries are bisectors of angles between two lines through an objective oint and two oints net to that oint. The war function maing from inut image sace to temlate sace is iecewise bilinear war using bilinear transform [8] for each quadrangle area. We consider a case of rojection about four vertices of a certain quadrangle area in temlate

7 868 K. Hoshiai, S. ujie, and T. Kobayashi nut image Edge image Smoothed edge image ig. 4. Process of forming the smoothed edge image used for matching with the temlate image sace to 00,y 00, 10,y 10, 01,y 01, 11,y 11 with the shae arameter.when each iel is denoted by =[, y ] T, the war function is described as follows. W =1 α1 β[ 00,y 00 ] T + α1 β[ 10,y 10 ] T +1 αβ[ 01,y 01 ] T + αβ[ 11,y 11 ] T 9 α and β are transform coefficients of bilinear transform from temlate sace to square sace. 5 ntegration of ASMs and AAMs The uer-body etraction using only edge information is difficult, when there are many edges in background as well as the uer-body contour. n this section, we describe how we integrate two models to tackle this roblem. n following art, we call normal AAMs for face as Active ace Aearance Models AAMs, and attach as suerscrit to arameters and model secific functions for AAMs. 5.1 ntegration of Shae Models The new shae model is a model obtained by integrating two models, the body shae model of ASMs and the face shae model of AAMs. We call this model integrated shae model. s =[ 1,y 1, 2,y 2,, v+u,y v+u ] T 10 An average shae s 0 and k rincial comonents s i are calculated by PCA for all training data. s = s 0 + k i s i 11 n addition, shae arameters are reresented as the vector =[ 1, 2,, k ] T. The shae of the model instance with resect to both the uer-body contour and the face region is secified by the. An aearance model is constructed for only face aearance model of AAMs. We call the model consisting of integrated shae model and face shae model and the fitting algorithm of it Active ntegrated Shae Models ASMs. i=1

8 Uer-ody Contour Etraction Using ace and ody Shae Variance nformation itting Algorithm The evaluation function of fitting ASMs is defined as weighted sum of error for uerbody and error of face. argmin 1 w [T W Δ s W ] 2 Δ s 0 + w s 0 [ T W Δ W ] 2 12 The range of weight is 0 <w<1. This weight reresents ratio of influence by face information and body information. n this study, we emirically determined w =0.4.We take the Taylor series eansion of T W Δ and T W Δ in Eq w [T + T W Δ s W ] 2 s 0 + w [T + T W Δ W ] 2 13 s 0 Δ minimizing Eq. 13 is comuted. Δ = H 1 1 w [ T W ][ s W T ] s 0 + w s 0 [ T W Then, we denote hessian matri H as follows. H =1 w s 0 + w s 0 [ T W [ T W ][ W T ] 14 ] T ] [ T W ] T ] [ T W Δ minimizing Eq.12 is calculated analytically by Eq.14. can be udated sequentially from obtained Δ and current. The otimal shae arameter is rovided by Δ sequentially, and fitting to the body contour and the face is erformed simultaneously. 6 Eeriments 6.1 ASMs Eeriment We evaluated the fitting to uer-body contour. We conducted comarative eeriments for evaluating accuracy with roosed ASMs of the modeling method described in 4 and CA algorithm described in

9 870 K. Hoshiai, S. ujie, and T. Kobayashi Setu. We erformed data collection to reare test data for the evaluation and learning data to construct the body shae model. Subjects were 15 students in our laboratory. The collected data included four kinds of atterns facing on the front facing sideways a little inclining to front and back inclining to right and left The subjects erformed the indicated movement of each attern. We collected 200 image data, caturing the video corresonding to each attern. The eamles of eeriment data are shown in ig. 5. We comuted the error between fitting result and labeled data. We regarded the distance from the correct feature oint to a nearest iel on the fitting result as the error. The image data for eeriment were 200 images described in 6.2. The data were divided into four sets, and then one set consists of 50 images. Three sets including 150 images were learning data, and one set was test data. We erformed eeriments with four combinations. We utilized Snakes which is widely used for the contour etraction as comarison method of ASMs. Drawing a line by connecting the start oint and the end oint of feature oints, we generated the edge, and artificially made closed region. We erformed the model fitting to the closed region of uer-body contour. We set the same shae of initial model instance. These initial model instances were randomly determined as slightly larger than hand labeled correct shaes. We set the distance d, of which there are ten kinds from 1 to 10, from the model instances. We reared one hundred combinations of the oints which draw aart from the three correct feature oints, and measure the error for each combination. Results and discussion. We calculated the error er one data of Snakes and roosed ASMs. Note that the unconverged data were eliminated for this evaluation. We calculated the mean of the errors of all test data. They are for Snakes and for ASMs. This result shows that ASMs is more accurate than the traditional method. The model instance of Snakes converged inside the closed region consisting of the uer-body contour, when the closed region was broken by lacking the edge over the uer-body contour. n contrast, ASMs fitted the contour successfully even in such a case, because the model instance kees the shae consistency for the body shae model. a b c ig. 5. The eamle of eeriment data: a facing sideway, b inclining to front, c inclining to left

10 Uer-ody Contour Etraction Using ace and ody Shae Variance nformation ASMs Eeriment We conducted an eeriment with ASMs and ASMs to measure the model fitting accuracy to uer-body contour. Setu. We erformed data collection to reare test data for the evaluation and learning data to construct the models. Subjects were 5 students in our laboratory. The collecting data included three kinds of attern. 1. facing on the front 2. facing sideway a little 3. inclining to right We gave instruction to the subjects to ose each attern. We catured the image at random from the video and collected in total 500 images which includes 100 images er subject. We labeled feature oints by hand to a face and a body contour on the catured images, and these oints coordinate were used as correct label reresenting shae. n this study, we used total 103 of the feature oints which consist of 68 face oints and 35 uer-body contour oints. We show an labeled image and the feature oint location in ig. 6. We used 100 images of one subject as test data and the other data as learning data, and erformed 5-fold cross validation. We used the images of training data sets and flied images of them to construct the models. 481 images succeeded in face detection. The shae arameter of ASMs converged for 440 images. The shae arameter of ASMs converged for 399 images. The shae arameter of both ASMs and ASMs converged for 393 image. The face osition was determined automatically by the face detector [9], and the osition of initial model instance was determined by the face osition around the body. We udated the shae arameter until it converges. When the arameter converges, we measured the distance error er oint between the model instance of feature oints and the hand-labeled correct feature oints. We counted the error of the data which converge both of ASMs and ASMs, and then we calculated the averaged error er feature oint. Results and discussion. The averaged errors er feature oint for ASMs and ASMs were 9.9 and 8.7 resectively. ig. 7 shows the cumulative frequency distribution of a b ig. 6. The eamle of labeled image: big oints are feature oints of uer-body, and small oints are feature oints of face region, a Labeled image, b eature oints location

11 872 K. Hoshiai, S. ujie, and T. Kobayashi ASMs ASMs Cumulative frequency[%] Distance error [iel] ig. 7. Cumulative frequency distribution of the distance error er feature oint for ASMs and ASMs a teration 1 b teration 50 c teration 100 d teration 150 ig. 8. Process of fitting convergence, the uer row:asms, and the lower row:asms teration count 1,50,100,150 the distance error er feature oint. This result concludes that ASMs often converge nearer than ASMs. We confirmed that the integration of the face information made the error reduced. The recision of the model fitting are imroved by the integration of ASMs and AAMs. The fitting rocess of the model instance is shown in ig. 8. The round oints show feature oints, and the line of the model instance shows the result of uer-body etraction. We discuss the eeriment result of ig. 8. The osition of the initial model instance was determined by the face region. Thereby, the uer-body was considerably away from correct osition with resect to this samle. n consequence, the model instance did not fit to the contour by ASMs. However, ASMs succeeded in the body art of the model fitting to the contour while the face art of the model instance turned around to the correct osition by using face information. The uer-body contour and face shae variance constrained each other by integration of body and face shae, while fitting to

12 Uer-ody Contour Etraction Using ace and ody Shae Variance nformation 873 the face using the inut image, the model instance fits to the contour along the osition, direction, and scale of face. Therefore the model instance fitted the uer-body contour. 7 Conclusions or uer-body contour etraction, we roosed ASMs as the fitting method using CA algorithm. ASMs has an uer-body contour shae model that reresents shae variance information. We confirmed that fitting recision of ASMs was high in comarison with the traditional method to the uer-body etraction by the eeriment. urthermore, we roosed ASMs as the fitting method that integrated AAMs and ASMs. We introduced weight for a face and hysical error function in ASMs and defined new evaluation function. Proosed method can estimate the most suitable shae arameter using CA algorithm for this evaluation function. We evaluated the fitting recision for ASMs and ASMs. As a result, high fitting recision of ASMs was achieved in comarison with ASMs and we showed the effectiveness of using face information for the issue of uer-body contour etraction. n this study, we used a fied weight for the error function of face and the uer-body contour etraction. n future, we are going to use a dynamic weight obeying a reliability with an uer-body contour etraction and a face information. This reliability can be calculated with the comleity of the edge by the background. Moreover we aim at recision imrovement by taking in the aearance information of clothes and using much aearance information. Acknowledgment This work was artly suorted by New Energy and ndustrial Technology Develoment Organization NEDO, Jaan. References 1. Cootes, T.., Edwards, G.J., Taylor, C.J.: Active aearance models. EEE Trans. on Pattern Analysis and Machine ntelligence 236, Matthews,., aker, S.: Active aearance models revisited. nternational Journal of Comuter Vision 602, Li, L., Huang, W., Gu,.Y., Tian, Q.: Statistical modeling of comle backgrounds for foreground object detection. EEE Trans. on mage Processing 1311, Kass, M., Witkin, A., Terzooulos, D.: Snakes: Active contour models. nternational Journal of Comuter Vision 14, aker, S., Matthews,.: Lucas-kanade 20 years on: A unifying framework. nternational Journal of Comuter Vision 563, Cootes, T.., Taylor, C.J., Cooer, D.H., Graham, J.: Active shae models their training and alication. Comuter Vision and mage Understanding 611, Lee, S.W., Kang, J., Shin, J., Paik, J.: Hierarchical active shae model with motion rediction for real-time tracking of non-rigid objects. ET Comuter Vision 11, Heckbert, P.S.: undamentals of teture maing and image waring. Master s thesis, University of California Viola, P., Jones, M.J.: Raid object detection using a boosted cascade of simle features. nternational Journal of Comuter Vision 572,