A Multi-View Nonlinear Active Shape Model Using Kernel PCA

Transcription

1 A Multi-View Nonlinear Active Shae Model Using Kernel PCA Sami Romdhani y, Shaogang Gong z and Alexandra Psarrou y y Harrow School of Comuter Science, University of Westminster, Harrow HA1 3TP, UK [rodhams sarroa]@wmin.ac.uk z Deartment of Comuter Science, Queen Mary and Westfield College, London E1 4NS, UK sgg@dcs.qmw.ac.uk Abstract Recovering the shae of any 3D object using multile 2D views requires establishing corresondence between feature oints at different views. However changes in viewoint introduce self-occlusions, resulting nonlinear variations in the shae and inconsistent 2D features between views. Here we introduce a multi-view nonlinear shae model utilising 2D view-deendent constraint without exlicit reference to 3D structures. For nonlinear model transformation, we adot Kernel PCA based on Suort Vector Machines. 1 Introduction Modelling the 3D shae of any rigid or non-rigid object in rincile requires the recovery of its 3D structure from 2D images. However, accurate 3D reconstruction has roved notoriously difficult to achieve and has never been fully imlemented in comuter vision. It can be shown that under certain restrictive assumtions, it is ossible to faithfully reresent the 3D structure of objects such as human faces and bodies without resorting to exlicit 3D models. Such a reresentation would consist of multile 2D views together with dense corresondence mas between these views, although ractically only sarse corresondence can be established quickly for a carefully chosen set of feature oints [5]. It should also be able to coe with object shae variations due to changes in viewoint and self-occlusion, and in the case of an articulated object, changes in configuration [10, 11]. Cootes, Lanitis and Taylor et al. [2, 4, 8] have shown that the 2D shae aearanceof objects can be modelled using Active Shae Models (ASM). An ASM consists of a Point Distribution Model (PDM) aiming to learn the variations of valid shaes, and a set of flexible models caturing the grey-levels around a set of landmark feature oints. While this aroach can be used to model and recover some changes in the shae of an object, it can only coe with largely linear variations. For instance, a single ASM is able to coe with shae variations from a narrow range of face oses (turning and nodding of 20 ). Nonlinear variations caused by changes in viewoints and self-occlusions from different hand gestures had to be catured through the use of five different models [8]. Active shae models are based on a number of imlicit but crucial assumtions: (i) the shae of the object of interest can be defined by a relatively small set of exlicit view models, (ii) the grey levels around a articular landmark oint are consistent for all the views of the 483 BMVC 1999 doi: /c.13.48

2 Figure 1: The tyical 2D shae of a face across views from -90 to +90 can be given by a set of facial landmark feature oints and their corresonding local grey-level structures. object and can be used to find corresondences between these views and, (iii) the shaes at different views vary linearly. These assumtions are valid when the variations allowed are well constrained. However, it is difficult for this aroach to coe with largely nonlinear shae variations and the inconsistency introduced in the landmarks as a result. This is illustrated in Figure 1. A set of tyical landmarks used for a 2D shae model of a face are suerimosed on face images varying from the left to the right rofile view. The local grey-levels around the landmarks vary widely. This is highlighted for landmark oint A which clearly cannot be established across views solely based on local grey-levels. Due to self-occlusion, 2D local image structures corresond to different arts of the 3D structure of an object. In this work, we describe a multi-view nonlinear active shae model that utilises 2D view-deendent contextual constraint without exlicit reference to 3D structures. Such a model catures all ossible 2D shae variations in a training set and erforms a nonlinear transformation of the model during matching. The model therefore is able to coe with both nonlinear shae and grey-level variations around the landmarks. In articular, using a face database across the view shere, we exlicitly reresent 2D view-based context sanned by the yaw variations of a face, referred to as the ose. For nonlinear model transformation, we adot a nonlinear Princial Comonents Analysis (PCA), known as the Kernel PCA [13] based on Suort Vector Machines [14]. The rest of this aer is arranged as follows. We first outline Kernel PCA in Section 2. In Section 3, we describe a new search algorithm that is used to simultaneously match nonlinear face shae variations and recover their oses. A set of exeriments demonstrating the effectiveness of the aroach are resented in Section 4 before conclusions are drawninsection5. 2 Kernel Princial Comonents Analysis The Active Shae Model (ASM) can only be used to model faithfully objects whose shae variations are linear [2, 8]. When the Valid Shae Region (VSR) in the shae sace is nonlinear, as in the case when large ose variations are allowed, the PDM of an ASM requires nonlinear transformations. If a linear PDM is used, the model would suffer from oor secificity and comactness (see exeriments shown in Section 4). The roblem can be addressed to some extend by aroximations using a combination of linear comonents [3, 7]. However, the use of linear comonents increases the dimensionality of the model and also allows for non valid shaes [1]. Although nonlinear shae variation can be catured by a set of structured linear models using hierarchical rincial comonents [10], this requires a very large database for learning the distribution of the linear subsaces. Kernel Princial Comonents Analysis (KPCA) is a nonlinear PCA method recently introduced by Sholkof et al. [13], based on Suort Vector Machines (SVM) [14]. The 484

3 essential idea of KPCA is both intuitive and generic. In general, PCA can only be effectively erformed on a set of observations that vary linearly. When the variations are nonlinear, they can always be maed into a higher dimensional sace which is again linear. If this higher dimensional linear sace is referred to as the feature sace (F), Kernel PCA utilises SVM to find a comutationally tractable solution through a simle kernel function which intrinsically constructs a nonlinear maing from the inut sace to F. As a result, KPCA erforms a nonlinear PCA in the inut sace. More recisely, if a PCA is aimed at decouling nonlinear correlations among a given set of shae vectors x j through diagonalising their covariance matrix, the covariance can be exressed in a linear feature sace F instead of the nonlinear inut sace, i.e. C = 1 M (x j )(x j ) T (1) j=1 where () is a nonlinear maing function which rojects the inut vectors from the inut sace to the F sace. To diagonalise the covariance matrix, the eigen-roblem = C must be solved in the F sace. As C = 1 M P M j=1 ((x j ) )(x j ) T,all nonsingular solutions with 6= 0 must lie in the san of (x1) ::: (x M ). This eigen-roblem is equivalent to ((x k ) ) =((x k ) C) (2) for all k =1 ::: M and there exists coefficients i such that = i(x i ): (3) i=1 Substituting Equation (2) with (1) and (3) gives i((x k ) (x i )) = i=1 1 M i=1 BMVC99 i( j=1 ((x k ) (x j ))((x j ) (x i ))) (4) It is imortant to note that this eigen-roblem only involves dot roducts of maed shae vectors in the feature sace F. This is the raison d être of this method. Indeed, the nature of structural risk minimisation suggests that maing () may not always be comutationally tractable although exists. However, it needs not be exlicitly comuted either. Only dot roducts of two vectors in the feature sace are needed. Even so, since the feature sace has high dimensionality, comuting such dot roducts could still be comutationally exensive if at all ossible. A suort vector machine can be used to avoid the needs for either exlicitly erforming maings () or dot roducts in the high dimensional feature sace F. Let us define an M M matrix K where kij =(x i ) (x j ), Equation (4) can then be rewritten as M = K (5) where = [1 ::: M ] T. Now, erforming PCA in the feature sace F amounts to resolving the eigen-roblem of (5). This yields eigenvectors 1 ::: M with eigenvalues 1 2 ::: M. Dimensionality can be reduced by retaining only the first 485

4 L eigenvectors. The rincial comonents b of a shae vector x are then extracted by rojecting (x) onto eigenvectors k where k =1 ::: L bk k (x) = BMVC99 i=1 k i ((x i) (x)) (6) To solve the eigen-roblem of Equation (5) and to roject from the inut sace to the KPCA sace using Equation (6), one can avoid the needs for comuting both the dot roducts in the feature sace and erforming the maings through constructing a SVM (Figure 2). This is achieved by finding a kernel function when alied to a air of shae vectors in the inut sace, it yields the dot roduct of their maing in the feature sace: K(x y) =(x) (y) (7) There exists a number of kernel functions which satisfy the above criterion [14]. This includes the Gaussian kernel we have adoted where K(x y) = ex ; kx;yk Feature Sace x 1 4 φ(x) P L φ(x) b? Inut Sace 2N dimensions Σ α k(x, x i ) i 3 KPCA Sace L dimensions 2 Figure 2: Concetually, KPCA erforms a nonlinear maing (x) to roject an inut vector to a higher dimensional feature sace F (ste 1). A linear PCA is then erformed in this feature sace giving a lower dimensional KPCA sace based reresentation (ste 2). To reconstruct an inut vector from the KPCA sace, its KPCA reresentation is rojected into the feature sace (ste 3) before an inverse (x) maing is erformed (ste 4). Comutationally, however, none of the four stes is erformed. The maing is in fact carried out directly by kernel functions k(x xi) between Pi the inut sace and its KPCA sace, shown as the dashed line in the diagram. For reconstruction, this kernel-based maing is only aroximated. Otimisation is required in the KPCA sace in order to find a best match between the model and the KPCA reresentation of the inut vector. This SVM based kernel function effectively rovides a low dimensional Kernel-PCA subsace which reresents the distribution of the maing of the training vectors in the high dimensional feature sace F. As a result, nonlinear shae transformation in the inut sace can be erformed by reconstructions from the KPCA subsace. However, this rocess can be roblematic [9, 12]. The vectors in the feature sace F which have a reimage in the inut sace are the ones which can be exressed as a linear combination of the vectors (x1) ::: (x M ). However, if the reconstruction in F is not erfect, there is no guarantee to find a re-image of the reconstruction in the inut sace (Figure 2). Indeed, if dimensionality reduction is used, then the reconstruction from the KPCA sace to F 486

5 can only be an aroximation. Therefore the reconstruction (^x) of an inut observation vector (x), whose rincial comonents is truncated to the first L comonents, must be aroximated by minimising k(^x) ; PL(x)k 2 (8) where PL is a truncation oerator. To solve this minimisation roblem, there exists otimisation techniques tailored to articular kernels [9]. 3 View-Context Based Nonlinear Active Shae Model The Active Shae Model alied to the modelling of faces exhibits only limited ose variations. One imlicit but crucial assumtion of the existing method is that corresondences between landmark oints of different views can be established solely based on the grey-level information. However, when large nonlinear shae variations are introduced due to changes in object ose, the grey-level values around the landmarks are also view deendent. In general, 2D image structures do change according to 3D context. In order to find corresondences between landmarks across large variations of shae, we make exlicit use of this contextual information in the model. In the case of face varying from rofile to rofile, this contextual information is indexed by the ose angle itself. Consequently, the shae vector for the PDM is augmented by the ose angle : (x1 y1 ::: xn yn ) where (xi yi) are the coordinates of the i th landmark. Similarly, a model for the Local Grey-Levels (LGLs) around each landmark is a concatenation of the grey-levels along the normal to the shae contour and the ose of the face. Both the PDM and the LGLs are built using Kernel PCA. Given that viewcontextual based constraints are built into the models, these models are used to match novel images of faces. It is assumed that a rough osition of a face in the image is known. However the ose is unknown and the matching of the models with the target image recovers both the shae of the face and its ose. The comutation is erformed as follows: 1. An iterative rocess starts from the frontal view of the shae located near the object on the image. Notice that it is better to start from a secific view rather than the average shae, as was adoted in [4]. This is because as we are dealing with large shae variations, the average shae is not a valid shae anymore, as illustrated in Figure To find lausible corresondences of landmarks between views, augmented local grey-level models are used. To this end, the KPCA reconstruction of the grey-level vector is minimised along the normal to the shae. To comute the KPCA reconstruction of a vector, one first rojects this vector to the KPCA sace using Equation (6), obtaining the kernel rincial comonents (b). The reconstruction is then erformed by minimising the norm given in Equation (8). During the first iteration the ose of the object is unknown. Therefore the reconstruction error must also be minimised with resect to oses. This rocess yields an estimation of both the landmark ositions and the ose for each landmark. The newly estimated ose is then the average ose of all the landmarks. This ose is to be used to constrain the shae within the Valid Shae Region (VSR) at ste The estimated shae is aligned as exlained in [4]. 4. To constraint the estimated shae within the VSR, it is rojected to the shae sace using the view-context based nonlinear PDM given by Equation (6), constrained to lie 487

6 within the VSR by limiting the values of b [4] and rojected back to the inut sace using Equation (8). This yields a new estimated shae. Its ose will be used to locate the corresondence of the landmark oints at the next iteration (ste 3). 5. Reeat ste 3 until convergence. Figure 3: Examles of training shaes (to) and the average shae at the frontal view (bottom). 4 Exeriments To illustrate our aroach, we use a face database comosed of images of 6 individuals taken at ose angles ranging from ;90 to +90 at 10 increments. The ose of the face is tracked by means of a magnetic sensor attached to the subject s head and a camera calibrated relative to the transmitter [6]. The landmark oints on the training faces were manually located. An examle of such a sequence was shown in Figure 1. On linear ASM coing with ose change: A linear PDM trained to cature face shae variation between a small range of oses (20 ) was comared to a PDM trained for a full range of oses between 90. Figure 4 shows the 2 main modes of variation for each of these linear PDMs. The Valid Shae Range (VSR) for training the 20 PDM was set to 3 i and the PDM for across 90 views was limited to 0:2 i. Figure 4: The first (to) and second (bottom) modes of shae variation for a linear PDM covering 50 views (left) and across 180 views (right). The range of variation for the 50 views PDM was set to 3 times of standard deviation whilst 0:2 times of standard deviation was limited for the model covering 180 views. The two PDMs in Figure 4 and their corresonding LGL models were used to fit ASMs to face images, as shown in Figure 5. Using the model trained for the 50 ose range, an ASM was able to fit shaes to face images quite well (left). However, when the PDM for the full ose range was used, an ASM was only able to fit shae satisfactorily near the frontal view. At most of the other oses, the ASM was unable to recover the shae within the Valid Shae Range. This is mainly because both the shae of the face and the local grey-levels at the landmarks vary significantly and nonlinearly across views. 488

7 Figure 5: Fitting shaes to images using linear ASMs trained across 20 (left) and 90 (right). On nonlinear ASM coing with ose change: Kernel PCA was used to train a nonlinear PDM and cature shae variations of a face across views (90 ). Figure 6 shows the three main modes of variation and illustrates that the nonlinear PDM succeeds in caturing valid variations of shae and extends the VSR of linear PDMs shown in Figure 4. Figure 6: First three modes of shae variation for a Kernel PCA based PDM. The range of variation is set to ;1:5 i b i 1:5 i. The nonlinear PDM and its corresonding LGL models were used to fit a nonlinear ASM on face images, as shown in Figure 7. The nonlinear ASM converges and recovers shaes within the VSR but not to the right shae. This is because sometimes the background grey-levels are very similar to the grey-levels around certain landmarks at secific oses, as can be seen from the examles in Figure 1. In such cases, using grey-levels alone will fail to find corresondences between views. To better discriminate object foreground and background, we use ose to imose a view-context based constraint. Figure 7: Examles of fitting shaes to images at different views using a nonlinear ASM. On view-context based nonlinear ASM: We used the view-context based nonlinear PDM to cature the shae variation across the full range of oses. Figure 8 shows the 3 main modes of variation similar to those of a nonlinear PDM shown in Figure 6. Figure 8: First three modes of shae variation for a view-context based nonlinear PDM. The range of variation is set to ;1:5 i b i 1:5 i. Figure 9: Fitting shaes to images at different views using a view-context based nonlinear ASM. 489

8 A view-context based nonlinear PDM and its corresonding LGL models were used to fit a view-context based nonlinear ASM to face images, as shown in Figure 9. The ASM converges to the right shae and is able to recover the ose. We used the frontal view shae to start fitting. For the first iteration, the landmarks were allowed to move along the normals to the shae contour for u to a distance of 12 ixels on each side. This was then adjusted roortionally to the fitting error after each iteration. A LGL model was built using 3 ixels on both sides of a landmark along the normal to the shae. Both the PDM and the LGLs were restrained to ten dimensional eigensaces. Figure 10 illustrates an examle of fitting a shae to a face image. From left to right, the to row deicts the shae transformation in the rocess. The bottom row shows both ose recovery (convergence towards ;80 ) and shae fitting errors in ixels. Figure 11 comares fitting errors of different ASMs. A linear ASM erforms better at mean oses than at extreme oses. A nonlinear ASM exhibits similar results excet at mean oses. For all oses, a view-context based nonlinear ASM erforms significantly better. Figure 10: An examle of fitting a shae to a face image and recovering its ose at ;80.Theto row shows estimated shae after iterations 0, 1, 4, 6, 12, 13, 15, 16, 20 and 25. The recovered ose and the fitting errors (in ixels) are shown at the bottom left and right resectively. Generalisation to novel views and novel faces: Two more exeriments were conducted to evaluate the caability of the view-context based nonlinear ASM for interolating shae of novel faces not in the training set and recovering oses at novel views. A view-context based nonlinear ASM was first trained at 20 ose intervals between 90. The model was then used to recover both the shae and ose of faces at novel views. Here the number of eigenvectors was increased to 20 and the Valid Shae Region was extended to 10 times the standard deviation. Examles of shae fitting at novel views between known oses are shown in Figure 12. A view-context based nonlinear model was also trained to recover both the shae and ose of novel faces not in the training set. A model was trained on all but one of the faces in a database and was then tested on all oses of an unknown face. The exeriment was erformed for a number of unknown faces and an examle is shown in Figure 13. A comarison of the fitting errors of both a model trained on all the oses and a model trained only on half of the oses in a database is shown on the left in Figure 14. Both 490

9 Figure 11: Comaring shae fitting errors across views. Tyical fitting errors of different ASMs in ixels are drawn against ose. Whilst the dashed-line reresents a linear ASM, the lain-line is for a nonlinear ASM and the bold-line for a view-context based nonlinear ASM. Figure 12: Examles of fitting shaes to images at novel views using a view-context based nonlinear ASM. Figure 13: An examle of fitting shaes to images of an unknown face across views using a viewcontext based nonlinear ASM. ASMs exhibit similar results which shows the generalisation ability of a view-context based nonlinear ASM to novel oses. A similar error comarison for generalisation to novel faces can be seen on the right in Figure Conclusion In this work, we resented a novel aroach to modelling nonlinear 2D shaes of nonrigid 3D objects and simultaneous recovering of object ose at multile views and across the view shere. Large ose variation in 3D objects such as human faces raise two difficult roblems. First, shae variations across views are highly nonlinear. Second, corresondences of landmark oints across views cannot be reliably established based solely on local grey-levels. The first roblem was addressed by erforming nonlinear shae transformation across views using Kernel PCA based on the concet of Suort Vector Machines. The second roblem was tackled by augmenting a nonlinear 2D active shae model with ose constraint. 491

10 Figure 14: Left: Comarable fitting errors between a view-context based nonlinear ASM trained on all oses (lain-line) and a model trained only on half of the oses (dashed-line). Right: Comarable fitting errors for a model trained on all faces (lain-line) and a model trained only on some of the faces and tested on a novel face (dashed-line). The horizontal axis shows the ose and the vertical axis shows fitting errors in ixels. References [1] R. Bowden, T. A. Mitchell, and M. Sarhadi. Reconstructing 3d ose and motion from a single camera view. In BMVC, ages , Southamton, UK, [2] T. Cootes, A. Hill, C. Taylor, and J. Haslam. The use of active shae models for locating structures in medical images. Image and Vision Comuting, 12: , [3] T. Cootes and C. Taylor. A mixture model for reresenting shae variation. In BMVC, ages , Essex, UK, [4] T. Cootes, C. Taylor, D. Cooer, and J. Graham. Active shae models - their training and alication. Comuter Vision and Image Understanding, 61(1):38 59, January [5] F. de la Torre, S. Gong, and S. McKenna. View-based adative affine tracking. In ECCV, volume 1, ages , Freiburg, Germany, [6] S. Gong, E-J. Ong, and S. McKenna. Learning to associate faces across views in vector sace of similarities to rototyes. In BMVC, ages 54 63, [7] T. Hea and D. Hogg. Imroving secificity in dms using a hierarchical aroach. In BMVC, ages 80 89, Essex, UK, [8] A. Lanitis, C. Taylor, T. Cootes, and T. Ahmed. Automatic interretation of human faces and hand gestures using flexible models. In FG, ages , Zurich, [9] S. Mika, B. Scholkof, A. Smola, G. Ratsch, K. Muller, M. Scholz, and G. Ratsch. Kernel ca and de-noising in feature saces. In NIPSS, [10] E-J. Ong and S. Gong. A dynamic human model using hybrid 2d-3d reresentations in hierarchical ca sace. In BMVC, Nottingham, UK, Setember [11] E-J. Ong and S. Gong. Tracking hybrid 2d-3d human models through multile views. In IEEE International Worksho on Modelling Peole, Corfu, Greece, Setember [12] B. Scholkof, S. Mika, A. Smola, G. Ratsch, and K. Muller. Kernel ca attern reconstruction via aroximate re-images. In ICANN. Sringer Verlag, [13] B. Scholkof, A. Smola, and K. Muller. Nonlinear comonent analysis as a kernel eigenvalue roblem. Neural Comutation, 10(5): , [14] V. Vanik. The nature of statistical learning theory. Sringer Verlag,