This paper is a postprint of a paper submitted to and accepted for publication in IET Biometrics and is subject to Institution of Engineering and

Transcription

1 This paper is a postprint of a paper submitted to and accepted for publication in IET Biometrics and is subject to Institution of Engineering and Technology Copyright. The copy of record is available at IET Digital Library

2 An Overview of Research on Facial Aging using the FG-NET Aging Database Gabriel Panis (1), Andreas Lanitis (1), Nicholas Tsapatsoulis (2), Timothy F. Cootes (3) (1) Visual Media Computing Lab Department of Multimedia and Graphic Arts Cyprus University of Technology (2) Department of Communication and Internet Studies Cyprus University of Technology (3) Imaging Sciences and the Biomedical Imaging Institute, Human and Medical Sciences, University of Manchester Abstract The FG-NET aging database was released in 2004 in an attempt to support research activities aimed at understanding the changes in facial appearance caused by aging. Since then the database was used for carrying out research in various disciplines including age estimation, age-invariant face recognition and age progression. Based on the analysis of published work where the FG-NET aging database was used, conclusions related to the type of research carried out in relation to the impact of the dataset in shaping up the research topic of facial aging, are presented. The paper also includes a review of key articles from different thematic areas, where the FG-NET aging database was used and the presentation of benchmark results. The ultimate aim of this article is to present concrete facts related to research activities in facial aging during the last decade, provide an indication of the main methodologies adopted, present a comprehensive list of benchmark results and most importantly provide roadmaps for future trends, requirements and research directions in facial aging. Keywords: FG-NET Aging Database, age estimation, age progression, face recognition 1. Introduction Since the 1990 s the topic of face image interpretation figures as one of the research areas that attracts significant interest among the computer vision community. Researchers in this area aim to develop techniques that perform reliably despite significant within and betweensubject variation encountered in face images. Among other types of variation, a number of researchers concentrated on the problem of facial aging in relation with key face image processing applications such as age invariant face recognition, age estimation and age progression. However, unlike other types of facial variation, the collection of face images displaying aging variation is a unique problem as aging occurs slowly over the years inhibiting in that way the process of data acquisition needed for experimentation in facial aging. For this reason the existence and distribution of suitable databases is a key issue for supporting research related to facial aging.

3 Due to the non-existence of publicly available databases, until 2004 only a small number of researchers considered the problem of facial aging, mainly based on small in-house face datasets containing age separated face images [43, 44, 45, 63, 76]. Back in 2004 two face aging datasets were made publicly available: The MORPH [74] and the FG-NET Aging Dataset (FG-NET-AD). When MORPH was first released it contained a large number of images but only about three instances of the same person. On the other hand the FG-NET-AD contained a small number of images and subjects, but included about 12 age separated images per subject. Despite the fact that neither of the two datasets was ideal, both datasets played an instrumental role in initiating systematic research activities in the area of facial aging during the last decade. Research in facial aging mainly focuses on the following topics: I. Facial Age Estimation: The process of estimating the age or the age group of a person based on facial information II. Age Invariant Face Recognition: The ability to recognize persons despite significant age related modifications III. Age Progression: The process of predicting the future facial appearance of a person Detailed coverage of the topics of facial aging can be found in related survey papers [22, 71] and books [19]. Also in [61] a technical report on the performance of age estimation algorithms is presented with emphasis on the results obtained rather than the adopted methodologies. Guo [32, 33] presents an overview of the state-of-the-art in human age estimation is provided that includes an overview of face representations, learning approaches and overview of existing databases. In [32] in addition to an overview of age estimation, state-of-the-art approaches to sex classification in conjunction with the mutual influence between age and gender is discussed. Guo [32] also points out future challenges of age estimation such as age estimation of moving faces, the lack of ideal age databases and the need for dealing with unconstrained faces. Unlike the aforementioned articles, in this paper we concentrate our attention on the use of the FG-NET-AD. In particular an analysis related to thematic areas of published papers over the years, an overview of the most representative papers reported in the literature and a comprehensive collection of benchmark results reported in the literature are presented. Based on the analysis of research activities in facial aging, research trends are outlined and future research directions are formulated. The remainder of the paper is structured as follows: In Section 2 information about the FG- NET-AD usage is presented. An overview of key papers that utilized the FG-NET-AD and benchmark results in facial aging are presented in Sections 3 and 4. In Section 5 a discussion related to future research directions and needs for additional aging datasets are described, followed by concluding comments. 2. Database Description 2.1. Database generation and contents The FG-NET-AD was generated as part of the Project FG-NET (Face and Gesture Recognition Network) [20]. FG-NET was funded by the European Union within the 5 th Framework Programme, Information Society Technologies (IST) in the category of initiative Support Measures Networks of Excellence and Working Groups. One of the major aims of the project was to encourage research technology development in the area of face and gesture recognition by specifying and supplying suitable image sets. Within this context, among other datasets, the FG-NET-AD was generated. The FG-NET-AD contains 1002 images from 82 different subjects with ages ranging between newborns to 69 years old subjects. However, ages up to 40 years are the most populated in the

4 database. The dataset is provided free of charge and can be used for academic research-related activities 1. With the exception of images showing individuals at more recent ages, for which digital images were available, images in the FG-NET-AD were collected by scanning photographs of subjects found in personal collections. As a consequence image quality of images in FG-NET-AD depends on the photographic skills of the photographer, the imaging equipment used, the photographic paper used and the overall photograph condition. FG-NET- AD images display considerable variability in resolution, image sharpness, illumination in combination with face viewpoint and expression variation. Occlusions in the form of spectacles, facial hair and hats are also present in a number of images Analysis of FG-NET-AD Usage So far the FG-NET-AD was distributed to more than 4000 researchers, supporting in that way widespread aging-related research activities. In this section we present key facts related to the FG-NET-AD usage, as derived from a Google Scholar-based search. As part of this process the number of papers where images from the FG-NET-AD were used, published between 2005 until October 2014 were located. The distribution of the 365 indexed publications over the past decade (see Figure 1) shows an increasing trend of articles reporting work using the FG-NET-AD, reflecting the growing interest of the research community in facial aging (The decrease in 2014 is mainly attributed to publications not indexed yet by Google Scholar). However, during the last three years the number of papers where FG-NET-AD was used, do not show significant increase indicating the need of researchers in facial aging to use more extended/dedicated databases. The fact that the 365 indexed publications originated from 167 different institutions from 37 countries from all six continents, highlights the wide spread interest in facial aging research. Figure 1: Number of papers where images from the FG-NET-AD were used, per year The majority of publications are conference and journal papers (see Figure 2). However, a significant number of PhD, Master s and Diploma Thesis as well as book chapters were published. The Other category groups together online publications and technical reports. Figure 2: Type of papers where images from the FG-NET-AD were used, per year 1 Researchers who wish to obtain the FG-NET-AD should send an to fgnet.aging@gmail.com.

5 Published papers describing research work using the FG-NET-AD were classified into the main research thematic areas of age estimation, face recognition, age progression/modeling, feature extraction/location, gender classification, biometrics, face modeling, face detection and pose estimation. Publications that were found to cover work extending across more than one thematic area were placed in all appropriate thematic areas. A significant number of other diverse thematic areas such as landmark detection [107], face interpretation [80], sketch matching [8], face synthesis [37], wrinkle detection [5], psychology and perception related publications [42, 75] have been grouped under the Other category. The distribution of papers into the main thematic areas over the years is shown in Figure 3. It is worth pointing out that although the primary scope of the dataset was to support research in age progression the highest share of papers is related with facial age estimation. Another important observation is the recorded increase in interest for age invariant face recognition reflecting the trend of researchers to deal with more challenging aspects of face recognition. Figure 3: Number of papers in different thematic areas per year The survey also recorded the total annual number of citations that each of the indexed publications received, as an indication of the visibility of these publications among the research community (see Figure 4). Although the thematic area of facial aging is a small part of the more general area of face image processing, papers dealing with facial aging received a considerable amount of citations reflecting the overall interest in facial aging research. Figure 4: Number of citations per year

6 3. Research Using the FG-NET-AD In this section we briefly review representative papers for each main thematic area addressed by researchers who used the FG-NET-AD Age Estimation According to the analysis presented in Figure 3, the topic of age estimation dominates research efforts in facial aging. The main reasons for this trend are attributed to the following factors: i) Machine-based age estimation methods could figure in a wide range of applications involving man-machine interfaces such as age-adaptive interfaces and the enforcement of age-based access restrictions both to physical and electronic sites. ii) Since humans are not perfect in the task of age estimation, automated age estimates could complement/aid the task of human operators. iii) Accurate age estimates are usually required for other facial aging related applications such as age invariant face recognition and age progression hence the starting point in dealing with facial aging is usually the task of age estimation. iv) Unlike age progression, there are concrete ways to test the performance of different algorithms allowing in that way the efficient comparative evaluation of different age estimation algorithms The output of an age estimation algorithm can be an estimate of the exact age of a person or the age group of a person. For exact age estimation the performance of an age estimator is usually based on the mean average error (MAE) between actual and estimated ages over a test set and plots of Cumulative Score (CS) that show the number of test cases which have an absolute error smaller than a given threshold. In the case of age-group estimation, errors usually refer to the percentage of correct classifications. Researchers who carried out research in facial age estimation investigated the use of both standard pattern recognition/regression approaches and techniques adapted to the age estimation problem. Most researchers conclude that the aging variability encountered in face images requires the use of dedicated techniques. The main trends of age estimation research activities are related with the determination of suitable feature vectors that better reflect aging information in conjunction with efforts of customizing classification algorithms to take into account certain characteristics of age classification such as the problem of data sparseness i.e. the fact that for a given individual it is unlikely to have available training samples covering all ages in the range of interest [23]. Geng et al. [24] generate aging patterns for each person in a dataset consisting of face images showing each subject at different ages. In this case the problem of data sparseness is addressed by filling in missing samples using the Expectation Maximization algorithm. The pattern position of a new face image that minimizes a reconstruction error indicates the age of the subject. Experimental results prove that this method outperformed previous approaches reported in the literature and also performed better than widely used classification methods. Guo et al. [28] propose a discriminative subspace learning based on manifold criterion for low-dimensional representation of the aging manifold. Regression is applied on the aging manifold patterns in order to learn the relationship between coded face representations and age. A key aspect of Guo s work is the use of a global SVR for obtaining a rough age estimate, which is then refined by a local SVR trained using only ages within a small interval around the initial age estimate. Luu et al. [57] also project faces in an AAM [15] subspace and

7 then adopt a two-stage hierarchical age estimation approach. The first stage involves the initial classification of faces into young and old followed by the use of a local SVM regressor in order to get the final age estimate. A number of researchers experimented with the use of Biologically Inspired (bio-inspired) features. Guo et al. [31] propose a model that contains alternating layers called Simple (S) and Complex (C) cell units that resemble object recognition models of the human visual system. Features for the S layer are extracted based on Gabor filters with different scales, standard deviations and orientations. The C layer involves the pooling of S-layer features at different bands, scales and orientations. The dimensionality of the feature vector is reduced using Principal Component Analysis and a support vector regressor is used for obtaining age estimates. The overall framework of using bio-inspired features has been studied extensively both in the area of age estimation and age invariant face recognition [94]. In a more recent approach, El Dib et al. [18] extract bio-inspired facial features at a fine level and information from the forehead is also utilized. Hong et al. [36] introduce the so-called biologically inspired active appearance model where instead of using pixel intensities, shape-free faces are represented by bio-inspired features during the process of AAM training. A regression-based age estimator is then used for estimating the age of samples based on coded face representations. The tendency of dealing with facial features at different levels was also adopted by Suo et al. [85] who propose an age estimation algorithm based on a hierarchical face model. The model represents human faces at three levels that include the global appearance, facial components and skin zones. An age estimator is trained from the feature vectors and their corresponding age labels. In [35] a hierarchical approach is presented where bio-inspired features are extracted from individual facial components. Facial components are then classified into one of four age groups and then within an age group an SVM regressor is trained to predict the age. It was found that the optimum performance was attained from a fusion of the best performing features, i.e. holistic bio-inspired features, shape and eye region bio-inspired features. As part of the efforts of using features related to the aging process Zhou et al. [109] describe an age classification method based on the Radon transform. Difference of Gaussians filtering is applied on the face image to extract perceptual features, which are processed using the Radon transform. An entropy-based SVM classification algorithm is then used to select features. Choi et al. [13] propose the extraction of wrinkles using a set of region specific Gabor filters in conjunction with the use of a local binary pattern (LBP) skin extractor. Li et al. [52] also attempt to provide a generalized framework for selecting Gabor features that preserve both global and local aging information and at the same time minimize the redundancy between features. A number of researchers concentrated their attention on dealing with age labels used for age estimation. Chang et al. [9] propose an ordinary hyperplane ranking algorithm that uses relative ranking information and a cost-sensitive property to optimize the age estimation process. Within this context the age estimation problem is decomposed into a number of binary decisions that classify a given face into a class of faces with age greater or less a given age. The combination of the results of all individual classifiers yields the final age estimation result. Chao et al. [10] propose a label-sensitive concept in an attempt to take advantage of correlations that exist between different classes in age estimation. As part of this effort the learning process of samples belonging to a certain age, takes also into account weighted samples belonging to neighboring ages. The problem of class similarity between adjacent ages is also addressed in [25] where the concept of using label distributions is introduced. Within this framework during the training process samples contribute both to the training process of the class they belong to and also to the training process for adjacent classes.

8 The use of Neural Network-based techniques for age estimation was also investigated. Zheng et al. [108] use a back propagation neural network, where the inputs are geometrical features and LBP vectors, in order to classify faces into juveniles and adults. Yin and Geng [104] use a Conditional Probability Neural Network where the inputs are a facial descriptor and an age estimate and the output is the probability that the face descriptor is extracted from a face at the input age. The majority of age estimation methods reported in the literature are based on texture-based features. In contrast Thukral et al. [88] use face shape information in a hierarchical approach where the test image is first classified into an age group using several classifiers fused using the majority rule. Then the Relevance Vector Machine regression model of that age group is used to estimate the age. Wu et al. [95] rely on facial shapes represented by point-coordinates on a Grassmann manifold. An aging signature is extracted for each sample, by considering the tangent vectors of the deformation needed to deform a given face shape to the average face shape. A regressor-based age estimator that relates aging signatures to age is used during the process of age estimation Age Invariant Face recognition Since the 1990 s face recognition poses as one of the most popular research themes in computer vision. Recently research in face recognition is mainly focused on scenarios involving unconstrained images that may be captured without the consent/cooperation of the subject. As part of the efforts to deal with challenging face recognition aspects, a number of researchers considered the problem of age invariant face recognition. According to the results of several studies [3, 64, 69, 81, 89], as the time interval between capturing a reference image and a test image increases, the performance of standard face recognition algorithms deteriorates hence there is a need of developing dedicated age-invariant face recognition techniques. In general researchers address the problem of age-invariant face recognition using two main approaches: The first approach involves the use of age-invariant facial features/regions whereas the second involves the use of age progression in an attempt to eliminate age variations between training and test images. As part of the efforts to define age invariant face representations Li et al. [51] extract from face images Scale-Invariant Feature Transform (SIFT) features and Multi-scale Local Binary Pattern (MLBP). The high dimensionality of the resulting feature vector is reduced by applying an extension of the Linear Discriminant Analysis (LDA) called the Multi-Feature Discriminative Analysis (MFDA). This results in multiple classifiers which are then combined to generate a decision through a fusion rule. Bereta et al. [7] test the performance of face recognition using feature sets based on local descriptors including variations of Local Binary Patterns. As part of the experiments local descriptors are applied to a given image, Gabor magnitude and Gabor phase images. According to the authors, among all descriptors the best performance is achieved when a multi-scale block LBP descriptor is applied on Gabor magnitudes. In [49] the general problem of assessing the intensity of aging effects on different facial areas is considered. As part of this effort different types of features are extracted from different facial areas and distributions of feature vectors from different age groups are compared, allowing in that way the definition of age sensitive and age invariant features. The results indicate that the upper part of a face has increased aging invariance when compared with the lower part. Juefei-Xu et al. [38] extract age-invariant facial features from the Periocular region. After a preprocessing stage that aims to remove lighting and pose variation, they obtain a feature vector based on the Walsh-Hadamard transform encoded Local Binary Patterns (WLBP) approach. Instead of using information from a single face region, a number of researchers focused their attention in fusing information from different facial regions.

9 Yadav et al. [99] extract Local Binary descriptors (LPB) for different facial regions. In particular LBP descriptors are extracted from the face, right and left eye regions, binocular region and the mouth region. Features derived from different regions are fused together based on a weighted scheme, where the weights are estimated through an evolutionary method that aims to maximize the correct classification score among training images. According to experimental results the proposed method outperforms classification methods based on features from single regions and fusion methods where weights were determined with other methods. Singh et al. [83] transform face images into polar coordinates so that training and test face images are normalized to a common domain, minimizing in that way aging-related variations. Recognition is performed using Gabor based features. Lu et al. [56] concentrated their attention on the case where only one training sample per subject is available. Within this context they first partition training images into a large number of overlapping blocks so that a single image is represented by multiple patches. Then they apply the so-called discriminative multi-manifold analysis (DMMA) that aims to identify the most discriminant features per subject. While the majority of methods involve the use of texture-based features for implementing age invariant face recognition methods, Ali et al. [4] use geometrical features derived from a number of triangles located within the facial area. Such features correspond to key metrics used in craniofacial anthropology for identifying characters and growth patterns. According to the results a 99% correct recognition rate is achieved on the FG-NET-AD. As an alternative to the definition and use of age-invariant features Sethuram et al. [81] address the problem of age invariant face recognition through the synthesis of age progressed face images. The age progression method uses support-vector regression (SVR) model that relates AAM [15] parametric descriptions of faces to age. A simulation method is used for generating a large number of face instances and classifying them to different age groups based on the SVR model so that each age group is populated with multiple instances. By considering differences between faces in different age groups, age effects can be simulated on new faces. The application of this method to raw face images prior to the face recognition task minimizes age differences between gallery and probe images resulting in improved recognition performance. Face verification experiments across aging also received the attention of the research community. Ling et al. [54] deal with the problem of face verification by using discriminant analysis aiming at enhancing id-information. An important aspect of the work presented is the facial feature vector which is comprised of gradient orientation parameters extracted at multiple resolution levels. In [58] hierarchical local binary pattern (HLBP) feature descriptors in combination with an Ada Boost classifier are used for addressing the problem of face verification of age separated faces. In [96] a craniofacial growth model is augmented with a set of linear equations of growth parameters. During the face verification processes the shape of a test face is warped to the face of a reference face based on the craniofacial growth model and the shape similarity between the warped and the reference face was used for determining whether the two images show the same person Age progression Traditionally age progression is performed by forensic artists [66] who make use of face aging theory as described in anthropometric studies [72, 73] in combination with images of relatives of a subject in order to create, either by hand or by using computer tools, age progressed drawings. In the case of automated age progression, the majority of age progression methods reported in the literature are data-driven.

10 Scandrett et al. [78] describe an age progression approach where a personal and consensus aging axis is defined in the PCA space, enabling the process of age progression to be carried out by considering the compounded effect of both person specific and global aging trends. The influence of each axis during the age progression process is determined by maximizing the probability that an age progressed face belongs both to the distribution of faces at the target age and the distribution of differences between age-progressed samples and the actual faces of the same subject at the target age. Rowland and Perret [76] propose an age progression method based on age prototypes generated by averaging faces belonging to the same age group. Age progression is achieved by adding on a given face image, differences between age prototypes corresponding to different ages. In a more recent approach Kemelmacher-Shlizerman et al [40] describe an age progression method based on age prototypes derived from images collected from the internet. A key element of the proposed method is the alignment and illumination normalization used that ensures the generation of accurate age prototypes despite illumination, pose and expression variation in the training images. The results of an extended human-based performance evaluation carried out mainly using images from the FG-NET-AD indicate that the proposed method is particularly suited to applying long term age progression on the faces of young children. In [46] three data-driven age progression algorithms are evaluated. The first method is based on age prototypes [76], the second method is based on aging functions [44] and the third method is based on the minimization of the weighted distance of a face projected in a low dimensional space to the centroids of the target age and subject distributions. The comparison is based on dedicated metrics that assess the success of an age progression algorithm to add appropriate age effects without affecting core id-related characteristics. The AGES age estimation method proposed by Geng et al. [24] can also be used for age progression where an Expectation Maximization algorithm is used for filling in missing faces in an aging pattern, accomplishing in that way the process of age progression. As an extension to the AGES algorithm, Tsai et al. [90] use guidance vectors during the process of completing missing faces in patterns. Within this context guidance vectors are formed by estimating the difference between a given face and the mean face in the corresponding age group and using the difference in subsequent calculations in order to preserve personal details. In [86] a multilayer And-Or graph used for face representation, is extended to include aging effects and hair features. This model includes age-specific aspects that relate to hair styles, shape variation, age related deformation of facial components and appearance of wrinkles. Face aging is modeled on the graph as dynamic Markov process based on graph-based face representations. A Markov-model for modeling aging was also used in [87] where AAM models of local facial regions are generated. During the model building process the main aging related modes of variation are extracted through correlation analysis. By considering training samples with limited age separation the short term aging process is approximated using polynomials and the long term aging process is modeled by concatenating short-term aging models based on a Markov model. In some cases age progression is regarded as a missing data recovery problem. For example Wang et al. [93] treat the problem of age progression in the framework of a super resolution methodology. During the age progression process the input face is down sampled to low resolution and a super-resolution algorithm trained using images at the target age is used for creating a high resolution image that incorporates aging effects on top of the basic id structure exhibited in the down-sampled image. In [60] the problem of age progression is treated as an occlusion removal problem where the appearance of a given face in conjunction with the recursive PCA method is used for predicting the appearance of a given face at the target age.

11 A number of researchers considered the problem of 3D age progression by utilizing either datasets with 3D scanned faces [79] or 3D reconstructed faces [65, 82]. Since age progression is in effect a face synthesis application, it could be accomplished more effectively in 3D as in that case facial pose and lighting variation can easily be eliminated. Park et al. [65] derive 3D models from FG-NET-AD images allowing in that way the generation of a 3D aging model. Modeling of the aging process is based on separate modeling of shape and texture aging. Based on the assumption that similar facial regions of young children will remain similar at older ages, Shen et al. [82] decompose a given 3D face into local regions and locate the most similar regions from the training set that contains 3D reconstructions of FG-NET-AD images. During the age progression process, they replace local facial regions of a given face with age progressed versions of the most similar regions. The methods described above are predominantly based on training data. In contrast the approach proposed by Ramanathan et al. [70] is primarily based on a physiological craniofacial growth model. The model is derived from the revised cardioidal strain transformation model proposed in psychophysical studies in the literature. The proposed model takes into consideration anthropometric evidence that characterize the growth of human faces using growth parameters across facial landmarks. However this model is focused on craniofacial growth and does not deal with age-related textural variations. A variation of the approach described by Ramanathan et al. [70] is reported in [95] in the context of age invariant face verification. 4. Comparative Evaluation Using the FG-NET-AD In this section we present a summary of performance evaluation results reported in the literature, in relation to experiments using the FG-NET-AD Age Estimation Most researchers reporting results using the FG-NET-AD (see Table 1) adopted the Leave One Person Out (LOPO) approach where for each of the 82 subjects in the database, an age estimator is trained using images of the remaining 81 subjects and the results are averaged over 82 trials. Given the small number of images available in the FG-NET-AD this is an optimum approach. The current benchmark for age estimation is the work of El Dib et al. [18] where a 3.17 years MAE is reported when the LOPO approach is used. In the case of human age estimation the recorded benchmark is 4.7 years when all images from the FG-NET-AD were processed through crowdsourcing by 10 volunteers [35]. Geng et al. [25] also report a human age estimation MAE of 6.23 years derived based on observations from 29 volunteers using a sub-sample of 51 FG-NET-AD images. Clearly a number of reported algorithms match and even overpass the indicative performance of humans as recorded in [25] and [35]. It is worth quoting that within three years of the publication of the first standardized age estimation results based on the LOPO method [24] reported MAE were almost halved, indicating in this way the benefits of comparative evaluation.

12 Table 1: Age Estimation Results using FG-NET-AD images Reference Method Training/Testing method Result (MAE) Ni2009 [62] Multi-Instance Regression 600 training 402 test images 9.49 Zhou2005 [110] Regression using Boosting 800 training 202 testing images 5.81 Xiao2009 [98] Regression using a learned 300 training 702 test images 5.04 distance metric Luu2009 [57] 2 stage SVR in AAM 802 training 200 test images 4.37 subspace Backpropagation Neural Network Zheng2013 [108] 602 train and 400 test images 83% correct classification into juvenile/adult Geng2007 [24] Aging Pattern Subspace LOPO 6.22 Thukral2012 [88] Shape-based age LOPO 6.2 estimation Günay2013 [27] Local radon Features LOPO 6.18 Yan2007 [101] Regressor from uncertain LOPO 5.78 labels Geng2013 [25] Label Distribution (IIS- LOPO 5.77 LLD) Yan2007 [100] Ranking with uncertain LOPO 5.33 labels Yan2009 [103] Synchronized Submanifold LOPO 5.21 Embedding Ylioinas2013 [105] LBP LOPO 5.09 Kernel Density Estimate Guo2008 [28] Manifold Learning and Locally Adjusted Robust Regressor LOPO 5.07 Kilinc2013 [41] Geometric and Gabor LBP LOPO 5.05 Guo2008 [29] Probabilistic Fusion LOPO 4.97 Approach Liang2014 [53] Hierarchical Framework LOPO 4.97 Yan2008 [102] Regression from patch LOPO 4.95 kernel Wu2012 [96] Grassmann manifold LOPO 5.89 Zhang2013 [106] Hierarchical Model LOPO 4.89 Li2012 [52] Ordinal Discriminative LOPO 4.82 Features Guo2009 [31] Biologically inspired LOPO 4.77 features Yin2012 [104] Conditional Probability LOPO 4.76 Neural Network Geng2013 [25] Label Distribution (CPNN) LOPO 4.76 Chen2013 [12] Cumulative Attribute SVR LOPO 4.67 Han2013 [35] Component and holistic LOPO 4.6 BIF Chen2013 [11] Pairwise Age Ranking LOPO 4.56 Chang2011 [9] Ordinal hyperplanes ranker LOPO 4.48 Chao2013 [10] Label-sensitive learning LOPO 4.38 Hong2013 [36] Biologically Inspired LOPO 4.18 AAM El Dib2010 [18] Enhanced Biologically - LOPO 3.17 Inspired features Han2013 [35] Human Observers Entire FG-NET-AD 4.7 Geng2013 [25] Human Observers 51 FG-NET-AD images 6.23

13 4.2. Face Recognition In the case of face recognition experiments it is common to encounter results based on training and testing using different parts of the FG-NET-AD (see Table 2). As a result the comparison of different approaches is not straightforward. Among all results reported in the literature the highest recognition rate was reported by Juefei et al. [38] when tested on the entire FG-NET-AD using the LOPO approach. Table 2: A summary of Face Recognition Results using the FG-NET-AD Reference Method Training/Testing method Result (Rank-1 Recognition Rate) Geng2007 [24] Aging Pattern Subspace 10 pairs of randomly selected 38.05% Park2010 [65] PCA coefficients of 3D shape and texture images Probe: 82 images in age range 0-30 Gallery: 82 images in age range % Li2011 [51] Discriminative analysis LOPO 47.50% Juefei-Xu2011 [38] WLBP fetures from LOPO 100% periocular region Yadav2013 [99] Bacteria For Aging Fusion Gallery: Images from 24 Probe: Images from 58 subjects 64.5% Sethuram2009 [81] Age progression Images from Morph and FG % NET-AD Singh2007 [83] Polar Coordinate FG-NET-AD 30%-50% Transformation and a private database. Bereta2013 [7] Multi-scale block LBP Leave-one-out 45% descriptor Ali2013 [4] Geometrical triangular Entire FG-NET-AD 99% features Lu2013 [56] Discriminative Multimanifold Analysis 4 subsets containing age separated pairs of each of the 82 subjects 15%-33% 4.3. Age Progression The lack of standardized age progression performance evaluation metrics in combination with the fact that in most cases researchers used different training/test sets, prevents the presentation of a comprehensive set of comparable age progression results. For this reason in Table 3 we only present typical evaluation methods used by different researchers rather than comparative results.

14 Table 3: A summary of Age Progression Evaluation Methods Reference Age Progression Method Evaluation Method Scandrett2006 [78] Aging Trajectories Shape difference, texture difference Ramanathan2006 [70] Craniofacial growth model Face recognition Geng2007 [24] Aging patterns subspace Mahalanobis distance in AAM space, face recognition results Suo2010 [86] Hierarchical And-Or graphs Human based, age estimation Lanitis2008 [46] Comparison of three methods (prototypes, aging functions and SVM-distance-based) Distance from id and age target distributions Sethuram2009 [81] AAM-based analysis by Face recognition synthesis Park2010 [65] 3D face aging model Face recognition Tsai2013 [90] Guided prediction Shape difference, user evaluation Suo2012 [87] Concatenational Graph Evolution Aging Model Wang2012 [93] Kemelmacher- Shlizerman2014 [40] Tensor Space Analysis and AAM Illumination invariant age prototypes Age similarity (user evaluation and age estimation algorithm), ID similarity (user evaluation and face recognition algorithm) Age similarity (user evaluation and age estimation algorithm), ID similarity (user evaluation and face recognition algorithm) Human based (Using Mechanical Turk) 5. Discussion The availability of two publicly available aging databases (MORPH and FG-NET-AD) played an important role in initiating an increased interest in research related to facial aging among the computer vision community during the last decade. As a result of the attention attracted, the topic of facial aging is now an established topic in biometrics. The key question that arises is concerned with the future research directions and trends in facial aging especially in relation with the current state of the art and future challenges. In the remainder of this section we briefly assess the state of the art in each of the aforementioned areas and also provide a roadmap of future research activities Facial Age Estimation According to the analysis of published work, the topic of facial age estimation has been the most popular among researchers active in the area of facial aging. The increased interest in age estimation resulted in the development of robust age algorithms capable of providing age estimates that can be used in most applications requiring user age information. In general two main trends form the current directions in age estimation: The first is the use of bio-inspired features [18, 31, 36] and the second is the exploitation of age label distributions and ranking [9, 10, 25]. In addition efforts in investigating feature extraction from facial areas that contain increased age related information [35] also show promise. It is anticipated that future research directions in age estimation will focus on: i) Unconstrained images: Developing age estimation algorithms that deal with images captured under completely unconstrained conditions such as the ones captured by surveillance cameras. Recent work by Guo and Zhang [34] on developing generalized age estimation algorithms that can be used for populations of different ethnic origins could also form the basis for developing age estimation algorithms that adapt to different imaging conditions.

15 ii) Age estimation based on video sequences. Currently almost all research efforts in age estimation deal with static images. However, temporal information that includes both face movements and expressions could also provide important age-related clues. A similar scenario was encountered in expression recognition that gradually moved away from dealing with static images as it became obvious that facial movements are important for interpreting expressions [14]. An early example of using face dynamics for improving facial age estimation is presented by Dibeklioğlu et al [16] where based on the location of 11 facial landmarks, metrics related to movement dynamics encountered in the eyes, cheek and mouth are defined. According to the results of an experimental evaluation using the UvA-NEMO Smile Database [17], age estimation results are improved when appearance-based metrics are combined with facial dynamic metrics. iii) Multi-modal age estimation: Apart from the face, aging also affects other body parts hence multi-modal approaches could lead to more accurate age estimation. Although some attempts in developing age estimation based on individual biometric modalities, such as gait [55], head movements [48] and fingerprints [26] were reported in the literature, it is anticipated that the topic of multi-modal biometric age estimation will attract substantial research interest in the near future. Along these lines the use of 3D images for age estimation [97] needs to be investigated. iv) Age estimation results have reached error levels that make them fit for most applications. However, there is still room for further improvements in age estimation tasks involving ages traditionally used as age thresholds (i.e ages of 12, 15, 18, 21). In order to support the scenarios stated above, there is a clear need for developing new aging datasets that contain unconstrained images, video sequences, extensive sets of age separated 3D scans and multi-modal biometric samples Age Invariant Face Recognition Research efforts in developing dedicated facial age invariant feature vectors resulted in facial representations that can provide excellent face recognition results [38] despite age differences between training and test images. It is expected that the recorded trend of attempting to focus on specific facial regions that do not exhibit increased age-invariance will be strengthened, as experimental results indicate that face region selection is important for age-invariant face recognition. The list of experimental results shown in Table 3, indicate that in general researchers who experiment with age-invariant face recognition do not use standardized training and test image subsets, making the process of comparative evaluation difficult. It is highly recommended that in the future the LOPO approach is adopted when results using the FG- NET-AD are reported. A severe limitation of face recognition experiments using the FG- NET-AD is the small number of classes (82 classes). Among the future challenges of age invariant face recognition is to develop systems that can deal with a considerably higher number of classes in order to test the viability of such systems to more realistic scenarios such as the ones encountered in passport control systems. Although the availability of the MORPH [74] and the Face Recognition Grand Challenge (FRGC) database [68] partly deal with the problem of limited number of classes for age invariant face recognition investigations, there is still room for the development of dedicated databases. Recently a growing trend in age-invariant face recognition is to use data extracted from the periocular region [39, 59, 77]. Juefei-Xu et al [38] indicate that the use of local information, and in particular the periocular region, is well suited to the problem of age-invariant face identification/verification. Mahalingam and Ricanek [59] indicate that although LPB-based

16 periocular face recognition achieves comparable performance to approaches utilizing the overall face, in the case of age-separated faces the performance obtained is inferior. However, both Juefei-Xu et al [38] and Mahalingam et al [59] highlight the importance of periocular face recognition along with the need of carrying out further research on defining the optimum coding schemes that result in truly age-invariant descriptions. Because periocular face recognition could fail in the cases that eye regions are heavily occluded, among the next challenges in age invariant face recognition is to deal, like in the case of age estimation, with completely unconstrained images that may include partially occluded faces, so that the performance of proposed systems is validated in real situations involving age separated images in combination with occlusions and other sources of extensive within-class face variability Age Progression Existing age progression algorithms are capable of generating face images with aging effects superimposed. Actually there are various web applications [1, 2] that are designed for adding aging related transformations. However, the key issue in automated age progression is not the generation of aesthetically pleasing results but the generation of accurate predictions. Age progression algorithms are not yet to a stage to produce highly accurate predictions for different subjects mainly due to the diversity of aging effects, the dependence of the aging process on external factors/events that may occur in a subject s life and also due to the compounded effect of aging variation with other within-subject variations (i.e. expression, pose and illumination). A key issue related to development of age progression methodologies, is the issue of performance evaluation [47]. Unlike other face image processing tasks (i.e. face recognition and age estimation) it is not straightforward to define standardized age progression performance evaluation metrics. Clearly both the issue of developing accurate age progression methodologies and the issue of performance evaluation are still open issues in the area of facial aging. Apart from developing accurate age progression evaluation metrics, an important step is the development of dedicated face datasets that support the idea of independent comparative evaluation. For this purpose it is necessary to develop and make publicly available the following datasets: i) Database with pairs of age separated images: Ideally such a database should contain both easy and difficult scenarios where in the case of the easy scenario age separated faces must be captured under identical imaging conditions so that the aging variation is isolated from other types of variations, allowing researchers to focus their attention on the problem of age progression. Early examples of developing dedicated datasets and performance evaluation metrics for age progression comparative evaluations are described in [50, 67]. ii) Aging Database Coupled with Personal Lifestyle Profiles: In order to study in a systematic way the influence of different lifestyle and health factors on the aging process, it is crucial to augment aging databases with lifestyle and health profiles of the subjects so that it will be possible to take personal profiles into account when dealing with age progression. Since age progression could be addressed more effectively using 3D models it is also important to generate 3D aging datasets. Given recent technology advances that enable 3D data acquisition using cheap equipment such as Kinects [91] or even smart phones, it is anticipated that in few years it will be feasible to have publicly available 3D face aging datasets that contain multiple samples of different subjects, enabling in that way systematic experimentation in 3D age progression.

17 5.4. Comparative Timeline of Research in Facial Aging Table 4 shows the key stages in research in facial aging, in comparison with related stages of the well-established task of face recognition. Based on the data in Table 4, it is obvious that further advances in the field heavily depend on the availability of new aging datasets. Although a number of aging databases exist (see Appendix A for more details), as explained in the previous sections there is need for new dedicated datasets. Table 4: Timeline of key stages in research in facial aging Face Age Estimation Age-Invariant Age Progression Recognition Face Recognition Initial 1980 s 1990 s 2000 s 1990 s Experimentation Publicly Mid 1990 s Available Datasets Standard 1990 s 2000 s 1990 s Not established Performance Evaluation Metrics Comparative N/A N/A Evaluations Commercial Yes Yes No No Systems Next Steps Dealing with totally unconstrained images Dealing with totally unconstrained images/temporal Dealing with totally unconstrained images. Systematic performance evaluation information. 3D age progression 6. Conclusions Dealing with large number of classes. Multimodal age estimation. Dealing with large number of classes. The FG-NET-AD was released in 2004 in an attempt to encourage and promote research in the new (at that time) research topic of facial aging. It was fortunate that the release of the FG-NET-AD coincided with the release of the MORPH aging database that provided different type of data and as a result supported complementary experimental investigations. Based on the analysis of published work and review of several key papers in the main areas related to facial aging, it is evident that the goals that lead to the generation and distribution of the FG-NET-AD are fulfilled. A large number of researchers have benefited from using the database and as a result the topic of facial aging is now an established research area in computer vision. Although the FG-NET-AD can still be used for supporting research related to facial aging, it is imperative that new aging databases are made available in order to support new type of experimentations that will further advance research in facial aging.