References: [FC1] [FC2] Biometrics technology: Faces Toshiaki Kondo and Hong Yan, "Automatic human face detection and recognition under nonuniform illumination ", Pattern Recognition, Volume 32, Issue 10, pp. 1707-1718, 1999. Dario Maio and Davide Maltoni, "Real-time face location on gray-scale static images", Pattern Recognition, Volume 33, Issue 9, pp. 1525-1539, 2000. [FC3] Chiunhsiun Lin and Kuo-Chin Fan, "Triangle-based approach to the detection of human face", Pattern Recognition, Volume 34, Issue 6, pp. 1271-1284,June 2001. [FC4] [FC5] [FC6] [FC7] Athanasios Nikolaidis and Ioannis Pitas, "Facial feature extraction and pose determination", Pattern Recognition, Volume 33, Issue 11, pp. 1783-1791,November 2000. Guo Can Feng and Pong C. Yuen, "Multi-cues eye detection on gray intensity image", Pattern Recognition, Volume 34, Issue 5, pp. 1033-1046,May 2001. Chengyu Wu, Ce Liu, Heung-Yeung Shum, Ying-Qing Xu, and Zhengyou Zhang, Automatic Eyeglasses Removal from Face Images,, IEEE Trans. on PAMI, Vol. 26, No. 3, March 2004, pp. 322-336. X. Lu, D. Colbry and A. K. Jain, "Three-Dimensional Model Based Face Recognition," To appear in Proc. International Conference on Pattern Recognition (ICPR), Cambridge, UK, August 2004. [FC8] W.Zhao, R.Chellappa, P.J.Phillips, A.Rosenfeld Face recognition: A literature survey December 2003, ACM Computing Surveys, vol. 35, No. 4, Dec. 2003, 399-548. [FC9] J.J. Weng, D.L. Swets, Face Recognition, Chapter 3, Biometrics: Personal Identification in Networked Society, Ed. Anil Jain, Ruud Bolle, Sharath Pankanti, Kluwer Academic Publishers, 1999. w05-faces Biometrics - Summer 2006 1
Controlled Environment How do you detect the face? w05-faces Biometrics - Summer 2006 2
Uncontrolled environment Where are the faces? What do you look for? w05-faces Biometrics - Summer 2006 3
Uncontrolled environment How do you locate the faces? w05-faces Biometrics - Summer 2006 4
Biometrics technology: Faces Major Challenges: Detect the Face remove the hair, eyeglasses from a mug-shot image; or from a crowd with different backgrounds, under different lightings, etc. Extract Face representation eyes, nose, mouth, cheek, chin, etc. How to represent these features? Recognize the Face verification or identification? Q: how does human do this so efficiently? w05-faces Biometrics - Summer 2006 5
Detecting Faces in Images Challenges in face detection includes Pose images of the face may vary due to the relative camera-face pose (frontal, 45 0, profile, tilted, etc.) or face is partially occluded Presence or absence of structural components such as beards, mustaches, glasses, etc. Facial expression smiling face, sad face, etc. will affect detection Occlusion by other objects, people in a group Orientation with respect to the camera s optical axis Imaging condition lighting, and camera characteristics References [FC1], [FC2] and [FC3] presents different approaches to detect (or locate) faces. w05-faces Biometrics - Summer 2006 6
Automatic human face detection and recognition under non-uniform illumination Ref.[FC1](1999) Detection of Face Haar Wavelet transform, i.e. detection of edges, to produce 3 subimages (Fig. 3) f LL low-pass filtered f LH horizontally low-pass and vertically high-pass horizontal edges surrounding the eyes and mouth f HL horizontally high-pass and vertically low-pass vertical edges of the nose Edge-blocks, facial-edge blocks detected Symmetry axis detected Face Recognition Eigenfaces Principal Component Analysis Result: Fig. 12 w05-faces Biometrics - Summer 2006 7
Real-time face location on gray-scale static images Ref.[FC2](2000) Operating condition: Structured background Tolerate illumination changes Scale variation Small head rotation A directional image is obtained from the original grayscale image. Two stages: Approximate Location (AL) looking for elliptical blobs in the directional image. Hough transform is used. (Fig. 4, 5) Fine Location and Face Verification (FLFV) mask F defined in terms of directional elements is used. (Fig. 7) Results: Fig. 11, 12, 13. w05-faces Biometrics - Summer 2006 8
Triangle-based approach to the detection of human face Ref.[FC3](2000) believe that the 2 eyes and the mouth form an isosceles triangle Process: Input image converted to binary image first Get the 4-connected components Label them Find the center of each block Find an isosceles triangle from any 3 different blocks expand the triangle to a rectangle which is the potential face region (Fig. 1) Can apply to the profiles where the ear, one eye and the mouth will form a right-angled triangle. Results: Fig. 13 21 under different conditions; limitations Fig. 22, 23. w05-faces Biometrics - Summer 2006 9
Face Recognition Feature Extraction In general, we can categorize into three types of feature extraction methods (section 3.1.2 of [FC8]): 1. Generic methods based on edges, lines and curves; 2. Feature-template-based methods that are used to detect facial features such as eyes; 3. Structural matching methods that take into consideration of geometrical constraints on the features. Reference [FC8] gives a good literature survey on Face Recognition. References [FC4], [FC5], [FC6] and [FC7] are some approaches to feature extraction in face recognition. w05-faces Biometrics - Summer 2006 10
Facial feature extraction and pose determination Ref.[FC4](2000) Use of complementary techniques to extract a set of features for recognition: Geometrical shape parameterization, e.g. ellipses, circles, triangles, etc. Template matching, e.g. prototype block, gradient operator, etc. Dynamic deformation of active contours, e.g. Hough transform, snake contour, etc. w05-faces Biometrics - Summer 2006 11
Facial feature extraction and pose determination Ref.[F4](2000) Features to be extracted: Eyes, nostrils and mouth extracted by minima analysis of the x- and y-gray level Checks and chin by an adaptive Hough transform to detect curves (Fig. 1) Upper eyebrows using a binary template Face contour by means of a dynamic deformation of active contour Gaze direction by the face symmetry properties Results: Fig. 3-6. w05-faces Biometrics - Summer 2006 12
Multi-cues eye detection on gray intensity image Ref.[FC5](2001) Facial feature is the eye : Process: Locate the head boundary snake technique (Fig. 2) Locate the face region Thresholding to give a binary image (Fig. 3) Eliminate the false regions, e.g. ear, head boundary or nose Erosion operator makes an object (white region) shrink (Fig. 4) dilation operator makes contour smoother, and suppresses small islands in the image. (Fig. 5) w05-faces Biometrics - Summer 2006 13
Multi-cues eye detection on gray intensity image Ref.[FC5](2001) Facial feature is the eye : (Cont d) Locate the eye windows (cue 1) detect holes in the face region (Fig. 6) Estimate the direction of the line joining the two centers of the eyes (cue 2) applying edge detector (e.g. sobel) and PCA (Fig. 7,8) Construct an eye variance filter (template) (cue 3) Combining these cues and assuming the eye windows are in the upper half of the head region, the exact eye windows can be identifies. (Fig. 16) Experiments and Results: Fig. 17, 18 w05-faces Biometrics - Summer 2006 14
Automatic eyeglasses removal from face images Ref.[FC6] (2004) Sample-based approach (Fig. 1) Learn the statistical mapping between face images with glasses and their counterparts without glasses. estimate the joint distribution of the pair of images (Fig. 2) Infer the best fit glasses-free region from the calibrated region with glasses based on the learned distribution. (Fig. 2) Paste the inferred glasses-free region onto the input image with boundary blended. w05-faces Biometrics - Summer 2006 15
Automatic eyeglasses removal from face images Ref.[FC6](2004) Training procedure: Key points are marked 7 face key points (eyes, nose, mouth) and 15 glasses frame points. (Fig. 3) Calibrated pair is trained (Fig. 4) normalize the glasses and glasses-free regions. Local features are captured by different bandpass filters (Fig. 6,7) and refined iteratively (Fig. 8) Optimization is done by the Markov-chain Monte Carlo technique to locate the eyeglasses. Results: Fig. 11, 12, 13, 14 w05-faces Biometrics - Summer 2006 16
Three-dimensional model based face recognition Ref.[FC7](2004) From 2.5-D face images to 3D models: 2.5D face images is a simplified 3D (x,y,z) surface representation where (x,y) is a point (pixel) in the 2-D plane and z (the z-direction) is the depth value of the point (x,y). 2.5D scanners are readily available (e.g. Minolta VIVID 910 scanner) 3D VRML model is used to produce the 2.5D image to a 3D model. A data base of 3D models is built during enrollment. w05-faces Biometrics - Summer 2006 17
Three-dimensional model based face recognition Ref.[FC7](2004) Verification or identification: i.e. match an input 2.5D facial image to 3D face models which are already in the database: Automatic feature detection in the 2.5D image (Fig. 3,4) Rigid transform to coarsely align the 2.5D image with the full 3D model (Fig. 5) Fine iterative registration using the iterative closest point (ICP) algorithm (Fig. 6,7) The root-mean-square distance minimized by the ICP algorithm is used as the matching score. Experiments and results: Fig. 8, 9, 10 w05-faces Biometrics - Summer 2006 18
Face Recognition: A Literature Survey Ref.[FC8](2003) Up-to-date survey of still- and video-based face recognition. In the early 70 s, typical pattern recognition and classification techniques based on measured attributes of features, e.g. distances between important points in faces were used. In the 1980 s, research in this area has been dormant In the 1990 s, activities have picked up due to advances and availability in H/W and increase importance in surveillance-related applications. w05-faces Biometrics - Summer 2006 19
Face Recognition: A Literature Survey Ref.[FC8](2003) Applications: (Table I of [8]) Entertainment Video game, virtual reality, training programs, human-robot-interaction, humancomputer-interaction Smart cards Drivers licenses, entitlement programs, immigration, national ID, passports, voter registration, welfare fraud Information security TV Parental control, personal device login, desktop logon, application security, database security, file encryption, intranet security, internet access, medical records, secure trading terminals Law enforcement and surveillance Advanced video surveillance, CCTV control, Portal control, post event analysis, shoplifting, suspect tracking and investigation w05-faces Biometrics - Summer 2006 20
Face Recognition: A Literature Survey Ref.[FC8](2003) It is futile to even attempt to develop a system using existing technology, which will mimic the remarkable face recognition ability of humans. Issues that are of potential interest to designers, developers of face recognition system: (section 2 of [FC8]) Is face recognition a dedicated process? Any difference between object recognition and face recognition? w05-faces Biometrics - Summer 2006 21
Face Recognition: A Literature Survey Ref.[FC8](2003) Issues that are of potential interest to designers, developers of face recognition system: (section 2 of [FC8]) (cont d) Is face perception the result of holistic (global) or feature analysis? Global descriptions usually serve as a front end for finer, feature based recognition. In face recall studies, odd features are remembered. Ranking of significance of facial features Hair, face outline, eyes, nose, mouth are important features for recognition. Upper part of the face Beauty, attractiveness, pleasantness, etc. w05-faces Biometrics - Summer 2006 22
Face Recognition: A Literature Survey Ref.[FC8](2003) Issues that are of potential interest to designers, developers of face recognition system: (section 2 of [FC8]) (cont d) Caricatures It is a symbol that exaggerates measurements relative to any measure which varies from one person to another. Line drawings of image data manage to capture the important characteristics of the face. The role of spatial frequency analysis Gender classification can be successfully accomplished using low frequency components. Low-frequency components global description High-frequency components finer details for recognition w05-faces Biometrics - Summer 2006 23
Face Recognition: A Literature Survey Ref.[FC8](2003) Issues that are of potential interest to designers, developers of face recognition system: (section 2 of [FC8]) (cont d) Viewpoint-invariant is the orientation of the face important? Effect of lighting changes Movement and face recognition study shows that human remembers a face better when shown a moving sequence than a still photo. Facial expressions analysis of facial expression is performed in parallel in recognition!! w05-faces Biometrics - Summer 2006 24
Face Recognition: A Literature Survey Ref.[FC8](2003) Methods in face recognition from intensity images: (section 3.2, Table III) Holistic matching the whole face region is used as raw input to the recognition system, e.g. eigenpictures. Principal component analysis is the major method used. Feature-based (Structural) matching local features such as eyes, nose, mouth, are extracted first. Their location, statistics, etc. are formed as feature vector. Classifiers are used in matching. Hybrid methods combination of the whole face region and features is used. w05-faces Biometrics - Summer 2006 25
Face Recognition: A Literature Survey Ref.[FC8](2003) Face recognition from image sequences: (section 4) Main challenges to video-based recognition include: The quality of the video is low Face images are small It is difficult to characterize faces/human parts from a sequence of images. Basic techniques required: Face segmentation Pose estimation Face/feature tracking Face modelling w05-faces Biometrics - Summer 2006 26
Face Recognition: A Literature Survey Ref.[FC8](2003) Evaluation of face recognition systems: (section 5) Various databases of faces are available for research and evaluation. Competitions among researchers and vendors are conducted regularly based on these data bases. w05-faces Biometrics - Summer 2006 27