Face detection, face alignment, and face image parsing

Transcription

1 Lecture overview Face detection, face alignment, and face image parsing Brandon M. Smith Guest Lecturer, CS 534 Monday, October 21, 2013 Brief introduction to local features Face detection Face alignment and landmark localization Face image parsing CS 534: Computation Photography 12/6/ CS 534: Computation Photography 12/6/ Local features: broad goal What are local features trying to capture? The local appearance in a region of the image David G. Lowe, "Distinctive image features from scale-invariant keypoints," International Journal of Computer Vision, 60, 2 (2004) CS 534: Computation Photography 12/6/ CS 534: Computation Photography 12/6/

2 o Image indexing and retrieval o Image indexing and retrieval o Aligning images, e.g., for panorama stitching Shen et al., CVPR 2012 CS 534: Computation Photography 12/6/ CS 534: Computation Photography 12/6/ o Image indexing and retrieval o Aligning images, e.g., for panorama stitching o Video stabilization o Image indexing and retrieval o Aligning images, e.g., for panorama stitching o Video stabilization o 3D reconstruction uomopisa500.jpg CS 534: Computation Photography 12/6/ CS 534: Computation Photography 12/6/

3 o Image indexing and retrieval o Aligning images, e.g., for panorama stitching o Video stabilization o 3D reconstruction o Object recognition, including face recognition Local features: types Types of features and feature descriptors o Image intensity or gradient patches o Shift Invariance Feature Transform (SIFT) very popular! o DAISY o SURF o Many more CS 534: Computation Photography 12/6/ CS 534: Computation Photography 12/6/ Face detection: goal Automatically detect the presence and location of faces in images. Face detection: motivation Automatic camera focus Shen et al., Detecting and Aligning Faces by Image Retrieval, CVPR 2013 CS 534: Computation Photography 12/6/ CS 534: Computation Photography 12/6/

4 Face detection: motivation Automatic camera focus Easier photo tagging Face detection: motivation Automatic camera focus Easier photo tagging First step in any face recognition algorithm CS 534: Computation Photography 12/6/ CS 534: Computation Photography 12/6/ Face detection: challenges Large face shape and appearance variation Scale and rotation (yaw, roll, pitch) variation Background clutter Occlusions Image noise Efficiency False positives Face detection: Viola-Jones* Paul Viola and Michael Jones, Robust Real-time Face Detection, International Journal of Computer Vision (IJCV), o Feature type? o Which features are important? o Decide: face or not a face * Next few slides are based on a presentation by Kostantina Pall & Alfredo Kalaitzis, available at CS 534: Computation Photography 12/6/ CS 534: Computation Photography 12/6/

5 Face detection: Viola-Jones Feature type? Useful domain knowledge: o The eye region is darker than the forehead or the upper cheeks o The nose bridge region is brighter than the eyes o The mouth is darker than the chin Encoding o Location and size: eyes, nose bridge, mouth, etc. o Value: darker vs. brighter Face detection: Viola-Jones Feature type? Rectangle features o Value = (pixels in black) - (pixels in white) o Three types: 2,3,4 rectangles o Very fast: integral image CS 534: Computation Photography 12/6/ CS 534: Computation Photography 12/6/ Face detection: Viola-Jones Which features are important? Tens of thousands of features to choose from AdaBoost (Singer and Schapire, 1997) o Given a set of weak classifiers: h t x { 1,1} o Iteratively combine classifiers to form a strong classifier: H x = 1 if α th t (x) > threshold t 0 otherwise Face detection: Viola-Jones Final decision: face or not a face Cascade of classifiers 1. Two-feature classifier: >99% recall, >60% precision 2. Five-feature classifier feature classifier feature classifier * From CS 534: Computation Photography 12/6/ CS 534: Computation Photography 12/6/

6 Face detection: Viola-Jones Face detection: recent approaches Xiangxin Zhu and Deva Ramanan, Face Detection, Pose Estimation, and Landmark Localization in the Wild, CVPR CS 534: Computation Photography 12/6/ CS 534: Computation Photography 12/6/ Face detection: recent approaches Shen et al., Detecting and Aligning Faces by Image Retrieval, CVPR Face detection: recent approaches Shen et al., Detecting and Aligning Faces by Image Retrieval, CVPR CS 534: Computation Photography 12/6/ CS 534: Computation Photography 12/6/

7 localization: goal localization: motivation Goal of face alignment: automatically align a face (usually non-rigidly) to a canonical reference Goal of face landmark localization: automatically locate face landmarks of interests Preprocess for: o Face recognition o Portrait editing wizards o Face image retrieval o Face tracking Expression recognition Facial pose recognition 169bedd4d /realeyes-facialrecognition.png CS 534: Computation Photography 12/6/ CS 534: Computation Photography 12/6/ localization: challenges Pose Expression Identity variation Occlusions Image noise Parametric appearance models o Cootes, Edwards, and Taylor, Active Appearance Models, ECCV 1998 CS 534: Computation Photography 12/6/ CS 534: Computation Photography 12/6/

8 Parametric appearance models o Cootes, Edwards, and Taylor, Active Appearance Models, ECCV 1998 Part-based deformable models o Saragih et al., Face Alignment through Subspace Constrained Mean- Shifts, ICCV 2009 CS 534: Computation Photography 12/6/ CS 534: Computation Photography 12/6/ Part-based deformable models o Saragih et al., Face Alignment through Subspace Constrained Mean- Shifts, ICCV 2009 Supervised descent o Xiong and De la Torre, Supervised Descent Method and its Applications to Face Alignment, CVPR 2013 CS 534: Computation Photography 12/6/ CS 534: Computation Photography 12/6/

9 Exemplar-based/non-parametric methods o Shen et al., Detecting and Aligning Faces by Image Retrieval, CVPR Face image parsing Smith, Zhang, Brandt, Lin, and Yang, Exemplar-Based Face Parsing, CVPR CS 534: Computation Photography 12/6/ CS 534: Computation Photography 12/6/ Face image parsing: goal Given an input face image, automatically segment the face into its constituent parts. Face image parsing: motivation Like face alignment, can be used as a preprocess for face recognition, automated portrait editing, etc. Encodes ambiguity Generalizes to hair, teeth, ears etc. across datasets CS 534: Computation Photography 12/6/ CS 534: Computation Photography 12/6/

10 Face image parsing: our approach Database Face image parsing: our approach Database Step 0: Rough alignment & Top exemplar selection 2K exemplar images Exemplar labels 11 landmarks ~150 SIFT features 2K exemplar images Exemplar labels 11 landmarks ~150 SIFT features Input 100 top exemplars CS 534: Computation Photography 12/6/ CS 534: Computation Photography 12/6/ Face image parsing: our approach Database Step 0: Rough alignment & Top exemplar selection Step 1: Nonrigid alignment Face image parsing: our approach Database Step 0: Rough alignment & Top exemplar selection Step 1: Nonrigid alignment Step 2: Exemplar label aggregation 2K exemplar images Exemplar labels 11 landmarks ~150 SIFT features Input 2K exemplar images Exemplar labels 11 landmarks ~150 SIFT features Input = * + * + CS 534: Computation Photography 12/6/ CS 534: Computation Photography 12/6/

11 Face image parsing: our approach Database 2K exemplar images Exemplar labels 11 landmarks ~150 SIFT features Input Output Step 0: Rough alignment & Top exemplar selection Step 1: Nonrigid alignment Step 2: Exemplar label aggregation Step 3: Pixel-wise label selection = w 1 * + w 2 * + * CS 534: Computation Photography 12/6/ * Label 1 Label 2 + w 9 * Label 9 Face image parsing: quantitative results CS 534: Computation Photography 12/6/ Face image parsing: qualitative results Face image parsing: qualitative results Input Soft segments Hard segments + Ground truth CS 534: Computation Photography 12/6/ Input Soft segments Hard segments Ground truth CS 534: Computation Photography 12/6/