Data Insufficiency in Sketch Versus Photo Face Recognition

Transcription

1 CVPR Workshop in Biometrics 2012 Data Insufficiency in Sketch Versus Photo Face Recognition 17 June 2012 Jonghyun Choi Abhishek Sharma, David W. Jacobs, Larry S. Davis Ins=tute of Advanced Computer Studies University of Maryland, College Park

2 Sketch-Photo Face Recognition Why is it important? - Automated criminal search by forensic sketch can reduce the 8me of crime inves8ga8on Probe Gallery Matching Image Courtesy by B. Klare from Matching Forensic Sketches to Mug Shot Photos, PAMI 2011

3 Popular Benchmark in Literature CUFS dataset [1] - Public benchmark dataset - Promo8ng the ini8al research A controlled dataset Well lit photos, neutral expression, frontal poses - Many approaches evaluated so far [1] Tang and Wang, Face Photo Recogni8on using Sketch, ICIP 2002

4 First CUFS dataset released (188) Baseline Timeline of Research On the CUFS Dataset CUFS dataset expanded (608), Bayesian Approach Common Discriminant Feature Extraction E-HMM and selected ensemble E-HMM and selected ensemble CUFS dataset expanded (1,800), Coupled Information Theoretic Encoding Accuracy ICIP 2003 ICCV Journal Extension 2004 T.CSVT 2005 CVPR 2006 ECCV Non-linear approach ICASSP T.CSVT Journal Extension CVPR ECCV Coupled Spectral Regression CVPR PAMI Local Feature based Discriminant Analysis (LFDA) Baseline Lighting and Pose Invariant 65

5 Summary of Previous Results Approach #Train #Test Rate (%) Sketch Synthesis Tang and Wang Tang and Wang Is the problem solved? Nonlinear E-HMM MS MRF+LDA MS MRF+LDA MS MRF+W.PCA Nearly perfect result! Modelling Modality Gap PLS-subspace Klare et al CITP

6 Yes

7 Yes for CUFS Dataset

8 CUFS Dataset 606 photo- sketch pairs - Random par88oning for training/tes8ng Combined with CUFSF (2011) from FERET - Total 1,800 photo- sketch pairs viewed sketch dataset - Provides well- aligned photo- sketch pairs Good for analysis of difference in sketch and photo domains without any other factors interven8ons

9 CUFS Dataset (Cont d) viewed sketch dataset - Sketch is drawn by ar8sts Capturing subtle edge similarity (e.g. hair style) - Pre- processed to make them well aligned as well

10 CUFS Dataset (Cont d) viewed sketch dataset - Sketch is drawn by ar8sts Capturing subtle edge similarity (e.g. hair style) - Pre- processed to make them well aligned as well

11 Real-World Scenario for Sketch-Photo Face Recognition 1. Eye- witness describes the criminal s facial traits verbally 2. Forensic ar;sts draw the sketch according to the verbal descrip;on Forensic sketch [1] Viewed sketch is not realistic [1] B. Klare et al., Matching Forensic Sketches to Mug Shot Photos, PAMI 2011 *Images courtesy from B. Klare

12 Insufficiency of the CUFS Dataset Well- alignment of CUFS dataset - Good for ini8al research on domain difference (photo- sketch) w/o interven8on of other factors - But simplifies the problem too much Simple edge matching techniques might work Ignores true variability of sketch- face recogni8on: Mis- alignment of fiducial components Seman8c descrip8on of shapes of fiducial component No precise descrip8on for subtle difference (e.g. hair style)

13 Today, We Show To obtain good result on CUFS dataset - Discrimina8ve edge matching technique is enough Outperforms state of the art in face iden8fica8on sebng - No effort in reducing modality gap is required Thus no training set except gallery set is required But even outperforms state of the art in bigger set

14 Discriminative Edge Matching Edge features - Gabor wavelet response Blurred edge tendency: Macro edge - CCS- POP [1] Micro- edgelet Discrimina8ve weight on the feature - Build a one- vs- all PLS model [1] Choi et al., A Complementary Local Feature for Face Iden8fica8on, WACV 2012

15 Partial Least Sqaures A supervised dimension reduc8on technique by maximizing covariance of weighted independent variable (X) and weighted dependent variable (Y) feature label ŵ = max w =1 cov(xw,y )2 Using NIPALS algorithm [1] to obtain the regression solu8on from X to Y [1] H. Wold, Par8al Least Squares, 1985

16 Overall System Diagram [1] Model Building (Training) D Gallery A C E F G B Z Build One- vs- All PLS regression models Model A PLSR Model Z Posi=ve Samples Nega=ve Samples PLSR Posi=ve Samples Nega=ve Samples Tes=ng Probe Regression Regression responses Model A Model B Model C Model D Model Z [1] Schwartz et al., A Robust and Scalable Approach to Face Iden8fica8on, ECCV 2010 Iden=fica=on Result C

17 Experimental Setup CUFS+CUFSF (CUFS) dataset - 1,800 pairs of sketch- face - No extra training set: Use all pairs for test - Photo to Sketch / Sketch to Photo experiments Comparison to previous work Various image cropping - Tight/Loose crop - Horizontal/Ver8cal strip crop - Fiducial component crop

18 Experimental Results Approach #Train #Test Rate (%) Sketch Synthesis Tang and Wang [24,26] Tang and Wang [25] Nonlinear [13] E-HMM [4,35] MS MRF+LDA [30] MS MRF+LDA (from [31]) MS MRF+W.PCA [31] Modelling Modality Gap PLS-subspace [21] Klareet al. [9] CITP [32] Ours (Gabor only) ± 0.44 Ours (CCS-POP only) ± 0.90 Ours(CCS-POP+Gabor) Ours (Gabor only) 0 1, Ours (CCS-POP only) 0 1, Ours(CCS-POP+Gabor) 0 1,

19 Experimental Results Test with 100 samples Accuracy (%) Approach #Train #Test Rate (%) Sketch Synthesis Tang and Wang [24,26] Tang and Wang [25] Nonlinear [13] E-HMM [4,35] MS MRF+LDA [30] MS MRF+LDA (from [31]) MS MRF+W.PCA [31] Modelling Modality Gap PLS-subspace [21] Klareet al. [9] CITP [32] Ours (Gabor only) ± 0.44 Ours (CCS-POP only) ± 0.90 Ours(CCS-POP+Gabor) Our Method Ours (Gabor only) 0 1, Ours (CCS-POP only) 0 1, Ours(CCS-POP+Gabor) 0 1, [24] 2006 [26] 2010 [31] 2010 [31] 2011 Ours 70

20 Experimental Results Test with 300 samples Accuracy (%) Approach #Train #Test Rate (%) Sketch Synthesis Tang and Wang [24,26] Tang and Wang [25] Nonlinear [13] E-HMM [4,35] MS MRF+LDA [30] MS MRF+LDA (from [31]) MS MRF+W.PCA [31] Modelling Modality Gap PLS-subspace [21] Klareet al. [9] CITP [32] Ours (Gabor only) ± 0.44 Ours (CCS-POP only) ± 0.90 Ours(CCS-POP+Gabor) Ours (Gabor only) Method 0 1, Ours (CCS-POP only) 0 1, Ours(CCS-POP+Gabor) 0 1, [25] 2005 [13] 2007,8 [4,35] 2009 [30] 2011 [9] 2011 [32] Ours (G+C) 80

21 Experimental Results Test with 1,800 samples - Compared to the results tested with 300 samples Accuracy (%) Our Method Approach #Train #Test Rate (%) Sketch Synthesis Tang and Wang [24,26] Tang and Wang [25] Nonlinear [13] E-HMM [4,35] MS MRF+LDA [30] MS MRF+LDA (from [31]) MS MRF+W.PCA [31] Modelling Modality Gap PLS-subspace [21] Klareet al. [9] CITP [32] Ours (Gabor only) ± 0.44 Ours (CCS-POP only) ± 0.90 Ours(CCS-POP+Gabor) Ours (Gabor only) 0 1, Ours (CCS-POP only) 0 1, Ours(CCS-POP+Gabor) 0 1, [9] w/ [32] w/300 Ours (G+C) w/

22 Different Cropping Tight/Loose Crop Fiducial Component Crop Horizontal/Ver8cal Strip Crop

23 Results on Tight/Loose Cropping Accuracy (%) Tight Medium Loose 97

24 Results on Fiducial Component Cropping Accuracy Ocular Nose Mouth Hair

25 Results on Strip Cropping 100 H V Hc Vc

26 Discussion & Conclusion A simple discrimina8ve edge analysis can perform well in overly- reduced problem of sketch- photo matching Now is the 8me to move on to more challenging dataset We suggest a guideline for new dataset (Please refer to our paper)

27 Thank you! Q/A Authors are supported by MURI Grant N from Office of Naval Research