Department of Computer Science & Engineering Michigan State University December 10, 2010

Transcription

1 Automatic Face Recognition: State of the Art Anil K. Jain Department of Computer Science & Engineering Michigan State University December 10, 2010

2 Birth to Age 10 in 85 Seconds

3 Outline Why face recognition? Challenges Applications Matching algorithms State-of-the art performance Current research 1,130 papers with face recognition in the title published in 2009 alone (Google Scholar)

4 Cameras Everywhere 1M CCTV cameras in London & 4M in U.K.; average Briton is seen by 300 cameras/day; 400K cameras in Beijing provide 100% coverage of public places; 150K cameras in Seoul

5 Identification at a Distance 5

6 Cameras Everywhere Number of mobile phones worldwide hits 4.6B (ITU, Feb 2010) Most computing devices are now mobile phones!

7 Automated Face Recognition Given a query face (probe), identify it from a target population (gallery) Probe Gallery MATCH Gallery size can vary from 1 to N

8 Computing Similarity "This recognition problem is made difficult by the great variability in head rotation and tilt, lighting intensity and angle, facial expression, aging, etc. Bledsoe, Chan and Bisson (1966) Used 20 inter-point distances for matching

9 Intra-class Variability Pose, lighting, expression (PIE) Occlusion Aging

10 Inter-class Similarity news.bbc.co.uk/hi/english/in_depth/americas/2000/us_elections

11 Why Face? Face recognition: most common human experience Social interaction: expression, emotion, intent, age Easy to capture: covert acquisition (surveillance) Legacy databases: passport, visa, driver license

12 Multidisciplinary Nature Neuroscience (face encoding) Cognitive science (perceptual information) Image processing (compression, image quality) Computer graphics (synthesis, rendering) Multimedia/HCI (expression, gaze) Computer vision (features, matcher, tracking) Increased knowledge about the ways people recognize each other may help to guide efforts to develop practical automatic face-recognition systems. Sinha et al. (2007)

13 Identification of Human Faces To ask how a human goes about face identification is probably an intractable question at present. a human may name and use specific features on demand but may not use them in his normal perceptual processes. Experiments with 256 subjects showed unique ID with ~7 features (hair texture, ear length, wide set eyes, lip thickness, ) Goldstein, Harmon and Lesk, Proc. IEEE, vol. 59, May 1971

14 Representation How to learn salient features?

15 Applications Law enforcement De-duplication of ID Documents Border crossing Surveillance Entertainment Large gallery, high throughput & accuracy, low cost

16 Bertillon System (1882) H.T. F. Rhodes, Alphonse Bertillon: Father of Scientific Detection, Harrap, 1956

17 Detecting Multiple Enrollment Matching 700K faces against 51M gallery (Florida DMV) found 5K duplicates

18 Border Crossing HK-Schenzen border crossing SmartGate, Australia

19 IPTV Fam ily Determine viewer demographics and sentiment for real-time audience measurement, content recommendation, targeted ads, e-commerce Courtesy I2R, Singapore

20 Virtual Makeover

21 Face Synchro Rate Tae Hee Kim &prodGrdCd=PD004401&t_top=DP000503

22 Automatic Face Recognition Face Detection Image Normalization Feature Extraction Matching

23 Face Sensing 2D (still, video), 3D (shape, texture), multipsectral Visible to Shortwave Infrared (SWIR) Spectrum (Bourlai et al., 2010)

24 Face Sensing: Seeing in the Dark

25 Appearance-based Methods Input face EigenFaces Fisherfaces PCA LDA Reconstructed face Minimize reconstruction error Maximize between-class to within-class scatter

26 Feature-based Methods Local Binary Patterns Represent a local face region as distribution of LBP features normalized histogram Improved accuracy over appearancebased methods Histogram of LBP feature values 5 0

27 Model-based Recognition Model Fitting M M Training set with landmarks AAM model shape and texture Input image Iterative process Model parameters obtained from the fitting process can be used for face recognition

28 Attribute & Simile Classifiers Figure 1: An attribute classifier is trained to recognize the presence or absence of a describable aspect (65) of visual appearance. Responses for 13 attribute classifiers are shown for a pair of images of Halle Berry. Figure 2: A number of simile classifiers are trained to recognize the similarities of parts of faces to 60 reference people (Rj). The responses to 13 simile classifiers are shown for a pair of images of Harrison Ford. Amazon Mechanical Turk used to obtain ~1000 training examples/attribute; $5k Kumar et al., Attribute and Simile Classifiers for Face Verification, ICCV, 2009

29 Face Recognition Vendor Test (FRVT 2006) Goal: FAR=0.1%, an order of magnitude better than FRVT2002 Four input face modalities Large image database (up to 100K) High res., controlled lighting, neutral Controlled lighting, smiling (400 IPD) Uncontrolled lighting, smiling (190 IPD) 3D shape + texture Phillips, FRVT 2006 and ICE 2006, Large-Scale Results, March,

30 Performance: State-of-Art Best performance: high resolution 2D (controlled lighting) & 3D images

31 Video Surveillance Trial ~60% true ID: German Federal Police at Mainz Train Station (2007)

32 Human vs. Machine O Toole (2007) compared humans and 7 algorithms in matching face pairs; 3 algorithms surpassed avg. human performance on difficult pairs; six on easy pairs Ding (2010): TH algorithm was better than 4,500 customs inspectors on easy pairs in operational data; proposed a cascaded system involving both machine & human Easy Pair Difficult Pair O Toole et al., Face recog. alg. surpass humans matching faces over changes in illumination, TPAMI, 2007 Ding et al. Computers do better than experts matching faces in a large population, 2010

33 Face Marks Ongoing Research Periocular Age Invariance Face at a Distance Face Individuality IR Face Recognition Sketch Recognition Avatar Recognition Heterogeneous FR

34 Medium Infrared (Thermal) vs. Visible What is the baseline performance and gallery size? Accuracy Rank HFR FaceVACS 333 thermal probe vs. 10,333 visible gallery

35 Age Invariance Facial shape and texture change over time Applications Age specific access control (vending machines) Missing children, multiple enrollment Age 18 Age 31 Age 17 Age 29 Lack of longitudinal data

36 Age Invariant Face Recognition Approach #1: aging invariant subspace learning Φ Φ Φ Φ SIFT MLBP (1) M ( M ) (1) M ( M ) SIFT MLBP Feature extraction & subspace learning Build classifiers: Minimize withinsubject variation & maximize between-subject variation Approach #2: appearance aging model Training set (age-separated images) L O Learn appearance aging pattern Φ = { ϕ, ϕ, K, ϕ } ' 0 1 N + Input 3D aging model Aging simulation

37 Matching Results Gallery Images Probe Images FaceVACS and generative model fail; discriminative model succeeds Discriminative model fails; FaceVACS and generative model succeed All three methods fail; fusion of generative and discriminative models succeeds Park, Tong & Jain, "Age Invariant Face Recognition", IEEE Trans. PAMI, 2010

38 Sketch to Mug shot Matching Sketch drawn based on eye witness description Klare, Li, and Jain, "Matching Forensic Sketches to Mug shot Photos," IEEE Trans. PAMI, 2011

39 Matching Sketch to Photo (sketch, photo) pairs Overlapping patches Patch features TRAINING Group patch vectors into slices Learn discriminant projection for each slice N Probe Sketch Feature extraction and grouping into slices MATCHING Discriminant projection Matching Gallery Photos

40 Forensic Sketch Matching Successful Matches Failed Matches

41 Periocular Biometric Augment face (or iris) recognition ability Periocular region has ~80% of ID information in face Useful when face/iris is partially occluded Periocular Information around the eye Face overall appearance Robust against face occlusion Park, Jillela, Ross and Jain, " Periocular Biometrics in the Visible Spectrum", IEEE Trans. Inf. Forensics & Security 2010

42 Face Marks Large birth mark Large birth mark Gang tattoo Some marks are distinctive & permanent

43 Recognition with Face Marks Score fusion = = FaceVACS fails at rank-1 Recognition succeeds by FaceVACS + face marks Matching with face marks Park & Jain, "Face Matching and Retrieval Using Soft Biometrics," IEEE Trans. on Inf. Forensics and Security, 2010

44 Biometrics of twins Iris and fingerprint are claimed to be unique for identical twins Iris Fingerprint Template Identical Twin Unrelated Person Face Template Identical Twin Unrelated Person Template Identical Twin Unrelated Person

45 Face Recognition At A Distance PTZ camera system (up to 15 m) One static camera controls the PTZ camera Coaxial-concentric camera configuration provides distance invariant face acquisition 96% recognition accuracy (20 probe and 10K gallery subjects) Dark box Beam splitter PTZ view Global view Static view PTZ camera Static camera Coaxial concentric camera system PTZ view Choi, Park, and Jain, "PTZ Camera Assisted Face Acquisition, Tracking & Recognition," BTAS, 2010

46 Face Recognition At A Distance Failed to capture Pose var. Blur Expression Correctly matched at rank-1 Gallery (3 images/subject) Failed to match at rank-1

47 Face Recognition At A Distance Telescope + NIR illuminator (up to 300 m) Telescope provides high res. face image with large standoff NIR illuminator enables face acquisition at night; NIR light is not visible to human eye (covert applications) Telescope, Sky-Watcher (USA) Aperture diameter: 180 mm, Focal length: 2700mm NIR-illuminator, KM-1000M Lanics (Korea) Using regular video camera with NIR illuminator (30 m) Using telescope + NIR illuminator (30 m)

48 Apsaras of Angkor Wat Hindu temple built in 1,150AD; French explorers discovered the hidden ruins ~1890 Do these faces represent different ethnicities?

49 Facial Landmarks 140 landmarks Facial components allow domain experts to assign different weights Klare, Mallapragada, Jain, Davis, "Clustering Face Carving: Exploring the Devatas of Angkor Wat", ICPR, 2010

50 Procustes Alignment A Point Distribution model (PDM) fit to each facial component; face similarity is the weighted sum of component similarities

51 Generating Clusters

52 Summary Face recognition, whether by man or machine, is a topic of great interest to several disciplines Progress in automatic face recognition is driven by: searching large databases in real-time with high accuracy; humans are not the best for this task Good performance achieved in constrained environments: frontal pose, neutral expression, controlled illumination & background, small age gap Unconstrained FR will require: better sensing & modeling, additional cues, contextual information,..

53

54 End of slides