Evaluation of Biometric Systems Christophe Rosenberger
Outline GREYC research lab Evaluation: a love story Evaluation of biometric systems Quality of biometric templates Conclusions & perspectives 2
GREYC Lab Le pôle TES et le sans-contact Research Group in Computer science, Automatics, Image processing and Electronics of Caen Laboratory staff: 7 CNRS researchers 25 Full professors 11 Associate professors 56 Assistant professors 79 PhD students 17 permanent staff 30 Engineers and post-doc Research topics: Electronics Image processing Algorithmic Document analysis Multi-agents Robotics navigation Automatics Computer security Natural language processing Biometrics Cryptography 3
GREYC Lab E-payment & biometrics research unit E-transactions ( E-secure Transactions Cluster) 4
Research topics Biometrics: Operational authentication that respects the privacy of users Le pôle TES et le sans-contact Biometric authentication (palm veins, keystroke dynamics ) Evaluation of biometric systems (acceptability, security ) Protection of biometrics (cancelable biometrics, smartcards ) GREYC Keystroke Keystroke dynamics authentication 5
Research topics Biometric systems Iris Finger Knuckle Print Face Keystroke dynamics Signature dynamics Fingerprint 6 Touch screen interaction Hand shape, palm vein
Outline GREYC research lab Evaluation: a love story Evaluation of biometric systems Quality of biometric templates Conclusions & perspectives 7
Evaluation: a love story PhD thesis: 1996-1999 «Adaptive segmentation: application to hyperspectral images» Application: Algae detection Questions: How to choose the best segmentation method for a type of region? How to validate my results? 8
Evaluation: a love story Supervised evaluation: relevance Use of an a priori information: Synthetic Easy to obtain (by construction), Not always representative. Expert Dedicated to an application, Costly. Image processing algorithm Reference 9
Evaluation: a love story Unsupervised evaluation: consistency Statistical parameters Example: PSNR, EQM Advantage Easy computation Automatic evaluation Image processing algorithm Drawback Difference with an expert jugment 10
Evaluation: a love story Segmentation 1 Segmentation 2 Which one is the best? Segmentation 3 Original image 11
Evaluation: a love story Paradox Compression: PSNR known as a poor metric but still used Segmentation : many metrics exist but not very often used Interpretation : needs to use a benchmark (not easy), very few metrics for the evaluation of a single result. Still a long way to go 12
Outline GREYC research lab Evaluation: a love story Evaluation of biometric systems Quality of biometric templates Conclusions & perspectives 13
Evaluation Context Biometrics: high level application with well defined tasks Multiple organizations: BIOAPI consortium: development a biometric API standard (1998) National committee 37 (AFNOR): normalisation (2004) Three important aspects: Performance Security Usability 14
Evaluation 15 M. El-Abed, R. Giot, B. Hemery, J.-J. Schwartzmann, C. Rosenberger, "Towards the Security Evaluation of Biometric Authentication Systems", International Conference on Security Science and Technology (ICSST) 2011
Evaluation General process In order to quantify the efficiency of a biometric system, we generally use two databases: Learning database 1-Learning database: used for the enrolment of individuals (can use different captures for the model definition); Testing database 2-Testing database: used for verification or identification with captures of known individuals (impostors and genuine users). 16
Evaluation Performance evaluation FAR : False Acceptance Rate FRR : False Rejection Rate EER : Equal Error Rate FAR ROC curve: FRR vs FAR EER AUC : Area under the curve FRR 17
Evaluation Data for the tests: Real users: time consuming, operators are needed Benchmark databases: http://biosecure.it-sudparis.eu/ab/ http://www4.comp.polyu.edu.hk/~biometrics/ http://www.cbsr.ia.ac.cn:8080/iapr_home.jsp http://www.nist.gov/itl/biometrics/ Large datasets allowing a relative comparison Different formats, naming conventions Lots of computations (scenarios, size of the database ) Sometimes not free 18
Evaluation 19 J. Mahier, B. Hemery, M. El Abed, M. El-Allam, M. Bouhaddaoui, C. Rosenberger. «Computation EvaBio: A Tool for Performance Evaluation in Biometrics», International Journal of Automated Identification Technology (IJAIT), 2011.
Evaluation Security audit 20 M. El-Abed, P. Lacharme, and C. Rosenberger, "Security EvaBio: An Analysis Tool for the Security Evaluation of Biometric Authentication Systems", the 5th IAPR/IEEE International Conference on Biometrics (ICB), New Delhi, India, p. 1-6, 2012.
Evaluation Usability analysis 21 M. El Abed, R. Giot, B. Hemery, C. Rosenberger, «Evaluation of Biometric Systems : A Study of Users' Acceptance and Satisfaction» Inderscience International Journal of Biometrics (IJBM), pages 1-27, 2011.
Evaluation Update strategies analysis Performance analysis of an update strategy method for keystroke dynamics considering attacks 22 R. Giot, B. Dorizzi, C. Rosenberger, "Analysis of Template Update Strategies for Keystroke Dynamics", SSCI 2011 CIBIM - 2011 IEEE Workshop on Computational Intelligence in Biometrics and Identity Management 2011
Evaluation Evaluation of cancelable biometric systems 23 R. Belguechi, E. Cherrier, M. El Abed and C. Rosenberger, "Evaluation of Cancelable Biometric Systems : Application to Finger-Knuckle-Prints", IEEE International Conference on Handbased Biometrics, 2011
Outline GREYC research lab Evaluation: a love story Evaluation of biometric systems Quality of biometric templates Conclusions & perspectives 24
Quality of biometric data Objective Quantifying the quality of a biometric raw data to optimize the performance of biometric systems 25 Patrick Grother, Elham Tabassi, "Performance of Biometric Quality Measures", IEEE Transactions on Pattern Analysis and Machine Intelligence archive, Volume 29 Issue 4, April 2007
Quality of biometric data A generic approach Image quality Pattern-based quality Multi-class SVM Quality Image quality: blind evaluation (without any reference) Pattern-based quality: is there any interesting information in the image for the recognition? 26 M. EL-Abed, B. Hemery, C. Charrier, and C. Rosenberger, "Evaluation de la qualité de données biométriques", RNTI journal, special issue on "Qualité des Données et des Connaissances", p. 1-22, 2011.
Quality of biometric data Image quality BLIINDS [2] is a NR-IQ index combining three kinds of information: Contrast distortion (v1) Structure distortion (v2) Anisotropy orientation (v3 & v4) BLIINDS is entirely based on a DCT framework It uses local DCT patch of size 17*17 to calculate the three information For each patch k Local contrast (pooled -> v1) Kurtosis of the non-dct coefficients (pooled -> v2) 2 measures based on Renyi Entropy (pooled -> v3 & v4) 27 M. Saad, A. C. Bovik and C. Charrier, A DCT statistics-based blind image quality index, IEEE Signal Processing Letters, 2010.
Quality of biometric data Pattern-based quality The Scale-Invariant Feature Transform (SIFT) SIFT (X, Y) Scale Descriptor (128- elements) 1) Number of keypoints detected from the image 2) DC coefficient of the m-by-n descriptors matrix (m = number of the keypoints detected and n = 128) 3) and 4) mean and standard deviation of scales 28 D. G. Lowe, Distinctive image features from scale-invariant keypoints. International journal of computer vision, 2004.
Quality of biometric data Benchmark databases Database Individuals/samples FACES94 152 / 20 ENSIB 100 / 40 FERET 725 / average of 11 AR 120 / 26 29
Quality of biometric data Synthetic alterations a) Blurring alteration b) Gaussian noise alteration c) Resize alteration An example of alterations for a reference image from FACES94. From left to right, reference image then alteration level 1, 2 and 3 30
EER (%) Quality of biometric data EER evolution 1 2 5 8 3 6 9 4 7 10 31 Good Fair Poor Very poor
Quality of biometric data Validation A good metric should be able to predict the performance of the biometric system based on the quality category Database Good Fair Poor Very poor FACES94 0.4744 0.2936 ENSIB 10.6397 10.413 FERET 31.88 26 0.6843 0.5131 13.2912 13.4 32.23 30.187 1.8078 1.661 16.5495 16.53 32.52 32.12 5.7983 5.044 17.787 17.774 34.37 33.757 The predicted EER /real EER values for each quality set 32 P. Grother and E. Tabassi. Performance of biometric quality measures. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), p. 531-543, 2007.
Quality of biometric data Comparison with NFIQ Use the FVC2002 Db2 database (100 individuals, 8 samples) Kolmogorov-Smirnov test on the intraclass and interclass distributions Method Good Fair Poor Very poor Contribution 0.869 0.828 0.797 0.626 NFIQ 0.82 0.698 0.632 0.64 Kolmogorov-Smirnov (KS) test of the genuine and impostor scores distribution 33 M. El Abed, R. Giot, B. Hemery, C. Charrier, C. Rosenberger, "A SVM-Based Model for the evaluation of biometric sample quality" SSCI 2011 CIBIM - 2011 IEEE Workshop on Computational Intelligence in Biometrics and Identity Management 2011.
Quality of biometric data Applications Optimization of the enrolment process Comparison of biometric sensors Use quality information in soft biometrics approaches Use quality as a weighting factor in multibiometrics 34
Outline GREYC research lab Evaluation: a love story Evaluation of biometric systems Quality of biometric templates Conclusions & perspectives 35
Conclusion Evaluation of biometric systems: a major trend Transfer to industry Make the users confident in the technology Facilitate the work for researchers Perspectives: Certification process for biometric systems? Biometrics Alliance Initiative 36
37 http://www.epaymentbiometrics.ensicaen.fr/