Evaluation of Biometric Systems. Christophe Rosenberger

Transcription

1 Evaluation of Biometric Systems Christophe Rosenberger

2 Outline GREYC research lab Evaluation: a love story Evaluation of biometric systems Quality of biometric templates Conclusions & perspectives 2

3 GREYC Lab Le pôle TES et le sans-contact Research Group in Computer science, Automatics, Image processing and Electronics of Caen Laboratory staff: 7 CNRS researchers 25 Full professors 11 Associate professors 56 Assistant professors 79 PhD students 17 permanent staff 30 Engineers and post-doc Research topics: Electronics Image processing Algorithmic Document analysis Multi-agents Robotics navigation Automatics Computer security Natural language processing Biometrics Cryptography 3

4 GREYC Lab E-payment & biometrics research unit E-transactions ( E-secure Transactions Cluster) 4

5 Research topics Biometrics: Operational authentication that respects the privacy of users Le pôle TES et le sans-contact Biometric authentication (palm veins, keystroke dynamics ) Evaluation of biometric systems (acceptability, security ) Protection of biometrics (cancelable biometrics, smartcards ) GREYC Keystroke Keystroke dynamics authentication 5

6 Research topics Biometric systems Iris Finger Knuckle Print Face Keystroke dynamics Signature dynamics Fingerprint 6 Touch screen interaction Hand shape, palm vein

7 Outline GREYC research lab Evaluation: a love story Evaluation of biometric systems Quality of biometric templates Conclusions & perspectives 7

8 Evaluation: a love story PhD thesis: «Adaptive segmentation: application to hyperspectral images» Application: Algae detection Questions: How to choose the best segmentation method for a type of region? How to validate my results? 8

9 Evaluation: a love story Supervised evaluation: relevance Use of an a priori information: Synthetic Easy to obtain (by construction), Not always representative. Expert Dedicated to an application, Costly. Image processing algorithm Reference 9

10 Evaluation: a love story Unsupervised evaluation: consistency Statistical parameters Example: PSNR, EQM Advantage Easy computation Automatic evaluation Image processing algorithm Drawback Difference with an expert jugment 10

11 Evaluation: a love story Segmentation 1 Segmentation 2 Which one is the best? Segmentation 3 Original image 11

12 Evaluation: a love story Paradox Compression: PSNR known as a poor metric but still used Segmentation : many metrics exist but not very often used Interpretation : needs to use a benchmark (not easy), very few metrics for the evaluation of a single result. Still a long way to go 12

13 Outline GREYC research lab Evaluation: a love story Evaluation of biometric systems Quality of biometric templates Conclusions & perspectives 13

14 Evaluation Context Biometrics: high level application with well defined tasks Multiple organizations: BIOAPI consortium: development a biometric API standard (1998) National committee 37 (AFNOR): normalisation (2004) Three important aspects: Performance Security Usability 14

15 Evaluation 15 M. El-Abed, R. Giot, B. Hemery, J.-J. Schwartzmann, C. Rosenberger, "Towards the Security Evaluation of Biometric Authentication Systems", International Conference on Security Science and Technology (ICSST) 2011

16 Evaluation General process In order to quantify the efficiency of a biometric system, we generally use two databases: Learning database 1-Learning database: used for the enrolment of individuals (can use different captures for the model definition); Testing database 2-Testing database: used for verification or identification with captures of known individuals (impostors and genuine users). 16

17 Evaluation Performance evaluation FAR : False Acceptance Rate FRR : False Rejection Rate EER : Equal Error Rate FAR ROC curve: FRR vs FAR EER AUC : Area under the curve FRR 17

18 Evaluation Data for the tests: Real users: time consuming, operators are needed Benchmark databases: Large datasets allowing a relative comparison Different formats, naming conventions Lots of computations (scenarios, size of the database ) Sometimes not free 18

19 Evaluation 19 J. Mahier, B. Hemery, M. El Abed, M. El-Allam, M. Bouhaddaoui, C. Rosenberger. «Computation EvaBio: A Tool for Performance Evaluation in Biometrics», International Journal of Automated Identification Technology (IJAIT), 2011.

20 Evaluation Security audit 20 M. El-Abed, P. Lacharme, and C. Rosenberger, "Security EvaBio: An Analysis Tool for the Security Evaluation of Biometric Authentication Systems", the 5th IAPR/IEEE International Conference on Biometrics (ICB), New Delhi, India, p. 1-6, 2012.

21 Evaluation Usability analysis 21 M. El Abed, R. Giot, B. Hemery, C. Rosenberger, «Evaluation of Biometric Systems : A Study of Users' Acceptance and Satisfaction» Inderscience International Journal of Biometrics (IJBM), pages 1-27, 2011.

22 Evaluation Update strategies analysis Performance analysis of an update strategy method for keystroke dynamics considering attacks 22 R. Giot, B. Dorizzi, C. Rosenberger, "Analysis of Template Update Strategies for Keystroke Dynamics", SSCI 2011 CIBIM IEEE Workshop on Computational Intelligence in Biometrics and Identity Management 2011

23 Evaluation Evaluation of cancelable biometric systems 23 R. Belguechi, E. Cherrier, M. El Abed and C. Rosenberger, "Evaluation of Cancelable Biometric Systems : Application to Finger-Knuckle-Prints", IEEE International Conference on Handbased Biometrics, 2011

24 Outline GREYC research lab Evaluation: a love story Evaluation of biometric systems Quality of biometric templates Conclusions & perspectives 24

25 Quality of biometric data Objective Quantifying the quality of a biometric raw data to optimize the performance of biometric systems 25 Patrick Grother, Elham Tabassi, "Performance of Biometric Quality Measures", IEEE Transactions on Pattern Analysis and Machine Intelligence archive, Volume 29 Issue 4, April 2007

26 Quality of biometric data A generic approach Image quality Pattern-based quality Multi-class SVM Quality Image quality: blind evaluation (without any reference) Pattern-based quality: is there any interesting information in the image for the recognition? 26 M. EL-Abed, B. Hemery, C. Charrier, and C. Rosenberger, "Evaluation de la qualité de données biométriques", RNTI journal, special issue on "Qualité des Données et des Connaissances", p. 1-22, 2011.

27 Quality of biometric data Image quality BLIINDS [2] is a NR-IQ index combining three kinds of information: Contrast distortion (v1) Structure distortion (v2) Anisotropy orientation (v3 & v4) BLIINDS is entirely based on a DCT framework It uses local DCT patch of size 17*17 to calculate the three information For each patch k Local contrast (pooled -> v1) Kurtosis of the non-dct coefficients (pooled -> v2) 2 measures based on Renyi Entropy (pooled -> v3 & v4) 27 M. Saad, A. C. Bovik and C. Charrier, A DCT statistics-based blind image quality index, IEEE Signal Processing Letters, 2010.

28 Quality of biometric data Pattern-based quality The Scale-Invariant Feature Transform (SIFT) SIFT (X, Y) Scale Descriptor (128- elements) 1) Number of keypoints detected from the image 2) DC coefficient of the m-by-n descriptors matrix (m = number of the keypoints detected and n = 128) 3) and 4) mean and standard deviation of scales 28 D. G. Lowe, Distinctive image features from scale-invariant keypoints. International journal of computer vision, 2004.

29 Quality of biometric data Benchmark databases Database Individuals/samples FACES / 20 ENSIB 100 / 40 FERET 725 / average of 11 AR 120 / 26 29

30 Quality of biometric data Synthetic alterations a) Blurring alteration b) Gaussian noise alteration c) Resize alteration An example of alterations for a reference image from FACES94. From left to right, reference image then alteration level 1, 2 and 3 30

31 EER (%) Quality of biometric data EER evolution Good Fair Poor Very poor

32 Quality of biometric data Validation A good metric should be able to predict the performance of the biometric system based on the quality category Database Good Fair Poor Very poor FACES ENSIB FERET The predicted EER /real EER values for each quality set 32 P. Grother and E. Tabassi. Performance of biometric quality measures. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), p , 2007.

33 Quality of biometric data Comparison with NFIQ Use the FVC2002 Db2 database (100 individuals, 8 samples) Kolmogorov-Smirnov test on the intraclass and interclass distributions Method Good Fair Poor Very poor Contribution NFIQ Kolmogorov-Smirnov (KS) test of the genuine and impostor scores distribution 33 M. El Abed, R. Giot, B. Hemery, C. Charrier, C. Rosenberger, "A SVM-Based Model for the evaluation of biometric sample quality" SSCI 2011 CIBIM IEEE Workshop on Computational Intelligence in Biometrics and Identity Management 2011.

34 Quality of biometric data Applications Optimization of the enrolment process Comparison of biometric sensors Use quality information in soft biometrics approaches Use quality as a weighting factor in multibiometrics 34

35 Outline GREYC research lab Evaluation: a love story Evaluation of biometric systems Quality of biometric templates Conclusions & perspectives 35

36 Conclusion Evaluation of biometric systems: a major trend Transfer to industry Make the users confident in the technology Facilitate the work for researchers Perspectives: Certification process for biometric systems? Biometrics Alliance Initiative 36

37 37