Some thoughts on safety of machine learning

Transcription

1 Pattern Recognition and Applications Lab Some thoughts on safety of machine learning Fabio Roli HUML 2016, Venice, December 16th, 2016 Department of Electrical and Electronic Engineering University of Cagliari, Italy

2 The black cloud

3 The black cloud The novel tells the arrival of an enormous cloud of gas that appears to destroy the life on Earth by blocking the Sun's radiation. The motion of the cloud doesn t follow physical laws, so one scientist argues that it might be a life-form with a degree of intelligence. Supposing that the cloud might be intelligent, the scientists try to communicate with it But human paradigms for information communication do not work! Could I put in this way? Said Kingsley. Between two absolutely identical individuals, no communication at all would be necessary because each individual would automatically know the experience of the other...between two widely different individuals a vastly more complicated communication system is required - F. Hoyle, The black cloud,

4 First issue 4

5 Is antropomorphism good? Is antropomorphism good for the human use of machine learning? o Should we fabricate robots with human appearance? o Should we be disappointed if algorithms make errors that humans wouldn t have never done? But if cattle and horses and lions had hands or could paint with their hands and create works such as men do, horses like horses and cattle like cattle also would depict the gods' shapes and make their bodies of such a sort as the form they themselves have Xenophánes; BC 5

6 Vision of a humanoid robot The icub is the humanoid robot developed at the Italian Institute of Technology as part of the EU project RobotCub and subsequently adopted by more than 20 laboratories worldwide. It has 53 motors that move the head, arms and hands, waist, and legs. It can see and hear, it has the sense of proprioception (body configuration) and movement (using accelerometers and gyroscopes). [ The object recognition system of icub uses visual features extracted with CNN models trained on the ImageNet dataset [G. Pasquale et al. MLIS 2015] 6

7 icub object recognition: example images [ 7

8 The icub object recognition pipeline Credits: Lorenzo Natale, Visual Learning of Objects and Tools on the icub Robot,

9 Generation of adversarial noise against icub Disclaimer: unpublished work, work in progress at PRA Lab f(x) l The adversarial image x + r is visually hard to distinguish from x Plate Adversarial Noise Cup 9 9

10 icub is not a unique case 10

11 Black swans in ImageNet [Szegedy et al., Intriguing properties of neural networks, 2014] 11

12 Adversarial faces [M. Sharif et al., ACM CCS 2016] M. Sharif et al. developed a systematic method to automatically generate attacks to face recognition systems, attacks which are realized through printing a pair of eyeglass frames. When worn by the attacker whose image is supplied to a state-of-theart face-recognition algorithm, the eyeglasses allow her to evade being recognized or to impersonate another individual. 12

13 Adversarial images in mission-critical apps [Patrick McDaniel et al., IEEE Security & Privacy, 2016] 13

14 Adversarial images in mission-critical apps [Patrick McDaniel et al., IEEE Security & Privacy, 2016] To humans, adversarial images are indistinguishable from original images. Left an ordinary image of a stop sign Right an image manipulated with adversarial noise and classified as a yield sign 14

15 Big disappointment? 15

16 Low probability blind spots [M. Gori and F. Scarselli, PAMI 1998; B. Biggio et al., MCS 2015] 5 2 class classification Blue: legitimate class Red: illegal class 0 Spam/Ham for example Adversarial inputs in low probability regions can evade easily the classifier 16

17 Low probability blind spots [M. Gori and F. Scarselli, PAMI 1998; B. Biggio et al., MCS 2015] 5 2 class classification 5 1.5C classification (MCS) Better enclosing legitimate data in feature space may improve classifier security 17

18 Second issue 18

19 Can machine learning be safe? M. Barreno et al., Can machine learning be secure? ASIACCS '06 Proceedings of the 2006 ACM Symposium on Information, computer and communications security 19

20 Android malware detection! About one billion of users of Android mobile operating system! Thousands of new Android malware samples every day 20

21 DREBIN: Android malware detector D. Arp et al. DREBIN: Effective and Explainable Detection of Android Malware in Your Pocket NDSS The Network and Distributed System Security Symposium w 1 w 2 w 3... w n s s th th s < th Malware Benign 21

22 Attacking a linear classifier Attacker s goal: evasion SEND_SMS CAMERA SEND_SMS weights READ_SMS Android.hardware.wifi weights READ_SMS Android.hardware.wifi The Attacker can evade easily the classifier by manipulating a few features if weights are sparse! 22

23 Securing linear classifiers [A. Demontis et al., S+SSPR 2016] We learn feature weights evenly-distributed by solving this optimization problem: There is a trade off between sparsity and security! 23

24 Third issue 24

25 Interpretability comes at a price? Supposed that sparsity can facilitate interpretability, there is a trade off between interpretability and security in machine learning? A. Demontis et al. On security and sparsity of linear classifiers for adversarial settings S+SSPR

26 Last issue People are disappointed, and worried, for such errors Can we trust in algorithms as well as we trust in humans? 26

27 Trust in humans or machines? Algorithms are biased, but also humans are as well When should you trust in humans and when in algorithms? 27

28 Biases of humans and machines A bat and a ball together cost $ 1.10 The bat costs $ 1.0 more than the ball How much does the ball cost? 28

29 Back to the future? Maybe we should reconsider again the distinction between two operation modes, System 1 and System 2? The discussion of the early days of machine learning about symbolic and sub-symbolic processing? NIPS Cognitive Computation: Integrating Neural and Symbolic Approaches 29

30 Thanks for listening! Any questions? Engineering isn't about perfect solutions; it's about doing the best you can with limited resources (Randy Pausch, ) 30