Sketch-a-Net that Beats Humans

Transcription

1 Sketch-a-Net that Beats Humans Qian Yu Queen Mary University of London 1

2 Authors Qian Yu Yongxin Yang Yi-Zhe Song Tao Xiang Timothy Hospedales 2

3 Let s play a game! Round 1 Easy fish face pizza 3

4 Let s play a game! Round 2 Hard piano violin guitar comb 4

5 Free-hand Sketch Definition: drawn by non-artists on touchscreen Professional sketch Free-hand sketch 5

6 Motivation It has been used as a basic visual communication media since pre-historic times 1500 BC AD 1800 AD 1970 A wide range of sketch-related applications have been investigated 6

7 Why difficult? Sketches are highly iconic and abstract 7

8 Why difficult? Sketches exhibit hugely varied levels of details/abstraction face bicycle 128 Less abstract

9 Why difficult? Sketches lack visual cues, like color, texture etc.? 119

10 Prior Work Prior work generally follows the conventional image classification paradigm Feature Extraction Feature Representation Classification HOG SIFT shape context BoW FV star graph SVM Limitations No specific or effective features No sequential ordering information 1410

11 Our Method Sketch-a-Net (1) Specifically designed CNN for feature representation learning (2) Multi-channel CNN architecture to embed sequential ordering information (3) Multi-scale network ensemble to address the variability of levels of abstraction 1611

12 Our Method Architecture of Sketch-a-Net 1512

13 Sketch-a-Net vs. AlexNet 1) A CNN for Sketch Recognition Modification of AlexNet [1] Larger first layer filters: 15*15 Remove local response normalization layer Less filters, larger pooling size, higher dropout rate etc. 1713

14 Sketch-a-Net 2) Modelling sketch stroke order with multiple channel Whole Sketch Part 1 Part 2 Part 3 Whole Sketch Part 1 Part 2 Part

15 Sketch-a-Net 2) Modelling sketch stroke order with multiple channels 6 channels Part 1 Channel 1 Channel 2 Part 2 Channel 3 Channel 4 Part 3 Channel 5 Channel

16 Sketch-a-Net 3) A multi-scale network ensemble with Joint Bayesian fusion 216

17 Sketch-a-Net How do we get multi-scale network Original size: 256*256 Downsample size: 256*256, 224*224, 192*192, 128*128, 64*64 Upsample to original size different blur levels Joint Bayesian fusion [2] Testing: using the likelihood ratio as a distance for KNN matching 217

18 Experiments and Results Dataset: TU-Berlin sketch dataset [3] Collected on Amazon Mechanical Turk (AMT) from 1350 participants 250 categories, 80 sketches per category Data pre-processing Scale to 256*256 Augmentation: reflection, rotation (in the range [-5 5] degrees), horizontal and vertical shifts (up to 32 pixels in each direction) ( 20,000*0.67 )*( 32*32 *11*2 ) = 300 millions 2218

19 Deep Classic Experiments and Results HOG-SVM [3] Ensemble [4] MKL-SVM [5] FV-SP [6] 56% 61.5% 65.8% 68.9% AlexNet- SVM [1] AlexNet- Sketch [1] LeNet [7] Sketch-a- Net Human [3] 67.1% 68.6% 55.2% 74.9% 73.1% Table 1: Comparison with state of the art results on sketch recognition 2319

20 Experiments and Results Full Model Multiple Channel & Single Scale Single Channel & Single Scale AlexNet-Sketch 74.9% 72.6% 72.2% 68.6% Table 2: Evaluation on the contributions of individual components of Sketch-a-Net 2420

21 Success at Fine-grained Classification Success cases Human Sketch-a- Net Bird* (average) 24.8% 42.5% Pigeon Seagull 2.5% 23.9% *Bird (average) stands for the average accuracy of seagull, flying-bird, standingbird and pigeon Seagull 2621

22 Failure with Bad Drawings Failure cases ice-cream-cone (pizza) lobster (bear) moon (pizza) windmill (Fan) dog (pig) 2622

23 Insufficient Training Data Failure cases bench ladder rifle Different kinds of bridge. piano Mouse (animal) 2623

24 References [1] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, [2] D. Chen, X. Cao, L. Wang, F. Wen, and J. Sun. Bayesian face revisited: A joint formulation. In ECCV, [3] M. Eitz, J. Hays, and M. Alexa. How do humans sketch objects? In SIGGRAPH, [4] Y. Li, Y. Song, and S. Gong. Sketch recognition by ensemble matching of structured features. In BMVC, [5] Y. Li, T. M. Hospedales, Y. Song, and S. Gong. Free-hand sketch recognition by multikernel feature learning. CVIU, [6] R. G. Schneider and T. Tuytelaars. Sketch classification and classification-driven analysis using fisher vectors. In SIGGRAPH Asia, [7] Y. LeCun, L. Bottou, G. B. Orr, and K. Mu ller. Efficient backprop. Neural networks: Tricks of the trade, pages 9 48,

25 Questions 3025