Kernels and Support Vector Machines

Transcription

1 Kernels and Support Vector Machines Machine Learning CSE446 Sham Kakade University of Washington November 1, Sham Kakade 1 Announcements: Project Milestones coming up HW2 You ve implemented GD, SGD, etc HW3 posted this week. Let s get state of the art on MNIST! It ll be collaborative Today: Review: the perceptron, margins, and separability Kernels & SVMs 2016 Sham Kakade 2 1

2 Support Vector Machines (Two Ideas Mixed up) 1) An attempt to better optimize the classification loss? Questionable? Latent SVMs are interesting. 2) Kernels Warp the feature space This idea is actually more general The success of SVMS? 2016 Sham Kakade 3 Linear Separability: More formally, Using Margin Data linearly separable, if there exists a vector a margin Such that 2016 Sham Kakade 4 2

3 Perceptron Analysis: Linearly Separable Case Theorem [Block, Novikoff]: Given a sequence of labeled examples: Each feature vector has bounded norm: If dataset is linearly separable: Then the number of mistakes made by the online perceptron on any such sequence is bounded by 2016 Sham Kakade 5 Beyond Linearly Separable Case Perceptron algorithm is super cool! No assumption about data distribution! Could be generated by an oblivious adversary, no need to be iid Makes a fixed number of mistakes, and it s done for ever! Even if you see infinite data However, real world not linearly separable Can t expect never to make mistakes again 2016 Sham Kakade 6 3

4 Kernels Machine Learning CSE446 Sham Kakade University of Washington November 1, Sham Kakade 7 What if the data is not linearly separable? Use features of features of features of features. Feature space can get really large really quickly! 2016 Sham Kakade 8 4

5 Higher order polynomials number of monomial terms number of input dimensions d=4 d=3 d=2 m input features d degree of polynomial grows fast! d = 6, m = 100 about 1.6 billion terms 2016 Sham Kakade 9 Perceptron Revisited Given weight vector w (t), predict point x by: Mistake at time t: w (t+1) ç w (t) + y (t) x (t) Thus, write weight vector in terms of mistaken data points only: Let M (t) be time steps up to t when mistakes were made: Prediction rule now: When using high dimensional features: 2016 Sham Kakade 10 5

6 Dot-product of polynomials exactly d 2016 Sham Kakade 11 Finally the Kernel Trick!!! (Kernelized Perceptron Every time you make a mistake, remember (x (t),y (t) ) Kernelized Perceptron prediction for x: y (j) j2m (t) sign(w (t) (x)) = X (x (j) ) (x) = X y (j) k(x (j), x) j2m (t) 2016 Sham Kakade 12 6

7 Polynomial kernels All monomials of degree d in O(d) operations: How about all monomials of degree up to d? Solution 0: exactly d Better solution: 2016 Sham Kakade 13 Common kernels Polynomials of degree exactly d Polynomials of degree up to d Gaussian (squared exponential) kernel 2 Sigmoid 2016 Sham Kakade 14 7

8 Support Vector Machines Machine Learning CSE446 Sham Kakade University of Washington November 1, Sham Kakade 15 Linear classifiers Which line is better? 2016 Sham Kakade 16 8

9 Pick the one with the largest margin! confidence = y j (w x j + w 0 ) 2016 Sham Kakade 17 Maximize the margin max,w,w 0 y j (w x j + w 0 ), 8j 2 {1,...,N} 2016 Sham Kakade 18 9

10 But there are many planes 2016 Sham Kakade 19 Review: Normal to a plane x j = x j + w w 2016 Sham Kakade 20 10

11 A Convention: Normalized margin Canonical hyperplanes x j = x j + w w x + x Sham Kakade 21 Margin maximization using canonical hyperplanes Unnormalized problem: max,w,w 0 y j (w x j + w 0 ), 8j 2 {1,...,N} Normalized Problem: min w,w 0 w 2 2 y j (w x j + w 0 ) 1, 8j 2 {1,...,N} 2016 Sham Kakade 22 11

12 Support vector machines (SVMs) min w,w 0 w 2 2 y j (w x j + w 0 ) 1, 8j 2 {1,...,N} Solve efficiently by many methods, e.g., quadratic programming (QP) Well-studied solution algorithms Stochastic gradient descent Hyperplane defined by support vectors 2016 Sham Kakade 23 What if the data is not linearly separable? Use features of features of features of features Sham Kakade 24 12

13 What if the data is still not linearly separable? min w,w 0 w 2 2 y j (w x j + w 0 ) 1, 8j If data is not linearly separable, some points don t satisfy margin constraint: How bad is the violation? Tradeoff margin violation with w : 2016 Sham Kakade 25 SVMs for Non-Linearly Separable meet my friend the Perceptron Perceptron was minimizing the hinge loss: NX j=1 y j (w x j + w 0 ) + SVMs minimizes the regularized hinge loss!! w C NX j=1 1 y j (w x j + w 0 ) Sham Kakade 26 13

14 Stochastic Gradient Descent for SVMs Perceptron minimization: NX y j (w x j + w 0 ) + j=1 SGD for Perceptron: SVMs minimization: NX w C 1 y j (w x j + w 0 ) + j=1 SGD for SVMs: w (t+1) w (t) + h i y (t) (w (t) x (t) ) apple 0 y (t) x (t) 2016 Sham Kakade 27 SVMs vs logistic regression We often want probabilities/confidences (logistic wins here) For classification loss, they are comparable Multiclass setting: Softmax naturally generalizes logistic regression SVMs have What about good old least squares? 2016 Sham Kakade 28 14

15 Multiple Classes One can generalize the hinge loss If no error (by some margin) -> no loss If error, penalize what you said against the best SVMs vs logistic regression We often want probabilities/confidences (logistic wins here) For classification loss, they are Latent SVMs When you have many classes it s difficult to do logistic regression 2) Kernels Warp the feature space 2016 Sham Kakade Sham Kakade 30 15

16 Slack variables Hinge loss If margin 1, don t care If margin < 1, pay linear penalty 2016 Sham Kakade 31 Side note: What s the difference between SVMs and logistic regression? SVM: Logistic regression: Log loss: 2016 Sham Kakade 32 16

17 What about multiple classes? 2016 Sham Kakade 33 One against All Learn 3 classifiers: 2016 Sham Kakade 34 17

18 Learn 1 classifier: Multiclass SVM Simultaneously learn 3 sets of weights 2016 Sham Kakade 35 Learn 1 classifier: Multiclass SVM 2016 Sham Kakade 36 18