Learning Permutations with Exponential Weights

Transcription

1 Learning Permutations with Exponential Weights David P. Helmbold Manfred K. Warmuth University of California - Santa Cruz Last update - June 8, 007 D. Helmbold & M.Warmuth (UCSC) Learning Permutations with Exponential Weights / 5

2 Commuter Airline Example Motivating Problem: Match planes to routes D. Helmbold & M.Warmuth (UCSC) Learning Permutations with Exponential Weights / 5

3 Commuter Airline Example n airplanes fly n daily routes Must assign different aircraft to each route Aircraft have various sizes, don t (yet) know # of passengers Too small/big aircraft can be bad At end of day, get the loss (regret) for each aircraft assignment Goal: have loss over many days close to best fixed assignment in hindsight On-line assignment problem want to pick good assignments (or permutations) D. Helmbold & M.Warmuth (UCSC) Learning Permutations with Exponential Weights / 5

4 Outline The problem The Algorithm The Analysis 4 Lower Bound 5 Open problems D. Helmbold & M.Warmuth (UCSC) Learning Permutations with Exponential Weights 4 / 5

5 Outline The problem The problem The Algorithm The Analysis 4 Lower Bound 5 Open problems D. Helmbold & M.Warmuth (UCSC) Learning Permutations with Exponential Weights 5 / 5

6 Learning Model The problem On line learning of permututions on n elements on each trial: Algorithm chooses distribution D over permutations (Π s) Nature picks any loss matrix L in [0, ] n n ; L i,j is loss of assigning element i to position j. n Loss of perm. Π is L i,π(i) = Π L (matrix dot product) i= Algorithm incurs expected loss E Π D [Π L] Goal: expected loss not much more than best permutation in hindsight D. Helmbold & M.Warmuth (UCSC) Learning Permutations with Exponential Weights 6 / 5

7 The problem Special Case: number of incorrect assignments Nature picks permutation; loss is number of items assigned to wrong position. If nature picks permutation abcd: Loss matrix Lis a b c d For Π = bacd = the loss is. Nature can permute the rows to pick different permutations. a 0 b Loss matrix for pick bcad is: 0 c 0 d D. Helmbold & M.Warmuth (UCSC) Learning Permutations with Exponential Weights 7 / 5

8 Some other special cases The problem Ranking: put best two items in top three positions (best two change each trial) some row L = permutation Note: any permutation of has loss, or Permutation reflects list order, minimize number of links to find the desired element: some row L = permutation better algorithm for this case in of Blum-Chawla-Kalai D. Helmbold & M.Warmuth (UCSC) Learning Permutations with Exponential Weights 8 / 5

9 Note that: The problem Loss of permutation decomposed into sum of per-assignment losses Entries of L in [0, ], per trial losses in [0, n]. Loss matrix provides important structure allowing reasonably efficient algorithm with L bestp + Lestn ln n + n ln n loss bounds Standard Hedge (or WMR, Experts) too expensive - n! permutations (and gets worse bound) Could use FPL* (Kalai-Vempala 05), faster but worse bounds (n ln n terms become n ln n) D. Helmbold & M.Warmuth (UCSC) Learning Permutations with Exponential Weights 9 / 5

10 We will: The problem Use exponential weights with difficult normalization Weight of permutation proportional to product of assignment weights keep n rather than n! weights. Bound with relative entropy analysis and Bregman projection techniques Get bounds for un-normalized version of WMR/Hedge D. Helmbold & M.Warmuth (UCSC) Learning Permutations with Exponential Weights 0 / 5

11 Outline The Algorithm The problem The Algorithm The Analysis 4 Lower Bound 5 Open problems D. Helmbold & M.Warmuth (UCSC) Learning Permutations with Exponential Weights / 5

12 The Algorithm Distributions on Permutations and Weight Matrix For any D over permutations, define n n weight matrix W where W i,j = Pr(item i gets mapped to position j) W is doubly stochastic (each row and col sums to ) Expected loss on trial is L W Many D s lead to same W; and can expand any n n doub.stoch. W into a D using few ( n ) permutations. Example: = = Π perms Π D. Helmbold & M.Warmuth (UCSC) Learning Permutations with Exponential Weights / 5

13 The Algorithm A rose by any other name... Permutation Learning: PermLearn D. Helmbold & M.Warmuth (UCSC) Learning Permutations with Exponential Weights / 5

14 The Algorithm A rose by any other name... Permutation Learning: PermLearn Permutation Learning with Exponential Weights: PermLearnEW D. Helmbold & M.Warmuth (UCSC) Learning Permutations with Exponential Weights / 5

15 The Algorithm A rose by any other name... Permutation Learning: PermLearn Permutation Learning with Exponential Weights: PermLearnEW Permutation Learning with Exponential-Weights: PermLearnE D. Helmbold & M.Warmuth (UCSC) Learning Permutations with Exponential Weights / 5

16 The Algorithm A rose by any other name... Permutation Learning: PermLearn Permutation Learning with Exponential Weights: PermLearnEW Permutation Learning with Exponential-Weights: PermLearnE Permutation Learning with Exponential-Weights: PermELearn D. Helmbold & M.Warmuth (UCSC) Learning Permutations with Exponential Weights / 5

17 Algorithm PermELearn The Algorithm Keeps n n doubly stochastic weight matrix W: initially each entry /n Need to predict and update each trial. Predict step: Decompose W into D on few permutations Randomly pick Π according to D Update step: Loss update: W i,j := W i,j e ηl i,j (η is learning rate) Normalize W back into doubly stochastic W. D. Helmbold & M.Warmuth (UCSC) Learning Permutations with Exponential Weights 4 / 5

18 The Algorithm Making a matrix doubly stochastic Use Sinkhorn Balancing (Sinkhorn 64), also called matrix scaling finds row and column factors (r i s and c j s) such that W i,j := W i,j r i c j doubly stochastic. Many iterative algorithms, pretty combinatorics, connection to permanent (Lineal-Samorodnitsky-Wigderson 00) but: no closed form for r i, c j! is D. Helmbold & M.Warmuth (UCSC) Learning Permutations with Exponential Weights 5 / 5

19 Matrix Scaling Example The Algorithm ( ) D. Helmbold & M.Warmuth (UCSC) Learning Permutations with Exponential Weights 6 / 5

20 Matrix Scaling Example The Algorithm ( ) ( ) D. Helmbold & M.Warmuth (UCSC) Learning Permutations with Exponential Weights 6 / 5

21 Matrix Scaling Example The Algorithm ( ) ( ) ( ) D. Helmbold & M.Warmuth (UCSC) Learning Permutations with Exponential Weights 6 / 5

22 Matrix Scaling Example The Algorithm ( ) ( ) ( ) D. Helmbold & M.Warmuth (UCSC) Learning Permutations with Exponential Weights 6 / 5

23 Matrix Scaling Example The Algorithm ( ) ( ) ( ) Why? heft of permutation is product of its assignment weights re-scaling rows and/or cols preserves heft ratios ( ) ( ) a b scales to where a + b = and a = b. b a D. Helmbold & M.Warmuth (UCSC) Learning Permutations with Exponential Weights 6 / 5

24 Outline The Analysis The problem The Algorithm The Analysis 4 Lower Bound 5 Open problems D. Helmbold & M.Warmuth (UCSC) Learning Permutations with Exponential Weights 7 / 5

25 The Analysis Method (Kivinen-Warmuth 97): for every trial and every doubly stochastic comparator U show the key invariant Alg s loss U s loss (U, W) (U, W ) ( e η )(W L) η(u L) where relative entropy (U, W) = i,j U i,j ln U i,j W i,j + W i,j U i,j, D. Helmbold & M.Warmuth (UCSC) Learning Permutations with Exponential Weights 8 / 5

26 The Analysis Method (Kivinen-Warmuth 97): for every trial and every doubly stochastic comparator U show the key invariant Alg s loss U s loss (U, W) (U, W ) ( e η )(W L) η(u L) where relative entropy (U, W) = i,j Summing over trials gives U i,j ln U i,j W i,j + W i,j U i,j, (U, W init ) (U, W fin ) ( e η )(Alg s tot loss) η t (U L t ) Alg s tot loss (U, W init) + η t (U L t) e η = n ln n + ηl bestp e η D. Helmbold & M.Warmuth (UCSC) Learning Permutations with Exponential Weights 8 / 5

27 The Analysis Method (Kivinen-Warmuth 97): for every trial and every doubly stochastic comparator U show the key invariant Alg s loss U s loss (U, W) (U, W ) ( e η )(W L) η(u L) where relative entropy (U, W) = i,j Summing over trials gives U i,j ln U i,j W i,j + W i,j U i,j, (U, W init ) (U, W fin ) ( e η )(Alg s tot loss) η t (U L t ) Alg s tot loss (U, W init) + η t (U L t) e η = n ln n + ηl bestp e η Tune η (Freund-Schapire 97): loss L bestp + Lestn ln n + n ln n D. Helmbold & M.Warmuth (UCSC) Learning Permutations with Exponential Weights 8 / 5

28 The Analysis Problem: don t know r i s, c j s can t work with (U, W ) directly Solution: Prove key invariant for U, un-normalized W Normalization projects onto convex sets containing U, so (U, W ) (U, W ) (Herbster-Warmuth 0) Implications: W already close enough to U, normalization to W only helps. Can interleave element loss updates and row/column normalization Can bound Un-normalized versions of Weighted Majority etc. (what does this mean?) D. Helmbold & M.Warmuth (UCSC) Learning Permutations with Exponential Weights 9 / 5

29 Outline Lower Bound The problem The Algorithm The Analysis 4 Lower Bound 5 Open problems D. Helmbold & M.Warmuth (UCSC) Learning Permutations with Exponential Weights 0 / 5

30 Lower Bound Lower Bound Consider just one row of L (where item gets mapped to) Nature can assign arbitrary [0..] loss to positions positions are experts in allocation (hedge) setting (FS 97) allocation algorithms generalize experts setting for large enough n, L beste any algorithm can have loss L beste + ( ɛ) L beste ln n (CFHHSW 97) D. Helmbold & M.Warmuth (UCSC) Learning Permutations with Exponential Weights / 5

31 Lower Bound Lower Bound Repeat on different rows each row uses up position. Use n positions on n rows. Bound becomes: ( n L beste + ( ɛ) L beste ln n ) n = L bestp + ( ɛ) L bestp ln n where L bestp = n L beste upper bound: L bestp + Lestn ln n + n ln n D. Helmbold & M.Warmuth (UCSC) Learning Permutations with Exponential Weights / 5

32 Outline Open problems The problem The Algorithm The Analysis 4 Lower Bound 5 Open problems D. Helmbold & M.Warmuth (UCSC) Learning Permutations with Exponential Weights / 5

33 Open problems Open problems Can interleaving update and normalization operations improve efficiency? Can bounds for FPL* be improved for permutations? (Trials can t be decomposed as in expert setting) What kind of other matrices can be learned? (Orthonormal?) D. Helmbold & M.Warmuth (UCSC) Learning Permutations with Exponential Weights 4 / 5

34 Open problems Open problems Can interleaving update and normalization operations improve efficiency? Can bounds for FPL* be improved for permutations? (Trials can t be decomposed as in expert setting) What kind of other matrices can be learned? (Orthonormal?) Thank You! D. Helmbold & M.Warmuth (UCSC) Learning Permutations with Exponential Weights 4 / 5

35 Notes: Open problems D. Helmbold & M.Warmuth (UCSC) Learning Permutations with Exponential Weights 5 / 5