Recommender systems and the Netflix prize. Charles Elkan. January 14, 2011

Transcription

1 Recommender systems and the Netflix prize Charles Elkan January 14, 2011 Solving the World's Problems Creatively

2 Recommender systems We Know What You Ought To Be Watching This Summer

3

4 We re quite curious, really. To the tune of one million dollars. Netflix Prize rules 1. Goal: Improve the Netflix recommendation algorithm, Cinematch 2. Criterion: Reduction in error (RMSE) 3. Oct 06: Contest start 4. Oct 07: $50K progress prize for 8.43% improvement 5. Oct 08: $50K progress prize for 9.44% improvement 6. Sept 09: $1 million grand prize for 10.06% improvement

5 Movie rating data Training data 100 million ratings 480,000 users 17,770 movies 6 years of data: Test data Last few ratings from each user (2.8 million) Dates of ratings are given Training data score movie user Test data movie user? 62 1? 96 1? 7 2? 3 2? 47 3? 15 3? 41 4? 28 4? 93 5? 74 5? 69 6? 83 6

6 What is RMSE? RMSE stands for root mean squared error. Let p be the prediction for user u and movie m; let r be the true rating. RMSE = (p - r)² / n RMSE measures the average mistake, with higher penalty for big mistakes, i.e. large values of (p-r)². You can t have a contest without a precise goal!

7 #ratings per user 1. Avg #ratings/user: 208

8 Most Active Users User ID # Ratings Mean Rating , , , , , , , , The dataset contains 17,770 movies!

9 #ratings per movie 1. Avg #ratings/movie: 5627

10 Movies Rated Most Often Title # Ratings Mean Rating Miss Congeniality 227, Independence Day 216, The Patriot 200, The Day After Tomorrow 194, Pretty Woman 190, Pirates of the Caribbean 188, The Green Mile 180, Forrest Gump 180,

11 Important RMSE levels Global average: erroneous User average: Personalization Movie average: Cinematch system: Prize 07 (BellKor): Prize 08 (BellKor+BigChaos): Grand Prize (BellKor s Pragmatic Chaos) : accurate Inherent noise:????

12 Major Challenges 1. Size of data Need memory management and efficiency of algorithms 2. Training and test data are different Test ratings are later in time 3. 99% of data are missing Eliminates many standard methods 4. Countless factors affect ratings: Genre, movie vs. TV vs. other Style of action, dialogue, plot, music Director, actors movie #16322

13 Types of recommender systems 1. Personalized recommendations of items (e.g. Amazon products) to users 3. Content-based: Pre-specified attributes measured for items Users interests estimated for same attributes Examples: eharmony, Pandora 4. Collaborative filtering (CF): Does not require content information about items or user surveys Infers relationships from purchases or ratings Nearest neighbor methods Hidden attribute methods

14

15 Hidden attribute methods Serious The Color Purple Amadeus Braveheart Geared towards females Sense and Sensibility Ocean s 11 Lethal Weapon Geared towards males Dave The Princess Diaries The Lion King Independence Day Gus Dumb and Dumber Escapist

16 Lessons from the Netflix contest Movie # 13043

17 Lesson #1: Look at the data 1. Major steps forward were based on including new aspects of the data: Time-based effects Selection bias: Which movies a user rates is predictive of rating values Daily rating counts are predictive 2. Use human intelligence to define new features of the data.

18 Multiple sources of temporal dynamics Item-based effects: Product perception and popularity change constantly Seasonal patterns influence popularity User-based effects: Customers continually change their tastes Transient, short-term bias; anchoring Drifting rating scale Change of rater within household

19 Something happened in

20 Are old movies better than new ones?

21 Lesson #3: Mathematics helps! Matrix factorization is the leading approach Gradient-descent-based optimization Integration of biases Incorporation of implicit feedback Accounting for temporal effects Combination with a neighborhood model

22 Matrix factorization method items users = items users This is a rank-3 linear algebra approximation!

23 Estimate unknown ratings as dot-products of factor values: items users ? = items users

24 Estimate unknown ratings as inner-products of factors: items users ? = ~ items users

25 Estimate unknown ratings as inner-products of factors: items users ~ ~ items users * *0.5 + (-0.5)*1.4 = 1.6

26 Matrix factorization model = Why can t we use standard linear algebra? 1. Standard linear algebra only applies to matrices where every entry has a known value. 3. Smoothing is necessary: We must learn as much signal as possible where there are sufficient data, but not overfit where data are scarce.

27 Matrix approximation and the Netflix contest Probably the most popular contest method Powerful, fast, and easy to program Simon Funk described first gradient-descent SVD method. Immediately ranked 3 rd place on leaderboard Still today: many related discussions in the Prize forum Monday, December 11, 2006 Netflix Update: Try This at Home Simon Funk is the pseudonym of Brandyn Webb, UCSD CSE B.S. alumnus.

28 Ideas needed to win the Netflix prize 1. Matrix factorization (see your linear algebra class) 2. RMSE cost function (see your statistics class) 3. Gradient descent (see your calculus class) 4. Stochastic gradient descent (machine learning) 5. Regularization (machine learning) 6. Baseline factors 7. A different target: which movies does a user rate? 8. Time-dependent factor values 28

29 Conclusion Learn to be computer scientists, then go out and change the world! Solving the World's Problems Creatively