DEEP LEARNING FOR MUSIC RECOMMENDATION:

Transcription

1 DEEP LEARNING FOR MUSIC RECOMMENDATION: Machine Listening & Collaborative Filtering ORIOL NIETO SEMINAR ON MUSIC KNOWLEDGE EXTRACTION USING MACHINE LEARNING POMPEU FABRA UNIVERSITY BARCELONA DECEMBER 4, 2016

2 Outline Pandora Overview Large Scale Music Recommendation with Deep Learning: Machine Listening Collaborative Filtering

3 Outline Pandora Overview Large Scale Music Recommendation with Deep Learning: Machine Listening Collaborative Filtering

4

5 TODAY 78 M Monthly Listeners 22 H Average Listening per Month 75 B + 10 B 180 K 55 % Stations Artists Spinning per Month Thumbs of all Music Streaming in USA

6 Collective) Intelligence Content&Based Personalized Filtering

7 Outline Pandora Overview Large Scale Music Recommendation with Deep Learning: Machine Listening Collaborative Filtering

8 Collective) Intelligence Content&Based Personalized Filtering

9 Short Time Fourier Transform MY GUITAR GENTLY WEEPS - THE BEATLES

10 Extracting Information Beats Downbeats Chords Structure electric guitar, drums, male singer, pop, classic rock

11 Hand Crafted Features MFCCs Spectral Rolloff Spectral Centroid Structural Features Chromagrams Tempograms CERNs Beats Downbeats Chords Structure electric guitar, drums, male singer, pop, classic rock

12 Learned Features Beats Downbeats Chords Structure DEEP LEARNING (Mostly Convolutional Neural Networks) images.forbes.com/kevinmurnane/files/2016/03/google- deepmind- artificial- intelligence x0-970x646.jpg electric guitar, drums, male singer, pop, classic rock

13 The Music Genome Project Attribute Examples Breathy Voice Nasal Voice Odd Meter Has Banjo Joyful Lyrics >1.5 Million tracks manually analyzed ~400 attributes per track

14 Recommending Music using the MGP Query Track Attributes Ranked Tracks

15 Recommending Music using the MGP EXAMPLE Arjst Title Query Track The Beatles While My Guitar Gently Weeps Ranked 1 Paul McCartney Freedom Ranked 2 Badly Drawn Boy What Is It Now Ranked 3 Jefferson Starship Fading Lady Light

16 Recommending Music using the MGP EXAMPLE Arjst Title Query Track The Beatles While My Guitar Gently Weeps Ranked 1 Paul McCartney Freedom Ranked 2 Badly Drawn Boy What Is It Now Ranked 3 Jefferson Starship Fading Lady Light

17 Recommending Music using the MGP EXAMPLE Arjst Title Query Track The Beatles While My Guitar Gently Weeps Ranked 1 Paul McCartney Freedom Ranked 2 Badly Drawn Boy What Is It Now Ranked 3 Jefferson Starship Fading Lady Light

18 Estimating MGP Attributes USING DEEP LEARNING images.forbes.com/kevinmurnane/files/2016/03/google- deepmind- artificial- intelligence x0-970x646.jpg

19 Estimating MGP Piano Attribute USING D EEP L EARNING images.forbes.com/kevinmurnane/files/2016/03/google- deepmind- artificial- intelligence x0-970x646.jpg

20 Piano Detector USING DEEP LEARNING

21 Piano Detector USING DEEP LEARNING 128 bins Patch 1 Patch 3 Patch 2 N N N

22 Piano Detector DEEP A RCHITECTURE 1024 N Convolutional Layers Dense Layers

23 Piano Detector ESTIMATING F ULL T RACKS

24 Piano Detector RESULTS

25 Piano Detector OUTPUT EXAMPLE

26 Outline Pandora Overview Large Scale Music Recommendation with Deep Learning: Machine Listening Collaborative Filtering

27 Collective) Intelligence Content&Based Personalized Filtering

28 Collaborative Filtering PROBLEM OVERVIEW Explicit Thumbs (up and down) Station Creation [Items (Tracks)?????????? [ Implicit Track Completion Track Skips Users

29 Collaborative Filtering MATRIX FACTORIZATION f Items (Tracks) [???????? [ [ Items (Tracks) [ f [?? Users [ Users (Koren et al., 2009)

30 Collaborative Filtering EXAMPLE Arjst Title Query Track The Beatles While My Guitar Gently Weeps Ranked 1 The Beatles A Day In The Life Ranked 2 The Beatles A Day In The Life (Love Version) Ranked 3 The Beatles While My Guitar Gently Weeps (Love Version)

31 Collaborative Filtering THE GOOD AND THE BAD Rich preference- driven similarity space Powerful at matching the right song with the right listener Latent space is generally not interpretable Can only recommend items that have already been rated

32 Collective) Intelligence Content&Based Personalized Filtering

33 Approximating Item Factors WITH DEEP LEARNING f N Convolutional Layers Dense Layers (van den Oord et al., 2013)

34 Approximating Item Factors ESTIMATING FULL TRACKS

35 Recommending from Learned Factors EXAMPLE Arjst Title Query Track La Bossa d Urina Mama i Papa Ranked 1 Johnny Cash I'm Going To Memphis Ranked 2 Merle Haggard The Woman Made A Fool Out Of Me Ranked 3 Waylon Jennings Lonesome On'ry And Mean

36 ?????????? Collective) Intelligence Content&Based Personalized Filtering

37 ENSEMBLE OF RECOMMENDERS MAY PRODUCE OPTIMAL RECOMMENDATIONS

38 MAN vs MACHINE?

39 MAN + MACHINE

40 MAN + MACHINE Mix,of,Art,and,Science THANKS! ONIETO@PANDORA.COM WE RE HIRING!

41 References Van Den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kavukcuoglu, K. (2016). Wavenet: A Generative Model for Raw Audio, Simonyan, K., & Zisserman, A. (2015). Very Deep Convolutional Networks for Large- Scale Image Recognition. International Conference on Learning Representations, Mcfee, B., Humphrey, E. J., & Bello, J. P. (2015). A Software Framework For Musical Data Augmentation. In Proc. of the 16th International Society for Music Information Retrieval Conference (pp ). Málaga, Spain. Koren, Y., Bell, R., & Volinsky, C. (2009). Matrix Factorization Techniques for Recommender Systems. Computer, 42(8), Oord, A. Van Den, Dieleman, S., & Schrauwen, B. (2013). Deep Content- based Music Recommendation. Advances in Neural Information Processing Systems, Retrieved from deep- content- based- music- recommendation