Large Scale Topic Detection using Node-Cut Partitioning on Dense Weighted-Graphs

Size: px

Start display at page:

Download "Large Scale Topic Detection using Node-Cut Partitioning on Dense Weighted-Graphs"

Walter Barrett
5 years ago
Views:

1 Large Scale Topic Detection using Node-Cut Partitioning on Dense Weighted-Graphs Kambiz Ghoorchian Šarūnas Girdzijauskas

2 Motivation Solution Results Conclusion 2

3 What is a Topic (Trending Topic)? #ChewbaccaMom

4 What is a Topic (Trending Topic)? #susanboyle #IranElection #FacebookIsDown #FindingDory #ChewbaccaMom #Politics #Aylan #Russia #TweetDeck #JobMarket #Apple #Wimbledon #Sport #Euro206 #Superbowl #Immigration رمضان# #Trump #uselections206 #Stefanlöfven 4

5 Why Topics (Trends) are Important? 5

6 Why Topics (Trends) are Important? 6

7 Why Topics (Trends) are Important? 7

8 What is Topic Detection? Given a large number of documents (e.g., tweets), how can we extract the most frequent (significant) topics (trends)? 8

9 Current Solutions 9

10 Current Solutions Statistical Topic Modeling Machine Learning 0

11 Current Solutions Document-Topic Statistical Topic Modeling Matrix Factorization Latent Dirichlet Allocation (LDA)[] Hierarchical LDA (HLDA) Machine Learning Document-Term W W2 W W4 D 0 D2 0 0 D 0 0 Dn 0 T T2 T Tk D D D Dn Word-Topic T T2 T Tk W W W Wm David M. Blei, Andrew Y. Ng, Michael I. Jordan; Latent Dirichlet Allocation (Jan):99-022, 200.

12 Current Solutions Document-Topic Statistical Topic Modeling Matrix Factorization Latent Dirichlet Allocation (LDA)[] Hierarchical LDA (HLDA) Machine Learning. Document Modeling Vector Modeling Graph Modeling 2. Topic Detection Unsupervised - Clustering Supervised - Classification Document-Term W W2 W W4 D 0 D2 0 0 D 0 0 Dn 0 T T2 T Tk D D D Dn Word-Topic T T2 T Tk W W W Wm David M. Blei, Andrew Y. Ng, Michael I. Jordan; Latent Dirichlet Allocation (Jan):99-022,

13 Limitations

14 Limitations Sparsity Short messages have Less informative co-occurrence patterns which results in[]:. False segmentation of topics. 2. Difficulty in identification of ambiguous words (Apple, Computer vs Fruit). [] - Liangjie et al, Empirical Study of Topic Modeling in Twitter. SOMA 200 [2] - 4

15 Limitations Sparsity Short messages have Less informative co-occurrence patterns which results in[]:. False segmentation of topics. 2. Difficulty in identification of ambiguous words (Apple, Computer vs Fruit). Dynamism Constant emergent of New phrases or Acronyms (e.g., Selfie, Unlike, Phablet, IAVS = I am very sorry, IWSN = I want sex now). [] - Liangjie et al, Empirical Study of Topic Modeling in Twitter. SOMA 200 [2] - 5

16 Limitations Sparsity Short messages have Less informative co-occurrence patterns which results in[]:. False segmentation of topics. 2. Difficulty in identification of ambiguous words (Apple, Computer vs Fruit). Dynamism Constant emergent of New phrases or Acronyms (e.g., Selfie, Unlike, Phablet, IAVS = I am very sorry, IWSN = I want sex now). Scalability 0M active-users/month [2] 500M messages/day [2] [] - Liangjie et al, Empirical Study of Topic Modeling in Twitter. SOMA 200 [2] - 6

17 Solution Unsupervised learning: -Graph Modeling 2-Node-cut Partitioning 7

18 Solution Unsupervised learning: -Graph Modeling 2-Node-cut Partitioning Documents D D2 D D4 D5 D6 8

19 Solution Unsupervised learning: -Graph Modeling 2-Node-cut Partitioning Documents D D2 D D4 D5 D6 - Graph Modeling 9

20 Solution Unsupervised learning: -Graph Modeling 2-Node-cut Partitioning Documents D D2 D D4 D5 D6 Random Indexing Knowledge Base Word W W2 W W4 W5 W6 W7 W8. RI Vector V V2 V V4 V5 V6 V7 V8 - Graph Modeling 20

21 Solution Unsupervised learning: -Graph Modeling 2-Node-cut Partitioning Documents D D2 D D4 D5 D6 Random Indexing Knowledge Base Word W W2 W W4 W5 W6 W7 W8. RI Vector V V2 V V4 V5 V6 V7 V8 - Graph Modeling 2 - Node-Cut Partitioning 2

22 - Graph Modeling using Random Indexing 22

23 Random Indexing (RI) Is a dimensionality reduction method (similar to hashing). 2

24 Random Indexing (RI) Is a dimensionality reduction method (similar to hashing). Documents D = {W, W4, W8, } D2 D D4 D5 D6 24

25 Random Indexing (RI) Is a dimensionality reduction method (similar to hashing). Random Indexing Knowledge Base Documents D = {W, W4, W8, } D2 D D4 D5 D6 Random Indexing Wor RI Vector W V = {a, b, c, d, e, f} W2 W W4 V4 = {a4, b4, c4, d4, e4, f4} W5 W6 W7 W8 V8 = {a8, b8, c8, d8, e8, f8}. 25

26 Random Indexing (RI) Is a dimensionality reduction method (similar to hashing). Random Indexing Knowledge Base Documents D = {W, W4, W8, } D2 D D4 D5 D6 Random Indexing Wor RI Vector W V = {a, b, c, d, e, f} W2 W W4 V4 = {a4, b4, c4, d4, e4, f4} W5 W6 W7 W8 V8 = {a8, b8, c8, d8, e8, f8}.. Unique 2. Fixed length. Captures Co-occurrence patterns of the words 26

27 Random Indexing (RI) Is a dimensionality reduction method (similar to hashing). Random Indexing Knowledge Base Documents D = {W, W4, W8, } D2 D D4 D5 D6 Random Indexing Wor RI Vector W V = {a, b, c, d, e, f} W2 W W4 V4 = {a4, b4, c4, d4, e4, f4} W5 W6 W7 W8 V8 = {a8, b8, c8, d8, e8, f8}.. Unique 2. Fixed length. Captures Co-occurrence patterns of the words 27

28 Graph Modeling 28

29 Graph Modeling RI - Knowledge Base Documents D = {W, W4, W8, } D2 = {W2, W, W7, } D = {W4, W, W, } D4 = {W2, W6, W9, } D5 = {W, W4, W8, } D6 = {W, W, W7, } Wor RI Vector W V = {a, b, c, d, e, f} W2 V2 = {a2, b2, c2, d2, e2, f2} W V = {a, b, c, d, e, f} W4 V4 = {a4, b4, c4, d4, e4, f4} W5 V5 = {a5, b5, c5, d5, e5, f5} W6 V6 = {a6, b6, c6, d6, e6, f6} W7 V7 = {a7, b7, c7, d7, e7, f7} W8 V8 = {a8, b8, c8, d8, e8, f8}. 29

30 Graph Modeling RI - Knowledge Base Documents D = {W, W4, W8, } D2 = {W2, W, W7, } D = {W4, W, W, } D4 = {W2, W6, W9, } D5 = {W, W4, W8, } D6 = {W, W, W7, } Wor RI Vector W V = {a, b, c, d, e, f} W2 V2 = {a2, b2, c2, d2, e2, f2} W V = {a, b, c, d, e, f} W4 V4 = {a4, b4, c4, d4, e4, f4} W5 V5 = {a5, b5, c5, d5, e5, f5} W6 V6 = {a6, b6, c6, d6, e6, f6} W7 V7 = {a7, b7, c7, d7, e7, f7} W8 V8 = {a8, b8, c8, d8, e8, f8}. 0

31 Graph Modeling a b RI - Knowledge Base f Documents D = {W, W4, W8, } D2 = {W2, W, W7, } D = {W4, W, W, } D4 = {W2, W6, W9, } D5 = {W, W4, W8, } D6 = {W, W, W7, } Wor RI Vector W V = {a, b, c, d, e, f} W2 V2 = {a2, b2, c2, d2, e2, f2} W V = {a, b, c, d, e, f} W4 V4 = {a4, b4, c4, d4, e4, f4} W5 V5 = {a5, b5, c5, d5, e5, f5} W6 V6 = {a6, b6, c6, d6, e6, f6} W7 V7 = {a7, b7, c7, d7, e7, f7} W8 V8 = {a8, b8, c8, d8, e8, f8}. a e e b d c b f c d

32 Graph Modeling a b RI - Knowledge Base f Documents D = {W, W4, W8, } D2 = {W2, W, W7, } D = {W4, W, W, } D4 = {W2, W6, W9, } D5 = {W, W4, W8, } D6 = {W, W, W7, } Wor RI Vector W V = {a, b, c, d, e, f} W2 V2 = {a2, b2, c2, d2, e2, f2} W V = {a, b, c, d, e, f} W4 V4 = {a4, b4, c4, d4, e4, f4} W5 V5 = {a5, b5, c5, d5, e5, f5} W6 V6 = {a6, b6, c6, d6, e6, f6} W7 V7 = {a7, b7, c7, d7, e7, f7} W8 V8 = {a8, b8, c8, d8, e8, f8}. a e e b d c b f a e b d c f c d 2

33 Graph Modeling b RI - Knowledge Base f c Documents D = {W, W4, W8, } D2 = {W2, W, W7, } D = {W4, W, W, } D4 = {W2, W6, W9, } D5 = {W, W4, W8, } D6 = {W, W, W7, } Wor RI Vector W V = {a, b, c, d, e, f} W2 V2 = {a2, b2, c2, d2, e2, f2} W V = {a, b, c, d, e, f} W4 V4 = {a4, b4, c4, d4, e4, f4} W5 V5 = {a5, b5, c5, d5, e5, f5} W6 V6 = {a6, b6, c6, d6, e6, f6} W7 V7 = {a7, b7, c7, d7, e7, f7} W8 V8 = {a8, b8, c8, d8, e8, f8}. a e e a d d c f a e b d c f e d

34 Graph Modeling RI - Knowledge Base Documents D = {W, W4, W8, } D2 = {W2, W, W7, } D = {W4, W, W, } D4 = {W2, W6, W9, } D5 = {W, W4, W8, } D6 = {W, W, W7, } Wor RI Vector W V = {a, b, c, d, e, f} W2 V2 = {a2, b2, c2, d2, e2, f2} W V = {a, b, c, d, e, f} W4 V4 = {a4, b4, c4, d4, e4, f4} W5 V5 = {a5, b5, c5, d5, e5, f5} W6 V6 = {a6, b6, c6, d6, e6, f6} W7 V7 = {a7, b7, c7, d7, e7, f7} W8 V8 = {a8, b8, c8, d8, e8, f8}. 4

35 Graph Modeling RI - Knowledge Base Documents D = {W, W4, W8, } D2 = {W2, W, W7, } D = {W4, W, W, } D4 = {W2, W6, W9, } D5 = {W, W4, W8, } D6 = {W, W, W7, } Wor RI Vector W V = {a, b, c, d, e, f} W2 V2 = {a2, b2, c2, d2, e2, f2} W V = {a, b, c, d, e, f} W4 V4 = {a4, b4, c4, d4, e4, f4} W5 V5 = {a5, b5, c5, d5, e5, f5} W6 V6 = {a6, b6, c6, d6, e6, f6} W7 V7 = {a7, b7, c7, d7, e7, f7} W8 V8 = {a8, b8, c8, d8, e8, f8}. 2 - Node-Cut Partitioning 5

36 2 - Node-Cut Partitioning 6

. F Rahimian, AH Payberah, S Girdzijauskas, S Haridi: Distributed

37 Node-Cut Partitioning Ja-Be-Ja-VC[] balanced, k-way partitioning for un-weighted graphs based on node-cut minimization.. F Rahimian, AH Payberah, S Girdzijauskas, S Haridi: Distributed Vertex-cut Partitioning, in Distributed Applications and Interoperable Systems, ,

38 Node-Cut Partitioning 8

39 Node-Cut Partitioning k = 2 Random Initialization 9

40 Node-Cut Partitioning k = 2 e e Random Initialization Iteration C = Blue C = Red Gain Heat 40

41 Node-Cut Partitioning k = 2 e e e e Random Initialization Iteration C = Blue C = Red Gain Heat 4

42 Node-Cut Partitioning k = 2 e e e e e e e e Random Initialization Iteration Iteration C = Blue C = Red Gain Heat 42

43 Node-Cut Partitioning k = 2 e e e e e e e e Random Initialization Iteration Iteration C = Blue C = Red Minimum Cut Size Gain Heat 4

44 Modifications Same Utility Function Gain Heat Weighted Gain factor Weighted Cut 44

45 Un-Weighted Graph 5, 5 5, 5 Modifications e e e Weighted Graph 5 5 e e e,, 9 45

46 Modifications Un-Weighted Graph Weighted Graph 5, 5 5, e e e e, e e, Weighted Graph 5 5 e e e,, 9 46

47 Modifications Un-Weighted Graph Weighted Graph 5, 5 5, e e e e, e e, Weighted Graph 5 5 e e e. Scalability,, 9 2. Convergence 47

48 Modifications 5 e e e 5, 9, 48

49 Modifications 49 49,, 9 e 5 e e 5 2, 0 e e 5 e 2

50 Experiments 50

51 Experiments. Accuracy (Quantitative) SNAP Twitter Trending Topics from 2009 [] EXP - Topics 25 Documents K = 00 Sam = 20% EXP2-8 Topics 275 Documents K = 00 Sam = 20% A. Scalability (Qualitative) TREC Tweets 20-6M Tweets [2] SNAP Twitter 2009 Topic Acronym EXP EXP2 Harry Potter (HP) HP 457 American Idol (AI) AI 424 Dollhouse (DH) DH 262 Slumdog Milliner (SM) SM 280 Susan Boyle (SB) SB Swine Flue (SF) SF Tiger Wood (TW) TW 2242 Tweetdeck (TD) TD 5860 Wimbledon (WI) WI 654 EXP 2756 Documents

52 Experiments Comparison GibsLDA - baseline [] BiTerm - Best known solution[2]. David M. Blei, Andrew Y. Ng, Michael I. Jordan; Latent Dirichlet Allocation (Jan):99-022, Yan, Xiaohui and Guo, Jiafeng and Lan, Yanyan and Cheng, Xueqi, A Biterm Topic Model for Short Texts, WWW. 52

53 Experiments - Evaluation F-Score (Quantitative) = [0 ] Average Coherence Score (Qualitative) = [Log(k/n) Log(+k/n)] = [ ] 5

54 LDA BiTerm Our s EXP - SNAP Topics - F-Score 54

55 LDA BiTerm Our s EXP2 - SNAP 8 Topics - F-Score 55

56 EXP - TREC - Coherency Tweets 00K Edges 7,9M Vertices 4000 Avg_Deg 948 Partitions 500 Duration LDA 684s BiTerm 97s Our Algorithm 7000s (Centralized) EXP - Twitter Large Large Dataset - Average Coherence Score - K=500 Num Top Words LDA BiTerm Our Algorithm

57 EXP - SNAP Topics - Coherency Tweets 2K Edges 2.M Vertices 994 Avg_Deg 75 Partitions 00 Duration LDA.s BiTerm 2s Our Algorithm 6000s (Centralized) EXP - Twitter Topics - Average Coherency Score - K=00 Num Top Words LDA BiTerm Our Algorithm

58 EXP2 - SNAP 8 Topics - Coherency Tweets 2K Edges 7,5M Vertices 4000 Avg_Deg 779 Partitions 00 Duration LDA 7S BiTerm 24S Our Algorithm 6000s (Centralized) EXP - Twitter 8 Topics - Average Coherence Score - K=00 Num Top Words LDA BiTerm Our Algorithm

59 Percentage Scalability Duration Growth Rate 59

60 Conclusion Achievements Efficient and scalable solution for topic detection. Solves Sparsity and Dynamism using RI Knowledge-base Meets Scalability using Graph Partitioning Future work Enhance initialization and language modeling Extend the algorithm to an streaming model since Graph construction is incremental 60

61 Thank You Questions? Bibliography. Sahlgren, M. (2005) An Introduction to Random Indexing, Proceedings of the Methods and Applications of Semantic Indexing Workshop at the 7th International Conference on Terminology and Knowledge Engineering, TKE 2005, August 6, Copenhagen, Denmark. 2. Kanevara, P: Sparse Distributed Memory and Related Models. Associative Neural Memories, Oxford University Press, 99.. Kanerava, P., Kristoferson, J., and Holst, A. (2000). Random indexing of text samples for latent semantic analysis. In Gleitman, L. R. and Josh, A. K., editors, Proceedings of the 22nd Annual Conference of the Cognitive Science Society, page 06, Mahwah, New Jersey. Erlbaum. 4. Johnson, W. and Lindenstrauss, J. (984). Extensions of Lipschitz mappings into a Hil- bert space. In Beals, R., Beck, A., Bellow, A., and Hajian, A., editors, Conference on Modern Analysis and Probability (982: Yale University), volume 26 of Con- temporary Mathematics, pages American Mathematical Society. 5. K Ghoorchian, F Rahimian, S Girdzijauskas: Semi Supervised Multiple Disambiguation, Trustcom/BigDataSE/ISPA, 205 IEEE 2, img. Img Img Img Img

Document Content-Based Search Using Topic Modeling

Document Content-Based Search Using Topic Modeling Jason Bello, Brian de Silva, Jerry Luo University of California, Los Angeles August 9, 2013 Jason Bello, Brian de Silva, Jerry Luo (UCLA) Topic Modeling