Topic-weak-correlated Latent Dirichlet Allocation 1

Transcription

1 opc-weak-correlate Latent Drchlet Allocaton Ymn AN, Zhan OU Department of Electronc Engneerng, snghua Unversty, Beng Corresponng emal: Abstract Latent Drchlet allocaton (LDA) has been wely use for analyzng large text corpora. In ths paper we propose the topc-weak-correlate LDA (WC-LDA) for topc moelng, whch constrans fferent topcs to be weak-correlate. hs s techncally acheve by placng a specal pror over the topcwor strbutons. Reucng the overlappng between the topcwor strbutons makes the learne topcs more nterpretable n the sense that each topc wor-strbuton can be clearly assocate to a stnctve semantc meanng. Expermental results on both synthetc an real-worl corpus show the superorty of the WC-LDA over the basc LDA for semantcally meanngful topc scovery an ocument classfcaton. eywors - topc moelng, weak-correlate topcs I. INRODUCION Poneere by latent Drchlet allocaton (LDA) [], probablstc topc moelng s becomng a popular tool for analyzng large unstructure screte ata such as text corpora. he basc ea s that the wors of each ocument are assume to be nepenently rawn from a mxture of multnomals. Each multnomal component s a wor strbuton over the vocabulary, whch we call the topc-wor strbuton. he topc-wor strbutons, or topcs, are share by all ocuments. Each ocument has ts own mxng proporton, whch we call the ocument-topc proporton. Learnng wth topc moels allows us to scover the latent topcs from unstructure text corpora. Posteror nference for the ocument-topc proporton s useful for mensonalty reucton, classfcaton, an nformaton retreval. Snce the ntroucton of the basc LDA, there are a lot of works evelopng topc moels an ther applcatons. Among them, there are many works relate to explorng prors for the LDA, namely, the prors over the topc proporton an over the topc-wor strbuton respectvely. In the basc LDA, the prors are both assume to be rchlet. In contrast to the researches on explorng fferent prors over the topc proporton such as usng the logstc normal pror [][3] or the Drchlet tree pror [4] to evelop correlate topc moels, there have been relatvely few works on explorng prors over the topc-wor strbutons, whch s the man ssue aresse n ths paper. Frst, t shoul be ponte out that the prors over the topcwor strbutons s not merely for smoothng n estmatng the topc-wor probabltes, as ntrouce n the orgnal paper hs work s supporte by NSFC (607500) an 863 program (006AA0Z49). []. hey have practcal effects. For example, usng neste Chnese restaurant process as the prors can learn topc herarches from ata [5]. Usng Gaussan Markov ranom fels as the prors can capture the relatonshps between topcs across multple corpora [6]. Secon, note that the topc term n the LDA s more a metaphor, wth no epstemologcal clams []. he learne topcs are usually name after we nspect the top wors from the learne topc-wor strbutons. opcs are expecte to be stnct n orer to convey nformaton [7]. We thnk, the stncton between fferent topcs can be quantfe by the weak-correlaton between fferent topc-wor strbutons. So f we can reuce the overlappng between the topc-wor strbutons, t wll make the learne topcs more nterpretable n the sense that each topc-wor strbuton can be clearly assocate to a stnct semantc meanng. he above conseratons motvate us to propose the topc-weak-correlate LDA (WC-LDA) for topc moelng, whch constrans fferent topcs (.e. topc-wor strbutons) to be weak-correlate. hs s techncally acheve by placng a specal pror over the topc-wor strbutons, whch exponentally ecreases as the correlaton between fferent topcs ncreases. Varatonal nference proceure s erve for the moel. Expermental results on both synthetc an real-worl corpus show that the WC-LDA can successfully scover the weak-correlate topcs whch have clearer an more stnctve semantc meanngs than topcs learne by the basc LDA. A rect consequence of ths s that WC-LDA elmnates the nee to manually remove stop-wors. In the ocument classfcaton task on Reuters-578 ataset [8], the propose WC-LDA acheves hgher classfcaton accuracy than the basc LDA. It s worthwhle to compare WC-LDA wth some relate researches. Frst, note that whle the prevous correlate topc moel [] ams at capturng the correlaton between the occurrences of latent topcs, WC-LDA focuses on ncorporatng the weak correlaton between the topcs themselves (.e. between topc-wor strbutons). hese approaches complement to each other. Secon, t s recently foun n [7] that LDA usng asymmetrc rchlet pror over ocument-topc strbutons can be robustness to stop-wors. Its man motvaton s that some topcs are assume a pror to occur more frequently n each ocument; these more frequently use topcs are thus force to absorb stop-wors after moel learnng. hs moelng motvaton s fferent from WC- LDA, whch rectly places a weak-correlate pror over topcwor strbutons, thus makes that the topc-wor strbutons are less-overlappe an each topc has stnctve semantc meanng. Although the seemng consequence of the LDA n [7] ISBN /0/$ IEEE 4

2 an WC-LDA s smlar - beng robustness to stop-wors, ther moelng motvaton are from fferent aspects. Moreover, the moel n [7] employs computatonal-ntensve Gbbs samplng, whle WC-LDA uses effcent varatonal nference. he rest of paper s organze as follows. Secton II escrbes the WC-LDA moel an the varatonal nference. he expermental results on both synthetc an realworl corpus are shown n Secton III. Fnally, we present the conclusons n Secton IV. II. W Z N D (a) 3 4 OPIC-WEA-CORRELAED LDA A. he basc LDA he basc LDA [] shown n Fg. (a) assumes that n the w n,, N arses from a corpus, each ocument mxture strbuton over latent topcs. Each wor w s assocate wth a latent topc z accorng to the ocumentspecfc topc proporton vector, whose pror s rchlet wth parameter. he wor w s sample from the topcwor strbuton parameterze by a V matrx, where each row, n, n, s nepenently rawn from an exchangeable rchlet wth parameter. Here an V enotes the number of topcs an the vocabulary sze respectvely. he generatve process for the basc LDA s as follows.. for each ocument, ~ Dr ;. for each of N wor n ocument Choose a topc z ~ Mult ; Choose a wor w ~ Mult. z B. WC-LDA moel formulaton Note that the topc term n the LDA s more a metaphor, wth no epstemologcal clams. he learne topcs are usually Z N D (b) 3 4 Fgure. Graphcal moel representaton for (a) basc LDA, an (b) WC-LDA. Here we set the number of topcs to be four for rawng convenence. W name after we nspect the top wors from the learne topcwor strbutons. So f we can reuce the overlappng between the topc-wor strbutons, t wll make the learne topcs more nterpretable n the sense that each topc-wor strbuton can be clearly assocate to a stnctve semantc meanng. he above conseratons motvate us to propose the topc-weak-correlate LDA (WC-LDA) for topc moelng as shown n Fg. (b), whch constrans fferent topcs (.e. topc-wor strbutons) to be weak- correlate. hs s techncally acheve by placng a specal pror over the topc-wor strbutons, whch exponentally ecreases as the correlaton between fferent topcs ncreases. hs specal pror s a non-conugate pror over the parameters, encong our specal pror knowlege. p( ) exp mn () Z mn where 0 controls the strength of the pror, Z s the normalzng constant. he negatve-log of the pror ensty s proportonal to the sum of all the nner proucts for every par of fferent rows n matrx. he larger the correlaton between fferent topcs s, the smaller the pror s. In ths way, the pror ncorporates the nteracton of fferent topcs an forces them to have weak correlatons. An approxmate formula for s gven n secton II-D. C. Varatonal nference for WC-LDA Here for formula smplcty, we llustrate the posteror nference for a sngle ocument. he nference problem for the WC-LDA s to compute the posteror p, z,, whch s ntractable n general. he basc ea of varatonal nference s to use a tractable strbuton q to approxmate the true posteror strbuton p, an then to mnmze the ullback- Lebler vergence between the strbutons as measure L q p q log q p. Here we use the mean-fel by approxmate strbuton q q z q, where : N,: N,,: N are the varatonal parameters for ocument. he resultng varatonal upate equatons are as follows: N () k k k n k exp k k E log k k, w k (3) V k ' B q( ) exp (4a) k ' k k k D N k k n where z w (4b) B k ' k, kk E k (4c) 5

3 an w s the ncator functon efne as w w From the equatons above, we can see how the nonconugate pror works. Conserng topc k ' an wor, f the occurrng probabltes k of wor n other topcs (.e. k k ' ) are large, t wll lea to a hgh value for B n (4c), whch subsequently encourages a low value of by (4a). herefore, n WC-LDA moel, the wor-probabltes for a gven wor n fferent topcs suppress each other. When 0, (4a) gves a rchlet strbuton, an we can easly compute (3) usng the gamma functon as n the basc LDA. Otherwse, f 0, computng the expectaton n (3) usng (4a) s ntractable. For ths reason, we further constran q k k k k,... (5) an perform the maxmum a-posteror (MAP) estmate for k s. As a result, the expectatons n (3) an (4c) can be easly compute usng the MAP pont estmates. We use the lnesearch technque to calculate the moe of (4a) for MAP estmate. For learnng wth the WC-LDA over multple ocuments, the varatonal upates of () an (3) are terate untl the convergence for each ocument, whle (4) s terate for the corpus scale. he emprcal Bayes estmate for parameter s the same as n the basc LDA moel. D. An approxmate formula for As sa n secton II-B, the nfluence of weak-correlate pror s auste through the strength parameter, when ong posteror nference for. Conserng the lkelhoo lower-boun wth regar to : V D N V L( ) w log n (6) It can be seen that weghts the weak correlate tem k ' k ', whch encourages weak-correlate topcs. We efne as the total number of wors n the corpus. We wll see below that an approxmate formula for s relate to the topc number, the vocabulary sze V an the wor number. If we assume unform strbuton for every topc-wor strbuton n, an the occurrences of every topc n the ocuments are unform, then we have w. For the V unform matrx, we have log V log/ V V we obtan the frst tem n L( ) as. hus ABLE : OPICS LEARNED BY LDA (LEF) AND WC-LDA (RIGH) RESPECIVELY. EACH COLUMN IS HE OP-EN WORDS IN HE LEARNED OPIC-WORD DISRIBUION. LDA WC-LDA topc topc topc 3 topc 4 topc topc topc 3 topc V D N L w V n log log/ (7) an the weak correlate tem n L( ) as L V V V V * * * / (8) Fnally, we obtan an approxmate formula for, whch says s proportonally to a value efne by, V an as follows, L log / V V approxmate log / V (9) L / V In practce n orer to obtan a reasonable value for, we only nee to tune the proporton factor frst n a small-scale experment, an then fxe n later large-scale experments. III. EXPERIMENAL RESULS A. Synthetc ataset Suppose that there s an magnary vocabulary of 400 wors, wth the wor- beng from 0 to 399. he 400 wors are equally ve nto 4 topcs. Every wor s har assgne to one topc, an every topc has ts own 00 wors. Specfcally, the topc-wor assgnments are that: wor 0~99 for topc, wor 00~99 for topc, wor 00~99 for topc 3, wor 300~399 for topc 4. he wor assgnment probabltes over 00 wors n each topc are ranomly generate. he topc that conssts of wor 0~99, s chosen to be the smulate topc of syntactc-wors whch occur more frequently. herefore we set a relatvely larger hyperparameter for topc n the rchlet pror ( 5, ). A total of 6000 ocuments (30 wors per ocument) are generate. Usng the above synthetc ataset, we learn the four topcs wth LDA an WC-LDA respectvely. It can be seen from able that the WC-LDA can successfully learn the four topcs of the smulate moel, but the basc LDA fals. he he orer of the learne topcs s not guarantee. For example, the learne topc by WC-LDA actually correspons to the real topc 4. 6

4 four topcs learne by the basc LDA are almost occupe by the wors from topc (.e. wor 0~99), whch represent smulate syntactc-wors. B. Real-worl text corpus For all the followng experments on real-worl corpus, we set the number of topcs to be 30, hyperparameter to be 0.5. ) Qualtatve assessment of the learne topcs We use the subset of the REC AP corpus [5] contanng 6333 artcles wth 3075 unque terms, whch was the same as the corpus use n [], an Year 994 Chna aly spaper (CDN) corpus. We remove the stop-wors n REC AP corpus before runnng topc moelng. We use the raw CDN corpus, all the wors are kept. he topcs learne are shown n able - 5. able an 4 shows the topcs learne by the basc LDA. Wthout eletng the stop-wors, the topcs learne by the basc LDA are mostly occupe by the syntactc wors (see able ), an thus t s ffcult to tell the semantc meanng of the topcs. In experments wth eletng the stop-wors, such problem s allevate to some extent. he topc n able 4 has the clear semantc meanng of law, whle topc 3 an topc 4 n able 4 are stll occupe by some semantc-vague wors whch are marke n re, such as I,,,,. able 3 an 5 shows topcs learne by WC-LDA. able 3 an 5 show that whether eletng the stop-wors or not, the WC-LDA can successfully learn the topcs wth clear semantc meanngs. Incorporatng weak-correlaton among topcs makes that each topc has ts own stnctve semantc meanng. Moreover, the WC-LDA can also scover the topcs wth fferent syntactc functons. For example, topc 3 n able 3 nclues preposton wors an topc 4 n able 3 nclues numeral wors. o make clear the semantc meanngs of learne topcs by eletng preefne stop-wor lst s subectve an nonaaptve. he stop-wor lst may be corpus-specfc. Weakcorrelate topcs mprove ths problem by constranng the structure among topcs. For example, preposton wors have hgh probabltes n topc 3 of able 3. he weak correlaton between topc 3 an other topcs prevents preposton wors spreang nto other topcs, an thus helps other topcs to have clearer semantc meanngs. ) Quanttatve Analyss of the learne topcs We conuct quanttatve analyss to see whether WC- LDA learn more stnctve topcs than LDA. We compare the correlaton between topcs extracte by LDA an WC-LDA. We efne the confuson matrx C, whose off-agonal elements represent the value of the cross-correlaton between fferent topcs, an W whch s the sum of all the mn m off-agonal elements n the confuson matrx C. he measurement of W gves an overall evaluaton of the correlaton between fferent topcs. It s clear from able 6 n an Fg. that the learne topcs of WC-LDA have sgnfcantly weaker correlaton than that of LDA. ABLE OPICS LEARNED BY BASIC LDA FROM CDN (RAW CORPUS). EACH CLOLUMN IS HE OP-EIGH WORDS IN HE LEARNED OPIC-WORD DISRIBUION. topc topc topc 3 topc 4 s s -e prouct n an enterprse market ABLE 3: OPICS LEARNED BY WC-LDA FROM CDN (RAW CORPUS). EACH CLOLUMN IS HE OP-EIGH WORDS IN HE LEARNED OPIC-WORD DISRIBUION. topc topc topc 3 topc 4 crme offce case safe polce attack court law ABLE 4: OPICS LEARNED BY BASIC LDA FROM AP-CORPUS (SOP-WORDS REMOVED). EACH CLOLUMN IS HE OP-EIGH WORDS IN HE LEARNED OPIC-WORD DISRIBUION. topc topc topc 3 topc 4 frst lke ust ABLE 5: OPICS LEARNED BY WC-LDA FROM AP-CORPUS (SOP-WORD REMOVED). EACH CLOLUMN IS HE OP-EIGH WORDS IN HE LEARNED OPIC-WORD DISRIBUION. topc topc topc 3 topc 4 state tme frst s to an have he come s culture publsh culture reaer epoch traton reaer book court case attorney tral uge charge prson sentence feeral court case rug uge attorney tral charges prson nvestgaton s sports -e an game tran have to s n an on m have to for sovet gorbachev ar afrca flght sovet unte government mltary states present war foregn offcal s s n an art auence musc -e ten three eght hunre nne seven thousan government present natonal communst congress bll senate commttee buget congress tax rep sen house 7

5 opcs opcs 3) Document Classfcaton he ocument-specfc posteror topc proportons can be use as a reuce-mensonal representaton of the ocument, whch serves as the feature for ocument classfcaton. We conuct bnary text classfcaton usng Reuters-578 ataset. After removng the space character, non-englsh characters, an a lst of 5 stop wors, we obtan a ataset wth 3476 ocuments an 353 wors. In the experment, we ve the whole ataset nto tranng set an test set. Utlzng the SVMLght package [9], we tran support vector machnes wth the feature prove by LDA an WC-LDA respectvely. he classfcaton experments focuse on man categores n Reuters-578 ataset - EARN an GRAIN. Expermental results n Fg.3 show that the ocument classfcaton accuraces of the basc LDA we mplemente are comparable to the results reporte n the orgn paper of LDA []. For fferent tranng ata proportons, the propose WC opcs Fgure. Confuson matrx for WC-LDA (left), an LDA (rght) (Lower value was showe n arker color) ABLE 6: COMPARISON OF CORRELAIONS BEWEENOPICS Dataset W of LDA W of WC-LDA REC-AP Chna Daly Newspaper Accuracy % LDA feature WC-LDA feature Proporton of tranng ata Accuracy % opcs LDA feature WC-LDA feature Proporton of tranng ata Fgure 3. Classfcaton Accuracy of opc EARN vs. NO EARN (left), GRAIN vs. NO GRAIN (rght) LDA moel consstently acheves hgher classfcaton accuraces than the basc LDA moel. IV. CONCLUSIONS In ths paper, we propose the topc-weak-correlate LDA (WC-LDA) for topc moelng, whch constrans fferent topcs (.e. topc wor-strbutons) to be weak-correlate. Such weak correlaton forces each topc to have clear an stnctve semantc meanng. Wthout manually eletng stopwors, the WC-LDA can scover topcs wth clear semantc meanngs. In the task of ocument classfcaton, the propose WC-LDA moel acheves hgher classfcaton accuraces than the basc LDA moel. opc moelng has been use n computer vson to learn natural scene categores [0]. However, t becomes harer for researchers to efne approprate stop-patches lst for mages topc moelng. Although ths paper focuses on text analyss, the WC-LDA moel can also be apple n other applcatons. It s worthwhle further stuyng the applcaton of weak-correlate topc moelng for computer vson. REFERENCES [] D. M. Ble, A. Y. Ng, an M. I. Joran. (003) Latent Drchlet allocaton. Journal of Machne Learnng Research, 3:993 0, 003 [] D. M. Ble an J. D. Lafferty (007). A correlate topc moel of Scence. Annals of Apple Statstcs, ():7 35. [3] D. Mmno, H. Wallach, an A. McCallum. (008) Gbbs Samplng for Logstc Normal opc Moels wth Graph-Base Prors. NIPS Workshop on Analyzng Graphs, 008 [4] am, Y.-C., & Schultz,. (007). Correlate latent semantc moel for unsupervse LM aaptaton. IEEE Intl. Conf. on Acoustcs, Speech an Sgnal Processng [5] D. Ble,. Grffths, M. Joran, an J. enenbaum. (003). Herarchcal topc moels an the neste Chnese restaurant process. Avances n Neural Informaton Processng Systems, 6, MI Press [6] C.Wang, B. hesson, C. Meek, an D. Ble. Markov topc moels. In he welfth Internatonal Conference on Artfcal Intellgence an Statstcs (AISAS), pages , 009. [7] H. M. Wallach, D. Mmno, an A. McCallum. Rethnkng la: Why prors matter. In Proceengs of the 3r Annual Conference on Neural Informaton Processng Systems,009 [8] [9]. Joachms, Makng large-scale SVM Learnng Practcal. Avances n ernel Methos - Support Vector Learnng, B. Schölkopf an C. Burges an A. Smola (e.), MI-Press, 999 [0] Fe-Fe, L. an Perona, P. (005). A Bayesan herarchcal moel for learnng natural scene categores. IEEE Computer Vson an Pattern Recognton, pages