Comparison of Global Term Expansion Methods for Text Retrieval

Transcription

1 Proceedngs of NTCIR-5 Worshop Meetng, December 6-9, 2005, Toyo, Japan Comparson of Global Term Expanson Methods for Text Retreval Yuen-Hsen Tseng, Yu-Chn Tsa*, and Ch-Jen Ln** Natonal Tawan Normal Unversty, Tape Tawan, R.O.C., 106 *Fu Jen Catholc Unversty, Tape Tawan, R.O.C., 242 **WebGene Informaton LTD., Tape Tawan, R.O.C., 106 Abstract Ths paper descrbes our wor at the ffth NTCIR worshop on the subtass of sngle language nformaton retreval (SLIR). Several automatc global uery expanson strateges were explored based on a machne-derve thesaurus. These term selecton strateges were compared wth manual selecton and local expanson. Experments show that all the global expanson strateges perform worse than the smple local expanson. Furthermore, even wth the help of a human n selectng the global terms, the performance may not be better than an automatc local feedbac method, f the human does not fully understand the nformaton need of the search topc. However, the machne-derved thesaurus does extract relevant global terms for expanson. Proper term selecton can yeld great mprovement n performance than local feedbac alone. Keywords: Chnese IR, term assocaton, global expanson. 1. Introducton In NTCIR-3, we partcpated n the Chnese, Japanese, and Korean sngle-language retreval tass (SLIRs) usng an nformaton retreval system that dealt wth these three languages n exactly the same way wthout usng language-dependent nowledge or resources [1]. Results showed that our retreval effectveness does not show any dfference among these three SLIRs. However, the effectveness of our system was lower than the average of all runs submtted to the NTCIR by all partcpants. Post-analyss showed that the basc retreval strateges used n our and others systems dd not match top-performng systems that used other sophstcated technues such as blnd relevance feedbac (BRF), probablstc retreval model, hybrd term ndexng, and ttle words re-weghtng [2-4]. Among these sophstcated technues, BRF s the maor approach that mproves performance most. Relevance feedbac s a technue that modfes the orgnal uery based on the ntal retreval results. If relevant (or rrelevant) terms can be dened from the ntal results, addng them to (or subtractng them from) the orgnal uery for another run of retreval often mproves the retreval effectveness. However, snce relevant terms are unnown to the system untl the ntal results were nspected and feedbaced by a searcher, most BRF methods under automatc retreval mode smply assume that the top-raned documents retreved from the ntal uery are relevant. Terms are then extracted from these relevant documents so as to add to the ntal uery. Ths way of uery modfcaton or uery expanson s called local expanson snce only a handful of documents relevant to the ntal uery are used. Informaton from the rest of the documents s not exploted at all. In contrast, f the feedbac nformaton comes from the entre collecton, we call ths way of relevance feedbac global expanson. Our prelmnary experments n NTCIR-4 showed that f best term selecton can be acheved, uery expanson based on the feedbac terms from the entre collecton can perform smlarly well wth local BRF and combnng both local and global expanson can outperform each method alone [5]. A number of global expanson methods has been proposed n the past lterature. A common way s to use a thesaurus, whch s a powerful search ade tool heavly referred to n nformaton access servces. A thesaurus lsts canddate search terms and relatonshps among them. Ths vocabulary nowledge can be used for broadenng or narrowng the user s search topcs, or for suggestng synonyms or related terms to reduce the vocabulary msmatch problem, whch has long been one of the maor

2 Proceedngs of NTCIR-5 Worshop Meetng, December 6-9, 2005, Toyo, Japan causes of search falure. However, tools le ths often reure laborous human nvolvement to add nowledge to them, mang usng thesaur an expensve choce. Besdes, the use of general purpose thesaur does not mprove retreval effectveness n a number of nformaton retreval experments [6-8]. Query expanson based on manual thesaur only succeeds when the terms covered by the thesaur correspond closely to the vocabulary used n the document collecton [9-10]. Thus, thesaur that are collecton-specfc and that can be easly generated and mantaned wth mnmum cost are of great value. For term suggeston and collecton exploraton durng an nteractve search scenaro, we have proposed an automatc and effcent way for generatng a thesaurus based on term co-occurrence. We beleve that f best thesaurus terms can be selected, the thesaurus can also help n automatc search scenaro such as n those topc-based retreval tass of NTCIR worshops. Ths wor reports our efforts n selectng the thesaurus terms for global expanson. We expermented on the collectons of NTCIR-3 and appled the best results to the collectons of NTCIR-5. Next secton wll ntroduce our ways of selectng global terms for uery expanson. Secton 3 wll descrbe the retreval strateges and expermental results on NTCIR-3 s collectons. Secton 4 reports our retreval results submtted to the NTCIR-5. Fnally we conclude and summarze our observatons n Secton Selecton of Global Relevant Terms There are a number of approaches to generate a co-occurrence thesaurus from the entre document collecton. However, most methods reure a tremendous amount of computaton. Specfcally, O(m 2 ) term-to-term smlartes were calculated wth each term-par smlarty reurng O(n) steps, where m s the number of collecton terms and n s the number of collecton documents. Ths leads to an O(m 2 n) method. Due to ths dffculty n obtanng a thesaurus, Tseng proposed another method that s far more effcent [11]. The maor dea of hs method s to lmt the terms to be assocated to those that co-occur n the same cal segments of a smaller text sze, such as a sentence or a paragraph, rather than n the entre document. Assocaton weghts are computed n ths way for each document and then accumulated over all documents. Ths changes t nto a roughly O(n 2 s) algorthm, where s the number of selected eywords for assocaton n a document and s s the average number of sentences n a document. In ths way, a global term relaton structure can be obtaned effcently. For the 381,375 Chnese documents n the NTCIR-4 collecton (469 MB of texts), t only taes 133 mnutes on a noteboo computer wth a 1.7 GHz CPU and 512 Mega RAM for ndexng, eyword extracton, and term assocaton computaton. As to the effectveness, two experments have been conducted. In one experment, 30 topcs (sngle uery terms) were selected from the ndex terms of 25,230 Chnese news artcles, from whch term assocaton was analyzed. Fve assessors (all maored n lbrary scence) were nvted for relevance udgment. For each topc, ts top N (N = 50) related terms were examned. Users were ased f they thought the relatonshp between a topc and each of ts assocated terms was relevant enough. If they are not sure (mostly due to lac of doman nowledge), they are advsed n advance to retreve those documents that may explan ther assocatons. The results show that n terms of percentage of relatedness, there are 69% assocated terms udged relevant to the topc terms [11]. In another smlar experment based on a much larger collecton (154,720 documents), the percentage of relatedness ncreases to 78.33% [12]. As can be seen, the more the documents for analyss, the better the effectveness of the term assocaton. To have an dea of the term assocaton results from the NTCIR collectons, Fgure 1 shows an example. The hghlghted search term: Ara Kurosawa s from the topc 012 of NTCIR-4 s CLIR topc set. Twelve top-raned co-occurred terms were shown, wth more relevant terms n closer dstance to the search term. In our mplementatons, at most 64 co-occurred terms for each eyword were ept n the term relaton structure for later use. Fgure 1. An example of global term expanson. The example from Fgure 1 seems to suggest that the assocated terms are all good for expanson. But ths s only a postve example. The assocated terms ndeed are related to the search terms wth

3 Proceedngs of NTCIR-5 Worshop Meetng, December 6-9, 2005, Toyo, Japan respect to some topcs or events mentoned n the collecton. But some of such topcs or events may not relate to the search topc represented by the sngle search term. Therefore, the goal n ths wor s to add to the uery wth proper terms that are assocated wth a set of search terms whch together represent a search topc. Specfcally, each uery strng was segmented by the top-2 longest ndex terms. The resultant uery terms, called eywords (KWs) hereafter, are used to fetch ther related terms (RTs) from the thesaurus generated above. Our goal s to select the proper terms from these RTs for global expanson. Four automatc strateges for global term selecton have been tred. They are denoted as from S1 to S4 and are descrbed as follows. S1: Canddate RTs are selected based on ther document freuences relatve to ther correspondng eywords and based on whether they are common RTs of more than one KW. Specfcally, hgh-freuency KWs (whose df > 20/n) were dscarded. RTs havng larger df than ther KWs are also dscarded. Fnally, only those RTs assocated to at least two KWs are used for expanson. Ths lmtaton on the added terms s to avod topc drft, a phenomenon that changes the topc of the orgnal uery as more (rrelevant) terms were added. But ths also lmts the number of terms for global uery expanson such that most topcs have only a few addtonal terms. Actually ths s the strategy that was used n our experment n NTCIR-4. S2: Ths strategy uses only common RTs wthout the freuency lmtaton. That s, common RTs of orgnal KWs are used to retreve new RTs. The ntersecton of the new RTs and the orgnal RTs are used for expanson. Expandng RTs addtonally may expand the topc foc. But by selectng the common RTs, we may have the chance to lmt the focus of the topc to those that we need whle broadenng the base for RT canddates. Indeed, we have observed more addtonal RTs ncluded by ths strategy than those by S1. S3: The above two strateges do not use the assocaton strength nformaton between the KWs and RTs. To calculate the strength of an RT wth respect to the search topc, the asymmetrc dce coeffcent s used: df ( RT KW ) S( RT ) = df ( RT ) KW Query The RTs are raned by ths strength n descendng order. The best RTs are then chosen for expanson. By varyng from 5, 10, 20, 30, to 40, our experments showed that the best performance was acheved when =5. S4: In another attempt to now whether the RTs are relevant to the search topc, we mae another document search by each RT and chec whether ttles of the top-raned documents contan the orgnal KWs. If yes, the RT s ncluded for expanson. Agan, by varyng from 5, 10, 15, to 20, the best performance was observed when =15. Fnally, to have an dea of how these RTs help n NTCIR s topc-based retreval tas, human udgment were conducted n comparson wth the above automatc methods. From the Chnese SLIR n NTCIR-3, over 200 eywords were extracted from the descrptons of the 42 topcs. Based on these eywords, a total of 8144 RTs were extracted from the thesaurus. Each of these 8144 RTs was then udged manually relevant or not to the search topcs based on the topc descrpton feld. However, our prelmnary experment showed that the manual selected RTs from ths udgment dd not mprove performance sgnfcant as expected. A second run of udgment was conducted. Ths tme the RTs have to appear n the relevant documents of the search topcs so as to be udged relevant or not. By lmtng the RTs n ths way, the udgment of these RTs can be more topc-specfc and thus accurate. 3. Results for the NTCIR-3 Collecton Based on the Chnese SLIR collecton of NTCIR-3, a number of experments were conducted wth varous strateges (wth or wthout global and local expanson) under dfferent retreval models. The used retreval models are BS (byte sze normalzaton), pvoted normalzaton method, BM11, BM25, and BM25m (modfed BM25). The BS method s an approxmaton of the cosne method as follows [13]: T d = 1,, BS( d, ) = T 2 ( byteszed ) = 1, where the bytesze denotes the number of bytes of a document. The document term weght d n the above s calculated by the term freuency,.e., (1+). The uery term weght, s calculated by the term freuency and the nverse document freuency,.e., (1+) x (1+n/df), where n s the collecton sze. BS s the fast method among these 5 retreval methods and was used n our NTCIR-3 and NTCIR-4 experments. The pvoted method s an mprovement of the BS method proposed by Snghal et al [14]: T n (1 + ( )) Pvot( d = ) = 1, df dl (1 s) + s Avgdl where T s the number of uery terms, dl s the document length of document Avgdl s the average

4 Proceedngs of NTCIR-5 Worshop Meetng, December 6-9, 2005, Toyo, Japan document length n the collecton, and s s a parameter, whch s 0.2 n our experment. The BM11 and BM25 are probablstc retreval models that compute the oap weght of a document wth respect to a uery: T n df BM11( d, ) = = 1, df + dl Avgdl = T ( 3 + 1), n df ( 1 + 1) BM 25( d ) = 1 3 +, df dl + ((1 ) + ) 1 b b Avgdl where 1, 3 and b are parameters and are set as 1 =1.2, 3 =1000, and b=0.75 n our experment. BM25 often outperforms BM11 for Englsh documents. However, the (n-df+0.5) term may lead to unexpected effect for hgh-freuency terms, as was analyzed by Fang et al [15]. Therefore, they proposed a modfed BM25 method by elmnatng ths effect as follows: = T ( 3 + 1), n ( 1 + 1) BM 25m( d, ) = dl 3, df + ((1 ) + ) 1 b b Avgdl As to the local expanson, thrty best terms from sx top-raned documents retreved by the ntal uery were used. These sx documents were frst concatenated nto one text strng and then the eyword extracton algorthm [11] was appled to extract maxmally repeated patterns. The extracted terms were sorted n decreasng order of occurrence. The frst 30 terms were then selected for local uery expanson. The decson on the number of best terms and the number of top-raned documents was ute arbtrarly. We chose these numbers from the begnnng almost wthout any tunng. Table 1. Performance of varous expanson strateges under dfferent retreval models for the Chnese SLIR descrpton run. BS Pvot BM11 BM25 BM25m Basc L S S S S H H S3+L H2+L Max of C-C-D n NTCIR Avg of C-C-D n NTCIR Table 1 shows our experment results for the Chnese SLIR tas n NTCIR-3. The uery strng s from the descrpton feld of the search topc. The Basc row denotes a baselne result wthout any expanson. The L row denotes the result from local expanson. S1 to S4 are those from global expanson strateges S1 to S4, respectvely. H1 and H2 are those results obtaned by addng the global terms udged relevant manually. H1 denotes our frst udgment attempt. H2 denotes the second, as s explaned n the above. As to the last two rows, they denote the combned strateges by usng the global term expanson frst, followed by the local expanson. As Table 1 shows, all these four strateges perform worse than the smple local expanson. However, f proper terms can be selected from the suggested RTs, the performance can be better than the local feedbac, as shown by the results of H2. Furthermore, combnng both local and global expanson can outperform each method alone. Ths shows that the automatcally generated thesaurus dd extract relevant global terms for expanson. It s our automatc methods that fal to select good enough terms, despte we have tred out four dfferent strateges. Also note that the results of H1 are better than the local expanson for the BS and Pvot methods; but they are slghtly worse for all the probablstc models. Ths mples that even wth the help of a human n selectng the expanson terms n an nteractve search mode, the performance may not be better than a fully automatc feedbac method, f t s eupped wth a hgh-performng retreval model. Only when the human fully understands what he/she needs (le the udgment n H2), can he/she outperform a machne-drven feedbac method. Another observaton from Table 1 s that BM25 dd perform better than the others. However, the pvoted normalzaton method dd not perform well as expected. Ths may due to the mproper parameter settng for ths collecton. But t also reveals ts senstvty n parameter tunng for obtanng a good result. 4. Results for the NTCIR-5 Collectons In NTCIR-5, the document collectons are new and larger than those n NTCIR-3 and -4. The uery topcs prepared by NTCIR-5 consst of ttle, descrpton, narratve, and concept felds. A total of about 50 topcs for SLIR were provded for retreval. Mean average precson (MAP) for these topcs was calculated by the well-nown trec_eval algorthm. Partcpants can submt multple results, each comes from the run that uses dfferent felds of uery topcs and/or dfferent retreval strateges to see how MAP

5 Proceedngs of NTCIR-5 Worshop Meetng, December 6-9, 2005, Toyo, Japan changes. Each submtted run s evaluated n two crtera, one s relax, meanng that the relevance udgment s done n a less strct way; the other s rgd, meanng that the relevance s udged n a more rgorous sense. Our results for the SLIR (Chnese, Japanese, and Korean) were shown n Table 2. In the RunID, the letter T denotes the run that submts the ttles as ueres, D denotes the descrptons, and C the concepts. The used retreval models are shown n the parentheses. All the runs apply the global expanson strategy S3 followed by the local expanson method. Table 2. Performance of dfferent runs. RunID Rgd Relax C-C-T (BS) C-C-T (BM11) C-C-D (BS) C-C-D (BM11) C-C-C (BM11) Max of C-C-T Avg of C-C-T Max of C-C-D Avg of C-C-D J-J-T (BS) J-J-T (BM11) J-J-D (BS) Max of J-J-T Avg of J-J-T Max of J-J-D Avg of J-J-D K-K-T (BS) K-K-T (BM11) K-K-D (BS) K-K-D (BM11) K-K-C (BM11) Max of K-K-T Avg of K-K-T Max of K-K-D Avg of K-K-D As can be seen from Table 2, the probablstc retreval model performs slghtly better than the vector space model regardless of long or short ueres for the Chnese SLIR. But t performs worse for the Japanese and Korean SLIR. The reason may be due to the poor ndexng schemes for the Japanese and Korean documents, for whch our term ndexng method do no use any vocabulary nowledge from these two languages, as was explaned n our prevous report [1]. 5. Conclusons We have tred several global expanson strateges based on an automatcally generated thesaurus. Although the thesaurus does capture the relevant global terms for expanson, as shown n our human selecton experment, our automatc selecton strateges do not come out wth desred results. The dffcultes le n that among the 8144 related terms n the NTCIR-3 Chnese topcs, only 214 are hghly relevant to these topcs, a rato of only 2.63% (4252 are vaguely relevant and the remanng 3678 are rrelevant). The strategy S3 has a recall of 70/214= and a precson of 70/(70+104)= , whle H2 has a recall of 162/214= and a precson of 162/( )= Detaled analyss of these strateges was dscussed n [16]. Generally speang, the contexts of the related terms are mportant n udgng ther relevance. But these contexts are not easly found for a specfc search topc, especally when the nformaton need s constraned by some lmtatons n the narratve feld of a search topc. Nevertheless, we plan to explore more expanson strateges n the future to better explot the co-occurrence thesaurus generated by the machne. From ths year s results and our past experences, we conclude that the factors that affect the retreval effectveness are the ualty of uery terms, the term ndexng schemes, the use of uery expanson, retreval models, and term weghtng methods. Future wor wll consder these factors together to obtan better retreval performance. Acnowledge Ths wor s supported n part by NSC under the grant number NSC E References [1] Da-We Juang and Yuen-Hsen Tseng, "Unform Indexng and Retreval Scheme for Chnese, Japanese, and Korean," Proceedngs of the Thrd NTCIR Worshop on Evaluaton of Informaton Retreval, Automatc Text Summarzaton and Queston Answerng, Oct. 8-10, 2002, Toyo, Japan, pp [2] K. L. Kwo, NTCIR-3 Chnese, Cross Language Retreval Experments Usng PIRCS, Proceedngs of the Thrd NTCIR Worshop on Evaluaton of Informaton Retreval, Automatc Text Summarzaton and Queston Answerng, Oct. 8-10, 2002, Toyo, Japan, pp [3] Masa Murata, Qng Ma, and Htosh Isahara, Applyng Multple Characterstcs and Technues to Obtan Hgh Levels of Performance n

6 Proceedngs of NTCIR-5 Worshop Meetng, December 6-9, 2005, Toyo, Japan Informaton Retreval, Proceedngs of the Thrd NTCIR Worshop on Evaluaton of Informaton Retreval, Automatc Text Summarzaton and Queston Answerng, Oct. 8-10, 2002, Toyo, Japan, pp [4] Robert W. P. Lu, K. F. Wong, and K. L. Kwo, Dfferent Retreval Models and Hybrd Term Indexng, Proceedngs of the Thrd NTCIR Worshop on Evaluaton of Informaton Retreval, Automatc Text Summarzaton and Queston Answerng, Oct. 8-10, 2002, Toyo, Japan, pp [5] Yuen-Hsen Tseng, Da-We Juang and, Shu-Han Chen "Global and Local Term Expanson for Text Retreval," Proceedngs of the Fourth NTCIR Worshop on Evaluaton of Informaton Retreval, Automatc Text Summarzaton and Queston Answerng, June 2-4, 2004, Toyo, Japan. [6] Chen, H.-H., Ln, C.-C., and Ln, W.-C., "Constructon of a Chnese Englsh wordnet and ts applcaton to CLIR," In Proceedngs of the ffth Internatonal Worshop on nformaton retreval wth Asan languages, pp , [7] Smeaton, A.F., & Berrut, C., "Thresholdng postngs lsts, uery expanson by word word dstances and POS taggng of Spansh text. In Proceedngs of the fourth text retreval conferences., 1996 [8] Voorhees, E.M., "Query expanson usng lexcal-semantc relatons," Proceedngs of the 17th ACM SIGIR conference on research and development n nformaton retreval, pp , [9] Fox, E.A., "Lexcal relatons enhancng effectveness of nformaton retreval systems," SIGIR Forum, 15(3), 6 36, [10] Mandala, R., Tounaga, T., & Tanaa, H., "Combnng multple evdence from dfferent types of thesaurus for uery expanson," In Proceedngs of the 22nd ACM SIGIR conference on research and development n nformaton retreval, pp , [11] Yuen-Hsen Tseng, "Automatc Thesaurus Generaton for Chnese Documents", Journal of the Amercan Socety for Informaton Scence and Technoy, Vol. 53, No. 13, Nov. 2002, pp [12] Ja-Yun Ye, Evaluaton of Term Suggeston n an Interactve Chnese Retreval System, Master Thess, Department of Lbrary and Informaton Scence, Fu Jen Catholc Unversty, (n Chnese) [13] Amt Snghal, Gerard Salton, and Chrs Bucley, "Length Normalzaton n Degraded Text Collectons," Proceedngs of Ffth Annual Symposum on Document Analyss and Informaton Retreval, Aprl 15-17, 1996, pp [14] Amt Snghal, Chrs Bucley, and Mandar Mtra, Pvoted Document Length Normalzaton, Proceedngs of the 19th annual nternatonal ACM SIGIR conference on Research and development n nformaton retreval, Zurch, Swtzerland, August 1996, Pages: [15] Hu Fang, Tao Tao, and ChengXang Zha A Formal Study of Informaton Retreval Heurstcs, Proceedngs of the 27th Internatonal ACM SIGIR Conference on Research and Development n Informaton Retreval, July Sheffeld, U.K., 2004, pp [16] Yu-Chn Tsa Term-Selecton for Query Expanson n Topc-based Informaton Retreval, Master Thess, Department of Lbrary and Informaton Scence, Fu Jen Catholc Unversty, (n Chnese).