Incident Threading for News Passages

Transcription

1 Incdent Threadng for News Passages Ao Feng Amazon.com 705 5th Ave S. Seattle, WA 98104, USA James Allan Center for Intellgent Informaton Retreval Department of Computer Scence Unversty of Massachusetts Amherst Amherst, MA 01003, USA ABSTRACT Wth an overwhelmng volume of news reports currently avalable, there s an ncreasng need for automatc technques to analyze and present news to a general reader n a meanngful and effcent manner. We explore ncdent threadng as a possble soluton to ths problem. All text that descrbes the occurrence of a real-world happenng s merged nto a news ncdent, and ncdents are organzed n a network wth dependences of predefned types. Earler attempts at ths problem have assumed that a news story covers a sngle topc. We move beyond that lmtaton to ntroduce passage threadng, whch processes news at the passage level. Frst we develop a new testbed for ths research and extend the evaluaton methods to address new granularty ssues. Then a three-stage algorthm s descrbed that dentfes on-subect passages, groups them nto ncdents, and establshes lnks between related ncdents. Fnally, we observe sgnfcant mprovement over earler work when we optmze the harmonc mean of the approprate evaluaton measures. The resultng performance exceeds the level that a calbraton study shows s necessary to support a readng comprehenson task. Categores and Subect Descrptors H.3.3 [Informaton Search and Retreval]: Clusterng; H.3.4 [Systems and Software]: Informaton Networks General Terms Algorthms, Desgn, Expermentaton, Human Factors, Languages, Measurement, Performance Keywords Informaton overload, Automatc news organzaton, Incdent threadng, Passage threadng 1. INTRODUCTION Assocated wth the fast development of modern technologes, the amount of accessble nformaton s ncreasng n an exponental manner [15]. Every day there s a large amount of new nformaton avalable to us, and a maor part s news. News comes Permsson to make dgtal or hard copes of all or part of ths work for personal or classroom use s granted wthout fee provded that copes are not made or dstrbuted for proft or commercal advantage and that copes bear ths notce and the full ctaton on the frst page. To copy otherwse, or republsh, to post on servers or to redstrbute to lsts, requres pror specfc permsson and/or a fee. CIKM 09, November 2 6, 2009, Hong Kong, Chna. Copyrght 2009 ACM /09/11...$ from many dfferent sources, ncludng tradtonal meda such as newspaper, rado or TV, and modern sources lke the Web. Wthout proper arrangement of the nformaton, one can easly become lost because of ts vast sze. Ths phenomenon s called nformaton overload. For an nformaton acquston task, there are tools avalable to help the web users: search engnes provde general knowledge related to a query; queston answerng systems help a user fnd drect answers to hs/her queston; onlne forums and mal lsts offer addtonal communty-based support through human-tohuman nteracton. Nevertheless, news remans an area that has not been fully explored. Many webstes publsh a large amount of news, and some provde categorzaton nformaton and/or search functons. Unfortunately, the problem of servng nterestng news to a user remans unresolved. It s nfeasble for a user to sort through all avalable news nformaton wthout any pre-processng, because the news a person can read n a certan tme s much less than the amount that s generated wthn the same perod. To help the user obtan nterestng nformaton wth the smallest cost, we desre a system that automatcally processes news and converts t nto a more user-effcent format. Dfferent people have ther own ways of comprehendng news nformaton, but there are some common rules that most would follow. For an automatc system to facltate users effectvely n ther readng process, t s recommended that ths system have smlar abltes. Each user has hs/her nformaton need. For example, a resdent of New York Cty mght be nterested n a crme that happened n the Cty, but may not care f there s a mltary conflct n Kosovo. A good system should group news accordng to the man topc dscussed. People remember nterestng nformaton for a long tme, and care about new messages rather than repettons, even f the repeated nformaton s descrbed n a dfferent vocabulary. It s not advsable that the system provde duplcate nformaton. Snce human bengs have reasonng abltes, they do not treat news events as solated facts. Hypothetcally, they would compare new nformaton to memory and assocate t wth exstng nformaton that s correlated. It would be preferable f the system takes smlar actons to lnk related (but not duplcate) events, because people are very lkely to be nterested n both (or nether). Fgure 1 shows summares of four news reports from CNN (the text below a box s the document dentfer). As we can see, three

2 of them are from the same news topc ( Pope vsts Cuba ) and the last one s about the Monca Lewnsky scandal case. An deal news organzaton, as shown n Fgure 1, should place the three related reports together and dsplay ther contextual lnks, leavng the rrelevant nformaton asde. Castro urges Cubans to welcome the Pope CNN Pope arrves n Cuba on Tuesday CNN Starr nvestgates whether Clnton urges Lewnsky to le CNN Pope celebrates mass n Santago de Cuba CNN Fgure 1: Sample News Organzaton Our goals algn wth the dea of ncdent threadng as proposed by Nallapat et al [14] and contnued by Feng and Allan [8]. Those papers lad the foundaton but were dsadvantaged by makng a clearly ncorrect (and acknowledged) assumpton that a story talks about a sngle ncdent. In ths paper we extend ther story threadng work to produce passage threadng. In the next secton, we revst the nfrastructure of ncdent threadng n the scenaro of passage-based news analyss. Secton 3 ntroduces the motvatons behnd ncdent threadng, llustrates ts two earler mplementatons, and shows the possblty of extensons. Secton 4 descrbes the nnovatve framework of passage threadng, whch conducts news analyss at the passage level. Experments n Chapter 5 show the performance mprovement of a three-stage algorthm over the earler work. Chapter 6 summarzes contrbutons of the paper and proposes potental research topcs for the future. 2. INCIDENT THREADING The dea of ncdent threadng was manly motvated by Topc Detecton and Trackng (TDT) [1], n whch news stores are assgned to ndvdual news topcs. Each topc ncludes a semnal event and all drectly related events. However, the dscusson of how these events are organzed s not the man concern of TDT. To obtan a clearer vew of the news evoluton, t s necessary to go beyond topcs and dve deeper nto ther nternal structure [14]. As related news s usually connected by semantc contextual nformaton, t s desrable to establsh a fact network, where each vertex represents an ndvdual news event, and an edge shows the connecton between the two events that t ons. That s the basc dea of ncdent threadng. 2.1 Incdent We defne the basc concepts before dscussng the detals of ncdent threadng. 1. News story: Any news n text format s dssemnated n unts, and each of them s a news story. A story has a unque ID and a seres of characters contanng ts content. A news story usually descrbes one or more real-world occurrences (events). 2. Man characters (WHO): The most mportant named enttes that show who or what s nvolved n the descrpton of an event. 3. Tme stamps (WHEN): Two tme features are consdered for a news report. One s called publcaton tme, whch s when the news s released. The other s actvty tme that contans the tme stamp of the descrbed occurrence, whch may be a tme pont or a perod. 4. Locaton (WHERE): It descrbes where the occurrence happened, whch s usually an absolute geographcal poston or a relatve reference towards another locaton. 5. Acton (WHAT): The key verb(s) n the descrpton of the event. Sometmes t may also be a noun. Wth a clear noton of these concepts, what an ncdent s can be naturally defned. Defnton 1a: An ncdent 1 s somethng that happens n the real world. It nvolves certan man characters, occurs at a defnte tme or durng a perod, happens at a geographcal locaton, and ncludes a specfc acton. Defnton 1b: An ncdent also refers to all news snppets that descrbe the same real-world occurrence, despte the vocabulary, language or medum of the report. Bascally, an ncdent s a real-world occurrence, whch nvolves some named enttes, and happens at a specfc tme and locaton. It can also be used to descrbe the unon of text that contans the same (or smlar) features (who, when, where, what) and descrbes the same epsode. 2.2 Incdent Network In order to accurately model the contextual relaton among ncdents, we need to specfy a lmted vocabulary of the possble relaton types. Dscourse analyss [5, 20] provdes a framework that reflects the structure of a news report, and some concepts n t can also be appled to defnng the relaton between two ncdents. Surprse attack 7/12/2006 Reacton Smlar rad by Palestnan gunmen 6/25/2006 Reacton Comment Israel bombs Palestne Israel sends Troops to Lebanon 7/12/2006 Reacton Israel refuses negotaton 7/12/2006 Ar strke on Lebanon by Israel army 7/12/2006 Consequence Hamas and Hezbollah request prsoner exchange Ar and sea blockade on Lebanon Reacton 55 Cvlans klled, >200 wounded Rockets fred nto Israel Consequence Fgure 2: Sample Incdent Network Defnton 2: An ncdent network s one or more ncdents connected by edges that represent certan types of contextual dependency. 1 The concept event s often used n Informaton Extracton (IE) where t has a dfferent meanng [11], so consstent wth earler work [8] we replace t wth ncdent to avod confusng. An event n IE s an actvty descrbed by a sentence that nvolves zero or more enttes. The event extracton task s usually lmted to certan types of events (e.g., conflcts) and ts focus s on the accurate dentfcaton of ther arguments. Descrptons of the same semantc content at dfferent places are often handled separately.

3 Defnton 3: Incdent threadng s the process of dentfyng ncdents n a news stream and generatng an ncdent network. Fgure 2 shows an ncdent network that represents some news reports about an Israel-Lebanon conflct. The text next to each edge s the relaton type of the correspondng lnk, and many of them are borrowed from a news schema n dscourse analyss [21]. There are three man classes of connectons n an ncdent network. Here they are descrbed startng from the strongest type of relatons. The frst class s logcal relatons. A connecton of ths type s establshed between two ncdents that have logcal causal relatons,.e., the occurrence of one ncdent drectly causes the other to happen. It s represented by a drected edge n the ncdent network, whch goes from the logcal premse to the result or consequence. Ths class ncludes Predcton, Comment, Reacton, Analyss, Background, and Consequence. The second class, here called progressons, requres weaker lnks than the prevous class. In TDT, they are usually two ncdents n the same topc, and one happens after another. However, one ncdent may not necessarly lead to the occurrence of the other (the lnk may not be causal). The only relaton type n ths class s named follow-up, and the sequence s decded by the tme order of the ncdents. Lnks n ths class are shown as drected edges, pontng from the earler ncdent to the later one. The thrd class s called weak relatons. From the usual perspectve, two ncdents wth a lnk n ths class do not have any drect relaton, except that they menton somethng n common. The overlappng factor may be the same man character, the same geographcal locaton, the same type of occurrence, etc. As there s usually no prorty defned by the common feature, lnks n ths class are represented by undrected edges. The frst two classes are strong relatons, because they often connect drectly-related ncdents to form a topc, as defned n TDT. Lnks n the last class usually go between topcs, but they are as valuable snce they connect dfferent topcs wth these common factors to form a global ncdent network. Ths feature s especally useful to lead the user to a new topc that cannot be found otherwse. 3. PREVIOUS WORK As we have mentoned, the dea of ncdent threadng s motvated mostly by TDT and dscourse analyss. TDT montors a news stream and places the stores nto ndvdual topcs, where each topc ncludes all the news events that are closely related. In addton to the effort of automatc news organzaton, dscourse analyss studes the nformaton flow n a press artcle. To some extent, dscourse analyss s the parallel work of ncdent threadng n another area, but the vast nvolvement of human bengs greatly lmts ts applcaton to large corpora. In ths secton, we also show some key decsons n the mplementaton of ncdent threadng. Dependng on the choces made, there can be varous systems based on the ncdent threadng framework. We brefly ntroduce two earler mplementatons here: story threadng and relaton-orented story threadng. Our passagebased model wll be descrbed n the next secton. 3.1 TDT TDT s a research program that focuses on event-based news organzaton. It breaks an ncomng news stream nto a lst of topcs, and each topc s a set of news stores that are strongly related by some semnal real-world event. [2] As t nvolves subectve understandng of news, whch may dffer by human beng, great dffculty s expected when the process s replcated n every detal. Several assumptons are made n TDT to reduce the complexty n ts mplementaton. Topcs do not overlap. Topcs are ndependent. The nternal structure of a topc does not affect evaluaton results. Startng from the plot study n 1997 [3], there were a total of eght evaluatons up to TDT-2004 [9]. The concept topc s emprcally defned wth detaled nstructons, and reasonable accuracy has been observed when buldng topcs from a contnuous news stream. However, the TDT framework does not provde a clear vew how a news topc s formed, plus the nonoverlappng and ndependence assumptons of topcs are often challenged. 3.2 Dscourse Analyss As a TDT topc s defned as a semnal event together wth all related events, a natural response would be an attempt to fnd these ndvdual events and ndcate the relatons among them. However, the descrpton of a relaton s subectve. A lmted vocabulary of connecton types and a detaled descrpton (deally a defnton) of each are necessary to avod the possble confuson. In the Informaton Retreval (IR) communty, we are unaware of any prevous attempt before ncdent threadng, but dscourse analyss n ournalsm deals wth smlar problems [5, 20, 21]. Dscourse analyss s a general term that ncludes many approaches to analyzng the use of languages, and one mportant applcaton of t s on news. Wthn the news doman, dscourse analyss deals wth the formaton of a complete news report (manly for news n the press), whle broadcast news s usually released n shorter peces and the context s often assumed to be avalable for the audence. However, models n dscourse analyss may also work for broadcast news, f each pece s regarded as a part of a press artcle. 3.3 Implementatons of Incdent Threadng In an ncdent threadng system, there are two mportant decsons to make. The frst s the selecton of the basc text unt. A news story usually contans a lot of semantc nformaton, whch makes t easer to understand, but n many cases a story mentons multple real-world occurrences. In contrast, a passage s shorter and often requres contextual nformaton to comprehend ts content entrely, but has better semantc coheson. The second choce s on the contextual lnks. It s relatvely smpler to determne f a relaton exsts between two ncdents, but markng ther lnk type may be a hghly subectve task. We can ether go wth bnary lnks, whch are easer to annotate and mplement, or requre the relaton type to be explctly marked for each lnk. Wth dfferent answers to the two questons above, there can be four combnatons n the system mplementaton. Story threadng [14] selects a news story as the basc semantc unt and gnores

4 lnk types. When type nformaton s consdered, t becomes relaton-orented story threadng [8]. Passage threadng (Secton 4) analyzes news at a smaller granularty (passages nstead of stores), and lmts the range of news to a specfc subect (volent actons n the experment). Under that scenaro, the vocabulary of relatons s lmted, so we choose to gnore the lnk types for now. We have also tred passage-level ncdent analyss of general news for rcher relatons. Unfortunately, the poor nter-annotator agreement suggests t s nsuffcently understood to be tackled Story Threadng As the earlest attempt to organze news at the ncdent level, story threadng [14] tres to capture the news ncdents wthn a TDT topc and the organzaton among them. Incdents n the same topc are shown n a Drected Acyclc Graph (DAG). An edge from ncdent A to ncdent B means that there s some correlaton (or dependency) between them, ether logcal (A causes B to happen) or progressonal (A precedes B n tme). However, the logcal and progressonal relatons are nontrval to dstngush, and a clear boundary s not establshed between them n ths work. Fgure 3 dsplays the deal ncdent model n ths framework for the data n Fgure 2. After the surprse attack of Hezbollah, Israel sends troops to Lebanon NYT_ENG_ Conflcts escalates as Israel bombs Southern Lebanon and Hezbollah fres rockets nto Israel NYT_ENG_ Foregn leaders call for nterventon of the Unted Natons NYT_ENG_ Fgure 3: Incdent Model n Story Threadng In the mplementaton of a story threadng system, there are manly two steps. Frst, all stores n the same topc are compared to each other, and smlar ones are merged nto a cluster. Each cluster at the end of the frst step corresponds to a news ncdent. In the second step, two ncdents wth hgh smlarty are lnked by an edge. The edge shows a precedng relaton, as t goes from the earler ncdent to the later one. In the experments [14], moderate accuracy can be acheved n story threadng wth smple algorthms and easy-to-extract features, but the smplfcatons n ths model leave ample space for further development Relaton-Orented Story Threadng Understandng the contextual nformaton n news reports seems straghtforward to a normal person, but desgnng a computer program wth the same capablty s dffcult. It requres abltes n natural language understandng and artfcal ntellgence that are stll beyond state-of-the-art research. Fortunately, certan relatons among ncdents often exst n analogous scenaros. For example, legal cases usually nvolve a crme, an nvestgaton, zero or more suspects, arrests, a tral, a verdct and a sentence. Furthermore, relatons among these parts are generally fxed. Schank and Abelson [18] fnd smlar phenomena n the understandng of human knowledge, and they create scrpts for scenaros n real lfe (e.g., restaurant scrpt 2 ). 2 The man steps n the restaurant scrpt nclude: customer enters restaurant, customer fnds seat, customer sts down, water/watress gets menu, etc. Here the term scrpt s borrowed from ther work and one scrpt s generated for each crcumstance, whch ncludes a lst of rules for possble lnk types under that scenaro. After defnng the lnk types, contextual nformaton can be represented more accurately n an ncdent network [8]. The ncdent network composed of the stores n the Israel-Lebanon conflct s shown n Fgure 4. After the surprse Conflcts escalates as Reacton attack of Hezbollah, Israel bombs Southern Foregn leaders call Israel sends troops to Lebanon and for nterventon of the Lebanon Hezbollah fres Unted Natons NYT_ENG_ rockets nto Israel NYT_ENG_ NYT_ENG_ Fgure 4: Incdent Model n Relaton-Orented Story Threadng To establsh the ncdent network, the same two-step process s used to create the ncdents and buld the lnks. In the second step, type-specfc rules are used to assgn the approprate type label to each lnk. Another method s to consder the possble relatons between any story par, and expand the par-wse competton process to a global optmzaton problem. For a collecton of n stores, an n*n relaton matrx s formed that defnes a global score functon, and smulated annealng s appled to fnd a global maxmum for t. From the expermental results [8], the revsed two-step algorthm has moderate performance n the ncdent formaton step, but creates lnks of low accuracy. In contrast, global optmzaton usually returns clusters of slghtly lower qualty, but the lnk performance s much hgher, especally for the assgnment of relaton types. Overall, global optmzaton s regarded as more approprate for the applcaton snce lnks are very mportant n formng the structure of the ncdent network, but the hgh computatonal complexty restrcts ts applcaton. The man dsadvantage of the two-stage algorthm, as shown by falure analyss, s that t cannot correct clusterng errors n later steps. That observaton also makes clusterng the performance bottleneck. 4. PASSAGE THREADING From the begnnng of TDT, a one-event-per-story assumpton has been consstently appled. That assumpton allows earler research to provde useful nsghts n automatc news analyss. However, t s clearly not always true, despte ts effectveness n reducng the complexty of the problem. Sometmes t also ntroduces unsolvable problems wthout removng that assumpton frst. For the example n Fgure 2, a sngle news story NYT_ENG_ contans multple ncdents n the network: Surprse attack on Israel troops. Israel sends troops to Lebanon. Hamas and Hezbollah request prsoner exchange. Israel refuses peace negotaton.

5 If we treat the story as a whole, the semantcs relaton among these ncdents cannot be modeled, as t s not allowed to have a relaton from one ncdent to tself. These lnks can only be approprately formed to establsh an ncdent network after the story s broken nto smaller peces. Usually a news story s composed of at least two or three paragraphs, each descrbng some detals of a certan happenng or related nformaton. When a user fnshes a complete story, t s assumed that suffcent context has been ncluded n the story and background nformaton s not essental (but stll benefcal) to understand ts content. In short, a news story s often a semantcally complete unt. However, t s not the case for passages 3. A passage s often short, composed of one or more sentences, and t descrbes a certan occurrence. For most cases, t s mpossble to understand a passage completely wthout the contextual nformaton from the full story. In an applcaton of passage-based news analyss, ths phenomenon drectly threatens performance, as the accurate dentfcaton of context usually requres semantc understandng of adacent or even remote passages. 4.1 Data Annotaton The frst obstacle encountered n passage-level news analyss s the avalablty of approprate data collectons. There are research topcs focusng on fner graned text snppets, ncludng passage retreval, fact fndng, novelty detecton, and other smlar areas. Some corpora are avalable for each of these applcatons, but none of them has provded rch enough annotaton that can be drectly appled to ncdent descrpton and contextual analyss. The top prorty to prepare for an mplementaton s to buld a data collecton suffcently annotated wth relable relevance udgments, so that the output from algorthms can be compared to the ground truth and an evaluaton score wll be assgned to measure each algorthm s performance. A large proporton of the news corpora avalable to us are well formatted, at least for newswre reports, n whch paragraph and sentence margns are avalable. To avod unnecessary nose ntroduced by segmentaton errors, we elect to use the exstng text boundares nstead of applyng segmentaton algorthms [4, 12]. For the choce between sentences and paragraphs, we opt to treat each paragraph as an ndependent semantc entty and the basc unt n annotaton, because t contans more nformaton to help understand the content. Wth ths selecton, the whole annotaton process and all evaluaton measures treat paragraphs and passages as equvalent. There are cases where one paragraph dscusses multple ncdents, but consderng them makes t very dffcult to acheve good nter-annotator agreement. Therefore, we make an assumpton that each paragraph s an ndvsble semantc unt. We use part of the Global Autonomous Language Explotaton (GALE) [16] corpus n our data annotaton, n Englsh newswre reports only. In ths collecton, queres are ssued to collect nformaton on specfc subects, whch are slghtly dfferent but smlar to the tradtonal TDT topcs. Some of these queres are 3 A passage s a contnuous subset of a news story that contans a complete descrpton of certan news nformaton. It usually follows the natural paragraph or sentence boundares, but t s also possble that a passage spans multple paragraphs. general and collect all nformaton related to certan topcs or persons/organzatons. Others focus more on specal scenaros and further lmt the range wth query arguments. When general queres are submtted to the collecton, the top documents returned usually contan news reports n varous subects. Manual analyss shows that many of them are solated ncdents, and do not have any relaton wth other reports. An ncdent network generated from such data would be wdely dstrbuted on multple subects wthout a coherent theme, whch s not an effectve representaton of nterestng nformaton. As publc nterest usually focuses on a few specal subects or topcs, we beleve that shrnkng the scope of news nto a sngle type of nformaton wll help mprove the coherence between dfferent reports, thus makng the contextual analyss more meanngful. Research results n such a topc can be further expanded to other categores, makng t generally nterestng. In ths experment, we focus on queres that relate to strong or volent actvtes n news, whch s one of the man topcs n the GALE collecton wth broad nterest. The selected queres are matched aganst the ndex of Englsh newswre collectons, and the 10 most smlar documents are selected for annotaton. The purpose of startng wth queres s to provde a set of documents that are lkely to be on the same topc, as well as some nonrelevant documents to supply background nose. Usng an actual retreval step also makes the process more realstc: t s beng appled to the output of an nformaton retreval system, not a set of hand-selected documents. The annotaton process conssts of three steps. The frst step s for the annotator to walk through each paragraph and dentfy f t contans any descrpton of a volent acton. The second step s to mark the ndvdual volent actons and fnd co-references of the same actvty. The last step scans an ncomplete lst of ncdent pars and the annotator s requred to determne f there s any logcal or progressonal relaton n each par, and mark the drecton f such a lnk exsts. Statstcs of the annotaton are dsplayed n Table 1. For each lne (except the number of queres), we show the total number of obects as well as the mnmum and maxmum n 17 queres. Table 1: Statstcs of Annotated Corpus n Passage Threadng Queres 17 Documents 170 (10 10) Passages 3,618 ( ) Volent passages 792 (10 93) Percentage of volent passages 21.8% ( %) Incdents 376 (4 45) Lnks 156 (0 47) As the annotaton s subectve n the last two steps, we calculate the nter-annotator agreement only for the frst step. Fless Kappa [10] among 4 annotators s 0.595, and Cohen s Kappa [7] for any two annotators s between Although there s no unversal defnton of a good Kappa value, these numbers show far agreement among annotators, and t would be safe to clam that the problem defnton s clear enough for the annotators to make ratonal choces. Ths also dsplays the advantage of annotatng a specfc topc nstead of a general area, as Fless Kappa n a smlar annotaton attempt for general ncdents s only 0.193, and Cohen s Kappa ranges from to

6 4.2 Evaluaton The evaluaton s manly composed of two parts, each correspondng to one porton of the mplementaton process. The frst part evaluates the clusterng step, whch measures how smlar the ncdents and the system-generated clusters are. The second focuses on lnks, manly on the overlap between the lnks that appear n the annotaton and those n the system output. However, evaluatons n these two are partally ndependent and cannot provde a sngle metrc for system comparson. For a good estmate of the overall performance, these ndvdual measures are combned to generate a total score. We evaluate the clusterng performance usng concentraton and purty scores. Suppose that an ncdent I ncludes p passages from the annotaton. There are n clusters n the system output, and the numbers of passages n each cluster that belong to ncdent I are p 1,p 2,,p n, respectvely. These numbers should add up to p. Concentraton for ncdent I s then defned as n p ( p 1) 1 Conc( I ) p( p 1) If the passages n an ncdent are evenly dvded nto two clusters, the concentraton score of the ncdent s approxmately 0.5. The score s 0 f every cluster contans 0 or 1 passage n the ncdent. The concentraton score can be calculated for all ncdents wth sze larger than 1, and an average of them s taken based on the sze. Concentraton Conc( I ) I Purty s defned n the other way that measures the dstrbuton of ncdents n a cluster. If there are q passages n a cluster C, and the number of passages n t that belong to the annotated m ncdents are q 1,q 2,,q m, respectvely (ther sum may not be equal to q, as there can be passages that do not belong to any ncdent. Refer to the frst step of the annotaton process), the purty score for cluster C s q ( q 1) 1 Pur( C) q( q 1) All purty scores of clusters wth sze larger than 1 are averaged to compute the overall purty. Purty m C I Pur( C ) C Concentraton and purty both evaluate the qualty of the system clusters. For the same clusterng algorthm, the parameter settng changes ts performance, but these two measures are usually negatvely correlated,.e., the ncrease of one often leads to the decrease of the other. A drect lnk evaluaton s non-trval as the clusters do not always match the ncdents exactly. Therefore, we take an alternatve approach by assumng that the lnks exst between passages nstead of ncdents or clusters. If there are s passages, an s*s matrx M s formed, and an element n t s Mean M 1 p p 1 p p 0 otherwse Wth the defnton above, two lnk matrces can be easly generated, one (MT) comes from the ground truth, and the other (MS) from the system output. Then the par-wse lnk precson and recall wll be defned as smple matrx calculatons, but they can also be expressed as when there s a lnk n MS, chance of fndng a lnk at the correspondng place n MT and vce versa. P lnk,, MS MT MS MS R lnk,, MS MT MT MT In the calculaton above, a specal case s gnored when the correspondng element s 1 n one matrx but -1 n the other. The proporton of arrows pontng to the wrong drecton can also be calculated wth these lnk matrces. Err lnk (1,, MS MS MT ) / 2 MT These evaluaton matrces are combned nto a sngle measure to facltate comparson. cluster Mean all 2 Mean Mean cluster cluster Mean + Mean Inspred by the matrx comparson method n the lnk evaluaton, more complex relatons can also be encoded n a matrx. Smlarly to the lnk matrx, a dstance matrx D can be defned as lnk lnk 0 p p 1 p p D 1 p p p o p The four dfferent relatons n the equaton above are, members of the same ncdent (cluster), lnk to, lnk from, and unrelated, respectvely. The value table of a score functon s shown below. Table 2: Value Table of Score Functon f(a,b) b a Then the score s added up throughout all locatons n the dstance matrx and normalzed to lmt the fnal value wthn the [0, 1] SQ _ Sm( DT, DS) range. 2 concentraton purty concentraton + purty,, max( f ( DT, DT ), f ( DS, DS )) f ( DT, DS ) Mean lnk 2 Plnk R P + R lnk lnk (1) lnk (1 Err lnk (2) )

7 If DT and DS are the same, the smlarty score s 1. Dependng on the percentage of elements n DS and DT that are dentcal, ths evaluaton measure can be n the range [0, 1]. 4.3 Calbraton Study In prelmnary experments, performance evaluated by Equaton 2 ranges from 0% to 70%, dependng on the algorthm and the dffculty of the query. However, t s an open queston when we can clam a system to be good enough. In order to explore ts utlty n a real applcaton, the ncdent threadng framework needs to go through a calbraton experment to show the performance level at whch t proves to work. The calbraton study s desgned n the followng way. Gven an exstng query, a certan number (10) of top-ranked documents are collected. These documents are processed through an ncdent threadng system, and the system outputs an ncdent network. Then an annotator s served one verson of the data, ether the orgnal documents or an ncdent network, together wth a lst of questons that are drectly related to the orgnal query and based on the content of the documents. In a lmted tme (5 mnutes n ths experment), the annotator skms the nformaton he/she has, and tres to fnd as many answers as possble. In order to fnd a precse obectve for the performance level, multple versons of the ncdent network are suppled. The orgnal documents have no varance, but the ncdent networks can nclude dfferent proportons of nose, whch changes ther performance n the evaluaton. Multple annotators are requred for the study. Each of them receves one verson, ether the orgnal documents (ordered by tme, statstcs of the documents are n Table 1) or an ncdent network (as an mage) at a certan performance level. Ther results are checked aganst the standard answers, and a score s assgned to each. There are certan restrctons n the process of queston desgn and readng comprehenson, so that the comparson can be far across annotators. The questons are mostly fact fndng, where the answer can be found wthn a sngle passage. The questons are approxmately evenly dstrbuted n the documents. The order of questons s rearranged so that t does not follow the order they appear n the documents. Search n the source documents s prohbted. Readng the questons before startng the tmer s allowed and encouraged. Table 3 shows the result of the calbraton study for three queres. The performance level of each verson s the number of questons correctly answered by the annotator, and the ncdent networks also provde the matrx smlarty scores (Equaton 2). Because these compressed networks do not nclude all nformaton n the orgnal documents, an upper bound s lsted for each of them, whch means how many questons can be answered gven unlmted tme (Note that a hgher score does not necessarly mean better coverage of the questons, whch account for only a small porton of the source documents). Items n an talc font are ncdent networks that perform worse than the baselne (orgnal documents) n the study, and the underlned ones are better than the baselne. Table 3: Result of Calbraton Study by Query Docs Network 1 Network 2 Network 3 Network 4 1 4/10 6/7(21%) 1/5(25%) 5/6(28%) 6/6(32%) 2 3/10 3/4(19%) 2/7(26%) 5/7(30%) 5/8(37%) 3 2/10 2/3(19%) 3/5(24%) 4/6(26%) 6/6(34%) As personal dfference always exsts among human bengs, some annotators are faster than others n readng. Therefore, t s not always the case that one person does better than another when gven a better representaton. Nevertheless, the pattern s clear n the table, as ncdent networks start to perform better than the orgnal documents n the 25-30% range of matrx smlarty. Out of the 10 cells n Table 3 that have a smlarty score of at least 20%, 8 of them show more correct answers than the baselne, even when the upper bound s lower (5-8 n comparson to 10). 5. Experments In ths secton, results from two systems are compared usng the evaluaton measures descrbed n Secton 4.2. The baselne algorthm s borrowed from story threadng [14], wth passages as the basc semantc unts nstead of news stores. The other method mtates the annotaton process n Secton 4.1, and establshes an ncdent network n three steps. 5.1 Baselne The baselne algorthm starts wth an agglomeratve process, wth each passage formng a sngleton cluster. A passage s represented by a tf df vector, where tf s calculated wth Okap [17], and df uses the Inquery normalzed formula [6]. In each round, the most smlar cluster par, evaluated by average lnk, s merged. Ths process contnues untl all pars have smlarty below a predefned threshold. The fnal clusters become ncdents. Lnks are generated among the ncdents, based on ther smlarty. If the average smlarty between two ncdents s over the threadng threshold (lower than the one used n clusterng), a drected arrow s formed that ponts from the earler ncdent to the later one, where the order s determned by the tme stamp of the earlest passage n each. If they have the same tme stamp, the one that appears earler n the news stream takes precedence. The lnks created do not have type nformaton, as the smlarty between two ncdents does not provde suffcent semantc nformaton to assgn a relaton type. Smlarly, the three-stage algorthm descrbed below does not generate lnk types ether. 5.2 Three-Stage Algorthm In the data annotaton phase, each query needs to go through three steps. The frst step tells f there exsts any volent acton n the current paragraph; the second annotates these actons n detal and shows ther co-reference; the last step creates lnks between the key ncdents and all others. Lkewse, we mplement a threestage algorthm to reproduce ths process Passage Classfcaton Frst, a bnary classfer s traned to separate volent passages from others. We tred SVM [22], MaxEnt [13] and BoosTexter [19] n ths step usng the followng features: Number of terms n the passage Number of terms that appear n the man characters

8 Number of terms descrbng locatons Number of terms n the tme stamps Percentage of acton verbs that descrbe volence-related events Percentage of terms for all varances of be, do, have and say Percentage of terms that express certan extent of uncertanty, e.g., lkely, may, can, often, sometmes, etc. Combnatons of the three features above. The full text of the passage. It s avalable only to BoosTexter, as the other two do not accept text features. The performance of the three classfers n a leave-one (query)-out cross valdaton s shown n Table 4. The man advantage of BoosTexter s that t takes plan text features. Table 4: Performance Comparson of Three Bnary Classfers Algorthm Text feature Average error rate P-value n t-test BoosTexter Yes 14.43% - SVM lght No 17.47% * MaxEnt No 19.60% * Incdent Formaton The second stage runs a clusterng algorthm on the volent passages. Although passages are shorter than news stores, snppets that belong to the same ncdent must have some overlap, ether n terms or n semantcs. The overlap may be dentcal terms used n both passages, a reference to the same person or organzaton, or a menton of the same geographcal locaton, etc. These are the features used n the clusterng process: Smlarty of all terms Smlarty of man characters Smlarty of geographcal locatons Match between tme stamps. As many passages are mssng tme stamps, we combne ths feature wth the term smlarty by takng the product of them. In the earler work that utlzes multple features [8], a weghted sum of smlartes from varous elements s calculated to determne the resemblance between two stores. Here a strcter requrement s enforced that matches n all attrbutes must be acheved for two passages to be declared smlar. An agglomeraton algorthm s appled to form the ncdents Lnkng Incdents Analyss shows that most contextual lnks n the volence subect belong ether to consequence or reacton n the logcal category, or follow-up n the progressonal form. Lnks n these types usually contan two ncdents, whch happen at dfferent yet close tme, nvolve the same geographcal locaton or locatons near each other, menton smlar man characters, but often show poor term overlap. We stll use the same features n the prevous step, and one threshold s set for each. A lnk s created between two ncdents only when all thresholds are met. Lke the baselne algorthm, a lnk s bnary and does not have a relaton type. 5.3 Parameter Tunng For both algorthms, there are some parameters that need to be adusted based on a tranng set. The baselne algorthm nvolves only two parameters the clusterng threshold and a smaller lnk threshold. As the three-stage algorthm contans more features, the number of parameters s also larger. 1. The flterng threshold for BoosTexter 2. The clusterng thresholds for term vectors, named enttes and locatons 3. The lnk thresholds for term vectors, named enttes and locatons When the number of parameters s large, the search space grows exponentally, makng t ntractable to calculate the performance for each parameter combnaton. In the parameter tunng for both algorthms, one parameter s optmzed wthn ts range n each round, whle others are fxed. Ths hll-clmbng process contnues untl the performance does not mprove wth any change of a sngle parameter. We are aware of the rsk that hll-clmbng may return a local optmum. From our observaton, the fnal parameters are usually n the rght range, so we expect the performance to be close enough to the optmal soluton. A formal tranng/test dvson s necessary to ustfy the experment results, as complex models usually have advantages n achevng better performance on the tranng set. At the same tme, more heurstc nformaton and more parameters also ncrease the rsk of overfttng, whch wll hurt the evaluaton result on the test set. Unfortunately, the passage-based experment does not have a large data collecton wth relevance udgment, whch lmts the scope of tranng. From Table 1, we have 17 queres that have been fully annotated. They are smlar to some extent, as these are all queres related to volent actvtes. On the other hand, ther statstcs are wdely dstrbuted, partally caused by the nature of the source documents, and also because of the dfference n annotators. So here each query becomes an ndependent enough sub-collecton. Snce the number of queres s small, leave-one-out cross valdaton s performed, where the data n one query are reserved for evaluaton n each round and then all others can be used for tranng. 5.4 Results Two sets of parameter tunng are performed on the tranng set, where dfferent evaluaton crtera are optmzed. When the harmonc mean n Equaton 1 s used, the performance on the test set s shown n Table 5. Table 6 contans smlar data, but the matrx comparson score n Equaton 2 s optmzed nstead. Changes wth an astersk are sgnfcant mprovements over the baselne by a one-taled t-test. Note that smaller numbers are better for the lnk drecton error. Clusterng precson and recall [14] are also ncluded n the tables for comparson wth earler experments.

9 Table 5: Performance Comparson for Passage-Based Systems Mean all Optmzed Evaluaton Baselne Three-stage Change n % Incdent concentraton % Cluster agreement %* Clusterng precson %* Clusterng recall %* Lnk precson %* Lnk recall % Lnk drecton error % Mean all %* SQ_SIM(DT,DS) 19.10% 26.40% +38.2%* Table 6: Performance Comparson for Passage-Based Systems SQ_SIM(DT,DS) Optmzed Evaluaton Baselne Three-stage Change n % Incdent concentraton % Cluster agreement %* Clusterng precson %* Clusterng recall % Lnk precson % Lnk recall % Lnk drecton error % Mean all % SQ_SIM(DT,DS) 22.58% 25.05% +10.9% Wth dfferent measures to optmze, the two systems show nterestng performance patterns. In Table 5, the harmonc mean of varous scores s the obectve for the parameter tunng, whch focuses on the qualty of both clusterng and lnks. As the threadng step s often the bottleneck of performance, moderate numbers are shown for both systems, but the three-stage algorthm s comparably more successful. For both sngle-valued evaluaton measures, the three-stage algorthm s sgnfcantly better than the baselne. However, there s a large proporton of lnks that are assgned wrong drectons by the three-stage method, and t does not receve a hgh score on the recall part. From our falure analyss, the reason s that many postve (volent) passages are erroneously fltered out n the frst step. When the matrx comparson measure s used to optmze the parameters (Table 6), the performance dfference between the two systems becomes more complex. As ths evaluaton algorthm favors par-wse relatons that domnate the dstance matrces, clusterng performance s weghted more than lnks, because the number of par-wse connectons s small for most queres. Under that condton, the three-stage algorthm outperforms the baselne n clusterng and the overall matrx comparson, but acheves worse results n lnks, whch also leads to a smaller cluster-lnk mean. The approprate evaluaton crteron to use hghly depends on the applcaton. For fact-fndng scenaros, the matrx comparson measure seems to be a better opton. The calbraton study n Secton 4.3 s such an applcaton. We are glad to see that the performance of the three-stage algorthm falls n the 25-30% range, whch mples that the output ncdent network s useful n comparson to the orgnal documents. On the other hand, general news representaton should adopt the harmonc mean, as contextual nformaton s a very mportant factor n the understandng of a large collecton of news reports. More experments would help understand the correlaton and dfference between these evaluaton measures. 6. CONCLUSIONS In order to stay up to date, t s mportant to keep track of the newest reports at any tme. However, the rapd growth of nformaton demands external ad, otherwse users may be easly overwhelmed by the huge amount of news. As each of the exstng news processng models has bult-n defcences, ths paper restates ncdent threadng, whch analyzes news based on the real-world occurrences dscussed n a report and dentfes contextual nformaton among news ncdents. In ths artcle we descrbe passage threadng that extends earler work and breaks each news story nto fner granules. Ths s a research area that has not been extensvely studed before, so t possesses both great potental and challenges. The mplementaton starts wth a fully-annotated data collecton and approprate evaluaton measures for the new applcaton. Then two algorthms are provded for a reference of the performance. The three-stage algorthm acheves sgnfcant mprovement over the baselne when we tune on the cluster-lnk mean metrc, but mxed performance s observed when the matrx comparson evaluaton s used n tranng. Moreover, a calbraton study shows that the current performance of an ncdent network s at least comparable to the orgnal documents. Therefore, the applcaton of ncdent threadng s ustfable n a real system. As an early attempt n the new research area, the paper has provded a detaled framework and suffcent support for addtonal developments. The current progress s encouragng, and further research n ths drecton s promsng. Ths work has made contrbutons on both theoretcal and techncal aspects. Currently the man challenge stll les n the proper representaton of a short text snppet. Although term vectors, together wth the automatcally extracted man characters, geographcal locatons and tme stamps, have been the foundaton of a system wth moderate performance, further research wll probably rely on the accurate modelng of semantc nformaton represented n the short pece. Wth the lmtaton of a sngle man subect, the possble types of relatons are restrcted n the current mplementaton. An expanson to general news would be deal, although an annotaton attempt for that case faled for lack of user agreement. Clearer nstructons and extensve tranng should mprove the nterannotator agreement, makng t possble to mark up general ncdents. Wth a rcher background, type-specfc relaton analyss s a foreseeable outcome, and t wll certanly help the comprehenson of news evoluton at a hgher level. 7. ACKNOWLEDGMENTS Ths work was supported n part by the Center for Intellgent Informaton Retreval and n part by the Defense Advanced Research Proects Agency (DARPA) under contract number HR C Any opnons, fndngs and conclusons or

10 recommendatons expressed n ths materal are the authors' and do not necessarly reflect those of the sponsor. 8. REFERENCES [1] Allan, J. Topc Detecton and Trackng: event-based nformaton organzaton. Kluwer Academc Publshers, [2] Allan, J. Introducton to Topc Detecton and Trackng. In Topc Detecton and Trackng: event-based nformaton organzaton, Kluwer Academc Publshers, pp. 1-16, [3] Allan, J., Carbonell, J., Doddngton, G., Yamron, J., and Yang, Y. Topc Detecton and Trackng Plot Study: Fnal Report. Proceedngs of the DARPA Broadcast News Transcrpton and Understandng Workshop, pp , [4] Beeferman, D., Berger, A., and Lafferty, J. Text Segmentaton usng Exponental Models. Proceedngs of the Second Conference on Emprcal Methods n Natural Language Processng, pp , [5] Brown, G. and Yule, G. Dscourse Analyss. Cambrdge Unversty Press [6] Callan, J., Croft, W. B., and Hardng, S. The INQUERY Retreval System. Proceedngs of the 3rd Internatonal Conference on Database and Expert Systems Applcaton, pp , [7] Cohen, J. A coeffcent of agreement for nomnal scales. Educatonal and Psychologcal Measurement, vol. 20, pp , [8] Feng, A. and Allan, J. Fndng and Lnkng Incdents n News. Proceedngs of the ACM Sxteenth Conference on Informaton and Knowledge Management, pp , [9] Fscus, J. and Wheatley, B. Overvew of the TDT 2004 Evaluaton and Results. Topc Detecton and Trackng 2004 Evaluaton Workshop, NIST, Dec 2-3, [10] Fless, J. L. Measurng nomnal scale agreement among many raters. Psychologcal Bulletn, vol. 76(5), pp , [11] Grshman, R. and Sundhem, B. Message Understandng Conference 6: A Bref Hstory. Proceedngs of the 16th Internatonal Conference on Computatonal Lngustcs (COLING), pp , [12] Hearst, M. A. TextTlng: A Quanttatve Approach to Dscourse Segmentaton. Proceedngs of the 32nd annual meetng on Assocaton for Computatonal Lngustcs, pp. 9-16, [13] Jaynes, E. T. Informaton Theory and Statstcal Mechancs. Physcs Revews, vol. 106, pp , [14] Nallapat, R., Feng, A., Peng, F., and Allan, J. Event Threadng wthn News Topcs. Proceedngs of CIKM 2004 conference, pp , [15] O Leary, D. E. The Internet, ntranets, and the AI renassance. Computer, Vol. 30(1), pp , [16] Olve, J. Global Autonomous Language Explotaton (GALE). DARPA/IPTO Proposal Informaton Pamphlet, [17] Robertson, S. E., Walker, S., Honcock-Beauleu, M., Gull, A., and Lau, M. Okap n TREC-7: Automatc ad hoc, flterng, VLC and nteractve track. The Seventh Text REtreval Conference (TREC-7), NIST, [18] Schank, R. C. and Abelson, R. P. Scrpts, Plans, Goals, and Understandng: an Inqury nto Human Knowledge Structure. Lawrence Erlbaum Assocates, [19] Schapre, R. E. and Snger, Y. BoosTexter: A Boostngbased System for Text Categorzaton. Machne Learnng, vol. 39(2/3), pp , [20] van Dk, T. A. Dscourse Analyss: Its Development and Applcaton to the Structure of News. The Journal of Communcaton, 33(2), pp , [21] van Dk, T. A. News as Dscourse. Lawrence Erlbaum Assocates, [22] Vapnk, V. N. The Nature of Statstcal Learnng Theory. Sprnger, 1995.