Comparison of Novel Semi supervised Text classification using BPNN by Active search with KNN Algorithm

Transcription

1 Comparson of Nove Sem supervsed Text cassfcaton usng BPNN by Actve search wth KNN Agorthm Mahak Motwan 1 Assstant Professor, Computer Scence Department, TCST Bhopa,M.P.,462062, Inda mahak.motwan@trubansttute.ac.n Aruna Twar 2 Assstant Professor, Computer Scence Department, IIT Indore, M.P, Inda artwar@t.ac.n Abstract Wth the avaabty of huge amount of text n nternet, news, nsttutes, organzaton etc need of automatc text cassfcaton aso ncreases, The proposed work comprsed to dea wth the major chaenge of gettng abeed data for tranng n cassfer, snce the avaabty of abeed data s expensve, tme consumng, t aso requres the nvovement of annotator. A nove sem supervsed test cassfcaton agorthm based on Back Propagaton Neura Network s proposed whch makes use of web asssted unabeed data by Actve search, ths agorthm s compared wth standard KNN agorthm on test data and standard data Mn Newsgroup. Expermenta resuts state that the proposed agorthm outperforms KNN wth Mcro averaged F 1 measure. INTRODUCTION A Major ssue n the fed of text cassfcaton s to organze arge amount of documents nto a number of meanngfu casses. Text cassfcaton has appcaton n the fed of securty, Bo medca, Company Resource Pannng [1]. In exstng Agorthm of Text cassfcaton Documents are represented usng Vector space mode whch treats document as bag of words. Text representaton s one of the cruca steps of text cassfcaton. Dependng on the data avaabe text cassfcaton can be categorzed as supervsed or unsupervsed. Supervsed earnng s earnng from abeed exampes. It s an area of machne earnng that has reached deveopment, t has generated genera purpose and practcay successfu agorthms[2], whereas earnng wthout the use of sampes or abeed data s unsupervsed earnng where one fnds an nterestng structure wth sampe ndependenty drawn from unknown dstrbuton, Unsupervsed earnng s cosey reated to the probem of densty estmaton n statstcs[3], major probems of unsupervsed earnng are mnmum doman knowedge, nosy data, nsenstve to nstance order etc. The arge quanttes of data s requred to obtan hgh accuracy, and the dffcuty of obtanng abeed data ed the Research drect towards fed where one can use a ot of unabeed data whch are easy avaabe rather than abeed data whch are manuay assgned by experenced anayst whch makes t tme consumng and abor ntensve job. Sem supervsed s a way to make use of ths huge amount of easy avaabe unabeed data and few abeed data whch makes t perform better than unsupervsed agorthm, our proposed agorthm s makng use of ony few root words and easy avaabe reevant data to tran the cassfer. The Nove approach to text cassfcaton starts wth just few root words usng actve search we coect web asssted data that undergoes the pre-processng, effcent text representaton technque s used foowed by BPNN, Our agorthm s compared wth standard KNN on the bass of Mcro averaged F 1 measure.the rest of the paper s structured as foows. In Secton 2 the Pre processng steps are descrbed aong wth the term weghtng method. Secton 3 proposes the agorthm and comparson wth KNN. Secton 4 depcts the expermenta Methodoogy and resuts. Secton 5 concudes the research and dscusses future prospects. 2. Pre processng n Text cassfcaton 2.1 Tokenzaton Text document s a coecton of sentences. In order to extract a words that are used n a gven text, a tokenzaton [4] process s requred for convertng text document nto stream of words by removng a punctuaton marks such as commas, spaces, tabs, speca characters etc, a text documents are merged to obtan set of dfferent words whch are coectvey caed the dctonary of a document coecton. ISSN : Vo. 6 No.05 May

2 2.2 Fterng Stop words Fterng [5] s a method to remove words from the dctonary and thus from the documents. A standard fterng method s stop word fterng. The set of dfferent words.e. dctonary whch s output of tokenzaton phase s now taken as nput for the stop word fterng. The dea of stop word fterng s to remove words that have tte or no content reated nformaton, ke artces a,an, the, conjunctons and, but, prepostons on, above, etc. It reduces compexty wthout any oss of nformaton for typca appcaton 2.3 Stemmng A stem s a natura group of words wth equa meanng. (Andreas Hotho; 2005) [5]. Stemmng method dentfes the root of words for exampe run s the root word of runnng and ran. Ths methods try to construct the basc forms of words.e. to remove ng from verbs, pura s from nouns, ed from past tense or other affxes. A we-known rue based stemmng agorthm has been orgnay proposed by Porter [Por80] [6]. He defned a set of producton rues to teratvey transform (Engsh) words nto ther stems. Each document words are preprocessed usng Porter s stemmng agorthm. After the stemmng process, every word s represented by ts stem. 2.4 Supervsed Term weghtng Method based on Reevance Factor Term Weghtng Methods The term weghtng methods assgns an approprate weght to the term to mprove the performance of text cassfcaton[7] paper nvestgates severa wdey used unsupervsed and supervsed term weghtng methods, a new smpe supervsed term weghtng method, tf,rf, (term frequency, reevance frequency)s used to mprove the terms dscrmnatng power for text categorzaton task, here emphass has been made on term dscrmnatng power anayss,reevance factor refers to the degree of reevance of the term to the category t beongs to as compared wth ts reevance to other documents. It has been proved that t has a consstenty better performance than other term most wdey used term weghtng methods Term frequency [8]. In text cassfcaton of mutpe casses, a term may have hgh term frequency (t.f) and may beong to amost a the casses n ths case the term actuay do not posses a hgh dscrmnatng power and so the nverse term document frequency factor and ts varant has been used, our proposed agorthm uses a supervsed term weghtng method whch s a mutpcaton of t.f and reevance factor r.f. where reevance factor s defned as r.f= og (2+ (a/max (1, c)) (1) Here a: tota number of document n the postve category that contan ths term c: number of document n the negatve category that contan ths term Here we assgn a term as postve category f t beongs to the document that beongs to the category and a other categores combned together as negatve category 3. Proposed Work 3.1 Proposed Agorthm Labeng arge amount of text spans for tranng systems s tme consumng and unreastc for many appcatons. We consder here the use of sem-supervsed technques, whch ets to tran a system wth ony a few abeed documents together wth arge amounts of unabeed documents, It s dffcut to bud reabe cassfer that s abe to acheve hgh cassfcaton accuracy wth of sma number of avaabe abeed documents, one way to overcome ths probem s by usng actve search. Actve search s a way to frst dentfy a number of mportant keywords, root words beongng to dfferent category and then utze search engnes to retreve from the web a muttude of reevant documents [9], we use Googe to get reevant documents. Though ntay we have unreated keywords, query word the web data or document coected w undergo effectve Preprocessng and feature seecton term weghtng method to remove the rreevant words and proceed for tranng. Ths data undergoes through the pre processng method of tokenzaton, stop word remova, appcaton of porter stemmng, we reduce the dmenson by consderng ony those words that appear n more than one document usuay words appearng n ony one document has ts correaton wth that document exampe names,such words do not specfcay have dscrmnatng power, such word are not consdered. Our Agorthm appes Supervsed Term weghtng Method based on Reevance Factor, t posses hgh dscrmnatng capabty of text words to the category. Ths data s fed to Neura Network cassfer based on Back Propagaton Neura Network. One of an effcent and popuar approach for text categorzaton s Neura network, t can hande near and nonnear probems for text categorzaton, and both of near [10] and nonnear [11] cassfer can acheve good resuts. There have been dfferent neura networks appcatons to text categorzaton. Perceptron s the earest and smpe form of neura networks, whch has ony one nput and an output ayer, Ng, Goh, and Low frst used the perceptrons to construct a text cassfer, and reported a ISSN : Vo. 6 No.05 May

3 surprsngy hgh performance [12]. Nakayama and Shmzu deveoped a tranng procedure for subject categorzaton usng mutayer perceptrons [13]. The nonnear neura networks are the more sophstcated neura networks wth some hdden ayers between the nput and the output ayers. Ruz and Srnvasan compared the back propagaton earnng mechansm and counter propagaton earnng mechansm[14]. Back propagaton neura network (BPNN) s the most popuar n a of the neura network appcatons. It has the advantages of yedng hgh cassfcaton accuracy. Back Propagaton Neura Network based Cassfer Mutayer feed forward network whch uses a supervsed earnng method, a generazaton of deta rue s known as back propagaton earnng agorthm.back propagaton neura network. The tranng of a network by back propagaton nvoves three stages: the feed-forward of the nput tranng pattern, the cacuaton and backpropagaton of the assocated error, and the adjustment of the weght and the bases. Input pattern feed-forward. Cacuate the neuron s nput and output. For the neuron j, the nput Ij and output Oj are I j = W j * O j; (2) O j= f(i j + j ) (3) where wj s the weght of the connecton from the th neuron n the prevous ayer to the neuron j, f(i j + j )s an actvaton functon of the neurons, Oj s the output of neuron j, and j s the bas nput to the neuron. In ths paper, we use a tanh(n )sgmod actvaton functon defned wth the equaton: tansg(n) = 2/(1+exp(-2*n))-1; (4) Ths functon s a good trade off for neura networks. The error, E, s cacuated n ths paper, the mean absoute error functon s used n the output ayer The mean absoute error s used to evauate the earnng effects and the tranng w contnue unt the mean absoute error fas beow some threshod or toerance eve. E T O q (5) Here n s the number of tranng patterns, s the number of output nodes, and O n and T n are the output vaue and target vaue,respectvey. The mean absoute error s used to evauate the earnng effects and the tranng w contnue unt the mean absoute error fas beow some threshod or toerance eve. The back propagaton errors both n the output ayer, and the hdden ayer,, are then cacuated wth the foowng formuas: j ( T O W ) f f j ' ( ' ( O ) O ) j (5) (6) Here T s the desred output of the th output neuron, O s the actua output n the output ayer, O j s the actua output vaue n the hdden ayer, and k s the adjustabe varabe n the actvaton functon. The back propagaton error s used to update the weghts and bases n both the output and hdden ayers. Weghts and bases adjustment : The weghts, wj, and bases,, are then adjusted usng the foowng formuas: W j ( K 1) W j ( k) jo (7) ( k 1) ( k) Here k s the number of the epoch and g s the earnng rate. The back propagaton error s used to update the weghts and bases n both the output and hdden ayers. 3.2 KNN Agorthm KNN agorthm KNN s aso known as nstance based earnng agorthm, Nearest Neghbor cassfer are based on earnng by anaogy that s by comparng a test data wth tranng data that s smar to t[15], after preprocessng each text document s now represented as a set of words and ts correspondng numerca vaue specfyng weghtage of that term n document. KNN agorthm [16] s a stabe and effcent method of cassfcaton based on exampes. Usng KNN agorthm the process of document cassfcaton are as foows: In document set, we fnd the most smar K tranng documents for one gven test documentaton d. Then gve each document cass a vaue that s the smarty sum between the test documentaton and the documentaton n the K tranng documents beongng to the cass. That s to say, f there are some documentaton beongng to ths cass n the K documents, the vaue of ths cass s j (8) ISSN : Vo. 6 No.05 May

4 the smarty sum between these documentaton and the test documentaton. Sortng by scores after gettng the statstca vaue of the casses contan the K documents, we just consder the score more than threshod. Specfc steps are as foows: 1) Assume K = the nearest number; 2) Cacuate the smarty between the test documentaton d and a tranng text; 3) Choose K documents, whch s the most smar to the documentaton d, as the nearest text of the documentaton d. 4) Coect these casses of the nearest documents that have been choose. 5) Gve each cass a vaue based on the nearest K documents;,,, b s threshod. 6) The cass wth the bggest vaue s the cass of the test document. 4.1Expermenta Methodoogy, 34 Query words of Computer scence fed,39 Query word of Medcne and 42 Query words of Sports are used to retreve 100 documents of the threee feds that are dvded n 70 to 30 rato of tranng and test documents, Ths tranng documents undergo Tokenzaton.e remova of speca character, numerc vaues etc, after tokenzaton we get terms from 210 tranng text fes, foowed by remova of 428 stop words, porter stemmng agorthm s apped the terms reduces to words are retreved that are ftered on the bass of occurrencee n number of document, we retreve ony those terms that occur n more than one document and thus 6458 terms are consdered. Term weghtng method based on reevance factor s used for feature representaton, ths data beongng to three category computer scence,medcne and sports undergoes tranng n BPNN wth foowng parameters The parameters used for BPNN are neurons n nput ayer and 20 neurons n hdden ayer, tranng functon used s gradent descent adaptve tranng functon, tansg as actvaton functon for hdden ayer and near functon for output ayer, earnng rate used s 0.3, momentum of Evauaton Crtera Precson and Reca are two popuar performance measures for text cassfcaton, precson s the fracton of retreved documents thatt are reevant, reca s the fracton of reevant documents that are retreved. The set of documents that are both reevant and retreved s denoted as reevant retreved, Precson= Reevant retreved/retreved (9) Precson =true postves/ /(true postvess + fase postves) ( 10) Reca: ths s the percentage of document that are reevant to the documents that are reevant to the query and were n fact, retreved. t s formay defned as Reca= reevant retreved/reevant ( 11) Reca = true postves/(true postves + fase negatves) ( 12) However, nether precson nor reca makes sense n soaton from each other as t s we known from the IR practce that hgher eves of precson may be obtaned at the prce of ow vaues of reca. To combne precson and reca, the two most wdey used measures,.e mcro-averaged and Reca. F 1 and macro-averaged F 1 measure the F 1 measures are harmonc mean of Respectve Precson Mcro-averaged vaues are cacuated by constructng a goba contngency tabe and then cacuatng precson and reca usng these sums. In contrast macro-averaged scores are cacuated by frst cacuatng precson and reca for each category and then takng the average of these. The notabe dfference between these cacuatons s that mcro-averagng gves equa weght to every document (t s caed a document-pvoted measure) whe macro-averagng gves equa weght to every category (category-pvoted measure) ). (13) ISSN : Vo. 6 No.05 May

5 (14) 4.3 Expermenta Resuts The proposed agorthm s mpemented usng MATLAB verson 2012 MATLAB s a hgh-eve anguage and nteractve envronment for numerca computaton, vsuazaton, and programmng. The MATLAB neura network toobox provdes a compete set of functons and a graphca user nterface for the desgn, vsuazaton, mpementaton, and smuaton of neura networks,t s checked wth the test data as we as on standard mn newsgroup data and compared wth KNN agorthm. Fgure 1,2. shows the performance of KNN on test data for dfferent vaues of K on the bass of Mcro F 1measure and Macro F 1measure, expermenta resuts depcts that Knn performance s best for vaue k=3 whch s 0.5 McroAverageF 1measure and 0.60 MacroAverage F1measure Fgure 3,4 shows the performance of proposed agorthm on the bass of Mcro Averaged and Macro Averaged F 1 measure on test data for dfferent number of epochs, Proposed agorthm gves qute good resuts for test data, resuts are best McroAverageF 1measure 0.955, McroAverageF 1measure at 2400 epochs ISSN : Vo. 6 No.05 May

6 Fgure 3 shows the performance of proposed agorthm and KNN on the bass of Mcro Averaged F 1 measure on standard Mn Newsgroup data on random sets of 10 to 50 documents of computer and sports category of standard Mn newsgroup. 5. Concuson and Prospects The proposed work has been ntated to address varous probems dentfed n the fed of Text Cassfcaton.e. unavaabty of abeed documents, Usng Actve search, few numbers of keywords are used to get the reevant data of the categores of Computers, sports and Medcne. Effcent supervsed Term weghtng method based on Reevance factor s used for Text representaton, ths nput s fed to BPNN, the agorthm output s cacuated on dfferent set of test data and standard mn newsgroup data wth KNN agorthm on the bass of Mcro Averaged measure F 1 measure.it has been found that proposed agorthm outperforms KNN agorthm. Improvement n tranng tme coud be done by modfyng BPNN. References [1] Fagun N. Pate, Neha R. Son Text mnng: A Bref survey Internatona Journa of Advanced Computer Research (ISSN (prnt): ISSN (onne): ) Voume-2 Number-4 Issue-6 December [2] Mara-Forna Bacan, Avrm Bum,2010 A dscrmnatve mode of sem supervsed earnng ACM DOI / [3] Jordan, Mchae I.; Bshop, Chrstopher M. (2004). "Neura Networks". In Aen B. Tucker. Computer Scence Handbook, Second Edton (Secton VII: Integent Systems). Boca Raton, FL: Chapman & Ha/CRC Press LLC. [4] Andreas Hotho,Andreas N rnberger, Gerhard Paaß, "A Bref Survey of Text Mnng", May 13, 2005 [5] Vakannu Ramanathan, T. Meyyappan, "Survey of Text Mnng", n Internatona Conference on Technoogy and Busness Management, March 18-20, [6] M. Porter. An agorthm for suffx strppng. Program, pages , 1980 [7] Lan M,Tan C L,Su J,Lu Y, Supervsed and tradtona term weghtng methods for automatc text categorzaton, IEEE Trans Pattern Ana Mach Inte Apr;31(4): [8] Mahak Motwan, Aruna Twar Comparatve Study and Anayss of Supervsed and Unsupervsed Term Weghtng Methods on Text Cassfcaton Internatona Journa of Computer Appcatons ( ) Voume 68 No.10, Apr 2013 [9] Zengn Xu, Rong Jn, Kazhu Huang Mchae R. Lyu, Irwn Kng, Sem-supervsed Text Categorzaton by Actve Search, CIKM 08, October 26 30, 2008, Napa Vaey, Caforna, USA, ACM /08/10. [10] Ma L, Shepherd J, Zhang Y (2003) Enhancng text cassfcaton usng synopses extracton. In: Proceedng of the fourth nternatona conference on web nformaton systems engneerng, pp [11] Savo LY Lam, Dk Lun Lee (1999). Feature reducton for neura network based text categorzaton, 6th nternatona conference on database systems for advanced appcatons (DASFAA 99) [12] Ng HT, Goh WB, Low KL (1997). Feature seecton, perceptron earnng, and a usabty case study for text categorzaton. In: Proceedngs of the 20th annua nternatona ACM-SIGIR conference on research and deveopment n nformaton retreva, pp [13] Nakayama M, Shmzu Y (2003) Subject categorzaton for web educatona resources usng MLP. In: Proceedngs of 11 th European symposum on artfca neura networks, pp 9 14 [14] Ruz ME, Srnvasan P (1998). Automatc text categorzaton usng neura network. In: Proceedngs of the 8th ASIS SIG/CR workshop on cassfcaton research, pp [15] Eu-Hong (Sam) Han,George Karyps, Vpn Kumar Text Categorzaton Usng Weght Adjusted k-nearest Neghbor Cassfcaton, Advance n Knowedge Dscovery and Data Mnng,Lecture notes n computer scence,voume 2035, 2001 p [16] Dempster A, Lard N, Rubn D. Maxmum kehood etmaton from ncompete data va EM agorthm [ J ]. J. Roya Statstca Socety Seres B, 1997, 3 9. ISSN : Vo. 6 No.05 May