B inary classification refers to the categorization of data

Transcription

1 ROBUST MODULAR ARTMAP FOR MULTI-CLASS SHAPE RECOGNITION Chue Poh Tn, Chen Chnge Loy, Weng Kin Li, Chee Peng Lim Abstrct This pper presents Fuzzy ARTMAP (FAM) bsed modulr rchitecture for multi-clss pttern recognition known s Modulr Adptive Resonnce Theory Mp (MARTMAP). The prediction of clss membership is mde collectively by combining outputs from multiple novelty detectors. Distnce-bsed fmilirity discrimintion is introduced to improve the robustness of MARTMAP in the presence of noise. The effectiveness of the proposed rchitecture is nlyzed nd compred with ARTMAP-FD network, FAM network, nd One-Aginst-One Support Vector Mchine (OAO-SVM). Experimentl results show tht MARTMAP is ble to retin effective fmilirity discrimintion in noisy environment, nd yet less sensitive to clss imblnce problem s compred to its counterprts. I. INTRODUCTION B inry clssifiction refers to the ctegoriztion of dt in the feture spce into two regions. In contrst, multi-clss pttern recognition is tsk to clssify the feture spce into more thn two regions using set of discriminte functions, in which ech region corresponds to pttern clss. More specificlly, given n vectors of instnces x =(x 1, x,,x n ) drwn from feture spce, multi-clssclssifierhsto clssify the inputs into k pre-defined clsses C =(C 1, C,, C k ), where C C b for b nd k >. Anumberofstudieshvebeencrriedoutinthereof multi-clss clssifiction [1][][3]. At present there re two min pproches tht cn be used to extend binry clssifier for multi-clss problems. The first pproch is by ssembling severl binry clssifiers to form recognition network. Exmples of this type of implementtion re oneginst-ll, one-ginst-one, one-ginst-higher-order, nd P-ginst-Q, wherep nd Q re greter thn one [1]. Aprt from deploying multiple binry clssifiers, nother pproch is to optimize single clssifier to recognize multiple clsses. Note tht it is computtionlly more expensive in Mnuscript received December 14, 007. This work ws supported in prt by the MIMOS Berhd, Mlysi. C. P. Tn is with the MIMOS Berhd, Mlysi (corresponding uthor) phone: ext.: 451; mobile: ; fx: ; e-mil: chue.poh@mimos.my ). C. C. Loy ws with MIMOS Berhd, Mlysi. He is now with the QueenMry University, United Kingdom. (e-mil: ccloy5@gmil.com ). W. K. Li is with MIMOS Berhd, Mlysi (e-mil: li@mimos.my) C. P. Lim is with Fculty of Engineering, University of Science Mlysi, Mlysi (emil : cplim@eng.usm.my ). solving multi-clss problem s compred with two-clss problem by using the sme mount of dt, since multi-clss clssifiction involves n ensemble of severl binry clssifiers or more complex optimiztion. In other words, multi-clss clssifiction is not merely trivil extension from binry clssifiction becuse multi-clss clssifiction my involve more complex boundry formtion nd optimiztion. Typicl differences in constructing multiclss clssifier nd binry clssifier re the network rchitectures, encoding schemes, nd trining methodologies [1], where ech of them hs high influence to the network ccurcy nd computtionl speed. This pper proposes modulr ARTMAP (MARTMAP), n extension of Fuzzy ARTMAP (FAM) [4] neurl network for multi-clss pttern recognition. It inherits the unique fetures from FAM, such s fst convergence nd incrementl lerning cpbility. The proposed network is bsed on novelty detection pproch, which differs from conventionl pproches, such s one-ginst-one, oneginst-ll nd strtegies forementioned. Bsiclly MARTMAP is built up with multiple novelty detectors tht re modeled independently by using only single clss of informtion. For ech novelty detector, distnce-bsed fmilirity function is introduced to determine whether n unknown pttern is fmilir to the respective pttern clss or not. The outputs from ll novelty detectors re then ggregted to form collective decision on the clss membership. In this study, three dt sets re employed in the performnce evlution. Aprt from compring with the originl FAM, the performnce of the proposed method is studied side-by-side with nother similr ART-bsed neurl network known s ARTMAP-FD [5]. The pper lso compred the proposed method with the stte-of-the-rt technique, multi-clss SVM tht implementing oneginst-one strtegy (OAO-SVM). It ws chosen becuse number of studies hve reported tht OAO-SVM gives good results s compred to other multi-clss clssifiction lgorithms [3]. The orgniztion of the pper is s follows. A detil explntion of the proposed method is provided in Section II. The dt sets used in this study is described in Section III. The experimentl results re reported nd discussed in Section IV. Finlly, Section V concludes the pper with some suggested future works /08/$5.00 c 008 IEEE 405

2 II. METHODOLOGY A. Modulr ARTMAP (MARTMAP) This section will commence with n overview of the MARTMAP rchitecture nd followed by more detils on the underlying lgorithms. As cn be seen from Fig. 1, MARTMAP is built up with multiple novelty detectors to form recognition network. These novelty detectors re bsiclly modified version of FAM to perform one clss clssifiction. Ech novelty detector should be ble to identify novel ptterns tht it is not wre of during trining [6]. In the trining stge, ech novelty detector is trined seprtely by using dt from single pttern clss, i.e., i th novelty detector is trined with ll the instnces from i th clss. Therefore, the number of novelty detector is proportionl to the number of pttern clss: for k-clss problem, k novelty detectors re required. In the prediction stge, the unknown pttern is fed into individul novelty detector. Ech novelty detector then computes ffinity score to mesure how fmilir the unknown pttern to the clss it recognized during trining. The ffinity score of ech novelty detector serves s the inputs to decision lyer which mke the finl decision. Note tht there is no direct connectivity between ech novelty detector. Thus, new novelty detector cn be dded or the existing one cn be removed from the MARTMAP clssifiction module s the need rises, without ffecting other trined novelty detectors. The decision lyer in MARTMAP plys n importnt role s it decides the finl clssifiction result bsed on the output from individul novelty detector. There re mny different implementtions tht re suitble for the decision lyer. It cn be simple rule-bsed clssifier (e.g. mx-win, min-win strtegy) or neurl networks depending on the complexity nd the nture of the pplictions. For certin types of pplictions, the decision lyer cn mke use of historicl prediction results to increse the prediction hits. For instnce, time series prediction such s object recognition in video strem, the decision lyer cn mke use of pst prediction results to form the finl decision. In this cse, the decision lyer performs clssifiction bsed on estimtion from multiple hypotheses [7], whereby outputs from the novelty detectors re used to construct decision histogrm tht records the clssifiction hypotheses. Hypotheses re ccumulted nd verged over period of time nd the finl clssifiction result is derived from the histogrm. A threshold cn be set to prevent the decision lyer of mking ny meningless guess before the estimtion chieves certin confidence level. This method is ble to reduce the generliztion error by ccumulting clssifier s predictions over time. Similr to the rchitecture of FAM s shown in Fig., ech novelty detector in MARTMAP consists of two fuzzy Adptive Resonnce Theory (ART) modules designted s ART nd ART b,whichcretestblerecognitionctegories in response to rbitrry sequences of input ptterns [3]. Both ART modules re linked together by mp field module, F b, n ssocitive lerning network to estblish n ssocition between input ptterns nd trget clsses C. Similr to FAM, there re two key prmeters tht influence the performnce of MARTMAP. The first prmeter is bse vigilnce prmeter [0, 1] which determine the ctegory formtion of the network. Less ctegories re formed by using lower, which in turn leds to more generlized boundry. Inthecontrry,higher will leds to firmer ctegory formtion nd the close boundry is tighter. The lerning prmeter, [0, 1], determines the lerning modes of the network. There re two lerning modes: fst lerning ( =1forlltimes)ndfstcommit slow recode lerning ( =1fornuncommitted node nd <1forcommittednode). The trining stge of individul novelty detector in MARTMAP is identicl to FAM. The following is brief explntion on the typicl opertion in ART,whichlso occurs in ART b.initilly,inthetriningstge,theoriginl M-dimensionl input vector is complement-coded into M-dimensionl vector A: c 1 M 1 M,,...,,1 A,...,1 (1) A is propgted from the input lyer F 1 to the dynmic output lyer F through set of dptive weights w. Activtion of the j th F node is determined by the choice function T j (A) sdefinedineqution(),withw j denoting the ctegory weight vector of the j th F node. T j A w j ( A) () w According to the winner-tke-ll strtegy, the node with the highest response vlue, denoted s node, isselectedsthe winning node, while ll other nodes j re dectivted. The winning node remins ctive if the mtch function of the chosen ctegory meets the vigilnce criterion: A w A Fig. shows the rchitecture of Fuzzy ARTMAP (FAM) neurl network. Ech novelty detector in MARTMAP is equivlent to modified FAM. If the vigilnce test is stisfied, the network will proceed to the mp field ssocition. However, if the existing winning node fils to predict the output clss, i.e., c() C, mtch trcking process is triggered until the best winning node tht stisfies both the ART nd mp field vigilnce test is found. Subsequently, lerning tkes plce by updting the ctegory weight vector of the winning node in ART ccording to Eqution (4). The process forementioned is repeted until j (3) Interntionl oint Conference on Neurl Networks (ICNN 008)

3 Novelty Detector 1 score 1 Unknown Pttern Novelty Detector score 1 Decision Lyer Decision Novelty Detector n score n Fig. 1 ArchitectureofmodulrARTMAP(MARTMAP) Fig. ThefigureshowstherchitectureofFuzzyARTMAP(FAM)neurlnetwork. Ech novelty detector in MARTMAP is equivlent to modified FAM 008 Interntionl oint Conference on Neurl Networks (ICNN 008) 407

4 the novelty detector lerns ll the trining instnces ssign to it. At the end of the trining stge, ech novelty detector would hve t lest one ctegory tht code ll the instnces of the respective clss C. Sometimes, instnces from clss my be coded by severl ctegories. This is minly due to the underlying distribution nd internl structure of the trining dt spce. But gin, the number of ctegory formed cn be lso controlled by the selection of. w (new) (old) (old) A w w 1 (4) The prediction phse is divided into two stges: the first stge is the internl competition of ctegories within novelty detector, wheres the second stge is the competition mong novelty detectors. Initilly, n unknown pttern is presented to every novelty detector. In the first stge, the response of ech ctegory to the unknown pttern is mesured using the Eqution (). The node tht hs the highest response, denoted s node, is selected s the winning node. All other nodes j re dectivted in ccordnce with the winner-tke-ll competition. As result, ech novelty detector would hve winner ctegory tht cn join the subsequent competition. In the second stge, fmilirity function is used to mesure the fmilirity of novelty detector to the new input pttern. The fmilirity function is bsiclly the Eucliden distnce between the input pttern to the centroid of the winner ctegories. The resulting distnces re then trnsmitted s the ffinity scores to the decision lyer. If the input pttern ppers more fmilir to the novelty detector, the distnce vlue would be smller. Therefore, min-win strtegy is used in the decision lyer, the i th novelty detector with the smllest Eucliden distnce will be selected s winner in the second stge, nd the clss lbel C i will be ssigned to the unknown input pttern. In the cse where two novelty detectors give the sme smllest vlue, the decision lyer is forced to mke decision by selecting the novelty detector with lower index s the winner. -c -c -c -c w w,w,, w ) (5) j ( j1 j Note tht the dimension of the centre weight vectors covers only the originl dimension of the input spce. At the beginning, the centre weight vectors re initilised to zero. When lerning tkes plce, the centre weight vectors of the th winning node re updted s follows, where N inputs denotes the number of inputs tht the ctegory hs coded. -c new -c old 1 -c w w w (6) - Ninputs In order to compute the Eucliden distnce, new set of weight vectors is introduced in the ART module of MARTMAP clled the centre weight vectors, jm old M -c -c, w i w i dist (7) i 1 There is nother ART-bsed neurl network clled ARTMAP-FD which is similr to the novelty detector in MARTMAP. ARTMAP-FD is n extension of FAM network with improvement in performing novelty detection. In contrst to the Eucliden distnce-bsed method proposed in this pper, the fmilirity function in ARTMAP-FD is the choice function expressed in Eqution (). Although novelty detector in MARTMAP nd ARTMAP-FD re both trined with locl knowledge, using choice function s fmilirity mesurement, however, my esier to prone to clssifiction errors in the presence of noise nd outliers, nd my fce clssifiction uncertintyinoverlppedboundries. Fig. 3 gives n exmple of two decision boundries generted from two novelty detectors. A decision boundry bsiclly is hyper-rectngulr R formed in F ctegory to enclose ll the dt points fll in tht prticulr clss region. Fig. 3 showstwodecisionboundriesthtrecorrupted with noisy dt locted t the lower right corner of the hyper-rectngulr. As cn be seen from Fig. 3, lthough the unknown pttern X is nerer to the right hyper-rectngulr lbeled s C b,itishoweverenclosed by the hyper-rectngulr lbeled s C due to unwnted noise in the trining dt. If choice function ws used s fmilirity discriminte function, X will be clssified s C insted of C b,whichclerlynerrorinclssifiction.such error cn be mitigted by mesuring the Eucliden distnce between X nd the clusters centroid s depicted in Fig. 3, where the distnces re denoted s d 1 nd d, respectively. As result, pttern X is clssified s C b since d < d 1. Fig. 3b illustrtesnunknownptternxloctedinside two overlpped decision boundries. In this cse, more thn one novelty detector will clim the unknown pttern s their pttern clss, which cuses clssifiction mbiguity. Agin, by using Eucliden distnce, one cn esily discriminte the point X from two overlpping clusters () Interntionl oint Conference on Neurl Networks (ICNN 008)

5 from the imges in this study, nmely dispersedness, compctness, xis rtio of fitted ellipse, roughness, occupncy, nd rtio of squred hull perimeter to hull re. The dt set ws chrcterized s imblnced dt set becuse some of the clsses re represented by significntly more number of instnces compred with other clsses. For instnce, the number of instnces in clss 1 (bird) ws four times more thn the number of instnces in clss 4 (cr) (b) Fig. 3 Clssifictionmbiguitiescusedbyoutliersintriningdt III. DATA USED In this study, three dt set were used, nmely, Gussin dt set, Gussin dtset corrupted with noise nd shpe dtset. The three dt sets were used to evlute the performnce of the clssifiers. The synthetic dt (herefter clled s Gussin dt set) s depicted in Fig. 4 ws drwn from four overlpping Gussin distributions centered t different men but with the sme stndrd devition =0.,resultingtotlof50 dt points for ech distribution. The min purpose of generting the Gussin dt set ws to simulte dt spce with four clsses, so s to exmine the bility of clssifier in seprting the clsses during prediction. The second dt set ws derived from the originl Gussin dt set with 15% of the trining dt corrupted with zero-men Gussin noise with vrince of 1.0. Fig. 4 SyntheticdtgenertedfromfourGussindistributions Fig. 5 Thefigureshowssomesmpleshpesintheshpedtset III. RESULTS AND DISCUSSION A. Comprison with FAM nd ARTMAP-FD The objective of the experiments is to compre the performnce of MARTMAP with ARTMAP-FD nd FAM. Rndom smple cross-vlidtion ws employed in this study, whereby hundred rndom prtitions were generted by prtitioning the originl dt set into 80%/0% trining/testing sets rndomly. Bootstrpping method ws implemented to compute the confidence intervls (CI) for the performnce metrics. The ccurcies obtined in ech sub-experiment were bootstrpped into 1000 smples. The verge of the estimted ccurcy long with the 95% CI ws then reported. Fst lerning pproch ( = 1) ws dopted throughout the experiments. Bse vigilnce prmeter ws chnged from low vlue ( =0.00)to high vlue ( = 0.90) in order to exmine the effect of this prmeter. Note tht =1.00isnotpplicbleinthis study s this vlue cuses the clssifier to ssume ech input is essentilly from different clsses. The results obtined by using the dt set re shown in Fig. 6. As cn be seen from Fig. 6, the performnce of ARTMAP-FD nd FAM remined stble cross different vlues of.althoughmartmaphdsuddendropin ccurcy t =0.50nddegrdedgrdullywhen ws further incresed to 0.9, MARTMAP generlly is significntly superior thn ARTMAP-FD nd FAM t lower vlues of,withthehighestccurcyt97.99%[ci= 97.83, 98.16] for =0.10. The third dt set consists of 193 shpes in eight ctegories. The dtset were selected from [8] nd the MPEG-7 test dtbse. Six simple fetures were extrcted 008 Interntionl oint Conference on Neurl Networks (ICNN 008) 409

6 Accurcy (%) MARTMAP ARTMAP-FD FAM Bse Vigilnce Prmeter Fig. 6 Thisfigureshowstheresultsvergedover1000bootstrpccurcieslongwith95%confidenceintervls. Actul Clss Actul Clss Actul Clss Predicted Clss Predicted Clss Predicted Clss () OAO-SVM (b) MARTMAP (c) Idel Fig. 7 ConfusionmtricesforshpeclssifictionbyusingMARTMAPndOAO-SVM.Echrowrepresentstheprobbilitiesofthtclssbeingconfused with ll the other clss verged over 100 runs Number of Clssifiers OAO-SVM MARTMAP Number of Clsses, k Fig. 8 ThisfigureillustrtesthenumberofclssifierneededbyOAO-SVMndMARTMAP Interntionl oint Conference on Neurl Networks (ICNN 008)

7 B. Comprison with Multi-Clss SVM In this study, the pproch used for multi-clss SVM ws One-Aginst-One (OAO-SVM) pproch, with Gussin rdil bsis kernel. The clssifier ws implemented by using LIBSVM librry [9]. In OAO-SVM, k(k 1)/ binry clssifiers were trined for k-clss problem. Ech binry clssifier is trined with dt from two clsses, C nd C b (1, b k). In the prediction stge, binry clssifier produces vote indicting whether feture vector belongs to clss C or C b.avotingschemeisusedto select the clss with the most votes nd ssign it to the feture vector. Since there re four clsses in the Gussin dt set, the vlue k ws set to 4. With the Gussin dt set, the result obtined using OAO-SVM ws 97.63% [CI = 97.46, 97.81]. In the cse where Gussin dt ws corrupted with noise, OAO-SVM chieved n ccurcy of 97.73% [CI = 97.5, 97.93]. In both cses, there were no obvious difference between the performnce chieved by MARTMAP nd OAO-SVM. In the experiment bsed on shpe dt set, the best ccurcy obtined by using OAO-SVM ws 6.17% [CI = 61.10, 63.6], compred with 9.37% [CI = 91.64, 93.08] chieved by using MARTMAP. As cn be seen from the confusion mtrices depicted in Fig. 7, the shpes were more likely to be misclssified by OAO-SVM s clss 1. Clerly, the performnce deteriortion of OAO-SVM ws cused by the imblnce trining dt. The mount of trining dt contributed by minority clsses ws fr less thn the mount in mjority clss (clss 1). As result, OAO-SVM hd tendency to produce outputs skewed to the mjority clss. In other words, it clssified fr more ptterns s belonging to mjority clss thn it should. Hung et l. [10] suggested tht the undesirble bising problem of OAO-SVM might be due to the equl error penlty of misclssifiction for ll the clsses. In contrst, the sme problem did not occur in MARTMAP. As depicted in Fig. 7b, MARTMAP ws less sensitive to uneven trining clss size s compred with OAO-SVM nd it ws ble to clssify ptterns, tht re from the minority clsses. Another observtion in this study ws the number of clssifiers needed by both OAO-SVM nd MARTMAP. As cn be observed from Fig. 8, the number of clssifiers needed in MARTMAP increses linerly with number of clsses k. Ontheotherhnd,thenumberofbinryclssifiers required by OAO-SVM increses in qudrtic with k. As result, OAO-SVM my computtionlly be more expensive since lrge number of binry clssifiers hs to be trined to hndle ech binry sub-problem when k is lrge. C. CONCLUSION AND FUTURE WORK The pper hs presented new FAM-bsed modulr rchitecture known s MARTMAP for multi-clss pttern recognition. The dynmics of the proposed network hve been described in detil to explin how multiple novelty detectors cn be used to drw collective decision. In MARTMAP, individul novelty detector is employed to discover nd lern the nturl groupings of the pttern clss ssigned to it by forming hyper-rectngulrs to enclose the pttern region. In the prediction stge, Eucliden distnce ws used to mesure the fmilirity between unknown ptterns with the centroid of the clusters formed in ech novelty detector. Fmilirity scores re ggregted to mke collective decision in severl wys such s min-win strtegy or multiple hypothesis estimtion. By using three dt sets, the pper hs demonstrted tht the proposed rchitecture is cpble of clssifying multiclss ptterns with higher ccurcy s compred to ARTMAP-FD nd FAM. In prticulr, the pper hs shown the cpbility of MARTMAP in retining its clssifiction ccurcy when the trining dt is corrupted with noise. Although individul novelty detectors in both ARTMAP-FD nd MARTMAP re trined with locl knowledge, but MARTMAP is ble to resolve the clssifiction uncertinty problem fced by ARTMAP-FD in overlpped boundries nd regions tht re not enclosed by hyper boxes. In the comprison study ginst OAO-SVM, MARTMAP ws found to be less sensitive in deling with clss imblnce problem. In ddition, it is better in terms of implementtion simplicity nd network complexity. The implementtion simplicity rises from the flexibility in mking chnges to individul novelty detector including retrining without ffecting the whole clssifiction module. Besides, MARTMAP is comprtively simpler thn OAO- SVM s it requires less clssifiers in solving the sme multiclss problem. The pper hs reveled the potentil of MARTMAP s multi-clss clssifier with good robustness to noisy nd imblnce trining dt. However, there re still number of res tht cn be enhnced nd pursued s further work. Firstly, some studies hve shown tht hyper-rectngulr my not be good geometricl representtion for certin types of dt. Therefore, it is worthwhile to investigte other geometricl representtion such s ellipsoid or hyper-sphere, which my give better generliztion nd representtion of ctegory. Secondly, insted of using Eucliden distnce s the discrimintion function, effectiveness of other distnce metrics such s Mhlnobis distnce cn be exmined. Finlly, the effectiveness of the MARTMAP hs to be vindicted ginst more dt sets. REFERENCES [1] G. Ou, Y. L. Murphey, Multi-clss pttern clssifiction using neurl networks, Pttern Recognition,Vol.40,No.1,007,pp [] F. Schwenker, Solving multi-clss pttern recognition problems with tree-structured support vector mchines, Proceedings of the 3 rd DAGM-Symposium on Pttern Recognition,001,pp [3]. Weston nd C. Wtkins, Support vector mchines for multiclss pttern recognition, Proceeding of the 7 th Europen Symposium On Artificil Neurl Networks,April1999. [4] G. A. Crpenter, S. Grossberg, N. Mrkuzon,. H. Reynolds, nd D. B. Rosen, Fuzzy ARTMAP: A neurl network rchitecture for 008 Interntionl oint Conference on Neurl Networks (ICNN 008) 411

8 incrementl supervised lerning of nlogue multidimensionl mps, IEEE Trnsctions on Neurl Networks,Vol.3,199,pp [5] G. A. Crpenter, M. A. Rubin, W. W. Streilein, ARTMAP-FD: fmilirity discrimintion pplied to rdr trget recognition, Proceeding of the Interntionl Conference on Neurl Networks, 1997, pp [6] M. Mrkou, S. Singh, Novelty detection: review prt 1: sttisticl pproches, Signl Processing,Vol.83,No.,003,pp [7] A.. Lipton, H. Fujiyoshi, nd R. S. Ptil, Moving trget clssifiction nd trcking from rel-time video, Proceedings IEEE Workshop Applictions of Computer Vision,1998,pp [8] T. B. Sebstin, P. N. Klein, B. B. Kimi, Shock-bsed indexing into lrge shpe dtbses, Proceedings of 7 th Europen Conference on Computer Vision,00,pp [9] C. C. Chng nd C.. Lin, LIBSVM : librry for support vector mchines, Softwre vilble t [10] Y. M. Hung nd S. X. Du, Weighted support vector mchine for clssifiction with uneven trining clss sizes, Proceedings of the 4 th IEEE Interntionl. Conference on Mchine Lerning nd Cybernetics,005,pp Interntionl oint Conference on Neurl Networks (ICNN 008)