A Novel Hybrid Neural Network for Data Clustering

Transcription

1 A Novel Hybrd Neural Network for Data Clusterng Dongha Guan, Andrey Gavrlov Department of Computer Engneerng Kyung Hee Unversty, Korea Abstract. Clusterng plays an ndspensable role for data analyss. Many clusterng algorms have been developed. However, most of em suffer eer poor performance of unsupervsed learnng or lackng of mechansms to utlze some pror knowledge ab data (sem-supervsed learnng) for mprovng clusterng result. In an effort to archve e ablty of semsupervsed clusterng and better unsupervsed clusterng performance, we develop a hybrd neural network model (HNN). It s e sequental combnaton of Mult-Layer Perceptron (MLP) and Adaptve Resonance Theory-2 (ART2). It nherts two dstnct advantages of stablty and plastcty from ART2. Meanwhle, by combnng e merts of MLP, t not only mproves e performance for unsupervsed clusterng, but also supports for sem-supervsed clusterng f partal knowledge ab data s avalable. Experment results show at our model can be used bo for unsupervsed clusterng and semsupervsed clusterng w promsng performance. 1 Introducton In general, data analyss meods consst of two categores: classfcaton and clusterng. Classfcaton s supervsed learnng. In classfcaton, we are provded w a collecton of labeled data tems and e problem s to label a newly encountered data tem. Typcally, e labeled patterns are used to learn e descrptons of classes whch n turn are used to label a new pattern. In case of clusterng, t s usually performed when no nformaton s avalable concernng e membershp of data tems to predefned classes. For s reason, clusterng s tradtonally seen as part of unsupervsed learnng [1][2]. Recently, a knd of new data analyss meods s proposed, called sem-supervsed clusterng. It s dfferent w tradtonal clusterng by utlzng small amount of avalable knowledge concernng eer par-wse (must-lnk or cannot-lnk) constrans between data tems or class labels for some tems [3][4][5]. Sem-supervsed clusterng s especally sutable for ose applcatons w partal but not much pror knowledge avalable. Alough many unsupervsed clusterng meods have been developed, most of em are unable to support sem-supervsed clusterng. In oer words, even some useful nformaton ab data s avalable, but we have no way to effectvely utlze em rough ose meods. So developng a concrete clusterng model at supports bo unsupervsed and sem-supervsed learnng s urgently needed.

2 In s paper, we develop a hybrd neural network (HNN) model. Ths model s orgnally proposed by us for nvarant recognton of vsual mages [7]. In s work, we propose to use s model for unsupervsed and sem-supervsed clusterng. Ths model s a sequental combnaton of Mult-Layer Perceptron (MLP) and Adaptve Resonance Theory-2 (ART2) [6]. HNN combnes e advantages of MLP and ART2. On one hand, t nherts stablty and plastcty from ART2 [7]. One e oer hand, by combnng e merts of MLP, sem-supervsed cluster s supported. We have tested our meod on two popular datasets: Irs and Balance Scale dataset, whch are avalable at UCI Machne Learnng Repostory [8]. The experments show e dstnct merts of HNN whch are also our contrbutons as follow: Its unsupervsed clusterng accuracy s better an most exstng clusterng meods. When t s used for sem-supervsed clusterng, small amount of pror nformaton could greatly mprove e clusterng accuracy. The structure of e paper s as follows. In secton 2, we present e HNN s archtecture and learnng algorm n detal. Secton 3 s e experments and comparsons w oer clusterng meods. We make e concluson and descrbe future work n secton 4. 2 Our Meod 2.1 HNN Archtecture As shown n Fg.1, our proposed hybrd neural network s a combnaton of MLP and ART2 w MLP n front and ART2 back. Fg. 1. Archtecture of hybrd neural network When t s used for unsupervsed data clusterng, e unlabeled data wll be sent to e nput layer of MLP frstly. Then e put of MLP wll be e nput of ART2. In

3 HNN, MLP could be treated as a data preprocessng layer, because t can provde data (features) converson rough ts hdden layers. Approprate data converson depends on e connecton weghts of MLP. In our model, MLP utlzes error back propagaton (EBP) to adust ts connecton weghts. We should note at e goal of tranng here s dfferent w tranng of tradtonal MLP for classfcaton. The goal here s to provde some addtonal help to ART2 rough data transformaton, so e tranng s secondary. Long tranng tme for tradtonal MLP s avoded here. In secton 2.2 and 2.3, e detaled algorm for unsupervsed and sem-supervsed clusterng wll be presented. 2.2 HNN for Unsupervsed Learnng The notatons used n our algorm are shown n Table 1. Table 1. Notatons Notaton Descrptons S, O, S O R d R m S : nput pattern. d : dmenson of S. : ndex of nput pattern O : put of MLP gven. : dmenson of. : S m O ndex of put pattern NI Number of neurons n e nput-layer of MLP, NI = d NK Number of neurons n e put-layer of MLP HLN Number of hdden layers n MLP NH Number of neurons n e hdden layer of MLP (supposed HLN = 1) N Number of clusters (number of neurons n put-layer of ART2) NS Number of samples n cluster Vglance value of ART2 ρ w, (1 NI,1 NH ) In MLP, connecton weght between neuron of nput-layer and neuron of hdden layer (supposed HLN = 1 ) w k,(1 NH,1 k NK) In MLP, connecton weght between W neuron of hdden layer and k neuron of put layer (supposed HLN = 1 ) The prototype (centrod) of cluster D The Eucldean dstance between O and W When HNN works as unsupervsed clusterng, ts learnng process s: Unlabeled data S s nputted nto MLP, O s e put of MLP.

4 O s nputted nto ART2 for clusterng. If O s recognzed belongng to class, en W (e prototype of class ) wll be treated as e target put of MLP for S. MLP tranng wll be adusted based on error back propagaton (EBP) algorm. The detaled C-lke algorm proceeds as follows: Algorm 1: HNN used for unsupervsed learnng Input: multple S (supposed totally n nput patterns) Output: Cluster number at each nput pattern belongs to Stage 1: HNN ntalzaton 1) MLP ntalzaton: 1/ 2) ART2 ntalzaton: N = 0 Stage 2: Clusterng 3) The sample 4) If ( 1 w = NI, w = 1/ NH k S s nputted nto MLP = ); else, goto step 5 == ), ( N = 1, and W1 O1 5) For ( 1: N = ), calculatng D. Then, select e mnmal one * 6) Vglance test: * If ( D * < ρ ), successful, S s recognzed belongng to cluster W updatng. W W D NS * * * * * Goto step 8; else, Goto step 7 7) 1 N = + /(1 + ), NS * = NS * + 1, = N +, W * = WN = O, In s algorm, we should note at tranng of MLP here s totally dfferent w tradtonal MLP tranng. In tradtonal MLP tranng, EBP need to reduce e errorfuncton of MLP to a very small value. Whle n HNN, EBP s used only for decreasng e dstance between actual put and target put of MLP. So long tme tranng s not needed. D S s recognzed belongng to s new cluster 8) MLP tranng by EBP w a small number of teratons ( O s actual put, W s * target put)

5 2.3 HNN for Sem-supervsed Learnng Sem-supervsed clusterng can be used n case of a small amount of pror knowledge avalable. The knowledge here means partal samples labels are known before clusterng and ey wll be e teacher of MLP. The algorm works as follows: Algorm 2: HNN used for sem-supervsed learnng Input: multple S (some samples labels avalable, S ) Output: Cluster number at each nput pattern belongs to y Stage 1: HNN ntalzaton 1) MLP ntalzaton: 1/ w = NI, w = 1/ NH 2) ART2 ntalzaton: N = 0 Stage 2: Learnng from e samples w labels known 3) Cluster prototype calculaton: W = W + Sy W /(1 + N) 4) MLP tranng by EBP. Stage 3: Clusterng The clusterng here s same w stage 2 n Alg. 1 k From s algorm, we can see s sem-supervsed learnng could acheve better result snce ose labeled data adust weghts of MLP to more approprate values. In oer words, based on ese labeled data, e put converson s more sutable for ART2 to get a better result. 3 Experments and Comparsons We test our model on two popular datasets, Irs and Balance Scale. Bo of em are avalable at UCI Machne Learnng Repostory. 3.1 Experment for Unsupervsed Learnng The dataset n s part s rs, whch s one of e most popular data sets to examne e performance of novel meods n pattern recognton and machne learnng. There are ree categores n e data set (.e., rs setosa, rs verscolor and rs vrgncal), each havng 50 patterns w four features. Irs setosa can be lnearly separated from rs verscolor and rs vrgncal, whle rs verscolor and rs vrgncal are not lnearly separable. Table 2 summarzes some of e clusterng results reported n e lterature. From e table, we can see at our approach provdes better result an most exstng meods (except Mercer Kernel Based Clusterng). The parameters used for

6 s experment are shown n Table 3 and exstng meods we use n our experments are as follow: GLVQ: general learnng vector quantzaton; GFMM: general fuzzy mn-max neural network; SVC: support vector clusterng; FCM; fuzzy c-means; CDL: cluster detecton and labelng network; HC: herarchcal clusterng: RHC: relatve herarchcal clusterng; FA: fuzzy adaptve resonance eory. Table 2. Experment results on Irs Algorms Number of Percentage of errors errors GLVQ[9] % FCM [10] % GFMM [11] 0~7 0~4.7% Mercer Kernel Based Clusterng [12] 3 2% SVC[13] 4 2.7% CDL[14] 6 4% HC [15] 13~17 8.7~11.3% RHC[15] 5~6 3.3~4% FA [16] 6.77~ ~30.9% K-Means % ****HNN (our approach)**** 4 2.7% Table 3. Parameters n HNN for clusterng Irs MLP ART2 1 hdden layer; 4 neurons n hdden layer 4 neurons n put layer Exponental Sgmod actvaton functon, a=1 Learnng rate=0.1 Iteratons=1 Vglance value R=0.08 In addton to HNN, we also use k-means to cluster Irs. For bo k-means and HNN, we fnd at almost all e ms-clustered samples are n versclor or vrgncal. It s not surprsed snce verscolor and vgncal are not lnearly separable. In fact, bo of k-means and HNN explots Eucldean dstance as smlarty measure, however, HNN can greatly mprove e clusterng performance compared w k- means. The reason s at e MLP part provdes feature converson (or mappng). As a result, most samples n verscolor and vrgncal are lnearly separable after feature converson (or mappng).

7 3.2 Experment for Sem-supervsed Learnng We test e performance of HNN for sem-supervsed clusterng on two datasets. One dataset s Irs, whch has been used n last experment. The oer one s extracted from Balance Scale dataset. We randomly select 60 samples, 20 samples for each class. For Balance Scale dataset, ere are ree categores w four features. The result of clusterng s shown n Table 4. For bo of e two datasets, 10% of total samples (15 samples n Irs, 6 samples n Balance Scale Dataset) are used for tranng. We can see at clusterng errors can be greatly reduced. The parameters we used n s experment are shown n Table 5. Table 4. Performance comparson between unsupervsed and sem-supervsed clusterng IRIS Balance Scale (60 samples) HNN: Unsupervsed Clusterng 4 errors 16 errors HNN: Sem-supervsed clusterng (10%) 1 error 7 errors Table 5. Parameters n HNN for sem-supervsed clusterng MLP ART2 1 hdden layer; 4 neurons n hdden layer 4 neurons n put layer Exponental Sgmod actvaton functon, a=1 Learnng rate=0.1 Iteratons=10 Vglance value R=0.1 4 Conclusons and Future Work In s paper, we propose a new data clusterng meod. It s a combnaton of Mult- Layer Perceptron and Adaptve Resonance Theory 2. To testfy e performance of our meod, we have done a set of experments on two known dataset: Irs and Balance Scale. Experment results show at our proposed meod surpasses most exstng meods n e followng two aspects: It provdes better unsupervsed clusterng accuracy. It also supports sem-supervsed clusterng, whch s crucal for ose applcatons w a small amount of nformaton avalable. Most exstng clusterng meods cannot be used for sem-supervsed clusterng.

8 Alough we have developed s meod and tested t one some known datasets, n e future, many ssues should be consdered. Two man ssues are: In fact, no unversal clusterng meods exst. We should explore at whch knd of data and applcatons our algorm s more sutable for. Most parameters used n our meod are fxed, such as learnng rate, teratons number and vglance value. We wll consder how to make em dynamc and adaptve for dfferent tasks. Reference [1] Ru Xu, Wunsch, D.: Survey of Clusterng Algorms. In IEEE Transacton on Neural Networks, Vol. 16, (2005) [2] A.K.Murty M.N. and Flynn P.J.: Data Clusterng: A Revew, In ACM Computng Surveys, Vol. 21, (1999) [3] Sugato Basu.: Sem-supervsed Clusterng w Lmted Background Knowledge, In Proc. of e Nn AAAI/SIGART Doctoral Consortum, (2004) [4] Sugato Basu, Arndam Baneree, and Raymond J. Mooney: Sem-supervsed Clusterng by Seedng, In Proc. of e Nneteen Internatonal Conference on Machne Learnng (ICML), (2002) [5] Nzar Grra, Mchel Crucanu and Nozha Bouemma, Unsupervsed and Sem-supervsed Clusterng: a Bref Survey, A Revew of Machne Learnng Technques for Processng Multmeda Content, 2004, [6] G.A. Carpenter and S.Crossberg: ART2: Self-organzaton of stable category recognton codes for analog nput patters, In Appl. Opt., Vol. 26, (1987) [7] Andrey Gavrlov, Young-Koo Lee and Sungyoung Lee.: Hybrd Neural Network Model Based on Mult-layer Perceptron and Adaptve Resonance Theory, In Proc. of Internatonal Symposum on Neural Networks 2006, (2006) [8] [9] N. Pal, J. Bezdek, and E. Tsao.: Generalzed Clusterng Networks and Kohonen s Selforganzng Scheme, In IEEE Transacton on Neural Networks, Vol. 4, (1993) [10] R. Haaway and J. Bezdek.: Fuzzy C-Means Clusterng of Incomplete Data, In IEEE Transacton on Systems, Man, Cybern, Vol. 31, (2001) [11] B. Gabrys and A. Bargela.: General Fuzzy Mn-Max Neural Network for Clusterng and Classfcaton, In IEEE Transacton on Neural Networks, Vol. 11, (2000) [12] M. Grolam.: Mercer Kernel Based Clusterng n Feature Space, In IEEE Transacton on Neural Networks, Vol. 13, (2002) [13] A. Ben-Hur, D. Hom, H. Segelmann, and V. Vapnk.: Support Vector Clusterng, In J. Of Machne Learnng Research, Vol. 2, (2001) [14] T. Eltoft and R. defgueredo.: A New Neural Network for Cluster Detecton and Labelng, In IEEE Transacton on Neural Networks, Vol. 9, (1998) [15] R. Mollneda and E. Vdal: A Relatve Approach to Herarchcal Clusterng, In Pattern Recognton and Applcatons, Fronters n Artfcal Intellgence and Applcatons, The Neerlands: IOS Press, (2000) [16] A. Barald and E. Alpaydn: Constructve feedforward ART clusterng Networks Part I and II, In IEEE Transacton on Neural Networks, Vol. 13, (2002)