MULTIPLE LAYAR KERNEL-BASED APPROACH IN RELEVANCE FEEDBACK CONTENT-BASED IMAGE RETRIEVAL SYSTEM

Transcription

1 Proceedngs of the Fourth Internatonal Conference on Machne Learnng and Cybernetcs, Guangzhou, August 2005 MULTIPLE LAYAR KERNEL-BASED APPROACH IN RELEVANCE FEEDBACK CONTENT-BASED IMAGE RETRIEVAL SYSTEM KIEN-PING CHUNG, CHUN-CHE FUNG School of Informaton Technology, Murdoch Unversty, Perth, Australa k.chung, Abstract: Relevance feedback has drawn ntense nterest from many researchers n the feld of content-based mage retreval (CBIR). In recent years, kernel-based approach has been a popular choce for the mplementaton of the relevance feedback based CBIR system. Ths s largely due to ts ablty to classfy patterns wth lmted sample data. Snce most of the kernel approaches reported have been treatng the nput as a long flat vector, such arrangement may ncrease the chances of pollutng the feature element that unquely dentfes the selected mage group. Ths paper proposes a two layer kernel confguraton wth an objectve to mprove the retreval accuracy. Whle the performance of the two confguratons s smlar n certan condtons, the proposed confguraton has shown to superor when domnant feature element ests that s capable to unquely dentfy the selected mage group. Keywords: Content-Based Image Retreval System (CBIR); Relevance Feedback; Kernel Method; Kernel Bas Dscrmnant Analyss (KBDA) 1. Introducton Content-based mage retreval (CBIR) system has been one of the most actve areas of research n recent years. One of the man reasons s due to ts potental n many commercal applcatons such as facal recognton for securty survellance system, or, n computer aded dagnostc systems for medcal applcatons. The other reason for the hgh level of actvtes may be due to the fact that such system s relatvely new, and most of the systems reported n the lteratures are stll n the prototype stage. Issues such as semantc gap and ndeng structure are stll largely unanswered. Recently, relevance feedback n CBIR system has ganed much attenton from the research communty. It s a strategy that nvtes nteractve nputs from the user to refne the query for subsequent retreval. Ths approach generally starts from promptng users to search the system va keywords, mage eamples or a combnaton of both. The system then prompts the user to select the relevant mages from the search results. After the user selected the mages, the system wll refne the orgnal query by analyzng the common features among the selected mages. Ths process s contnued teratvely untl the target s found. The selecton of the common features wll be the most approprate for applcatons ncorporatng ntellgent technologes such as neural network and fuzzy logc. Ths s due to the need for fne-tune and modfcaton of the process wth human nput. Evolutonary computaton technques could also be used n optmzng the process. Over the past ten years, relevance feedback has been evolved from a smple machne learnng problem such as frequency of occurrence [1], Bayesan classfcaton [2], and now, to the more popular kernel based approach [3-6]. Kernel technque has long been used n the statstc pattern recognton applcatons. Its recent popularty gan n CBIR applcatons s mostly due to ts ablty to analyze small sample data and the classfcaton of non-lnear data. The ntegraton of kernel based approach wth dscrmnant analyss reported by authors such as Tao and Tang [3], and, Zhou and Huang [6] have shown mprovement from the earler proposals. In a general CBIR system, t s mpossble to know what feature model/s can be used to capture the unque dentty of certan groups of mages. Hence, one dea s to employ as many mage features as possble n hope that one has the ablty to capture the unque feature of the targeted mages. Such dea ntroduces problems f the mage features are treated as a cascade of one flat vector. Such arrangement may ncrease the chances of pollutng the feature element that unquely dentfes the selected mage group. Inspred by the kernel based dscrmnant analyss as ntroduces by Zhou and Huang [6] and the herarchcal framework as proposed n MARS [1] system, ths paper proposes a kernel-based dscrmnant analyss herarchcal relevance feedback framework for the content-based mage retreval system. The dea s to project each mage feature separately to a feature space and calculate the Eucldean /05/$ IEEE 405 Authorzed lcensed use lmted to: Murdoch Unversty. Downloaded on June 15, 2009 at 04:10 from IEEE Xplore. Restrctons apply.

2 Proceedngs of the Fourth Internatonal Conference on Machne Learnng and Cybernetcs, Guangzhou, August 2005 dstance of these features n the new space wth reference to the postve centrod. Smlarly, the calculated dstance of each feature vector s agan treated as nputs to another feature space, and thereby calculatng the Eucldean dstance of each mage and fnally rankng the mages. Such arrangement has the advantage of treatng each mage feature ndependently and thus, ncreases the weght alas wth the more mportant feature elements. Ths paper wll frst provde a bref descrpton on the background of the theory, and follow by the descrpton of the proposed framework. The paper wll then look at the eperment conducted on the proposed method and lastly, concluson wll be drawn based on the eperment fndng. 2. Background 2.1. Kernel Bas Dscrmnant Analyss (KBDA) 2.2. Proposed Method Fgure 1 llustrates the abstract computaton model of the proposed method. It s essentally what Zhou and Huang [6] proposed ecept the authors of ths paper have restructured the analyss of the nput vector nto two layer. In ths confguraton, each mage feature vector s processed separately by ndvdual KBDA module. The outcome of the low-level KBDA module s the Eucldean dstances of projected pont wth reference to the postve centrod. These outcomes serve as nputs to another KBDA module whch wll yeld a new pont n the fnal projected space. Ths new pont wll be used to compute the fnal Eucldean dstance for rankng purpose. The authors called ths confguraton as the herarchcal kernel based dscrmnant analyss (HKBDA). The dea of KBDA [6] s to transfer data from the orgnal space to a new feature space that can best dscrmnate the postve from negatve gven samples. Ths s done be applyng a set of weght vectors W that mamze the rato between the postve covarance matr S and the based matr epressed as: S y. The problem can be W opt T W S yw = arg ma (1) T w W S W matr The postve covarance matr S y are defned as: S where S y = = { y 1,..., N y Ny = 1 Ny = 1 S ( ( y ) m )( ( y ) m ) ( ( ) m )( ( ) and the based ) T (2) m (3) { = 1,..., N } } denote the postve eamples, = are the negatve eamples gven, and, Φ s the kernel mappng functon. One can vew Equaton (1) as a problem of generalzed egenanalyss where the optmal egenvectors assocated wth the largest egenvalues are the weght factor for the new feature space. By knowng the weght, one can now project the new nput pattern z onto the new space: ( z) new _ space = w T (4) T Fgure 1: The proposed HKBDA framework. The steps of HKBDA can be summarzed as follows: 1. Project each dfferent feature vectors of the gven postve and negatve eamples nto a new feature space by usng KBDA. 2. In the new space, calculate the Eucldean dstances of each eample from the postve centrod. 3. Smlar to Step 1, project the calculated dstances to another new feature space by usng KBDA. 4. In ths new space, return the ponts correspondng to the Eucldean nearest neghbors from the postve centrod. Wat for the user feedback then go to Step Authorzed lcensed use lmted to: Murdoch Unversty. Downloaded on June 15, 2009 at 04:10 from IEEE Xplore. Restrctons apply.

3 Proceedngs of the Fourth Internatonal Conference on Machne Learnng and Cybernetcs, Guangzhou, August Eperment and Evaluaton 3.1. Can The Data Be Polluted? Fgure 2 and Fgure 3 show the results produced from KBDA and HKBDA. In ths eperment, 24 mages from Corel dgtal lbrary were selected wth three labeled as postve mages wth a common theme and the non-selected mages as negatve samples. These mages are selected wth pror knowledge that the two mage groups can be separated by one of the feature elements n HSV color coherent vector (CCV) [7]. The frst plot on Fgure 2 shows the Eucldean dstances produced by KBDA when only CCV s appled n analyzng the mages. In ths plot, the three selected postve mages clearly have shorter Eucldean dstances wth reference to the postve centrod than all the other negatve mages. On the second plot, HSV moments [8] has also been added to the same system and the plot showng the Eucldean dstances between the postve selected mages and two other negatve mages have dmnshed. On the last plot, another feature, global edge hstogram [9], has agan been added to the system. By now, the Eucldean dstance of one of the negatve mages s shorter than one of the selected postve mages. Together, these plots clearly show the unque feature dentfyng the two groups of mages have been polluted by the etra mage features added to the system. features. (c), dstances calculated by usng HSV CCV, HSV moments and global edge hstogram as features. The same procedures have been appled HKBDA and the results are shown n Fgure 3. The plots have shown that although the scales of the plots vary, the shape of the curves has not changed. More mportantly, regardless of the addtonal features, the three selected postve mages have the shortest Eucldean dstances on all three plots. Fgure 3: Eucldean dstances computed usng HKBDA The Prototype System Fgure 2: Eucldean dstances computed usng KBDA. (a), dstance calculated by usng HSV CCV as mage feature. (b). dstances calculated by usng HSV CCV and moments as mage Fgure 4: User nterface for the prototype system. 407 Authorzed lcensed use lmted to: Murdoch Unversty. Downloaded on June 15, 2009 at 04:10 from IEEE Xplore. Restrctons apply.

4 Proceedngs of the Fourth Internatonal Conference on Machne Learnng and Cybernetcs, Guangzhou, August 2005 To evaluate the performance of the proposed approach, the authors have desgned and mplemented a prototype mage retreval system usng Matlab. The user nterface s shown n Fgure 4. Ths prototype system provdes the user wth the ablty to query the mage database va an mage sample. After the frst retreval teraton, users can select the relevant mage whle gnorng the non-relevant mages. The system wll label the selected mages as postve mages whle treatng the gnored mages as negatve mages. The retreval procedure of the prototype system s as follows: 1. User nputs a query mage; 2. The vsual features of the query are etracted by the system; 3. All mages n the database are sorted n ascendng order based on the dstance of dssmlarty; 4. User selects the postve mages and the rest wll be automatcally labelled as negatve mages. 5. Query and all mages n the database are projected n the kernel space based on the HKBDA approach. For ths eperment, Radal Bass Functon (RBF) s used as the kernel for both confguratons. The purpose s to see how the two confguratons perform under smlar settngs. As for the mage features, the authors have selected the water-fllng edge hstogram [10], HSV colour coherent vector [7], HSV hstogram, global edge drecton hstogram [9], HSV colour moments [8] and colour ntensty hstogram. Together, 66 feature elements have been used for ths testng. It s the authors ntenton to use as many feature elements as possble. Ths s specfcally done by pollutng the mportant features wth nose. It s etremely dffcult to test and verfy the retreval result f one s to let the system performng search on a database. In addton, relevancy of an mage can be subjectve. Thus, the authors have developed a set of testng strateges to ensure the test result can be easly compared and measured. The dea s to let a user to select the relevant mages from a group of mages. After the user s selecton, the system s to re-rank ths group of mages, and deally, the user selected mages wll have a hgher rank than the other mages. The eact testng steps are as follow: 1. User nputs a query mage. 2. User selects an mage database. 3. On the frst teraton, the system wll retreve and rank the mage based on the Eucldean dstance measure on the ncluded mage features. In ths teraton, the weght s set equally to all the features. 4. User selects the relevant mages from the retreved mages. 5. Base on the current retreved mages, the system wll re-calculate Eucldean dstances of each mage usng the two confguratons and re-rank the already retreved mages accordngly Results of the Prototype System The performance of the proposed approach s evaluated accordng to the retreval accuracy and dstance rato between the closest negatve mage and the furthest postve mage. The bgger the rato mples the further the postve mages are from the negatve mages, and hence, better separaton. The accuracy of the retreval s calculated by dvdng the number of correctly dentfy postve mages by the number of postve mages selected by users. Table 1, Retreval result from KBDA Concept Accuracy Dstance rato Tank 50% Landscape wth 78% rver Brd 100% Mouse 50% Yellow flower 50% Table 2, Retreval result from HKBDA Concept Accuracy Dstance rato Tank 75% Landscape wth 100% 3.02 rver Brd 100% Mouse 50% Yellow flower 50% Table 1 and 2 are the retreval results gathered from 130 mages from the Corel mage database. The mages were retreved and classfed under fve dfferent themes. These results show both confguratons performed poorly on the mage groups mouse and yellow flower. The relatvely low accuracy n both tables ndcate both confguratons are unable to dstngush the selected mages from the negatve labeled mages. As for the other three groups of mages, HKBDA out-performed KBDA n both retreval accuracy and dstance rato. After careful analyss on the mage groups, the authors have found the reason for the poor performance of the two mage groups s because the mage analyss algorthms ncluded are unable to etract the feature/s that clearly dstngush the two mage groups. Ths observaton mples that n terms of retreval accuracy and classfcaton ablty, HKBDA out-performs KBDA when the mages can be dentfed by a unque feature. 408 Authorzed lcensed use lmted to: Murdoch Unversty. Downloaded on June 15, 2009 at 04:10 from IEEE Xplore. Restrctons apply.

5 Proceedngs of the Fourth Internatonal Conference on Machne Learnng and Cybernetcs, Guangzhou, August Concluson A new mult-layer kernel based framework for the relevance feedback content-based mage retreval system has been ntroduced n ths paper. The proposed framework combnes kernel based dscmnant approach wth mult-layer analyss framework. Usng ths framework, an mprovement has been shown n retreval and classfcaton ablty as compare to the orgnal framework n whch the nput was treated as a flat vector. In ths paper, the authors have only appled the proposed framework to a kernel-based algorthm. More detaled study s gong to be conducted by the authors to analyze the characterstc of such framework further by applyng t to several other kernel-based algorthms. One of the future drectons s to ncorporate relevance feedback n the mplementaton of computatonal ntellgent technques for the tasks of mage retreval. The other area of research wll be proflng and personalzaton of the user n order to provde frst level of mplct relevancy feedback. Acknowledgement Ths project s supported by a grant awarded by the 2004 Murdoch Unversty Research Ecellence Grant Schemes (REGS). Ken Png s also a recpent of the Murdoch Research Scholarshp (MURS) from References [1] Y. Ru, T. S. Huang, and S. Mehrotra, "Content-Based Image Retreval wth Relevance Feedback n MARS," Proceedngs of the Internatonal Conference on Image Processng, Washngton, DC, October [2] I. J. Co, M. L. Mller, S. M. Omohundro, and P. N. Yanlos, "PcHunter: Bayesan Relevance Feedback for Image Retreval," Proceedngs of the 13th Internatonal Conference on Pattern Recognton, Venna, Austra, August [3] D. Tao and X. Tang, "A Drect Method to Solve the Based Dscrmnant Analyss n Kernel Feature Space for Content Based Image Retreval," Proceedngs of IEEE Internatonal Conference Acoustcs, Speech, and Sgnal Processng, Montreal Canada, May [4] S. Tong and E. Chang, "Support Vector Machne Actve Learnng for Image Retreval," Proceedngs of 9th ACM Internatonal Multmeda Conference, Ottawa, Cananda, September 30 - October [5] G.-D. Guo, A. K. Jan, W.-Y. Ma, and H. J. Zhang, "Learnng Smlarty Measure for Natural Image Retreval Wth Relevance Feedback," IEEE Transactons on Neural Networks, vol. 13, pp , [6] X. S. Zhou and T. S. Huang, "Small Sample Learnng Durng Multmeda Retreval usng BasMap," IEEE Conference on Computer Vson and Pattern Recognton, Hawa, Unted States, December [7] G. Pass, R. Zabh, and J. Mller, "Comparng Images Usng Color Coherence Vectors," The 4th ACM nternatonal conference on Multmeda, Boston, Massachusetts, Unted States, November [8] M. Strcker and M. Orengo, "Smlarty of Color Images," Storage and Retreval for Image and Vdeo Databases III, San Dego/La Jolla, CA, USA, February [9] D. K. Park, Y. S. Jeon, and C. S. Won, "Effcent Use of Local Edge Hstogram Descrptor," Proceedngs of the 2000 ACM workshops on Multmeda, Los Angles, Calforna, Unted States [10] X. S. Zhou and T. S. Huang, "Edge-based Structural Features for Content-based Image Retreval," Pattern Recognton Letters, vol. 22, pp , Authorzed lcensed use lmted to: Murdoch Unversty. Downloaded on June 15, 2009 at 04:10 from IEEE Xplore. Restrctons apply.