Skech-based Image Rerieval Using Conour Segmens Yuing Zhang #1, Xueming Qian *2, Xianglong Tan #3 # SMLESLAB of Xi an Jiaoong Universiy, Xi an CN710049, China 1 zhangyuing@su.xju.edu.cn 2 qianxm@mail.xju.edu.cn 3 xjuicemaple@sina.com Absrac The paper presens a skech-based image rerieval algorihm. One of he main challenges in skech-based image rerieval (SBIR) is o measure he similariy beween a skech and an image in conour wih high precision. To ackle his problem, we divided he conour of image ino wo ypes: he firs is global conour, suggesing ha we can use i o reduce he similariy beween he images wih complex background. The second, called salien conour, is helpful o rerieve images wih objecs similar o he query. Besides, we propose a new descripor, namely angular radial orienaion pariioning (AROP) feaure, which makes full use of he gradien orienaion informaion o decrease he gap beween skech and image. Using he wo conours as candidae conours for feaure exracion could increase he rerieval rae dramaically. Finally an applicaion of rerieval sysem based on his algorihm is esablished. The experimen on 0.42 million image daase shows excellen rerieval performance of he proposed mehod and comparisons wih oher algorihms are also given. I. INTRODUCTION Developmens in Inerne and mobile devices have increased he demand for powerful and efficien image rerieval ools. Conen-based image rerieval (CBIR) mainly uses he ex or an image as a query. Tex feaures are less accurae and migh ake mismach beween he user s expression and he user s expecaion. Alhough he imagebased search echnology develops rapidly and works well, here are some rouble in obaining relevan images when he user does no have he query images and ex. To avoid his problem, he user could draw a skech and hen use he skech as he inpu for an image rerieval sysem, his becomes more and more convenien for users. Skech-based image rerieval (SBIR) echnology becomes an acive research area. SBIR mehods use a hand-drawn skech composed of rough and simple black and whie o rerieve he corresponding images. Alhough SBIR had been sudied since 1990s, i sill remains challenge o measure he similariy beween a skech and an image wih high precision. Image rerieval mus deal wih he ambiguousness in he query skech caused by a lack of semanic, besides a large majoriy of poenial users fail o precisely express fine deails in heir drawings [14, 15, 16]. To improve he precision, many descripors are proposed. Thus 2015 IEEE 17h Inernaional Workshop on Mulimedia Signal Processing (MMSP), Oc 19-21, 2015, Xiamen, China. 978-1-4673-7478-1/15/$31.00 2015 IEEE many sudies have been focussed on how o choose a good descripor. Some works focus on global descripors, bu he oher works focus on local descripors. Some researchers design a robus global descripor o represen he skech and image individually. Global feaures can be beer used in image analysis, maching, and classificaion, such as HOG (hisogram of gradiens) [1], EHD (edge hisogram descripor) [2], and ARP (angular radial pariioning) [3]. However, global feaures are unsaisfacory as hey are unreliable under affine variaions. To overcome such drawbacks, Eiz e al. [4], [5] use local descripors o achieve sae-of-ar performance. And QVE (query by visual example) [6] is a ypical mehod using blocks and local feaures. Cao e al. also propose a local feaure mehod, edgel index mehod [7], for skech-based image search by convering a shape image o a documen-like represenaion. In order o develop an image rerieval sysem which is able o find ou more images wih objecs similar o he query, we develop a global feaure based on he global and salien conours. The global conour is a global feaure, and is defined o find he relevan image wih simple background. The salien conour is local feaure, and is defined o ackle he problem ha one objec is similar o he query. Besides, he AROP feaure is refined he ARP feaure, and makes full use of conour orienaion o consrain he shape informaion. The main conribuions of his paper are summarized as follows. 1) We propose he global conour, which inroduces he salien region o make he conour more discriminaive. 2) We propose he salien conour o make he rerieve images wih objecs similar o he query. 3) The orienaion pariioning scheme is inroduced based on he original ARP feaure. Thus AROP feaure conains more informaion, which makes he rerieval resul more accurae and reliable. The remainder of his paper is organized as follows. Work relaed o skech-based rerieval is reviewed in Secion II. We describe he proposed approach in Secion III, our experimens in Secion IV, and he discussion in Secion V. Finally, we presen our conclusions in Secion VI. II. RELATED WORK There have been a lo of sudies in skech-based image rerieval sysem recenly and skech based image rerieval echniques have been well discussed in [18]. In he following, we briefly describe some approaches which are widely used in SBIR sysem.
f and f 1 2 1 2 f and f Fig. 1. Illusraion for feaure exracion. The edgel index approach is a shape-based indexing mehod [7]. I solves he shape-o-image maching problem using pixel-level maching. Oriened chamfer maching [8] is used o compue he disance beween conours o convenienly build he index srucure, Wang e al. [7] used a binary similariy map (a hi map) insead of he disance map [7]. For each inpu skech, N hi maps are creaed, which correspond o he N orienaions. They also designed a simple hi funcion. Specifically, if a poin falls in he valid region on a hi map in he same channel, i is considered as one hi. The sum of all he his is he similariy beween he image D (represened by is conours) and he query skech Q. Then, hey build an edgel index srucure for fas rerieval. The ARP mehod based SBIR approach is firs proposed by Chalechale e al. in [3]. In ARP, he edge is firsly exraced by he Canny operaor and Gaussian mask, and hen he edge is hinned o obain he absrac image. Finally hey define angular pariions in he surrounding circle of he absrac image. Each number of pixels would be couned o form a hisogram defined as ARP (Angular Radius Pariioning) feaure. Roman-Rangel e al. [17] propose a shape descripor Hisogram of Orienaion Shape Conex (HOOSC), which exends he Generalized Shape Conex (GSC) [19] using a hisogram of orienaions and disance-based normalizaion. Differen from he HOOSC feaure, which is based on double sides of wide conours, we propose he AROP feaure based on wo candidae conours. Besides, he ype of similariy is differen. They use bag-of-shapemes, however, we compue he wo AROP feaure wih weigh. Zhou e al. [9] use he human percepion mechanism o idenify wo ypes of regions in one image: he firs ype of region is defined by a weighed cener of image feaures, and i is used o rerieve objecs in images regardless of heir size and posiions. The second ype of region is o find he mos salien par of an image. They firsly exrac orienaion feaures and hen organize hem in a hierarchal way o generae global-o-local feaures by a series of hierarchical and overlapping paches. Finally, a hierarchical daabase index srucure is buil. Cheng e al. [10] propose a simple, efficien, naurally muli-scale, and produces full-resoluion, high-qualiy saliency maps. Firsly, hey propose hisogram-based conras mehod o define saliency value for image pixels using color saisics of he inpu image (i.e., pixels wih he same color have he same salienc. And hen hey use hisogram comparison and spaial weigh o obain region conras (RC). They also inroduce SaliencyCu, which uses he compued saliency map o assis in auomaic salien objec segmenaion o auomae salien region exracion. Used in SBIR sysem, he auhor rank he images by SC [11] disance beween heir salien region oulines and user inpu skeches based on heir SaliencyCu algorihm. As a resul, heir rerieval mehod is more effecive. Berkeley deecor [12] o employ Brighness Gradien (BG), Color Gradien (CG), Texure Gradien (TG), and combine informaion from hese feaures in an opimal way. Berkeley deecor can accuraely deec he localize boundaries in naural scenes han ha of Canny. III. THE RETRIEVAL SYSTEM BASED ON AROP FEATURE The framework of he proposed SBIR sysem is shown in Fig.1. I consiss of wo pars: he offline par and he online par. In he offline par, we obain global conour and saliency conour based on conour map use Berkeley deecor [12] and saliency region use RC [10]. Then, we exrac AROP feaure from every conour. In he online par, for a given inpu query skech, we exrac he global conour and saliency conour based on he conour map. Then we exrac he AROP feaure for he wo conours. Finally, we calculae he similariy beween he inpu and he daase images. A. Conour Map and Orienaion Map Exracion The Berkeley deecor [12] exracs conours. For an image, we apply he Berkeley deecor o each image (resized o 200 200 ). We will ge he rue poserior probabiliy (defined P x, y > h, we define as (, ) (, ) P x y ) and orienaion. When ( ) Bh xy as he raw conour map, where h is he hreshold value, Bg ( xy, ) as he raw orienaion map and Bo ( xy, ) as he quanizaion orienaion map. 1, P( x, > h Bh ( x, = (1) 0, oherwise
( j 1) π jπ jifb, h ( xy, ) = 1& Bg ( xy, ), Bo ( x, = O O (2) 0, oherwise where j is he orienaion channel, O is he number of quanized orienaions. Bo ( xy, ) is obained by quaniaing xy, ino O orienaion channels. Bg ( ) B. Candidae Conours Exracion We divide he conour of image ino wo ypes: he firs ype is he global conour, suggesing ha we can use i o reduce he similariy beween he images wih complex background. The second ype is he salien conour, is helpful o rerieve images wih objecs similar o he query. 1) Offline Candidae Conours Exracion We use conour map and he saliency region [10] o obain he global conour and saliency conour. The global conour is used o presen he conour of image accuraely. The saliency conour is used o presen he main objec in he image and is used o find he relevan images conaining a common objec. We apply RC [10] o ge he saliency region in one image. The saliency map is defined as 1, if Bh ( x, in hesaliency region RC ( x, = (3) 0, oherwise The global conour map is shown in Fig.2. When he background is complex, he global conour map conains more edge pixels in he background region. This will makes he feaure value very big. So he global conour has discriminaion for complex background images. (b) Salien conour map exracion In mos cases, users are more concerned abou wheher he inpu skech can be found mainly in he image. So we can obain he saliency conour map hrough he following seps. Firsly, we obain he saliency region in each image and we use bounding-box o obain he minimum recangle of each saliency region. Secondly, we obain he saliency image region wih main objec. Tha is o say, if here is a saliency region, we make he region as he saliency image region. If he number of saliency region is more han 2, we make he maximum recangle (defined as r1), which conains a maximum of edge pixels, as he main recangle. Then we find he neares recangle (defined as r2). Then we make a new recangle (defined as r) wich conain he wo recangles as he saliency image region resized o 100 100. Thirdly, we exrac he saliency image region s conour map using Berkeley deecor and make he conour wihin he saliency region as he conour map Bh ( xy, ). In order o make he objec more clearly, we se h=0.3 and he B0.3 ( xy, ) is he saliency conour map. Fig. 2. The example of global conour maps. The firs row is he images, he second row is conour maps, and he hird row is global conour maps. (a) Global conour map exracion As previously menioned, we obain he conour map and saliency region in one image. We obain he saliency map RC from (3) and he conour map from (1). From [7], [12], we can know ha when h = 0.5, he conour map can presen he conour of he image and we choose his as he conour map of image. When h < 0.5, here will be more small edge. When h > 0.5, here will lose more conour informaion. In order o remove complex background images, we mus inroduce more deail edges bu no all. So we choose h = 0.3 for he saliency region. The global conour map is defined as 1, if B0.5 ( x, = 1 & RC = 1 MCM ( x, = 1, if B0.3 ( x, = 1 & RC = 0 (4) 0, oherwise Fig. 3. The example of saliency conour maps. The yellow border is he saliency image region in he firs row. The second row is saliency conour maps. From Fig. 3, we can find he conour map jus conain he main objec in he image even when he number of objec is wo. So our saliency conour map can be used o find more relevance images. 2) Online Candidae Conours Exracion Differen from he daase images processing, we apply Berkeley deecor o obain he inpu skech conour map, and hen we ake he conour map as he global conour map. We choose he image region which is he smalles recangle ha conains he larges pixel value as he saliency region resized o 100 100. Finally, we ake he conour map as he saliency conour map.
(a) (b) (c) (d) Fig. 4. AROP feaure exracion. (a) is he conour map, (b) is he angle, radius and orienaion pariion (red line is he gradien orienaion, which is quanized o 8 direcions), (c) is he hisogram of every secor and (d) is he AROP feaure. C. AROP Exracion In order o inroduce he AROP feaure, we firs give brief recommendaion for ARP. 1) The Angular Radial Pariioning Feaure Exracion The ARP [3] -based SBIR approach refines he angular pariioning (AP) feaure [13] using radial pariioning. The ARP feaure is obained by pariion he image ino M N secors uses he image cener as he cener of circles. M is he number of radius pariions and N is he number of angular pariions. The range of each angle θ = 2 π / N and he radius of successive concenric circles is φ = R / M where R is he radius of he surrounding circle of he image [3]. The conour is divided o N = 8 angular and M = 4 radials. Based on he obained conour map of he original image, he corresponding edge pixel number in each secor is uilized o represen each secor. Then, for he oal M N secors, he final ARP vecor is wih dimension M N. 2) Angular Radial Orienaion Pariioning Feaure ARP [3] is a coarse represenaion for he conour image. I jus coun he number of edge pixels in each secor, and hey don consider he gradien orienaion of edge pixel, which is proved o be effecive for maching. Thus, in he proposed mehod, we make full use of he gradien orienaion. Same wih he ARP mehod, we divided he conour map ino M N secors, where M is he number of angle pariion, N is he number of radius pariion. And hen we use he number of edge pixel number under in differen orienaion maps Bo ( xy, ). Tha is o say, we represen each secor by an O dimensional orienaion vecor, as shown in Fig.4 (b). Finally, we cascade he feaure of he oal O orienaion channel o represen he image. By his means, he oal dimension of AROP feaure is M N O. For each daase image and he inpu skech, we will exrac he AROP feaures based on he wo ype conour maps. Compared o he M N dimensional ARP feaure, he AROP feaure can represen more local spaial informaion. This local spaial informaion can narrow he scope of mach and also enhance accuracy rae. In a word, AROP capures cerain local spaial informaion of he image. D. AROP Feaure Maching In he offline, we exrac daase images AROP feaures and k we define hem as f, k = 1,2and = 1,2,, T, where T is he number of daase images, k denoes in he case of differen 1 conours. f is he AROP feaures based on he global 2 conour map and f is he AROP feaures based on saliency conour map. In he online, we exrac AROP feaure of inpu k query skech and define i as f. Le sim1( ) and sim2( ) denoe he similariy of he global conours f 1 1 and f and he 2 2 similariy of he salien conours f and f. Then we can measure he similariy of he inpu query skech and he daase images by aking he global conour similariy and salien conour similariy as follows: sim() = w sim1() + (1 w) sim2(), 1,2,, T (5) Finally, we sor he similariy scores in descending order, and deermine he resuls (he op-n ranked). IV. EXPERIMENTS AND DISCUSSION In order o show he effeciveness of he proposed approach, we compare our algorihm AROP wih he mehod Edgel [7], he mehod ARP [3], he mehod in [9] and he proposed mehod on our crawled daase. Besides, we exrac he AROP feaure on he conour map corresponding o RC region and use his mehod (named RC-AROP) as a compare. All experimens are carried ou on he same environmen. (a). Example of daase image. (b). Example of skech Fig. 5. The example of daase image and he inpu skech. A. Daase The experimen daase consiss of 433,790 images, and he sorage cos is 130 GB. Our daase is crawled from Google using key words. We selec 81 opics ha can be easily described by skeches, so ha here could be sufficien similar images for a query skech. And here are approximaely 1,000 images in each opic. This daase also conains he GOLD se [20, 21], which mainly conains landmarks and landscapes. Topics mainly include living goods, fruis, animals, and some landmarks easy o be described by skeches. Some examples are shown in Fig.5 (a). We draw 162 query skeches which cover mos of he 81 opics in he Skech-describable Daase, and hen se hem as queries o skech rerieval sysems. Some of hem are shown in Fig.5(b). In he following pars of his paper, experimens used he 162 skeches. B. Performance Evaluaion We used he precision under deph n (denoed as Precion@n) o measure he objecive performance, defined as Precion 1 1 @ n Z n = R () i m= 1 i= 1 m (6) Z n
where Rm () i is he relevance of he i-h resul for query m, i [ 1, 2,, n], and m [ 1, 2,, Z] relevan o he query skech hen m () 1 precision 0. 8 0. 7 0. 6 0. 5 0. 4 0. 3 0. 2. Z=162 for our daase. If i is R i =, oherwise R () i = 0. edgel ARP RC- AROP SBI R our s 0. 1 1 5 9 13 17 21 25 29 33 37 41 45 49 n Fig. 6. Precision comparison wih oher mehods. Edgel is he mehod in [7], ARP is he mehod in [3], RC-AROP is he AROP feaure based on RC [10] mehod and SBIR is he mehod in [9]. C. Objecive Comparisons Correspondingly, he Precision@n curves of oher mehods and he proposed mehod wih he deph varying in he range [1, 50] are shown in Fig.6. The curve is drawn by he average resuls of 162 queries on our daabase. For fair comparison, he parameers M and N are se o be 8 and 4 respecively for ARP and AROP. The pariion of radius is uniform. The orienaion channel (O) of AROP and edgel mehods are boh se o be 8 and in our mehod, w = 0.8 (he weigh of he similari. In our objecive comparison, we find ha he proposed algorihm improve 10% han he oher mehods in he op 10 resuls. For n = 1, our mehod is also more accurae han he edgel mehod and he RC-AROP mehod. The proposed mehod makes he image comprising a common objec more similariy and irrelevan image more differen. These experimens were implemened using Malab. The average compuaional cos of he proposed mehod is 1.423 s. When he orienaion channel O is increase by O, he dimension of AROP feaure will increase by M N O'. This can increase he compuaional complexiy. However, he increased ime can be ignored. D. Discussions We now discuss he impacs of he parameers on he performance of our skech-based rerieval sysem. In he proposed mehod, he parameer w in (5) is used o compue he score of similariy. We se w = 0.8 in our baseline experimens. This parameer deermines he conribuions of he global conour map and he saliency conour map. Accordingly, w should range beween 0 and 1. When w = 0, i means no AROP feaure of global conour map. When w = 1, i means no AROP feaure of saliency conour map. As shown in Fig.7, he mehod performed bes when w was approximaely 0.7. From Fig.7, we find ha he global conour map and he saliency conour are more imporan o he final performance for he following reasons. m 1) The global conour map conains more edge informaion and presens he conen of he image clearly. Besides i makes he image wih complex background have more edge informaion. So we can use he global conour map o filer he image wih complex background. 2) The saliency conour map conains he main objec in he image. This informaion can make he image wih common objec more similariy. precision 0. 88 0. 78 0. 68 0. 58 0. 48 0. 38 w=0 w=0. 1 w=0. 3 w=0. 5 w=0. 7 w=0. 8 w=0. 9 w=1 0. 28 1 5 9 13 17 21 25 29 33 37 41 45 49 n Fig. 7. Precision@n curves for various w. E. Subjecive Comparisons In Fig.8, compared o he SBIR [9], edgel mehod and he proposed mehods use hree inpu skeches. Fig. 8 shows he rerieval resuls of he mehod in [9] (he firs row), he edgel mehod (he second row) and our mehod (he hird row). As shown in Fig. 8 (a), our op 10 resuls were all correc and he oher mehods reurned several irrelevan images. In Fig. 8 (b, c), he edgel mehod and he mehod in [9] reurned more irrelevan images, bu our op five resuls were all correc. Fig. 8 shows ha our resuls also conained some incorrec images, bu hey are all similar in shape o he queries, and he resuls were beer han hose of he oher mehods. V. CONCLUSION We have inroduced a novel approach for image represenaion based on conour segmens. Considering ha false maches could degrade rerieval performance, we propose he global conour map and he saliency conour map. The proposed mehod can find he image wih simple background and find he image wih he common objec. This is very imporan for he chaoic daase. The experimenal resuls show ha he AROP feaures have cerain advanages over he oher mehods in rerieval precision. Various experimens proved ha skech rerieval algorihm is beyond he oher mehods. ACKNOWLEDGMENT This work is suppored in par by he Program 973 No.2012CB316400, by NSFC No.60903121, 61173109, 61332018, and Microsof Research Asia. REFERENCES [1] M. Eiz, K. Hildebrand, T. Boubekeur, and M. Alexa, A descripor for large scale image rerieval based on skeched feaure lines, Skech Based Inerfaces and Modeling, 2009, pp. 29-36.
[2] P. Salembier, T. Sikora and B. S. Manjunah. Inroducion o MPEG-7: mulimedia conen descripion inerface, John Wiley & Sons, Inc., 2002. [3] A. Chalechale, G. Naghdy, and A. Merins, Edge image descripion using angular radial pariioning, IEEE Proceedings-Vision, Image and Signal Processing, vol. 151(2): 93 101, April, 2004. [4] M. Eiz, K. Hildebrand, T. Boubekeur, and M. Alexa, Skech-based image rerieval: Benchmark and bag-of-feaures descripors, IEEE Trans. Visualizaion and Compuer Graphics, 2011, pp.1624-1636. [5] M. Eiz, K. Hildebrand, T. Boubekeur, and M. Alexa, An evaluaion of descripors for large-scale image rerieval from skeched feaure lines, IEEE Trans. Visualizaion and Compuer Graphics, vol. 34, no. 5, pp. 482-498, 2010. [6] K. Hira, and T. Kao, Query by visual example, in EDBT'92. Springer Berlin Heidelberg, 1992, pp. 56-71. [7] Y. Cao, C. Wang, L. Zhang, and L. Zhang, edgel index for large-scale skech-based image search, IEEE CVPR, 2011, pp. 761-768. [8] B. Senger, A. Thayananhan, P. Torr, and R. Cipolla. Model-based hand racking using a hierarchical bayesian filer, IEEE TPAMI, vol. 28, no. 9, pp. 1372-1384, 2006. [9] R. Zhou, L. Chen, and L. Zhang, Skech-based image rerieval on a large scale daabase, ACM MM, 2012, pp. 973-976. [10] M. Cheng, N. Mira, X. Huang, P. Torr, and S. Hu, Global Conras Based Salien Region Deecion, IEEE TPAMI., vol. 37, no. 3, pp. 569-582, 2014. [11] S. Belongie, J. Malik, and J. Puzicha, Shape maching and objec recogniion using shape conexs, IEEE Trans. Paern Anal. Mach. Inell., 2002, pp. 509-522. [12] D. R. Marin, C. C. Fowlkes, and J. Malik. Learning o deec naural image boundaries using local brighness, color, and exure cues, IEEE TPAMI., vol. 26, no. 5, pp. 530-549, 2004. [13] A. Chalechale, and A. Merins, Skech-based image maching using angular pariioning, IEEE TSMCA, vol.35, no. 1, pp. 28-41, 2005. [14] T. Chen, M. Cheng, P. Tan, A. Shamir, and S. Hu, Skech2Phoo: inerne image monage, ACM Trans. Graph., vol.28, no. 124, aricle 124, 2009. [15] M. Eiz, J. Hays, and M. Alexa, How do humans skech objecs? ACM Trans. Graph, vol. 31, no. 4, aricle. 44, 2012. [16] S. Ren, C. Jin, C. Sun, and Y. Zhang, Skech-Based Image Rerieval via Adapive Weighing, ACM ICMR, 2014, pp. 427-430. [17] E. Roman-Rangel, C. Pallan, J. Odobez, and D. Gaica-Perez. Analyzing Ancien Maya Glyph Collecions wih Conexual Shape Descripors, IJCV 2011. [18] Y. Rui, T. S. Huang, and S. Chang, Image Rerieval: curren echniques, promising direcions, and open issues, JVCIR, 1999. [19] Mori, G., Belongie, S., & Malik, J. (2005). Efficien shape machingusing shape conexs. IEEE Transacions on Paern Analysis and Machine Inelligence, 27(11), 1832 1837. [20] J. Li, X. Qian, Y. Tang, L. Yang, and T. Mei, GPS esimaion for places of ineres from social users uploaded phoos, IEEE Trans. Mulimedia 2013, vol.15, no.8,pp.2058 2071. [21] J. Li, X. Qian, Y. Tang, L. Yang, and C. Liu, GPS esimaion from users phoos, in Proc. MMM 2013, pp.118 129. (a) (b) (c) Fig.8. Skech rerieval performances, he firs row conains he op- ranked resul using proposed mehod in [9], and he second row conains he resuls using edgel mehod, and he hird row conains he resuls using he proposed mehod