Multiscale directional filter bank with applications to structured and random texture retrieval

Transcription

1 Pattern Recognition 40 (2007) Multiscale directional filter bank with applications to structured and random texture retrieval K.-O. Cheng, N.-F. Law, W.-C. Siu Centre for Multimedia Signal Processing, Department of Electronic and Information Engineering, The Hong Kong Polytechnic University, Hung Hom, Hong Kong Received 15 July 2005; received in revised form 16 June 2006; accepted 31 July 2006 Abstract In this paper, multiscale directional filter bank (MDFB) is investigated for texture characterization and retrieval. First, the problem of aliasing in decimated bandpass images on directional decomposition is addressed. MDFB is then designed to suppress the aliasing effect as well as to minimize the reduction in frequency resolution. Second, an entropy-based measure on energy signatures is proposed to classify structured and random textures. With the use of this measure for texture pre-classification, an optimized retrieval performance can be achieved by selecting the MDFB-based method for retrieving structured textures and a statistical or model-based method for retrieving random textures. In addition, a feature reduction scheme and a rotation-invariant conversion method are developed. The former is developed so as to find the most representative features while the latter is developed to provide a set of rotation-invariant features for texture characterization. Experimental works confirm that they are effective for texture retrieval Pattern Recognition Society. Published by Elsevier Ltd. All rights reserved. Keywords: Texture characterization; Texture retrieval; Directional filter bank; Multiscale directional filter bank; Rotation-invariant features 1. Introduction As texture is one of the basic attributes of natural images, texture analysis has attracted much attention on areas such as computer vision, content-based image retrieval (CBIR), remote sensing, medical imaging and quality inspection, etc. In particular, much research [1 4] has been done on texture description for CBIR so as to manage the continuously growing multimedia data. The commonly used methods for texture characterization can be divided into three categories; statistical, model-based and filtering approaches [5]. Statistical methods such as co-occurrence features [6,7] describe the tonal distribution in textures. Model-based methods such as Markov random field (MRF) [8] and simultaneous autoregressive (SAR) models [9] provide a description of texture in terms of spatial interaction while Corresponding author. Tel.: ; fax: addresses: k.o.cheng@polyu.edu.hk (K.-O. Cheng), ennflaw@polyu.edu.hk (N.-F. Law), enwcsiu@polyu.edu.hk (W.-C. Siu). filtering approaches including wavelet [10,11], Gabor filters [1,12], steerable pyramid [13] and directional filter bank (DFB) [14] characterize textures in the frequency domain. Among the three categories, MPEG-7 has adopted Gaborlike filtering for the texture description [15]. The rationale behind is that visual cortex is sensitive to localized frequency components [16]. It has been shown that the direction together with scale information is important for texture perception. The filtering schemes, such as Gabor filters and steerable pyramid, are developed for image analysis in a multiple scale and direction manner. Although Gabor filters and steerable pyramid provide higher angular resolution than the wavelet transform, they are overcomplete in both scale and directional decomposition. This in turn implies that they are less computationally efficient than the wavelet approach [17]. For directional decomposition, DFB [18 20] has been proposed as a highly computationally efficient tool. DFB is maximally decimated and so is not overcomplete. One of the main disadvantages of DFB is the lack of multiscale /$ Pattern Recognition Society. Published by Elsevier Ltd. All rights reserved. doi: /j.patcog

2 K.-O. Cheng et al. / Pattern Recognition 40 (2007) property. Recently, pyramidal directional filter banks (PDFB) or contourlet transform [21,22] have been proposed to solve this problem by combining the DFB with Laplacian pyramid (LP) [23]. Although the LP is somehow redundant, the combined approach is still computationally efficient while providing a high angular resolution. In Ref. [24], the PDFB is modified as multiscale directional filter bank (MDFB) to have a fine high-frequency decomposition. In both the PDFB and MDFB, various lowpass filters can be used in the LP while still maintaining perfect reconstruction. Usually, the filters have the stopband edge greater than π/2, e.g. quadrature mirror filter (QMF) and 9-7 biorthogonal filter. However, the use of these lowpass filters has a shortcoming that aliasing occurs in the passbands after downsampling, i.e. those at scales other than the first one. When the DFB is applied on the decimated lowpass image, the aliasing components will be decomposed at the same time. However, the orientations of the aliasing components can be very different from those of non-aliasing components for some directional subbands [25]. Therefore, further analysis should be performed to study and remove the aliasing effect so as to improve the use of MDFB for texture characterization. Besides the use of lowpass filters in the LP, there are still many issues about the use of MDFB for texture characterization. In particular, the retrieval performance for random textures should be improved since most features commonly extracted from filtering approaches lack statistical description. One way to enhance the retrieval performance of the MDFB is to unify it with statistical or model-based approaches. It has been shown in Refs. [4,26] that the unified approaches can take the advantages of filtering approaches for structured textures and statistical or modelbased approaches for random textures. Thus, we will first study ways for characterizing structured and random textures using MDFB. After this pre-classification, MDFB will be used for retrieval of structured textures while modelbased approach will be utilized to retrieve random textures. Furthermore, feature reduction and rotation-invariance are often concerned in practice. The retrieval time usually increases with the number of features so that features used for texture description should be as few as possible while maintaining good retrieval accuracy. On the other hand, the query texture image provided by a user may have different orientations from the texture images stored in the databases of the retrieval system. Therefore, the features representing the texture images should be insensitive to rotation. We studied ways to reduce the number of features as well as use of rotation-invariant features for texture description. In this paper, we focus on various issues associated with the use of the MDFB for texture characterization and retrieval. This includes the aliasing problem, the combination of MDFB and model-based texture description algorithm, feature reduction for structured textures and the development of a rotation-invariant texture description. This paper is organized as follows. Background about MDFB is given Fig. 1. Frequency partitioning in DFB. in Section 2. Then Section 3 addresses the aliasing problem associated with the directional decomposition of bandpass images in the LP. We will also describe a way to adapt the bandlimiting constraint in Refs. [17,25] to the MDFB so as to alleviate the aliasing problem. In Section 4, a measure of texture regularity based on the MDFB features is proposed. With the use of this measure, a hybrid system combining the MDFB and the model-based algorithm is then developed. A feature reduction method targeted for structured textures is also developed so as to speed up the searching. Section 5 describes the development of the rotation-invariant MDFB features. Finally, Section 6 concludes the paper. 2. Background 2.1. Directional filter bank The DFB [18 20] performs directional decomposition with partitioning of a frequency plane in wedge-shaped regions as shown in Fig. 1. The DFB is computationally efficient due to its tree structure. In the first two stages of the tree structure, the DFB has a two-band filter bank structure given in Fig. 2(a). The two-band filter bank consists of two complementary fan-shaped filters H0 2-D (ω) and H1 2-D (ω). To split the input spectrum into two wedge-shaped regions, the input signal is fed into the respective filters for decomposition. Two directional subbands are obtained after decimation on[ a quincunx ] lattice[ using the ] downsampling matrix, Q 0 = or Q =. 1 1 For stages subsequent to the second stage, a twoband filter bank shown in Fig. 2(b) is used. This kind of

3 1184 K.-O. Cheng et al. / Pattern Recognition 40 (2007) Fig. 3. System block diagram for construction of each level in LP, where hl 2-D[n] is the 2-D lowpass filter implemented using the 1-D filter h L[n] through separable filtering. Fig. 2. Two-channel fan-shaped filter bank (a) for the first two stages and (b) for the third and later stages. two-band filter bank still uses two complementary fanshaped filters for spectrum splitting. Since the next level directional components have a parallelogram-shaped support, resampling using some kind of unimodular matrix R i is required to change the support into fan-shaped for decomposition. [ There ] are four [ kinds] of unimodular [ ] matrices, R 0 =, R =, R = and [ ] R 3 =. Their uses depend on the shape of the 1 1 support region of the required directional components. After splitting, the signals are then decimated by matrix Q k to produce two subband signals. Further decomposition can be achieved by feeding the subband signals into this type of two-channel filter bank. Apart from direct implementation, there is an efficient implementation of the two-band filter banks with a ladder structure [27]. Under the ladder structure, signals are filtered after downsampling. In addition, separable filtering can be used. For the downsampling matrix Q 1, the resulting two complementary fan-shaped filters are given as follows: H0 2-D (z 0,z 1 ) = z 2N 0 z0 1 β( z 0z1 1 )β( z 0z 1 ), (1) 2 H1 2-D (z 0,z 1 ) = z0 4N+1 β( z 0 z1 1 )β( z 0z 1 ) H 0 (z 0,z 1 ), (2) where β(z) is a 1-D lowpass filter, which has even length, N, and linear phase for the FIR design Multiscale DFB As the DFB does not possess multiscale property, it has been combined with certain multiscale schemes for image analysis [21,22,24]. For example, the LP is used in Refs. [21,22]. Unlike the critically sampled wavelet scheme, its bandpass images do not suffer from frequency scrambling. The oversampled multiscale scheme has been improved for texture characterization [24] to provide one more bandpass image by splitting the first bandpass image in the LP into two bandpass images. The lowpass image is not considered for texture characterization. This is due to the non-ideal frequency responses in real implementation which causes uneven distribution of low-frequency components into different directional subbands [14]. Another reason for omitting the lowpass image is that textures are often well characterized by the high- and mid-frequency components [28]. In the MDFB, a 1-D lowpass filter h l [n] of cut-off frequency ω l is firstly applied on the input image on rows and columns separately. In Ref. [24], the cut-off frequency ω l is 3π/4. The first scale subimage is then obtained as the difference of the input image and the lowpass image. Subsequent multiscale decomposition on this lowpass image is performed by using the LP, as depicted in Fig. 3. Therefore, the subimage i(i 2) is the bandpass image of the level i 1 in the LP. The 1-D prototype lowpass filter h L [n] used in the LP usually has stopband edge ω s which is greater than π/2 and gives rise to aliasing. Despite that, the LP still allows perfect reconstruction by using the difference of the original image and the approximation from the lowpass image. After performing multiscale decomposition, the DFB is applied independently on each subimage for directional analysis. For subsequent discussion, a multiscale directional decomposition is represented as (d 1,d 2,...,d n ), where d i is the number of directional decompositions at scale i and n is the total number of bandpass images (scales). An example of frequency partitioning is given in Fig Choice of h L [n] without aliasing effect The function of the 1-D lowpass filter h L [n] in LP is to decompose an image iteratively into a set of bandpass images of passbands [ π, π] 2 \[ ω c, ω c ] 2, {[ ω c /2 i 1, ω c /2 i 1 ] 2 \[ ω c /2 i, ω c /2 i ] 2 i = 1, 2,...,S 1} and [ ω c /2 S 1, ω c /2 S 1 ] 2, where S is the number of decomposition levels and ω c is the cut-off frequency of h L [n]. In MDFB, ω c is normally selected to be π/2. However, non-ideal filters h L [n] usually have non-zero values at frequency higher than π/2. According to Shannon sampling theorem, there would be aliasing after downsampling by 2. The effect of aliasing on a multiscale decomposition is usually termed a shift

4 K.-O. Cheng et al. / Pattern Recognition 40 (2007) Fig. 4. Frequency partitioning (colored regions) of a frequency plane for multiscale directional decomposition (4, 8, 4). Fig. 6. Illustration of aliasing in the frequency domain. (a) Ideal frequency response of a non-decimated directional subband, HD 0,z 1 ). (b) Ideal frequency response of the directional subband for the decimated bandpass image at the second level of the LP, HB 0,z 1 )HD 0,z 1 ). (c) Frequency response of the 2-D lowpass filter HL 0,z 1 ) commonly used in the LP. (d) Ideal frequency response of the non-decimated directional subband in the second scale HB 2 0,z2 1 )H 2-D D (z2 0,z2 1 ) with dashed line indicating the filtering of the non-ideal lowpass filter HL 0,z 1 ). Fig. 5. Equivalent filter bank for one of the directional subbands in the second scale (a) with upsampled filters in the DFB (HD 0,z 1 )) and (b) with upsampled filter (HSD 0,z 1 )) in the MDFB. variance problem in the time/spatial domain [17,29]. One can also perform a frequency domain analysis on aliasing [25]. Consider the PDFB that consists of the three levels LP and three levels DFB at the second scale. The filter bank for one of the directional subbands in the second scale can be realized as in Fig. 5(a). Using noble identity [30], the block diagram of the filter bank can be redrawn in terms of one non-decimated filter HSD 0,z 1 ) and one downsampler as shown in Fig. 5(b). The non-decimated filter response is given by HSD 2-D (z 0,z 1 ) = HL 2-D (z 0,z 1 )HB 2-D (z2 0,z2 2-D 1 )HD (z2 0,z2 1 ), (3) where HL 0,z 1 ) is the 2-D lowpass filter in the LP, HB 0,z 1 ) is the bandpass filter used to model the system response of the bandpass image of the LP and HD 0,z 1 ) is the upsampled filter of one of the subbands in the DFB. As the transition band of the bandpass response of LP will not be concerned, the use of HB 0,z 1 ) to model the bandpass response will not affect the following discussion. Furthermore, HB 0,z 1 ) can be approximated in terms of (z 0,z 1 ) as H 2-D L HB 2-D (z 0,z 1 ) = HL 2-D (ω) 2. (4) Hence, HB 0,z 1 ) is a bandpass filter with passband [ π, π] 2 \[ π/2, π/2] 2. The ideal frequency responses of the bandpass directional component before and after upsampling, i.e. represented by terms HB 0,z 1 )HD 0,z 1 ) and HB 2 0,z2 1 )H 2-D D (z2 0,z2 1 ), are illustrated in Fig. 6(b) and (d), respectively. In practice, the filterhl 0,z 1 ) is non-ideal which means that there is non-zero frequency response outside the passband region [ π/2, π/2] 2, especially near the transition band as shown in Fig. 6(c). This results in passing undesired response of HB 2 0,z2 1 )H 2-D D (z2 0,z2 1 ) as indicated by the dashed line in Fig. 6(d). This undesired response comes from adjacent scale. However, its orientation is very different from the desired response. There is about 73.7 deviation. In fact, this deviation in orientation is subband dependent. For subbands closer to direction ±45, the difference in orientation is larger. For example, with directional subband given in Fig. 7(a), less than half of deviation in the case of Fig. 6(d) is found as illustrated by Fig. 7(b). Whatever thedirectional subbands involved, the aliasing must be avoided in order to have a precise decomposition for directional analysis in each scale.

5 1186 K.-O. Cheng et al. / Pattern Recognition 40 (2007) Fig. 7. Aliasing in the frequency domain for another directional subband. (a) Ideal frequency response of a non-decimated directional subband, HD 0,z 1 ). (b) Ideal frequency response of the non-decimated directional subband in the second scale HB 2 0,z2 1 )H 2-D D (z2 0,z2 1 ) with dashed line indicating the filtering of the non-ideal lowpass filter HL 0,z 1 ). drawback of using a lowpass filter with stopband edge at π/2 is broadening of the bandwidth of the highest frequency subband. In these cases, the additional splitting of the highest frequency band in the MDFB becomes more important in order to maintain the radial frequency resolution for texture characterization. In this paper, the lowpass filter h l [n] for the additional splitting is designed so that the highest frequency band is split into two bands of nearly equal bandwidth. In particular, h l [n] is designed such that its frequency response H l (ω)=0.5 when ω=0.65π for non-aliasing filters, binom7 and ER13. For aliasing filters, QMF13 and Daub14, H l (ω)= 0.5 when ω = 0.75π. In summary, with the use of bandlimiting constraints for h L [n], precise directional decomposition can be achieved in downsampled bandpass images in LP. However, there would be a drawback of broadening the highest frequency passband. In order to maintain high radial frequency resolution, the highest frequency passband is split into two passbands of approximately equal bandwidth using a lowpass filter h l [n] Retrieval based on MDFB Fig. 8. Filter response of the QMF filter of length 13 (QMF13) and Daubechies filter of length 14 (Daub14), binomial filter of length 7 (binom7) and the designed equiripple filter of length 13 (ER13). In order to reduce the undesired filtering, the aliasing in the LP is minimized by imposing the bandlimiting constraint to ensure zero response above π/2 onh L [n] [17,25]. For example, a binomial filter of length 7 can be selected because its stopband energy is significantly small at frequency larger than π/2. Apart from binomial-type filters, the LP filter can be designed using Parks McClellan algorithm [31] by requiring the stopband edge to be π/2. Fig. 8 shows the magnitude responses of four types of filters, (1) binomial filter of length 7 (binom7), (2) an equiripple filter of length 13 (ER13) designed using Parks McClellan algorithm given the bandlimiting constraint and two lowpass filters with cutoff frequency π/2 of similar length, (3) QMF of length 13 (QMF13) and (4) Daubechies filter of length 14 (Daub14). As binom7 and ER13 have stopband edge at π/2 so they are referred as non-aliasing filters whilst QMF13 and Daub14 filters have non-zero values beyond π/2 so they are referred as aliasing filters. From Fig. 8, it is obvious that the Each subband in MDFB represents texture characteristics in a particular scale and direction. Thus, energy signatures of subbands in MDFB can be used as texture features. The commonly used energy signatures are L 1 and L 2 norms [3]. From our experiences, L 1 norm gives slightly better retrieval performance for textures and is used throughout this paper. The L 1 norm energy signature for the subband of scale s and direction d is defined as f s,d = 1 N N x s,d,n, (5) n=1 where {x s,d,n } is the set of the subband coefficients and N is the total number of coefficients in that subband. The feature vector is obtained by concatenating the features at different scales and directions as f = (f 1,1,f 1,2,...,f S,DS 1,f S,DS ), (6) where S and D i are the total number of scales and the total number of directions for the ith scale, respectively. For similarity measure of two texture images, the weighted sum of absolute difference is used. The weighted sum of absolute difference between images u and v is computed by d(u, v) = S D s s=1 d=1 f u s,d f v σ s,d s,d, (7) where the weight σ s,d is the standard deviation of features at scale s and direction d over the entire database. In retrieval, K textures with the least distance from a query texture are returned as the results, where K is set to be the size of the query texture s class in the database. The retrieval accuracy is evaluated as the average percentage of correctly retrieved images for the queries.

6 K.-O. Cheng et al. / Pattern Recognition 40 (2007) Table 1 Retrieval accuracy (%) of MDFB based on aliasing and non-aliasing filters for different number of scales Decomposition QMF13 Daub14 Binom7 ER13 (8, 8, 8) (8, 8, 8, 8) (8, 8, 8, 8, 8) Table 2 Retrieval accuracy (%) for same number of directional decompositions in all the four scales Decomposition Binom7 ER13 (4, 4, 4, 4) (8, 8, 8, 8) (16, 16, 16, 16) (32, 32, 32, 32) Retrieval experiment using non-aliasing LP decomposition To investigate texture retrieval based on the MDFB with non-aliasing filters h L [n], a texture retrieval experiment on a database derived from 111 images in the Brodatz texture album [32] is performed. Each of the 111 album images is a bit gray level image. To construct the database, nine non-overlapping subimages are extracted at the center of each album image. This results in 999 images of 111 texture classes. Every subimage extracted from the same album image is considered to be in the same texture class. To prevent the bias due to similar intensity in images of the same class, all images are subjected to histogram equalization. In order to study the influence of aliasing, we vary the number of scales but fix the number of directional decomposition to be 8 for each scale in this experiment. Filters used for h L [n] include those discussed in the previous section, i.e. QMF13, Daub14, binom7 and ER13. A lowpass filter h l [n] of half amplitude at frequency 0.75π is associated with aliasing filters h L [n] for further decomposition in the highest frequency band. For non-aliasing filters h L [n], h l [n] with half amplitude at frequency 0.65π is used. Table 1 summarizes retrieval accuracies for the MDFB based on different scale decompositions. For the same number of scales, the retrieval accuracy obtained using nonaliasing filters is always higher than that using aliasing filters. With non-aliasing filters, the highest retrieval accuracy is achieved when four scales are used. The retrieval accuracy is about 72%, which is higher than those with aliasing filters by approximately 4% for the same number of scales and 2% for the best results. From the results of non-aliasing filters, it can also be found that the retrieval accuracy increases up to four scales and has no improvement when the fifth scale, which corresponds to a scale of frequency below π/16, is added. This confirms our discussion in Section 2.2 that the mid- to high-frequency texture information is more important than the low-frequency information. In the cases of aliasing filters, the retrieval performance is improved up to five scales. However, it should be noted that the aliasing filters have bandwidth about double of that of the non-aliasing ones (as illustrated in Fig. 8) so their fifth scale corresponds to the frequency range of the fourth scale in the cases of the non-aliasing filters. In addition, the retrieval accuracy also tends to be stable when the number of scales increases. The low-frequency components, probably Table 3 Retrieval accuracy (%) for various combinations of 4 and 8 directional decompositions in the four scales Decomposition Binom7 ER13 (4, 4, 4, 8) (4, 4, 8, 4) (4, 8, 4, 4) (8, 4, 4, 4) (4, 4, 8, 8) (4, 8, 4, 8) (8, 4, 4, 8) (4, 8, 8, 4) (8, 4, 8, 4) (8, 8, 4, 4) (4, 8, 8, 8) (8, 4, 8, 8) (8, 8, 4, 8) (8, 8, 8, 4) The highest retrieval accuracy among the combinations of same set of directional decompositions is highlighted. the scales higher than the fifth one, should have no or very little improvement in the cases of aliasing filters as well Retrieval with different directional decomposition in scales As shown in Table 1, four scales are important for texture retrieval in the MDFB with the non-aliasing filters h L [n]. In this section, we investigate the retrieval performance of different directional decomposition in the scales through intense simulations. First, let us fix the number of directional decompositions to be 4, 8, 16 and 32 in all the four scales. The results are given in Table 2. It can be seen that when the number of directions is 8, the retrieval accuracy is the highest for both the two non-aliasing LP filters. Second, we perform further analysis using different combinations of 4 and 8 directional decompositions in the four scales. Note that, increasing the number of directional components increases the angular resolution, while at the same time increases the number of features used in texture characterization and retrieval. The retrieval accuracies for these combinations are summarized in Table 3. With two or more scales of 8 directional decompositions, the retrieval accuracy can

7 1188 K.-O. Cheng et al. / Pattern Recognition 40 (2007) be maintained very close to the decomposition (8, 8, 8, 8). For ER13 filter, decomposition (4, 8, 8, 4) has retrieval accuracy about 0.4% less than that of decomposition (8, 8, 8, 8) but with 8 features less. The retrieval accuracy of decomposition (4, 8, 8, 8) is 0.1% less while the number of features used is 4 less. From these results, retrieval performance is higher in most cases when higher angular resolution is used in mid-frequency range. For example, decomposition (4, 8, 8, 8) has retrieval accuracy which is 0.4% higher than that of decomposition (8, 8, 8, 4). This can be explained by the brief that mid-frequency components are more important for characterizing textures [10]. Similar experiments have been conducted using different combinations of 8 and 16 directional decompositions in the four scales. However, the highest improvement was found to be about 0.1% despite that more features were used. From these experiments, we can see that the optimal number of orientations is about 8. For the first and fourth scales, the number of orientations can be reduced to be 4 without significant drop in retrieval accuracy because the mid-frequency components can well describe textural features Performance comparison The performance of the MDFB for texture retrieval has been compared with the Gabor filters in polar form [12] and steerable pyramid [17] in terms of retrieval accuracy and computational complexity. The comparison results are summarized in Table 4. For texture retrieval, the database used is the same as before. The L 1 norm and sum of absolute difference are used as subband features and similarity measure, respectively. The number of scales of Gabor filters and steerable pyramid is 4 as it is frequently used in literature. In fact, our results of the MDFB in Section 3.2 that the frequency components below π/16 are insignificant for texture characterization agree with the use of the first four scales. For Gabor filters, different numbers of orientations have been tested and the results are provided in Table 5. The highest retrieval accuracy of 72.5% is achieved for six orientations in the four scales. For the steerable pyramid, six orientations and four scales decomposition is performed. The retrieval accuracy is found to be 69.6%. In Table 4, these results are compared with the MDFB approach using decomposition (4, 8, 8, 8) for ER13 filter. It can be seen that the retrieval performance of the MDFB is comparable to the Gabor filters while higher than steerable pyramid by 2.5%. In terms of the number of features, the MDFB approach uses 28 features, which are more than the 24 features in both the Gabor filters and steerable pyramid cases. However, in Section 4.3, a feature reduction scheme for the MDFB approach will be proposed which can significantly lower the computational complexity. For the complexity in feature extraction, the MDFB is the best. Unlike these two schemes, the directional decomposition in MDFB is not overcomplete. For decomposition with S scales and D directions of an image of size N N, MDFB has complexity of O(N 2 log 2 D) as given in Table 4. The complexity of steerable pyramid is higher and proportional to number of directions D because of its overcomplete directional decomposition. For Gabor filters, if the filtering is performed in frequency domain, the complexity is O(DSN 2 log 2 N). The complexity of Gabor filters is the highest as the decomposition is overcomplete in both scales and directions. In summary, the MDFB approach has a low computational complexity while gives comparable retrieval accuracy as Gabor filters and steerable pyramid. 4. Use of MDFB in classification of structured textures A way to classify textures is to specify its perceptual characteristics such as regularity. Regularity is closely related to directionality [33]. We have seen that the MDFB can be used to capture texture characteristics at various scales and orientations. Hence, the MDFB can be used to characterize texture regularity. Regular or structured textures usually consist of dominant periodic patterns. Therefore, orientations of the patterns are consistent throughout. For random or unstructured textures, there is no well-defined directionality so the distribution of signal energy should be roughly uniform over all orientations. In our classification algorithm, the directionality is measured based on the similarity of energy signatures. Shannon entropy, which gives a measure of concentration for a sequence [34], is used. For a directional energy sequence, {e d }, Shannon entropy is calculated as E({e d }) = ( ed ) 2 ( ed ) 2, ln e e d ( 1/2 where e = ed) 2. (8) d A large value of E is obtained if energy values are nearly the same, and vice versa. In MDFB, each scale will give one value of E. To determine whether a texture is structured or random, the minimum of the entropy over all scales is used as the overall regularity measure of textures. The reason is that a structured texture should have dominant directional features in at least one scale. The regularity of a texture is thus given by R = min s E s, (9) where E s is the entropy of the normalized directional energy sequence at scale s. If the regularity is below a pre-set threshold, the texture is classified to be structured. Otherwise, the texture is classified as a random texture. The threshold can be determined to have the value for the best retrieval performance on a given training set of textures. It should be noted that random textures may be confused with textures with circular patterns such as tree ring patterns. The textures with circular patterns would give low value of R even if the patterns are arranged regularly.

8 K.-O. Cheng et al. / Pattern Recognition 40 (2007) Table 4 Summary of comparison results Approach Gabor filters Steerable pyramid MDFB Retrieval accuracy (%) Number of features Complexity in feature extraction O(DSN 2 log 2 N) O(DN 2 ) O(N 2 log 2 D) Table 5 Retrieval accuracy of four scales Gabor wavelet with various number of orientations Number of orientations Retrieval accuracy (%) Number of orientations Retrieval accuracy (%) Table 6 Entropy-based regularity of six texture samples Texture D1 D17 D74 D86 D91 D102 (8,8,8,8) (16,16,16,16) The dynamic range of entropy is different for different number of directions Finding structured and random content In this part, classification performance of structured and random textures using the MDFB-based features is examined. Table 6 gives the regularity value R based on decomposition (8, 8, 8, 8) and (16, 16, 16, 16) of six texture samples from the Brodatz texture album as illustrated in Fig. 9. Images D1, D17 and D102 are regarded as structured while D74, D86 and D91 are considered to be random. The regularity value of structured texture D1 is significantly smaller than that of random texture D74 in both decompositions. However, the more complicated structured texture D17 has higher regularity value than the random textures D86 and D91 when decomposition (8, 8, 8, 8) is used. For decomposition (16, 16, 16, 16), the situation is improved so that the regularity value of D17 is smaller than that of D86 and D91. If the threshold for classification is set to be 2.36, the structured and random textures can be classified successfully. This implies that having higher directional decomposition such as (16, 16, 16, 16) is necessary for classifying structured textures Pre-classification of textures before retrieval As a filtering approach, the MDFB features can characterize periodic structured textures effectively. For random textures, the MDFB approach may not work as effective as statistical approaches and model-based approaches due to ambiguous directionality in their patterns. In order to optimize the performance of the MDFB method for a more general texture database, i.e. the one with structured textures as well as random textures, a hybrid system which combines features from MDFB and a model-based approach called multiresolution simultaneous autoregressive (MR- SAR) model [35] is investigated and implemented. First, a query texture is classified to be either structured or random using the method described in Section 4.1. If the query texture is classified to be structured, features extracted from MDFB are selected for retrieval. Otherwise, features extracted from MRSAR models are used. As shown in Section 4.1, the multiscale directional features can distinguish structured and random textures. Therefore, the multiscale directional features can be used for pre-classification of textures in structure as well as in the retrieval of any query texture classified to be structured. Experiments have been performed to investigate the retrieval performance of the hybrid system. To avoid the bias due to unequal number of structured and random textures, a database composed of 20 classes for each structured and random textures is selected. The images are derived from the Brodatz album images, D1, D4, D6, D8, D15, D21, D22 D23, D27, D30, D34, D49, D50, D51, D52, D53, D54, D55, D56, D58, D62, D63, D66, D67, D68, D73, D75, D77, D79, D83, D85, D90, D92, D95, D99, D100, D106, D108, D111 and D112. For the hybrid retrieval system, the MDFB approach with decomposition (16, 16, 16, 16) is used. The lowpass filter in LP is ER13. For MRSAR approach, Gaussian filter is chosen for multiscale decomposition and the number of scales is four. From each scale, four SAR coefficients and standard deviation of estimation error are calculated as the features. This gives 20 features in total. The weighted sum of absolute difference similar to Eq. (7) is

9 1190 K.-O. Cheng et al. / Pattern Recognition 40 (2007) Table 7 Comparison for the hybrid retrieval system, the retrieval system based on MDFB only and that based on MRSAR only Database Retrieval accuracy (%) for hybrid system Retrieval accuracy (%) for MDFB Retrieval accuracy (%) for MRSAR Quantity of texture images All Structured Random lower retrieval accuracy but is compensated in the case of the hybrid system. The experimental results justify the use of pre-classification based on entropy for optimizing the retrieval performance of MDFB Feature reduction for structured textures Fig. 9. Six texture samples used in regularity measurement in Table 6. Images from left to right and top to bottom: D1, D17, D74, D86, D91 and D102. used as their similarity measure. In the pre-classification, the threshold for regularity is chosen so as to achieve the highest retrieval accuracy. The experimental results are summarized in Table 7. The overall retrieval accuracy of the hybrid system is 81.5% which is 2.2% and 2.3% higher than those of MDFB and MRSAR approaches, respectively. In the retrieval, 175 samples are classified to be structured while 185 samples are regarded as random. The pre-classification results closely match the texture distribution in the databases. As shown in the table, the retrieval accuracy of MDFB for structured textures is 96.8% and higher than that of MRSAR by 4.7%. For random textures, the MDFB approach gives Besides the retrieval accuracy, another main concern in texture retrieval is the searching time. As the number of computations in similarity measurement increases with the number of features, an efficient retrieval scheme should use features as few as possible to speed up the searching. As discussed in Section 4.2, structured textures are extracted and processed using the MDFB approach in the hybrid system. Feature reduction method for MDFB can then be focused for structured textures only. For the wavelet packet energy features used in Ref. [36], a few most dominant energy signatures were selected for texture representation whilst the performance could still be maintained at a high level. A similar feature reduction scheme is used here because structured textures usually have significant energy in the dominant orientations. The m highest energy signatures of a query texture and the corresponding subband energy signatures of textures in the database are selected for similarity measurement. However, somewhat unlike in the scheme of wavelet packet energy features, the corresponding features of a texture in the database, which do not belong to its m most dominant energy signatures, are set to zero. The reason is that the non-dominant energy signatures usually correspond to subbands of little directional information and can be greatly affected by noises. By setting them to zero, the retrieval error can be reduced. Fig. 10 shows an example of reconstruction of a structured texture using the lowpass subband and different numbers of dominant subbands. Here, dominant subbands refer to the subbands with large energy signature. With only the lowpass subband, most of textural characteristics are lost. By increasing the number of dominant subbands, the structure of the texture in the reconstructed images becomes clearer. The reconstructed image looks similar to the original one when first 28 dominant subbands are included in the reconstruction. This shows the importance of dominant subbands for structured texture description. Further experiments were performed to evaluate the feature reduction scheme for the MDFB features in texture retrieval. The setup of the experiment was the same as in Section 4.2 except that the proposed feature reduction

10 K.-O. Cheng et al. / Pattern Recognition 40 (2007) structured textures versus the number of features is plotted in Fig. 11. From the results, high retrieval accuracy can be found even for small feature sets. The highest retrieval rate is achieved when 37 features are used. The value is 97.8%, which results in 82.0% overall retrieval accuracy in the hybrid system. The minimum number of features for performance comparable to the complete set of features is 19, which is less than one-third of the complete feature set. The results show that the proposed feature reduction method based on the MDFB features is effective for structured textures. 5. Rotation-invariant features Each band of the DFB corresponds to a frequency region at a particular orientation. If the input image is rotated, the sequence of subband energies is circularly shifted. Similar to the case of Gabor filtering [37], the magnitudes of discrete Fourier transform (DFT) coefficients of a directional energy sequence can be used as rotation-invariant features. To accommodate this idea to features extracted from the MDFB, the DFT is applied on the features weighted by global standard deviation in each scale, i.e. {ê s,d }=DFT {e s,d /σ s,d } for an energy sequence {e s,d } in scale s. A rotation-invariant feature vector can then be formed as fˆ s = ( ê s,1, ê s,2,..., ê s,ds ), (10) Fig. 10. Reconstruction of structured texture D1. Images from left to right and top to bottom: original image, reconstructed images using the lowpass subband and the first m dominant subbands, m = 0, 14, 28, 42 and 56. Fig. 11. Retrieval accuracy (%) versus number of features used. method was applied to the MDFB features. The 175 query textures, which were pre-classified to be structured, were investigated. A graph showing the retrieval accuracy of the where D s is the number of directions in scale s. Due to symmetry of magnitude spectrum, almost half of the size of the feature vector can be reduced. The feature vector can then be formed as f ˆ s = ( ê s,1,a ê s,2,...,a ê s,ds /2, ê s,ds /2+1 ), (11) where a is the scaling factor for coefficients except those at frequency 0 and π, which is used to adjust the similarity measure based on f ˆ s to be the same as that based on fˆ s. For example, a = 2 for Euclidean distance and a = 2 for sum of absolute difference, respectively. The rotationinvariant multiscale directional feature vector is given by concatenating the feature vector in each scale as f ˆ = ( f ˆ 1, f ˆ 2,..., f ˆ S ), (12) where S is the total number of scales. Since the feature vectors have changed, some modifications are required in the retrieval algorithms as presented in Sections 3 and 4. First, sum of absolute difference without weighting is used as the similarity measure. Second, if pre-classification for texture structure is desired, it should be performed before applying the DFT. Besides these two modifications, other steps remain unchanged. The Brodatz texture album is used to test the retrieval performance using the proposed rotation-invariant multiscale directional features. Each Brodatz texture image is divided into four non-overlapped regions of size After

11 1192 K.-O. Cheng et al. / Pattern Recognition 40 (2007) that, each divided region is rotated from 0 to 165 with a step size of 15 to generate 12 rotated subimages. The center part of each rotated subimage is then cropped to have a size of The database is thus formed with 5328 images and each class consists of 48 texture images. The performance for two decompositions (8, 8, 8, 8) and (16, 16, 16, 16) based on LP with filter ER13 are examined. The retrieval accuracy and number of features are given in Table 8. The results for features without rotation-invariant conversion are included for reference only. Obviously, the rotation-invariant features give much higher retrieval accuracy than the original ones. It can also be found that unlike the case without rotation, when the number of angular decomposition increases from 8 to 16, the retrieval accuracy is improved from 59.2% to 60.3%. This is due to the fact that high angular resolution is essential to distinguish finely rotated texture images. Comparative studies about rotation-invariant texture classification were performed with one existing approach using rotation-invariant polar-wavelet texture features [2]. Retrieval experiment for our proposed algorithm was performed on the same database used in the reference paper [2]. The database is a subset of the database used previously with 25 texture classes only. The retrieval accuracies of our proposed algorithm and the algorithm using polarwavelet features are provided in Table 9. Our proposed algorithm uses a significantly smaller number of features than the polar-wavelet features. Moreover, the retrieval performance of the case with decomposition (16, 16, 16, 16) provides about 2% increase in retrieval accuracy while the number of features used is only 36, in comparison to 96 in the polar-wavelet approach. This shows the effectiveness of our proposed rotation-invariant features based on MDFB. Table 8 Comparison in retrieval accuracy (%) between using the proposed rotationinvariant multiscale directional features and the proposed features in Section 3 for the rotated texture database Decomposition Rotation-invariant Non-rotation-invariant (8, 8, 8, 8) (16, 16, 16, 16) Conclusions The use of multiscale directional filter bank (MDFB) is analyzed for texture characterization and retrieval in this paper. First, we study the influence of aliasing in Laplacian pyramid (LP) on the directional decomposition of bandpass images. It is found that the aliasing mixes frequency components in different orientations which results in inaccurate directional decomposition. Hence, the multiscale decomposition in MDFB is designed by considering a bandlimiting constraint on the lowpass filter used in LP so as to eliminate the aliasing. Experimental results show that the features extracted based on MDFB without aliasing in the LP decomposition give higher retrieval accuracy. On the other hand, it is found that finer directional decomposition in mid-frequency range usually results in better texture characterization as most texture patterns are quasi-periodic in nature. Our comparative study shows that the MDFB approach is comparable to Gabor filters in polar form but better than steerable pyramid in retrieval accuracy. However, directional decomposition in MDFB is maximally decimated. At the same time, the filtering in MDFB is separable in nature. These imply that the computational complexity of the MDFB is much smaller than that of the Gabor filters and steerable pyramid. The MDFB has been utilized for classifying structured and random textures. The classification is based upon the similarity measurement of directional subband energy signatures in each scale using the Shannon entropy. With a pre-classification of structured textures, a hybrid retrieval system is developed in which the MDFB features are selected for characterizing structured textures, while a modelbased method multiresolution simultaneous autoregressive (MRSAR) model is used for characterizing random textures. Experimental results show that for a database of 40 classes with equal number of structured and random textures, the overall retrieval accuracy of the proposed hybrid system can be improved by about 2.2% as compared with the approach based only on MDFB. In addition, a feature reduction scheme has been proposed to select the most characteristics MDFB features for texture representation. Finally, MDFB-based rotation invariant features have been proposed. They are computed as the magnitudes of the discrete Fourier transform (DFT) of the directional energy signatures in each scale. Compared with the existing Table 9 Comparisons of the proposed rotation-invariant multiscale directional features and rotation-invariant polar-wavelet features Features used in retrieval Proposed approach with decomposition (8, 8, 8, 8) Proposed approach with decomposition (16, 16, 16, 16) Rotation-invariant polarwavelet features a Retrieval accuracy (%) Number of features a The results of rotation-invariant polar-wavelet features are cited from Ref. [2].

12 K.-O. Cheng et al. / Pattern Recognition 40 (2007) approach using rotation-invariant polar-wavelet features, our approach improves retrieval performance for rotated textures but uses less number of features. Hence, the MDFB is practical for most applications. Throughout all the experiments, it is also observed that the number of directions required varies for different applications. For non-rotated texture retrieval, moderate angular resolution, e.g. 8 directions, is sufficient. For rotated texture retrieval or classification of structured textures, more directional decompositions, e.g. 16 directions, is required to have higher precision. Besides rotation, images encountered in real situations may be different in scales. In the future, it is interesting to develop a set of features based on MDFB, which are insensitive to scale change as well as rotation, for texture retrieval. Acknowledgments The authors thank the anonymous reviewers for their constructive comments. This work is supported by the Department of Electronic and Information Engineering, Centre for Multimedia Signal Processing, The Hong Kong Polytechnic University, and the CERG Grant (PolyU 5222/03E) of the Hong Kong SAR Government. K.-O. Cheng thanks The Hong Kong Polytechnic University for the support he receives under the research Grant A452. References [1] B.S. Manjunath, W.Y. Ma, Texture features for browsing and retrieval of image data, IEEE Trans. Pattern Anal. Mach. Intell. 18 (8) (1996) [2] C.-M. Pun, M.-C. Lee, Rotation invariant texture feature for content based image retrieval, in: Proceedings of IEEE International Conference on Multimedia and Expo, 2002, pp [3] M.N. Do, M. Vetterli, Wavelet-based texture retrieval using generalized Gaussian density and Kullback Leibler distance, IEEE Trans. Image Process. 11 (2) (2002) [4] F. Liu, R.W. Picard, Periodicity, directionality, and randomness: wold features for image modeling and retrieval, IEEE Trans. Pattern Anal. Mach. Intell. 18 (7) (1996) [5] T. Randen, J.H. HusZy, Filtering for texture classification: a comparative study, IEEE Trans. Pattern Anal. Mach. Intell. 21 (4) (1999) [6] R.M. Haralick, Statistical and structural approaches to texture, Proc. IEEE (1979) [7] G.V.D. Wouwer, P. Scheunders, D. Van Dyck, Statistical texture characterization from discrete wavelet representations, IEEE Trans. Image Process. 8 (4) (1999) [8] G.R. Cross, A.K. Jain, Markov random field texture models, IEEE Trans. Pattern Anal. Mach. Intell. PAMI-5 (1) (1983) [9] R.L. Kashyap, R. Chellappa, Estimation and choice of neighbors in spatial-interaction models of images, IEEE Trans. Inf. Theory IT-29 (1) (1983) [10] T. Chang, C.-C.J. Kuo, Texture analysis and classification with treestructured wavelet transform, IEEE Trans. Image Process. 2 (4) (1993) [11] M. Unser, Texture classification and segmentation using wavelet frames, IEEE Trans. Image Process. 4 (11) (1995) [12] G.M. Haley, B.S. Manjunath, Rotation-invariant texture classification using a complete space frequency model, IEEE Trans. Image Process. 8 (2) (1999) [13] W.T. Freeman, E.H. Adelson, The design and use of steerable filters, IEEE Trans. Pattern Anal. Mach. Intell. 13 (9) (1991) [14] J.G. Rosiles, M.J.T. Smith, Texture classification with a biorthogonal directional filter bank, in: Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing, May 2001, pp [15] T. Sikora, The MPEG-7 visual standard for content description an overview, IEEE Trans. Circuits Syst. Video Technol. 11 (6) (2001) [16] J.J. Kulikowski, P.O. Bishop, Fourier analysis and spatial representation in the visual cortex, Experientia 37 (1981) [17] E.P. Simoncelli, W.T. Freeman, The steerable pyramid: a flexible architecture for multi-scale derivative computation, in: Proceedings of International Conference on Image Processing, October 1995, pp [18] R.H. Bamberger, M.J.T. Smith, A filter bank for the directional decomposition of images: theory and design, IEEE Trans. Signal Process. 40 (4) (1992) [19] S.-I. Park, M.J.T. Smith, R.M. Mersereau, A new directional filter bank for image analysis and classification, in: Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing, March 1999, pp [20] S.-I. Park, M.J.T. Smith, R.M. Mersereau, Improved structures of maximally decimated directional filter banks for spatial image analysis, IEEE Trans. Image Process. 13 (11) (2004) [21] M.N. Do, M. Vetterli, Pyramidal directional filter banks and curvelets, in: Proceedings of International Conference on Image Processing, October 2001, pp [22] M.N. Do, M. Vetterli, The contourlet transform: an efficient directional multiresolution image representation, IEEE Trans. Image Process. 14 (12) (2005) [23] P.J. Burt, E.H. Adelson, The Laplacian pyramid as a compact image code, IEEE Trans. Commun. COM-31 (4) (1983) [24] W.Y. Chan, N.F. Law, W.C. Siu, Multiscale feature analysis using directional filter bank, in: Proceedings of the 2003 Joint Conference of the Fourth International Conference on Information, Communications and Signal Processing, 2003 and the Fourth Pacific Rim Conference on Multimedia, December 2003, pp [25] T.T. Nguyen, The multiresolution directional filter banks, Ph.D. Thesis, Department of Electrical Engineering, The University of Texas at Arlington, US, August [26] S.C. Zhu, Y. Wu, D. Mumford, Filters, random fields and maximum entropy (FRAME): towards a unified theory for texture modeling, Int. J. Computer Vision 27 (2) (1998) [27] S.-M. Phoong, C.W. Kim, P.P. Vaidyanathan, R. Ansari, A new class of two-channel biorthogonal filter banks and wavelet bases, IEEE Trans. Signal Process. 43 (3) (1995) [28] T. Chang, C.-C. Jay Kuo, Texture analysis and classification with tree-structured wavelet transform, IEEE Trans. Image Process. 2 (4) (1993) [29] E.P. Simoncelli, W.T. Freeman, E.H. Adelson, D.J. Heeger, Shiftable multiscale transforms, IEEE Trans. Inf. Theory 38 (2) (1992) [30] P.P. Vaidyanathan, Multirate Systems and Filter Banks, Prentice-Hall, Englewood Cliffs, NJ, USA, [31] J.H. McClellan, T.W. Parks, L.R. Rabiner, FIR Linear Phase Filter Design Program, in: The Digital Signal Processing Committee, IEEE Acoustics, Speech, and Signal Processing Society (Eds.), Programs for Digital Signal Processing, IEEE Press, New York, USA, 1979, pp [32] tranden/brodatz.html. [33] B.S. Manjunath, J.-R. Ohm, V.V. Vasudevan, A. Yamada, Color and texture descriptors, IEEE Trans. Circuits Syst. Video Technol. 11 (6) (2001) [34] R.R. Coifman, M.V. Wickerhauser, Entropy-based algorithms for best basis selection, IEEE Trans. Inf. Theory 38 (2) (1992) [35] J. Mao, A.K. Jain, Texture classification and segmentation using Multiresolution Simultaneous Autoregressive Models, Pattern Recognition 25 (2) (1992)