MOST of Earth observation satellites, such as Landsat-7,

Transcription

1 454 IEEE GEOSCIENCE AND REMOTE SENSING LETTERS, VOL. 11, NO. 2, FEBRUARY 2014 A Robust Image Fusion Method Based on Local Spectral and Spatial Correlation Huixian Wang, Wanshou Jiang, Chengqiang Lei, Shanlan Qin, and Jiaolong Wang Abstract To solve the potential color distortion problem of synthetic-variable-ratio method, an improved fusion method based on local spectral and spatial correlation (SSC) is presented. The proposed method, which uses SSC characteristics and local optimization strategy to simulate a low-resolution panchromatic image, can effectively reduce the spectral distortion of the fused image. QuickBird and other satellite images are used to assess the quality of the method. Visual and quantitative analysis demonstrates that the proposed approach can significantly improve the fusion quality. Index Terms Image fusion, local adaptive process, spectral and spatial correlation (SSC), spectral distortion, synthetic variable ratio (SVR). I. INTRODUCTION MOST of Earth observation satellites, such as Landsat-7, Spot 5, IKONOS, QuickBird, GeoEye-1, and WorldView-2 provide both panchromatic (PAN) images with high spatial resolution but low spectral resolution and multispectral (MS) images with high spectral resolution but low spatial resolution. To make full use of the spatial and spectral information, image fusion techniques have been applied in various remote sensing applications, such as image classification, object detection, and forest-type mapping [1], [2]. For more than two decades, many image fusion methods have been proposed, which can be classified into two categories by their protocol [3], i.e., component substitution (CS) methods and methods belonging to multiresolution analysis (MRA). Generalized intensity hue saturation adaptive (GIHSA) [4], Brovey transform [5], synthetic variable ratio (SVR) [6], principal component analysis [7], and Gram Schmidt context- Manuscript received January 20, 2013; revised March 17, 2013, April 16, 2013 and May 8, 2013; accepted May 28, Date of publication July 4, 2013; date of current version November 25, This work was supported in part by the National High Technology Research and Development Program (863 Program) of China under Grant 2007AA120203, by the National Basic Research Program (973 Program) of China under Grant 2011CB707105, and by the National Natural Science Foundation of China under Grant H. Wang, W. Jiang, and J. Wang are with the State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan , China ( whxyaogan@whu.edu.cn; jws@whu.edu.cn; wangjiaolong623@126.com). C. Lei is with the Department of Electronic and Optical Engineering, Mechanical Engineering College, Shijiazhuang , China ( lcqguangxue@163.com). S. Qin is with the Second Crustal Monitoring and Application Center, CEA, Xi an , China ( shanlan_qin@163.com). Color versions of one or more of the figures in this paper are available online at Digital Object Identifier /LGRS adaptive sharpening (GSA-CA) [3] are well-known CS methods. In CS methods, the spatial details can be well injected into the fused products, but spectral distortion may happen [8]. In the recent ten years, the methods based on the MRA framework become well known for their reduced color distortions, such as wavelets [9], Laplacian pyramids [10], and multiscale toggle contrast operator [11], but the spatial enhancement that is attainable is generally not satisfactory, particularly in the case of dissimilarities and aliasing [12], [13]. Several recent methods take advantages of the aforementioned two categories [14] and bring in new theories, such as compressed sensing [15] and optimized approaches [16] into a single method; they are called hybrid in [8]. As the representative fusion method of the CS technique, the SVR techniques have clear physical meaning and theoretical basis, particularly the modified SVR fusion method, i.e., UNB-Pansharp [6]. It has been a popular method since it was commercialized in PCI Geomatica software. However, it was found that the spatial enhancement or spectral preservation of the fused image may not be occasionally satisfactory [12]. According to the analysis of SVR methods, it can be shown that an important drawback of the considered SVR methods is the direct use of the high-resolution PAN for the regression of the coefficients, which are utilized for determining the synthetic PAN. However, in this case, the calculation of the coefficients could be undermined by the decoupling between the high spatial frequencies present in the PAN images and absent in the MS images. For this reason, in the more recent GIHSA [4] and GSA-CA methods [3], the calculation of the coefficients has been performed by considering a linear regression between the low-resolution MS image and a lowresolution PAN image. In view of the aforementioned, in this letter, we proposed a solution of spectral and spatial correlation (SSC)-based synthetic variable ratio (SSCSVR) to remove the spectral distortion of fused image while preserving the spatial characteristic of the PAN image. First, the regression model of the SVR is improved with a spatial correlated component. Second, we adopted a modified localized adaptive processing strategy to better preserve the spectral information. This paper is organized as follows: The principle of the SVR-based method is briefly described in Section II. The scheme and the subparts of our proposed method are reported in Section III. Section IV provides the experimental results and discussion. Finally, the conclusions are given in Section V. II. PRINCIPLE OF SVR FUSION METHOD AND ITS DRAWBACKS Munechika et al. [17] first developed the SVR method to merge MS images and a high-resolution PAN image. They X 2013 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See for more information.

2 WANG et al.: ROBUST IMAGE FUSION METHOD 455 assumed that the low-resolution synthetic PAN image can be created from low-resolution MS bands whose spectral response overlap the one of the input high-resolution PAN band. The principle is as follows: Pan Syn = ϕ i MS i (1) where Pan Syn is the low-resolution synthetic PAN band, MS i is the ith band of the MS image to be fused, and ϕ i is the weight of the ith band of the MS image. The SVR method supposes that the gray value ratio between the high-resolution PAN band and the low-resolution Pan Syn band reflects the spatial detail difference between the PAN and MS images. If these differences are distributed to every MS band by proportion, a high-resolution MS image can be reproduced, whose spatial resolution is the same as the PAN image, i.e., Fused i = MS i Pan ori Pan Syn (2) where Pan ori is the high-resolution PAN image to be fused, and Fused i is the ith band of fused MS image. The SVR method assumes that weight ϕ i can be obtained from the multiple linear regressions between the highresolution PAN and low-resolution MS images, i.e., Pan ori = ϕ i MS i. (3) Various methods have been suggested to determine the ideal weight ϕ i. In [17], weights ϕ i were calculated by using the five land cover types, i.e., urban, soil, water, grass, and trees. Zhang [6] took a further research on the SVR method and clearly explained that the MS bands used to calculate ϕ i should be from the MS bands whose spectral ranges were overlapped by the spectral range of the PAN image but not from all MS bands. At the same time, Zhang suggested that ϕ i should not be calculated only with the five land cover classes but with the global information of image. In his UNB-Pansharp method, Zhang first performs histogram standardization on the MS and PAN images to improve the solution stability, and then, the whole image is used to calculate ϕ i by least-square fitting. Although UNB-Pansharp has been popular, it may occasionally produce fused image with much spectral distortion due to the following factors. 1) If the spectral wavelength of the MS image cannot overlap that of the PAN image completely, the spatial enhancement or the spectral preservation of the fused image may not be satisfied [12]. 2) Even if the wavelength satisfies the overlapping condition, there may still be spectral distortion due to the uneven distribution of the land cover type in the image. According to the property of the least-square fitting, dominating the land cover type may hide the contribution of the minority type, resulting in low accuracy of weights ϕ i. 3) As well known from [8], the PAN and MS images may present some local instability or dissimilarity, such as object occultation and contrast inversion. Thus, using the whole information of the image to calculate ϕ i will make local Pan Syn not close to the original PAN image. 4) The multiple linear regression of (3) simply considers the spectral relationship between the PAN image and the corresponding MS image, in which high-resolution PAN is directly used for the regression of the coefficients, while neglecting the spatial correlation between the pixels in the PAN image. In this case, a correct calculation of the coefficients could be undermined by the decoupling between the high spatial frequencies present in the PAN images and absent in the MS images. Moreover, histogram standardization in the UNB-Pansharp method, which may cause the looseness of spectral information, is not necessary in theory. As previously mentioned, it is necessary to put forward a robust SVR fusion method, which is insensible to band overlapping and land-cover-type distribution and can adapt to the local instability of the image and the detail difference between the PAN and MS images. III. PROPOSED IMAGE FUSION SCHEME In this section, we propose a robust SVR-based image fusion method that uses local SSC (which is called the SSCSVR method). The proposed method is described as follows: A. Improved Model Based on SSC In the image fusion, the spectral preserving correlates with the interband structure of the MS image, and the spatial preserving correlates with the injection of the spatial structure from the PAN image. The traditional fusion method such as SVR only uses the spectral correlation between the PAN and MS images to build the multiple-linear-regression model. However, the pixels in an image are spatially correlated, meaning that, for a source image, if one pixel contributes to the fused image, its neighbors are likely to contribute to the fused image as well. Therefore, the decision making during the fusion process should exploit the property of spatial correlation. In addition, the wavelength of the PAN image is relatively wide; thus, the information in the PAN image not only contains the spectral characteristics of the MS bands but also contains more spatial details than the MS bands. Based on the aforementioned reasons, we build a more appropriate regression model to calculate weight ϕ i, including both the SSC fraction. The improved SSC model is Pan ori = ϕ i MS i + βg s (4) where β is the weight of the spatial correlation fraction, and G s is the spatial correlation fraction, i.e., high-frequency information. In order to get the high-frequency G s, we adopt the Gaussian filter method. denotes the convolution, and Pan L denotes the low frequency of the PAN image. Here, we employ the standard normal distribution σ =1. The expression is as follows: Pan L = G(x, y; σ) Pan ori ; G s = Pan ori Pan L. (5)

3 456 IEEE GEOSCIENCE AND REMOTE SENSING LETTERS, VOL. 11, NO. 2, FEBRUARY 2014 The matrix form of the SSC model is expressed as ϕ 1 ϕ 2 Pan ori =[MS 1 MS 2... MS N G S ]... =BX (6) ϕ N β where X =[ϕ 1,ϕ 2,...,ϕ N,β] T is the weight coefficients vector to be solved. B =[MS 1, MS 2,...,MS N,G s ] is the known spectral and spatial fractions. Equation (4) can be written as Pan ori = ϕ i MS i + β(pan ori Pan L ) Pan ori (1 β)+βpan L = ϕ i MS i. (7) The strategy of directly using the high-resolution PAN can keep well spatial details but bad spectral preserving, while the strategy of directly considering the low-resolution PAN is in contrast. Despite (7), we can see that the proposed approach is practically a tradeoff between the two strategies of considering the high- and low-resolution PAN images in computing the coefficients. β is not a fixed value but an adaptive value that is the same as the other coefficients, which could be obtained by (6). If β is 1, the strategy of considering the low-resolution PAN image is adopted, whereas if β is 0, the strategy of using high-resolution PAN is considered. B. Adaptively Local Processing Method and Exception Handling Traditional SVR fusion methods use the global information to calculate the weight coefficients and do not take the local information into account. Local treatment has shown the greater advantage than the global processing [3] in spectral information preserving. Different from context-adaptive (CA) models [3] for Gram Schmidt adaptive procedure, which takes local statistics on a sliding window for each pixel of an image, our SSCSVR method adopts a blockwise computation to reduce the computational cost. In addition, we employ the fast nonnegative constrained least-square method (FNNLS) [18] to solve the regression model to get the spatial and spectral weight coefficients and synthetic for the low-resolution PAN image. In the solution procedure, the invertibility of a matrix should be considered. When the block of interest locates in a low contrast area, e.g., in a large area of water, the matrix of regression is usually noninvertible. If the matrix is invertible, we can easily use the FNNLS to obtain the accurate weight coefficients. Otherwise, we adopt formula (8) to calculate the weight coefficients, i.e., the coefficient is equal to the ratio between the sum of all PAN pixels and the sum of all MS pixels in the image block, i.e., ϕ i = j=np i=n j=np i=1 Pan j MS i,j (8) where np is the total pixels used to calculate the coefficients. Pan j is the gray value of the j pixel for the PAN image, and MS i,j is the gray value of the j pixel for the ith band for the MS image. Fig. 1. True color (R, G, and B) composition of QuickBird data set. (a) Degraded high-resolution PAN (2.8m). (b) Degraded low-resolution MS (11.2m). (c) Original high-resolution MS image (2.8m). (d) GSA-CA. (e) HCS. (f) HPF. (g) GIHSA. (h) UNB-Pansharp. (i) Proposed method. Considering the process speed and the fusion quality, the block size can be assumed by the following formula: w =5 l/h +1 (9) where h and l denote the resolutions of PAN and MS, respectively. The block strategy can make full use of the local feature. Nonetheless, the coefficient calculated block by block may bring out a block effect. To overcome the problem, the coefficients of pixels at the block borders are interpolated with a bilinear method. IV. EXPERIMENTAL RESULTS AND DISCUSSION Experiments were conducted to evaluate the performance of the proposed method using QuickBird, Spot-5, IKONOS, Landsat-7, GeoEye-1, and WorldWiew-2 images. In order to save space, only the QuickBird data experiments and comparisons are reported in this section. For other five sensor experiments, we only show source images and proposed method results in the Appendix. To evaluate the fusion quality, we adopted Wald s protocol [19]. Fusion is performed on degraded data sets, and the fused high-resolution MS images are then compared with the original low-resolution MS images, which are seen as the reference images. We compare our method with GSA-CA [3], hyperspherical color sharpening (HCS) [20], high-pass filter addition (HPF) [21], GIHSA [4], and UNB- Pansharp [6]. For visual estimation, the display of all images was made consistent by employing at 2% linear stretch in environment for visualizing images program, and for clear visualization, we only display small chips in all demonstrations.

4 WANG et al.: ROBUST IMAGE FUSION METHOD 457 TABLE I COMPARISON OF PROPOSED ALGORITHM WITH THE EXISTING METHODS ON THE QUICKBIRD IMAGE QuickBird satellite data provide the PAN band at 0.7-m resolution and the MS image at 2.8-m resolution. Fig. 1(a) and (b) gives degraded high-resolution PAN and low-resolution MS images of one region in Sundarbans. Original true 2.8-m RGB bands of the MS data are presented in Fig. 1(c) for visual reference. Fused results of GSA-CA [3] (the size of local window is 11 11), HCS [20], HPF [21], GIHSA [4], UNB-Pansharp [6], and the proposed method are illustrated in Fig. 1(d) (i), respectively. By visually comparing the fused images with the original source images, we can see that all the experimental methods can effectively enhance the spatial detail of the MS data. However, compared with the original reference MS image, the proposed method shows the best spectral preserving performance. The fused images with the other methods have color distortion to some extent for the vegetated and water areas, e.g., the fused image of the GSA- CA method is redder than the reference image. Table I lists the quantitative scores for all methods by considering metrics as the correlation coefficient (CC), relative global-dimensional synthesis error (ERGAS) [22], relative spectral fidelity (SF), and Q4 [23]. The relative spectral ( X(i, j) F (i, j) /X(i, j)), where X and F denote the reference MS image and the fused image with the size of M N, respectively. ERGAS and SF should be as low as possible, whereas CC and Q4 should approach to 1. In Table I, the best results for each quality index are labeled in bold. We can see that the proposed method outperforms the existing methods in terms of the measures. In order to verify the advantage of our SSC model, experiments on modified GSA-CA (GSA-SSC-CA in Table I) in which the linear regression is replaced with our SSC model are conducted. From Table I, the GSA-SSC-CA shows better results than the GSA-CA, which demonstrates that the linear coefficients of our model are better than directly only using the filtered PAN and the MS bands. The reason for that is that the SSC model can give better weights of different spatial frequencies between the high-resolution PAN and the filtered PAN than directly reducing the PAN image to the MS spatial scale. From the visual analysis of the fused image in the Appendix, we can see that spatial and spectral characteristics for the fidelity is given by SF =1/(M N) M i=1 N fused image of the proposed method are similar to the original MS image for GeoEye-1, IKONOS, Landsat-7, SPOT-5, and WorldView-2. On the whole, the quantitative assessment results and the visual evaluations demonstrate that the proposed method not only provides high-quality spatial details but also satisfactorily preserves spectral information. V. C ONCLUSION In this paper, a robust local SSC model-based fusion algorithm has been presented to keep the spatial details from the PAN image and preserve spectral characteristics from the MS image. We have addressed the spectral distortions in the traditional SVR method and have resolved them by improving the regression model and adopting an adaptive local process. The proposed method has been compared with the typical GSA-CA, HCS, HPF, GIHSA, and UNB-Pansharp methods. The experimental results suggest that the images obtained with the proposed algorithm have higher fusion quality than that produced with other well-known methods in terms of visual analysis and the pertained quantitative evaluation indicators. APPENDIX The study area of the GeoEye-1 image, which provides the PAN band at 0.5-m resolution and the MS bands at 2-m resolution, is located in Hobart, Australia. The study area of the IKONOS image, which provides the PAN band at 1-m resolution and the MS band at 4-m resolution, is located in Beijing, China. The study area of the Landsat-7 image, which provides the PAN band at 15-m resolution and the MS band at 30-m resolution, is located in Xuzhou, China. The study area of the SPOT-5 image, which provides the PAN band at 5-m resolution and the MS band at 10-m resolution, is located in Marseille, France. The study area of the WorldView-2 image, which provides the PAN band at 0.5-m resolution and the MS band at 2-m resolution, is located in Rome, Italy. Fig. 2 illustrates the source images and the proposed-method fused results for GeoEye-1, IKONOS, Landsat-7, SPOT-5, and WorldView-2, respectively, where Fig. 2(a), (d), (g), (j), and (m) give degraded PAN images; Fig. 2(b), (e), (h), (k), and (n)

5 458 IEEE GEOSCIENCE AND REMOTE SENSING LETTERS, VOL. 11, NO. 2, FEBRUARY 2014 Fig. 2. Source images and proposed-method fused results for GeoEye-1, IKONOS, Landsat-7, SPOT-5, and WorldView-2. (a), (d), (g), (j), and (m) degraded PAN images. (b), (e), (h), (k), and (n) Original true RGB bands of the MS images. (c), (f), (i), (l), and (o) Proposed-method fused results. show the original true RGB bands of the MS images for visual reference; and Fig. 2(c), (f), (i), (l), and (o) illustrate the proposed-method fused results. ACKNOWLEDGMENT The authors would like to thank the editors and the reviewers for their constructive comments and help concerning the improvement to this paper in a consequence of the reviewing process. REFERENCES [1] M. Chikr El-Mezouar, N. Taleb, K. Kpalma, and J. Ronsin, An IHSbased fusion for color distortion reduction and vegetation enhancement in IKONOS imagery, IEEE Trans. Geosci. Remote Sens., vol. 49, no. 5, pp , May [2] I. Ulusoy and H. Yuruk, New method for the fusion of complementary information from infrared and visual images for object detection, IET Image Process., vol. 5, no. 1, pp , Feb [3] B. Aiazzi, S. Baronti, F. Lotti, and M. Selva, A comparison between global and context-adaptive pansharpening of multispectral images, IEEE Geosci. Remote Sens. Lett., vol. 6, no. 2, pp , Apr [4] B. Aiazzi, S. Baronti, and M. Selva, Improving component substitution pansharpening through multivariate regression of MS + Pan Data, IEEE Trans. Geosci. Remote Sens., vol. 45, no. 10, pp , Oct [5] A. R. Gillespie, A. B. Kahle, and R. E. Walker, Color enhancement of highly correlated images. II: Channel ratio and "chromaticity" transformation techniques, Remote Sens. Environ., vol. 22, no. 3, pp , Aug [6] Y. Zhang, System and method for image fusion, US Patent B2, Mar. 4, [7] V. K. Shettigara, A generalized component substitution technique for spatial enhancement of multispectral images using a higher resolution data set, Photogramm. Eng. Remote Sens., vol. 58, no. 5, pp , May [8] C. Thomas, T. Ranchin, L. Wald, and J. Chanussot, Synthesis of multispectral images to high spatial resolution: A critical review of fusion methods based on remote sensing physics, IEEE Trans. Geosci. Remote Sens., vol. 46, no. 5, pp , May [9] P. S. Pradhan, R. L. King, N. H. Younan, and D. W. Holcomb, Estimation of the number of decomposition levels for a wavelet-based multiresolution multisensor image fusion, IEEE Trans. Geosci. Remote Sens., vol. 44, no. 12, pp , Dec [10] L. Alparone, B. Aiazzi, S. Baronti, and A. Garzelli, Sharpening of very high resolution images with spectral distortion minimization, in Proc. IEEE IGARSS, 2003, vol. 1, pp [11] X. Bai, F. Zhou, and B. Xue, Edge preserved image fusion based on multiscale toggle contrast operator, Image Vis. Comput., vol. 29, no. 12, pp , Nov [12] L. Alparone, L. Wald, J. Chanussot, C. Thomas, P. Gamba, and L. M. Bruce, Comparison of pansharpening algorithms: Outcome of the 2006 GRS-S data-fusion contest, IEEE Trans. Geosci. Remote Sens., vol. 45, no. 10, pp , Oct [13] S. Baronti, B. Aiazzi, M. Selva, A. Garzelli, and L. Alparone, A theoretical analysis of the effects of aliasing and misregistration on pansharpened imagery, IEEE J. Sel. Topics Signal Process., vol. 5, no. 3, pp , Jun [14] M. Gonzalez-Audicana, J. L. Saleta, R. G. Catalan, and R. Garcia, Fusion of multispectral and panchromatic images using improved IHS and PCA mergers based on wavelet decomposition, IEEE Trans. Geosci. Remote Sens., vol. 42, no. 6, pp , Jun [15] L. Shutao and Y. Bin, A new pan-sharpening method using a compressed sensing technique, IEEE Trans. Geosci. Remote Sens., vol. 49, no. 2, pp , Feb [16] Z. Zhou, S. Peng, B. Wang, Z. Hao, and S. Chen, An optimized approach for pansharpening very high resolution multispectral images, IEEE Geosci. Remote Sens. Lett., vol. 9, no. 4, pp , Jul [17] C. K. Munechika, J. S. Warnick, C. Salvaggio, and J. R. Schott, Resolution enhancement of multispectral image data to improve classification accuracy, Photogramm. Engi. Remote Sens., vol. 59, no. 1, pp , Jan [18] R. Bro and S. D. Jong, A fast non-negativity-constrained least squares algorithm, J. Chemomtr., vol. 11, no. 5, pp , Sep./Oct [19] L. Wald, T. Ranchin, and M. Mangolini, Fusion of satellite images of different spatial resolutions: assessing the quality of resulting images, Photogramm. Eng. Remote Sens., vol. 63, no. 6, pp , Jun [20] C. Padwick, F. Pacifici, and S. Smallwood, WorldView-2 Pansharpening, in Proc. ASPRS Annu. Conf., San Diego, CA, USA, 2010, pp [21] P. S. P. Ute, G. Gangkofner, and D. W Holcomb, Optimizing the high-pass filter addition technique for image fusion, Photogramm. Eng. Remote Sens., vol. 74, no. 9, pp , Sep [22] L. Wald, Data Fusion: Definitions and Architectures Fusion of Images of Different Spatial Resolutions. Paris, France: Presses de l Ecole Ecole des Mines de Paris, [23] L. Alparone, S. Baronti, A. Garzelli, and F. Nencini, A global quality measurement of pan-sharpened multispectral imagery, IEEE Geosci. Remote Sens. Lett., vol. 1, no. 4, pp , Oct