DATA FUSION IN REMOTE SENSING AND IMPROVEMENT OF THE SPATIAL RESOLUTION OF SATELLITE IMAGES

Transcription

1 DATA FUSION IN REMOTE SENSING AND IMPROVEMENT OF THE SPATIAL RESOLUTION OF SATELLITE IMAGES T.RANCHIN Ecole des Mines de Paris Centre d'energetique, Groupe Teledetection & Modelisation BP 207, Sophia Antipolis cedex France 1. Introduction Since the beginning of the 1990's, the availability of Earth Observation satellite data has increased dramatically. Spatial resolutions available range from a meter or less to few kilometers and with finer and finer spectral resolutions. This new situation opens new applications and new fields of research in Earth observation. But this wealth of information is difficult to manage for the user. He can select the most representative data for his application, or he can process all the available data separately. But the most interesting approach will be to combine the available set of information in the most efficient way. For example, a panchromatic image from any satellite such as SPOT, IRS or IKONOS is of interest for visualizing details and structures over an area. The structure and characteristic scales of a city, the buildings, the hierarchy of streets, the vehicles, even more details are detected, recognized or identified according to the high spatial resolution. The exploitation of the multispectral set of images provided by the same satellite, but with coarser spatial resolution achieves spectral recognition and further vegetation monitoring, studies on marine or air pollution, thematic mapping, precision farming, etc., will be achieved. The use of color composition for visualizing multispectral images also facilitates the interpretation of the area of interest. Combining both sets of information will obviously bring much more added value to the user, opening much more applications with the best spatial resolution available. Hence, data fusion is becoming of paramount importance, in Earth Observation. Data fusion is an approach oriented to information extraction adopted in several domains. It is based on the synergetic exploitation of data originating from different sources. It aims to produce a better result than the one obtained by a separate exploitation of the same sources. According to Wald [1): "The exploitation of satellite images and more generally of observations of the Earth and our environment is presently one of the most productive in data fusion. Observation of tile Earth is performed by means of satellites, planes, ships, and ground-based instruments. It results into a great variety of measurements, partly redundant, partly complementary. These measurements may be punctual and time-integrated, bi-dimensional and instantaneous A. K. Hyder et al. (eds.). Multisensor Fusion, Kluwer Academic Publishers. 633

2 634 DATA FUSION IN REMOTE SENSING (images), vertical profiles with time-integration or not, three-dimensional information (oceanic I atmospheric profiler I sounder at ground level, or satellite-borne, or ship-borne). Adding the large amount of archives and numerical models representing the geophysical I biological processes, one should conclude that the quantity of information available to describe and model the Earth and our environment increases rapidly. Data fusion is a subject becoming increasingly relevant because it efficiently helps scientists to extract increasingly precise and relevant knowledge from the available information. The operation of data fusion by itself is not new in environment. For example, meteorologists predict weather for several tens of years. In remote sensing (i.e. Earth observation from spacecraft or aircraft), classification procedures are performed since long and are obviously relevant to data fusion. Data fusion allows formalizing the combination of these measurements, as well as to monitor the quality of information in the course of the fusion process. " In Europe and thanks to the impulsion of the European Association of Remote Sensing Laboratories (EARSeL), a Special Interest Group "Data Fusion" was created in This group contributes to a better understanding and use of data fusion in the field of Earth Observation by organizing regular meetings of its members and tackling fundamentals of Data Fusion in remote sensing. A series of bi-annual international conference called "Fusion of Earth Data - merging point measurements, raster maps and remotely sensed images" was launched in 1996 with the aim of browsing this field of research and to help the scientific community to fully understand the benefits of data fusion in the Earth Observation domain [2, 3, 4]. From the work of this Special Interest Group a set of reference terms emerged [5]. The definition of data fusion in the field of remote sensing was adopted as: "Data fusion is a formal framework in which are expressed means and tools for the alliance of data originating from different sources. " This definition aims to conduct scientists to a formal approach of data fusion and to the benefits of a global reflection on data fusion. Data fusion aims at obtaining information of greater quality; the exact definition of 'greater quality' will depend upon the application. In this case, quality is a generic word denoting that the user is better satisfied by the results obtained through a fusion process. In this paper, data fusion is illustrated through a specific application, the synthesis of images at the best spatial resolution available in the set of images processed. This application can be a step to further processing or applications. Some solutions proposed in this aim are described and a discussion from the quality point of view is addressed. An example is presented and exploited. Conclusions on this approach are drawn. 2. Improvement of the spatial resolution: methods Several studies and publications have shown that merging broadband high spatial resolution images with low spatial resolution and high spectral resolution images proves to be of great benefit in many applications. Many methods have been developed in that purpose and produce multispectral images having the highest spatial resolution available within the data set. They apply on a data set comprising multispectral images Bil at a low

3 RANCHIN 635 spatial resolution I and images Ah at a higher spatial resolution h but with a different spectral content. Examples of such a data set are the SPOT-XS (3 bands, 20 m) and SPOT-P (panchromatic, 10 m) images, or the SPOT-4 case, with 3 bands at 20 m (XSI, XS3, and MIR) and the band XS2 at 10 m, or IRS-IC with the LlSS (3 bands) at 23.2 m and a PAN at 5.8 m, or IKONOS with 4 multispectral bands at 4 m and a PAN at 1 m. The number of methods is fairly large. Of interest in this paper are only concerned those methods which claim to provide a synthetic image close to reality when enhancing the spatial resolution, and not those which only provide a better visual representation of the image [e.g., 6]. The latter are vel)' useful for photo-interpretation. This is particularly true when the number of spectral bands is much larger than the usual three bands for describing colors: red, green, blue. However, such methods have their limitations, especially with the new space-borne sensors and the most recent techniques, which allow the reconstruction of high spatial resolution landscapes with objects having their natural colors. Here, in this context, natural colors mean the colors that are perceived by the human eye. These methods under discussion in this paper aim at constructing synthetic multispectral images B*ih having the highest spatial resolution available within the data set (e.g. the 3 XS bands at 10 m in the case of SPOT 1-3) which are close to reality by performing a high-quality transformation of the multispectral content when increasing the spatial resolution. All the methods can be associated in three groups of techniques currently used: Projection of original data sets into another space, substitution of one vector by the high resolution image and inverse projection into the original space, such as the IRS (Intensity, Hue, and Saturation) method [6]. Relative spectral contribution such as the Brovey transform [7] which can be applied to any set of image and the CNES P+XS method [8] dedicated to the SPOT case. It should be noted that the Brovey transform does not well represent this group because of its poor principles in construction. Nevertheless, it is often used. Scale by scale description of the information content of both images and synthesis of the high-frequency information missing to transform the low spatial resolution images into high spatial resolution high spectral content images. This group of methods can be presented through the ARSIS concept [9]. Two algorithms will be described, the HPF [10] and the XS-HR algorithms [9] PROJECTION METHODS: THE IRS METHOD The IHS method [6] is based on the projection of the original set of images into another space. It can be applied when three multispectral images and one high spatial resolution image are available. Figure I presents this algorithm. Each of the three bands Bi are interpolated to the spatial resolution of Ah and labeled as red, green and blue respectively. Then, these color components are converted into intensity (I), hue (H) and saturation (S) components using for example, the model for colors of "Commission Intemationale pour l'eclairage". The next step is the substitution of the intensity by the high spatial resolution

4 636 OAT A FUSION IN REMOTE SENSING image A h Refinements can be made which include the substitution of a linear combination of the Ah values and the original intensity. The last step perfonns the inverse model converting IHS components into blue, green and red components, which are the searched synthetic images B*ih. The method can apply to either digital counts or to radiances. In any case, the dynamics of the signal in each bands, including A h, should be adjusted in order to make them similar. This may cause a distortion of the spectral content. High spatin! resolution image Replacclllcni of Ihe Inlcnslly by Ihc lligh palin! resolulion imagc Figure 1. The IHS method 2.2. RELATIVE SPECTRAL CONTRIBUTION METHODS The Brovey Transform Let Ah be the high spatial resolution image, Bi/ the multispectral image, h the original spatial resolution of A and / the original spatial resolution of Bi (I < h). The Brovey transfonn applies to the digital counts of three spectral bands B'ih (i = ), where B'ih is the image Bi/ resampled at resolution h, and of an image Ah of a better resolution - though it should deal with radiances and the software should request the calibration coefficients. An example is the SPOT case, where B'ih are the XSI, XS2 and XS3 bands at original resolution of 20 Ill, resampled to 10 m, andah is the P band with a spatial resolution of 10 m. The synthetic bands B *ih are given by: This method can be generalized to any number of spectral bands The P+XSMethod The P+ XS algorithm of CNES takes into account the modulation transfer function and the spectral filter of each band P and XS. It should be applied to images acquired at the same

5 RANCHIN 637 time. However, its mathematical expression is so simple that many people are using it even for non-coincident dates. In this algorithm, it is considered that the radiance of P is equal to the half-sum of the radiance of XSI and XS2. Additionally, the ratio of the radiance of XS lover XS2 should be equal to the ratio of the radiance of the synthesized images at 10 m called XPl and XP2. This leads to the following equations: L L LXSl XPl = 2 p ---""'"'-- LXSl + LXS2 L L LXS2 XP2 = 2 p ---"::::"=""- LXSI +LXS2 where Lp, Lxst. LXS2, Lxpt. LXP2 are respectively the radiance of the P, XSI, XS2 original images and of the XPl and XP2 synthesized images at 10 m. The XP3 synthesized image is obtained by a simple duplication of the original pixels of XS3 image THE ARSIS CONCEPT This method uses the wavelet transform and the multiresolution analysis to decompose the two images to be merge as in Figure 2. A multiresolution analysis using the wavelet transform [11] is applied to image Ah and image Bi/, describing image Ah and image Bi/ at different spatial resolutions and the differences of information between the successive approximations of image Ah and image Bi/. The wavelet coefficients provided by the multiresolution analysis of the high spatial resolution image A", between the scale ofimageah and the scale of image Bi/, describe the missing information for the synthesis of the image Bil at the same spatial resolution than the one of image A h high resolution imagea" low resolution imagebii Image A Image B Figure 2. The use of the multiresolution analysis in the ARSIS method.

6 638 DATA FUSION IN REMOTE SENSING The simplest solution is to shift the wavelet coefficients from pyramidah to pyramid Bi/ and to use them to synthesize image B*ih at the spatial resolution of image Ah. If the wavelet coefficients provided by image Ah are used without modifications, the synthesized image B*ih will not be equivalent to "what would be seen by sensor Bil if it had the spatial resolution of sensor Ah". Hence, to improve the quality of the synthesized image, the model, to transform the wavelet coefficients provided by the multiresolution analysis of image Ah in the wavelet coefficients needed for the synthesis should take into account the physics of the environment. Whatever this model is, by construction, the ARSIS method preserves the spectral content of original image. A multiresolution analysis applied to the synthesized image B* ih will allow the computation of an approximation similar to original image Bi/. Figure 3 presents the general scheme of the ARSIS method. First a multiresolution analysis using the wavelet transform is used to compute the wavelet coefficients and the approximations of image Ah (1). The same operation is applied to the image Bi/ (2). The wavelet coefficients provided by each decomposition are used to compute a model of transformation of the known wavelet coefficients of image Ah to the known wavelet coefficients of image Bi/. This model takes into account the physics of both images and the correlation or anti-correlation existing between both wavelet coefficients images (3). The model can have various forms and take into account more than one scale. This model is then used to compute the missing wavelet coefficients (4). The inversion of the multiresolution analysis (Wrl) allows the synthesis of the image B*ih with the spatial resolution of image Ah (5) r t t t I Resolution n01 I Resolution n02 I Resolution no] I (... J I (Resolution n) r I t I- - -I I Spect,aI ' (.HiilltOililtiạ, )-I'0J I banda I ".re~.. Wavelet Transform CW.T.) MODEL, I Spectral b... B :/ (.::~:... :. ~.. Otlie4:.. ::g.. ::.'. ~-(~.~~... \;.. ~ :.:-y)~~:: <.J. -'--G ---' Figure 3. General scheme for the application of the ARSIS concept using wavelet transform (WT) and inverse wavelet transform (Wf"l)

7 RANCHIN The HPF method The High Pass Filtering (HPF) algorithm [10] was first applied to the fusion of Landsat TM and SPOT P images. It is based on the extraction of the high frequencies of the image with the highest spatial resolution through a Laplacian filtering, and on the addition of these high frequencies to the interpolated lower spatial resolution image. Chavez et al. [10] discussed the size of the filter and its effects on the resulting images. The filter can be derived from the computation of the second derivate of an apodisation function. In the case of the fusion of SPOT P and XS images, the filter was limited to a 3x3 dimension. The value of the filter coefficient are: This filter is convoluted with the P image for the extraction of the high frequencies (corresponding mainly to the structures between 10 and 20 m). These high frequencies are added to the three XSi images interpolated at the resolution of 10 m The XS-HR method In the case of the application to the P and the XS images, the ARSIS concept is implemented through the XS-HR method. Figure 4 presents the application of the ARSIS method to the case of the SPOT imagery. Spatial resolution 10m Spatial resolution 20 m Spatial resolution 40 m P Wavelet P,. H,o.'ZQ Wavelet transform transform p.. o H 2O VIG.2Q 0".10 I,. Model l V2D D2D 1 Model estimation Hi,o.2Q Inverse XSi Hilt. XSi-HR,wavelet Wavelet transform transform Vi. DiXl r Vi IG Di ID 2CI Figure 4. Application of the ARSIS method to the SPOT imagery

8 640 OAT A FUSION IN REMOTE SENSING The set of images is composed in this case of a panchromatic image at the spatial resolution of 10 m and three multispectral images XS1, XS2, XS3 at the spatial resolution of 20 m. The aim of the XS-HR method is to compute the three XSi images at the spatial resolution of 10 m and to perform a high quality transformation of the original spectral content. Two itemtions of the multiresolution analysis using the wavelet transform [11] are applied to the original panchromatic (P) image and one itemtion to the original XSi image. A model of transformation, for each direction, from the panchromatic wavelet coefficient images to the XSi wavelet coefficient images is estimated at the spatial resolution of 40 m. This model can be of various types. The simplest one is the identity model. A convenient one can be an adjustment of the mean and the variance of the histogmm of the wavelet coefficient images. The model must take into account the physics of both images and the correlation and anti-correlation between both images. Several models have been tested. The best results were achieved with a model taking into account the local variation between the P and the XSi wavelet coefficient images. The estimated model is inferred at the spatial resolution of 20 m. Then, it is applied to the transformation of the wavelet coefficient images representing the information between 10 and 20 m of the P image, into those corresponding to the XSi image. Finally, the multiresolution analysis is inverted and the XSi image at the spatial resolution of 10m, called XSi-HR, is synthesized from the original XSi image and from the wavelet coefficient images computed through the model. The complete description of the implementation of the XS-HR method is provided in [9]. 3. A Formal Approach for Evaluation of the Quality of resulting images The merging methods under concern aim at constructing synthetic images B *" close to the reality. Wald et al. [12] established the properties of such synthetic images: Any synthetic image B*" once degmded to its original resolution I, should be as identical as possible to the original image B,. Any synthetic image B*" should be as identical as possible to the image Bh that the corresponding sensor would observe with the highest spatial resolution h. The multispectral set of synthetic images B*" should be as identical as possible to the multispectral set of images B" that the corresponding sensor would observe with the highest spatial resolution h. Testing the firsf property: Any synthetic image B*1o once degraded to its original resolution I, should be as identical as possible to the original image B,. To achieve this, the synthetic image B*" is spatially degraded to an approximate solution B', of B,. If the first property is true, then B', is very close to B,. The difference between both images is computed on a per-pixel basis. This difference image should be visually compared to the original image in order to detect trends of error, if any, possibly related to the type of landscape. Then some statistical quantities are to be used to quantitatively express the discrepancies between both images. These quantities are similar to the first and second sets of criteria described under the second property below.

9 RANCHIN 641 There is an influence of the filtering operator upon the results, but it can be kept very small provided the operator is appropriate enough. Testing the second property: Any synthetic image B*h should be as identical as possible to the image Bit that the corresponding sensor would observe with the highest resolution h. The second and third properties are difficult to test, because they refer to Bh, an image that would be sensed if the sensor had a better resolution. This image, of course, is not available; otherwise, all the above-cited methods would not have been developed. The difficulty is partly overcome by the following approach: The available images A" and BI are degraded to respectively Al and Ba, respectively. For the SPOT case, the P and XS images are degraded to Pzo (20 m resolution) and XS40 (40 m resolution). The images Al and Ba are very close to what the corresponding sensor would have measured with a degraded resolution. as discussed previously. Then the synthesizing method under assessment is applied to A I and B s. It provides a synthetic image B*I (XS*zo in the SPOT case). This synthetic image B*I is compared to the image-truth BI (XS in the SPOT case) by means of some criteria described below. The numerical comparison should be made preferably in physical units and also in relative values. Thus, different tests made over different scenes may be compared. This comparison provides an assessment of the quality of B*I. It is assumed that this quality is fairly similar to that of the synthesized high-resolution image B*h. Two sets of criteria are proposed to quantitatively summarize the performance of a method in synthesizing an image in one spectral band. The first set of criteria provides a global view of the discrepancies between the original image BI and the synthetic one B*I. It contains: The bias, as well as its value relative to the mean value of the original image. Recall that the bias is the difference between the means of the original image and of the synthetic image. Ideally, the bias should be null. The difference in variances (variance of the original image minus variance of the synthetic image), as well as its value relative to the variance of the original image. This difference expresses the quantity of infonnation added or lost during the enhancement of the spatial resolution. For a method providing too many innovations (in the sense of infonnation theory), i.e., "inventing" too much infonnation. the difference will be negative because the variance of the synthetic image will be larger than the original variance. In the opposite case, the difference will be positive. In information theory, the entropy describes the quantity of information. However, we selected the variance difference because most researchers and engineers are much more familiar with variance, and entropy and variance act quite similarly for our purpose. Ideally, the variance difference should be null.

10 642 DATA FUSION IN REMOTE SENSING The correlation coefficient between the original and synthetic images. It shows the similarity in small size structures between the original and synthetic images. It should be as close as possible to l. The standard deviation of the difference image, as well as its value relative to the mean of the original image. It globally indicates the level of error at any pixel. Ideally, it should be null. The error at pixel level may be more detailed. Let us compute at each pixel the absolute relative error (the absolute value of the difference between the original and synthetic values, divided by the original value). Then the histogram of these relative errors is computed. It can be seen as the probability density function. Therefore, we can compute the probability of having at a pixel a relative error (in absolute value) lower than a given threshold. This probability denotes the error made at pixel level, and hence indicates the capability of a method to synthesize the small size structures. The closer to 100 percent the probability for a given error threshold, the better the synthesis. The ideal value is a probability of 100 percent for a null relative error. Here, for reasons of computer precision, the lowest threshold "no relative error or null error" is set to percent Testing the third property: The multispectral set of synthetic images B*h should be as identical as possible to the multispectral set of images Bh that the corresponding sensor would observe with the highest resolution h. Visual inspection may be made through color composites of, for example, the first three principal components of the set of images. Both color composites should agree visually. Most methods for color composites are using dynamical adjustment for color coding (e.g., [13]). If the sets of images are different, even slightly, then the color coding will be different for both composites and no comparison will be possible. Practically, we recommend the following approach. For each spectral band, the Bl and B*l images are juxtaposed into a single computer file. The principal components analysis as well as the color coding are performed on this set of files. The projected Bl and B*l images are then extracted from these projected files and the color composites are displayed, simultaneously or alternatively, onto the screen. This approach guarantees that the color composites are comparable. Of course, if only three spectral bands are available as in the SPOT case, there is no need to perform a principal components analysis. The advantage of this visual assessment is that it does show trend in errors, if any, possibly related to landscape features. The drawback of it is that it is a subjective assessment and also that this assessment may be limited either by physiological factors (e.g., color contrast perception by humans), or by technical factors (e.g., when a large number of spectral bands are present). In the latter case, and if the landscape offers a large variety of objects, the color re-coding of the first three principal components reduces dramatically the differences between the Bl and B*l images, particularly if these differences are random, i.e., not related to a peculiar landscape feature or to a spectral band. A quantitative assessment can be made using the following three additional sets of criteria which quantify the performance of a method to synthesize the spectral signatures during the change in spatial resolution. The third set (numbered after the two sets described above for the second property) deals with the information correlation between

11 RANCHIN 643 the different spectral images taken two at a time. This dependence can be expressed by the correlation coefficients, with the ideal values being given by the set of original images B/. As an example, for the case of SPOT, the correlation coefficient between Pzo and XSl *zo is computed and compared to the correlation coefficient for Pzo and XSlzo. This is done for every pair. The fourth set of criteria partly quantifies the synthesis of the actual multispectral n-tuplets by a method, where n-tuplet means the vector composed by each of the n spectral bands at a pixel. It comprises the number of different n-tuplets (i.e., the number of spectra) observed in the original Mt and in the synthesized B*/ sets of images, as well as the difference between these numbers. A positive difference means that the synthesized images do not present enough n-tuplets; a negative difference means too many spectral innovations. The previous criteria do not guarantee that the synthesized n-tuplets are the same as in the original image B/. The fifth and final set of criteria assesses the performance in synthesizing the actual n-tuplets. It deals with the most frequent n-tuplets, because they are predominant in multispectral classification. For a given threshold in frequency, only the n-tuplets having a frequency (relative number of pixels) greater than this threshold are used. The threshold is set to 0.01 percent, 0.05 percent, 0.1 percent, and 0.5 percent, successively. The greater the threshold, the lower the number of n-tuplets, but the greater the number of pixels exhibiting one of these n-tuplets. For each of the n-tuplets, the difference is computed between the original frequency and the one observed in the synthesized images. These differences are summarized by the following quantities: the number of actual n-tuplets, the number of coincident n-tuplets in the synthesized images, and the difference between these numbers, expressed in absolute and relative terms, the number of pixels in these n-tuplets, in absolute and relative terms, and the difference between the above number of pixels for the original and synthesized images, in absolute and relative terms. Munechika et al. [14] partly quantify the performances in synthesizing the multispectral information by first computing the root mean square (RMS) of the differences, pixel per pixel, between the synthesized B*/ and original B/ images, for each spectral band, and then by summing up these spectral RMS values to obtain a global error, which should be as low as possible. This global error can easily be computed from our first set of parameters, i.e., the bias and the standard deviation. Other criteria may be further dermed. 4. Case studies: the fusion of SPOT Panchromatic and XS In this section two case studies are presented. The first one enhances the influence of the time acquisition of the images to merge. The second one enhances the benefits of the evaluation for users, allowing a better knowledge of the different algorithms for improving the spatial resolution of images.

12 644 OAT A FUSION IN REMOTE SENSING 4.1. THE THREE GORGES DAM, CHINA: INFLUENCE OF THE TIME-GAP ON THE SYNTHESIZED IMAGES The Three Gorges Dam is the largest water conservancy project ever built in China, and so in the world. With normal pool level (NFL) at 175 m, the total storage capacity of the reservoir is 39.3 billion m 3. The preparation of the Three Gorges project started in On Dec. 14, 1994, China Premier Li Peng declared the formal beginning of construction. Over the past three years, significant changes have taken place on both sides of the Yangtze river at Sandouping area owing to the effort of Three Gorges builders. Due to the very large area to cover, remote sensing was selected as a mean for studying the upstream geological impact of this project, and detecting and surveying geological hazards. In order to detect geological problems such as faults, landslides, landflows, the highest spatial resolution and the most recent information are simultaneously required. The set of data available is composed of a SPOT panchromatic (P) images from 1997 and a multispectral SPOT XS image from In this section, the problem of the fusion of panchromatic and XS images acquired at different dates, and the influence of landscape changes on the results of the algorithms are addressed. Because of the very large changes due to the construction of the dam, this area is well-suited to this study. Five algorithms are considered: the IHS algorithm, the Brovey and the P+ XS algorithm, the High-Pass filtering (HPF) algorithm, and the XS-HR algoritiun model RWM within the ARSIS concept. Figure 5 presents the original P image from Figure 6a presents the original XS 1 image from Tremendous modifications occur on the Dam site. The most visible one is the evolution of the island in the middle of the Yangtze river. The shiplock is close to completion in Figure 6a. Figure 5. Extract ofa SPOT panchromatic image from the Three Gorges site acquired in Copyright CNES SPOT Image. Figure 6 presents the original XSI image and the results of the different algorithms of fusion of the P and XS 1 image. The XS 1 10 images were synthesized by the IHS (Figure 6b), the Brovey (Figure 6c), the P+XS (Figure 6d), the HPF (Figure 6e) and the XS-HR (Figure 61) methods. Recalls that the aim is to perform a synthesis of images with a high quality transformation of the missing information.

13 RANCHIN 645 te) Figure 6. Original XSI image of the Three Gorges Dam, China, acquired in November 10, Copyright CNES SPOT Image (a). XSI"IO image synthesized with the IHS (b), Brovey (c), P+XS (d), HPF (e) and XS-HR (I) algorithms. A more complete evaluation of the influence of landscape changes on some of these algorithms was achieved from a geological point of view [IS).

14 646 DATA FUSION IN REMOlE SENSING The visual inspection over the site of the Dam of the resulting images demonstrated the failure of the IHS, Brovey, P+ XS and HPF methods when the time acquisition of the images are different. On the contrary, the XS-HR method is able to fulfil the objectives of the fusion. In the case of the IHS, Brovey and P+ XS algorithms, the predominance of the structures coming from the P image is due to the inherent construction of the synthesized images. All the structures at all scales that are in the P image are introduced in the synthesized images. In the case of the P+ XS algorithm, the fusion of non contemporary images is not recommended by CNES and SPOT Image. The later even refused to apply this algorithm when the two dates of acquisition are different. Figure 6c is a very good illustration of the problem occurring in this case. As the Brovey algorithm is very close to the P+ XS algorithm, the resulting image presents the same characteristics due to the same behavior. In the case of the image obtain from the HPF algorithm, it is obvious that Figures 6e presents an artificial aspect. The information added to the XS 1 image is extracted from the P images by application of a Laplacian filter. Only the high frequencies comprised between both resolutions are extracted from the P images and added without any transformation to the XS 1 image. The modification of the landscape from 1997 to 1998 was important in this area and the introduction of a non-contemporary information gives the impression of a superimposition of two images. Figures 6f looks very close to Figure 6a. The XS-HR method derived from the ARSIS concept is more stable when applied to non-contemporary images. The synthesized image is still exploitable for further processing such as classification processes or interpretation of geological features and faults BARCELONA, SPAIN: EVALUATION OF THE QUALITY The protocol of Wald et aj. [12], presented section 3 is followed to assess the quality of the results of the different methods. The methods are applied to the same SPOT image of the city of Barcelona, Spain, as in their paper (Figure 7). Such an urban area has been selected for illustration because it is certainly the most difficult type of landscape to deal with according to our knowledge. Urban areas often point out the qualities and drawbacks of algorithms because of the high variability of information in space and spectral band, induced by the diversity of features both in size and nature. In the SPOT case, the multispectral images Bi/ are the XSI, XS2 and XS3 bands at original resolution of 20 m, and the high spatial resolution image Ah is the panchromatic band P with a spatial resolution of 10 m. The synthetic bands B*ih are the XS1, XS2 and XS3 bands synthesized at 10 m. Table 1 gives the means, standard-deviations and calibration coefficient of the original images. To assess the first property, the synthetic image B*ih made at 10 m are filtered before resampling to degrade the resolution down to 20 m: (B*ih)l. They are then compared to the original images Bih. The filtering function is a sine cardinal (sinc) kernel truncated by a Hanning apodisation function of size 13x13.

15 RANCHIN 647 Figure 7. SPOT P image of Barcelona used for test. The dark area in the upper left comer is the slope of a hill with Mediterranean vegetation. The size is 512x512 pixels. Copyright CNES SPOT Image (1990). TABLE I. Means, standard deviations, and calibration coefficients of original images (in W m 2 sf' 1m"') XSI XS2 XS3 P Mean Standard deviation Calibration coefficient To test the second and third properties, the P and XS images are degraded to a resolution of 20 (A 2h) and 40 m «(BJ2/), respectively. Then, images B*i/ are synthesized at a 20-m resolution and compared to the original XS images Ba by a visual inspection on the one hand, and by performing a difference pixel per pixel. The discrepancies are analyzed and synthesized in five sets of criteria, which deal respectively with: each spectral band in a global way, the statistical distribution of errors at pixel level for each spectral band, information correlation between the different spectral images, the multispectral aspect, that is the errors in reconstructing spectral signatures, the reconstruction of the most frequent spectral signatures. Wald et 01. discussed the extrapolation of the quality assessments made at 20 m to 10 m. They underlined the unpredictability of such assessments when changing the

16 648 DATA FUSION IN REMOlE SENSING resolution. That is, it cannot be said whether the error at 10 m is larger or lower than that at 20 m. By testing several methods on SPOT images degraded to 40 and 80 m, they found in several cases that the quality was best at 20 m than at 40 m. They suggested that one can assume that the quality of the synthetic images at 10m may be considered as similar to that of the synthetic images at 20 m. A visual inspection of the synthesized images H*ih is performed first, once the contrast table adjusted for each. They are visually fairly close one to the others for all methods and of satisfactory quality, except for the IHS method which produces in that case an image of bad quality in the XS3 band. The HPF images contain too much high frequencies: the contours are enforced in an excessive manner. First property Like the H*ih images, the contrast-adjusted different (H*ih)' images are visually fairly close and of satisfactory quality, except for the IHS method in the XS3 band, the HPF images. Adjusting the contrast table for each (H*ilr)1 accommodates for linear changes in statistical distribution, and especially mean and variance. For the Brovey and IHS methods, these parameters are strongly modified relative to the original Hi/ images. The details of the quantitative comparison (not shown here) further demonstrate that the first property is clearly not satisfied by the Brovey and IHS methods, and also the P+ XS method. In these four methods, the synthesis of the band H*ih at 10 m is influenced by the high resolution image Air and the other spectral bands Hjl. This influence is irrespective of the size of the structures, that is that the large structures (i.e. larger than 20 m) observed in these images A" and Hli are partly included in the synthesized image Hih. A mathematical analysis of these methods clearly shows that the influence of Ah and the other spectral bands Hli in the synthesized image H*ih does not disappear when reducing the resolution to 20 m. The method built within the ARSIS concept using the wavelet transform is inherently built to satisfy this first property, with reservations regarding the degradation process as discussed by Wald et al. [12). On the contrary, the HPF technique does not satisfy this property, mostly due to a strong change in variance. This is confirmed by a visual inspection. Second property The conclusions of the visual analysis of the different H*il images are in accordance with that obtained for the first property. Compared to the first property, we found that the drawbacks of a method are enhanced by the testing oft he second property. This is why we put an emphasis on this comparison. Tables 2 to 4 provide some statistics on the differences between the original Hil images XSI, XS2 and XS3 at 20 m and the synthesized H*il images. They provide a global view of the quality of a method to synthesize each individual spectral band Hi. These tables show a very strong bias (difference between mean values) for the Brovey transform for the three bands. This bias amounts to approximately 0.65 times the original mean value. This is due to the very construction of this transform, which, briefly, written, is equal to the spectral band under concern, multiplied by the ratio of three times the panchromatic band P and the sum of the three bands. Since the method does not request the computation to be made in radiances, a difference in mean between spectral bands - as here between XS2 and the others (Table I) - may induce a strong bias for all

17 RANCHIN 649 TABLE 2. Some statistics on the differences between the original and synthesized images, in radiance (W m 2 sri ~.I) or relative value, for XSI band. r/j 0 - :r: ;>» r/j I1.l X o:l """ ~ ~:E I~ ~cg Bias (ideal value: 0) relative to the mean XS value -10% 64% 1% 0% 0% Actual variance-estimate relative to the actual variance 14% 77% -19 % -267% 3% Correlation coefficient between XS and estimate (ideal val ue: I) Standard-deviation of the differences (ideal value: 0) relative to the mean ofxs value 8% 17% 6% 43% 4% TABLE 3. As Table 2, but for XS2 band.» r/j r/j ~ ~ ~ """ ~ ~~ o:l ~cg Bias (ideal value: 0) relative to the mean XS value - 10 % 65% 0% 0% 0% Actual variance-estimate (ideal value: 0) relative to the actual variance 55% 81 % 11% -57 % 9% Correlation coefficient between XS and estimate (ideal value: I) Standard-deviation of the differences (ideal value: 0) relative to the mean ofxs value 11% 13 % 7% 38% 5%

18 650 OAT A FUSION IN REMOTE SENSING TABLE 4. As Table 2, but for XS3 band. For XS 3, the method "P+XS" reduces to duplication IH Br p+ HP X1R S ov XS F - W ey HM Bias (ideal value: 0) relative to the mean XS value -10% 64% 1% 1% 0% Actual variance-estimate(ideal value: 0) relative to the actual variance 22% 70% -35 % -420% 5% Correlation coefficient between XS and estimate (ideal value: 1) Standard-deviation of the differences (ideal value: 0) relative to the mean ofxs value 8% 10% 7% 36% 3% synthesized bands. This construction also implies that the variance of a synthesized band B *;1 is a combination of the variances of all bands, including the panchromatic. It follows that the variance of the B*i/ image strongly differs from that of the original image Bi/. This method adds too much variance by a relative amount exceeding 70 % of the original variance. The correlation between the B;I and B*i/ images is high as far as the correlation between the Bi/ and Ah images is high. The correlation between XS3 and P is only 0.35 instead of 0.97 for the two other bands, and the correlation between XS3 and XS3*20 is only 0.7, which is rather poor. Finally the relative error at pixel level in reconstructing the original image ranges from 10 to 17 % (standard-deviation). As a whole, the other methods perform better, though only a few provide satisfactory results. The IHS method exhibits a relative negative bias of 10 %, which is still too large and means an overestimation of the values as a whole. This bias may be partly overcome by an a priori equalization of the dynamics of the images Bil and AI. This would also reduce the differences in variance, and more generally would provide better results if the correlation between the images Bil and AI is large. This equalization step is made at the expenses of the physical significance of the images. This remark also holds for the HPF methods. The IHS method does not introduce enough high frequency signal in the synthesized image (the variance is too low), for which the variance is too large, except for the XS3 band. It should be noted for this substitution methods that the results are strongly dependent upon the original images. According to the mutual correlation between bands and the variance in each band, the introduction of high frequencies will be either too large or too low, and sometimes satisfactory. This is true for the other parameters under examination for this second property.

19 RANCHIN 651 The P+ XS method is Wlbiased but introduces too much signal from the P band into the XS*1 and 2. This method reduces to duplication for the XS3 band. Accordingly the variance of the XS*3 image is too low: there is no fusion and no addition of signal from another source. The HPF method is rather disappointing. The amowlt of excessive variance is huge. All the contours are enforced but excessively. The correlation between the synthesized and original images is low for all bands. Finally the error at pixel level is extremely high. The best results are attained by the XS-HR method using the wavelet transform. The method offers the same level of quality for the bands XS 1 and XS2. Finally it should be noted that the results are better for the bands XS 1 and XS2 than for the band XS3. This is due to the fact that the band P encompasses the bands XSI and XS2 and not the XS3. Third property Color composites have been created for each synthesized set of multispectral B*i/ images. The color coding for each set follows that used for the original set. in order to make them comparable and following the recommendations of Wald et al. [12]. This color composite are presented in Figure 8. Unsurprisingly, the color composite obtained by the Brovey transform does not show similarity with the original one. This also holds for the ills method but to a lesser extent. The other methods provide color composites closer to the original with various degrees in quality, which are better described through some quantitative parameters. Table 5 shows the performances of each method in synthesizing the multispectral information. It represents the difference between the actual number of triplets and the number fowld in the synthesized images for each method. These triplets may be different from the original ones; only their number is taken into account in this table. The number of original triplets is approximately 45,600 and is large compared to the number of pixels. This demonstrates the spectral diversity of urban areas. The Brovey transform only found approximately 8,000 triplets! It means that this transform flattens out the spectral diversity of a scene. The HPF method does not perform correctly for this parameter; it provides about twice more triplets. This is due to the enforcement of structures already noted. As expected, the duplication exhibits less triplets than the original. The high discrepancy (49 %) demonstrates the changes in the statistical distribution of spectral signatures when changing the spatial resolution. The other methods perform from fairly correctly (p+xs) to very satisfactory (XS-HR). Actually, this table 5 partly summarizes the multispectral performances of each method. Most of the triplets have a low frequency, i.e. most ofthem are carried by a very few number of pixels. The average number of pixels per triplet is 5.7. Many of the triplets are superfluous; they are carried by 1 or 2 pixels and are not taken into account in further classification processes, or visual analysis of the synthesized images as colored composition. Table 6 shows the performances of each method in synthesizing the most frequent actual triplets. Each triplet Wlder consideration has a frequency of at least 0.01 percent, which corresponds to 26 pixels in this case. The total of pixels they represent amounts to 23 percent of the total number of pixels in the image. Hence synthesizing them accurately is of prinuuy importance in classification purposes. In this Table, for each of these

20 652 DATA FUSION IN REMOTE SENSING (b) (c) (e) (I) Figure 8. Color composite of Barcelona, Spain, from the original data set (a); from images synthesized by IHS (b), Brovey (c), P+XS (d), HPF (e) and XS-HR (f) methods.

21 RANCHIN 653 triplets, the number of pixels carrying this triplet in the synthesized images is compared to the corresponding number in the original images. The differences are summed up for all the triplets, giving the "difference with original" in Table 6. A difference equal to 0 means that all the predominant triplets are exactly the same than in original images. Because of its bias in all bands, the Brovey transform is unable to retrieve any of these 1,675 triplets. Very bad results are also obtained by the IHS method: it retrieves only 721 triplets (43 %) and only 12 % of the corresponding pixels. It means that it does not synthesize correctly the triplets and even for those it retrieves, they are not correctly allotted to the pixels: this would induce errors in cartography after classification. This bad result is not contrary to Table 5, for which synthesized triplets and actual triplets may differ: only the numbers of triplets are compared. TABLE 5. Performances in synthesizing the multispectral information. Difference between the actual frequency ofa triplet (XSl, XS2, and XS3) and the estimates. Average number of pixels per triplet is 5.7. o;j >.!!) r./).s r./) ;;. X I-< S - a: Number of triplets Difference with original (in%) 10% 83% -17 % -90% 4% TABLE 6. As Table 5 but only the most frequent triplets are taken into account. Each triplet has a frequency of at least 26 pixels (0.01 percent of the total number). The total of pixels they represent amounts to 23 percent of the total number of pixels in the image. o;j >. r./) ~ r./)!!) 50 ;;. X ~ 0 + I-< 0... a:1 S '"'- ~~ co ::r:: 0 o~ + ~ ~~ '"'- ~ ~~ ~~ Number of predominant triplets Difference with original (ideal: 0) (in%) 57% 100% 0% 0% 0% Number of pixels Difference with original (ideal: 0) (in%) 88% 100% 41 % 52% 0%