PLEASE SCROLL DOWN FOR ARTICLE

Transcription

1 This article was downloaded by: [University of Connecticut] On: 10 June 2009 Access details: Access Details: [subscription number ] Publisher Taylor & Francis Informa Ltd Registered in England and Wales Registered Number: Registered office: Mortimer House, Mortimer Street, London W1T 3JH, UK International Journal of Remote Sensing Publication details, including instructions for authors and subscription information: Restoration of clouded pixels in multispectral remotely sensed imagery with cokriging Chuanrong Zhang a ; Weidong Li a ; David J. Travis b a Department of Geography and Center for Environmental Sciences and Engineering, University of Connecticut, Storrs, CT 06269, USA b Department of Geography and Geology, University of Wisconsin- Whitewater, Whitewater, WI 53190, USA Online Publication Date: 01 January 2009 To cite this Article Zhang, Chuanrong, Li, Weidong and Travis, David J.(2009)'Restoration of clouded pixels in multispectral remotely sensed imagery with cokriging',international Journal of Remote Sensing,30:9, To link to this Article: DOI: / URL: PLEASE SCROLL DOWN FOR ARTICLE Full terms and conditions of use: This article may be used for research, teaching and private study purposes. Any substantial or systematic reproduction, re-distribution, re-selling, loan or sub-licensing, systematic supply or distribution in any form to anyone is expressly forbidden. The publisher does not give any warranty express or implied or make any representation that the contents will be complete or accurate or up to date. The accuracy of any instructions, formulae and drug doses should be independently verified with primary sources. The publisher shall not be liable for any loss, actions, claims, proceedings, demand or costs or damages whatsoever or howsoever caused arising directly or indirectly in connection with or arising out of the use of this material.

2 International Journal of Remote Sensing Vol. 30, No. 9, 10 May 2009, Restoration of clouded pixels in multispectral remotely sensed imagery with cokriging CHUANRONG ZHANG*{, WEIDONG LI{ and DAVID J. TRAVIS{ {Department of Geography and Center for Environmental Sciences and Engineering, University of Connecticut, Storrs, CT 06269, USA {Department of Geography and Geology, University of Wisconsin-Whitewater, Whitewater, WI 53190, USA (Received 18 June 2007; in final form 15 January 2008 ) The presence of clouds and their shadows in remotely sensed images limits their potential uses for extracting information. The commonly used methods for replacing clouded pixels by land cover reflection estimates usually yield poor results if the images being combined exhibit radical differences in target radiance due, for example, to large date separation and high temporal variability. This study focuses on introducing geostatistical techniques for interpolating the DN values of clouded pixels in multispectral remotely sensed images using traditional ordinary cokriging and standardized ordinary cokriging. Two case studies were conducted in this study. The first case study shows that the methods work well for the small clouds in a heterogeneous landscape even when the images being combined show high temporal variability. Although the basic spatial structure in large size clouds can be captured, image interpolation-related artefacts such as smoothing effects are visually apparent in a heterogeneous landscape. The second case study indicates that the cokriging methods work better in homogenous regions such as the dominantly agricultural areas in United States Midwest. Various statistics including both global statistics and local statistics are employed to confirm the reliability of the methods. 1. Introduction Multispectral remotely sensed images have now become important in monitoring and utilizing natural resources on the Earth, and they are providing data for many studies such as land-use change analysis, hydrological analysis and environmental analysis because of their rapid availability, consistency and comprehensive coverage. However, the presence of clouds and their shadows in remotely sensed images limits their potential uses for extracting information. For larger area studies such as those at sub-continental scales, it is difficult to obtain images completely free of clouds (Addink and Stein 1999, Arellano 2004). Thus, it is important to remove the effect of cloud contamination from remotely sensed imagery. Various methodologies have been proposed to solve the problem of missing information caused by clouds and their shadows. Common methods for replacing clouded pixels by land cover reflection estimates are the Maximum Value Composite of the Normalized Difference Vegetation Index (NDVI) values (Holben 1986, Moody and Strahler 1994), Minimum Value Composite and Simple Replacement (Arellano 2004). Other *Corresponding author. chuanrong.zhang@uconn.edu International Journal of Remote Sensing ISSN print/issn online # 2009 Taylor & Francis DOI: /

3 2174 C. Zhang et al. strategies such as histogram matching, regression trees, and image fusion techniques were also suggested for cloud-free satellite image mosaics (e.g. Homer et al. 1997, Pohl 1999, Arellano 2004, Howard and Lacasse 2004, Helmer and Ruefenacht 2005). However, these methods do not take full advantage of the spatial information in the clouded imagery. Spatial independence rarely occurs in image scenes and there is spatial dependence of variability in images, which is reflected in the variability of digital number (DN) values (Jupp et al. 1988, 1989). Adjacent pixels tend to be spatially auto-correlated and it is expected that two adjacent pixels will generally be more similar in their DN values than would two pixels separated by a greater distance (Lark 1996). Spatial autocorrelation in images has been measured and verified by many studies in the literature (e.g. Box and Jenkins 1976, Labovitz et al. 1982). Without taking full consideration of the spatially autocorrelated information in the clouded imagery, the aforementioned methods may yield poor quality results if the clouded and merged images do not satisfy certain criteria such as minimum date separation and low temporal variability. For example, the local linear histogram algorithm can yield poor results if the scenes being combined exhibit radical differences in target radiance due to large date separation and high temporal variability (USGS 2004, Commonwealth of Australia 2006). Image fusion techniques are inappropriate for computer aided processing if input images differ greatly in either their spectral or spatial resolutions and the radiometric values of the replaced clouded pixels are not close to the original input data (Arellano 2004). Because spatial structure occurs in remotely sensed images and DN values do not vary uniformly across the landscape, understanding the magnitude and pattern in spatial variability is necessary for accurately interpolating the missing pixels in Landsat 7 ETM + imagery. Geostatistical techniques, which were founded and initially developed by Georges Matheron in France in the 1960s, are designed for spatial data and take full advantage of the spatial correlation information (Matheron 1962, 1963, 1971, 1973, Journel and Huijbregts 1978, Isaaks and Srivastava 1989, Goovaerts 1997). These techniques can be used to explore and describe spatial patterns and variations in remotely sensed data. In this paper, alternative geostatistical interpolation methods traditional ordinary cokriging and standardized ordinary cokriging are described and used to fill the missing pixel information in Landsat 7 ETM + imagery and Landsat 5 ETM imagery. Ordinary cokriging methods would enable geoscientists to make use of cloudy ETM + /ETM data by considering spatial structure in the original image and the spatial cross correlation between the original image and a secondary image. By quantifying the dependence and spatial variability of radiometric data in both the original and secondary images and incorporating them into the interpolation, geostatistical cokriging interpolation approaches provide unbiased estimates with minimum and known error (Matheron 1971, Cressie 1990). Thus, they may offer an alternative means for replacement of clouded pixels in remotely sensed imagery. Although geostatistical techniques have been used in many situations and well documented in the remote sensing literature (Curran and Atkinson 1998, Stein et al. 1999, Curran 2001), most studies have focused on exploring a semivariogram (or variogram), which is the traditional measure of spatial dependence in geostatistics, to describe an image s spatial structure (Curran 1988, Jupp et al. 1988, 1989, Woodcock et al. 1988a, b, Smith et al. 1989, Cohen et al. 1990). Remote sensing research has not paid much attention to kriging interpolation (Ramstein and Raffy

4 Restoration of clouded pixels in multispectral remotely sensed imagery , Ishida and Ando 1999). Some studies have considered the application of cokriging to the problem of estimating environmental properties such as soil, vegetation, and sea surface temperature using classified soft information of remotely sensed imagery (e.g. Bhatti et al. 1991, Atkinson et al. 1992, 1994, Dungan 1998, Ishida and Ando 1999, Wang et al. 2002). However, their main purposes were focused on interpolation of environmental properties instead of dealing with the remotely sensed imagery, and they only used the classified information from remotely sensed imagery as secondary information for the interpolation of these environmental properties. Studies on applying kriging or cokriging directly to interpolate radiometric DN values of missing pixels of remotely sensed imagery are rare and limited (e.g. Addink and Stein 1999, Pardo-Igúzquiza et al. 2006, Zhang et al. 2007). Moreover, there are few studies that address the problem of missing information caused by clouds and their shadows in remote sensing literature. Rossi et al. (1994) introduced indicator kriging approach for estimating whether clouded pixels belong to a class pasture or not using classified images. Addink and Stein (1999) compared seven methods to replace clouded pixels in NOAA-AVHRR images. However, their study was intended for images which are almost free of clouds (Addink and Stein 1999). The objective of this study is to introduce ordinary cokriging for replacing clouded pixels. Several issues that were not introduced in Addink and Stein (1999) are addressed here. These include: (1) modelling spatial dependence between the primary and secondary images via a linear model of coregionalization; (2) estimating the DN values of clouded pixels in the primary image using both standardized ordinary cokriging and traditional ordinary cokriging; (3) comparing ordinary cokriging and ordinary kriging to illustrate whether the addition of a secondary variable can improve interpolation results or not; and (4) comparing ordinary cokriging and other image fusion methods such as using simple replacement, simple regression, and Brovey transform under conditions that the secondary image differs greatly in the radiometric values from that of the original image. Further, while Addink and Stein (1999) worked on images almost free of clouds, in this study, ordinary cokriging is conducted on an image where a high percentage of cloud pixels exist with both small clouds (a small cloud refers to a clouded area with a small number of pixels relative to the total pixels available in the whole image) and large size clouds (a large size cloud refers to a clouded area with a large number of pixels relative to the total pixels available in the whole image). The efficacy of ordinary cokriging methods for interpolating different cloud sizes is evaluated. In addition to using root mean square deviation (RMSE) to assess the accuracy of cokriging interpolation results, several other validation methods, such as Q Q plots and spatial distribution maps of interpolation errors, are employed in this study. While global estimates of accuracy are useful in their own right, visualizing the spatial distribution of interpolation errors is desirable. This is because the interpolation accuracy is not constant across the map and map errors are generally spatially autocorrelated (Foody 2002). Finally, this study also tests whether the interpolated image can be used for computer aided processing such as unsupervised classification. 2. Methods Geostatistics comprise a set of spatial statistical techniques for evaluating the autocorrelation observed in spatial data and estimating the local values of properties that vary in space from sample data (Matheron 1963, Isaaks and Srivastava 1989).

5 2176 C. Zhang et al. Kriging is a family of generalized least-square regression algorithms that account for the spatial dependence represented by the variogram (Matheron 1971, Cressie 1990). A variogram is useful not only for characterizing autocorrelation in an image but also for deriving a model that can be used in kriging (interpolation), which allows for the prediction of values at unsampled locations. Cokriging is the extension of simple kriging and incorporates data from related variables to increase the accuracy of estimates by taking simultaneously into account the autocorrelation in each variable and the cross correlation between the variables (Oliver and Webster 1991). Cokriging was originally developed to save cost and sampling time and has been traditionally used in mining and geostatistical applications (Journel and Huijbregts 1978, Matheron 1979, Isaaks and Srivastava 1989, Goovaerts 1997). There are various types of cokriging methods. The distinction arises from the way in which constraints are imposed. Three common types of cokriging are: simple, traditional ordinary, and standardized ordinary cokriging (Isaaks and Srivastava 1989, Goovaerts 1997, Deutsch and Journel 1998). The following sections briefly introduce the methods used in this study linear model of coregionalization for modelling spatial dependence, traditional ordinary cokriging and standardized ordinary cokriging for estimating the missing pixel values in a cloudy image, and verification methods used in this study. 2.1 Linear model of coregionalization Cokriging methods are based on the theory of regionalized variables (Matheron 1971), which assumes that the variables involved are random and spatially correlated at some scale. For remotely sensed imagery, the digital number Z of pixel x a is a regionalized variable, because the position of pixel x a in space is known. Note that DN has been interpreted as a regionalized variable in a lot of remote sensing literature (e.g. Curran 1988, Woodcock et al. 1988b, Atkinson et al. 1994, Atkinson and Curran 1995, Chica-Olmo and Abarca-Hernández 1998, 2000, Pardo- Igúzquiza et al. 2006). Variograms can be used to characterize the spatial structure and measure the spatial variation of pixel DN values in remotely sensed images (Curran 1988, Chica-Olmo and Abarca-Hernández 2000). Variograms model the spatial dependence in a regionalized variable Z under the intrinsic hypothesis that the increments Z(x a + h)2z(x a ) associated with a small distance h are weakly stationary (Matheron 1971). This implies that the mean value of the sensed radiation is locally stationary and that the squared difference is constant for a given small distance h. The usual equation for calculating the variogram is: cðhþ~ 1 2NðÞ h XN ðhþ a~1 ½zx ð a Þ{zðx a zhþš 2, ð1þ where z(x a ) denotes the DN values of pixels, and N(h) is the number of pairs of data locations a vector h apart. The variogram is a useful measure of dissimilarity between spatially separate pixels (Jupp et al. 1988). The larger the value of c, the less similar are the pixels. As variance characterizes the distribution of a non-spatial random variable, variogram characterizes the distribution of a regionalized variable. Variogram has already found widespread use in remote sensing (e.g. Smith et al. 1989, Cohen et al. 1990, Bhatti et al. 1991, Gohin and Langlois 1993, Raffy 1993, Rossi et al. 1994). The joint spatial dependence of two co-dependent variables (in this study they refer to the two co-dependent images) is often measured using a cross-variogram.

6 Restoration of clouded pixels in multispectral remotely sensed imagery 2177 The cross-variogram can be estimated similarly by: c ij ðhþ~ 1 2NðhÞ XN ðhþ a~1 ½z i ðx a Þ{z i ðx a zhþ : zj ðx a Þ{z j ðx a zhþ, ð2þ where z i (x a ) and z j (x a ) represent the DN values of pixels in two random variables Z 1 (x) and Z 2 (x). To describe the experimental variograms and cross variograms for use in cokriging, a mathematical model of coregionalization (Journel 1989) has to be fit to the experimental variograms and cross variograms. Based on the model of coregionalization, cokriging considers the spatial dependence of two or more sets of variables and their interdependence simultaneously. The linear model of coregionalization builds each random variable Z(x) as a linear combination of L independent random variables, Y(x), whose variograms are given by g l (h), l51, 2, L. The linear model of coregionalization is the set of direct and cross variogram models c ij (h) defined as: c ij ðhþ~ XL l~1 b l ij g lðhþ V i, j, where c ij (h) represents variogram models between any two random variables Z 1 (x) and Z 2 (x), l represents each of the L variables Y(x), and g l (h) is a permissible variogram model. Here, the coefficients b l ij ~bl ji for all values of l, and for each l the matrix of coefficients: " # b l ii b l ij ð4þ b l ji must be positive definite. Note coefficients b l ij correspond to the sill or slope of the model g l (h). Since the matrix is symmetrical, it is sufficient that its determinant b l ii and b l jj and all its principal minor determinants are non-negative: qffiffiffiffiffiffiffiffiffi b l ij~ b l jiƒ b l ii bl jj: ð5þ b l jj ð3þ Equation (5), which is known as Schwarz s inequality, means that every basic structure appearing on a cross variogram model c ij (h) must be present in both direct variogram models c ii (h) andc jj (h). However, it is not necessary for the structure g l (h) appearing on both direct variogram models c ii (h) and c jj (h) to be present on the cross variogram model c ij (h). Further, a basic structure g l (h) must be absent on all cross variograms involving the variable if it is absent on a direct variogram. In practice, the following procedure may be taken to fit a linear model of coregionalization (Goovaerts 1997): (1) select the smallest set of basic structures g l (h) that describes the major features of all omnidirectional direct variograms; (2) for each basic structure g l (h), consider anisotropy only if the anisotropy is clearly evident on all directional variograms; (3) estimate the contributions (sill, slope) b l ij under the constraint of positive definite; (4) change a range or the type of basic variogram model to improve the overall goodness of the fit. Whenever a compromise is necessary, priority is given to the direct variograms, especially the primary variable.

7 2178 C. Zhang et al. 2.2 Traditional ordinary cokriging and standardized ordinary cokriging Cokriging is an optimal technique which provides unbiased estimates with minimum and known error using data defined on different attributes (Matheron 1973, 1979). By incorporating related secondary information, cokriging considers the spatial cross correlation between primary and secondary variables, thus improving the interpolation accuracy. Considering the problem of estimating a continuous DN value z of the clouded pixels in a primary remotely sensed image Z 1 at location x using the n 1 primary data fz 1 ðx a1 Þ, a 1 ~1,...,n 1 g available over the primary image Z 1 and the n 2 secondary data fz 2 ðx a2 Þ, a 2 ~1,...,n 2 g available over the secondary image Z 2, the traditional ordinary cokriging is written as: Z1 ðxþ~ Xn 1ðxÞ a 1 ~1 l a1 ðxþz 1 ðx a1 Þz Xn 2ðxÞ a 2 ~1 l a2 ðxþz 2 ðx a2 Þ ð6þ with two constraints: Xn 1 ðxþ l a1 a 1 ~1 ðxþ~1 and Xn 2 ðxþ l a2 a 2 ~1 ðxþ~0, where n 1 (x) is the known data in primary image Z 1, the l a1 ðxþ are the weights applied to the n 1 primary data Z 1, n 2 (x) is the known data in secondary image Z 2, and the l a2 ðxþ are the weights applied to the n 2 secondary data in Z 2. The weights are obtained by solving the following cokriging system: 8 np 1 ðxþ n2 P ðxþ l b1 ðxþc 11 xa 1 {x b1 z l b2 ðxþc 12 x a1 {x b2 zm1 ðxþ~c 11 ðx a1 {xþ a 1 ~1, :::, n 1 ðxþ >< >: b 1 ~1 np 1 ðxþ l b1 b 1 ~1 ðxþc 21 x a2 {x b1 b 2 ~1 n2 P ðxþ z l b2 b 2 ~1 np 1 ðxþ l b1 b 1 ~1 ðxþc 22 x a2 {x b2 ðxþ~1 np 2 ðxþ l b2 b 2 ~1 zm2 ðxþ~c 21 ðx a2 {xþ a 2 ~1, :::, n 2 ðxþ ðxþ~0, where C 11 and C 22 are the auto covariances of the primary and secondary images; C 12 and C 21 are cross covariances between the primary and secondary images. To reduce the occurrence of negative weights and avoid artificially limiting the impact of secondary data, a standardized ordinary cokriging (Isaaks and Srivastava 1989, p. 416) may be employed. The standardized ordinary cokriging estimator is written as: ð7þ ð8þ Z1 ðxþ~ Xn 1ðxÞ a 1 ~1 l a1 ðxþz 1 ðx a1 Þz Xn 2ðxÞ a 2 ~1 with the single condition that all weights must sum to 1: l a2 ðxþ½z 2 ðx a2 Þzm 1 {m 2 Š ð9þ np 1 ðxþ l a1 a 1 ~1 n 2ðxÞ ðxþz P l a2 a 2 ~1 ðxþ~1, where m 1 5E{Z 1 (x)} and m 2 5E{Z 2 (x)} are the stationary means of Z 1 and Z 2, respectively. 2.3 Verification methods To confirm the feasibility and reliability of the methods and assess their accuracies, we manually cut off a polygon area (test polygon) and assumed it represented a

8 Restoration of clouded pixels in multispectral remotely sensed imagery 2179 clouded area in the primary image. We examined how cokriging would perform by comparing the interpolated DN values obtained using standardized ordinary cokriging with the original DN values in the test polygon area. The standardized ordinary cokriging with the same parameters and the same linear coregionalization models in the previous interpolations were used to estimate the missing values of the pixels in the test polygon, where the truth is known. Root-mean-square-error (RMSE), standardized RMSE, and Q Q plots were employed to measure the overall accuracy of the interpolated pixels. Error distribution map is used to spatially assess the performance RMSE. The RMSE can be used to measure the overall accuracy of the interpolated pixels and the global average deviation between the interpolated pixels and the true pixels. The RMSE is calculated by: sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi P N z a {z 2 a a~1 RMSE~, ð10þ N where z* a represents DN values of the interpolated pixels and z a represents DN values of the true pixels. To enable mutual comparison between bands the RMSE can be normalized by the standard deviation of the corresponding bands. For each band, the standardized RMSE is calculated by: Standardized RMSE~ RMSE RMSE ~ q ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi s P, ð11þ 1 N ts N a~1 ðz a {zþ 2 where s is the standard deviation of the corresponding band, and z is the mean of Z and is defined as: z~ 1 N X N a~ Quantile Quantile plot. Quantile Quantile (Q Q) plot is a graphical data analysis technique for determining if two data sets have a common distribution. A Q Q plot is a plot of the quantiles of the first data set against the quantiles of the second data set. In a Q Q plot a 45u reference line is also plotted. If the two data sets have the same distribution, the points should fall approximately along this reference line. The greater the departure from this reference line, the greater the evidence that the two data sets have different distributions. Q Q plot can provide insight into the nature of the difference in many distributional aspects such as shifts in location, shifts in scale, changes in symmetry, and the presence of outliers. Q Q plot was employed in this study to determine whether or not DN values of interpolated pixels and those of truth pixels share the same distribution Error distribution map. Although RMSE, histograms and Q Q plots can help evaluate overall performance of cokriging interpolation, these global statistics cannot spatially assess the effectiveness of standardized ordinary cokriging as the performance of cokriging may be location-dependent. To spatially assess cokriging s effectiveness corresponding error (true DN value minus estimated DN value) z a : ð12þ

9 2180 C. Zhang et al. distribution maps for different bands for the test polygon area were created in this study. Error distribution maps can demonstrate the spatial distribution of interpolation errors Accuracy assessment of classification results of the interpolated images. To further confirm that the interpolated image is not only suitable for visual interpretation but also facilitates computer aided processing, we classified the interpolated images into thematic images using the unsupervised ISODATA clustering (Leica Geosystems 2003). A stratified random method was used to select verification points over the original masked areas. GPS data from fieldwork, a 1-m resolution grey IKONOS satellite image, and a 4-m resolution colour IKONOS satellite image were used as ground data for accuracy assessment. A confusion matrix and Kappa coefficient were used for accuracy assessment of classification results. 3. Case study 3.1 Background The first case study, located among the plateau karst lands of Yunnan Province, in south-west China, is a heterogeneous area with strong orographic uplift causing cloud cover to occur frequently. There are valley regions and hilly areas in the study area with diverse vegetation covers. As a means to test the application of the geostatistical techniques to interpolate the clouded pixels in multispectral remotely sensed satellite imagery, two Landsat 7 ETM + satellite imagery scenes, covering the same area but acquired at different time periods, were selected. The selection of the two image scenes was based on consideration of cloud coverage, area of interest and temporal variability. For illustrative purposes, a subset of each scene was used in the case study. Each subset contains pixels. Figure 1(a) shows the subset of the first ETM imagery scene with clouds and cloud shadows. This image, acquired on 1 January 2005, has a large amount of cloud coverage including both small and large clouds. Figure 1(b) illustrates the cloud and cloud shadow mask image with missing values, which served as the primary image in the case study to conduct the cokriging interpolation. The cloud and cloud shadow mask image was derived from a combination of unsupervised classification with ISODATA, using ERDAS Imagine v. 9.0 and manual editing. Figure 1(c) displays the subset of the secondary cloud-free ETM image, acquired on 26 April 2005, and this image was used as the secondary image in the case study for cokriging. From figure 1 it can be seen that the primary (a) (b) (c) Figure 1. Images used for cokriging interpolation. (a) Original ETM with clouds and cloud shadows (Band 4, 3, and 2). (b) Cloud and cloud shadow mask image (Bands 4, 3 and 2). (c) Secondary ETM for cokriging (Bands 4, 3, and 2).

10 Restoration of clouded pixels in multispectral remotely sensed imagery 2181 Table 1. Correlation coefficients and mean differences between cloudy ETM and secondary ETM. Band 1 Band 2 Band 3 Band 4 Band 5 Band 7 Correlation coefficient Difference between means for each band image has a different radiance from the secondary image due to the vegetation cover change. Table 1 shows that the correlation coefficients between these two images are different with different ETM bands. The correlation coefficients vary from to The highest correlation coefficient is with Band 4, while Bands 1 and 2 have the lowest correlation coefficients. Low correlation coefficients between the two images for Bands 1 and 2 imply weak direct correlation between the two datasets. Despite the weak correlation, the cokriging techniques described earlier were applied to assess whether the secondary data could enhance the interpolation accuracy of the clouded data. It can also be seen from table 1 that the mean DN values of the six optical bands for the two images are not close and the differences for the six optical bands are 8.2, 10.1, 13.5, 25.9, 16.9, and 23, respectively. The large differences of mean DN values between the primary and secondary images imply that common methods for replacing clouded pixels such as Maximum Value Composite, Minimum Value Composite, Simple Replacement and Image Fusion techniques may not work well enough in conducting computer aided image processing such as unsupervised classification. The objective of the case study is to explore another approach cokriging techniques for estimating the DN values of clouded pixels in the multispectral remotely sensed imagery. 3.2 Results Modelling spatial dependence between the primary and secondary images. Experimental variograms and cross variograms representing the coregionalization of the two images were computed using equations (1) and (2). Variogram and cross variogram calculation requires judgement and decisions by the analyst (Isaaks and Srivastava 1989). An appropriate lag distance for experimental variogram and cross variogram was determined by generating and visually comparing several experimental variograms and cross variograms calculated with different lag distances. After comparing several lags, the lag distance selected for the case study was 30 m, which is consistent with the spatial resolution of the Landsat 7 ETM + sensor. Visual inspection of preliminary experimental variograms showed this distance was long enough to capture the spatial character within the data, yet small enough to avoid the unreliable values calculated for larger lags. This distance was the one best describing the radiometric differences in the immediate neighbourhood of the central pixel. Figure 2 shows the experimental variograms of the original cloudy image and the secondary ETM image (dots) and their cross variograms (dots) with the lag distance of 30 m for the six optical bands. Because the number of pixels with DN values in both images used to compute the variogram is large, the experimental variograms and cross variograms had smooth, gradual, round shapes and were gently sloping near the sill. This made it easier to fit the experimental variograms and cross variograms using a linear model of coregionalization with

11 2182 C. Zhang et al. Figure 2. Experimental variograms of the original clouded image and the secondary ETM image (dots), their experimental cross variograms (dots), and corresponding fitted mathematical models (lines) for the six optical bands. standard mathematical models such as linear, spherical and exponential models for ordinary cokriging interpolation. The ultimate goal for variogram modelling in the case study was to interpolate values of clouded pixels using cokriging. Thus, it was important to make the variogram models fit precisely to the experimental variograms and cross variograms. We used the aforementioned procedure for fitting a linear model of coregionalization. The lines in figure 2 illustrate the fitted linear model of coregionalization for the six optical bands. Each variogram or cross variogram was fit with the same basic model as in intrinsic coregionalization (Journel and Huijbregts 1978). The coefficient matrices of these models were positive definite. Each fitted variogram or cross variogram model was composed of three structures: a nugget structure, an exponential structure and a spherical structure: cðhþ~c 0 zc 1 1{exp { 3h a 1 zc 2 " 3h { 1 # h 3 2a 2 2 a 2, ð13þ where c 0 is the nugget variance, c 1 is the sill (contribution) of the exponential structure, a 1 is the range parameter of the exponential structure, c 2 is the sill (contribution) of the spherical structure, and a 2 is the range parameter of the spherical structure. Table 2 shows the parameters of the fitted linear model of coregionalization including variograms of the primary image and the secondary image and cross variograms between the two images for cokriging. Parameters of the variogram models include the following information: (1) nugget, standing for the level of random variation within the data; (2) sill, representing the total magnitude

12 Restoration of clouded pixels in multispectral remotely sensed imagery 2183 Table 2. Parameters of the fitted variogram models and cross variogram models for cokriging. Band Variogram Model type Nugget Sill Structured variance Range Band 1 Clouded image Exponential Spherical Secondary image Exponential Spherical Cross variogram Exponential Spherical Band 2 Clouded image Exponential Spherical Secondary image Exponential Spherical Cross variogram Exponential Spherical Band 3 Clouded image Exponential Spherical Secondary image Exponential Spherical Cross variogram Exponential Spherical Band 4 Clouded image Exponential Spherical Secondary image Exponential Spherical Cross variogram Exponential Spherical Band 5 Clouded image Exponential Spherical Secondary image Exponential Spherical Cross variogram Exponential Spherical Band 6 Clouded image Exponential Spherical Secondary image Exponential Spherical Cross variogram Exponential Spherical of spatial variability; (3) structured variance, revealing the contribution of the spatial variability except for random variation; and (4) range, describing the spatial dependence of the spatial variability. Ideally the variogram nugget, which is nonzero variance and represents unexplained or random variance (Deutsch and Journel 1998), should be zero for remotely sensed imagery. This is because nuggets are usually caused by short range variation missed by sparse samples. Because remotely sensed imagery data are exhaustive data, there is no missed short range variation. However, there is a nonzero value for Bands 1, 2, and 4 of the secondary image, respectively. One reason for the nugget effect in those bands may be that there is a distinct jump in variance at an initial short increment of distance. Another reason may be measurement errors from sources such as discretization (analogue to digital conversion), transmission, and ground segment processing (Woodcock et al. 1988b).

13 2184 C. Zhang et al Estimating DN values of clouded pixels using both standardized ordinary cokriging and traditional ordinary cokriging. The fitted linear models of coregionalization in figure 2 together with the available DN values in the primary image and all DN values in the secondary image were then used to estimate missing DN values of clouded pixels in the primary image. Figures 3(a) and (b) show the interpolated results using traditional ordinary cokriging and standardized ordinary cokriging, respectively. Visual inspection of the results demonstrates that both methods produced similar results and there was no large difference existing between the traditional ordinary cokriging and standardized ordinary cokriging. To more precisely quantify the difference between the two methods, correlation coefficients between interpolated DN values from both methods over the original masked areas for the six optical bands were calculated as shown in Table 3. The high correlation coefficients confirm that the two methods produced similar results. Standardized ordinary cokriging generated a slightly improved similarity of spatial structure to the secondary ETM image than traditional ordinary cokriging (figure 1(c)), most likely because it added the influence of the secondary image by creating a new secondary image with the same mean as the primary image. Both methods were effective for interpolating the missing values in the small clouds, while the reliability of the estimates decreased in the large size clouds. Both methods captured the linear river features well (near the bottom left of the image). Note that it is difficult to interpolate the linear features for most interpolation methods. The traditional ordinary cokriging and standardized ordinary cokriging captured the missing linear river features in the case study well due to the incorporation of the information from the secondary image. From figures 3(a) and (a) (b) (c) (d) (e) (f ) Figure 3. Comparison of traditional ordinary cokriging, standardized ordinary cokriging and ordinary kriging. (a) Traditional ordinary cokriging results (Bands 4, 3 and 2). (b) Standardized cokriging results (Bands 4, 3, and 2). (c) Ordinary kriging results (Bands 4, 3, and 2). (d) Simple replacement. (e) Image fusion using ordinary linear regression. (f) Image fusion using Brovey transform.

14 Restoration of clouded pixels in multispectral remotely sensed imagery 2185 Table 3. Correlation coefficients between interpolated DN values from traditional ordinary cokriging and standardized ordinary cokriging over the original masked areas for the six optical bands. Band 1 Band 2 Band 3 Band 4 Band 5 Band 7 Correlation coefficient (b), we can also see that the replaced locations for the largest cloud and cloud shadow areas are easy to identify and their edges have visible seam-lines. Comparing the interpolation results with the secondary ETM image (figure 1(c)), it is evident that the detailed spatial structures or characteristics of the largest cloud and cloud shadow areas were not captured well although the basic spatial structural characteristics were still retained. Obvious image interpolation-related artefacts such as smoothing effects were visually apparent throughout the cloud areas. However, for the small cloud and cloud shadow areas the interpolation results appeared to be good and they matched spatially in the image. We also can observe that for the small cloud and cloud shadow areas the spatial continuity and patterns were well interpolated with little interpolation artefacts. Smoothing effects were not significant and detailed spatial variation could be found in the interpolated small cloud and shadow areas Comparing ordinary cokriging and ordinary kriging. To confirm whether the incorporation of the information from the secondary image could improve the interpolation results or not, we estimated DN values of clouded pixels in the primary image using ordinary kriging. Figure 3(c) shows the results of ordinary kriging. Compared with figure 3(a) and (b), it can be seen that without the incorporation of the information from the secondary image ordinary kriging cannot capture even the general spatial structure and there is obvious smoothing effect in the results of the large and medium size clouds. The linear river feature (near the bottom left of the image) disappeared in figure 3(c). This verified that cokriging could capture well the spatial structure and missing linear features and this should attribute to the incorporation of the information from the secondary image Comparing ordinary cokriging and other image fusion methods. To confirm the advantages of cokriging, we also estimated the DN values of clouded pixels in the primary image using other methods such as simple replacement, image fusion using ordinary linear regression, and image fusion using Brovey transform. Figure 3(d), 3(e) and 3( f ) show the interpolated images using these methods. Due to the variable nature of the vegetation phenology, the fusion results from these methods are not good and the fused boundaries are clear even for small clouds. The simple replacement method (figure 3(d)) generated the least impressive results and the fused areas are radiometrically different from the original image. The image fusing using Brovey transform worked better than both simple replacement and image fusing using ordinary linear regression. However, for the small clouds the image fusing using Brovey transform still has problems with generating radiometrically matched imagery. Although ordinary cokriging methods produced obvious smoothing effects for large size clouds, they were the most effective methods for interpolating the missing values for small clouds Evaluating the accuracy of interpolation results. Figure 4(a) shows the original image with the test polygon for validation purposes and a thin yellow

15 2186 C. Zhang et al. (a) (b) Figure 4. Images with a test polygon for validation purpose. (a) Original image (Bands 4, 3, and 2, and the yellow thin line indicates the test polygon location). (b) Interpolated image (Bands 4, 3, and 2) with standardized cokriging (the yellow thin line indicates the interpolated polygon location). line indicates the test polygon location. Figure 4(b) illustrates the interpolated image (the thin yellow line indicates the interpolated polygon location). To see clearly the cokriging performance, we zoomed into both images with the test polygon in the centre (figure 5). Figure 5(a) and (b) show the zoomed-in original and interpolated images, respectively. Compared visually with the original image, the continuity of the majority of the scene in the interpolated pixel region was good and spatial patterns were restored well. However, there were a few noticeable deleterious smoothing effects on the red linear features (agricultural fields), which showed a less consistent resemblance to the true data. Table 4 shows the RMSE and Standardized RMSE for the six optical bands. The RMSE values range from 0.12 to 0.32, which suggests that the standardized ordinary cokriging performed well overall. Small RMSE values can be achieved even when the correlation coefficients between the primary and secondary images are small. For example, while the correlation coefficients between the primary and secondary images for Bands 1 and 2 are only 0.42 and 0.54, respectively, their corresponding RMSE values are still small (0.12 and 0.13, respectively). This means that it is possible to improve the estimation accuracy of a primary image by incorporating information of the secondary image even though it may lack simple (a) (b) Figure 5. Zoomed-in images with a test polygon for validation purpose. (a) Zoomed-in original image (Bands 4, 3, and 2, and the yellow line indicates the test polygon location). (b) Zoomed-in interpolated image (Bands 4, 3, and 2) with cokriging (the yellow line indicates the interpolated polygon location).

16 Restoration of clouded pixels in multispectral remotely sensed imagery 2187 Table 4. Root-mean-square-error (RMSE) and standardized RMSE for the six optical bands (Note: the unit is Digital Number). Band 1 Band 2 Band 3 Band 4 Band 5 Band 7 RMSE Standardized RMSE correlation with the primary image. The standardized RMSE values reveal that Bands 4, 5, and 7 produced smaller standardized RMSE values, while Bands 1 and 2 showed somewhat higher values. It is interesting to note that the results are coincident with the correlation coefficients between the primary and secondary images for different bands. The cokriging interpolation performed better in Bands 4, 5 and 7 because in those bands the secondary images are more highly correlated than in other bands (table 1). Figure 6 shows Q Q plots of the distribution of true DN values versus those of estimated DN values in the test polygon for the six optical bands. While the Q Q plot for Band 4 has values slightly below the 45u line, plots for most bands show values plotted near the 45u line. This indicates that there are similar distributions between interpolated pixel values and truth pixel values in the test polygon for most bands. For Band 4, the estimated DN values are smaller than the corresponding true DN values indicating that the DN values of pixels were underestimated. This may be because the estimated DN values for Band 4 were influenced more by the corresponding pixels in the secondary image due to a higher correlation between the primary and secondary images. All bands show discrepancies at the upper tails of their plots. This indicates that the few pixels with extremely large or small DN values were either underestimated Figure 6. Q Q plots of the distribution of true DN values versus that of estimated DN values in the test polygon area for the six optical bands.

17 2188 C. Zhang et al. Figure 7. Spatial distribution maps of errors (true DN value minus estimation DN value) for bands 1 and 2 (for the test polygon area shown in figures 4 and 5). or overestimated and that it is difficult to estimate accurately the extreme DN values even using secondary information through cokriging. Figure 7 shows the corresponding error distribution maps for bands 1 and 2. The error distribution map is useful in analysing the reliability of the DN value of each pixel in the test polygon. A yellow colour shows the locations of accurate estimates. Red and green colours show locations of positive and negative errors, respectively. One expected pattern is that the error is greatest in those areas that have extreme DN values. Note that these errors correspond closely to the upper tail deviations in the Q Q plots. In general, most pixels in the test polygon show small deviation (yellow colour, light red colour and light green colour), and this means that it is feasible to interpolate the missing values of clouded pixels using the standardized ordinary cokriging. Although some pixels have relatively large deviations (deep red colour and deep green colour), they only account for a small percentage of the interpolated pixels. From the error distribution maps it also can be seen that map errors are generally spatially autocorrelated (pixels with relatively large deviations tend to occur together). One expected pattern is that the error is greatest in those areas that have the least consistent DN values or the extreme DN values. For example, locations with relatively large deviations are composed of pixels whose DN values are different from those of their neighbouring pixels. It is difficult to accurately interpolate this kind of pixels according to the DN values of their neighbouring pixels. The pixels sharing similar DN values with a few neighbouring pixels are underestimated. To further confirm that the interpolated images can be used for classification, the interpolated images using standardized ordinary cokriging and image fusion with Brovey transform were classified into six-class thematic images. The data sets used to calculate confusion matrix and Kappa coefficient include a set of 100 randomly selected points over the whole original masked areas and a set of 50 randomly selected points over the small cloud areas. Table 5 shows the accuracy assessment results. The interpolated image using standardized ordinary cokriging correctly classified 66.34% of points over the whole original masked areas and yielding an error matrix with a Kappa coefficient of agreement of , while the interpolated image using image fusion with Brovey transform only correctly classified 30.00% of points and yielding an error matrix with a Kappa coefficient of agreement of

18 Restoration of clouded pixels in multispectral remotely sensed imagery 2189 Table 5. Accuracy assessment of classification results of the interpolated images using standardized ordinary cokriging and image fusion with Brovey transform. Percentage correct overall Kappa coefficient agreement Whole original masked areas using cokriging 66.34% Small cloud areas using cokriging 83.67% Whole original masked areas using image fusion 30.00% Small cloud areas using image fusion 61.22% Both methods generated more accurate classification results over small cloud areas. The interpolated image using standardized ordinary cokriging correctly classified 83.67% of points over small cloud areas and yielding an error matrix with a Kappa coefficient of agreement of , while the interpolated image using image fusion with Brovey transform correctly classified 61.22% of points and yielding an error matrix with a Kappa coefficient of agreement of The results show that the interpolated image using standardized ordinary cokriging may be used for computer aided processing such as unsupervised classification even under conditions where the radiometric values of the secondary image are different from those of the original cloudy image. However, the image fusion methods may yield poor results if the radiometric values of the secondary image are not close to those of the original cloudy image. 3.3 Second case study The second case study was conducted to further confirm the effectiveness of the cokriging method in a different landscape. This case study is located in the United States Midwest region in Kent, Ohio. This area is a relatively homogenous region with dominantly agricultural landscape. Two Landsat 5 ETM satellite imagery scenes, covering the same area but acquired at different time periods, were used for the case study. Figure 8(a) and (b) show the Landsat 5 ETM image with clouds and cloud shadows and its mask image (Band 3), respectively. Figure 8(d) illustrates the secondary Landsat 5 ETM image used in standardized ordinary cokriging (Band 3). Figure 8(c) displays the cokriging results. Visual examination of the resulting image indicated that the standardized ordinary cokriging interpolation approach restored the image well even for large size cloud and cloud shadow areas and the continuity of image features in the interpolated areas was good. No obvious visual differences appeared between interpolated and non-interpolated pixels. The summary statistics, correlation coefficients, histograms and accuracy assessment results indicated that the cokriging results were more accurate than the results in the first case study. Classification of the interpolated image gave results similar to that of analysis from clear (i.e. non-cloudy) images within the same season (almost no temporal change). The better results in the second case study may be due to the more homogenous landscape, which exhibits small spatial changes. 4. Discussion and conclusions The two case studies show that it is practical to replace cloud pixels of multispectral remotely sensed imagery using a coherent model of coregionalization and ordinary cokriging techniques. The advantage of the ordinary cokriging method is that it can restore cloudy imagery by using imagery with different vegetation phenology. While

19 2190 C. Zhang et al. (a) (b) (c) Figure 8. Images used for standardized ordinary cokriging interpolation in the second case study. (a) Original Landsat 5 ETM image with clouds and cloud shadows (Bands 3). (b) Cloud and cloud shadow mask image (Bands 3). (c) Standardized ordinary cokriging results (Bands 3). (d) Secondary Landsat 5 ETM image for cokriging (Bands 3). various methods such as simple replacement and image fusion can restore cloudy imagery well if the images combined satisfy some criteria such as minimum date separation and low temporal variability, they don t address the intractable restoration problem when using images from different dates with large phenological differences. The ordinary cokriging approach can resolve this problem by taking full advantage of the spatial information in the cloudy imagery. This occurs by taking into account explicitly the spatial correlation information in the original cloudy imagery and does not require that the secondary imagery has vegetation phenology similar to the original cloudy imagery. It is possible to restore cloudy imagery using ordinary cokriging by incorporating information of secondary imagery that may lack simple correlation with the cloudy imagery. For validating and obtaining the real estimation error, a cloud-free area was intentionally cut off from the image in the first case study. The validation results demonstrate that the continuity and accuracy of the interpolated image are good, although the interpolation approach cannot perfectly restore the image. No obvious visual differences were noted between the interpolated and non-interpolated pixels in most areas. While there are still some deviations or errors of the DN values in the interpolated pixels, such deviations are typically minor and within the acceptable error limits of variability for some image classification methods. As pointed out by Arellano (2004), images derived by image fusion techniques are inappropriate for computer aided processing if input images differ greatly in the radiometric values. (d)