Fusion of Landsat 8 OLI and Sentinel-2 MSI data

Transcription

1 1 Fusion of Landsat 8 OLI and Sentine-2 MSI data Qunming Wang a, George Aan Backburn a, Aex O. Onojeghuo a, Jadu Dash b, Lingquan Zhou b, Yihang Zhang c,d, Peter M. Atkinson b,e,f a Lancaster Environment Centre, Lancaster University, Lancaster LA1 4YQ, UK b Geography and Environment, University of Southampton, Highfied, Southampton SO17 1BJ, UK c Institute of Geodesy and Geophysics, Chinese Academy of Sciences, Wuhan , China d University of Chinese Academy of Sciences, Beijing , China e Facuty of Science and Technoogy, Engineering Buiding, Lancaster University, Lancaster LA1 4YR, UK f Schoo of Geography, Archaeoogy and Paaeoecoogy, Queen's University Befast, BT7 1NN, Northern Ireand, UK E-mai: wqm11111@126.com Abstract: Sentine-2 is a wide-swath and fine spatia resoution sateite imaging mission designed for data continuity and enhancement of the Landsat and other missions. In this paper, a new approach is presented for fusion of Landsat 8 and Sentine-2 data to coordinate their spatia resoutions for continuous goba monitoring. The advanced area-to-point regression kriging (ATPRK) approach was empoyed for the fusion probem, where the 30 m spatia resoution Landsat 8 bands are downscaed to 10 m using 10 m Sentine-2 bands as covariates. To account for the and-cover/and-use (LCLU) changes that may have occurred between the Landsat 8 and Sentine-2 images, the Landsat 8 PAN band was aso incorporated in the fusion process. The experimenta resuts showed that the proposed approach is effective for fusing Landsat 8 with Sentine-2 data, and the use of the PAN band can decrease the errors introduced by LCLU changes. By fusion of Landsat 8 and Sentine-2 data, more frequent observations can be produced for continuous monitoring, and the observed 30 m Landsat 8 data can be downscaed to a finer spatia resoution of 10 m. The products have great potentia for timey monitoring of rapid changes on the Earth s surface. Keywords: Landsat 8, Sentine-2, image fusion, downscaing, area-to-point regression kriging (ATPRK), goba monitoring. 1. Introduction Landsat data have been used widey for goba monitoring, due to their free avaiabiity and reguar revisit capabiities. The Landsat 5 sateite equipped with Thematic Mapper (TM) sensor was aunched in 1984, but the TM sensor stopped transmitting in November As the successor of Landsat 5, the Landsat 7 sateite was aunched in Apri In May 2003, however, the scan-ine corrector (SLC) of the Landsat 7 Enhanced Thematic Mapper Pus (ETM+) sensor faied permanenty, resuting in SLC-off images with around 22% dead pixes from May 2003 to present [1]. To cope with the SLC-off issue of the Landsat 7 ETM+ sensor, the new generation Landsat 8 sateite equipped with Operationa Land Imager (OLI) and Therma Infrared Sensor (TIRS) sensors was aunched in February 2013 and the sensors are now in operation routiney acquiring goba remote sensing data [2]. The imitation of Landsat is that the sateites can ony revisit the same area every 16 days. In most cases, the acquired Landsat data for specific areas can be contaminated by coud and shadow, meaning that obtaining one cean Landsat image per month is considered a good outcome [3]. The temporay sparse time-series Landsat data are, therefore, unsuitabe for goba monitoring of rapid changes on the Earth s surface, such as urbanization (especiay in highy deveoped cities, such as Shenzhen in China) [4]-[5], deforestation and forest degradation (such as in the Amazon rainforest) [6], and rapid phenoogy changes (e.g., due to crop harvesting) [7]. To obtain more frequent Landsat data for timey monitoring, spatio-tempora fusion methods have been deveoped to downscae 500 m spatia resoution images from the Moderate Resoution Imaging Spectroradiometer (MODIS) to 30 m resoution Landsat-ike images [8]-[13]. The MODIS sensor can revisit the same area on a daiy basis, which is of great use for near rea-time monitoring at the goba scae. Some

2 2 spatio-tempora fusion methods require at east one pair of MODIS-Landsat images acquired on the same day to guide the downscaing process of MODIS on other days [8]-[10]. This can be demanding for a given period. Moreover, from the observed spatia resoution of 500 m to the target resoution of 30 m, the downscaing process invoves a arge zoom factor of 16, indicating arge uncertainties given the i-posed nature of the probem. In addition, the difference between the geographic coordinate system of MODIS (in the Sinusoida projection) and Landsat (in the UTM/WGS84 projection) may potentiay ead to additiona uncertainties in downscaing. Very recenty in June 2015, the Sentine-2 sateite was aunched for data continuity and enhancement of the Landsat and SPOT missions. Sentine-2 is a wide-swath and fine spatia resoution sateite imaging mission of the European Space Agency (ESA) deveoped in the framework of the European Union Copernicus programme [14]-[16]. Sentine-2 data cover 13 spectra bands, with four bands at 10 m, six bands at 20 m and three bands at 60 m spatia resoution. The sensor revisits the same area every ten days with a constant viewing ange. The free access, fine spatia resoution, goba coverage and fine tempora resoution make the Sentine-2 data of great utiity for a wide range of appications based on remote sensing. The Sentine-2 bands have corresponding waveengths to the Landsat bands; see the exampe for Landsat 8 and Sentine-2 data in Tabe 1 [17]. Moreover, the Sentine-2 products pubished onine have the same geographic coordinate system as Landsat products. The free access of both Sentine-2 and Landsat data, the same waveengths, and the same geographic coordinate system provide an exceent opportunity to combine these two types of data for more continuous monitoring at the goba scae. As seen from Tabe 1, however, the spatia resoutions of the two types of data are different and Sentine-2 has finer spatia resoutions (10 and 20 m) than Landsat (30 m). Tabe 1 Band waveengths for Landsat 8 and Sentine-2 data Landsat 8 Sentine-2 Band number Waveength (nm) Spatia resoution (m) Band number Waveength (nm) Spatia resoution (m) In this paper, for the first time, we fuse Landsat 8 and Sentine-2 data by addressing the incompatibiity probem of spatia resoution to achieve potentiay continuous monitoring. There are two potentia schemes for the fusion task. The first is to upscae the 10 or 20 m Sentine-2 data to 30 m to match the spatia resoution of Landsat 8. This scheme is straightforward, but wastes the vauabe 10 m information obtained by Sentine-2. The second scheme is to downscae the 30 m Landsat 8 data to 10 m to match the spatia resoution of Sentine-2. This scheme aims to take fu advantage of the avaiabe information in both Landsat 8 and Sentine-2. For goba monitoring, anaysts aways prefer to obtain as much detaied spatia information as possibe. In this paper, the second scheme is considered. Specificay, we downscae the 30 m bands 2-7 of Landsat 8 to 10 m, with the aid of 10 or 20 m resoution data in the corresponding Sentine-2 bands 2, 3, 4, 8, 11 and 12. We suggest that the Sentine-2 bands provide vauabe information at the 10 m target spatia resoution, which can greaty decrease the uncertainty in downscaing Landsat 8 data. This downscaing issue is aso termed image fusion in remote sensing. The significance of fusing Landsat 8 with Sentine-2 data is twofod. 1) It can produce more frequent time-series images for continuous goba monitoring. More precisey, in theory, every month Sentine-2 can provide an additiona three observations as suppementary information to the two Landsat observations, thus, increasing the number of observations every month to five. 2) The observed 30 m Landsat 8 data can be downscaed to 10 m and continuous monitoring can be made at

3 3 a finer spatia resoution than the origina Landsat data. Many features (e.g., urban fabric, sma residentia buidings, roads and other inear features) that cannot be observed ceary in the origina 30 m Landsat data can be shown more expicity in the 10 m downscaed products. Over the past decades, various approaches have been deveoped for image fusion, such as the intensity-hue-saturation [18], Brovey [19], principa component anaysis [20], a trous waveet transform (ATWT) [21], high-pass fiter (HPF) [22], smoothing fiter-based intensity moduation (SFIM) [23], and sparse representation [24] methods. There are severa reviews of the avaiabe image fusion approaches [25]-[28]. Recenty, area-to-point regression kriging (ATPRK) in geostatistics was proposed for image fusion [29], [30]. ATPRK treats the coarse band as the primary variabe and the fine spatia resoution band (hereafter, fine band) as a covariate. It is an advanced image fusion approach which has the appeaing advantage of precisey preserving the spectra properties of the observed coarse images (i.e., perfecty coherent). The advantages of ATPRK over other geostatistica approaches (such as kriging with externa drift [31] and downscaing cokriging [32]-[34]) have been presented both theoreticay and experimentay in our previous research [29], [30]. ATPRK is a user-friendy approach and accounts expicity for size of support (pixe), spatia correation, and the point spread function (PSF) of the sensor. Motivated by the advantages and encouraging performance of ATPRK in our previous research, in this paper, ATPRK is empoyed for fusion of Landsat 8 and Sentine-2 data. When fusing Landsat with Sentine-2 data, and-cover/and-use (LCLU) changes may have taken pace during the period of time between the acquisition of both datasets and this can be a critica probem bringing great chaenges. This is prominent when abrupt changes occur from Sentine-2 to (reativey) coarse Landsat images. For exampe, hypotheticay, in a Sentine-2 image a region may be entirey covered by bare soi, but in a Landsat 8 image some parts of the region may be changed to be mixed with both bare soi and sma residentia buidings (smaer than 30 m). In this case, the 10 or 20 m Sentine-2 image of this region cannot provide usefu spatia information for downscaing the Landsat data (e.g., downscaing the sma residentia buidings that cannot be observed by Sentine-2 at a). Therefore, using ony the Sentine-2 image may sometimes be insufficient for accuratey reproducing the LCLU changes that have occurred between the acquisition time of the Landsat and Sentine-2 data. It is worth noting that the Landsat 8 OLI sensor aso provides a 15 m panchromatic (PAN) band (band 8) covering the same scene with the 30 m mutispectra bands. Athough coarser than the 10 m target spatia resoution, the PAN band is acquired at the exacty same time with the 30 m mutispectra bands and can revea the changes at a finer spatia resoution than 30 m which may not be observed by the 10 m Sentine-2 image (e.g., the abrupt changes). In this paper, we fuse Landsat 8 with Sentine-2 data by taking fu advantage of a the avaiabe information provided by the two types of sensors. Specificay, based on the advanced ATPRK approach, we propose a new fusion approach to incorporate 10 or 20 m Sentine-2 and 15m PAN images to downscae 30 m Landsat 8 images to 10 m. The remainder of this paper is organized as foows. Section 2 first briefy introduces the theoretica basis of ATPRK, and then the principes of the proposed ATPRK-based fusion approach. The experimenta resuts are provided in Section 3 to demonstrate the appicabiity of the proposed fusion approach. Section 4 provides some further discussions, foowed by a concusion in Section Methods 2.1 ATPRK ATPRK consists of regression-based overa trend (i.e., the spatiay varying mean of a spatia process) estimation and area-to-point (ATPK)-based residua (i.e., the variation remaining after remova of the trend) downscaing [29], [35], [36]. The principe of ATPRK is briefy introduced beow.

4 4 Suppose ZV( x i) is the random vector of pixe V centered at x i (i=1,,m, where M is the number of pixes) k in coarse band (=1,,L, where L is the number of coarse bands), and Z ( x ) is the random vector of pixe v centered at 2 x (j=1,, j MF, where F is the spatia resoution (zoom) ratio between the coarse and fine bands) in fine band k (k=1,,k, where K is the number of fine bands). Using the primary variabe Z ( x ) at coarse k spatia resoution and covariate Z ( x ) at fine spatia resoution as inputs, ATPRK aims to predict the target v j variabe Zv( x ) for a fine pixes in a coarse bands. Denoting the predictions of the regression and the ATPK parts as ˆ Z ( ) v1 x and ˆ Z ( ) v2 x, the ATPRK prediction is given by ˆ ˆ ( ) ˆ Zv x Zv 1( x) Zv2( x ). (1) At a specific ocation x 0, the regression prediction is a inear combination of the K fine pixes in K fine bands K k 0 v1 0 k v 0 v 0 k 0 v Zˆ ( x ) a Z ( x ), Z ( x ) 1. (2) Assuming the reation in (2) is invariant with spatia scae, the coefficients { a k 0,..., K} in (2) are cacuated according to the reationship between the observed coarse band and the upscaed bands (k=1,,k) from the origina K fine bands. K k 0 V ( ) k V ( ) V ( ), V ( ) 1 k 0 V Z x a Z x R x Z x x (3) where the coefficients are estimated by ordinary east squares [37]. R ( x ) is a residua term that needs to be downscaed to the fine spatia resoution in the foowing. ATPK is performed in the second-stage to downscae the coarse residuas R ( x ) in (3) to the target fine spatia resoution. The fine residua at ocation x 0 is predicted as N N v2 0 i RV i i i 1 i 1 Zˆ ( x ) ( x ), s.t. 1 (4) in which i is the weight for the ith coarse residua centered at x i and N is the number of neighboring coarse pixes. The weights { i i 1,..., N} are cacuated according to the kriging matrix VV ( x1, x1)... VV ( x1, xn ) 1 vv ( x0, x1) VV ( xn, x1)... VV ( xn, xn ) N vv ( x0, xn ) where ( x, x ) is the coarse-to-coarse semivariogram between coarse pixes centered at x i and VV i j j V k V i k Z V (5) x j in band, vv ( x0, x j) is the fine-to-coarse semivariogram between fine and coarse pixes centered at x 0 and x j in band, and is the Lagrange mutipier. Suppose s is the Eucidean distance between the centroids of any two pixes and h () s is the PSF of the sensor. () s and () s are cacuated by convouting the fine-to-fine V semivariogram () s with the PSF h () s as foows vv V VV vv ( s) ( s)* h ( s ) (6) vv vv V VV vv hv hv vv ( s) ( s)* ( s)* ( s ) (7) where * is the convoution operator. () s can be estimated by deconvoution of the coarse semivariogram

5 5 cacuated from the coarse residua image RV ( x ). Detais on the deconvoution approach can be found in [29]. Based on the hypothesis that the coarse pixe vaue is the average of the fine pixe vaues within it, the sensor PSF can be defined as foows [32]-[34] 1, if x V ( x ) hv ( x ) S (8) V 0, otherwise in which S V is the size of pixe V and V ( x ) is the spatia support of pixe V centered at x. 2.2 The proposed ATPRK-based approach for fusion of Landsat 8 and Sentine-2 The proposed fusion approach makes fu use of the information in 10 or 20 m Sentine-2 bands, 30 m Landsat 8 mutispectra bands and the 15 m Landsat 8 PAN band to produce 10 m Landsat 8 mutispectra bands. This section introduces the principe of the ATPRK-based fusion approach. As seen from Tabe 1, 20 m Sentine-2 bands 11 and 12 have the same waveength as 30 m Landsat 8 bands 6 and 7 and the former wi be fused with the atter in downscaing. Obviousy, the two 20 m Sentine-2 bands need to be downscaed to 10 m in advance to provide the 10 m reference for Landsat 8 bands 6 and 7. For the downscaing process, the 10 m information in Sentine-2 bands 2, 3, 4, and 8 can be used. This process can be achieved using the ATPRK approach, where the 10 m Sentine-2 bands 2, 3, 4, and 8 are treated as fine bands (covariates). A four 10 m bands can be used straightforwardy in ATPRK by mutipe regression in (2) and (3). However, the consideration of a fine bands in the regression mode in ATPRK may over-generaize. In this paper, for each of the 20 m bands 11 and 12, a 10 m band with the greatest correation (quantified by the correation coefficient (CC)) with it was seected from the 10 m bands 2, 3, 4, and 8 and used as the covariate in ATPRK. The waveength of the Landsat 8 PAN band is nm, which covers those of Landsat 8 bands 2-4 but not bands 5-7. Thus, compared with 10 m Sentine-2 images, the Landsat 8 PAN image may not aways be abe to provide more usefu spatia information for downscaing bands 5-7. On the other hand, if the LCLU changes between the Sentine-2 and Landsat 8 images are arge or the acquisition time of the two types of data is very different, using ony Sentine-2 bands 8, 11 and 12 that have the same waveength with Landsat 8 bands 5-7 may aso not be abe to provide sufficient textura information at 10 m spatia resoution for the changed areas. Based on this hypothesis, in the proposed fusion approach, Landsat 8 bands 2-4 are downscaed using both Sentine-2 (corresponding bands 2-4) and Landsat 8 PAN images as auxiiary data. For Landsat 8 bands 5-7, Sentine-2 images are sti used for fusion as they provide spatia information at the desired 10 m spatia resoution, which is particuary vauabe for unchanged areas. However, whether the Landsat 8 PAN image shoud be considered for Landsat 8 bands 5-7 depends on its correation with them. Fig. 1 iustrates schematicay the fusion approach. For the th (=5, 6 and 7) band of Landsat 8, we denote the CC between the band and the Landsat 8 PAN band as CC( L, L ), and the CC between the band and the corresponding kth Sentine-2 band (k=8, 11 and PAN 12) as CC( L, S ). If CC( L, L ) > CC( L, S ), the PAN image is used in fusion; otherwise, the PAN image k PAN k is not considered as hepfu auxiiary data and ony the Sentine-2 images are used. Note that before cacuating the CCs, the Landsat 8 PAN band and Sentine-2 bands 8, 11 and 12 need to be upscaed to 30 m to match the spatia resoution of the Landsat 8 bands 5-7. The proposed fusion approach is, thus, impemented as foows. 1) Based on ATPRK, the 20 m Sentine-2 bands 11 and 12 are downscaed to 10 m using the 10 m bands 2, 3, 4, and 8 as covariates. For each 20 m band, a 10 m band with the argest CC is seected from bands 2, 3, 4, and 8. 2) Based on ATPRK, the 30 m Landsat 8 bands 2-4 are downscaed to 15 m using the 15 m Landsat 8 PAN band as the covariate. 3) The 10 m Sentine-2 bands 2-4 are downscaed to 5 m by the simpe bicubic interpoation.

6 6 4) Treating the 5 m Sentine-2 bands as covariates, the 15 m pan-sharpening resuts of step 2) are further downscaed to 5 m using ATPRK. In this process, each 15 m band of Landsat 8 is downscaed using the 5 m band with the same waveength (see Tabe 1). 5) The 5 m resuts of step 4) are upscaed to 10 m to produce the fina resuts for Landsat 8 bands ) For Landsat 8 bands 5-7, the CCs between them and the Landsat 8 PAN band (i.e., CC( L, L ), =5, 6 and 7) and Sentine-2 bands 8, 11 and 12 (i.e., CC( L, S ), k=8, 11 and 12) are cacuated. 7) For the th of Landsat 8 bands 5-7, if CC( L, L ) > CC( L, S ), the PAN image is used, and the PAN downscaing process for this Landsat 8 band is simiar to that for Landsat 8 bands 2-4 (see steps 2-5), where Sentine-2 band 8, 11 or 12 is invoved instead. Otherwise, ony the corresponding 10 m Sentine-2 band is considered as the covariate and fused with the 30 m band to produce the fina 10 m downscaing resut for this Landsat 8 band. k k PAN 10 m Sentine-2 bands 2-4 at t 1 30 m Landsat 8 bands 2-4 at t 2 15 m Landsat 8 PAN band at t 2 10 m Landsat 8 bands 2-4 at t 2 20 m Sentine-2 bands 11, 12 at t 1 10 m Sentine-2 30 m Landsat 8 bands 8, 11, 12 at t 1 bands 5-7 at t 2 10 m Landsat 8 bands 5-7 at t 2 Fig. 1 The proposed image fusion approach for downscaing Landsat 8 mutispectra bands 2-7 at t2, which uses both a Sentine-2 image at t1 and a Landsat 8 PAN image at t2 as auxiiary data. The bue ines indicate that the Sentine-2 bands 11 and 12 are downscaed to 10 m using the 10 m Sentine-2 bands 2, 3, 4 and 8. The dashed ine indicates that whether the Landsat 8 PAN image shoud be incorporated in downscaing Landsat 8 bands 5-7 depends on its correation with them. As the spatia resoution ratio between the 15 m PAN and 10 m Sentine-2 bands is not an integer, step 3) of downscaing and step 5) of backward upscaing are introduced, which means that it is essentiay 10 m Sentine-2 information incorporated in the fusion process. The uncertainty in downscaing in step 3), which invoves direct interpoation without auxiiary data, can be eiminated by the upscaing process in step 5). In the proposed ATPRK-based fusion approach, the 10 m Sentine-2 bands are used mainy to provide vauabe information at the target fine spatia resoution, whie the 15 m PAN band is used to provide information for those changes that cannot be sufficienty characterized by 10 m Sentine-2 bands (e.g., abrupt changes). For downscaing Landsat 8 bands 5-7, whether the Landsat 8 PAN band shoud be incorporated is determined by a competey automatic soution, which makes the proposed approach more user-friendy. Based on the appeaing advantage of the perfect coherence of APTRK, the origina Landsat mutispectra

7 7 information at 30 m spatia resoution is precisey preserved. Note that when no LCLU changes occur between Sentine-2 and Landsat 8 observations, using ony Sentine-2 image as the covariate woud ead to amost the same performance as using both Sentine-2 and Landsat 8 PAN images as covariates. 3. Experiments 3.1 Data and experimenta setup In the experiments, the Landsat 8 and Sentine-2 datasets used cover a scene in Verona, Itay. The study area has a spatia extent of 18 km by 18 km, and correspondingy, the 30 m Landsat bands contain 600 by 600 pixes, the 15 m Landsat PAN band contains 1200 by 1200 pixes, and the 10 m Sentine-2 bands contain 1800 by 1800 pixes. The study area is covered mainy by a mix of vegetation and urban fabric. The Sentine-2 data were acquired on 18 August They are Leve-1C products and provide geo-coded top of atmosphere refectance with a sub-pixe mutispectra registration in the UTM/WGS84 projection [14]. As for Landsat 8 datasets (mutispectra bands 2-7 and PAN band 8), we used four observations acquired on 6 September 2015, 5 August 2015, 10 Apri 2015, and 1 November 2014, respectivey. A Landsat 8 datasets are aso in the UTM/WGS84 projection. The origina digita number was converted to top of atmosphere refectance using the radiometric rescaing coefficients and Sun ange provided in the product metadata fie [38]. The Sentine-2 and Landsat 8 images are shown in Fig. 2. As observed from the bottom right corner of the five images, obvious LCLU changes exist among a images. (a) (b) (c) (d) (e) Fig. 2 The Sentine-2 and Landsat 8 datasets used in the experiments with a spatia extent of 18 km by 18 km (bands 432 as RGB). (a) 10 m Sentine-2 image (18 August 2015). (b) 30 m Landsat 8 image (6 September

8 8 2015). (c) 30 m Landsat 8 image (5 August 2015). (d) 30 m Landsat 8 image (10 Apri 2015). (e) 30 m Landsat 8 image (1 November, 2014). The 30 m Landsat 8 bands coud be fused with the 10 m Sentine-2 bands to produce the 10 m Landsat 8 images. However, no reference at 10 m coud then be used to examine the performance of downscaing objectivey. Thus, for reiabe assessment, synthetic datasets were used in the experiments. More precisey, a avaiabe data were upscaed by a factor of three. Accordingy, the 30 m Landsat 8 bands, 15 m Landsat 8 PAN band and 10 m Sentine-2 bands were upscaed to 90 m (200 by 200 pixes), 45 m (400 by 400 pixes) and 30 m (600 by 600 pixes). The objective of Landsat 8 and Sentine-2 fusion was to reproduce the 30 m Landsat bands with the aid of the 45 m PAN and 30 m Sentine-2 bands. With this scheme, the 30 m Landsat bands are known perfecty and can be used for objective assessment. This is a scheme used commony in experimenta studies to evauate downscaing approaches [39]. The HPF [22], SFIM [23] and ATWT [21] methods were used as three benchmark methods for comparison. They fused directy the 90 m Landsat bands with 30 m Sentine-2 bands to produce 30 m Landsat resuts. Moreover, three different versions of ATPRK were aso impemented. The first one uses ony 30 m Sentine-2 data, whie the second one uses ony the 45 m PAN band (i.e., the standard pan-sharpening probem). The third one is the proposed fusion approach that uses both the Sentine-2 and Landsat PAN images. For carity, we denote the three versions as ATPRK1, ATPRK2, and ATPRK3. For ATPRK2, the 45 m image band with 400 by 400 pixes was first interpoated to 600 by 600 pixes to match the spatia size of the 30 m spatia resoution images. The interpoated PAN image was then fused with the 90 m Landsat 8 mutispectra bands to produce 30 m predictions. Six indices were used for quantitative assessment, incuding the root mean square error (RMSE), CC, universa image quaity index (UIQI) [40], reative goba-dimensiona synthesis error (ERGAS) [41], spectra ange mapper (SAM) and coherence. For RMSE, CC and UIQI, they were first cacuated for each band, and then the vaues for a bands were averaged. For SAM, vaues for spectra of a pixes were first cacuated and then averaged. Coherence (quantified by the CC) is an index measuring the reation between the observed coarse image and the coarse image obtained by upscaing the sharpened image. For each mutispectra band, a coherence vaue was cacuated and the vaues for a bands were averaged. 3.2 Downscaing Landsat 8 images in the past In this section, the 90 m Landsat data of 5 August 2015 were downscaed to 30 m. The acquisition time of the Landsat data is before that of Sentine-2 (18 August 2015), and thus, can iustrate the performance of the proposed approach for downscaing Landsat images in the past (i.e., acquired earier than Sentine-2). The resuts are shown in Fig. 3 where bands 432 were seected as the RGB composite. For convenience of visua inspection, Fig. 4 shows the resuts for a 4.5 km by 4.5 km sub-area. From the date of Landsat 8 acquisition to Sentine-2, the study area was subject to severa LCLU changes; see the faint yeow objects in the Sentine-2 image in Fig. 4(b). As shown in the resuts, the downscaing resuts are visuay more satisfactory than the origina 90 m coarse image in Fig. 4(c) and many detais can be reproduced, suggesting the benefits of the fusion methods. Evauating the resuts of the six methods, HPF and SFIM produced specke artifacts for urban pixes. Moreover, due to the LCLU changes, saturation artifacts (especiay for the arge faint yeow objects in Fig. 4(b)) and ambiguous boundaries (see, for exampe, the faint yeow objects in the first few coumns) were produced in the resuts. ATWT produced smooth resuts for urban pixes, but the saturation artifacts become more obvious. Athough ATPRK using ony Sentine-2 (ATPRK1) can aeviate some saturation artifacts (e.g., around the boundaries of green pixes) to some extent and produces more accurate resuts than HPF, SFIM and ATWT, it sti faied to dea with some changes and some inear artifacts remain. Using ony the 45 m PAN image for downscaing (ATPRK2) the resut of is over-smooth and many detais cannot be observed as ceary as those in the 30 m resuts. By referring to the reference in Fig. 4(a), the proposed method (ATPRK3) is found to produce the most accurate resut amongst a methods and remarkaby, it is abe to remove amost a

9 9 saturation artifacts produced by the LCLU changes and reproduce more accuratey the LCLU boundaries (see again the faint yeow objects in the first few coumns). (a) (b) (c) (d) (e) (f) (g) (h) (i) Fig m Downscaing resuts for the Landsat 8 image on 5 August 2015 (bands 432 as RGB). (a) 30 m Landsat 8 reference (5 August 2015). (b) 30 m Sentine-2 image (18 August 2015). (c) 90 m Landsat 8 image used as input (5 August 2015). (d) HPF resut. (e) SFIM resut. (f) ATWT resut. (g) ATPRK resut produced by fusing the 90 m Landsat image with the 30 m Sentine-2 image (ATPRK1). (h) ATPRK resut produced by fusing the 90 m Landsat image with the 45 m PAN image (ATPRK2). (i) ATPRK resut produced by fusing the 90 m Landsat image with the 30 m Sentine-2 and 45 m PAN images (ATPRK3).

10 10 (a) (b) (c) (d) (e) (f) (g) (h) (i) Fig m Downscaing resuts for a sub-area in Fig. 3 (bands 432 as RGB). (a) 30 m Landsat 8 reference (5 August 2015). (b) 30 m Sentine-2 image (18 August 2015). (c) 90 m Landsat 8 image used as input (5 August 2015). (d) HPF resut. (e) SFIM resut. (f) ATWT resut. (g) ATPRK resut produced by fusing the 90 m Landsat image with the 30 m Sentine-2 image (ATPRK1). (h) ATPRK resut produced by fusing the 90 m Landsat image with the 45 m PAN image (ATPRK2). (i) ATPRK resut produced by fusing the 90 m Landsat image with the 30 m Sentine-2 and 45 m PAN images (ATPRK3). Quantitative assessment for the entire 18 km by 18 km study area is shown in Tabe 2. As isted in the tabe, HPF, SFIM and ATWT are ess accurate than the three ATPRK-based methods. For exampe, the CC and UIQI for HPF, SFIM and ATWT are beow 0.94, but for the three ATPRK-based methods, both indices are above A three ATPRK-based methods produced perfect coherence vaues of 1. The ATPRK1 and

11 11 ATPRK3 products are more accurate than ATPRK2. This is because ATPRK2 uses ony 45 m Landsat 8 PAN information in the fusion process, but ATPRK1 and ATPRK3 use the Sentine-2 data which can provide information at the desired 30 m spatia resoution. Furthermore, ATPRK3 has a arger CC and UIQI and smaer RMSE, ERGAS and SAM than ATPRK1. More precisey, ATPRK3 reduces the overa RMSE, ERGAS and SAM by , , and , respectivey, and increases both the overa CC and overa UIQI by This demonstrates that incorporation of the Landsat 8 PAN band can enhance the performance of ATPRK in fusing Landsat 8 with Sentine-2 data. It is noteworthy that ATPRK1 and ATPRK3 have the same performances for Landsat 8 bands 5-7. The reason for this is that the three bands have a arger correation with Sentine-2 bands 8, 11 and 12 than for the Landsat 8 PAN band, and according to the principe of the proposed ATPRK3 approach, the PAN band was not considered when downscaing these three bands. Tabe 2 Quantitative assessment of the downscaing methods for the entire Landsat 8 image of 5 August 2015 (ATPRK1 uses ony the 30 m Sentine-2 data, ATPRK2 uses ony the 45 m PAN band, and ATPRK3 uses both the 30 m Sentine-2 and 45 m PAN data; the bod vaues indicate the most accurate resut in each term) Idea HPF SFIM ATWT ATPRK1 ATPRK2 ATPRK3 Band Band Band RMSE Band Band Band Mean Band Band Band CC Band Band Band Mean Band Band Band UIQI Band Band Band Mean ERGAS SAM( ) Coherence Downscaing Landsat 8 images in the future In this section, to iustrate the performance of the proposed approach for downscaing Landsat images in the future (i.e., acquired ater than Sentine-2), the 90 m Landsat 8 image of 6 September 2015 was downscaed to 30 m. Fig. 5 shows the resuts of the six methods of the same 4.5 km by 4.5 km sub-area as in Fig. 4. Compared to the Landsat image of 5 August 2015, the Landsat image of 6 September 2015 shows fewer LCLU changes reative to the Sentine-2 image. Simiary to the previous experiment, HPF and SFIM produced specke artifacts for urban pixes, ambiguous boundaries for arge size objects and eongated artifacts, which are mainy caused by the LCLU changes. ATWT and ATPRK1 mitigated the phenomenon, but the LCLU boundaries are sti ambiguous. ATPRK2 using ony the 45 m Landsat PAN image cannot provide cear LCLU information at the desired finer spatia resoution. Focusing on the resut of the proposed ATPRK3 method, the

12 12 boundaries between the LCLU casses are much cearer than those in the other five resuts, and it is the cosest to the reference in Fig. 5(a). Tabe 3 dispays the accuracies of a tested methods for the entire 18 km by 18 km study area. Simiary to visua inspection, the three ATPRK-based methods are more accurate than HPF, SFIM and ATWT. As for the inter-comparison between the three ATPRK-based methods, ATPRK2 is the east accurate. In this experiment, the Landsat 8 bands 5-7 aso have a arger correation with Sentine-2 bands 8, 11 and 12 than for the Landsat 8 PAN band and, thus, the PAN band was not used for these three bands. As a resut, ATPRK1 and ATPRK3 have the same performances for Landsat 8 bands 5-7. Using both PAN and Sentine-2 data, however, ATPRK3 increased the overa CC and UIQI by and , respectivey. The accuracies of both ATPRK1 and ATPRK3 are greater than that in Tabe 2, and the accuracy gains of ATPRK3 over ATPRK1 is smaer than that in Tabe 2. This is because fewer LCLU changes occurred between the Sentine-2 image and the Landsat image used in this experiment. This experiment reveas that the proposed approach aso works we for downscaing Landsat images in the future. (a) (b) (c) (d) (e) (f) (g) (h) (i)

13 13 Fig m Downscaing resuts for a sub-area of the Landsat 8 image on 6 September 2015 (bands 432 as RGB). (a) 30 m Landsat 8 reference (6 September 2015). (b) 30 m Sentine-2 image (18 August 2015). (c) 90 m Landsat 8 image used as input (6 September 2015). (d) HPF resut. (e) SFIM resut. (f) ATWT resut. (g) ATPRK resut produced by fusing the 90 m Landsat image with the 30 m Sentine-2 image (ATPRK1). (h) ATPRK resut produced by fusing the 90 m Landsat image with the 45 m PAN image (ATPRK2). (i) ATPRK resut produced by fusing the 90 m Landsat image with the 30 m Sentine-2 and 45 m PAN images (ATPRK3). Tabe 3 Quantitative assessment of the downscaing methods for the entire Landsat 8 image of 6 September 2015 (ATPRK1 uses ony the 30 m Sentine-2 data, ATPRK2 uses ony the 45 m PAN band, and ATPRK3 uses both the 30 m Sentine-2 and 45 m PAN data; the bod vaues indicate the most accurate resut in each term) Idea HPF SFIM ATWT ATPRK1 ATPRK2 ATPRK3 Band Band Band RMSE Band Band Band Mean Band Band Band CC Band Band Band Mean Band Band Band UIQI Band Band Band Mean ERGAS SAM( ) Coherence Downscaing historica Landsat 8 images To examine the proposed method for downscaing Landsat 8 images that were acquired ong before the Sentine-2 image, the 90 m Landsat 8 images of 10 Apri 2015 and 1 November 2014 were downscaed to 30 m. Fig. 6 shows the resuts for the same 4.5 km by 4.5 km sub-area for the Landsat 8 data of 10 Apri Consistent with the observations in the previous two experiments, HPF and SFIM produced specke artifacts and ambiguous boundaries for changed paces, whie ATPRK2 cannot provide cear LCLU information at the desired 30 m spatia resoution. The proposed ATPRK3 method can restore most of the boundaries and produced resuts cosest to the reference in Fig. 6(a). The accuracies measured by the six indices in Tabes 4 and 5 for the Landsat 8 data of 10 Apri 2015 and 1 November 2014 (entire 18 km by 18 km study area) aso ead to the same concusion as the visua inspection. More importanty, two further observations can be made from the two tabes. First, as the time interva between the Sentine-2 image and the Landsat 8 image increases, the accuracy of the proposed ATPRK3 approach decreases. More precisey, the time intervas between the Sentine-2 image

14 14 and the Landsat 8 images of 5 August 2015, 10 Apri 2015 and 1 November 2014 are 13, 125 and 285 days, respectivey. Correspondingy, the CCs of ATPRK3 are , and , respectivey. Second, different from the previous two experiments where the PAN bands were not used for Landsat 8 bands 5-7, in this experiment, the PAN band was used for some of the Landsat 8 bands 5-7 (e.g., bands 6 and 7 of the Landsat 8 data of 10 Apri 2015). The reason is that as the time interva between the Sentine-2 image and the Landsat 8 image increases, the changes between the two types of data increase. As a resut, the correation between the Sentine-2 bands 8, 11 and 12 and Landsat 8 bands 5-7 decreases to be smaer than that between the Landsat 8 PAN and Landsat 8 bands 5-7, and the Landsat 8 PAN band needs to be incorporated into the fusion process to provide more reiabe information. (a) (b) (c) (d) (e) (f) (g) (h) (i) Fig m Downscaing resuts for a sub-area of the Landsat 8 image on 10 Apri 2015 (bands 432 as RGB). (a) 30 m Landsat 8 reference (10 Apri 2015). (b) 30 m Sentine-2 image (18 August 2015). (c) 90 m Landsat 8 image used as input (10 Apri 2015). (d) HPF resut. (e) SFIM resut. (f) ATWT resut. (g) ATPRK resut produced by fusing the 90 m Landsat image with the 30 m Sentine-2 image (ATPRK1). (h) ATPRK resut

15 15 produced by fusing the 90 m Landsat image with the 45 m PAN image (ATPRK2). (i) ATPRK resut produced by fusing the 90 m Landsat image with the 30 m Sentine-2 and 45 m PAN images (ATPRK3). Tabe 4 Quantitative assessment of the downscaing methods for the entire Landsat 8 image of 10 Apri 2015 (ATPRK1 uses ony the 30 m Sentine-2 data, ATPRK2 uses ony the 45 m PAN band, and ATPRK3 uses both the 30 m Sentine-2 and 45 m PAN data; the bod vaues indicate the most accurate resut in each term) Idea HPF SFIM ATWT ATPRK1 ATPRK2 ATPRK3 Band Band Band RMSE Band Band Band Mean Band Band Band CC Band Band Band Mean Band Band Band UIQI Band Band Band Mean ERGAS SAM( ) Coherence Tabe 5 Quantitative assessment of the downscaing methods for the entire Landsat 8 image of 1 November 2014 (ATPRK1 uses ony the 30 m Sentine-2 data, ATPRK2 uses ony the 45 m PAN band, and ATPRK3 uses both the 30 m Sentine-2 and 45 m PAN data; the bod vaues indicate the most accurate resut in each term) Idea HPF SFIM ATWT ATPRK1 ATPRK2 ATPRK3 Band Band Band RMSE Band Band Band Mean Band Band Band CC Band Band Band Mean UIQI Band Band

16 16 Band Band Band Band Mean ERGAS SAM( ) Coherence Use of the Landsat 8 PAN image in downscaing Landsat 8 bands 5-7 The proposed approach determines whether the PAN band shoud be incorporated into the downscaing of Landsat 8 bands 5-7 according to the correation between them. This necessitates a comparison between the proposed method (ATPRK3) and the method that uses consistenty the PAN band for a Landsat 8 bands 5-7. Fig. 7 shows the resuts of bands 5-7 for the 4.5 km by 4.5 km sub-area of the Landsat 8 images on 5 August 2015 and 1 November It is observed ceary that when using the PAN band for a three Landsat bands, the resuts are ambiguous and some inear features cannot be restored. The resuts of the proposed approach are visuay more satisfactory. The quantitative assessment (in terms of CC) of the two schemes for a four fu Landsat 8 observations in Fig. 2(b)-(e) is dispayed in Tabe 6. Athough the proposed seective approach sometimes produced smaer CCs for severa bands, the CCs for most of the bands as we as the overa accuracies are greater. This experiment vaidates the rationae of the seective scheme for downscaing Landsat 8 bands 5-7 in the proposed fusion approach. (a) (b) (c) (d) (e) (f) Fig m downscaing resuts for a sub-area of the Landsat 8 image on 5 August 2015 and 1 November 2014 (bands 567 as RGB). (a)-(c) are resuts for the Landsat image on 5 August 2015, and (d)-(f) are resuts for the Landsat image on 1 November (a) and (d) are 30 m Landsat 8 reference images. (b) and (e) are ATPRK resuts produced by fusing the 90 m Landsat image with the 30 m Sentine-2 and 45 m PAN images for a

17 17 bands 5-7. (c) and (f) are ATPRK resuts produced by fusing the 90 m Landsat image with the 30 m Sentine-2 and 45 m PAN images using the proposed seective approach. Tabe 6 Quantitative assessment (in terms of CC) of the two schemes for downscaing the Landsat 8 bands 5-7 on 5 August 2015 and 1 November 2014 ( A means considering the PAN band for a bands 5-7, whie Seective means the proposed seective approach, that is, APTRK3; Y means the PAN band is used for the Landsat 8 band, whie N means not; the bod vaues indicate the most accurate resut in each term) 6 September August Apri November 2014 A Seective A Seective A Seective A Seective Band (N) (N) (N) (N) Band (N) (N) (Y) (N) Band (N) (N) (Y) (Y) Mean Resuts of the 10 m downscaed Landsat 8 images To revea the appicabiity of the proposed fusion approach in practice, we appied it to downscae the observed 30 m Landsat 8 images to 10 m, using the 10 m Sentine-2 and 15 m PAN images. The resuts for two Landsat images on 5 August 2015 and 6 September 2015 are shown in Fig. 8. Fig. 9 exhibits the resuts for the 4.5 km by 4.5 km sub-area, where the resuts of the method using ony the Sentine-2 information is aso provided for visua comparison. The benefits of downscaing are very cear when comparing the 10 m resuts to the origina 30 m Landsat 8 images. For exampe, the roads, textures of urban fabric and white buidings in the 10 m resuts are obviousy much cearer than those in the origina 30 m observations. Furthermore, the inter-comparison between the two downscaing approaches (i.e., ATPRK1 and ATPRK3) suggests that by using the 15 m PAN bands, the proposed ATPRK3 approach can produce more satisfactory predictions where more eongated features are present and pixes of LCLU changes are more accuratey restored. (a) (b) Fig m Downscaing resuts of the proposed method for Landsat 8 images on (a) 5 August 2015 and (b) 6 September 2015 (bands 432 as RGB). (a)

18 18 (b) (c) (d) (e) (f) (g) Fig m Downscaing resuts for a sub-area of the Landsat 8 image on 5 August 2015 and 6 September 2015 (bands 432 as RGB). (a) 10 m Sentine-2 image on 18 August (b)-(d) are 10 m resuts for the Landsat image on 5 August 2015, and (e)-(g) are 10 m resuts for the Landsat image on 6 September (b) and (e) are the origina 30 m Landsat 8 images. (c) and (f) are 10 m ATPRK resuts produced by fusing the 30 m Landsat images with the 10 m Sentine-2 images (ATPRK1). (d) and (g) are 10 m ATPRK resuts produced by fusing the 30 m Landsat images with the 10 m Sentine-2 and 15 m PAN images (ATPRK3). 4. Discussion 4.1 Contributions This paper presents a new approach for fusing Landsat 8 with Sentine-2 images to coordinate their spatia resoutions for continuous monitoring at the goba scae. The spatia resoution of the Landsat 8 images is

19 19 downscaed to 10 m to match that of the Sentine-2 images. The contributions of this paper ie in the theoretica innovation, technoogica advancement and appication potentia. Theoreticay, for each 30 m Landsat 8 band, we borrow the information from the 10 m Sentine-2 band with the same waveength and incorporate it into the downscaing process. We aso consider the LCLU changes between the Landsat 8 and Sentine-2 images to increase the accuracy of the downscaed Landsat 8 images. The objective is to take fu advantage of a the information provided by the two types of sensors. Technoogicay, the advanced ATPRK approach is proposed for the fusion task, where the 10 m Sentine-2 images are treated as covariates and provide vauabe fine spatia resoution information. Based on ATPRK, the spectra properties of the origina Landsat data can be perfecty preserved. To account for the LCLU changes, a Landsat 8 PAN image is incorporated into the downscaing process for further enhancement. Since the PAN band is aways acquired at exacty the same time as the mutispectra bands, it encompasses information on changes that cannot be observed by the Sentine-2 images. Fusion of Landsat 8 and Sentine-2 has great potentia appication vaue. First, for the Landsat 8 data acquired after the aunch date of the Sentine-2 sateite, they can be downscaed to 10 m and embedded to the avaiabe Sentine-2 time-series data to produce finer tempora resoution data at 10 m spatia resoution, and more continuous goba monitoring can be achieved to observe rapid changes on the Earth s surface. The experimenta resuts in Sections 3.1 and 3.3 where the Landsat 8 data were acquired temporay cose to the Sentine-2 data suggested that the proposed ATPRK-based fusion approach is suitabe for coordinating the spatia resoutions of the two types of data for more continuous monitoring. Timey monitoring is critica in a wide range of appications, such as the urbanization process in highy deveoped cities [4], [5] and deforestation and forest degradation processes (for exampe in the Amazon rainforest where intervention is needed quicky foowing the detection of deforestation) [6]. Apart from LCLU changes, the finer tempora resoution Sentine-2 data wi aso have great potentia in monitoring rapid changes in vegetation phenoogy, especiay in agricutura areas. Second, the historica Landsat 8 data acquired between February 2013 (the aunch time of Landsat 8) and June 2015 (the aunch time of Sentine-2) can aso be downscaed to 10 m with the proposed fusion approach. The 10 m products may provide anaysts with more expicit LCLU information in these two years than the origina 30 m Landsat data. The experimenta resuts in Section 3.4 where the two Landsat 8 datasets were acquired ong before the Sentine-2 data (with time intervas of 125 and 285 days, respectivey) reveaed that the proposed fusion approach has potentia for downscaing historica Landsat 8 data to 10 m. 4.2 Comparison with fusion of Landsat and MODIS data Fusion of Landsat and MODIS images is an existing soution to provide more frequent fine spatia resoution data for goba monitoring, which can synthesize 30 m Landsat-ike data at the tempora resoution of the avaiabe MODIS data. However, it is substantiay different from the fusion probem presented in this paper. The essence of fusing Landsat with Sentine-2 data is to downscae the 30 m Landsat data to the 10 m Sentine-2 resoution, whie fusing Landsat with MODIS data means downscaing the 500 m MODIS data to the 30 m Landsat resoution. The former invoves a zoom factor of ony three, but the atter invoves a very arge zoom factor of 16 and arge uncertainties. Moreover, since Landsat and Sentine-2 data are provided in the same geographic coordinate system (the UTM/WGS84 projection), the geometric registration is more convenient for fusion of the two types of data. In theory, by fusing Landsat with MODIS data, 30 m Landsat-ike data at the tempora resoution of daiy MODIS observations can be generated. Due to atmospheric conditions, however, coudy pixes exist commony in daiy MODIS data and eight-day composite MODIS products are sometimes preferabe choices for reiabe goba monitoring. Thus, given eight-day composite data, approximatey four Landsat-ike datasets can be produced every month, which is comparabe to the five observations per month that can be produced from the combined Landsat and Sentine-2 time-series data. It is worth noting that the addition of the compementary Sentine-2B sateite wi be aunched in mid The twin Sentine-2 sateites wi be in the same orbit and 180 apart from each other, and this increases the frequency of coverage from the current ten