Classification of Multitemporal Remote Sensing Data of Different Resolution using Conditional Random Fields

Transcription

1 Classificaion of Muliemporal Remoe Sensing Daa of Differen Resoluion using Condiional Random Fields Thorsen Hoberg, Franz Roenseiner and Chrisian Heipke Insiue of Phoogrammery and GeoInformaion Leibniz Universiä Hannover, Nienburger Sr. 1, Hannover, Germany {hoberg, roenseiner, Absrac The increasing availabiliy of muliemporal opical remoe sensing daa offers new poenials for land cover analysis. We presen a novel approach for enhancing he classificaion accuracy of medium resoluion daa by combining hem wih high resoluion daa of an earlier acquisiion ime, hus saving daa acquisiion coss. Our approach uses Condiional Random Fields o model boh spaial and emporal dependencies. Temporal conex is considered by a novel exension of he CRF concep by an addiional emporal ineracion poenial, which can model dependencies beween idenical regions in images of differen acquisiion imes and scales. The model also considers differen levels of absracion in he class srucures a differen scales. The approach is esed wih wo se-ups of Ikonos, RapidEye, and Landsa imagery. 1. Inroducion An increasing number of opical high resoluion (HR) remoe sensing saellie sysems have become available in he las decade. I should hus be possible o improve he classificaion accuracy and o analyze land cover changes a a higher frequency han his is currenly done based on a muliemporal analysis. However, he purchase of HR muliemporal daa for hese purposes is ofen no economically viable, especially for large areas. Daa having medium resoluion do no offer as much deail, bu cover a larger area and may ofen be preferable from an economical poin of view. I is hus he goal of his paper o presen a mehod capable of combining HR images wih daa of lower resoluion for increasing he classificaion accuracy and deecing land cover changes. This is achieved by a combined classificaion using medium resoluion daa (e.g. 30 m Ground Sampling Disance - GSD), available for one or more poins in ime, ogeher wih daa of high resoluion (GSD 5 m) acquired a he beginning of he observaion period. In his way i should be possible o benefi from he higher informaion conen of HR imagery while performing change deecion in daa of lower resoluion. Up o now mos approaches for muliemporal land cover analysis do no make use of emporal dependencies, bu derive resuls by some kind of difference measure beween he monoemporal classificaion resuls of differen epochs (i.e., differen acquisiion imes) [1]. If daa from all epochs are available, i would seem o be advanageous o use he original observaions, i.e. he image daa, raher han derived daa. This has for insance been done in [2], where a model of emporal dependencies based on Markov chains is applied for deecing land cover changes in Landsa images (30 m GSD). As in mos echniques for muliemporal classificaion found in he lieraure, each pixel is classified individually wihou considering spaial conex, which leads o a sal-andpepper-like appearance of he change deecion resuls. Bruzzone e al. [3] ry o overcome his problem by using a cascade of hree muliemporal classifiers, one of hem considering he k-neares neighbors of each pixel. A sysemaic saisical model of spaial conex in image classificaion is given by Markov Random Fields (MRF) [4], which have also been used for change deecion [5], [6]. In [5], he MRF framework is exended by a emporal energy erm based on a ransiion probabiliy marix in order o improve he classificaion resuls for wo consecuive images. In [6], he MRF framework is applied o deec changes in opical saellie images based on muliscale feaures. The change deecion problem is formulaed as a hypohesis es, leading o a binary map of changes beween he wo given images. The mehod works in an unsupervised way, bu i does no disinguish he changed objec classes. Using MRF, he ineracion beween neighboring image sies (pixels or segmens) is resriced o he class labels, whereas he feaures exraced from differen sies are assumed o be condiionally independen. This resricion is overcome by Condiional Random Fields (CRF), originally inroduced for classifying one-dimensional daa in [7] and laer also applied o he classificaion of erresrial images [8], [9]. CRF provide a discriminaive framework ha can also model dependencies beween feaures from differen image sies and ineracions beween he labels and he feaures. In remoe sensing CRF have been used for he classificaion of selemen areas in HR opical saellie images [10], [11], for

2 generaing a digial errain model from LiDAR daa [12], and for classifying crop ypes and oher land cover classes in Landsa daa [13]. All hese approaches use monoemporal daa. The image sies ha are o be classified are eiher he image pixels or small squares of image pixels. In conras, [14] use an irregular graph derived from mean-shif segmenaion for CRF-based building deecion from aerial images and airborne InSAR daa. Such a definiion would no seem o be appropriae for change deecion, because image segmenaion canno be expeced o yield coinciden segmen boundaries (and, hus, 1:N or N:1 relaions beween segmens) in images from differen epochs. Muliemporal classificaion based on CRF for improving he overall classificaion accuracy as well as deecing changes has only been applied in [15]. Unlike mos sae-of-he-ar mehods for muliemporal classificaion, he mehod in [15] allows for emporal informaion passing in boh direcions, using an expansion of he CRF model by emporal ineracions. However, all he inpu images are assumed o have approximaely he same geomerical resoluion. Muliscale analysis is moivaed by he fac ha he appearance of objecs in a scene is a funcion of he image resoluion and because i is capable of providing a more global view on image conen and/or image analysis algorihms [16], [17]. The simples way of considering muliple scales in classificaion is o derive he feaures a muliple scales, e.g. [9], which has been applied for change deecion in [6]. Muliscale random fields were defined in [18] as a compuaionally more racable alernaive o MRF. The image is represened by a pyramidal srucure based on an ocree whose leaves are he individual pixels. Each node of he ocree is modeled o be only dependen on is predecessor on he coarser level, bu no on is spaial neighbors, resuling in a series of Markov chains in scale. The smoohing effec of MRF is achieved by ocree nodes a coarser scales propagaing heir informaion o he leaves. A similar approach was used for image classificaion in [19], where i was also noed ha a muliscale approach requires a redefiniion of he class srucure a he coarser scales because some classes migh be exinc. In [20], such a muliscale represenaion is combined wih spaial ineracions a each scale of he represenaion, bu no resuls are presened. There have also been approaches o combine a muliscale analysis wih CRF. In [21], a muliscale CRF is buil on an image grid ha in addiion o he spaial neighborhood relaions also considers neighbors in scale based on a regular pyramid srucure. This paper also considers he fac ha differen classes are represened a differen scale levels by defining a par-based objec model: a finer resoluions, he classes o be discerned correspond o objec pars, whereas a he coarser resoluions, hey correspond o compound objecs. This mehod is applied o deec objecs such as moorbikes in erresrial images. In [22], his mehod is expanded o an irregular pyramid based on a muli-scale waershed segmenaion of he original image. The class srucure seems o be consan over scale in his model. I has o be noed ha excep for [6] mos of he muliscale mehods are based on monoemporal images. A few of hem consider he fac ha he class srucure changes wih scale [19], [21]. Nearly all of hem require he images a full resoluion o have he same scale, excepions for he monoemporal case being given in [17]. To our knowledge, here is no mehod for muliemporal classificaion and change deecion ha uses images of differen resoluions a he individual epochs and ha can handle class ransiions beween he differen scales. In his paper, a novel approach for he classificaion of muliemporal remoe sensing daa is presened ha is designed o achieve his very goal. A se of mulispecral images of differen resoluion is classified simulaneously in order o increase of he accuracy and reliabiliy of he classificaion resuls and o deec land cover changes beween he individual epochs. No exising land cover map is required. This mehod is based on an exension of he CRF concep by an addiional emporal ineracion poenial in a similar way as [15]. Using his poenial i is possible o model dependencies beween image regions a idenical posiions in he differen epochs ha may addiionally be characerized by differen scales and, hence by differen (hough relaed) class srucures. The remainder of his paper is srucured as follows. In Secion 2, he principles of CRF and he exensions for he classificaion of muliemporal and muliscalar daa are presened. Secion 3 focuses on he descripion of he feaures and he class srucure. In Secion 4, he mehod is evaluaed based on Ikonos, RapidEye, and Landsa daa. Conclusions and an oulook are given in Secion Condiional Random Fields In many classificaion algorihms he decision for a class a a cerain image sie is jus based on informaion derived a he regarded sie, where a sie migh be a pixel, a square block of pixels in a regular grid, or a segmen. In fac, he class labels and also he daa of spaially and emporally neighboring sies are ofen similar or show characerisic paerns, which can be modeled using CRF. In monoemporal classificaion, we wan o deermine he vecor of class labels x whose componen x i corresponds o he class of image sie i S for given image daa y by maximizing he poserior probabiliy P(x y) [9]: 1 P ( xy ) = i i ij ( i j ) Z exp A ( x, ) + I x,x, y y i S i S j (1) Ni In (1), N i is he spaial neighborhood of image sie i (hus, j is a spaial neighbor o i), and Z is a normalizaion

3 consan called he pariion funcion. The associaion poenial A i links he class label x i of image sie i o he daa y, whereas he erm I ij, called ineracion poenial, models he dependencies beween he labels x i and x j of neighboring sies i and j and he daa y. The model is very general in erms of he definiion of he funcional model for boh A i and I ij. For insance, [9] use generalized linear models for boh poenials Muliemporal Approach In he muliemporal case, we have M co-regisered images. In addiion o he ineracions of spaial neighbors, he emporal neighborhood is aken ino accoun. The lef par of Figure 1 shows he muliemporal graph srucure for images having he same scale. Each node is only linked o is direc emporal neighbors a is spaial posiion. The componens of he image daa vecor y are sie-wise daa vecors y i, wih i S and S being he se of sies of all images (i.e., i does no refer o a paricular spaial posiion, bu i refers o one spaial posiion in one of he images). The index indicaes he membership of image sie i o he relaed epoch T and T = {1, M}. The componens of x are he class labels of image sie i, x i, also wih epoch index T. For each image sie we wan o deermine he class x i from a se of pre-defined classes. The class srucure and hus he number of classes are dependen on. In order o model he muual dependency of he class labels a an image posiion a differen epochs, he model for P(x y) in (1) has o be expanded: 1 P( xy) = exp A(x i, y ) + IS( x i,x j, y ) + Z i S i S j Ni (2) k k k + IT ( x i,x l, y, y ) i S k E k l Li As we use he same funcional model for he poenial funcions A, IS, and IT k for all image sies (hus applying a homogeneous CRF model), he subscrips of he poenial funcions in (1) have been omied in (2). In (2), he associaion poenial A corresponds o A i in (1) for he labels and he image daa in a specific epoch and can be modeled in he same way. The second erm of (2), IS, corresponds o he ineracion poenial I ij in (1) for he labels and he image daa in a specific epoch, bu i is called spaial ineracion poenial in order o disinguish i from he hird erm, he emporal ineracion poenial IT k. In IT k, y and y k are he images observed a epochs and k, respecively. E is he se of epochs in he emporal neighborhood of he epoch o which image sie i belongs, hus k is he ime index of an epoch in emporal neighborhood of. The se of image sies a epoch k E ha are emporal neighbors of he image sie i is denoed by L k i, hus l L k i is an image sie ha is a emporal neighbor o i in epoch k. The emporal ineracion poenial models he dependency beween he class labels and he observed daa a consecuive epochs. The image sies are chosen o be individual pixels and hus are arranged in a regular grid for each image. As shown in Figure 1, he spaial neighborhood N i of a pixel i consiss of is four direc neighbors in he image grid. The definiion of he emporal neighborhood is explained in Secion 2.4. Figure 1: Lef: Muliemporal graph srucure for images having he same scale. Red node: processed primiive; orange nodes: spaial neighbors; green nodes: emporal neighbors. Righ: Graph srucure for images having differen scales Associaion poenial The associaion poenial A(x i, y ) in (2) is relaed o he probabiliy of label x i aking a value c given he image y a epoch by A(x i, y ) = log{p[x i =c f i (y )]}. The image daa are represened by sie-wise feaure vecors f i (y ) ha may depend on he enire image a epoch, e.g. by using feaures a differen scales [9]. We use a simple Gaussian model for P[x i =c f i (y )] [23]: 1 ( y ) P x = = i c fi i = n ( 2π ) de ( Σ fc ) e T 1 ( ) ( ) 1 f 2 i y Efc Σ fc f i y Efc In (3), E fc and Σ fc are he mean and co-variance marix of he feaures of class c, respecively. I is imporan o noe ha boh he definiion of he feaures and he dimension of he feaure vecors f i (y ) may vary over he images, because he definiion of appropriae and expressive feaures depends on he scale and also on he specral informaion conained in he images (Secion 3) Spaial ineracion poenial The spaial ineracion poenial IS(x i, x j, y ) in (2) is a measure for he influence of he daa y and he neighboring labels x j on he class x i of image sie i a epoch. In his poenial, he daa are represened by siewise vecors of ineracion feaures μ ij (y ) [9]: (3)

4 ( ) 2 µ ij y β exp if x i = x j R IS ( x i,x j, y ) = 2 µ ij ( y ) β 1 exp if x i x j R In (4), µ ij (y ) denoes he Euclidean norm of µ ij (y ) and β is a weighing facor for he influence of he spaial ineracion poenial in he classificaion process. We use he componen-wise differences of he feaure vecors f i (y ) for he ineracion feaures μ ij (y ), i.e. μ ij (y ) = [µ ij1, µ ijr ] T, where R is he dimension of he vecors f i (y ) ha may vary wih. Thus, denoing he m h componen of f i (y ) by f im (y ), he m h componen of μ ij (y ) is µ ijm = f im (y ) f jm (y ). Dividing he Euclidean norm by he number of feaures R in (4) guaranees an idenical influence of he spaial ineracion poenials for all scales. This is a very simple model ha penalizes local changes of he class labels if he daa are similar and also penalizes idenical class labels if he feaures are differen Temporal ineracion poenial The emporal ineracion poenial IT k (x i, x k l, y, y k ) models he dependencies beween he daa y and he labels x i and x k l of sie i a epoch and sie l of epoch k. In principle, IT k could be modeled similarly o IS by penalizing emporal change of labels unless i is indicaed by differences in he daa. However, a more sophisicaed funcional model would be required o compensae for he effecs of differen amospheric and lighing condiions, differen scales, and seasonal effecs on he vegeaion. We use a simple model for he emporal ineracion poenial ha neglecs he dependency of IT k from he daa: ()( ) ( x,x i l ) ssk k (4) γ TM k k k k k IT ( x i,x l, y, y ) =IT ( x i,x l ) = (5) k Q In (5), γ is a weighing facor. TM s()s(k) is a emporal ransiion marix similar o he ransiion probabiliy marix in [3]. The elemens of TM s()s(k) (x i, x k l ) can be seen as condiional probabiliies P(x i =c k x l =c k ) of an image sie i belonging o class c a epoch if he image sie l ha occupies he same spaial posiion as i in epoch k belongs o class c k in ha epoch. The se of epochs in he emporal neighborhood E of x i is chosen o consis of he wo epochs -1 and +1 if hey boh exis and of one epoch for he firs and he las images of he sequence. In he muliscale case an image sie i a epoch migh have more han jus one emporal neighbor l in epoch k (righ par of Figure 1). The se of emporal neighbors L k i consiss of all i image sies a epoch k ha have a spaial overlap wih i. The number of elemens in L i k is denoed by Q i k. Q i k acs as a normalizaion facor ensuring an idenical influence of he sum of all emporal ineracion poenials in any epoch, no maer how many emporal neighbors exis. Our definiion of he emporal ineracion poenial implies ha we apply a bidirecional ransfer of emporal informaion raher han a cascade approach handing informaion from one image o he nex in a sequence as in [3]. As saed in Secion 1, no only he appearance of classes, bu also he srucure of classes ha can be discerned is a funcion of scale. This is considered in our mehod by defining one se of classes C s for each group of images having a similar scale s. The se of classes C s will be used for he images of all epochs having a scale similar o s. In our model for he emporal ineracion poenials, his is considered by he superscrip in he ransiion marix TM s()s(k) in (5), where s( ) denoes he scale of he respecive epoch. There is one such marix for any configuraion of scales [s(), s(k)] of epochs and k linked by he emporal ineracion poenial IT k. For example, if here are four epochs ( 1, 2, 3, 4 ) wih s( 1 ) = s( 2 ) = s 1 and s( 3 ) = s( 4 ) = s 2, here will be hree such marices, namely TM 11, TM 12, and TM 22. Noe ha he ransposed marix can be used for message passing in he oher direcion: TM s()s(k) = [TM s(k)s() ] T. The ransiion marices connecing classes a he same scale are square, bu hey are no symmeric due o he fac ha some changes are more likely o occur in one direcion han in he oher. For insance, farmland is more likely o be changed ino selemen as ime passes han vice versa. The elemens of hese marices also have o model he fac ha change is no a very likely even. The ransiion marices beween differen scales are recangular. In his case, he fac ha he mos likely even o occur is ofen ha nohing changes may no be modeled as easily because he class label migh change simply due o he differen class definiion. I is relaively simple if here is a 1:N relaion beween he classes of he coarser scale and hose of he finer scale. In his case, he classes a he coarser scale are aggregaed classes merging N componens ha do no occur in any oher aggregaed class (e.g. he classes building, garden and urban road defined a a GSD of 1 m migh be merged o a class selemen a a GSD of 30 m). Consequenly, all he componens will only suppor one aggregaed class and all aggregaed classes will only suppor heir componens. In case of N:M relaions beween classes defined a differen scales, his is no as easily achieved, because a class defined a a fine scale migh give suppor o more han one class a he coarser scale and vice versa (e.g. if he class garden in he previous example is replaced by grass, he class grass migh no only be relaed o class selemen a a GSD of 30 m, bu also o a class pasure). Currenly, we only consider 1:N relaions.

5 2.5. Training and Inference Exac raining and inference is compuaionally inracable for CRF [9]. In [24], several mehods for parameer learning and inference are compared. In our applicaion, we only rain he parameers of he associaion poenials, i.e. he mean E fc and he co-variance marix Σ fc of he feaures of each class c in (3). They are deermined from he feaures f i (y ) in raining sies individually for each epoch and each class c. The oher model parameers are he weighing facors β and γ of he spaial and emporal ineracion poenials, respecively, and he elemens of he ransiion marices TM s()s(k) ; cf. (4) and (5). Esimaing he elemens of TM s()s(k) from raining daa would require a large amoun of muliemporal daa wih a significan number of acual changes, which is no a our disposal. Thus, he ransiion marices TM s()s(k) are defined by he user based on exper knowledge abou he likelihood of class changes. The weigh facors β and γ could be deermined from raining daa in a way similar o [9] if fully labeled raining images for all epochs were available, bu currenly hey are defined by he user. The parameer values used in our experimens (cf. Secion 4) were found empirically. For inference, we use Loopy Belief Propagaion (LBP) [25], a sandard echnique for performing probabiliy propagaion in graphs wih cycles ha has shown o give good resuls in he comparison repored in [24]. In his conex, edges linking emporal neighbors are reaed in he same way as edges linking spaial neighbors in message passing, excep ha he messages are differen. 3. Feaures and Class Srucure 3.1. Feaure Vecors The sie-wise feaure vecors f i (y ) used boh for he associaion poenial and for deriving he ineracion feaure vecors mus be defined such ha hey can help o discriminae he differen classes. The definiion of he feaures for classificaion depends on he scale and he specral configuraion of he images. As far as he specral configuraion is concerned, only color infrared images conaining a green, a red, and a near infrared band were available for our experimens. We had images a wo scales: one group had a GSD of 4-5 m, whereas he GSD of he second group was 30 m (cf. Secion 4). From he HR imagery (GSD 4-5 m), feaures are exraced a hree differen scales λ 1, λ 2 and λ 3 for each pixel. In his way, dependencies beween he image daa of neighboring sies are modeled. The scale λ 1 corresponds o he original resoluion, and he only feaures exraced a his scale are he hree color values observed a each pixel. A he scales λ 2 and λ 3, he pixels in a square of size 5 pixels and 11 pixels, respecively, cenered a he paricular pixel, are aken ino accoun o compue wo groups of feaures. The color-based feaures a scales λ 2 and λ 3 are he mean of he green, red and infrared channels as well as he variance of he red channel. The variances of he oher channels were found ou empirically no o conain significanly more informaion. The gradienbased feaures are derived from a weighed hisogram of he orienaions of he inensiy gradien. We use he mean and he variance of hese orienaions along wih he number of bins conaining values above he mean. These feaures allow a good disincion beween exured and homogeneous areas. The sie-wise feaure vecors f i (y ) for he HR daa hus consis of 17 elemens (3 for λ 1, for λ 2 and λ 3, respecively). For he medium resoluion daa (GSD 30 m), he use of gradien based feaures and he applicaion of larger scales was found no o be suiable. Here he feaure vecors f i (y ) jus consis of 3 elemens, which are he green, red and infrared values a he respecive pixel posiion. The values for each feaure in each daa se are normalized so ha hey are in he inerval [0, 10] for he raining sies. Noe ha he principles described here could be easily expanded o oher specral configuraions and / or images of oher spaial resoluions Class Srucure As saed in Secion 2.4, he class srucure depends on he scale of he images, and we assume 1:N relaions beween he classes a he coarser scale and he classes a he finer scale. In he medium resoluion (GSD 30 m) images used in our experimens, hree classes are o be disinguished, namely buil-up areas (bui), foress (for), and cropland (crp). In he HR images (GSD 4-5 m), in he buil-up areas here is a clear disincion beween residenial areas (res) and indusrial areas (ind). Thus, he class bui in he medium resoluion images corresponds o he wo classes res and ind in he HR images. 4. Experimens 4.1. Tes Daa and Se-up of he Experimens Our es area is siuaed near Herne, Germany, and covers an area of 8.6 x 8.6 km² (Figure 2). We used mulispecral Ikonos daa wih 4 m GSD acquired in 2005, RapidEye daa wih 5 m GSD acquired in 2009, and Landsa daa of 30 m GSD acquired in All images were recorded in summer. Abou 10% of he scene was used for raining, he res for esing. Ground ruh was obained by manually labeling he images on pixel level. For change deecion analysis some regions conaining land cover changes are needed. Unforunaely in our es sie here are oo few and oo small changes o be deeced wih Landsa imagery. For ha reason, we manipulaed he Landsa scene, creaing 12 new buil-up areas by copying

6 daa from anoher Landsa scene. Some of hese new builup areas are conneced o buil-up areas in he original scene, whereas ohers correspond o compleely new selemens surrounded by cropland and foress (Figure 3). Figure 2: Ikonos (2005) image of he enire es sie. Figure 3: Lef: Manipulaed Landsa image (2010). Righ. Manipulaed areas (whie). We used he class srucure defined in Secion 3.2, he HR daa corresponding o he Ikonos and RapidEye images and he medium-resoluion daa o he Landsa image. We esed our approach for wo differen daa se-ups: Se-up I consiss of he Ikonos and he manipulaed Landsa scenes, he ypical se-up described in Secion 1. In se-up II we insered a RapidEye scene beween he wo epochs of se-up I o invesigae how addiional daa of a differen sensor influences he resuls. The emporal ransiion marix TM beween Ikonos / RapidEye and Landsa used in our experimens is shown in Table 1. A similar marix was defined for he ransiion beween he wo HR images in se-up II. As learning he parameers of TM was impossible given he available daa, hey were se based on exper knowledge. The choice of hese values is dependen on he land cover srucure and he assumed changes. We assume ha i is mos likely o have no changes in any region. Neverheless each class ransiion migh happen, bu wih differen probabiliy. In our case a ransiion from fores or cropland o buil areas is more likely han vice versa. The facors β and γ used in our models for he spaial and emporal ineracion poenials (Equaions 4 and 5, respecively) were se o β = 1 and γ = 1.5. These values were found empirically. For boh se-ups, we compared our mehod (scenario CRF muli ) o a Maximum Likelihood classificaion using he Gaussian model in (3) (scenario ML) and o a muliemporal MRF-classificaion (scenario MRF) using he same graph srucure as for our CRF muli -approach, obained by replacing he exponen in (4) by 0. For hese hree scenarios, he overall classificaion accuracy and he kappa coefficiens are compared for all epochs. We addiionally applied a monoemporal CRF-based classificaion o he Landsa image (scenario CRF mono ) and compared he classificaion accuracy and he confusion marices o hose achieved for he scenarios CRF muli and ML. We also assessed he capabiliy of our mehod for deecing he acual changes in he Landsa scene. x +1 i = bui +1 x i = for x +1 i = crp x i = res x i = ind x i = for x i = crp Table 1: Temporal ransiion marix; corresponds o Ikonos/RapidEye imagery, +1 corresponds o Landsa imagery Resuls and Evaluaion Figure 4 shows he reference for he Landsa scene and some of he classificaion resuls. Table 2 shows he overall accuracy and he kappa coefficiens achieved for he individual scenes. The scenario ML performed wors, wih an overall accuracy of only 64% for he Landsa scene. Figure 4 also shows he sal-and pepper-like appearance of he resuls. Taking ino accoun spaial conex in scenario CRF mono improves he overall accuracy for he Landsa scene by 8%. The impac of he muliemporal approach is highlighed by he overall accuracy achieved in he scenario CRF muli, where over 79% could be achieved in boh se-ups. The higher informaion conen of he HR images clearly propagaes o he medium resoluion scene and yields a significan increase of accuracy of 7% here. There was hardly any difference beween scenarios MRF and CRF muli. Only in a few regions finer srucures are beer preserved by he CRFapproach. I may come as a bi of a surprise ha regarding he daa in he spaial ineracion poenial of CRF muli does no improve he resuls. This may be aribued o he raher simple model ha, for insance, weigh differences in he muliscalar feaures in he HR images (which are no likely o change very much beween neighboring pixels) in he same way as hose for he scale λ 1. Inegraing he RadidEye image in se-up II did no affec he overall accuracy eiher. The poorer ML classificaion resuls of he RapidEye scene indicae ha he model of he associaion poenial does no fi as well as for he Ikonos daa. The confusion-marices of he differen classificaion scenarios for he manipulaed Landsa scene in se-up I are shown in Table 3.

7 (1) (2) (3) (4) Figure 4: (1) Reference (Landsa); (2) Resuls of ML; (3) Resuls of CRF muli (boh Landsa) (4) Resuls of CRF muli (Ikonos). Classes: yellow: crp; green: for; dark red: bui (Landsa); ligh red: res; ligh blue: ind (Ikonos); whie: raining areas. S/E CRF muli CRF mono ML MRF I / % / % / % / 0.68 I / % / % / % / % / 0.68 II / % / % / % / 0.68 II / % / % / % / 0.68 II / % / % / % / % / 0.68 Table 2: Overall classificaion accuracy / kappa coefficiens; S/E: Se-up / epoch; se-up I: 1 : Ikonos, 2005 ; 2 : Landsa, 2010 ; se-up II. 1 : Ikonos, 2005 ; 2 : RapidEye, 2009, 3 : Landsa, The resuls for se-up II and for MRF are no shown because here is hardly any difference o hose for CRF muli of se-up I. Again here is a clear rend ha he qualiy of he resuls is improved by considering he spaial conex and even more so by he emporal ineracions. The compleeness and he correcness (producer s and user s accuracy) are improved going from ML via CRF mono o CRF muli. The only excepions are he correcness of class bui, which is marginally higher in CRF mono han in CRF muli, an effec compensaed by an increase in compleeness by 5%, and he compleeness of for, which is lower in CRF mono han in ML. The bigges improvemens are achieved in he compleeness of class crp and he correcness of class for. The class crp has a very inhomogeneous appearance, which may be he reason why emporal conex is imporan for disinguishing i from for. A problem for all scenarios is he relaively high rae of false posiive bui pixels. Figure 5 shows he classificaion resuls of he Landsa scene for he changed areas in CRF mono and CRF muli. In boh se-ups, 70 % of he changed pixels were correcly classified as buil-up areas in CRF muli (18% fores, 12% cropland). Using CRF mono, even 87% of he changes could be deeced. Whereas in general he emporal ineracion erm improves he classificaion considerably, i also oversmoohes some areas of acual change. I has o be noed, hough, ha in 10 ou of 12 changed areas (83%), he majoriy of he pixels in he respecive areas correcly indicaed a change. To improve he mehod s capabiliies for change deecion, he model for he emporal ineracions could be augmened o consider a dependency of he class ransiions from he daa. CRF muli CRF mono ML x i ref x i = bui x i = for x i = crp Comp. bui % for % crp % Corr. 77.0% 66.9% 92.0% bui % for % crp % Corr. 78.7% 49.5% 84.0% bui % for % crp % Corr. 73.3% 41.8% 83.6% Table 3: Confusion marices for he Landsa scene (pixel numbers in se-up I). Comp. / Corr.: compleeness / correcness. Classes: bui = buil area; for = fores; crp = cropland. Figure 5: Resuls of classificaion in he changed areas in he Landsa scene. Lef: CRF muli of se-up I; righ: CRF mono. 5. Conclusions We have presened a novel CRF-based approach for muliemporal and muliscalar image classificaion wih he goal of deecing changes in and improving he classificaion accuracy of medium resoluion daa by combining hem wih HR daa of an earlier acquisiion ime. Besides incorporaing spaial conex, our mehod uses a model of emporal conex by inroducing a emporal ineracion poenial in order o ake ino accoun dependencies beween regions a idenical posiions in images acquired a differen imes and scales along wih a scale-dependen definiion of he class srucure.

8 I was shown ha he overall classificaion accuracy of he medium resoluion image was improved by abou 8% by including spaial conex and by anoher 7% by considering he emporal ineracions. These resuls are quie promising, even more so because hey were achieved wih a simple se of feaures. The overall accuracy could sill be improved by using beer feaures, bu his was no he focus of his work; he impac of he spaial and emporal ineracion poenials is quie impressive. The mehod is capable o deec mos of he changes in he scene despie he smoohing effec of he emporal ineracions, hough one has o noe ha here is a loss of accuracy in comparison o monoemporal classificaion. Neverheless, we hink ha he muliemporal / muliscale graph srucure can be applied o many oher asks in image classificaion. The fac ha he CRF model did no yield beer resuls han he MRF approach indicaes ha in he fuure, he model for he spaial ineracion poenial should be improved, e.g. by a beer selecion of he ineracion feaures and by applying a generalized linear model [9], whose parameers can be learn from he raining daa. Apar from ha, furher research will concenrae on an improvemen of he model for he emporal ineracion poenial o improve he compleeness of he deeced changes. Learning of a leas some of he parameers of he emporal ineracion poenial is also of ineres. Acknowledgemen Our implemenaion of he CRF-classificaion is based on he UGM Malab oolbox by Mark Schmid: hp://people.cs.ubc.ca/~schmidm/sofware/ugm.hml. The research was funded by he German Science Foundaion (DFG) under gran HE 1822/22-1. References [1] D. Lu, P. Mausel, E. Brondizio, E. Moran. Change deecion echniques. In. J. Remoe Sens.z, 25(12): , [2] R. Q. Feiosa, G. A. O. P. Cosa, G. L. A. Moa, K. Pakzad, M. C. O. Cosa. Cascade muliemporal classificaion based on fuzzy Markov chains. ISPRS J. Phoogrammery Remoe Sens. 64(2): , [3] L. Bruzzone, R. Cossu, G. Vernazza. Deecion of landcover ransiions by combining mulidae classifiers. Paern Recogniion Leers, 25(13): , [4] G. Geman, D. Geman. Sochasic relaxaion, Gibbs disribuion and Bayesian resoraion of images. IEEE- TGARS, 6(6): , [5] F. Melgani, S. B. Serpico. A Markov Random Field approach o spaio-emporal conexual image classificaion. IEEE-TGARS, 41(11): , [6] G. Moser, E. Angiai, S. B. Serpico. A conexual muliscale unsupervised mehod for change deecion wih muliemporal remoe-sensing images. Proc. 9 h Conf. Inelligen Sysems Design & Applicaions: , [7] J. Laffery, A. McCallum, F. Pereira. Condiional Random Fields: Probabilisic models for segmening and labeling sequence daa. Proc. In. Conf. Machine Learning: 8p, [8] S. Kumar, M. Heber. Discriminaive Random Fields: A discriminaive framework for conexual ineracion in classificaion. Proc. In. l Conf. Compuer Vision, 2: , [9] S. Kumar, M. Heber. Discriminaive Random Fields. In l. J. Compuer Vision, 68(2): , [10] P. Zhong, R. Wang. A muliple condiional random fields ensemble model for urban area deecion in remoe sensing opical images. IEEE-TGARS, 45(12): , [11] T. Hoberg, F. Roenseiner. Classificaion of selemen areas in remoe sensing imagery using Condiional Random Fields. In l. Arch. Phoogrammery, Remoe Sens., SIS XXXVIII (7), [12] W.-L. Lu, K. P. Murphy, J. J. Lile, A. Sheffer, H. Fu. A hybrid condiional random field for esimaing he underlying ground surface from airborne Lidar daa. IEEE- TGARS, 47(8/2): , [13] R. Roscher, B. Waske, W. Försner. Kernel discriminaive random fields for land cover classificaion. 6 h IAPR TC 7 Workshop Paern Recogniion in Remoe Sens.: 5p., [14] J.D. Wegner, U. Sörgel, B. Rosenhahn. Segmen-based building deecion wih Condiional Random Fields, Proc. 6 h Join Urban Remoe Sensing Even: , [15] T. Hoberg, F. Roenseiner, C. Heipke. Classificaion of Muliemporal Remoe Sensing Daa Using Condiional Random Fields. 6 h IAPR TC 7 Workshop on Paern Recogniion in Remoe Sensing: 4p., [16] Z. Kao, M. Berhod, J. Zerubia. Muliscale Markov random field models for parallel image classificaion. Proc. Fourh In. Conference on Compuer Vision: , [17] A. S. Wilsky. Muliresoluion Markov models for signal and image processing. Proc. IEEE, 90(8): , [18] C. A. Bouman, M. Shapiro. A muliscale random field for Bayesian image segmenaion. IEEE-TIP, 3(2): , [19] J. Kersen, M. Gähler, S. Voig. A general framework for fas and ineracive classificaion of opical VHR saellie imagery using hierarchical and planar Markov Random Fields. PFG 6(2010): , [20] M. J. Choi, V. Chandrasekaran, D. M. Maliouov, J. K. Johnson, A. S. Willsky. Muliscale sochasic modeling for racable inference and daa assimilaion. Compu. Mehods Appl. Mech. Engrg. 197 (2008): , [21] P. Schnizspan, M. Friz, B. Schiele. Hierarchical suppor vecor random fields: join raining o combine local and global feaures. Proc. ECCV II: , [22] M. Y. Yang, W. Försner, M. Drauschke. Hierarchical condiional random field for muli-class image classificaion. Proc. In l. Conf. Compuer Vision Theory and Applicaions (VISAPP): , [23] C. M. Bishop. Paern recogniion and machine learning. 1 s ediion, Springer New York, [24] S. Vishwanahan, N. N. Schraudolph, M. W. Schmid, K. P. Murphy. Acceleraed raining of condiional random fields wih sochasic gradien mehods. 23 rd In. Conf. on Machine Learning: , [25] J. Nocedal, S. J. Wrigh. Numerical Opimizaion. 2 nd ediion, Springer New York, 2006.