Detection of Compound Structures in Very High Spatial Resolution Images

Detection of Compound Structures in Very High Spatial Resolution Images Selim Aksoy Department of Computer Engineering Bilkent University Bilkent, 06800, Ankara, Turkey saksoy@cs.bilkent.edu.tr Joint work with Gö khan Akç ay, Ç ağ lar Arı, Daniya Zamalieva October 24, 2012 October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 1 / 55

Introduction.., A common approach for object recognition is to segment the images into homogeneous regions. October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 2 / 55

Introduction.., A common approach for object recognition is to segment the images into homogeneous regions..., However, such homogeneous regions often correspond to very small details in very high spatial resolution (VHR) images. October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 2 / 55

Introduction.., A common approach for object recognition is to segment the images into homogeneous regions..., However, such homogeneous regions often correspond to very small details in very high spatial resolution (VHR) images..., An alternative is to model the spatial arrangements of simple image regions to identify complex region groups. October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 2 / 55

Introduction.., A common approach for object recognition is to segment the images into homogeneous regions..., However, such homogeneous regions often correspond to very small details in very high spatial resolution (VHR) images..., An alternative is to model the spatial arrangements of simple image regions to identify complex region groups..., Examples of such region groups, also called compound structures, include different types of residential, commercial, industrial, and agricultural areas. October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 2 / 55

Introduction Figure 1: An Ikonos image of Baltimore, and some compound structures of interest: residential, commercial, park, marina, housing project, etc. October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 3 / 55

Introduction Figure 2: Compound structures in WorldView-2 images of Ankara and Kusadasi. October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 4 / 55

Introduction.., Compound structures are comprised of different spatial arrangements of primitive objects such as buildings, roads, and trees. October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 5 / 55

Introduction.., Compound structures are comprised of different spatial arrangements of primitive objects such as buildings, roads, and trees..., Our framework involves statistical modeling of the features of primitive objects and structural modeling of their arrangements using graphs. October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 5 / 55

Introduction.., Compound structures are comprised of different spatial arrangements of primitive objects such as buildings, roads, and trees..., Our framework involves statistical modeling of the features of primitive objects and structural modeling of their arrangements using graphs..., We have developed methods that use October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 5 / 55

Introduction.., Compound structures are comprised of different spatial arrangements of primitive objects such as buildings, roads, and trees..., Our framework involves statistical modeling of the features of primitive objects and structural modeling of their arrangements using graphs..., We have developed methods that use.., graph-based knowledge discovery, October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 5 / 55

Introduction.., Compound structures are comprised of different spatial arrangements of primitive objects such as buildings, roads, and trees..., Our framework involves statistical modeling of the features of primitive objects and structural modeling of their arrangements using graphs..., We have developed methods that use.., graph-based knowledge discovery,.., region co-occurrences, October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 5 / 55

Introduction.., Compound structures are comprised of different spatial arrangements of primitive objects such as buildings, roads, and trees..., Our framework involves statistical modeling of the features of primitive objects and structural modeling of their arrangements using graphs..., We have developed methods that use.., graph-based knowledge discovery,.., region co-occurrences,.., hierarchical clustering, October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 5 / 55

Introduction.., Compound structures are comprised of different spatial arrangements of primitive objects such as buildings, roads, and trees..., Our framework involves statistical modeling of the features of primitive objects and structural modeling of their arrangements using graphs..., We have developed methods that use.., graph-based knowledge discovery,.., region co-occurrences,.., hierarchical clustering,.., robust constrained clustering. October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 5 / 55

Detection using graph-based knowledge discovery.., Compound structures can be defined in terms of frequent occurrences of primitive region types in particular spatial arrangements..., Given a segmentation, features of neighboring region pairs are incorporated in a spatial co-occurrence space..., Density estimation in the co-occurrence space identifies groups of related region pairs. D. Zamalieva, S. Aksoy, and J. C. Tilton. Finding compound structures in images using image segmentation and graph-based knowledge discovery. In Proceedings of IEEE International Geoscience and Remote Sensing Symposium, volume V, pages 252 255, Cape Town, South Africa, July 13 17, 2009. October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 6 / 55

Detection using graph-based knowledge discovery Figure 3: A 2D illustration of the spatial co-occurrence space. October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 7 / 55

Detection using graph-based knowledge discovery.., A graph is used to encode the spatial structure where there is a vertex for each primitive region and the edges connect the vertex pairs that correspond to the modes of the density estimate..., Finally, a frequent subgraph discovery algorithm produces parts of high-level compound structures. October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 8 / 55

Detection using graph-based knowledge discovery Figure 4: Example substructures obtained by graph analysis, and the corresponding region groups in a multispectral Ikonos image of Antalya. October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 9 / 55

Detection using graph-based knowledge discovery Figure 5: Example substructures obtained by graph analysis, and the corresponding region groups in a multispectral Ikonos image of Antalya. October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 10 / 55

Detection using graph-based knowledge discovery Figure 6: Example segmentation obtained by clustering the subgraph histograms within sliding image windows. October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 11 / 55

Detection using region co-occurrences.., An alternative approach is to build a segmentation hierarchy by iteratively merging regions that appear together frequently..., Dense regions in the spatial co-occurrence space are assumed to correspond to significant relations..., After merging the significant co-occurrences, we obtain the next level in the hierarchy..., Consequently, the resulting regions represent complex structures in the image. H. G. Akcay, S. Aksoy, and P. Soille. Hierarchical segmentation of complex structures. In Proceedings of 20th IAPR International Conference on Pattern Recognition, pages 1120 1123, Istanbul, Turkey, August 23 26, 2010. October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 12 / 55

Detection using region co-occurrences Figure 7: Flowchart for hierarchical segmentation of region co-occurrences. October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 13 / 55

Detection using region co-occurrences October 24, 2012 Figure 8: An example hierarchy. @2012, Sel1m Aksoy (B1Ikent Umvers1ty) 14 /55 ' \ f I

Detection using region co-occurrences Figure 9: Different hierarchy levels for a pan-sharpened QuickBird image of Germany. October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 15 / 55

Detection using hierarchical clustering.., Our next model uses attributed relational graphs where the primitive objects form the vertices..., We connect every neighboring vertex pair with an edge..., We use a threshold on the distance between the centroids of object pairs to determine the neighbors. H. G. Akcay and S. Aksoy. Detection of compound structures using hierarchical clustering of statistical and structural features. In Proceedings of IEEE International Geoscience and Remote Sensing Symposium, pages 2385 2388, Vancouver, Canada, July 25 29, 2011. H. G. Akcay and S. Aksoy. Detection of compound structures using multiple hierarchical segmentations. In Proceedings of IEEE International Geoscience and Remote Sensing Symposium, Munich, Germany, July 23 27, 2012. October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 16 / 55

Detection using hierarchical clustering Figure 14: Examples of building detection on a multispectral WorldView-2 image of Ankara. October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 17 / 55

Detection using hierarchical clustering Figure 15: Examples of building detection on another multispectral WorldView-2 image of Ankara. October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 18 / 55

Detection using hierarchical clustering (a) Building mask (b) Neighborhood graph Figure 16: Examples of graph construction. The vertices considered as neighbors based on proximity analysis are connected with red edges in (b). October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 19 / 55

Detection using hierarchical clustering (a) Building mask (b) Neighborhood graph Figure 17: Examples of graph construction. The vertices considered as neighbors based on proximity analysis are connected with red edges in (b). October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 20 / 55

Detection using hierarchical clustering.., The statistical features that summarize the spectral content and the shape of each individual object consist of.., spectral means,.., area,.., eccentricity, and.., centroid location. October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 21 / 55

Detection using hierarchical clustering.., The structural features represent the spatial layout of each object with respect to its neighbors..., An important structural information is the amount of alignment among objects..., Aligned groups of objects are found by checking all possible subsets having at least three objects. Figure 18: Illustration of object alignment. October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 22 / 55

Detection using hierarchical clustering.., The candidate subsets are generated using a depth-first search on the neighborhood graph..., A group of three or more objects are accepted as aligned if their centroids lie on a straight line and the spacing among the objects is uniform..., The set of structural features computed for each object group consists of.., orientation of the fitted line, and.., mean of the centroid distances. October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 23 / 55

Detection using hierarchical clustering (a) Building mask (b) Alignment detection Figure 19: Examples of alignment detection in an image with 418 buildings. All groups of buildings satisfying the alignment criteria are shown in (b). October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 24 / 55

Detection using hierarchical clustering (a) Building mask (b) Alignment detection Figure 21: Examples of alignment detection. All groups of buildings satisfying the alignment criteria are shown in (b). October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 25 / 55

Detection using hierarchical clustering.., After each vertex is assigned statistical and structural features, the next step is to group these objects via clustering..., Once the statistical and structural distances are computed for each neighboring object pair, agglomerative hierarchical clustering iteratively groups these objects. October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 26 / 55

Detection using hierarchical clustering (a) Ankara image (b) Combined clustering Figure 24: Example clustering results on a multispectral WorldView-2 image of Ankara. Different groups are shown in different colors. October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 27 / 55

Detection using hierarchical clustering.., Each group is modeled using a Markov random field..., The cliques consist of individual regions and neighboring region pairs..., For each region r i, we compute.., a i : area,.., e i : eccentricity, and.., o i : orientation..., For each region r i and each neighbor r j, we compute.., d ij : proximity, and.., α ij : relative orientation. October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 28 / 55

Detection using hierarchical clustering.., We define a region process R as a set of regions R = (r i, i = 1,..., n) that follows a Gibbs distribution where 1 p(r β) = Z exp( < β, H (R) >).., Z is the partition function,.., H contains five marginal histograms as high-order statistics: H (R) = (H (d), H (α), H (a), H (e), H (o)), and.., β is the parameter set controlling each histogram bin. October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 29 / 55

Detection using hierarchical clustering.., Given a query region process, we learn the parameter set β using maximum likelihood estimation..., We extract the marginal histograms H (R) for each region process R in the data set..., We compute p(r β ) for each region process R..., Retrieval is achieved by ranking the region processes according to their probabilities. October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 30 / 55

Figure 25: Example retrieval results on a WorldView-2 image of Ankara. The query window is shown as red and the top 100 results are shown as blue. October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 31 / 55

Figure 26: The query window and the top 15 results in the WorldView-2 Ankara image. October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 32 / 55

Figure 27: Example retrieval results on a WorldView-2 image of Ankara. The query window is shown as red and the top 100 results are shown as blue. October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 33 / 55

Figure 28: The query window and the top 15 results in the WorldView-2 Ankara image. October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 34 / 55

Figure 33: Example retrieval results on a WorldView-2 image of Ankara. The query window is shown as red and the top 100 results are shown as blue. October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 35 / 55

Figure 34: The query window and the top 15 results in the WorldView-2 Ankara image. October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 36 / 55

Detection using robust constrained clustering.., Our final approach is based on clustering of image data using a robust Gaussian mixture model (GMM) that can group pixels that satisfy given spectral and spatial constraints..., We assume that the spectral and shape content of the primitive objects can be modeled using Gaussians. C. Ari and S. Aksoy. Detection of compound structures using a Gaussian mixture model with spectral and spatial constraints. In Proceedings of SPIE Defense, Security, and Sensing: Algorithms and Technologies for Multispectral, Hyperspectral, and Ultraspectral Imagery XVIII, Baltimore, Maryland, April 23 27, 2012. October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 37 / 55

Detection using robust constrained clustering.., Each pixel is represented using a feature vector that encodes both spectral and spatial information. x = x ms\ : multispectral data : coordinates x xy.., A group of pixels corresponding to a particular object is modeled using a Gaussian. Σ = µ ms\ : object color µ = µ xy : object position \ Σ ms 0 : homogeneity of the color content 0 Σ xy : shape of the object October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 38 / 55

Detection using robust constrained clustering.., The primitive objects can form different compound structures according to different spatial layouts..., Our layout model uses displacement vectors between object pairs. (a) RGB image (b) Spectral model (c) Spatial model Figure 35: An example model for six buildings in a grid formation. October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 39 / 55

Detection using robust constrained clustering.., We assume that we are given an example compound structure of interest with K primitive objects..., The total of Ñ pixels belonging to the given reference structure are fit a GMM with K components..., Given a target image with N pixels, the goal is to identify the pixels that are the most similar to the reference structure..., This is achieved by clustering the pixels of the target image using a robust constrained GMM model with K components. October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 40 / 55

Detection using robust constrained clustering Σ xy xy Σ1 Σ 2 xy xy ms ms µ 1 µ 2 2 Σ 1 µ ms ms 1 µ 2 µ ms 3 Σ ms 3 µ xy 3 Σ xy 3 (a) Spectral model (b) Spatial model Figure 36: An example GMM with K = 3 components. October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 41 / 55

Detection using robust constrained clustering ms ms T ms (µ 1 ms ms ms ms i µ P (i) ) (Σ P (i) ) (µ i µ P (i) ) β Σ i = Σ P (i) Σ ms 2 µ ms 2 Σ ms µ ms 1 ms 1 ms µ µ 1 2 ms ms Σ 2 Σ 1 µ ms 3 Σ ms 3 ms µ 3 ms Σ 3 (a) Reference spectral model (b) Mean constraints (c) Covariance constraints Figure 37: Spectral constraints. P (i) = j if the j th component of the reference GMM corresponds to the i th component of the target GMM. October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 42 / 55

Detection using robust constrained clustering µ xy xy xy xy λ min (Σ P (i) + d ij = µ P (j) i ) = λ min (Σ P (i) ) µ xy xy xy xy i + d ij µ = t j ij, t ij u λ max (Σ i ) = λ max (Σ P (i) ) xy xy Σ xy Σ xy µ xy xy Σ 1 Σ 2 1 2 1 µ 2 xy xy d 12 µ 1 µ 2 t 12 t 13 d 13 d 23 xy xy µ 3 µ 3 xy t xy Σ 23 Σ 3 3 (a) Reference spatial model (b) Mean constraints (c) Covariance constraints Figure 38: Spatial constraints. October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 43 / 55

Detection using robust constrained clustering.., We associate each pixel in the target image with an indicator variable z j, j = 1,..., N, where z j = 1 if a pixel is determined as an inlier and z j = 0 if it is not..., Then, the robust GMM model is estimated by maximizing the trimmed log-likelihood function N j=1 z j log K \ α k p k (x j θ k ) k=1 with the additional constraint LN z j = Ñ..., The parameters that do not satisfy the desired constraints after being updated are projected onto feasible sets. j=1 October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 44 / 55

Detection using robust constrained clustering 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 (a) RGB image (b) Reference structure (c) Unconstrained GMM result Figure 39: Detection of an example structure composed of four buildings with red roofs in a diamond formation in a multispectral WorldView-2 image of Ankara. 0 October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 45 / 55

Detection using robust constrained clustering 7 x 10 3 2.5 2 1.5 1 0.5 (a) RGB image (b) Reference structure (c) Constrained GMM result Figure 40: Detection of an example structure composed of four buildings with red roofs in a diamond formation in a multispectral WorldView-2 image of Ankara. 0 October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 46 / 55

Detection using robust constrained clustering Figure 41: The top eight structures that corresponded to the highest likelihood values at the end of all EM runs. October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 47 / 55

Detection using robust constrained clustering 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 (a) RGB image (b) Reference structure (c) Unconstrained GMM result Figure 42: Detection of an example structure corresponding to an intersection of four road segments in a multispectral WorldView-2 image of Ankara. October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 48 / 55

Detection using robust constrained clustering 7 x 10 6 5 4 3 2 1 (a) RGB image (b) Reference structure (c) Constrained GMM result Figure 43: Detection of an example structure corresponding to an intersection of four road segments in a multispectral WorldView-2 image of Ankara. 0 October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 49 / 55

Detection using robust constrained clustering Figure 44: The top six structures that corresponded to the highest likelihood values at the end of all EM runs. October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 50 / 55

Detection using robust constrained clustering 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 (a) RGB image (b) Reference structure (c) Unconstrained GMM result Figure 45: Detection of an example structure composed of four buildings and a pool in a multispectral WorldView-2 image of Ankara. October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 51 / 55

Detection using robust constrained clustering 1.2 1 0.8 0.6 0.4 0.2 (a) RGB image (b) Reference structure (c) Unconstrained GMM result Figure 46: Detection of an example structure composed of four buildings and a pool in a multispectral WorldView-2 image of Ankara. 0 October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 52 / 55

Detection using robust constrained clustering 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 (a) RGB image (b) Reference structure (c) Constrained GMM result Figure 47: Detection of an example structure composed of four buildings and a pool in another multispectral WorldView-2 image of Ankara. October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 53 / 55

Detection using robust constrained clustering 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 (a) RGB image (b) Reference structure (c) Constrained GMM result Figure 48: Detection of an example structure composed of four buildings and a pool in another multispectral WorldView-2 image of Ankara. 0 October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 54 / 55

Summary.., Groups of objects with different characteristics and spatial layouts that cannot be obtained by traditional segmentation methods can be successfully extracted using the proposed methods..., Exploiting spectral information, shape information, and spatial arrangements together is a promising research direction that enables new abstractions of remote sensing image content..., Future work includes additional experiments for different types of compound structures. October 24, 2012 Qc 2012, Selim Aksoy (Bilkent University) 55 / 55