Potential for automatic near realtime preliminary classification of Sentinel-2 (and Sentinel-3) imagery using the Satellite Image Automatic Mapper (SIAM ) Andrea Baraldi, Luigi Boschetti and Chris Justice University of Maryland, Dept. of Geographical Sciences, College Park, MD 20740, USA 1 Baraldi, Boschetti, and Justice: SIAM Automatic Preliminary Classification of Sentinel-2 and Sentinel-3 Optical Imagery 4/27/2012
GEO-CEOS QA4EO guidelines Group on Earth Observations (GEO)- Global Earth Observation System of Systems (GEOSS), ten-year implementation plan, 2005-2015. GEOSS objective: to allow the provision of and the access to the Right Information, in the Right Format, at the Right Time, to the Right People, to Make the Right Decisions. GEOSS two key principles: Accessibility/Availability and Suitability/Reliability of RS data and derived products. GEO-Committee of Earth Observations (CEOS) Quality Assurance framework for Earth Observation (QA4EO). QA4EO guidelines 1) An appropriate community-agreed coordinated programme of Cal/Val activities is required throughout all stages of a spaceborne mission, from sensor build to end-of-life. 2) Cal/Val guarantees harmonization and interoperability of EO data and derived information products generated from a variety of sources at all scales - global, regional and local. 3) Cal/Val is critical to data Quality Assurance (QA) and, therefore, data usability. Baraldi, Boschetti, and Justice: SIAM Automatic Preliminary Classification of Sentinel-2 and Sentinel-3 Optical Imagery
Problem Recognition Only 10% (or less) of usable spaceborne image data are ever downloaded by users. Possible cause: Low Accessibility/Availability. No existing commercial RS Image Understanding System (RS-IUS), but one, requires RS image radiometric calibration in compliance with the QA4EO guidelines. Effect: Low Suitability/Reliability. ecognition Server by Definiens AG Commercial RS-IUSs Radiometric calibration requirement NO RAD. CAL. statistical model, semi-automatic and site-specific. PCI Geomatics GeomaticaX NO RAD. CAL. statistical model, semi-automatic and site-specific. Pixel- and Segment-based versions of the Environment for Visualizing Images (ENVI) by ITT VIS ERDAS IMAGING Objective ERDAS ATCOR3 Proposed three-stage stratified hierarchical RS-IUS architecture: (i) rad. Cal. of DNs into top-of-atmosphere (TOA) reflectance (TOARF) or SURF, with TOARF SURF, (ii) SIAM preliminary classification and (iii) stratified battery of context-sensitive classifiers. NO RAD. CAL. statistical model, semi-automatic and site-specific. NO RAD. CAL. statistical model, semi-automatic and site-specific. Surface reflectance (SURF) physical model, inherently ill-posed atmospheric correction first stage. Consistent with QA4EO, but semiautomatic and site-specific. RAD. CAL. Of DNs into TOARF or SURF values, with TOARF SURF atmospheric correction is optional. Consistent with QA4EO. Fully automated (no user-defined parameter, no training sample). Baraldi, Boschetti, and Justice: SIAM Automatic Preliminary Classification of Sentinel-2 and Sentinel-3 Optical Imagery
Problem Recognition Existing commercial or scientific RS-IUSs score low in terms of operational quantifiable metrological/ statistical Quality Indicators (QIs) required by the QA4EO guidelines. Effect: Low Suitability/Reliability. Quality Indicators (QIs) Existing RS-IUS software products SIAM software product Degree of automation: (a) number, physical meaning and range of variation of user-defined parameters, (b)collection of the required training data set, if any. VL, L VH (fully automatic, it cannot be surpassed) Effectiveness : (a) semantic accuracy and (b) spatial accuracy. M, H, VH VH Semantic information level Land cover class (e.g., deciduous forest) Spectral semi-concept (e.g., vegetation) Efficiency: (a) computation time and (b) memory occupation. VL, L in training (hours per images) VH (5 m to 30 s per Landsat image in a laptop) Robustness to changes in input image VL (specific training per image) VH Robustness to changes in input parameters VL VH (it cannot be surpassed) Scalability to changes in the sensor s specifications or user s needs. Timeliness (from data acquisition to high-level product generation, increases with manpower and computing power). Economy (inverse of costs increasing with manpower and computing power). VL VH (e.g., the collection of reference samples is a difficult and expensive task) VL, L, high costs in manpower and also computing power VH (it works with any existing spaceborne sensor) VL, i.e., timeliness is reduced to almost zero VH, i.e., costs in manpower and computing power are reduced to almost zero Legend of fuzzy sets. Very low (VL), Low (L), Medium (M), High (H), Very High (VH). Legend of color highlights. Red = L, VL; Blue = M; Green = H, VH. Baraldi, Boschetti, and Justice: SIAM Automatic Preliminary Classification of Sentinel-2 and Sentinel-3 Optical Imagery
1 st key principle: Improve Accessibility to /Availability of existing large-scale multi-sensor multi-resolution spaceborne image databases Cost-free access to RS data (e.g., NASA-USGS access policy for Landsat data). Semantic querying systems. 2 nd key principle: Improve Suitability/Reliability of RS data and derived products Improve operational QIs of existing RS-IUSs. Levels of understanding of an RS-IUS eligible for improvement: 1. Computational theory (system architecture). 2. Knowledge/information representation. 3. Algorithms design (partitioning into highly cohesive modules mutually uncorrelated). 4. Implementation. Opportunity Recognition Baraldi, Boschetti, and Justice: SIAM Automatic Preliminary Classification of Sentinel-2 and Sentinel-3 Optical Imagery
SIAM computational theory SIAM is a top-down (deductive, prior knowledge-based) decision-tree classifier. Static, i.e., non-adaptive to input data. Context-insensitive, i.e., pixel-based. Based on spectral prior knowledge = reference dictionary of spectral signatures in TOARF values = spectral end-members = spectral categories. There are six super-categories that are mutually exclusive and totally exhaustive. 1. Water or shadow. 2. Snow or ice. 3. Clouds. 4. Vegetation. 5. Bare soil or built-up. 6. Outliers. Blue Light blue / white White Green Brown Rest of the world 6 Baraldi, Boschetti, and Justice: SIAM Automatic Preliminary Classification of Sentinel-2 and Sentinel-3 Optical Imagery 4/27/2012
SIAM prior knowledge base Expose bare rock 150 125 100 75 50 25 0 ETM1 ETM2 ETM3 ETM4 ETM5 ETM7 Discrete and finite dictionary of symbolic families of spectral signatures (spectral categories ) Water/ Shadow Snow Cloud Rnglnd Thick cloud (cumulus) 250 200 150 100 50 0 ETM1 ETM2 ETM3 ETM4 ETM5 ETM7 Continuous sub-symbolic color space Color-space quantization (mapping) into 3 discrete categorical variables Blue like WATER 1 SIAM Green like Vgt2 Green like Vgt1 7 Baraldi, Boschetti, and Justice: SIAM Automatic Preliminary Classification of Sentinel-2 and Sentinel-3 Optical Imagery 4/27/2012
SIAM functional characteristics SIAM is operational and fully automatic, i.e., it requires neither user-defined parameters nor training supervised (labeled) data to run. SIAM requires input MS images to be radiometrically calibrated into TOARF or SURF values (ala QA4EO). SIAM is multi-sensor and spatial resolution independent SIAM is near real-time (< 5 min per RS image in a laptop computer). SIAM is suitable for: RS optical image pre-processing (enhancement). For example: Stratified image co-registration, stratified image topographic correction, etc. Thematic mapping applications. Semantic querying of a large multi-sensor image database. 8 Baraldi, Boschetti, and Justice: SIAM Automatic Preliminary Classification of Sentinel-2 and Sentinel-3 Optical Imagery 4/27/2012
SIAM multi-sensor capabilities SIAM works at the spatial resolution of the sensor ranging from 0.5 m (pan-sharpened WorldView-2) to 3 km (Meteosat SEVIRI), i.e., it is spatial resolution-independent. SIAM can be input with any radiometrically calibrated RS image whose spectral resolution overlaps with Landsat s at least in part. The number of output spectral categories depends on the number and spectral location of the bands of the sensor. Six integrated SIAM sub-systems exist for 6 classes of sensors: 7 bands (Vis, NIR, MIR, TIR): Landsat-5 TM, Landsat-7 ETM+, ASTER, MODIS. 4 bands (VIS, NIR, MIR): SPOT-4 HRVIR, SPOT-5 HRG, IRS-1C/-1D/-P6. 4 bands (Vis, NIR, MIR, TIR): NOAA AVHRR, Meteosat Second Generation. 5 bands (Vis, NIR, MIR, TIR): ENVISAT AATSR. 4 bands (Vis, NIR): IKONOS-2, QuickBird-2, OrbView-3, GeoEye-1, WorldView2, RapidEye, ALOS AVNIR-2, ENVISAT MERIS. 3 bands (Vis, NIR): Landsat-1/-2/-3/-4 MSS, DMC, SPOT-1/-2/-3 HRV. 9 Baraldi, Boschetti, and Justice: SIAM Automatic Preliminary Classification of Sentinel-2 and Sentinel-3 Optical Imagery 4/27/2012
SIAM multi-sensor Sentinel-2/-3 capabilities Six integrated SIAM sub-systems exist for 6 classes of sensors: Landsat-like SIAM (L-SIAM ). 7 bands (Vis, NIR, MIR, TIR): Landsat-5 TM, Landsat-7 ETM+, ASTER, MODIS. SENTINEL-2 (without band TIR) SENTINEL-3 SLSTR (without band VisBlue) SPOT-like SIAM (S-SIAM ). 4 bands (VIS, NIR, MIR): SPOT-4 HRVIR, SPOT-5 HRG, IRS-1C/-1D/-P6. AVHRR-like SIAM (AV-SIAM ). 4 bands (Vis, NIR, MIR, TIR): NOAA AVHRR, Meteosat Second Generation. AATSR-like SIAM (AA-SIAM ). 5 bands (Vis, NIR, MIR, TIR): ENVISAT AATSR. QuickBird-like SIAM (Q-SIAM ). 4 bands (Vis, NIR): IKONOS-2, QuickBird- 2, OrbView-3, GeoEye-1, WorldView-2, RapidEye, ALOS AVNIR-2, ENVISAT MERIS. SENTINEL-3 OLCI DMC-like SIAM (D-SIAM ). 3 bands (Vis, NIR): Landsat-1/-2/-3/-4 MSS, DMC, SPOT-1/-2/-3 HRV. 10 Baraldi, Boschetti, and Justice: SIAM Automatic Preliminary Classification of Sentinel-2 and Sentinel-3 Optical Imagery 4/27/2012
SIAM multi-sensor Sentinel-2/-3 capabilities Landsat-4/-5 TM and Landsat-7 ETM+ Band Spectral region ( m) ENVISAT AATSR, ERS-2 ATSR-2, SENTINEL-3 SLSTR (bands S1 to S9) Band Spectral region ( m) ENVISAT MERIS, SENTINEL-3 OLCI (bands O1 to O23) SENTINEL-2 11 Baraldi, Boschetti, and Justice: SIAM Automatic Preliminary Classification of Sentinel-2 and Sentinel-3 Optical Imagery 4/27/2012 Band 1 (B) 0.45-0.52 - - (3+4), (O4+O5) Spectral region (nm) (490 10/2 + 510 10/2) Band Spectral region (nm) 2 (10 m) 490±65/2 2 (G) 0.52-0.60 1, S1 0.545-0.565 5, O6 560 10/2 3 (10 m) 560±35/2 3 (R) 0.63-0.69 2, S2 0.649-0.669 (7+8), (665 10/2 + 4 (10 m) 665±30/2 (O8+O9) 681.25 7.5/2) 4 (NIR) 0.76-0.90 3, S3 0.855-0.875 (11+ 12+ 13+ 14+ 15), 7 (20 m) + 8a (20 m) 783±20/2 + 865±20/2 (O12+ O13 (new)+ O23(new) + O14/15 + O16/17 + O18 + O19) MERIS: (760.625 3.75/2 + 778.65 15/2 + 865 20/2 + 885 10/2 + 900 10/2), OLCI: (760.625 3.75/2 + 764.375 3.75/2 + 767. 5 2.5/2 + 778.65 15/2 + 865 20/2 + 885 10/2 + 900 10/2), 5 (MIR1) 1.55-1.75 4, S5 1.46-1.76 - - 11 (20 m) 1610±90/2 7 (MIR2) 2.08-2.35 S6 2.25 - - 12 (20 m) 2190±180/2 6 (TIR) 10.4-12.5 (central wavelength 11.45) 6 OR (6 + 7), S8 OR (S8 + S9) (10.35-11.35) OR [(10.35-11.35) + (11.50-12.50)] - - - - Table 3. Spectral resolutions of Landsat-4/-5 TM and Landsat-7 ETM+ compared with those of the ENVISAT AATSR, ERS-2 ATSR-2, SENTINEL-3 SLSTR, ENVISAT MERIS, SENTINEL-3 OLCI and SENTINEL-2.
3-stage RS-IUS employing SIAM as its preliminary classification first stage First stage: operational automatic deductive prior knowledgebased per-pixel SIAM provided with a feedback loop RS multispectral image RS multispectral image pre-processing (enhancement) Enhanced RS multispectral image First-stage pixel-based preliminary classification (fully automated SIAM ) Spectral category preliminary classification map Second-stage stratified contextual classification (class- and applicationspecific): Divide-andconquer classification approach Land cover classification map Sub-symbolic digital numbers Second-stage driven-by-knowledge context-sensitive classification (under development): Traditional techniques (unsupervised data clustering, image segmentation, supervised data learning classifiers) are employed here on a novel stratified (driven-by-knowledge) context-sensitive basis! Increasing levels of symbolic information == Coarse-to-fine semantic granularity 12 Baraldi, Boschetti, and Justice: SIAM Automatic Preliminary Classification of Sentinel-2 and Sentinel-3 Optical Imagery 4/27/2012
SIAM preliminary classification map legends "Large" leaf area index (LAI) vegetation types (LAI values decreasing left to right) "Average" LAI vegetation types (LAI values decreasing left to right) Shrub or herbaceous rangeland Other types of vegetation (e.g., vegetation in shadow, dark vegetation, wetland) Bare soil or built-up Deep water, shallow water, turbid water or shadow Thick cloud and thin cloud over vegetation, or water, or bare soil Thick smoke plume and thin smoke plume over vegetation, or water, or bare soil Snow and shadow snow Shadow Flame Unknowns L-SIAM legend for 7-band Landsat-like imagery: 95 classes (fine semantic granularity). "Large" leaf area index (LAI) vegetation types (LAI values decreasing left to right) "Average" LAI vegetation types (LAI values decreasing left to right) Shrub or herbaceous rangeland Other types of vegetation (e.g., vegetation in shadow, dark vegetation, wetland) Bare soil or built-up Deep water or turbid water or shadow Smoke plume over water, over vegetation or over bare soil Snow or cloud or bright bare soil or bright built-up Unknowns Q-SIAM legend for 4-band QuickBird-like imagery: 52 classes (fine semantic granularity). 13 Baraldi, Boschetti, and Justice: SIAM Automatic Preliminary Classification of Sentinel-2 and Sentinel-3 Optical Imagery 4/27/2012
NASA-USGS WELD. Year: 2006. Landsat-7 ETM+ in TOARF values. 663 fixed location tiles. Spatial resolution: 30 m. Area coverage: Continental USA and Alaska. Period coverage: 7-year. 14 Baraldi, Boschetti, and Justice: SIAM Automatic Preliminary Classification of Sentinel-2 and Sentinel-3 Optical Imagery 4/27/2012
Automated Application to Weld WELD tile h22v08, annual composite true color SURF (20x30km subset) Baraldi, Boschetti & Roy Automatic Land Cover Classification
Automated Application to Weld WELD tile h22v08, annual composite false color Baraldi, Boschetti & Roy Automatic Land Cover Classification
The first stage is not landcover WELD tile h22v08, Preliminary classification Strong vegetation Average shrub Average barren lands Dark barren lands Baraldi, Boschetti & Roy Automatic Land Cover Classification
NASA-USGS WELD. Year: 2006. L-SIAM SENTINEL-2, SENTINEL-3 SLSTR preliminary classification: fine (95)/intermediate (47)/coarse (18) semantic granularity. 18 Baraldi, Boschetti, and Justice: SIAM Automatic Preliminary Classification of Sentinel-2 and Sentinel-3 Optical Imagery 4/27/2012
Relationship between spectral categories and vegetation land cover classes Strong Vegetation (SV) VHNIR Average Vegetation (AV) VHNIR Average Shrub Rangeland (ASR) VHNIR or HNIR Strong Vegetation (SV) HNIR Average Vegetation (AV) - HNIR Strong Vegetation (SV) MNIR Average Vegetation (AV) - MNIR Strong Vegetation (SV) LNIR Average Vegetation (AV) - LNIR Average Shrub Rangeland (ASR) MNIR or LNIR Average Herbaceous Rangeland (AHR) Strong Herbaceous Rangeland (SHR) Weak Rangeland (WR) - Crop field (Vegetated agricultural fields) - Pastures - Deciduous Broadleaved forests - Deciduous Permanent crop (deciduous fruit-trees) - Crop field (Vegetated agricultural fields) - Evergreen Broadleaved forests - Evergreen Permanent crop (evergreen fruit-trees, e.g., orange tree field) - Crop field (Vegetated agricultural fields) - Evergreen Coniferous forests - Forests in shadow areas - Open forest - Sparse trees (e.g., olive grows) - Regrowth - Transitional woodlands - Clear cuttings - Natural grassland 19 19 Baraldi, Boschetti & Roy Automatic Land Cover Classification
Complete coverage reference map for validation purposes: NLCD2006 Land Cover Map.. 20 Baraldi, Boschetti, and Justice: SIAM Automatic Preliminary Classification of Sentinel-2 and Sentinel-3 Optical Imagery 4/27/2012
ENVISAT AATSR SENTINEL-3 SLSTR (?) 5-band AA-SIAM Fig. A. ENVISAT AATSR image acquired on 2003-01-05, covering a surface area over the Black sea (R: band 7, G: band 6, B: band 4), spatial resolution: 1 km. Fig. B. SIAM classification map generated from Fig. A, consisting of 72 spectral categories depicted in pseudo color (refer to the map legend). 21 Baraldi, Boschetti, and Justice: SIAM Automatic Preliminary Classification of Sentinel-2 and Sentinel-3 Optical Imagery 4/27/2012
ENVISAT MERIS SENTINEL-3 OLCI 4-band Q-SIAM Fig. A. ENVISAT MERIS image acquired on 2010-09-28, covering a surface area over Northern Italy (R: band (7+8) = R, G: band (11+12+13+14+15) = NIR, B: band (3+4) = B), spatial resolution: 300 m. Courtesy of Chelys s.r.l., Rome, Italy. Fig. B. SIAM classification map generated from Fig. A, consisting of 52 spectral categories depicted in pseudo color (refer to the map legend). 22 Baraldi, Boschetti, and Justice: SIAM Automatic Preliminary Classification of Sentinel-2 and Sentinel-3 Optical Imagery 4/27/2012
8-band WorldView-2 at 2 m resolution: Q-SIAM classification and segmentation maps Fig. A. 8-band WorldView-2 VHR image of the city of Rome, Italy, acquired on 2009-12-10, at 10:30 a..m., depicted in false colors (R: band R, G: band NIR1; B: band B), radiometrically calibrated into TOA reflectance. Spatial resolution: 2.0 m. Fig. B. SIAM classification map generated from Fig. A, consisting of 52 spectral categories depicted in pseudo color (refer to the map legend). 23 Baraldi, Boschetti, and Justice: SIAM Automatic Preliminary Classification of Sentinel-2 and Sentinel-3 Optical Imagery 4/27/2012
8-band WorldView-2 at 2 m resolution: Q-SIAM classification and segmentation maps Fig. A. 8-band WorldView-2 VHR image of the city of Rome, Italy, acquired on 2009-12-10, at 10:30 a..m., depicted in false colors (R: band R, G: band NIR1; B: band B), radiometrically calibrated into TOA reflectance. Spatial resolution: 2.0 m. Fig. D. Contour map generated from the preliminary classification map shown in Fig. B. Note: the bridge has disappeared (Undersegmentation error!!!) 24 Baraldi, Boschetti, and Justice: SIAM Automatic Preliminary Classification of Sentinel-2 and Sentinel-3 Optical Imagery 4/27/2012
8-band WorldView-2 at 2 m resolution (zoom): Q-SIAM classification and segmentation maps Fig. E. Zoomed image extracted from the 8-band WorldView-2 VHR image of the city of Rome, Italy, acquired on 2009-12-10, at 10:30 a..m., shown in Fig. A. Spatial resolution: 2.0 m. Fig. F. Zoomed image extracted from the preliminary classification map shown in Fig. B. Fig. F. Zoomed image extracted from the contour mapshown in Fig. D. 25 Baraldi, Boschetti, and Justice: SIAM Automatic Preliminary Classification of Sentinel-2 and Sentinel-3 Optical Imagery 4/27/2012
QuickBird-2, pan-sharpened, 0.6 m resolution (zoom) Urban areas, Q-SIAM Fig. A. Zoomed image extracted from the QuickBird-2 image of Campania, Italy (acquisition date: 2004-13-06, 09:58 GMT), depicted in false colors (R: band CH3, G: band CH4, B: band CH1), 2.44m resolution, calibrated into TOA reflectance, pan-sharpened at 0.61m resolution. Fig. B. Output map generated from Fig. A, consisting of 46 spectral categories depicted in pseudo colors (refer to the map legend). Fig. C. Sunlit Red Roof mask generated from the image shown in Fig. A and the preliminary classification map shown in Fig. B based on a stratified 2 nd -stage classspecific spectral rule. 26 Baraldi, Boschetti, and Justice: SIAM Automatic Preliminary Classification of Sentinel-2 and Sentinel-3 Optical Imagery 4/27/2012
QuickBird-2, pan-sharpened, 0.6 m resolution (zoom) Urban areas, Q-SIAM Fig. A. Zoomed image extracted from QuickBird-2 image of Campania, Italy (acquisition date: 2004-13-06, 09:58 GMT), depicted in false colors (R: band CH3, G: band CH4, B: band CH1), 2.44m resolution, calibrated into TOA reflectance, pan-sharpened at 0.61m resolution. Fig. B. Output map generated from Fig. A, consisting of 46 spectral categories depicted in pseudo colors (refer to the map legend). 27 Baraldi, Boschetti, and Justice: SIAM Automatic Preliminary Classification of Sentinel-2 and Sentinel-3 Optical Imagery 4/27/2012
SIAM is: Conclusions Fully operational (easy-to-use, efficient, accurate, fast, scalable). Sensor-independent, thus it is eligible for use with SENTINEL- 2, SENTINEL-3 SLSTR and OLCI. Completely automated no training, no user-defined parameter. Near real-time processing on a laptop computer. Capability demonstrated at local, regional and national scale. SIAM is eligible for use in: RS image pre-processing (e.g., stratified image co-registration, stratified topographic correction). RS image classification in a 3-stage RS-IUS architecture. Semantic querying of large-scale multi-sensor image databases. 28 Baraldi, Boschetti, and Justice: SIAM Automatic Preliminary Classification of Sentinel-2 and Sentinel-3 Optical Imagery 4/27/2012