DECLIMS. Vessel Classification Benchmark. DECLIMS: Detection, Classification and Identification of Marine Traffic from Space

Transcription

1 Institute for the Protection and the Security of the Citizen Agriculture and Fisheries Unit I Ispra (VA) Italy DECLIMS Vessel Classification Benchmark PROJECT TITLE REFERENCE DECLIMS: Detection, Classification and Identification of Marine Traffic from Space Contract Nr EVG2-CT DOCUMENT TITLE Vessel Classification Benchmark REFERENCE D4 (include D3) FILE D4v01n1.doc VERS 1 AUTHOR Harm Greidanus DATE 24 Apr 2007 DISTRIBUTION PERIOD - COORDINATOR HOME PAGE Public EC-JRC Public: Project:

2 DECLIMS Vessel Classification Benchmark 2 DISTRIBUTION LIST Affiliation Part. nr Person Copies EC-RTD Michele Galatola 2 EC-RTD Bernard Denore e (= electronic copy) QinetiQ 1 Rob Ringrose e JRC 2 Harm Greidanus e CCRS/DRDC 3 Paris Vachon e MDA 4 Gordon Staples e CEDRE 5 François Parthiot e TNO 6 Arthur Smith e (ixl-ag)* 7 - CNR-LaMMA 8 Alberto Ortolani e FFI 9 Richard Olsen e (DXGI)* 10 - CLS 11 Jean-Pierre Cauzac e IRD 12 Michel Petit e IRSA 13 Bingfang Wu e Absolute 14 Jeffery Douglas e GD 15 Chris Wackerman e MHI 16 Shinji Sato e ESA 17 Gordon Campbell e SPOT 19 Olivier Pronier e KSPT 20 Gunnar Rasmussen e KSAT 21 Tony Bauna e UPC 22 Jordi Mallorquí e Definiens 23 Ursula Benz e DSpD 24 Jake Tunaley e BOOST 25 Vincent Kerbaol e This distribution list shows for each project partner only a single name, the person in direct charge of the work. *: The partners ixl-ag and DXGI have gone out of business. Action leader Name Th. Barbas Date Signature Unit head Name J. Delincé Date Signature

3 DECLIMS Vessel Classification Benchmark 3 Legal notice The contents of this report do not necessarily reflect the official opinion of the European Communities or any European Communities institution. Neither the European Commission nor any person or company acting on the behalf of the European Commission is responsible for the use that may be made of the information contained in this report. DECLIMS PARTNERS JRC European Commission Joint Research Centre QQ QinetiQ CCRS/DRDC Canada Centre for Remote Sensing / Defence R&D Canada MDA MDA Geospatial Services CEDRE Centre de Documentation, de Recherche et d'expérimentation sur les pollutions accidentelles des eaux TNO The Netherlands Organization for Applied Scientific Research IXL ixl Satelliten-Informations-Aktiengesellschaft LaMMA Consiglio Nazionale di Ricerca Laboratorio per la Meteorologia e la Modellistica Ambientale FFI Norwegian Defence Research Institute DXGI Doxiades GeoImaging CLS Collecte Localisation Satellites IRD Institute de Recherche pour le Developpement IRSA Chinese Academy of Sciences Institute for Remote Sensing Applications Absolute Absolute Communications GD General Dynamics Advanced Information Systems MHI Mitsubishi Heavy Industries ESA European Space Agency SPOT SPOT Image KSPT Kongsberg Spacetec KSAT Kongsberg Satellite Services UPC Universitat Politècnica de Catalunya Definiens Definiens Imaging DSpD Defence Canada Directorate of Space Development BOOST BOOST Technologies

4 DECLIMS Vessel Classification Benchmark 4 EXECUTIVE SUMMARY The DECLIMS project is concerned with detection, classification and identification of maritime vessels using radar and optical imagery from commercial satellites. This report discusses an exercise to benchmark the capabilities of a number of automatic ship detection and classification systems using SAR (Synthetic Aperture Radar). Nine such systems have analysed the same dataset, which consisted of 19 SAR images plus ground truth in the form of known ships in the images. Already in the preparation of this exercise it was clear that none of the SAR ship detection and classification systems considered in DECLIMS have the capability to derive the class, or type, of the detected vessel (fishing, cargo, tanker, etc.). Classification therefore was limited to size estimation. The 9 systems provide different subsets of target parameters with their detections. Most, but not all, give a length estimate and a reliability indication of the detection; only about half of them give an estimate of width or heading; and only few include speed or RCS (Radar Cross Section) estimates. The benchmarking consisted of comparing the detection and classification outcomes of the different systems on the different images with each other, with the ground truth, and with visual estimates of the detected vessel sizes by a human operator. For some practical reasons, only 7 systems and 16 images were considered. Not all systems had analysed all images, and the benchmarking encompassed a total of 86 system/image combinations, each of which consisted of a list of detected ships with their attributes. Concerning the detection part of the benchmark, it was found that the different partner systems produced rather different numbers of detections on the same image. Some partner systems appeared to have a high false alarm rate. Although many partners had implemented improvements to their detection algorithms based on the outcomes of the previous SAR ship detection benchmark, some of the then signalled problems were still present. In particular, most systems still suffer from an excess false alarm rate near the coast. Concerning classification, it can be argued that a human operator will always produce better size estimates of vessels in SAR images than an automatic system. Therefore, first the performance of visual analysis was assessed. This gives an indication of the SAR imaging fidelity, regardless of the performance of any automatic analysis system. This showed that at the highest presently available spaceborne SAR resolution ( Fine, 8 m) and under calm conditions, visual ship length estimates are reasonably reliable (3-sigma error of the order of 25 m). At Standard resolution (25 m) or at higher sea state, visual ship length estimates are subject to an error of the order of 100 m. For either resolution, azimuth smearing due to vessel motions on the waves can introduce very large overestimates these cases are not included in the quoted error measures. Concerning the performance of the automatic systems, serious estimation errors were found for all 7 tested systems on the Standard images in the benchmark data set. However, there were also significant differences in performance between the systems. The better systems show a 3-sigma error in ship length estimation of the order of 150 m. Whereas length overestimates can be explained by azimuth smearing, many systems also showed many cases of length underestimation, which are less easily explained. The automatic systems could deal better with the images at Fine resolution, where most of them could estimate ship length with an error of 50 m (not counting outliers). When comparing the size estimates of the automatic systems with visual size estimates, instead of with real vessel size, it is found that the deviations are somewhat less but still large. Taken together with what was concluded on visual performance, this implies that the performance of the automatic systems is still limited by their inability to approach the performance of a human analyst, and not yet by fundamental SAR image limitations, except for the cases of azimuth smearing due to vessel motions. The present study could be further extended, on the basis of the existing data, by closer inspection of individual cases of large deviation, by examining the correlation of ship heading with size estimation errors, and by including the use of RCS to support size estimation.

5 DECLIMS Vessel Classification Benchmark 5 CONTENTS 1 Introduction Method Data set Analysis Partner system outputs Detection Classification Visual size estimation Automatic size estimation Conclusions...18

6 DECLIMS Vessel Classification Benchmark 6 ABBREVIATIONS AIS Automatic Identification System AP Alternating Polarisation BMK Benchmark EC European Commission HH Horizontal send, horizontal receive polarisation HV Horizontal send, vertical receive polarisation RCS Radar Cross Section SAR Synthetic Aperture Radar VHR Very High Resolution VMS Vessel Monitoring System VTS Vessel Traffic Services VV Vertical send, vertical receive polarisation

7 DECLIMS Vessel Classification Benchmark 7 1 INTRODUCTION The DECLIMS project aims to establish the state-of-the-art in ship detection and classification from commercial satellites, and to stimulate further developments and applications in that area. It includes optical sensors and radar (SAR, Synthetic Aperture Radar). It has now 24 partners who are active in maritime surveillance and use earth observation satellites. One of the activities in the project is benchmarking the ship detection and classification algorithms and systems in use with the partners. This report is about the second benchmarking exercise in DECLIMS concerned with SAR. In this type of benchmarking, partners analyse a common test set, each with his own algorithms and system, and the results are compared compared with ground truth (which should be available) and compared to each other. This procedure gives good insight in what is the general capability level amongst the partners, what are common capabilities or problems, and what are capabilities of problems specific to individual partners. In that way, the state-of-the-art is established and problems are identified. Partners can learn from each other, are indicated ways for improvements and are spurred to implement them. This report covers three goals of the project: Perform a second SAR ship detection benchmark; Perform a SAR classification benchmark; Include new sensors. The first SAR ship detection benchmark, BMK1, was executed in 2004 [1, 2]. The project plan called for a second SAR detection benchmark, so that partners could have the opportunity to test improvements that they had made to their algorithms and systems based on the results of BMK1. The project plan also stipulated the use of new sensors, i.e., sensors that became available during the project. The BMK1 test set included RADARSAT ScanSAR and Standard modes. In the present benchmark test set, RADARSAT Fine is included in addition to RADARSAT Standard, as well as ENVISAT-ASAR. The latter is an instrument that was not existing at the time the project proposal was written. It was decided to combine the second SAR detection benchmark exercise with the SAR classification benchmark. After all, detection has to be done before classification. A test set was compiled containing the above types of images, plus ERS-2. No ScanSAR or Wide Swath images were included, because the resolution provided by those two modes ( m) is too coarse for any useful classification. The test set was put together from contributions by a number of partners. Preliminary results of this SAR vessel classification benchmark exercise were published earlier in [3].

8 DECLIMS Vessel Classification Benchmark 8 2 METHOD 2.1 DATA SET The test set was compiled from contributions by different partners: MDA, FFI, KSAT, CCRS/DRDC, CLS, TNO and JRC. ESA has allowed ENVISAT images to be shared by the DECLIMS partners, and MDA has done the same for RADARSAT images. The essential element is not only the SAR image, but also the corresponding ground truth. Ground truth or in-situ information means that the image contains a number of known ships (with known sizes). In the beginning of the project, it was not easy to obtain SAR images with known ships. In the best cases, only a small number of ships in the image could be known. The VMS (Vessel Monitoring System) with which fishing ships are tracked could be used for this. Later, the introduction of AIS (Automatic Identification System) made it easier to have images that contain a large number of known ships. Still, AIS operates only on large vessels, and smaller ships in the image most often remain unidentified. The test set for the benchmark is specified in Table 1. Nr Campaign From Date Sat Mo Bea Pol Targets reported w/ size Of these detected 1 Oslofjord FFI/KSAT ASAR AP IS1 HX Oslofjord FFI/KSAT ASAR AP IS7 HX Oslofjord FFI/KSAT ASAR AP IS7 HX Oslofjord FFI/KSAT ASAR AP IS7 VX Norne FFI/KSAT ASAR AP IS1 HX Norne FFI/KSAT ASAR AP IS4 HX Norne FFI/KSAT ASAR AP IS7 HX Gijón CLS ASAR IM IS2 VV Gijón CLS ERS - - VV 10 Gijón CLS ERS - - VV 11 Greece JRC RSAT F F5 HH Greece JRC RSAT F F4 HH Gibraltar MDA/JRC RSAT F F4 HH Gibraltar MDA/JRC RSAT F F4 HH Halifax CCRS/DRDC RSAT F F4 HH Halifax CCRS/DRDC RSAT F F2 HH Vancouver MDA RSAT F F5 HH Den Haag TNO/JRC RSAT S S5 HH Den Haag TNO/JRC RSAT S S2 HH Table 1. The benchmark test set. The column Sat is the satellite (ENVISAT-ASAR, RADARSAT or ERS). Mo is the mode (Alternating Polarisation, Image Mode or Fine). Bea is the beam (named according to the sensor nomenclature; low numbers indicate steep incidence angle). Pol is polarisation (HX=HH+HV, VX=VV+VH). Table 1 specifies the number of ground-truth targets that are known with their size for each image. But these targets cannot always be detected; the final column gives how many of those could be seen in the image. This final number is the number of useable ground truth ships; it totals 132. Two images in the test set had to be left out of consideration: the ERS images. This was due to the fact that the software at JRC, who analysed all the partner results, was not able to ingest this type of image. Furthermore, it turned out that one image (number 11) contained no ground truth vessels (in contrast

9 DECLIMS Vessel Classification Benchmark 9 to what was earlier believed when it was introduced in the test set). Therefore, the total number of images on which the analysis in this report is based is 16. Table 2 gives some further information about the images and the ground truth. Nr Campaign Mo Bea Pol Det. In situ Remarks in situ 1 Oslofjord AP IS1 HX 1 VTS NW part of Oslofjord; steep incidence, difficult clutter 2 Oslofjord AP IS7 HX 1 VTS NW part of Oslofjord 3 Oslofjord AP IS7 HX 6 VTS W bay of Oslofjord and SW seacoast 4 Oslofjord AP IS7 VX 9 VTS Oslofjord and SE seacoast 5 Norne AP IS1 HX 3 Ship No coast 6 Norne AP IS4 HX 3 Ship No coast 7 Norne AP IS7 HX 1 Ship No coastl; several unknown targets 8 Gijón IM IS2 VV 3 AIS Coast 9 Gijón - - VV No coast 10 Gijón - - VV Coast 11 Greece F F5 HH 0 No known vessels 12 Greece F F4 HH 8 VMS A/D undersampled; coast; fishing ships 13 Gibraltar F F4 HH 9 Opt Known ships are stationary; many more unknown; coast 14 Gibraltar F F4 HH 6 Opt Known ships are stationary; many more unknown; coast 15 Halifax F F4 HH 1 Ship Coast 16 Halifax F F2 HH 1 Ship Coast 17 Vancouver F F5 HH 8 VTS Coast; some of the known ships on river 18 Den Haag S S5 HH 34 AIS Coast; 45 AIS targets 19 Den Haag S S2 HH 38 AIS Coast, 54 AIS targets Table 2. Notes about the benchmark test images. The first 6 columns are repeated from the previous table. The column Det. in situ is the number of ground truth targets that could be detected in the image (last column of previous table). The column In situ is the source of the ground truth (see text). Ground truth comes from VTS (Vessel Traffic Control) systems; from coastal AIS; from the known reported position of a single ship in the image (marked as Ship in Table 2); from VMS (in case of fishing ships); or from near-simultaneous VHR (Very High Resolution) optical satellite images, which allow to estimate the size of the ships. The last case concerns an experiment in the Bay of Gibraltar, and those data have been analysed in [4]. 2.2 ANALYSIS All images were analysed by JRC to verify whether the ground truth vessels could be detected. As Table 1 already showed, this was not always the case. For each image, a file was made listing the positions of the known ground truth vessels (both as sample, record in the image and as latitude, longitude) indicating whether they could be found on the image or not. These files were distributed to the partners. The satellite SAR images were analysed by various DECLIMS partners for ships and their attributes. Except for the list of detected ground truth vessel positions, the partners did not have access to the ground truth during this phase. (Of course, they knew the ground truth that they had supplied themselves.). So, the partners knew the positions of the ground truth vessels, and which ones were detectable, but not their sizes or other attributes. The partners subsequently submitted their analysis results to JRC. These submissions had to be in a predetermined format: a flat ASCII text file for each image carrying a name that identified image and partner, containing one line per detected target, and on that line in a fixed order the target attributes. The format is specified in [5].

10 DECLIMS Vessel Classification Benchmark 10 At the same time, JRC analysed all the targets in the images visually for a length and width estimate. This was done by displaying each target on a monitor and estimating size by eye. The ENVI software was used for that purpose. So for each detected ground truth vessel there is not only a real size available, but also a visually estimated size to which the size estimates of the partner systems can be compared. In addition, visual size estimates were made of other detected ships in the image, ones which are not part of the set of known ground truth ships. After all partner system results were received at JRC, they were inter-compared. Because of the large number of images, partners and detected ships, this could not be done by hand but had to be automated. Software was written in MATLAB to that end. The first step is to associate the detected targets of the partners with the ground truth targets. Some partners had used the supplied list of ground truth target positions and assigned their target attribute estimates (length, width, heading, etc.) to them, thus exclusively addressing the classification part of the benchmark. But most partners, aiming also at the detection part of the benchmark, had run their automatic detection systems, resulting in a list of targets (with attributes) that did not directly correspond to the list of ground truth targets. In that case the partner detections had to be associated with the ground truth targets via their location. The location of a detected target can be not exactly the same from different systems, because targets have an extended size, or because the conversion to geographical coordinates is not exactly the same. Software was written to associate targets on the basis of vicinity. The results were verified by plotting detected vessel positions of the partner systems with their associated ground truth vessel positions. After this, the attributes were analysed by scatter plots. Graphs were made plotting the vessel length from a partner system versus the real vessel length. And similar with vessel width; and comparing either to visually estimated vessel size. Also visually estimated vessel length was compared to real vessel length. With various combinations of plot attributes and selections of images and partners, the performance of the partner systems was analysed. It was found that results were rather different for two classes of images: Standard and Fine. With Standard is meant, images of 25 m multi-look resolution, including RADARSAT Standard, ENVISAT-ASAR IM, AP and ERS. Fine is RADARSAT Fine images of 8 m single look resolution. Therefore, in the analysis it was found useful to combine results over images, but split between the Standard and the Fine images.

11 DECLIMS Vessel Classification Benchmark 11 3 PARTNER SYSTEM OUTPUTS The DECLIMS partners who participated in analysing the test set, and sent their results to JRC to be compared, were: Number Partner System 01 JRC SUMO 02 QinetiQ MaST 09 FFI FFI 15 General Dynamics GD 21 KSAT MeosView 25 BOOST SARTool 31 CCRS/DRDC IAPro 32 CCRS/DRDC OMW 33 CCRS/DRDC VUSAR Table 3. Partners who have analysed the benchmark test set. Not all partners have analysed all images, either because their system could not ingest some image formats, did not properly work on the image types, or because it proved too much work. The overview of who analysed which images is given in Table 4. Image Partner F F F F F F 01 JRC x x x x x x x x x x x x x x QQ x x x x x x FFI x x x x x x x x x x GD x x x x x x x x x x x x x x x x x KSAT x x x x x x x x x x x x x x x BOOST x x x x x x x x x x x x x x x x x x IAPro x x x x x x x x x x x x x x x OMW x x x x x 5-33 VUSAR x x x x x x x x x x x x x 13 - Total Table 4. Overview of which partner analysed which images. The fine beam images are indicated on top with an F. The results that were not taken into account in the comparison are indicated by -. The last two columns show the total numbers of images analysed by each partner (column All in ) and the total number taken into account in the comparison (column All com ). All in All com Table 4 shows that a grand total of 113 ship detection and classification runs were carried out and submitted for comparison. Images 9, 10 and 11 were not taken along in the comparison, as mentioned because the JRC software that was used could not ingest ERS images, and for image 11 there was no ground truth. Furthermore, only the IAPro results of CCRS/DRDC were used in the comparison in this report. But that was for practical reasons, to save space and because not enough different symbols were available to discern the different partners in the plots. Of these three inputs from CCRS/DRDC, IAPro was chosen because VUSAR does not give ship size (on which this benchmark is based), and OMW only had 5 images analysed. Therefore, without the results of images 9, 10 and 11, and systems OMW and VUSAR, the grand total of ship detection and classification runs that was used in the comparison in this report was 86. The different partner systems provide different types of information about the targets, taken from the categories position, length, width, reliability, RCS (Radar Cross Section). Table 5 lists what target information was provided by the different systems.

12 DECLIMS Vessel Classification Benchmark 12 Partner system Pixel Geogr Reliab Length Width Class Head Speed RCS 01 JRC x x x x x x 02 QQ x x x x 09 FFI x x x x x x 15 GD x x x x x x 21 KSAT x x 25 BOOST x x x x x x 31 IAPro x x x x x 32 OMW x x x x x 33 VUSAR x x x x Table 5. The information about the vessels that is provided by the different systems. The first three columns of Table 5 concern detection attributes: Pixel means that the sample, record location in the image is given; Geogr that the latitude, longitude is given; and Reliab means that a reliability estimate of the detection is provided. All systems provide the geographical location of the detection (it could not be otherwise for an operational system), but all other attributes show variation. Most systems, but not all, give the pixel location in the image, which is arguably only useful for internal and control purposes. Also, most systems but not all give a reliability; this can however be useful information for end users. All systems provide a ship length estimate, except VUSAR which has for that reason not been taken along in the comparison. Only half of those systems also provide ship width. Ship heading is given by just more than half of the systems; speed only by one; and RCS only by the three Canadian systems. Conspicuously, the column in the table that is entirely empty is Class, which should indicate that a ship class or type is specified (such as fishing vessel, cargo ship, tanker, cruise liner, etc.). This is the purpose of classification. But no system in the DECLIMS consortium provides this output. This had become clear of course earlier, while defining the benchmark test. It was decided for that reason that the vessel classification benchmark should concentrate on vessel size. Vessel size is very valuable information in its own right, and it limits the types to which a vessel can belong. It is very relevant to take RCS into consideration, because it provides a handle on ship size. There is an order of magnitude relationship between ship length and ship RCS. However, this issue was not pursued in DECLIMS. The ENVISAT-ASAR Alternating Polarisation (AP) images have two polarimetric channels. Most partners have provided two output ship detection and classification sets corresponding to the two channels. But some partners have given only a single output set, presumably based on an integration of the two channels or a selection of the best one. In what follows, first the results relating to ship detection will be described, and after that the results relating to ship classification. Because of the large number of figures, they are put at the end of the report, but the text will refer to them. The reason to include the large amount of figures is so that the partners can use them to assess in detail the performance of their systems, and hopefully find directions for improvements.

13 DECLIMS Vessel Classification Benchmark 13 4 DETECTION Figure 4 illustrates the ship detection results of the participating systems. The figure spreads over 16 pages because it has one page per image. For each image, the partner detections are plotted as panels in the same order. The plots show the geographic locations of the detections. They are scaled to their own geographic extent, so the area plotted is different in each image. This was chosen because in that way in the situations were the targets are geographically close, more detail can be seen. The symbols and colours used in Fig. 4 are explained in its legend, before its first page. Partner plots which contain only black dots (ground truth vessel positions) indicate that that partner has not processed that image, but they are kept anyway to have the same partner always appear at the same location on the page. Partner 31 (CCRS/DRDC IAPro) has only analysed the provided ground truth positions, and not performed full detection runs on the images, so the inclusion of their detection plots is not really meaningful. However, it does illustrate in some cases problems that the association software has. This software associates a partner detection with a ground truth position when they are less than 500 m apart. However, it makes no association in case there is no one-to-one correspondence, i.e., if there are several partner detections within 500 m of a ground truth position or vice versa. Problems in association are due to the JRC benchmark analysis software, not to the IAPro system. It seems that some partners have manually limited the search area to be around the ground truth positions. This could explain why for example partner 15 (GD) in image 1 only finds targets within a small geographic area. There are clear differences in the number of detections on the same image between the partners. For example, partner 25 (BOOST) often finds many more targets than others. This can be an effect either of threshold setting, or of the values assigned to the reliability attribute. Concerning threshold, all the algorithms have a free parameter in the detection to exchange being more sensitive to smaller targets with picking up more false alarms. The benchmark did not specify a priori a false alarm rate, so variations in detection threshold setting and consequent false alarm rate from the different partners are to be expected. Concerning reliability, the benchmark did specify a convention as follows [5]: 1 = probable false alarm 2 = vessel with low confidence 3 = vessel with high confidence In the comparison analysis, only vessels with reliability 2 or 3 were retained. It can be seen that most systems find too many targets along coastlines. For example, in the bottom right plots of images 17 and 18 of Fig. 4 one can see lines that become visible because many detections coincide; these essentially trace out the coast line. This clearly points to an excess false alarm rate near the coast, due to either inaccurate land masking or effects from ambiguities or coastal sea clutter features. This problem was already signalled in the first SAR detection benchmark test.

14 DECLIMS Vessel Classification Benchmark 14 5 CLASSIFICATION It was already mentioned that one of the conclusions of DECLIMS is that there are no operational systems that can deduce vessel class from a commercial satellite SAR system (at least not among the 24 DECLIMS partners). And that, therefore, the classification focuses on size estimate. The remainder of this chapter is about that aspect; when classification is mentioned in the following, size estimation is meant. 5.1 VISUAL SIZE ESTIMATION Existing automatic systems for vessel size estimation are essentially based on emulating the interpretation by a human analyst. They are not based on fundamentals that would make them essentially better than what a human analyst can do. Indeed, it is thought that an experienced analyst would probably perform better in estimating vessel size than any of the existing automatic algorithms. This should be especially valid with respect to being able to deal with different and unexpected situations, vessel types, sensors and sensor modes. Automatic algorithms may be well tuned to specific circumstances, but can easily be thrown off when unexpected complications occur, which an experienced human analyst can recognise without problems. The reasons to use an automatic system are in the first place speed (a human would take much longer to analyse a full image), cost (computer system is cheaper), availability (analysis may need to take place at hours when there is no staff) and objectivity (different analysts, or the same analyst on different days will give different estimates). Considering that automatic software to classify ships is essentially an encoding of human knowledge, visual analysis by an experienced operator would probably give the best possible results, while results of any automatic system would be less accurate than that (except on the accounts of speed and consistency as mentioned above). Therefore, it makes sense to in the first place compare the results of visual analysis with the ground truth data; this can be expected to give an upper limit to what can be expected from the automatic systems. This comparison is done in Fig. 1. This figure combines the results of all Standard images and all Fine images, but keeping these two separate. It shows, for both these image classes, the comparison between visually estimated vessel size against real vessel size according to the ground truth data. It has different plots for length and width. In Fig. 1, colour and symbol refer to polarisation (see legend). Looking at the plot for length estimation of Standard images (25 m resolution, upper left), it can be seen that there is quite a large scatter. There are a number of serious overestimates (ships of around 100 m that appear to be around 400 m long) and there is one serious underestimate. Excepting these cases of gross mis-estimation, the scatter is such that a visual length estimate has a 3-sigma random error of the order of 100 m. It can not at all be excluded that some values on the in-situ axis are false, i.e. that the SAR target does not have the size it was thought to have. This can happen when the ground truth is erroneous or when the association between SAR target and ground truth is wrong. Some of the images contain quite dense shipping which may give rise to wrong associations, and this can be exacerbated in some cases by the fact that no correction was made for SAR displacement effects due to ship range speed. Concerning the large overestimations, a likely cause is the azimuth smearing that acts on vessel SAR signatures as a result of vessel rotational motions on the waves. It is possible to verify that by combining the ratios of visual size over real size as a function of ship heading in the image. A preliminary test has shown that indeed vessels with an apparent heading in azimuth direction are the ones with the largest size overestimations. This was presented in one of the DECLIMS meetings but is not followed up here. Considering the effect of polarisation, the plot gives the impression that HV (green X) has a tendency to overestimate length, VV (blue V) also but to a lesser extent, while HH (red +) shows no particular systematic effect.

15 DECLIMS Vessel Classification Benchmark 15 The plot of ship width (upper right) also shows a large scatter, and a systematic overestimation of 20 m. The latter is not too bad considering that the resolution is 25 m. Differences in polarisation are not pronounced. Fig. 1 in the bottom row shows the same results, but now for the Fine images (8 m resolution). Here, the scatter is much less, both for length (bottom left) and width (bottom right). Again, there are some serious length overestimations, especially for the smaller vessels, which can possibly be ascribed to azimuth smearing. And the width is systematically overestimated by some 8 m which is again of the order of the resolution. Figures 5 and 6 split out the scatter plots of Fig. 1 over all images; Fig. 5 for the Standard images and Fig. 6 for the Fine images; both run over more pages. From Fig. 5 it can be seen that most of the Standard images lead to overestimations of the visual size estimate, and that almost all underestimations come from image 19, a steep incidence angle (S2) RADARSAT Standard (HH) image. Fig. 6 similarly shows that most of the mis-estimations (overestimations) originate from a single image, namely image 12. This is an image with small fishing vessels, and azimuth smearing due to ship motion is a plausible cause. 5.2 AUTOMATIC SIZE ESTIMATION OVERALL RESULTS Figure 2 shows how the vessel size estimates from the partner systems compare with the real sizes. It is, in the same way as Fig. 1, split out in four plots between length and width, for Standard and Fine images. In this figure however, the symbols indicates the partners, as detailed in the legend, while the colour indicates polarisation as in the previous figure. The top left plot (length from Standard images) seems to show almost random scatter, and the plot for width (top right) shows the same behaviour. Individual ships receive length and width estimates form the various partner systems that can vary by huge amounts. From these plots, one is led to conclude that as an ensemble, the automatic detection systems give no reliable size estimate on Standard images whatsoever. The situation for the Fine images (bottom row) is a bit better. One can discern a number of small targets that receive much too large size estimates; this effect was also signalled in the analysis of the visual estimates, with a possible explanation of azimuth smearing. And one can discern a few outliers of large ships whose size is much underestimated. But for the rest, the estimates fall around the centre y=x line, albeit with significant scatter. Now it can be investigated whether the output of the automatic systems compares better with visual size estimates from the images than with real size. This would be expected, considering that the SAR imaging creates a distortion in the real size which is experienced both by the human interpreter and the automatic algorithm. This comparison is done in Fig. 3, which is organised in exactly the same way as the previous figure, except that the horizontal axis is visual size estimate instead of actual size. Note that the number of targets in this figure is larger: visual size estimates are available not only of the ground truth targets, but also of the other detected targets in the images. The top left plot of Fig. 3 (system length estimates on Standard images) shows indeed somewhat less scatter than its sister plot of Fig. 2. It seems to have two branches: one that follows the y=x line, and one which extends horizontally above the bottom. However, the scatter is still very large. The distribution of the points in the width estimates of the Standard images (top right) is rather different from that of the previous figure; but it can hardly be said that the correlation has much improved. This would seem to imply that the human operator does not make the same estimation errors as the automatic algorithms, and, in combination with what was seen in Fig. 1 where the visual estimation was validated, the automatic size extraction methods lack much of the capability of estimating SAR vessel sizes in Standard imagery that the human analyst has. For the case of the Fine images (bottom row), there is not much difference between the points distributions in the plots of Figs. 2 and 3. Combining this with the relatively good visual estimation capability for the Fine images as demonstrated in Fig. 1, this confirms that the errors in the automatic

16 DECLIMS Vessel Classification Benchmark 16 size estimation on Fine images are much larger than those in the visual estimates. (While they are still much smaller than those on the Standard images.) These overall results can be further analysed by looking into the effects of the individual images, and those of the individual detection systems INDIVIDUAL IMAGES The effects of the individual images are examined in Figs These figures show correlations of all partner system results together, but split out over individual images, length and width, and comparing to real size and visual size. In particular, Fig. 7 (Standard images) shows that most images display a similar scatter, and a general trend for overestimation. Images 3 and 4 show some extremely large partner length estimates of up to 1,000 m, which were not even displayed in Figs. 2 and 3 because the axis range was fixed for those plots. Images 18 and 19 are of particular interest, because they contain a large number of targets. Image 18 seems to have two branches: one which lies above y=x and overestimates the size, and one which lies above the bottom (underestimating). In image 19 also two branches can be recognised, but the top one follows y=x much better. In Fig. 8 (Fine images), it is foremost Image 17 which has a lot of points deviating very much from y=x. Figures 9 and 10, where the same analysis is made as in 7 and 8 except comparing system length with visual length instead of with real length, show essentially the same trends except the correlations are a bit better, as expected. In Fig. 11 which compares vessel width from Standard images we can see roughly the same behaviour as in Fig. 7 (length). In particular, image 18 shows much overestimation while image 19 much less so INDIVIDUAL SYSTEMS Figure 15 plots length partner estimates against real and visual length, with one separate plot for each partner that combines all Standard images. Figure 16 does the same for width. These plots clearly indicate the behaviour and performance of the individual systems. The plots of partners 2 (QQ) and 21 (KSAT) stand out because the size estimates seem to be nearly constant values, mostly underestimating the target size. The plots of most other partners contain a population of targets that shows up as a line near the bottom of the plots; these are targets of varying size for which a very low size estimate is given by the system (essentially, unresolved). One partner system which does not have this is 31 (CCRS/DRDC IAPro). For the rest, the partner systems scatter their estimates, as compared to real length, roughly around y=x. Partners 1 (JRC), 15 (GD), 25 (BOOST) and 31 (IAPro) tend to overestimation, whereas partner 9 (FFI) tends to underestimation. When compared to visual length (right column of Fig. 15), the scatter of the partner estimates lessens in most cases, except for partner 9 (FFI). Partner 31 (IAPro) has the least scatter of all when comparing its size estimates with visual estimates. The previous paragraph was pertained to length estimates on Standard images (Fig. 15). Concerning width estimates on Standard images (Fig. 16), partner 2 (QQ) does better than it did for the length (at least given the small dynamic range in width values). Partners 21 (KSAT) and 31 (IAPro) do not provide width estimates. For the rest, the behaviour of the width estimates is similar to that of the length estimates. Figure 17 is the equivalent of Fig. 15, but looks at the length estimate for the Fine images instead of the Standard. Most partners show reasonable estimates, as was already expected from the composite plots that were discussed earlier (Figs. 2 and 3). Like for the Standard images in the previous figure, however, partner 21 (KSAT) severely underestimates. Partner 2 (QQ) has too few Fine cases analysed to make a conclusion; and partner 9 (FFI) none at all. Interestingly, the comparisons do not seem to improve when using the visual estimates as a reference instead of the real sizes. In that case, a population of targets appears in the plots in a line near the bottom, i.e., targets with large visual length but small system estimates; the systems seem somehow to be better to approach the real (small) length values for those cases. Looking back at Fig. 3, this population can be recognised there as well. No

17 DECLIMS Vessel Classification Benchmark 17 explanation can be given for this effect at this point. The results of partner 31 (IAPro) show the least scatter, like they did for the Standard images. Finally, the width correlations on the Fine images (Fig. 18) do not add essential information to what was discussed above.

18 DECLIMS Vessel Classification Benchmark 18 6 CONCLUSIONS Concerning SAR ship detection is can be said that, although many partners that run detection algorithms have improved them since the first SAR detection benchmark test, this improvement was not quantified by the present benchmark. It could however be established that one of the main problems with automatic SAR ship detection, namely excess false alarms near coastlines, has not been solved. Concerning SAR ship classification, it has to be in the first place concluded that there is no operational capability to derive vessel class (like fishing vessel, ferry, tanker, ) from commercial satellite SAR imagery. The present SAR classification benchmark therefore concentrated on vessel size estimation. The partner algorithms produce different attributes with their detected ships. Of the 9 automatic SAR ship detection systems that participated in the benchmark, 7 give a reliability with the detection; 8 give a length estimate; 4 a width estimate; 5 a heading estimate; 1 a speed estimate; and 3 an RCS estimate. Only length and width estimates have been analysed here, supported by the reliability figure. Of the 19 original images in the benchmark test set, 16 were analysed in this report. A comparison of visual ship size estimates with in-situ information shows that even visual analysis of vessel SAR signatures by an expert is subject to significant errors as regards size extraction. The errors found in this study are quantified in Table 6, top part. The table makes a distinction between Standard images (25 m resolution) and Fine images (8 m resolution). Expressed in units of resolution, and disregarding outliers, the errors for visual size estimation are not vastly different for Standard and Fine. In absolute numbers, however, the 3-sigma random error for visual length estimation in Standard images is 100 m, which is quite large. The 3-sigma random error for visual length estimation in Fine images is at 25 m of a more practical value. Analysis of the 86 different input datasets (each one the result of a detection / classification run of a partner system on an image, containing a list of detected vessels with their attributes) shows that automatic size estimation on Standard images is essentially impossible, at least when all systems are viewed as an ensemble. The performance of the automatic systems is rather better on the Fine images, where the random error on length is a factor of 2 bigger than for visual estimation (50 m; disregarding outliers). See Table 6, bottom part. Visual Standard Fine 3σ Error Bias 3σ Error Bias Length 4 * +2 a 3 * 0 * Width 1 * System Standard Fine 3σ Error Bias 3σ Error Bias Length (6 * b ) - 6 * +4 * Width Table 6. Random error and bias in estimation of vessel size, in units of resolution (25 m for Standard, 8 m for Fine). Top table: errors for visual estimates. Bottom table: errors for system estimates, taking all partner systems together. *: not counting outliers a: except image 19 where underestimate b: not taking all systems together but only for the better systems When the classification results are inspected more closely, it is seen that there are significant performance differences between different images, and between different partner systems. Therefore, some of the systems perform significantly better than the figures mentioned in the bottom part of Table 6 (and some significantly worse). Differences between the partner systems are especially clear

19 DECLIMS Vessel Classification Benchmark 19 for length estimation on the Standard images (Fig. 15); for that case, the better systems have a random 3-sigma error of the order of 150 m. It is believed that some of the larger errors (outliers) are caused by azimuth smearing of the targets, which leads to a large extension in azimuth direction of the ship signatures. This effect can lead to a size overestimation in many cases, also with less extreme effects. Most of the ships in the Fine images, where size estimation performed better, were actually stationary and the images were acquired under low sea state, whereas most of the ships in the Standard images were moving and not especially at low sea state. However, also severe underestimations are found. Some of these occur in one of the images (19) with steep incidence angle, and may be due to the consequent low ship-sea contrast. This should lead to a less well defined and apparently smaller ship outline. Of course, some of the scatter in the size estimates may be due to wrong associations between the ships detected in the image and the ground truth ships, or even to wrong sizes specified in the ground truth. The classification results of the automatic systems can also be compared to the visual size estimates, instead of to the real sizes. Then the scatter is somewhat less in case of the Standard images, but not much different for the Fine images. This implies, together with the numbers quoted in table 6, that most of the automatic algorithms for ship size estimation still have much room for improvement before they approximate the performance of a human analyst, which is by itself far from perfect. In this respect the IAPro (Image Analyst Pro) system run by CCRS/DRDC lives up to its name: from the 9 systems tested, its performance comes closest to that of the human operator. The analysis presented in this report has not exhausted the information content of the benchmark data set, with all the partner results, that has been collected. The analysis could be further refined by looking into individual cases of large deviation to find possible causes. Size estimation errors could be correlated with apparent ship heading to further verify the notion that overestimation is caused by azimuth smearing, as a preliminary study had indicated earlier. Finally, the use of RCS to support size estimation could be pursued.

20 DECLIMS Vessel Classification Benchmark 20 REFERENCES 1. H. Greidanus, 2004: BMK1 Analysis description, WP2B1AnalysDescr2-7.doc, 22 Mar Harm Greidanus, Peter Clayton, Marte Indregard, Gordon Staples, Naoki Suzuki, Paris Vachon, Chris Wackerman, Tove Tennvassas, Jordi Mallorquí, Naouma Kourti, Robert Ringrose, Harm Melief, 2004: Benchmarking operational SAR ship detection, IGARSS 2004, Sept 2004, Anchorage, Alaska 4. H. Greidanus, N. Kourti, 2006: A detailed comparison between radar and optical vessel signatures, IGARSS 2006, 31 July - 4 Aug 2006, Denver, CO, USA 5. H. Greidanus, 2006: Format specification for output of vessel detection benchmark, WP3B1Format2-2.doc, 1 Feb Harm Greidanus, Naouma Kourti, 2006: Findings of the DECLIMS project Detection and classification of marine traffic from space, SEASAR 2006: Advances in SAR oceanography from ENVISAT and ERS, Jan 2006, ESA-ESRIN, Frascati, Italy

21 DECLIMS Vessel Classification Benchmark 21 Figure 1. Visual vessel size vs. real vessel size. Left column: Length. Right column: Width. Top line: all Standard images. Bottom line: all Fine images. Legend: Red + = HH, Blue v = VV, Green x = HV. Symbol size codes reliability (smaller is les reliable). Dotted line is y=x. Axis units are in meter.

22 DECLIMS Vessel Classification Benchmark 22 Figure 2. Partner system vessel size vs. real vessel size. Left column: Length. Right column: Width. Top line: all Standard images. Bottom line: all Fine images. Legend: Dotted line is y=x. Axis units are in meter. Colour codes polarisation, and symbol codes partner: Symbol Colour +: JRC Red = HH o: QQ Blue = VV : FFI Green = HV : GD Black = combined x: KSAT : BOOST : IAPro

23 DECLIMS Vessel Classification Benchmark 23 Figure 3. Partner system vessel size vs. visual vessel size. Left column: Length. Right column: Width. Top line: all Standard images. Bottom line: all Fine images. Legend as in Fig. 2.

24 DECLIMS Vessel Classification Benchmark 24 Figure 4 Legend. This figure occupies the next 16 pages. Each page shows the detections of all partners for a certain image. The bottom right plot combines all partner detections in one plot, the other seven plots show the detections of each partner separately. Black dot: Ground truth position Other symbol: Partner detection Colour of partner detection: Blue: Reliable detection, matched with a ground truth target Magenta: Unreliable detection, matched with a ground truth target Red: Reliable detection, not matched with a ground truth target In the seven separate partner plots, the symbols relate to polarisation: +: HH detection x: HV detection : VV detection o: Single detection combined from two polarimetric channels In the bottom right combined plot, the symbols identify the partner: +: JRC o: QQ : FFI : GD x: KSAT : BOOST : IAPro

25 DECLIMS Vessel Classification Benchmark 25 Figure 4 - Image 1, ship detections and associations.

26 DECLIMS Vessel Classification Benchmark 26 Figure 4 - Image 2, ship detections and associations.

27 DECLIMS Vessel Classification Benchmark 27 Figure 4 - Image 3, ship detections and associations.

28 DECLIMS Vessel Classification Benchmark 28 Figure 4 - Image 4, ship detections and associations.

29 DECLIMS Vessel Classification Benchmark 29 Figure 4 - Image 5, ship detections and associations.

30 DECLIMS Vessel Classification Benchmark 30 Figure 4 - Image 6, ship detections and associations.

31 DECLIMS Vessel Classification Benchmark 31 Figure 4 - Image 7, ship detections and associations.

32 DECLIMS Vessel Classification Benchmark 32 Figure 4 - Image 8, ship detections and associations.

33 DECLIMS Vessel Classification Benchmark 33 Figure 4 - Image 12, ship detections and associations.

34 DECLIMS Vessel Classification Benchmark 34 Figure 4 - Image 13, ship detections and associations.

35 DECLIMS Vessel Classification Benchmark 35 Figure 4 - Image 14, ship detections and associations.

36 DECLIMS Vessel Classification Benchmark 36 Figure 4 - Image 15, ship detections and associations.

37 DECLIMS Vessel Classification Benchmark 37 Figure 4 - Image 16, ship detections and associations.

38 DECLIMS Vessel Classification Benchmark 38 Figure 4 - Image 17, ship detections and associations.

39 DECLIMS Vessel Classification Benchmark 39 Figure 4 - Image 18, ship detections and associations.

40 DECLIMS Vessel Classification Benchmark 40 Figure 4 - Image 19, ship detections and associations.

41 DECLIMS Vessel Classification Benchmark 41 Figure 5 - Vessel visual size vs. real size. Left: length. Right: width. Images 1-3 (Standard). Legend as for Fig. 1.

42 DECLIMS Vessel Classification Benchmark 42 Figure 5 - Vessel visual size vs. real size. Left: length. Right: width. Images 4-6 (Standard).

43 DECLIMS Vessel Classification Benchmark 43 Figure 5 - Vessel visual size vs. real size. Left: length. Right: width. Images 7, 8 (Standard).

44 DECLIMS Vessel Classification Benchmark 44 Figure 5 - Vessel visual size vs. real size. Left: length. Right: width. Images 18, 19 (Standard).

45 DECLIMS Vessel Classification Benchmark 45 Figure 6 - Vessel visual size vs. real size. Left: length. Right: width. Images (Fine). Legend as for Fig. 1.

46 DECLIMS Vessel Classification Benchmark 46 Figure 6 - Vessel visual size vs. real size. Left: length. Right: width. Images (Fine).

47 DECLIMS Vessel Classification Benchmark 47 Legend of Figures 7-18 on the following pages. Dotted line is y=x. Axis units are in meter. Colour codes polarisation, and symbol codes partner: Symbol Colour +: JRC Red = HH o: QQ Blue = VV : FFI Green = HV : GD Black = combined x: KSAT : BOOST : IAPro

48 DECLIMS Vessel Classification Benchmark 48 Figure 7 - Images 1-6 (Standard), Vessel length from partner systems vs. real vessel length.

49 DECLIMS Vessel Classification Benchmark 49 Figure 7 - Images 7, 8, 18, 19 (Standard) Vessel length from partner systems vs. real vessel length.

50 DECLIMS Vessel Classification Benchmark 50 Figure 8 - Images (Fine), Vessel length from partner systems vs. real vessel length.

51 DECLIMS Vessel Classification Benchmark 51 Figure 9 - Images 1-6 (Standard), Vessel length from partner systems vs. visual vessel length.

52 DECLIMS Vessel Classification Benchmark 52 Figure 9 - Images 7, 8, 18, 19 (Standard) Vessel length from partner systems vs. visual vessel length.

53 DECLIMS Vessel Classification Benchmark 53 Figure 10 - Images (Fine), Vessel length from partner systems vs. visual vessel length.

54 DECLIMS Vessel Classification Benchmark 54 Figure 11 - Images 1-6 (Standard), Vessel width from partner systems vs. real vessel width.

55 DECLIMS Vessel Classification Benchmark 55 Figure 11 - Images 7, 8, 18, 19 (Standard) Vessel width from partner systems vs. real vessel width.

56 DECLIMS Vessel Classification Benchmark 56 Figure 12 - Images (Fine), Vessel width from partner systems vs. real vessel width.

57 DECLIMS Vessel Classification Benchmark 57 Figure 13 - Images 1-6 (Standard), Vessel width from partner systems vs. visual vessel width.

58 DECLIMS Vessel Classification Benchmark 58 Figure 13 - Images 7, 8, 18, 19 (Standard) Vessel width from partner systems vs. visual vessel width.

59 DECLIMS Vessel Classification Benchmark 59 Figure 14 - Images (Fine), Vessel width from partner systems vs. visual vessel width.

60 DECLIMS Vessel Classification Benchmark 60 Figure 15 All standard images combined: Vessel length from partner system vs. real vessel length (left column) and visual vessel length (right column). Partners 1, 2, 9.

61 DECLIMS Vessel Classification Benchmark 61 Figure 15 All standard images combined: Vessel length from partner system vs. real vessel length (left column) and visual vessel length (right column). Partners 15, 21, 25.

62 DECLIMS Vessel Classification Benchmark 62 Figure 15 All standard images combined: Vessel length from partner system vs. real vessel length (left column) and visual vessel length (right column). Partner 31.

63 DECLIMS Vessel Classification Benchmark 63 Figure 16 All standard images combined: Vessel width from partner system vs. real vessel width (left column) and visual vessel length (right column). Partners 1, 2, 9.

64 DECLIMS Vessel Classification Benchmark 64 Figure 16 All standard images combined: Vessel width from partner system vs. real vessel width (left column) and visual vessel length (right column). Partner 15, 21, 25.

65 DECLIMS Vessel Classification Benchmark 65 Figure 17 All Fine images combined: Vessel length from partner system vs. real vessel length (left column) and visual vessel length (right column). Partners 1, 2, 9.