GeoLAB. A geolocation algorithm testbed and benchmarking tool. Defence R&D Canada Ottawa. B.R. Jackson

Transcription

1 GeoLAB A geolocation algorithm testbed and benchmarking tool B.R. Jackson Defence R&D Canada Ottawa Technical Memorandum DRDC Ottawa TM June 2011

2

3 GeoLAB A geolocation algorithm testbed and benchmarking tool B.R. Jackson Defence R&D Canada Ottawa Defence R&D Canada Ottawa Technical Memorandum DRDC Ottawa TM June 2011

4 Principal Author Original signed by B.R. Jackson B.R. Jackson Approved by Original signed by M.W. Katsube M.W. Katsube Head/CNEW Section Approved for release by Original signed by C. McMillan C. McMillan Head/Document Review Panel c Her Majesty the Queen in Right of Canada as represented by the Minister of National Defence, 2011 c Sa Majesté la Reine (en droit du Canada), telle que représentée par le ministre de la Défense nationale, 2011

5 Abstract The Tactical Electronic Warfare Systems (TEWS) group within the Communications and Navigation Electronic Warfare (CNEW) section at DRDC Ottawa is focused on investigating and testing next-generation emitter geolocation techniques for the Canadian Forces. To that end, and as part of the Klondike Technology Demonstration Program (TDP), many geolocation algorithms have been developed using different techniques and several measurement campaigns have been conducted to produce data which could be used to validate the algorithms. Previously, each algorithm was tested independently, generally by the scientist that developed it, and each scientist had a different methodology and set of assumptions. This made comparing algorithm performance from different researchers very difficult. The GeoLocation Algorithm Benchmark and Testbed (GeoLAB) tool fills this gap by providing the ability to easily test new algorithms and fairly compare them with other approaches/algorithms in a single universal easy-to-use platform. GeoLAB, which was developed in MATLAB and has an easy-to-use graphical user interface, allows an arbitrary number of algorithms to be simultaneously run on a dataset (simulated or measured data) and provides many options as to how the calculations will be conducted and how the results will be displayed, including a Google Earth display option. It will be used to identify the most suitable geolocation method for the Klondike TDP and select the best performing algorithms for future field trials. Résumé Le groupe des systèmes de guerre électronique tactique (SGET) de la section Guerre électronique communications et navigation (GECN) de RDDC Ottawa se concentre sur l étude et la mise à l essai de techniques de géolocalisation d émetteurs de prochaine génération pour les Forces canadiennes. À cette fin, dans le cadre du Programme de démonstration de technologies (PDT) Klondike, de nombreux algorithmes de géolocalisation ont été développés au moyen de différentes techniques, et plusieurs campagnes de mesure ont été menées en vue de générer des données pour valider ces algorithmes. Auparavant, chaque algorithme était généralement évalué indépendamment par le scientifique qui l a élaboré, lequel possédait ses propres méthode et ensemble d hypothèses, ce qui rendait la comparaison des performances d algorithmes de différents scientifiques très ardue. Ce problème a été résolu grâce à l outil GeoLAB (GeoLocation Algorithm Benchmark and Testbed [Banc d essai et outil d analyse comparative d algorithmes de géolocalisation]), qui permet de facilement mettre à l essai de nouveaux algorithmes, puis de comparer ces derniers à d autres méthodes et algorithmes équitablement, au moyen d une seule plateforme universelle conviviale. GeoLAB, conçu dans MATLAB et doté d une interface graphique conviviale, permet d exécuter simultanément un nombre arbitraire d algorithmes à partir d un ensemble de donnes (simulées ou mesures) et offre un grand nombre de méthodes de calcul, ainsi que de façons d afficher les données, notamment une option DRDC Ottawa TM i

6 d affichage dans Google Earth. L outil servira à choisir la méthode de géolocalisation la plus appropriée pour le PDT Klondike et les algorithmes les plus performants pour des futurs essais sur le terrain. ii DRDC Ottawa TM

7 Executive summary GeoLAB: A geolocation algorithm testbed and benchmarking tool B.R. Jackson; DRDC Ottawa TM ; Defence R&D Canada Ottawa; June Background: Building upon the foundation laid by the Integrated Communications Electronic Warfare Analysis and RF Sensor (ICEWARS) Technology Demonstration Program (TDP), the Klondike TDP aims to enhance the current land forces emitter geolocation capabilities through the use of sensors on army assets not previously used for electronic warfare (EW). As part of this endeavour, new emitter geolocation techniques and algorithms need to be explored and a common testbed was required to make fair comparisons between competing methods. To this end, a testbed and benchmarking tool has been developed and is called GeoLAB (GeoLocation Algorithm Benchmark and Testbed). Principal results: The principal results are a fully-functional software tool developed in MATLAB that can be used to evaluate the performance of geolocation algorithms using simulated or measured data. The software tool, which consists of approximately 1,500 lines of MATLAB code, has an easy-to-use GUI and produces results in a very easy to understand and intuitive manner (including a Google Earth display option). Multiple geolocation algorithms can be tested simultaneously on the same set of data and conclusions can be reached regarding their relative performance (accuracy and required computation time). Significance of results: Using the GeoLAB tool, the TEWS group at DRDC Ottawa is now in an excellent position to efficiently explore numerous geolocation techniques/algorithms and select the most appropriate and optimal ones for use in the Klondike TDP, as well as future research projects on emitter geolocation. Since algorithms can now be compared in a fair and impartial manner, the best performing algorithms can be selected and further tested in field trials. Furthermore, since GeoLAB is not limited to a single geolocation technique, algorithms from different types of data can be compared (e.g. using angle-of-arrival, received signal strength, or two-channel interferometry) to not only determine the best algorithm, but also select an optimal technique that can be implemented in Klondike. GeoLAB has already proven to be very useful for algorithm performance characterization as evidenced by the production of a DRDC Technical Report and two scientific publications on received signal strength algorithms that relied upon GeoLAB to produce their results. DRDC Ottawa TM iii

8 Furthermore, GeoLAB has been provided to scientists at the Defence Science and Technology Organisation (DSTO) of Australia s Department of Defence through Subsidiary Arrangement (SA) No. 29 on Distributed Electronic Warfare Technology Collaboration under the Canada Australia Memorandum of Understanding (MOU) on Defence Science and Technology. Future work: While the current implementation of GeoLAB has all of the fundamental features required, there are several potential areas for future work. Currently, it is assumed that input data is entered correctly and there is no error-checking. Future version of GeoLAB could include code to ensure that appropriate inputs are used. For example, when non-positive numbers are entered for the frequency, a suitable and informative error message would appear. Furthermore, current error messages and some output data is displayed in the MATLAB command window. These messages could be sent to the GUI in future versions. The data file could also be pre-loaded (i.e. after the data file is selected, but before the Run button is pushed) and checked for errors and also provide options specific to that data file (e.g. select only a certain time range or only allow the user to select a channel number that actually exists in the data file). The Communications Research Centre s Spectrum Explorer software, which is generally used in TEWS to process the sensor data has the ability to assign GPS grids to the measured sensor data. For example, all of the measurements collected within a predefined 10 m 10 m cell could be tagged with an integer value representing this cell. A future addition to GeoLAB would be the ability to select only the measurements that were recorded within specified GPS grid cells. Lastly, the input data file type could be changed so that it is a native file type from a MiDAS Spectrum Explorer system. Currently, the data file produced by Spectrum Explorer is converted to a Microsoft Access database file, and then exported as a commadelimited (.csv) file from Microsoft Access. By reading in the data directly from the Spectrum Explorer output, this manual conversion step would be avoided and could open the possibility for GeoLAB to perform calculations directly on an output file produced by an unmanned, autonomous sensor. iv DRDC Ottawa TM

9 Sommaire GeoLAB: A geolocation algorithm testbed and benchmarking tool B.R. Jackson ; DRDC Ottawa TM ;R&Dpour la défense Canada Ottawa ; juin Contexte : Fond sur le PDT ICEWARS (Analyse intégrée de la guerre électronique, des communications et des capteurs de radiofréquence), le PDT Klondike a pour objet d améliorer la capacité de géolocalisation d émetteurs des forces terrestres au moyen de capteurs installés sur des biens de l armée auparavant non destinés à la guerre électronique. Pour ce faire, on devait étudier de nouvelles techniques et de nouveaux algorithmes de géolocalisation d émetteurs, ce qui a nécessité un banc d essai commun afin de comparer ceux ci de façon équitable. C est pourquoi GeoLAB, un banc d essai et un outil d analyse comparative, a été conçu. Résultats : Les recherches ont permis de créer dans MATLAB un outil informatique entièrement fonctionnel qui peut servir à évaluer les performances d algorithmes de géolocalisation à partir de données mesurées ou simulées. Constitué d environ 1500 lignes de code MATLAB, cet outil est doté d une interface graphique conviviale et génère des résultats de façon très simple et intuitive (notamment une option d affichage dans Google Earth). De nombreux algorithmes de géolocalisation peuvent être exécutés simultanément à partir du même ensemble de données, et des conclusions peuvent être tirées sur leurs performances respectives (précision et temps de calcul requis). Portée : Grâce à l outil GeoLAB, le groupe des SGET de RDDC Ottawa est maintenant dans une position des plus favorables pour étudier efficacement les nombreuses techniques et nombreux algorithmes de géolocalisation ainsi que pour choisir ceux qui conviennent le mieux au PDT Klondike et aux futurs projets de recherche sur la géolocalisation d émetteurs. Maintenant que les algorithmes peuvent être comparés de façon équitable, il est possible de choisir les plus performants, afin de les soumettre à des essais plus poussés sur le terrain. De plus, puisque GeoLAB ne se limite pas à une seule technique de géolocalisation, on peut comparer les algorithmes de différents types de données (p. ex. en utilisant l angle d incidence, la force du signal reçu ou l interférométrie à deux voies) non seulement pour trouver le meilleur algorithme, mais aussi pour choisir la technique optimale pouvant être mise en œuvre dans le cadre du PDT Klondike. GeoLAB s est déjà montré fort utile pour caractériser des performances des algorithmes, comme il a étédémontré dans un rapport technique de RDDC et deux publications scientifiques portant sur les algorithmes de puissance des signaux reçus, dont les résultats ont été DRDC Ottawa TM v

10 obtenus au moyen de GeoLAB. En outre, des exemplaires de GeoLAB ont été remis aux scientifiques de la Defence Science and Technology Organisation (DSTO) [organisation des sciences et de la technologie de la Défense] du ministère de la Défense de l Australie dans le cadre de l arrangement subsidiaire no 29 sur la collaboration technologique décentralisée en matière de guerre électronique, et selon le protocole d entente (PE) entre le Canada et l Australie sur les sciences et la technologie de la Défense. Recherches futures : Bien que la version présente de GeoLAB comporte toutes les fonctions requises, plusieurs domaines de recherches demeurent inexplorés. Actuellement, on tient pour acquis que les données sont entrées correctement et qu il n y a aucune vérification des erreurs. On pourrait intégrer du code à une version future de GeoLAB en vue de s assurer que les bonnes données sont utilisées. Par exemple, lorsqu on saisit un nombre négatif comme fréquence, un message d erreur pertinent et informatif pourrait s afficher. De surcroît, les messages d erreur actuels et certains résultats sont affichés dans la fenêtre de commande de MATLAB. Dans une version future de GeoLAB, ces messages pourraient être acheminés à l interface graphique. Le fichier de données pourrait être préinstallé (c.- à-d. une fois le fichier choisi, mais avant d avoir cliqué sur le bouton Run ), avoir fait l objet d une vérification des erreurs et fournir également des options qui lui sont propres (p. ex. choisir seulement une certaine échelle de temps ou permettre à l utilisateur de choisir uniquement un numéro de canal qui existe dans le fichier). L Explorateur de spectre (ES), élaboré par le Centre de recherches sur les communications, habituellement utilisé par les SGET pour traiter les données de capteur, peut attribuer des quadrillages GPS aux données de capteur mesurées. Par exemple, toutes les mesures effectuées dans une cellule prédéfinie de 10 m sur 10 m pourraient être étiquetées avec une valeur entière représentant cette cellule. Une future version de GeoLAB pourrait permettre de choisir uniquement les mesures enregistrées dans des cellules de quadrillages GPS précises. Enfin, le type de fichier de données d entrée pourrait être modifié afin qu il soit natif du système d exploration du spectre MiDAS. Actuellement, le fichier de données généré par l explorateur de spectre est converti en fichier de base de données Microsoft Access, puis exporté en tant que fichier comma-delimited (délimité par un point) (.csv) de Microsoft Access. Si la lecture des données pouvait se faire sans intermédiaire à partir de la sortie de l explorateur de spectre, cette étape de conversion manuelle serait éliminée ce qui permettrait peut-être à GeoLAB de faire des calculs soumettre à des calculs directs un fichier de résultats produit par un capteur autonome. vi DRDC Ottawa TM

11 Table of contents Abstract... i Résumé... i Executive summary... Sommaire... iii v Table of contents... vii List of figures... Acknowledgements... ix x 1 Introduction Motivation Goals GeoLAB description and capabilities System requirements and GeoLAB files GeoLAB algorithm interface Algorithm inputs from GeoLAB Algorithm outputs to GeoLAB Selecting algorithm/data files and data options Algorithm and data files Data options Settings Results display options Sample results and comments Conclusions References DRDC Ottawa TM vii

12 Annex A: Sample algorithm for use with GeoLAB Annex B: Sample data file for use with GeoLAB List of abbreviations/acronyms viii DRDC Ottawa TM

13 List of figures Figure 1: The GeoLAB graphical user interface (GUI) Figure 2: The GeoLAB GUI File Options Figure 3: The GeoLAB GUI Data Options Figure 4: The GeoLAB GUI Settings Figure 5: The GeoLAB GUI Results Display Options Figure 6: GeoLAB results displayed in MATLAB Figure 7: GeoLAB results displayed in Google Earth Figure 8: Geolocation estimate information included by GeoLAB in Google Earth. 20 Figure 9: Sensor information included by GeoLAB in Google Earth Figure 10: GeoLAB s estimation of the actual emitter position Figure 11: Google Earth time slider used for real-time simulation (on-the-fly) option. 22 Figure 12: Example of the Plot RMSE vs. # of Sensors option output (from [1]) Figure 13: Example of the Plot Histogram of Results option output DRDC Ottawa TM ix

14 Acknowledgements The author of this Technical Memorandum would like to thank the members of the Tactical Electronic Warfare Systems group for their participation in the collection of experimental data used to test and evaluate GeoLAB. In addition, Dr. Shanzeng Guo and Ms. Jenny Chuang deserve specific recognition for their many constructive comments and feedback during the development of this software tool. x DRDC Ottawa TM

15 1 Introduction The Tactical Electronic Warfare Systems (TEWS) group within the Communications and Navigation EW (CNEW) section at DRDC Ottawa is investigating new geolocation techniques and algorithms in support of the Klondike Technology Demonstration Program (TDP). Several scientists are involved in the geolocation algorithm development and testing and the need for a universal testing platform quickly arose as more and more measured datasets became available and multiple algorithms emerged as candidates for implementation in Klondike. To this end, a software tool called GeoLocation Algorithm Benchmark and Testbed (GeoLAB) was conceived as a solution to this problem. Since MATLAB is the platform used by TEWS researchers to develop geolocation algorithms, it was the obvious choice for the development of GeoLAB. In total, GeoLAB consists of approximately 1,500 lines of MATLAB code. The graphical user interface (GUI) of GeoLAB, shown in Figure 1, provides many user options regarding the data that will be made available to the algorithms, as well as the displaying of results. GeoLAB has already proven to be very useful for algorithm performance characterization as evidenced by [1 3], which are a DRDC Technical Report and two scientific publications on received signal strength algorithms that all used GeoLAB to produce their results. Furthermore, GeoLAB has been provided to scientists at the Defence Science and Technology Organisation (DSTO) of Australia s Department of Defence through Subsidiary Arrangement (SA) No. 29 on Distributed Electronic Warfare Technology Collaboration under the Canada Australia Memorandum of Understanding (MOU) on Defence Science and Technology. As will be detailed in this document, the results can be displayed using either built-in - MATLAB plotting functions or in Google Earth. The use of Google Earth to display the results can provide a very intuitive understanding of the geolocation accuracy, particularly when data from actual field experiments is used. The positions and signal power measurements associated with the sensors and the locations of the sensors with respect to the emitter and local obstructions, such as trees and buildings, or other obstructions are all easily discerned. Inferences can be readily drawn as to the effect of terrain features on the sensor measurements and how these affect the geolocation estimates and circular error probable (CEP). Furthermore, given Google Earth s global map coverage, results from sensor measurements taken anywhere in the world can be easily viewed by simply tagging the data with latitude and longitude information. DRDC Ottawa TM

16 Figure 1: The GeoLAB graphical user interface (GUI). 2 DRDC Ottawa TM

17 1.1 Motivation Prior to the development of GeoLAB, geolocation algorithms were tested independently, generally by the scientist that developed them. Since scientists often have different methodologies and assumptions it was very difficult to compare algorithm performance from different researchers, which could result in a suboptimal algorithm selection for the particular situation. The motivation for GeoLAB was the desire to fill this gap and provide the ability to easily test new algorithms using measured or simulated data and fairly compare different algorithms on a single universal platform. 1.2 Goals The goal of this work was to develop a flexible and capable software research tool for the evaluation and comparison of different geolocation algorithms and techniques in support of the Klondike TDP. The ability to quickly and easily compare the results of algorithms with different datasets was desired in order to select the best performing algorithm for the Klondike TDP. Specific capabilities such as the ability to view results in both MATLAB as well as Google Earth were desirable and the ability to have some control over the data that the algorithms have to compute their position fix. In most cases in the past, algorithms were tested in a batch processing mode, i.e., the algorithms were supplied all of the data contained in the data set at once. Another goal of GeoLAB was to implement the ability to simulate how the algorithms will perform in the realistic case where they have access to new data as it is collected. Therefore, GeoLAB was developed to have the ability to incrementally supply the algorithms with new data to see how the algorithm results converge on an emitter as a sensor acquires new data. Furthermore, the ability to play back the results was desirable in various speeds ranging from real-time to many times faster than real-time. A final goal was to be able to accept multiple data files and perform preprocessing and chronological sorting before passing a single combined dataset to the algorithms. This increases the relevance and flexibility of the tool as future experiments and field trials may include multiple sensors, each producing a distinct data file. DRDC Ottawa TM

18 This page intentionally left blank. 4 DRDC Ottawa TM

19 2 GeoLAB description and capabilities 2.1 System requirements and GeoLAB files GeoLAB was written and tested primarily on a PC running Microsoft Windows 7 using MATLAB version 7.1 (R2010a). GeoLAB was intentionally developed to not require any special toolboxes developed for MATLAB by MathWorks, and only requires a basic MATLAB installation. However, in order to use Google Earth to display GeoLAB results, the Google Earth Toolbox for MATLAB is required and is available as a free download [4]. While all computations and functionality within GeoLAB will work properly on any PC with at least a 1 GHz processor, 1 GB of RAM, 1 GB of free disk space, and MATLAB version 7 or higher, there may be minor formatting issues with the GUI on some platforms (e.g., if GeoLAB is run on in MATLAB on a Apple Mac OS X or Linux-based computer there may be minor GUI issues). This issue has not been found on any Windows-based MATLAB installation where GeoLAB was tested. In order to use GeoLAB s advanced results display capabilities in Google Earth, a local installation of Google Earth is necessary (available for free at [5]). GeoLAB does not have any specific computer hardware requirements and should be easily run by any computer capable of having a recent version of MATLAB installed. The computational requirements will generally be dictated by the size of the data file being used by GeoLAB as well as the computational complexity of the algorithms under test. GeoLAB consists of several files: GeoLAB GUI.m: the MATLAB m-file that sets up and controls the GUI, and passes the input data to GeoLAB.m. GeoLAB GUI.m is the file to run to start GeoLAB. GeoLAB GUI.fig: the MATLAB figure file for the GeoLAB GUI. GeoLAB.m: the main GeoLAB MATLAB file where the data processing occurs, algorithms are called, and output is generated. Info.fig: a MATLAB figure file contains general information about GeoLAB and is activated when the About... button is clicked on the GeoLAB GUI. images subfolder: this folder contains the two images used in the GeoLAB GUI, the DRDC logo (DRDC.jpg) and the GeoLAB title (GeoLAB.jpg). When Google Earth is selected to display the results, GeoLAB produces the output file GeolocationResult.kml and automatically opens this file in Google Earth. DRDC Ottawa TM

20 2.2 GeoLAB algorithm interface GeoLAB was developed for the primary purpose of comparing a variety of algorithms using measured or simulated data. For example, GeoLAB was designed so that an algorithm using power difference of arrival could be compared simultaneously with an angle-of-arrival algorithm and a two-channel interferometry algorithm (as well as combinations of different techniques used in conjunction with each other). To this end, GeoLAB can read in many parameters from a data file and provides this data to the algorithms irrespective of whether or not this data is actually used in the algorithms. The GeoLAB Algorithm interface will continue to evolve as requirements, equipment, and end-goals change. However, the interface as of this writing contains the major input and output elements, each of which will now be briefly described Algorithm inputs from GeoLAB In this subsection, the parameters that are available to the algorithms will be described. Most of these parameters consist of the data available from the Communications Research Centre s Spectrum Explorer software, which is used by TEWS scientists to process the data measured by the sensors. Frequency The frequency of the signal of interest (MHz). In future versions of GeoLAB this value can be automatically determined by using the Channel ID described below along with the channel definition in the data file. Currently, the frequency parameter is used primarily to calculate the wavelength, which is use in the two-channel interferometer algorithms. Detection ID A counter than increments each time a signal is detected above the specified threshold. Channel ID An integer value that is specified in the data file that corresponds to a specific signal frequency. Power The power of the signal (dbm). Angle-of-Arrival/Phase Difference This parameter is either the angle-of arrival or the phase difference, depending on the type of sensor (degrees). Direction/Phase Quality A measure of the quality of the AOA/phase difference measurement (degrees). 6 DRDC Ottawa TM

21 Sensor Heading The direction that the sensor platform is moving (degrees). Currently, GeoLAB computes the heading using successive GPS coordinates, however, in the future the heading will be measured using a vector GPS. Sensor Status An ID code that represents the sensor status and can be used to determine if the sensor is functioning properly. Time A time stamp for each measurement. GPS Status ID The status of the GPS (can be used to determine if the GPS is locked or not). Sensor x-coordinate A relative x-coordinate of the sensor location (m). GeoLAB converts the latitude and longitude information from the sensor into a relative coordinate system for ease of use by the algorithms. Sensor y-coordinate A relative y-coordinate of the sensor location (m). GeoLAB converts the latitude and longitude information from the sensor into a relative coordinate system for ease of use by the algorithms. Elevation The elevation of the sensor (m). GPS Grid ID A GPS grid can be configured during measurement set up and each grid has an ID. Area of Interest The area of interest in X (m) + j*y (m) Algorithm State For multiple iterations, the algorithms can save a state variable to save computation time on the next iteration. As requirements develop new inputs will be added. An example of an input that will be added is Sensor ID, which will be a parameter that represents the type of sensor (e.g. power sensor, angle-of-arrival sensor, two-channel interferometer sensor, etc.). DRDC Ottawa TM

22 2.2.2 Algorithm outputs to GeoLAB Transmitter Estimate The algorithm s estimate of the transmitter location in X (m) + j*y (m) relative coordinates. GeoLAB converts this to latitude and longitude for plotting in Google Earth. Circular Error Probable The radius of the circular error probable (m). If a CEP is not calculated by the algorithm a negative value can be assigned (e.g., CEP = -1;) Algorithm State For multiple iterations, the algorithms can save a state variable to save computation time on the next iteration. This parameter does not have to be assigned. There will be additions to this output schema in the future including the additional parameters required for an elliptical error probable. 2.3 Selecting algorithm/data files and data options Algorithm and data files Outlined in red in Figure 2 is the area of the GeoLAB GUI where the algorithm(s) and data file(s) are selected. The algorithm files used by GeoLAB are MATLAB.m-files, an example which is presented in Annex B. The data file(s) used by GeoLAB are commadelimited text files with the headers included on the first line (a short example is presented in Annex C). An unlimited number of algorithms can be selected so that their resulting performance can be easily compared. Furthermore, an unlimited number of separate data files can be used if multiple sensors are available. GeoLAB internally combines multiple data files into one data set and sorts in chronological order before providing the data to the algorithms. In order to select the algorithm and data file(s), the Browse button is clicked in the highlighted area in Figure 2. If multiple files are to be chosen then the Shift or Ctrl key on the keyboard should be held down while the files are selected with the mouse. The order of the data in the comma-delimited text file does not matter as long as the headers are included and specified properly. GeoLAB looks for specific headers and assigns the data in that column to the appropriate variable. The headers that are searched for include: Detection Measurement ID Channel ID Power dbm Direction Deg Direction Quality Deg Status Time 8 DRDC Ottawa TM

23 Figure 2: The GeoLAB GUI File Options. DRDC Ottawa TM

24 Milliseconds GPS Status ID Longitude Deg Decimal Longitude Deg Min Latitude Deg Decimal Latitude Deg Min Elevation m GPS Grid ID As new data types are included in the measured (or simulated) files, new headers will have to be added. Furthermore, if the current data s headers are changed in the data files then GeoLAB will also need to include these updated header definitions Data options The Data Options area of GeoLAB (highlighted in Figure 3) provides options for what data should be included for the algorithms to be used. Each of the three options will be described below: Use All Data If this option is selected then all of the data is used and is available for the algorithms subject to the constraint that the sensor to emitter separation must be greater than the Minimum Tx-to-Sensors Separation value specified in the Settings. Decimate The Decimate option allows a decimation factor to be specified. For example, if a Decimation Factor of 100 is specified then the only data available to the algorithms will be the data points: 1, 101, 201, 301, etc. This option is very useful to reduce the computation time with large data sets. The default value for Decimation Factor is 1,000. Randomly Choose Data The final option is Randomly Choose Data, which was included to provide Monte Carlo capabilities and statistical results with a specified number of sensors. When this option is selected there are three additional fields that become active. The first is Number of Sensors, which can be an array of sensor numbers (multiple values separated by commas). Next, Number of Runs specifies the number of iterations that GeoLAB will perform and over which the results will be averaged. Finally, Minimum Sensor Separation provides the ability to specify the minimum distance that sensors must be separated in order to be included in the dataset supplied to the algorithm(s). Essentially, GeoLAB will randomly choose a number of data points in the data set that corresponds to the Number of Sensors and ensure that their separation is greater than the Minimum Sensor Separation (selecting new values until this condition is met). Using this data, a geolocation estimate will be made 10 DRDC Ottawa TM

25 Figure 3: The GeoLAB GUI Data Options. DRDC Ottawa TM

26 by the algorithm(s), and then a new dataset will be generated in the same way up to the number of times specified by Number of Runs. These results are averaged to produce an average miss distance. 2.4 Settings Shown in Figure 4 is the Settings area of the GeoLAB GUI highlighted. There are several input areas in this area that will now be described. Figure 4: The GeoLAB GUI Settings. Frequency The frequency of the signal of interest in MHz. This should correspond to the frequency specified in the data file for the Channel Number input below. The default value is 462 MHz (in the FRS band). Area of Interest This is the size of the AOI in X (m) Y (m). The area of interest should be larger than the area of which measurements were made. For some algorithms, the estimated transmitter location will be constrained to this AOI, so it should be large enough to include all 12 DRDC Ottawa TM

27 reasonable potential emitter locations. However, the computation time of many algorithms depend critically on the size of this AOI, and a large area will increase the computation time dramatically. The default value is 3 km 3 km. Minimum Tx-to-Sensor Separation During measurement collection, the sensor may come very close to the transmitter. Since this situation is very unlikely for electronic warfare emitter geolocation applications, an option has been included in GeoLAB to remove these measurements. Any measured values that are within the distance specified in this field from the actual transmitter location will be excluded and will not be available for the algorithms. The minimum sensor-to-sensor separation can only be selected if Randomly Choose Data is chosen in the Data Options. The default value for Minimum Tx-to-Sensor Separation is 100 m. Channel Number This is the channel number of the transmitted signal to be geolocated. During the set up and configuration of Spectrum Explorer, the channel numbers and corresponding frequencies are manually set. This allows, for example, multiple targets (emitters) to be deployed at various locations, each transmitting at a different frequency with a corresponding channel number. In this case, a sensor can record measurements from all targets simultaneously tagged by channel number, and the algorithms can be tested against each emitter separately. The default value for this parameter is: 1. Automatically Determine Actual Tx Location In order to determine which algorithm is the most accurate, the actual transmitter location, or ground-truth, must be known. If Yes is selected for this option, a simple algorithm is used to determine the actual emitter location. As mentioned previously, in many cases during measurements the sensor comes very close to the emitter (often within 10 m). While these close-in measurements are generally discarded for the algorithms to maintain practicality for EW applications, these measurements can still be very useful. To calculate the algorithm estimation error for each emitter, the actual transmitter location is estimated by GeoLAB by using the location with the highest recorded power measurement when the Yes option is selected. If the data collection includes measurements that come within approximately 10 m of the transmitter at some point then this generally produces an acceptable location. Alternatively, if No is selected, the latitude and longitude of the actual emitter location must be entered for each transmitter (in decimal degrees), which may be cumbersome to determine if there are many measurement data files and/or multiple emitters. Note that if Yes is selected for this option, it is not possible to enter coordinates into the Latitude and Longitude fields (they are greyed-out) since they are not required or used. The default value is Yes. Simulate Real-Time System (on-the-fly) In the past, during algorithm development, the entire data file was generally processed and used by the algorithm all at once. However, in an actual implementation it would DRDC Ottawa TM

28 be desirable to have algorithms produce fixes in real-time as the data is streaming in. To simulate this, there is an option in GeoLAB to simulate a real-time system (or on-thefly) estimates. If No is selected for this option (which is the default), the algorithms are provided with all of the data at once, and a single fix is produced by each algorithm. However, if Yes is selected for this option the algorithms are given one new data point along with all previous data points during each iteration until all the data has been provided. In this case, a new fix is produced each time the algorithm is called, which is equal to the number of data points. A great benefit to this option is the ability to see how the emitter geolocation estimates converge on a target as more measurements are acquired. In order to visualize this, the time stamp information is used to produce a dynamic Google Earth file that includes a time slider with a Play option. Using this GeoLAB option, results can be played back in quasi-real-time as if the algorithms were running and producing fixes as the sensors were collecting data (with the benefit of being able to speed up time to quickly see how the results converge). 2.5 Results display options The final area of the GeoLAB GUI is highlighted in Figure 5, Plotting Options. This area provides several Yes or No options for displaying the algorithm results. The first two options control whether the output is plotted in Google Earth, MATLAB, or neither. If the results are to be shown in Google Earth, GeoLAB will generate a.kml file (the native Google Earth file type) and automatically open this file upon finishing calculations (required that Google Earth is installed). The free Google Earth Toolbox for MATLAB [4] is used to generate this.kml file. If the Plot in MATLAB option is chosen then GeoLAB will use all of the sensor positions in the data file to draw the roads that were travelled during measurement collection. The next four options dictate what is to be included in the Google Earth and/or MATLAB results window. If Yes is selected for Plot the Actual Tx an antenna icon will be included in the display to represent the ground truth. If Plot the CEP is selected then the circular error probable calculated by the algorithm(s) will be shown in the results. If Plot Sensors is selected then the display window will include an icon representing the sensor positions. In the case where there are many sensors measurements then it may be desirable to select No for this option. Finally, there is an option to Print Sensor Powers which will print the measured power for each sensor on the display. This can be useful, for example, in combination with the Google Earth display where you can see the terrain and buildings along with sensor power levels and see how this affects the emitter geolocation estimate. The final two options are only available if Randomly Choose Data is selected in Data Options. When multiple runs are specified with various numbers of sensors, Plot RMSE vs. # of Sensors will produce a plot that shows the average miss distance as the numbers of sensors varies. The last option, Plot Histogram of Results creates a histogram that shows 14 DRDC Ottawa TM

29 Figure 5: The GeoLAB GUI Results Display Options. DRDC Ottawa TM

30 the distribution of the miss distances for multiple runs. Consider the situation where an algorithm produces an excellent fix half the time and very wild fixes the other half. Using the histogram plotting option will show this very uneven distribution, which could lead to further investigation as to the cause of the poor fixes. 16 DRDC Ottawa TM

31 3 Sample results and comments In this section, examples will be provided of GeoLAB s capabilities and output. Shown in Figure 6 is the MATLAB results figure for one run of randomly selected data with 8 sensors separated by at least 100 m. The GeoLAB set up used to generate this result was as follows: Randomly Choose Data selected with Number of Sensors: 8 Number of Runs: 1 Minimum Sensor Separation: 100 m. Plot in MATLAB: Yes Plot the Actual Tx: Yes Plot the CEP: Yes Plot Sensors: Yes Print Sensor Powers: No Plot RMSE vs. # of Sensors: No Plot Histogram of Results: No The data used to generate this figure, as well as all of the figures in this section was collected using a single mobile sensor vehicle and a continuous wave (CW) transmitter on May 4, 2010 at the DRDC Ottawa Shirley s Bay campus (see [1] for more details on the sensor and how the measurement data was collected). As shown in Figure 6, the black lines represent the roads in and around the DRDC Ottawa campus, which were generated by using all sensor positions in the data (i.e. the mobile sensor platform was driven everywhere there is a black line). Alternatively, a separate data file could be used to generate the roads if the sensor(s) did not traverse the roads in the AOI. The sensor positions, target emitter, and geolocation estimates are also shown in Figure 6 along with the CEP for one of the estimates (in this case only one algorithm actually computed a CEP). Note that the legend includes the miss distance for each algorithm and informative text is printed on the figure so that the configuration used to generate the results is recorded. For the same computation that generated Figure 6, Google Earth was also selected to view the results (i.e. Plot in Google Earth: Yes) and the.kml file that GeoLAB produced is shown in Google Earth in Figure 7. Comparing Figures 6 and 7 it can be seen that the results are the same, but Google Earth provides a display with rich satellite imagery. GeoLAB uses light armoured vehicles (LAVs) icons to represent the sensor locations, an antenna icon to represent the transmitter location, and coloured dots to represent each algorithm s estimation of the transmitter position. From this display, the potential shadowing effects of buildings, trees, etc. can be intuitively seen and their effect on the geolocation accuracy of each algorithm can be explored. GeoLAB embeds information into the.kml file that describes the configuration that produced the results (however, the specific data points used to produce the results are not DRDC Ottawa TM

32 Figure 6: GeoLAB results displayed in MATLAB. 18 DRDC Ottawa TM

33 Figure 7: GeoLAB results displayed in Google Earth. DRDC Ottawa TM

34 saved). Figure 8 shows a pop-up display for the NLS POA algorithm estimation, which is obtained by clicking on the algorithm s estimate icon (the red dot in this case). This informative display includes information such as the algorithm name, the miss distance, the CEP radius, the minimum separation from the transmitter and from other sensors, as well as the computation time required to produce the estimate using this algorithm. Figure 8: Geolocation estimate information included by GeoLAB in Google Earth. Similarly, by clicking on a sensor icon in the Google Earth display an informative balloon window appears as shown in Figure 9. This display shows the received power by that sensor, the angle of arrival or phase difference (depending on sensor type), and the latitude and longitude of the sensor. If the option Plot Sensor Powers was chosen in GeoLAB then each sensor would display the received signal strength in dbm next to the sensor without having to click on the sensor icon to see this information. As discussed previously, the Automatically Determine Actual Tx Location option in Geo- LAB looks at all of the data in the specified data file(s) and finds the maximum power level. The sensor location that recorded this maximum received signal strength is then used as the 20 DRDC Ottawa TM

35 Figure 9: Sensor information included by GeoLAB in Google Earth. actual transmitter position in order to compute miss distances and plot the actual emitter on the results display(s). Figure 10 shows where GeoLAB has determined the actual transmitter to be. During the measurement, the emitter was set up approximately 5 m to the east of GeoLAB s calculated position shown in Figure 10 (on the short stub at the corner of the road). Since most of the geolocation estimates have errors that are much greater than 5 m, GeoLAB s approximation is very acceptable in this case and eliminates the need for a user to manually enter latitude and longitude coordinates. However, it is very important to note how this approximation is made since it could produce very inaccurate results with different datasets or if GeoLAB was used for a different application. When the Simulate Real-Time System (on-the-fly) option is selected, the results can only be viewed in Google Earth and not MATLAB. Google Earth has advanced dynamic display capabilities that allows the results of the simulated real-time system to be viewed effectively, whereas MATLAB does not. Using the time slider in Google Earth (shown in Figure 11) the results can be played back with control over time. The time slider can instantly be moved to any point in the measurement campaign and all results over the entire time duration can be overlaid if desired. Additionally, there are options for the speed of playback (anywhere from real-time to many times faster). In fact, when viewed in Google Earth using the GeoLAB generated.kml file, the mobile sensor(s) are shown in Google Earth moving along the roads in the exact same way they did during the actual measurement and DRDC Ottawa TM

36 Figure 10: GeoLAB s estimation of the actual emitter position. Figure 11: Google Earth time slider used for real-time simulation (on-the-fly) option. 22 DRDC Ottawa TM

37 new geolocation estimates are continually being produced and displayed. For example, a measurement campaign that was one hour in duration can be played back and increased in speed (anywhere from real-time to only seconds in duration). The movement of the mobile sensor(s) and the convergence of the geolocation estimates can be easily seen as well as the reduction in CEP radius as more measurements are available to the algorithm(s). The final two plotting options are only available when Randomly Choose Data is selected under Data Options. Shown in Figure 12 is the root mean square error (RMSE) plotted against the number of sensors from 3 to 12 (from [1]). For each number of sensors there were 1,000 runs and the results were averaged. The inputs under Randomly Choose Data were as follows: Number of Sensors: 3,4,5,6,8,10,12 Number of Runs: 1000 Minimum Sensor Separation: 100 m while the relevant inputs under Plotting Options were: Plot in Google Earth: No Plot in MATLAB: No Plot RMSE vs. # of Sensors: Yes Plot Histogram of Results: No For example, for 10 sensors, there were 1,000 runs of 10 randomly selected sensors, and the algorithm s miss distances for each run were averaged to produce a single RMSE for each algorithm. Shown in Figure 13 is a histogram of results for 1,000 runs with 15 sensors. The inputs under Randomly Choose Data were as follows: Number of Sensors: 15 Number of Runs: 1000 Minimum Sensor Separation: 100 m with the relevant inputs under Plotting Options: Plot in Google Earth: No Plot in MATLAB: No Plot RMSE vs. # of Sensors: No Plot Histogram of Results: Yes With this plot it is easy to see the distribution of miss distances and evaluate algorithm performance. DRDC Ottawa TM

38 Average Error Distance (m) NLS/DPD/ML 200 Intersection Density Y.T. Chan s LS 100 S. Wang s LS L. Zhu s LS Number of Sensors Figure 12: Example of the Plot RMSE vs. # of Sensors option output (from [1]). 24 DRDC Ottawa TM

39 Number of Occurrences Error Distance (m) Figure 13: Example of the Plot Histogram of Results option output. DRDC Ottawa TM

40 This page intentionally left blank. 26 DRDC Ottawa TM

41 4 Conclusions A new tool has been developed for evaluating geolocation techniques and algorithms. This tool, called GeoLAB, was developed using MATLAB and was designed to be highly capable and flexible. It can use either measured or simulated data and can simultaneously run multiple algorithms and provides several options for comparing results. More specifically, the results can be displayed in MATLAB, or, if the free Google Earth software is available on the computer running GeoLAB then the rich Google Earth display can be used. The sensor positions, actual antenna position, and geolocation estimates are overlaid on the satellite imagery in Google Earth, which provides significant insight into the effects of terrain and obstructions such as buildings on the accuracy of an algorithm. Furthermore, GeoLAB embeds information within the Google Earth.kml file so that these files can be shared and contain all of the information regarding the configuration used to produce the results. An advanced feature of GeoLAB is its ability to simulate a real-time environment by producing geolocation estimates on-the-fly by providing the algorithms with one new data point each iteration. For each new data point the algorithm(s) produce a new fix and Geo- LAB creates a Google Earth file that can play these back in real-time or faster, allowing the convergence of the geolocation result to be witnessed. By using this feature, it is possible to simulate the algorithm producing fixes in real-time as the sensor is measuring the signal. In summary, GeoLAB will provide an ideal platform upon which to test new algorithms and compare geolocation techniques (e.g. power-based techniques with those using twochannel interferometry). With the development of GeoLAB, a missing link in the geolocation research performed by the Tactical EW Systems group at DRDC Ottawa has been realized. Researchers within TEWS now have the ability to fairly compare algorithms and make independent assessments of which algorithm(s) and geolocation techniques will maximize the success of the Klondike TDP. DRDC Ottawa TM

42 This page intentionally left blank. 28 DRDC Ottawa TM

43 References [1] Jackson, B.R., Wang, S., and Inkol, R. (2011), Emitter geolocation estimation using power difference of arrival: an algorithm comparison for non-cooperative emitters (U), (DRDC Ottawa TR ) Defence R&D Canada Ottawa. [2] Jackson, B.R., Wang, S., and Inkol, R. (2011), Received signal strength difference emitter geolocation least squares algorithm comparison, In IEEE Canadian Conference on Electrical and Computer Engineering (CCECE). [3] Wang, S., Jackson, B.R., and Inkol, R. (2011), Impact of emitter-sensor geometry on accuracy of received signal strength based geolocation, In IEEE Vehicular Technology Conference (VTC20111-Fall). [4] Google Earth Toolbox for MATLAB (online), Google, (Access Date: November 2010). [5] Google Earth (online), Google, (Access Date: April 2011). DRDC Ottawa TM

44 This page intentionally left blank. 30 DRDC Ottawa TM

45 Annex A: Sample algorithm for use with GeoLAB %======================================================================= % Brad Jackson, Ph.D. % Sichun Wang, Ph.D. % DRDC Ottawa % Communication and Navigation Electronic Warfare % brad.jackson@drdc -rddc.gc.ca % A non-linear least squares power of arrival algorithm %======================================================================= function [TXNLS,CEP,Q] = NLS_POA(freq,Det_ID_Alg,Chan_Meas_ID_Alg,... Chan_ID_Alg,pwrsdBm,AOA_PhaseDiff,DirQual,heading,Status_Alg,... Time_Alg,Millisec_Alg,GPS_Status_ID_Alg,RxX,RxY,Elevation_Alg,... GPS_Grid_ID_Alg,AOI,state) RX = RxX + j*rxy; gamma = 3.75; Δ _x = 50; % Grid size in the x coordinate Δ _y = 50; % Grid size in the y coordinate direction direction % Generation of grid used for NLS n_x = ceil(real(aoi)/ Δ _x); n_y = ceil(imag(aoi)/ Δ _y); Q = zeros(1+n_x,1+n_y); mygrid = Q; for kk = 1:(n_x+1) for ll = 1:(n_y+1) mygrid(kk,ll) = (kk-1)* Δ _x+j*((ll-1)* Δ _y); end end %simulate geolocation min_q = eps(realmax); for Pt = 20:1:33 for alpha = gamma Q = compute_q_function_on_a_grid(rx,pt,pwrsdbm,gamma,mygrid); temp = min(q); end if min_q > min(temp) Ptx = Pt; [temp_x, column_index_vector] = min(q); [min_q,i] = min(temp_x); end DRDC Ottawa TM

46 end index_kk = column_index_vector(i); index_ll = I; TXNLS = real(mygrid(1))+(index_kk -1)* Δ _x+j*( imag(mygrid(1))+... (index_ll -1)* Δ _y); x = real(txnls); y = imag(txnls); sigma = 8; m = 1; sensors = [real(rx);imag(rx)]; [r,c] = size(sensors); s = sensors; for k = 1:r xk = s(k,1); yk = s(k,2); d(k) = sqrt((x-xk)ˆ2+(y-yk)ˆ2); phi(k) = atan2(y-yk,x-xk); tau(k) = cos(phi(k))/d(k); rho(k) = sin(phi(k))/d(k); end vector = tau-tau(m); term1 = mean(vector.ˆ2); vector = rho-rho(m); term2 = mean(vector.ˆ2); term3 = mean(tau-tau(m)); term4 = mean(rho-rho(m)); term5 = mean( (tau-tau(m)).*(rho-rho(m))); numerator = term1+term2-term3ˆ2-term4ˆ2; denominator = (term1-term3ˆ2)*(term2-term4ˆ2)-(term5-term3*term4)ˆ2; ratio = numerator/denominator; c = sigma*log(10); c = c/10; c = c/gamma; c = c/(r).ˆ0.5; CEP = c*sqrt(ratio); end Q = sqrt(ratio); if CEP > real(aoi) isnan(cep) CEP = real(aoi); end function z=compute_q_function_on_a_grid(rx,pt,pwrsdbm,gamma, mygrid) %compute_q_function.m 32 DRDC Ottawa TM

47 %RX: receiver locations represented as a nx2 matrix; % pwrsdbm: vector storing the RSS at the n receivers; % (x, y): prospective location of transmitter; z=zeros(size(mygrid)); n=length(rx); for k=1:n TxSens = sqrt(abs(mygrid-rx(k)).ˆ2)+eps; Ppred = Pt + 10*log10((3e8/462e6)ˆ2./((4*pi)ˆ2*TxSens.ˆgamma)); temp=pwrsdbm(k)-ppred; temp=temp.ˆ2; z=z+temp; end end DRDC Ottawa TM

48 This page intentionally left blank. 34 DRDC Ottawa TM

49 Annex B: Sample data file for use with GeoLAB Below is a very short sample of a comma-delimited data file formatted for use with Geo- LAB (the duration is approximately 3 seconds). Data files in practice may contain tens or even hundreds of thousands of measurements. Channel ID,Power dbm,direction Deg,Direction Quality Deg,Status,Time,Milliseconds, GPS Status ID,Longitude Deg Decimal,Latitude eg Decimal,Elevation m,gps Grid ID 1,-101.4,150,11,1,12:02:52 PM,659,2, , ,53.8,6660 1,-101.5,164,20,1,12:02:52 PM,845,2, , ,53.8,6660 1,-102.4,170,10,1,12:02:52 PM,878,2, , ,53.8,6660 1,-102.2,177,17,1,12:02:52 PM,911,2, , ,53.8,6660 1,-103.2,164,10,1,12:02:52 PM,944,2, , ,53.8,6660 1,-104.4,168,12,1,12:02:53 PM,547,2, , ,54.5,6660 1,-105.3,172,10,1,12:02:53 PM,580,2, , ,54.5,6660 1,-107,170,12,1,12:02:53 PM,612,2, , ,54.5,6660 1,-104.8,158,10,1,12:02:53 PM,645,2, , ,54.5,6660 1,-107.8,170,10,1,12:02:53 PM,677,2, , ,54.5,6660 1,-106.1,162,30,1,12:02:53 PM,710,2, , ,54.5,6660 1,-105.9,163,10,1,12:02:53 PM,743,2, , ,54.5,6660 1,-108.2,163,11,1,12:02:53 PM,776,2, , ,54.5,6660 1,-106,167,10,1,12:02:53 PM,808,2, , ,54.5,6660 1,-106.5,165,10,1,12:02:53 PM,841,2, , ,54.5,6660 1,-106.8,163,10,1,12:02:53 PM,874,2, , ,54.5,6660 1,-106.9,167,12,1,12:02:53 PM,907,2, , ,54.5,6660 1,-106.2,166,10,1,12:02:53 PM,939,2, , ,54.5,6660 1,-106.8,155,10,1,12:02:53 PM,972,2, , ,54.5,6660 1,-108.1,157,27,1,12:02:54 PM,5,2, , ,54.5,6660 1,-106.9,153,11,1,12:02:54 PM,38,2, , ,54.5,6660 1,-106.7,166,10,1,12:02:54 PM,71,2, , ,54.5,6660 1,-105.8,158,11,1,12:02:54 PM,656,2, , ,55.1,6660 1,-107.8,155,11,1,12:02:54 PM,689,2, , ,55.1,6660 1,-105.5,156,10,1,12:02:54 PM,722,2, , ,55.1,6660 1,-105.1,164,14,1,12:02:54 PM,755,2, , ,55.1,6660 1,-105,157,10,1,12:02:54 PM,788,2, , ,55.1,6660 1,-105.9,136,14,1,12:02:54 PM,821,2, , ,55.1,6660 * Note that the header line should be on one line in the data text file (the page width in this document forces the header line break). DRDC Ottawa TM

50 This page intentionally left blank. 36 DRDC Ottawa TM

51 List of abbreviations/acronyms AOA AOI CEP CF CNEW CRC DF DND DRDC EEP EW FRS GeoLAB GPS GUI LAV LOB MiDAS PDOA RF RMSE RSS TDP TEWS Tx Angle of Arrival Area of Interest Circular Error Probable Canadian Forces Communication and Navigation Electronic Warfare Communications Research Centre Direction Finding Department of National Defence Defence Research and Development Canada Elliptical Error Probable Electronic Warfare Family Radio Service Geolocation Algorithm Benchmark and Testbed Global Positioning System Graphical User Interface Light Armoured Vehicle Line of Bearing Military Digital Analysis System Power Difference of Arrival Radio Frequency Root Mean Square Error Received Signal Strength Technology Demonstration Project Tactical Electronic Warfare Systems Transmitter DRDC Ottawa TM

52 This page intentionally left blank. 38 DRDC Ottawa TM

53 DOCUMENT CONTROL DATA (Security classification of title, body of abstract and indexing annotation must be entered when document is classified) 1. ORIGINATOR (The name and address of the organization preparing the document. Organizations for whom the document was prepared, e.g. Centre sponsoring a contractor s report, or tasking agency, are entered in section 8.) Defence R&D Canada Ottawa 3701 Carling Avenue, Ottawa, Ontario, Canada K1A 0Z4 2. SECURITY CLASSIFICATION (Overall security classification of the document including special warning terms if applicable.) UNCLASSIFIED 3. TITLE (The complete document title as indicated on the title page. Its classification should be indicated by the appropriate abbreviation (S, C or U) in parentheses after the title.) GeoLAB: A geolocation algorithm testbed and benchmarking tool 4. AUTHORS (Last name, followed by initials ranks, titles, etc. not to be used.) Jackson, B.R. 5. DATE OF PUBLICATION (Month and year of publication of document.) June a. NO. OF PAGES (Total containing information. Include Annexes, Appendices, etc.) 54 6b. NO. OF REFS (Total cited in document.) 5 7. DESCRIPTIVE NOTES (The category of the document, e.g. technical report, technical note or memorandum. If appropriate, enter the type of report, e.g. interim, progress, summary, annual or final. Give the inclusive dates when a specific reporting period is covered.) Technical Memorandum 8. SPONSORING ACTIVITY (The name of the department project office or laboratory sponsoring the research and development include address.) Defence R&D Canada Ottawa 3701 Carling Avenue, Ottawa, Ontario, Canada K1A 0Z4 9a. PROJECT NO. (The applicable research and development project number under which the document was written. Please specify whether project or grant.) 12po 10a. ORIGINATOR S DOCUMENT NUMBER (The official document number by which the document is identified by the originating activity. This number must be unique to this document.) DRDC Ottawa TM b. GRANT OR CONTRACT NO. (If appropriate, the applicable number under which the document was written.) 10b. OTHER DOCUMENT NO(s). (Any other numbers which may be assigned this document either by the originator or by the sponsor.) 11. DOCUMENT AVAILABILITY (Any limitations on further dissemination of the document, other than those imposed by security classification.) ( X ) Unlimited distribution ( ) Defence departments and defence contractors; further distribution only as approved ( ) Defence departments and Canadian defence contractors; further distribution only as approved ( ) Government departments and agencies; further distribution only as approved ( ) Defence departments; further distribution only as approved ( ) Other (please specify): 12. DOCUMENT ANNOUNCEMENT (Any limitation to the bibliographic announcement of this document. This will normally correspond to the Document Availability (11). However, where further distribution (beyond the audience specified in (11)) is possible, a wider announcement audience may be selected.) Full unlimited announcement