Guidelines for the SAR operational evaluation of the AIMS system. G. Toussaint V. Larochelle DRDC Valcartier

Transcription

1 Guidelines for the SAR operational evaluation of the AIMS system G. Toussaint V. Larochelle DRDC Valcartier Defence R&D Canada Valcartier Technical Memorandum DRDC Valcartier TM March 2008

2

3 Guidelines for the SAR operational evaluation of the AIMS system G. Toussaint V. Larochelle DRDC Valcartier Defence R&D Canada Valcartier Technical Memorandum DRDC Valcartier TM March 2008

4 Principal Author Geneviève Toussaint Defence Scientist Approved by Dennis Nandlall C/PEA Approved for release by Christian Carrier Chief Scientist Her Majesty the Queen in Right of Canada, as represented by the Minister of National Defence, 2008 Sa Majesté la Reine (en droit du Canada), telle que représentée par le ministre de la Défense nationale, 2008

5 Abstract.. An airborne multi-sensor imaging system is presently being developed under the Canadian Technology Demonstration Program (TDP). The Advanced Integrated Multi-sensing Surveillance (AIMS) system should enhance Canadian Forces (CF) surveillance and reconnaissance capabilities, particularly at night and in degraded weather conditions. An important role for the system will be to support Search and Rescue (SAR) operations. Therefore, the need for a rigorous method to evaluate the effectiveness of AIMS was identified at the start of the project. In view of preparing flight trials, measures of effectiveness (MOEs) to evaluate the system, which include AIMS and the operator, were developed. This memorandum summarizes the AIMS TDP evaluation criteria, which include some MOEs previously published and modifications done to some of them, and proposes seven additional MOEs: three are related to the tracking capability, one to the search planning time, one to the experience of the operator, one to the search area covering time and finally, the last one to both the probability of detection and the time lost due to reduced visibility. Among other things, the MOEs of the system will allow to determine the data collection requirements for the Search and Rescue Exercise (SAREX) event and for other flight demonstrations. In addition to the MOEs, other criteria are proposed to help evaluate the AIMS system. Thereafter, the memorandum presents the results of the analysis of the data collected during the SAREX search event held in This collection of data improves the previously published database collected during other past SAREX events, giving us more confidence in the results. Eventually, the data compiled from past SAREX exercises will be compared to the results obtained using the AIMS system participating at the SAREX search event Further on, recommendations are given for the next SAREX, that was held in Goose Bay in 2007 and, finally, the last section presents factors to consider when choosing scenarios for the future flight trials. DRDC Valcartier TM i

6 Résumé... Un système aéroporté d imagerie de multi-capteurs est actuellement développé dans le cadre d un projet de démonstration technologique (PDT) canadien. Ce système perfectionné de surveillance multi-capteurs intégré (AIMS) devrait améliorer les capacités de surveillance et reconnaissance des Forces canadiennes (FC), particulièrement la nuit et dans des conditions météorologiques dégradées. Un rôle important pour le système sera de soutenir les opérations de recherche et sauvetage (SAR). C est pourquoi, dès le début du projet, une méthode rigoureuse d évaluation de l efficacité d AIMS a été identifiée. Dans le but de préparer les essais en vol, des mesures d efficacité (MOE) pour évaluer le système, qui inclut AIMS et l opérateur, ont été développées. Ce mémorandum résume les critères d évaluation du PDT AIMS, lesquels incluent des MOE publiées précédemment et des modifications apportées à certains d entre eux, et propose sept MOE additionnelles: trois sont liées à la capacité du capteur à suivre une cible, une au temps requis pour planifier une recherche, une à l expérience de l opérateur, une au temps requis pour couvrir une aire de recherche et finalement, la dernière, liée à la fois à la probabilité de détection et au temps perdu en raison de la visibilité réduite. Entre autres, ces MOE nous permettent de déterminer les données à recueillir durant l événement de l exercice de recherche et sauvetage (SAREX) et durant d autres démonstrations en vol. En plus des MOE, d autres critères sont proposés pour aider à évaluer le système AIMS. Par la suite, ce document présente les résultats de l analyse des données recueillies durant l événement de recherche SAREX tenu en Ces données augmentent l étendue des données déjà recueillies dans des SAREX passés, donnant ainsi plus de validité aux résultats. Éventuellement, les données recueillies durant les autres exercices SAREX seront comparées aux résultats obtenus en utilisant le système AIMS qui participera à l exercice de recherche du SAREX Ensuite, certaines recommandations sont formulées pour le SAREX suivant, qui a eu lieu à Goose Bay en 2007 et finalement, le dernier chapitre présente les facteurs dont on doit tenir compte lors du choix des scénarios pour les futurs essais en vol. ii DRDC Valcartier TM

7 Executive summary Guidelines for the SAR operational evaluation of the AIMS system G. Toussaint; V. Larochelle; DRDC Valcartier TM ; Defence R&D Canada Valcartier; March The Advanced Integrated Multi-sensing Surveillance Technology Demonstration Program (AIMS TDP) is a system that combines many sensors including a thermal imager, an active imaging system, a laser rangefinder, a robust sensor scanning patterns and colour cameras mounted on a stabilized platform and operated with an advanced human-machine interface. This system should enhance the Canadian Forces (CF) detection capability during night time and under bad weather conditions. In a previous publication it has been recommended that an analyst gather and analyse more data for the evaluation of Search and Rescue (SAR) measures of effectiveness (MOEs). The work presented in this memorandum was undertaken to respond to the aforementioned recommendation and requirement for additional analysis. The work was conducted by Defence Research and Development Canada Centre for Operational Research and Analysis (DRDC CORA) and DRDC Valcartier. In short, it presents evaluation criteria (which include MOEs and specifications), it improves the previously published data collected during the past Search and Rescue Exercise (SAREX) events and it identifies the data needed to fully demonstrate the performance of the AIMS system during future flight demonstrations. This document also presents recommendations for the data collection at the next SAREX and finally, it gives factors to consider before choosing scenarios for the flight trials. In this memorandum, the effectiveness of the system is represented by a combination of the effectiveness of AIMS and the effectiveness of the operator. Therefore, the authors summarize the evaluation criteria required to demonstrate the effectiveness for each component of the AIMS system and for the combination of these components. Some of these criteria are based on previous investigations but, seven additional MOEs are proposed in this memorandum: three are related to the tracking capability, one to the search planning time, one to the experience of the operator, one to the search area covering time and finally the last one to both the probability of detection and the time lost due to reduced visibility. Moreover, some modifications have been done to the previously identified evaluation criteria: the target location accuracy, the target confirmation time and the number of call-arounds. Other criteria are also suggested to help evaluate the AIMS system. Then, to ensure that all required information is collected for the MOEs assessment, the data to be gathered during SAREX and during the other trials are given. Also, a methodology to analyse these data is suggested. As an example of data analysis, some data collected during the SAREX 2006 were analysed. The outcomes include some recommendations proposed for the SAREX 2007 and for the future flight trials. Finally, the last section presents factors to consider when determining scenarios for the flight trials and provides some examples. DRDC Valcartier TM iii

8 The main conclusions of this memorandum are: 1. The flight trials during the SAREX could possibly demonstrate the performance of the AIMS system, but some factors need to be considered when analysing the results. These factors are: a. The aircraft on which the system will be installed is a Twin Otter and only two or three people including the pilot and the operator will be on board whereas there should be as many SAR technicians (techs) and spotters as in the other aircrafts participating in the SAREX (for example, sometimes there might be up to 6 SAR techs/spotters in a Hercules). b. Data will be collected for only one flight using the AIMS system and they will be compared to the data collected for five or six SAREX flights. Therefore, the data collected by the SAREX teams when combined altogether are more statistically valid than those collected by AIMS. c. Importantly, no trial is planned at night or in bad weather conditions, forsaking the conditions where the AIMS system would be most useful. d. During the SAREX, some of the targets chosen do not represent real SAR scenarios (for example, orange marker panels) whereas the system has a combination of sensors (for example, thermal imager and active imaging system) that help mostly to find passive targets. 2. A new approach will be in place for the SAREX Namely, seven permanent targets were chosen and will be installed at each future SAREX. This project could enhance SAR training and allow the possibility to evaluate and compare new techniques and equipment like AIMS. 3. AIMS sensors will be equipped with powerful zooming capabilities which should allow the operator to immediately recognize or identify targets of interest. This capability is not available to SAR techs/spotters and will present a significant impact on some criteria, like the target confirmation time and the number of call-arounds. Overall, the recommendations are: 1. To continue to collect data during SAREX events until the flight trials with the AIMS system take place. 2. The operational research (OR) analyst should continue to be involved in SAREX to facilitate the data collection activity during the AIMS flight trials at SAREX The OR analyst should propose to the SAREX committee a facultative night flight trial with night vision goggles (NVGs). 4. The OR analyst should develop some MOEs to specifically evaluate the knowledge and effectiveness of the operator using the AIMS system. In the future, these MOEs could help to develop a training program for its operators. iv DRDC Valcartier TM

9 Sommaire... Guidelines for the SAR operational evaluation of the AIMS system G. Toussaint; V. Larochelle; DRDC Valcartier TM ; R & D pour la défense Canada Valcartier; Mars Le projet de démonstration technologique d un système perfectionné de surveillance multicapteurs intégré (PDT AIMS) est un système qui combine de nombreux capteurs dont un imageur thermique, un système d imagerie active, un télémètre au laser et des capteurs à trame de balayage montée sur une plateforme stabilisée et activée à l aide d une interface homme/machine perfectionnée. Ce système devrait améliorer grandement la capacité de détection des Forces canadiennes (FC) la nuit et dans de mauvaises conditions météorologiques. Dans une publication précédente, il a été recommandé qu un analyste recueille et analyse davantage de données pour l évaluation de mesures d efficacité (MOE) de recherche et sauvetage (SAR). Le travail présenté dans ce mémorandum a donc été entrepris suite à cette recommandation d effectuer des analyses additionnelles. Le travail a été effectué par Recherche et développement pour la défense Canada Centre d analyse et de recherche opérationnelle (RDDC CARO) et RDDC Valcartier. En bref, il présente des critères d évaluation (qui incluent des mesures d efficacité et des spécifications), il augmente l étendue des données déjà publiées et recueillies durant les exercices de recherche et sauvetage (SAREX) précédents et il identifie les données nécessaires pour démontrer la performance du système AIMS durant les futurs essais en vol. Ce document fournit aussi des recommandations pour la cueillette de données du SAREX suivant et finalement, il présente les facteurs dont on doit tenir compte avant de choisir les scénarios des essais en vol. Dans ce mémorandum, l efficacité du système est représentée par la combinaison de l efficacité d AIMS et de l efficacité de l opérateur. C est pourquoi, les auteurs résument les critères d évaluation requis pour démontrer l efficacité de chaque composante du système AIMS et de la combinaison de ces composantes. Certains de ces critères sont basés sur des études précédentes, mais sept mesures d efficacité (MOE) additionnelles sont proposées dans ce mémorandum : trois sont liées à la capacité du capteur à suivre une cible, une au temps requis pour planifier une recherche, une à l expérience de l opérateur, une au temps requis pour couvrir une aire de recherche et finalement, la dernière, à la fois à la probabilité de détection et au temps perdu en raison de la visibilité réduite. De plus, des modifications ont été apportées aux critères d évaluation mentionnés précédemment : la précision de la position d une cible, le temps requis pour confirmer une cible et le nombre de retours en arrière. D autres critères sont aussi suggérés pour aider à évaluer le système AIMS. Ensuite, pour s assurer d avoir toute l information nécessaire à l évaluation des MOE, les données à recueillir lors du SAREX et durant les autres essais sont mentionnées. De plus, une méthodologie pour analyser ces données est suggérée. À titre d exemple, certaines données recueillies durant le SAREX 2006 sont analysées. Des recommandations sont alors formulées pour le prochain SAREX et pour de futurs essais en vol. Finalement, le dernier chapitre présente les facteurs dont on doit tenir compte lors du choix des scénarios pour les essais en vol et fournit quelques exemples. DRDC Valcartier TM v

10 Les principales conclusions de ce mémorandum sont: 1. Les essais en vol durant le SAREX pourraient démontrer la performance du système AIMS, mais certains facteurs entrent en ligne de compte lors de l analyse des résultats, notamment: a. L avion dans lequel le système sera placé est un Twin Otter et seulement deux ou trois personnes, incluant le pilote et l opérateur, seront à bord alors qu il devrait y avoir autant de techniciens SAR et d observateurs que dans les autres avions qui participent au SAREX (par exemple, parfois il peut y avoir jusqu à six techniciens SAR/observateurs dans un Hercule). b. Des données seront recueillies pour un seul vol avec le système AIMS et seront comparées aux données recueillies par cinq ou six équipes du SAREX. Par conséquent, en combinant les données recueillies par les équipes du SAREX, on obtient des données statistiquement plus valides que les données recueillies avec le système AIMS. c. Il est très important de tenir compte qu aucun essai n est prévu la nuit ni dans de mauvaises conditions météorologiques, alors que c est justement dans ces situations que le système est le plus utile. d. Durant le SAREX, certaines cibles ne représentent pas un incident réel de SAR, par exemple, des panneaux oranges, alors que le système possède une combinaison de capteurs, par exemple, un imageur thermique et un système d imagerie active, qui aident principalement à détecter des cibles passives. 2. Une nouvelle démarche a été mise en place dès le SAREX Plus précisément, sept cibles permanentes ont été choisies et seront dorénavant installées à chaque futur SAREX. Cette mesure pourrait améliorer l entraînement de SAR et offrir la possibilité d évaluer et de comparer de nouvelles techniques et équipements comme AIMS. 3. AIMS sera équipé de puissantes lentilles permettant d agrandir ou de réduire l image, ce qui devrait permettre à l opérateur de reconnaître ou d identifier immédiatement les cibles d intérêt. Les techniciens SAR et les observateurs n ont pas cette possibilité; c est pourquoi l utilisation de ces lentilles aura un impact significatif sur certains critères, comme le temps requis pour confirmer une cible et le nombre de retours en arrière. Les recommandations sont les suivantes : 1. Continuer la collecte de données lors des SAREX jusqu aux essais en vol avec AIMS. 2. L analyste en recherche opérationnelle (RO) devrait continuer de s impliquer dans le SAREX pour faciliter la cueillette des résultats lors des essais avec AIMS au SAREX L analyste en RO devrait proposer au comité du SAREX une recherche de nuit facultative avec une lunette de vision nocturne. 4. L analyste en RO devrait développer des MOE pour évaluer spécifiquement les connaissances et l efficacité de l opérateur à utiliser le système AIMS. Dans le futur, ces MOE pourraient alors aider à développer un programme d entraînement pour ces opérateurs. vi DRDC Valcartier TM

11 Table of contents Abstract..... i Résumé ii Executive summary... iii Sommaire... v Table of contents... vii List of figures... ix List of tables... x Acknowledgements... xi 1...Introduction Evaluation criteria AIMS evaluation criteria Tracking capability Data processing time Search planning time Ability to collect evidence on an incident Operator evaluation criteria Operator experience Operator comfort System evaluation criteria Search area covering time Target location accuracy Target confirmation time Number of call-arounds False detection rates Probability of detection Time lost due to reduced visibility Ability to disseminate information on a target Summary SAREX 2006 Event Summary of SAREX 2006 search event Target location accuracy Target confirmation time Number of call-arounds False detection rates Time lost due to a missed target Probability of detection DRDC Valcartier TM vii

12 3.6.2 Familiarity with the environment Number of targets SAREX 2007 Event Collaboration with SAREX community Data collection guidelines Scenarios for the flight trials Conclusion References Annex A.. SAREX Events description Annex B.. Search Event Scenario B.1 Search Event Scenario B.2 JRCC Tasking Annex C.. Matlab Program Annex D.. Examples of GPS tracks Annex E... Letter of authorization[20] List of symbols/abbreviations/acronyms/initialisms Glossary Distribution list viii DRDC Valcartier TM

13 List of figures Figure 1:. Types of criteria for the AIMS system... 3 Figure 2:. Sweep Width versus POD [10] Figure 3:. Map of the search area [13] Figure 4:. Number of targets detected within specific distance (NM) Figure 5:. Percentage obtained for good identification of shape versus percentage obtained for identification of at least one good colour Figure 6:. Flight 9 Faithfull 42 SAREX 2: Top View [16] Figure 7:. Flight 9 Faithfull 42 SAREX 2: 3D-Elevated View [16] Figure 8:. False detection rates/nm Figure 9:. Comparison between real target detected and false target detected Figure 10: Estimated PODs for cooperative and passive targets for the SAREX Figure 11: Estimated PODs for cooperative and passive targets for the past SAREX (1988 to 1991, 1993, 2004 to 2006) Figure 12: Estimated PODs for different types of cooperative and passive targets for the past SAREX (1988 to 1991, 1993, 2004 to 2006) Figure 13: Detection rates required to demonstrate improvements with 95% confidence using SAREX 2006 and past SAREX search events data Figure B-1: GPS track of the team Janke Figure B-2: GPS track of the team Duffy Figure B-3: GPS track of the team CSS - Siket SAREX Figure B-4: GPS track of the team Siket Sarex DRDC Valcartier TM ix

14 List of tables Table 1:... Background information on the operator s and SAR techs/spotters experience... 6 Table 2:... Example of target characteristics compilation... 9 Table 3:... Summary of the possibility to evaluate MOEs during different flight trials Table 4:... Teams participating to the SAREX 2006 search event Table 5:... Description of the targets Table 6:... Points allocation Table 7:... Evaluation of the accuracy of the description of the targets given by the teams at the SAREX 2006 search event Table 8:... Number of call-arounds Table 9:... False Detection Table 10:. Estimated PODs of SAREX Table 11:. Winner versus the region where the SAREX event was held x DRDC Valcartier TM

15 Acknowledgements Special thanks are given to the SAR community for their collaboration on the AIMS project (including the military and the Civil Air Search And Rescue Association (CASARA)), in particular to Major Bob Struthers for his excellent support and to Maj. Clinton Mowbray, Capt Heko Partenheimer and Capt Phil Bischoff. Also, Sgt. André Hotton was very supportive during the implementation of permanent targets at the SAREX and a lot of the future work would not be possible without his collaboration. Thanks to all others that offered their support and facilitated the data collection activity at the SAREX 2006 event. Finally, the authors would like to thank Mr. Terry Rea, Dr. Jacqui Crebolder and Capt. Jean- François Gaudreau for their comments, Dr. Étienne Vincent for the help he provided on the MOEs, to Mr. David Taylor for the global positioning system (GPS) and to Defence Research and Development Canada Centre for Operational Research and Analysis (DRDC CORA) for reviewing the document. DRDC Valcartier TM xi

16 This page intentionally left blank. xii DRDC Valcartier TM

17 1 Introduction The Advanced Integrated Multi-sensing Surveillance (AIMS) system is a fully integrated multisensor system in an airborne stabilized platform. AIMS includes an active imager integrated with thermal imaging and visible colour cameras, a geo-referencing system to precisely locate targets on the ground and an advanced operator workstation. The system will extend Canadian Forces (CF) capability to conduct diverse missions around the clock and in adverse weather conditions. Also, it will optimize the detection and identification of small objects 1 and improve the effectiveness of current airborne search vehicles by increasing surveillance and reconnaissance capabilities [1]. Thus, the AIMS system will improve search effectiveness by reducing search time, will improve situational awareness and will provide integrated Search and Rescue (SAR), maritime patrol and tactical surveillance capability in foul weather conditions and at night. In a previous publication [2] it has been recommended that an analyst gather and analyse more data for the evaluation of SAR measures of effectiveness (MOEs). The work presented in this memorandum was undertaken to respond to the aforementioned recommendation and requirement for additional analysis. The work was conducted by Defence Research and Development Canada Centre for Operational Research and Analysis (DRDC CORA) and DRDC Valcartier. In short, the objectives of this memorandum are: to summarize the evaluation process of the AIMS system using evaluation criteria (which include MOEs and specifications), to improve the previously published data collected during the past Search and Rescue Exercise (SAREX) events [2][3][4][5][6][7][8] and to identify the data needed to fully demonstrate the performance of the AIMS system during future flight demonstrations. It also presents recommendations for the data collection at the next SAREX and finally, it gives factors to consider before choosing scenarios for the flight trials. The SAREX is an annual event and an opportunity for the different SAR teams from different squadrons (SQNs) to evaluate their SAR skills but also to exchange ideas on this subject and to promote a sense of community between its members. Usually, the normal events taking place at the SAREX are: Belly Ringer, Parachute Accuracy Event, Medical Event, Search Event (including Civil Air Search And Rescue Association (CASARA) aircrafts), Rescue Event, Maintenance Event, Helicopter Accuracy Event and Ground Search Event (see Annex A for more details on the different events). For the search event, each team is briefed on the scenario and asked to find as many search objects as possible along a pre-defined path or within a specific area and usually within one hour of search. As an example, Annex B provides the SAREX 2006 search event Joint Rescue Coordination Center (JRCC) tasking. It was suggested that the AIMS system could be compared to unassisted experienced SAR technicians (techs) during this search event. Hence, the AIMS project team plans to be part of the SAREX 2009 search event. The second section of this memorandum summarizes the evaluation criteria, which include a brief review of some of the MOEs developed previously by Vincent [9], discusses modifications done to some of them and provides other criteria to be used to evaluate the AIMS system. Some of the criteria were developed to specifically evaluate AIMS, others look to evaluate the operator, as the remaining criteria are used to evaluate the system with the operator. Further on, the third section presents a summary of the data collected at SAREX 2006 and their analysis. The fourth section 1 A human-size object is considered as a small object: life jacket, dummy and ice box are some examples. DRDC Valcartier TM

18 explains the collaboration that was established with the SAREX community and suggests a guideline for the collection of data at SAREX 2007 and beyond, and finally, the last section presents factors to consider when choosing scenarios for the future flight trials and provides some examples. 2 DRDC Valcartier TM

19 2 Evaluation criteria Previous investigations [2][3][9] have developed different MOEs to be used in assessing the performance of the AIMS system. In these documents, the MOEs were divided into two categories, namely the measures of effectiveness related to the time criterion and the other MOE criteria. However, to be able to evaluate adequately the AIMS system, it was established that some criteria, which include MOEs and specifications, would be defined specifically for AIMS, some for the operator, and finally, some criteria would be developed for the whole system. This concept is shown in Figure 1. Figure 1: Types of criteria for the AIMS system In the present memorandum, MOEs developed previously [2][3][9] are presented where some have been modified to more accurately fit the objective of the evaluation. Also, seven new MOEs were added to this list: three are related to the tracking capability, one to the search planning time, one to the experience of the operator, one to the search area covering time and finally, the last one to both the probability of detection (POD) and the time lost due to reduced visibility. AIMS evaluation criteria include the following: tracking capability, data processing time, search planning time and ability to collect evidence on an incident. The operator evaluation criteria contain the following: operator experience and operator comfort. Finally, some criteria evaluate the combination of the performance of both the operator and AIMS. These evaluation criteria are related to: search area covering time, target location accuracy, target confirmation time, number of call-arounds, false detection rates, probability of detection, time lost due to reduced visibility and finally, ability to disseminate information on a target. In short, the aim of this section is to explain the evaluation criteria, to suggest a method to collect and analyse the data according to the MOEs and to determine during which flight trials the data could be collected. DRDC Valcartier TM

20 2.1 AIMS evaluation criteria The evaluation criteria presented in this section are related to the AIMS system only because they are not influenced by the performance of the operator. Several criteria are suggested, among them, five MOEs are related to: the tracking capability, the data processing time and the search planning time, and one specification is tied to the ability to collect evidence on an incident Tracking capability During a SAR search, since the aircraft is constantly moving and considering that sometimes the target may also move (for example a person floating in the water), a tracking capability would allow the operator to follow more easily a target of interest. In this case, supposing that the aircraft is able to follow the target at any time, several MOEs are proposed to evaluate the effectiveness of the tracking device; first, the alignment error between the actual location of the target and the one given by the tracking device could be measured for a certain period of time while both the aircraft and the target are moving, second, the maximum time that the sensor is able to stay on the target could be monitored and finally, the average time it takes for the sensor to recover the target when loosing it could be determined Data processing time According to Vincent [9], if images are not displayed in real time by the imaging sensor (the reason could be that some processing is required before an image allowing detection becomes available), there could be a significant interval of time (loss) between the fly-over and the detection. Moreover, if a call-around is needed, the delay caused by the processing would also impact the time before the detection as the aircraft will have to fly a greater distance to reach the site of interest. Time from fly-over to detection could be measured during trials on the ground or during flight trials. During the SAREX, this would require another analyst on board of the Twin Otter to gather the data and it is actually not possible because of space and operational contingencies. However, in the case of AIMS, images from the various imaging devices will be pre-processed before being displayed on the operator workstation, all done in real time with no delays expected Search planning time Some information on search planning methods is available in the National SAR Manual [10]. Among other things, it is mentioned that searches may be planned manually or with the help of appropriate computer programs and that usually five sequential events are followed to develop a search plan. They are: estimating the datum determining the position of the emergency and in maritime cases determining the effect of wind and current on the survivors; determining the size of the search area allowing for errors in position estimates, navigation errors of search units and drift variables; 4 DRDC Valcartier TM

21 selecting appropriate search patterns considering size of area, type of terrain and capabilities of search units; determining the desired area coverage considering factors affecting the probability of detection, track spacing and number of sweeps; and developing an optimum and attainable search plan considering the number of search units available and other limiting factors and circumstances. Other parameters such as the terrain, weather, availability and capability of the search units, limitations of time, and safety factors need to be considered. Also, the search plan has to be periodically reviewed and updated. Therefore, planning searches involves many factors and takes time. The presence of a planning tool incorporating the AIMS system could consider the evolution of the parameters of search in real time, which could possibly make the updating of the plan easier and faster. Vincent [9] proposed to report the existence of a planning tool for optimal searches, when evaluating different SAR sensors. It would be interesting to go further and to compare for different scenarios the time necessary to plan an optimal search by the planning tool and manually, by the appropriate person Ability to collect evidence on an incident Actually, in a SAR incident, pictures are taken using hand-held cameras with zoom capabilities. These capabilities are already integrated and made available to the operator of AIMS. However, with new equipment using advanced sensors like AIMS, it would be useful to capture images from the additional sensors like the thermal imagers or image intensifiers [9]. As proposed by Vincent in his memorandum [9], it should be sufficient to specify the presence or absence of a capability to store captured images. In that case, the presence of this capability would be considered as a specification or asset of the system. 2.2 Operator evaluation criteria An evaluation criteria related to the operator is a criteria that is not or almost not influenced by the performance of the system. For example, several capabilities incorporated on the system, such as the active imager and the thermal imager, could influence the detection capability, but will not change the number of hours of experience that an operator has before he comes to a SAREX event. In this case, the number of hours of experience will be more influenced by the availability of the equipment, the budget availability for training, the capability of the operator to learn and the availability of the operator. In the same line of thinking, the performance of AIMS is related to the capability of the system to detect targets, which does not influence directly how much time the operator can work effectively on the system. Thus, this sub-section presents the MOEs related to the operator experience and comfort Operator experience One very important measure to be considered in the evaluation of AIMS is the SAR operator s experience on the system. This new MOE would be to determine before each flight trial takes DRDC Valcartier TM

22 place the number of hours of experience as a SAR tech/spotter (spotting with naked eyes) and as an operator using the sensor suite. One year prior to SAREX 2009, it is suggested that five operators be trained (if possible) to use the AIMS system in order to fly the AIMS system as many times as the military aircraft during the SAREX 2009 search event. Also, at this event, the number of SAR techs/spotters and the number of hours of experience for each of them should be noted before each search flight. As well, the flight number and the type of aircraft should appear on the form. An example table to be completed is shown below (Table 1). With the results, it might be possible to determine if experience on AIMS and/or as a SAR tech/spotter has an influence on the number of targets detected. Table 1: Background information on the operator s and SAR techs/spotters experience NAMES OF SAR TECHS/SPOTTERS MILITARY OR CIVILIAN # HOURS OF EXPERIENCE SEARCHING WITH THE AIMS SYSTEM # HOURS OF EXPERIENCE SEARCHING WITH NAKED EYES TOTAL We suppose that the operator will have a sufficient knowledge of the system to use it correctly and will have sufficient training before he comes to SAREX 2009 for the search exercise. If the operator is not well trained on the system, poor results can be obtained during flight trials which will not reflect the actual capability of AIMS. In this regard, it would be important to evaluate the knowledge and effectiveness of the operator to use the AIMS system before SAREX To do that, MOEs should be developed by the OR analyst with the help of the scientific team of the AIMS project. Furthermore, in the future, these MOEs could help to develop a training program for operators using the AIMS system Operator comfort Another important MOE suggested by Vincent [9] is related to the fatigue of the operator. During a real search, most of the time spotters alternate every twenty minutes to combat fatigue [9]. This alternative will not be possible for the AIMS operator since there will be only one operator on board. Consequently, before the SAREX search takes place, field tests should be carried out and according to the opinion or experience of the operator for example, the maximum time that the sensor suite can be used without significant operator fatigue should be determined. The approach suggested in this memorandum to evaluate the level of fatigue would be to give to the operator video sequences to be analysed. These experiments would be performed on the ground; on one hand, this would allow the possibility to control and fix easily some parameters and would significantly be less expensive than flight trials; on the other hand, the sickness factor 6 DRDC Valcartier TM

23 would not be considered. Parameters like the background and the continuity in each video sequence could be fixed. Also, the number of targets to be found could be proportional to the length of the sequence. Different types of targets could be used: parachute, fuselage, dummy with life preserver, white panel marker with identification numbers, canoe, etc. If possible, the operator should ignore in advance the types of target he is looking for. There should also be a period of time for the operator to rest between each analysis. The duration of the sequences could be as follows: 10 min. rest 15 min. rest 20 min. rest 30 min. rest 45 min. rest 60 min. The period of rest should be sufficient for the operator to recover completely from possible annoyances (for example from headaches). We expect from the experiments that the type of search will influence the level of fatigue of the operator using the sensor suite. There are primarily two reasons for that: overload and vigilance fatigue. Also, the human/machine interface conviviality, the flight conditions and the general health of the operator could influence his fatigue level. If the result from the experiments shows that the operator cannot use the AIMS system for a long period of time with full concentration (less than one hour for example), it might raise other problems such as the need for a second operator to alternate with the first one during real SAR incident. 2.3 System evaluation criteria The MOEs and specifications presented in this section evaluate simultaneously the operator and AIMS effectiveness. They are related to the search area covering time, the target location accuracy, the target confirmation time, the number of call-arounds, the false detection rates, the probability of detection, the time lost due to reduced visibility and the ability to disseminate information on a target Search area covering time The search area covering time is defined by the time required to cover a search area from a starting point to target location excluding time losses due to call-around, loss of visibility and return trips for refuelling [9]. So, the search area covering time depends essentially on the aircraft speed and the search pattern [9], for a given altitude. Moreover, during SAR missions different patterns may be chosen such as track crawl, creeping line and parallel track patterns, expanding square, sector pattern or contour search [10] that may change the search area covering time. Using the AIMS system on board should not change the search area covering time (in the sense that aircraft speed, track spacing and search pattern are chosen according to actual search standards). On the other hand, if the flight trials with AIMS demonstrate that the velocity of the aircraft is too high for the operator to capture the data and identify the targets quickly, but that the system is really helpful at a lower speed, then AIMS could have an influence on the selection of the aircraft speed. Also, if a search planning tool is used, it could reduce the time taken to plan the search. Thus, a new MOE developed by the authors is to establish a maximum flight velocity for the operator to be able to analyse effectively the data in real time. Another factor to be considered in this evaluation is the fact that the operator is viewing the scene on a display and that a restricted field of view might influence his capability to cover the entire area while flying over. This factor is explained in detail in section DRDC Valcartier TM

24 The data related to this criterion should be collected during the first AIMS flight demonstration, i.e., before the SAREX flight trials. Two scenarios may be envisaged. In the first case, the pilot could increase incrementally the aircraft velocity for a while and at each selected speed the operator would tell him when he is not able to analyse the data effectively. In the second case, the operator could analyse video sequences passing with different speeds on the screen and determine an approximate velocity where he cannot analyse effectively the data anymore. Again, the advantages of ground trials are the possibility to control many parameters during the experiments and to reduce the cost of the experiments. Although the second method is not stringent, it will give an estimate of the maximum image velocity the operator can analyse. Other indicators could be developed to determine the maximum range velocity Target location accuracy According to Vincent [9], the degree of precision of the location should be sufficient to allow rescue teams (helicopter, vessel, ground resources) to find the target rapidly. As a MOE, he suggested to report the presence or absence of a geo-location capability sufficiently accurate to direct others to a location of interest. However, the authors of the actual memorandum believe that it is important to get the position of the target as precise as possible, particularly when SAR fixed-wing aircrafts are used and not helicopters where it is easier to go right on top of targets and obtain precise locations. In fact, even if the target geo-location data provided to the rescue teams are relatively near the real target, it can be difficult for ground teams to find it rapidly. In mountainous regions for example, a simple error on the position (1 nautical mile for example) can lead to a lot of time lost by the SAR teams on the ground to travel this extra distance. So, the MOE suggested for this criterion is the precision of the geo-location system. This MOE is in agreement with the approach used to analyse the data of SAREX 2005 by Vincent [2]. In that memorandum [2], the location of the targets reported by SAR techs/spotters was compared to the actual GPS position of these targets. Eventually, these data could be compared to the location accuracy of the AIMS system Target confirmation time In the memorandum written by Vincent [9], the target confirmation time corresponds to the time between the detection of an element and the confirmation that this element is the object of interest. This can be done for example by seeing the identification (registration) number of the aircraft, distress signals from survivors, damages on the scene, etc. Two MOEs were suggested by Vincent [9]: the average time required between the detection and the identification of a target, and the average time required between the detection and the recognition of a target. For the operator the line is thin between the terms identification and recognition and it would be difficult to evaluate them separately. Consequently, the authors suggested that no distinction be made between the two terms. The MOE suggested would rather be simply the average time required between the detection and the identification/recognition of a target. The time will start as soon as someone sees a target but is not able to either identify or recognize it right away, and will stop when there is a confirmation by one member of the crew (usually a SAR tech or a spotter) that it is or not the target. The operation could imply a call-around or not. With this MOE, it will be possible to verify for example if it is preferable to have a reduced number of false alarms with 8 DRDC Valcartier TM

25 a longer average time to confirm or not a target, or on the contrary, a higher number of false alarms, but a shorter time for target confirmation. If the time of search is limited, it is possible that some targets are not well identified or recognized. To avoid that problem, enough time should be allowed to cover adequately the search area. Besides, the ability to provide information on a target will influence the target confirmation time. Two main factors influence this ability: the shape of the target and its colours. It is known, for example, that one main limitation of night vision goggles (NVGs) at night is the absence of colour. Consequently, the ability of the sensor suite to see colours and identify shapes could be an advantageous specification. A method to evaluate this ability is presented below. During SAREX search events, a comparison between the description of the targets reported by SAR techs/spotters and the actual description could be done according to a pre-determined chart. An example is given in Table 2. In this table, five points are accorded for a correct definition of each object and five points are given if one of the colours is provided. Since some colours are very similar when looking from above, we assumed that orange, yellow, pink and red are similar and can be confused. Points will be equally distributed between shapes and colours for a fair analysis. The idea is that it might be possible to determine if shapes have more, less, or equal influence than colours to identify objects. If for example, a new sensor provides zooming capabilities, shape might be easier to see and consequently the identification of the target could be faster. On the other hand, a sensor able to discriminate colours could have the same effect on the identification. For sure, the more accurate details SAR techs/spotters can provide, the better the chances are for rapid and correct identification of the target. Eventually, the results compiled from SAREX events could be compared to the results obtained by the operator using the AIMS system at the SAREX Table 2: Example of target characteristics compilation TARGET # OBJECT COLOURS TOTAL A Canoe (5pts) Green (5pts) 10 B Parachute (5pts) Blue - Orange - Green - Yellow - White (5 pts ) 10 C Dummy (5pts) Orange (5 pts) 10 TOTAL Number of call-arounds During a search, it is expected that many false detections would be made before finding the true search object. When a potential target is detected but cannot be rapidly confirmed or dismissed, a call-around is required. The total time spent on call-arounds depends on the number of such DRDC Valcartier TM

26 incidents and on the time required to do each call-around. During a SAREX search event, it also depends on the allocated time to fly the search area, which will limit the number of call-arounds that can be performed. Now, we are interested to know if the use of a SAR sensor suite will influence the number of targets to be identified during a search and if this will consequently change the number of call-arounds required. When searching for a specific vessel in a busy sea where there are many ships, usually multiple call-arounds are required to check out each vessel but, we expect a new sensor suite like AIMS to reduce this number. The MOE suggested previously by Vincent [9] was to determine the average number of false positives per unit of area by measuring the number of call-arounds. In this memorandum, the MOE is simply to determine the number of call-arounds. During the SAREX event, the number of call-arounds will be counted using the GPS history maps for each search and as previously presented by Vincent in [2], the number of call-arounds obtained should be divided into categories. Later, the data obtained will be compared with the results of a future search done with the AIMS system. Also, it is not necessary to measure the time spent doing call-arounds since it is generally related to the type of aircraft. However, since during the SAREX event the time given for the search limits the number of call-arounds, it is recommended that enough time be given to allow the crews to cover adequately the entire area False detection rates Another MOE developed by Vincent in [9] is the average number of false positives per unit of area. In another memorandum [2], this MOE was expressed as the number of false positives per unit of distance. In this memorandum, we will keep the average number of false positives per unit of area since in a SAREX search event, the task can be a track to follow or an area to search. It is easier to determine the area of a search by multiplying the track length by the width, than dividing the search area into corridors and then measuring the track lengths Probability of detection Vincent mentioned in his memorandum [9] that time loss due to missed targets can vary enormously between missions. For example, time can be lost if a target is not detected on the first fly over and the time interval before the next pass will depend on many variables (weather, fuel consumption, crew fatigue, etc.). So, it is really important to maximize the chance of seeing a target. One important measure given by Vincent [9] was the ability of a sensor to see the target when flying over and was quantified by the probability of detection. The probability of detection refers to the odds of detecting the target [10]. Besides, as discussed in [2][3], it is necessary to determine the number of targets required to demonstrate with a certain level of statistical confidence an increase in POD. The approach used to do that is explained in [2][3]. The Matlab program used to determine the confidence rate that a particular method is better than another one is given in Annex C and was provided by [11]. In Figure 2, it is possible to see the relationship between the probability of detection and the sweep width covered. 10 DRDC Valcartier TM

27 Figure 2: Sweep Width versus POD [10] At closer range, more targets are detected than at the maximum range except for the area below the aircraft. In that case, it is difficult for SAR techs/spotters to see right under the aircraft, but the use of a SAR sensor suite mounted on a platform located under the aircraft should avoid this problem. The new SAR sensor suite is thus expected to increase the POD within the same detection range or, if the POD is similar, then a larger detection range is expected from the use of this sensor suite. Therefore, a new MOE proposed is the maximum detection range of the sensor suite to be compared to the one of SAR techs/spotters searching with naked eyes. An additional issue influencing the PODs is reported in [3] and is related to the capability of the sensor to quickly scan the entire area being covered as the aircraft flies over. Since aircraft speed would be determined by the needs of spotters working with naked eyes, it is possible that a sensor with a limited field of view would be unable to scan the selected area as required, thus covering only a portion of it. Alternatively, a narrow FOV sensor might be able to sweep the entire area covered by the aircraft, but only at a high sweeping rate that lowers the chance of detection. DRDC Valcartier TM

28 Moreover, if the sweeping rate is too high, the images might be blurred. Such an inability to sweep an entire viewing area as the SAR aircraft flies by would essentially influence the time required to find a target by lowering the POD. Thus, during the SAREX event, it is possible to calculate the PODs by calculating for each team the number of real targets detected divided by the total number of targets to be detected. It should also be possible to determine from past SAREX the number of targets required to determine with a certain level of statistical confidence an increase in POD. For comparison purposes (between the PODs calculated for the past SAREX events and the PODs measured with the AIMS system at SAREX 2009), the same track space and search pattern should be chosen for all the SAREX flight trials and the aircraft speed be chosen according to the rules and regulations of that specific SAREX event Time lost due to reduced visibility The minimum visibility conditions where the sensor is still effective and the minimum illumination where the sensor is still operational were MOEs suggested in [9]. As discussed in [9], it is difficult to evaluate those parameters during flight trials since nobody controls the weather. Further, at the SAREX, it will be impossible to measure the time lost due to reduced visibility since the search is cancelled if the weather is bad. Moreover, at this event, there are actually no searches during night time, so the minimum illumination condition will also be difficult to evaluate during this same event. Therefore, instead of measuring the minimum and maximum visibility conditions and illumination during flight trials, the authors suggest a qualitative response to the following question: In what situations will the use of a new sensor provide improvement to the SAR missions? The answer should include the weather conditions, the time of day and the types of scenarios. The question could be addressed to a scientific team with expertise in the same field who would answer based on experience, experiments already done and on the potential of the AIMS system. This would provide an idea of the ability or effectiveness of the sensor suite to remain effective during different visibility and illumination conditions. Moreover, for each visibility and illumination conditions mentioned above, the maximum detection range of the sensor suite could be provided. The answer to the previous question and the detection range values could help determine the influence of reduced visibility and condition of illumination on the effectiveness of the searches during day and night. Moreover, even if weather conditions will have an influence on probability of detection, it will not be evaluated since as said previously, during the SAREX no searches take place during bad weather and thus, no comparison is possible in those conditions. However, if possible, some future flight trials should be conducted under different weather conditions and at night time to demonstrate the improvements provided by the use of a new sensor suite like AIMS. In short, due to harsh environmental conditions, searches can be stopped or simply delayed with the consequent time lost, but the use of a new SAR sensor suite could improve the effectiveness of the search in those conditions. 12 DRDC Valcartier TM

29 2.3.8 Ability to disseminate information on a target The ability to disseminate information on a target is related to the performance of the communications link to transmit images [9]. Vincent [9] proposed to report the presence or absence of a reasonable image transmission capability and mentioned that image dissemination is not of primary concern for the SAR role. The presence of this capability on a SAR system could be considered as an asset and could be used to compare different SAR sensor suite together. 2.4 Summary This sub-section presents a summary of the MOEs proposed for the evaluation of the AIMS system and establishes which flight trials would be adequate to measure them (see Table 3). For some MOEs, additional information is provided in [9]. Table 3: Summary of the possibility to evaluate MOEs during different flight trials MOEs SAREX 2009 FLIGHT TRIAL OTHER FLIGHT TRIALS GROUND TRIALS 1. The alignment error between the actual location of the target and the one given by the tracking device. 2. The maximum time that the sensor is able to stay on the target. 3. The average time it takes for the sensor to recover the target when loosing it. Yes Yes No Yes Yes No Yes Yes No 4. The time from fly-over to detection. No Yes No 5. The time necessary to plan an optimal search. 6. The number of hours of experience as a SAR tech/spotter (spotting with naked eyes) and as an operator using the sensor suite. 7. The maximum time that the sensor suite can be used without significant operator fatigue. 8. The maximum flight velocity for the operator to be able to analyse effectively the data in real time. Yes Yes Yes Yes No No No Yes Yes No Yes Yes DRDC Valcartier TM

30 9. The precision of the geo-location system. Yes Yes Yes 10. The average time required between the detection and the identification/recognition of a target. Yes Yes No 11. The number of call-arounds. Yes Yes No 12. The average number of false positives per unit of area. Yes Yes No 13. The probability of detection. Yes Yes No 14. The maximum detection range of the sensor suite. Yes Yes No Also, in this section some other criteria were mentioned and considered as assets for a SAR sensor suite, they are: 15. The presence of a capability to store captured images. 16. The ability of the sensor suite to see colours and identify shapes. 17. The ability or effectiveness of the sensor suite to remain effective during different visibility and illumination conditions. 18. The presence of a reasonable image transmission capability. The following section presents a summary of the SAREX 2006 event and the data collected during this event. These data will be compared to the data collected during future flight demonstrations with AIMS and therefore, will be helpful to demonstrate the utility of the AIMS system during SAR operations. 14 DRDC Valcartier TM

31 3 SAREX 2006 Event This section presents a summary of the SAREX 2006 event that was hosted by 424 SQN (Trenton) and took place in North Bay (Ontario). An analysis of the data collected at the SAREX 2006 is provided. Results from CASARA teams were not analysed since for now, the AIMS system is indented to be installed on military aircrafts only. 3.1 Summary of SAREX 2006 search event During the search event, each team receives a brief verbal tasking with the same SAR scenario (detailed in Annex B) and a map (1: scale) of the search area with entry and exit points. As soon as they receive the brief, the crew has 30 minutes to take off. Table 4 contains general information on the teams. During the search event, the crew usually flew the aircraft at normal search speed and as a general guidance, time spent searching was not to exceed 60 minutes of flying time [12]. Table 4: Teams participating to the SAREX 2006 search event TEAM SQUADRON HOME BASE AIRCRAFT SAREX Greenwood C130 - Hercules SAREX Comox C115 - Buffalo SAREX Winnipeg C130 - Hercules SAREX4 (Meunier) 424 Trenton C130 - Hercules SAREX 5 CSS Bagotville CH146 - Griffon SAREX Bagotville CH146 - Griffon SAREX Gander CH146 - Griffon 417 SQN 417 Goose Bay CH146 - Griffon 424 SQN (Duffy) 424 Trenton CH146 - Griffon Each crew was tasked to find as many targets as possible in the search area. The targets are described in Table 5. However, given the fact that they were encouraged to over-report, i.e. they report targets that they would probably not report in a real search and the fact that crews use as many spotters as possible during the search flight, teams increase their chance of finding targets when doing the search event compared to a real search of a SAR incident. DRDC Valcartier TM

32 Table 5: Description of the targets TARGET # POSITION DESCRIPTION COOPERATIVE TARGET 23 A N W B N W C N W D N W E N W F N W G N W H N W I N W Green canoe Parachute in trees: orange, green, yellow, white Yellow bicycle with orange dummy Two orange/yellow dummies with orange life preservers Pink and orange 16 feet panel markers, in arrow figure Multi-coloured parachute Panel marker, visible fore and aft only White parachute with orange dummy in trees Orange and green dummy with yellow bicycle NO NO YES YES YES YES NO NO YES Figure 3 presents the map of the search area [13]. The path to follow is represented by a red line starting on the left and ending on the right; the position of each target is represented by a black triangle. The last target corresponds to the Emergency Locator Transmitter (ELT) position. 2 A cooperative target is a target placed by survivors in an effort to be seen [2]; it could also be a target easily seen from the air. 3 Passive target can be defined as a target that result directly from a SAR incident [2]. 16 DRDC Valcartier TM

33 Figure 3: Map of the search area [13] 3.2 Target location accuracy As reported in [2][9], the target location accuracy is relevant for the evaluation of SAR sensors. As a matter of fact, reporting the target location accurately is relevant for the rescue team. For example, a helicopter can be tasked to rescue people at a location reported by a search team. In that case, it is very important for the helicopter crew to pinpoint the target as fast as possible when they arrive over the area. Also, for ground teams, the location is very important considering that sometimes an object seen from the air is not easy to find from the ground perspective. As done in [2], the sighting and the actual GPS position were compared to evaluate the target location accuracy. Figure 4 presents the data collected at the SAREX 2006 and compares the actual GPS position of the targets [14][15] and their reported position. It appears that most DRDC Valcartier TM

34 sightings (66%) are identified at less than 0.3NM (0.556km) of the truth location, which would probably allow the rescue assets to find the target s position in a reasonable time. However, even if the location given is near to the actual GPS position, it might take time for the rescue teams to find the target Number of targets Error (NM) Figure 4: Number of targets detected within specific distance (NM) Since the AIMS system is equipped with a geo-positioning system; the flight trials with AIMS should demonstrate the precision of the target location. 3.3 Target confirmation time For SAREX 2006, it was not possible to measure the average time required between the detection and the identification/recognition of each target. However, as discussed in section 2.3.3, the ability to see colours and to identify shapes influences the target confirmation time and for SAREX 2006, it was possible to evaluate this ability. The data collected at this event were analysed using Table 6. In this table, five points were given for each object well identified and another five points if at least one exact colour was given. When the object was not detected, or wrongly identified, or only wrong colours were given, then no point was awarded for the item. 18 DRDC Valcartier TM

35 Table 7: Evaluation of the accuracy of the description of the targets given by the teams at the SAREX 2006 search event TARGET # 103 POITRAS 413 DORT 417 JANKE 424 DUFFY 424 MEUNIER 435 DORT 439 SIKET 442 EDWARDS CS SIKET SHAPE / COLOUR SHAPE / COLOUR SHAPE / COLOUR SHAPE / COLOUR SHAPE / COLOUR SHAPE / COLOUR SHAPE / COLOUR SHAPE / COLOUR SHAPE / COLOUR A B C D E F G H I TOTAL: # PTS % (max=160) % TOTAL DRDC Valcartier TM

36 Table 6: Points allocation TARGET # OBJECT (5pts) COLOURS (5pts) MAX. POINTS ALLOCATED A Canoe Green 10 B Parachute Orange - green - yellow - white 10 C Bicycle Yellow 10 Dummy Orange 10 D Life preserver 1 Orange 10 Life preserver 2 Orange 10 Dummy 1 Orange - yellow 10 Dummy 2 Orange - yellow 10 E Panel marker Pink - orange 10 Arrow figure 5 F Parachute Multi-coloured 10 G Panel marker Yellow 10 V figure 5 H Parachute White 10 Dummy Orange 10 I Bicycle Yellow 10 Dummy Orange - green 10 TOTAL 160 The accuracy of the description given by each team who participated to the SAREX 2006 search event was evaluated using Table 6. The results obtained are presented in Table 7. DRDC Valcartier TM

37 Actually, if we plot the percentage obtained by each team for a good identification of colours and shapes, we obtain the following figure: 35 Colour % Poitras Dort Janke Duffy Meunier Dort Siket Edwards CS - Siket Equivalence Shape % Figure 5: Percentage obtained for good identification of shape versus percentage obtained for identification of at least one good colour By looking at Figure 5, at first it seems that SAR techs/spotters identify better the colours than the shapes, but more data will be needed to confirm this hypothesis. If it shows in future reports that it is true, then it could be more important for new sensors to be able to discriminate colours than identify shapes. Also, with the data obtained using AIMS at SAREX 2009, we should be able to verify the influence of colours and shapes on detection when sensors are used. 3.4 Number of call-arounds GPS coordinates of the tracks were recorded for the majority of the search paths during SAREX 2006 [16]. The free trial version of Oziexplorer [17] allowed us to create a visual map of the aerial track flown by the search unit. The objective was to compare the number of call-arounds made on detected targets, on missed target and for nothing. Figure 6 shows an example of the path seen from above. It is possible to count the number of call-arounds conducted on the track and link them to a real target when it is the case (real targets are represented by a white flag). Annex D contains all the other maps recorded for this search event [16]. DRDC Valcartier TM

38 Figure 6: Flight 9 Faithfull 42 SAREX 2: Top View [16] Figure 7 presents the same search but from an elevated view. It is interesting to see in this figure that sometimes, the aircraft comes closer (in altitude) to the target to help its identification. A new sensor with zooming capabilities would probably reduce the need to do that. 22 DRDC Valcartier TM

39 Figure 7: Flight 9 Faithfull 42 SAREX 2: 3D-Elevated View [16] Table 8 contains the number of call-around occurrences compiled during SAREX The data are not available for all teams since the tracks were not recorded for all search exercises. The table includes the total number of call-arounds, including multiple call-arounds on a single target (since time is lost on all call-arounds). Table 8: Number of call-arounds TEAM ON DETECTED TARGETS ON MISSED TARGET FOR NOTHING TOTAL Janke Duffy SAREX SAREX 5 -CSS - Siket SAREX Siket AVERAGE DRDC Valcartier TM

40 In Table 8, note that 2/3 of all call-arounds are done for nothing. In amount of time, it represents a lot of time lost. The number of call-arounds carried out by a search team depends on different factors including the level of activity in a search area [9], the experience of spotters and the type of aircraft. Moreover, we need to consider that during the search event, because they are constrained to one hour of search, the aircrew cannot do as many call-arounds as they would do during a real search. So, for this MOE, it would be preferable if the data were collected during a search where the time would be sufficient to do the number of call-arounds needed to perform a good search. 3.5 False detection rates As previously discussed in [2][9], measuring the false detection rates might be important in the evaluation of SAR sensors. Reporting false targets can lead to a loss of time, especially when they lead to call-arounds to verify the information. Table 9 provides a summary of the false detection rates obtained at the SAREX 2006 search event. The track length of the SAREX 2006 search was NM (see Figure 3) and it was assumed that the SAR techs/spotters could cover approximately 1NM on each side of the aircraft. Table 9: False Detection TEAM #FALSE TARGETS REPORTED #FALSE TARGETS PER NM 2 SAREX SAREX SAREX SAREX SAREX SAREX 5 - CS SAREX Janke Duffy AVERAGE Figure 8 shows a comparison between the average of false target rates reported/nm 2 from 1988 to 1991, 1993 and 2005 to The average number of false targets reported by teams at the 24 DRDC Valcartier TM

41 SAREX event 1988 to 1991 and 1993 were extracted from [3] and for SAREX 2005 data were taken from [2] Average of false target reported/nm^ False detection rates Year Figure 8: False detection rates/nm 2 In general, the false detection rates seem to vary significantly. However, as mentioned in [3], the Trenton (1989) event seems to have encountered higher false detection rates, which were mainly attributed to higher human activity around the search area. According to Vincent [3], the way the false target detections are actually collected do not represent a good measure to evaluate spotting performance. He mentions several weaknesses of false detection rates: the fact that spotters are encouraged to over report sightings, targets do not necessarily look like real targets (orange panels for example), and presence of clues such as broken top trees are disregarded by the spotters. However, false detection rates could be compared for the same SAREX event, because the same weaknesses would apply for all searches performed during that specific event. One interesting figure to look at is the possible relationship between the number of detections and the number of false detections. That is to say, do more sightings lead necessarily to more detection of targets? Figure 9 shows the average number of detections at SAREX 1988 to 1991, 1993, 2005 and 2006 compared to their corresponding average number of false detections. On one hand, the number of real targets detected seems relatively constant at five targets/team for each SAREX. On the other hand, the number of false targets reported does not seem to be correlated to the number of real targets detected. Many factors including the number of spotters, the experience and judgment of spotters, human activity in the search area, the fact that the types of targets change from one year to another, the environment where the targets are placed which is different from year to year and the type of aircraft/helicopter might influence the number of false DRDC Valcartier TM

42 detections. Again, due to the uncertainty on the false detections, it is difficult to establish a relationship between real and perceived targets AVERAGE FALSE TARGET REPORTED AVERAGE TARGET DETECTED Comox Trenton Edmonton Comox Trenton PEI North Bay Figure 9: Comparison between real target detected and false target detected In short, considering that the false detection rates vary significantly between SAREX events, false detection rates obtained in the context of a SAREX event cannot be compared to values obtained during another SAREX event or during other flight trials, because of the weaknesses mentioned earlier. Also, actually it is not possible to establish a relationship between the number of detections and the number of false ones. In that regard, a more qualitative and subjective evaluation of false target detection could possibly be helpful and was suggested by Vincent [3]. Two effects on false detection could be expected with the use of the AIMS system: on one hand, there will probably be more objects detected with the AIMS system due to its thermal (warm objects will be detected) and active imager (reflective objects will be seen) capabilities; on the other hand, the resolution of the AIMS sensors should allow the spotter to confirm rapidly whether it is a real target or not. Thus, as a general guidance, data collected on false detection when flying with the AIMS system at SAREX 2009 event, should only be compared with the data compiled when searching visually at this same event. Future trials will tell us if more false objects are detected when using the AIMS system and if it takes more time overall to cover the same area because of the additional time required to identify a higher number of false targets. 3.6 Time lost due to a missed target The first section presents some results on the PODs and many factors influencing those PODs will be discussed below. Then, a short analysis is done to evaluate the influence of the familiarity 26 DRDC Valcartier TM

43 of the environment on the POD. Thereafter, a comparison between the data collected during SAREX 2006 and other SAREX events on the number of targets needed to demonstrate with a certain level of statistical confidence that an increase in POD is made Probability of detection OR scientists at the former Air Transport Group Headquarters (ATGOR) [4][5][6][7][8] and at DRDC Valcartier [2][3] attended the past SAREX search events and collected data on the PODs of the various targets. One of the authors also participated in SAREX 2006 and collected data to improve the sample size. The results obtained are presented next. Table 10 summarizes the PODs of different targets at SAREX Table 10: Estimated PODs of SAREX 2006 TARGETS ATTEMPTS DETECTIONS FRACTION DETECTED Panel markers Two dummies with life preserver Cooperative parachutes ALL COOPERATIVE TARGETS Passive parachute Bicycle with dummy Canoe ALL PASSIVE TARGETS Many documents on previous SAREX [2][3][4][6][7][8] demonstrated that the POD for passive targets is significantly less than that for cooperative targets. In Figure 10, the estimated PODs for cooperative and passive targets with confidence intervals of 90%, 95% and 99% are given for SAREX 2006 only. In Figure 11, one can observe the PODs for SAREX 1988 to 1991, 1993 and 2004 to 2006 with confidence intervals of 90%, 95% and 99%. The mean value obtained for SAREX 2006 is similar to the one obtained for the previous SAREX events. However, the lengths of the confidence intervals are larger for SAREX 2006, which is normal since less data were compiled. DRDC Valcartier TM

44 Cooperative Passive Probability of detection (POD) 99% 95% 90% confidence intervals Figure 10: Estimated PODs for cooperative and passive targets for the SAREX 2006 Cooperative Passive Probability of detection (POD) 99% 95% 90% confidence intervals Figure 11: Estimated PODs for cooperative and passive targets for the past SAREX (1988 to 1991, 1993, 2004 to 2006) Again, the results demonstrate that passive targets are harder to find than cooperative targets. Figure 12 displays the PODs for some types of cooperative and passive targets obtained in the past SAREX 1988 to 1991, 1993 and 2004 to 2006 and also shows the average of the PODs obtained with all the cooperative targets together (64%) and the average of the PODs obtained with all the passive targets together (21%). 28 DRDC Valcartier TM

45 Marker panels Parachutes (coop.) Dinghies All coop. targets Aircraft parts Thermal blankets Parachutes (pass.) All passive targets Probability of detection (POD) 99% 95% 90% confidence intervals Figure 12: Estimated PODs for different types of cooperative and passive targets for the past SAREX (1988 to 1991, 1993, 2004 to 2006) A summary of the factors that might influence the PODs were found in several documents [2][3][4][5][6][7] and are explained below: Weather and time of day (clouds, lighting, fog, precipitation); the weather might be different each year and might be different for different teams. The search event location is different each year, therefore, the background contrast and the environment change also. Search and aircraft parameters (size of the area to search, type of aircraft, airspeed, altitude, track spacing, sweep width, time flying). Condition of the crew (crew fatigue, crew motivation, experience of the crew members is not constant within a team and between the teams. Number of spotters might change between aircrafts, there are usually more spotters during SAREX searches than there are during real SAR incidents and spotters search for the entire hour during the search rather than alternate every minutes as is standard practice. Target characteristics: each year the number and type of targets change. It is thus difficult to establish a solid baseline for comparison. Subjectivity of the scientist to class a target as passive or cooperative when calculating the PODs. Ability of the sensor to sweep the entire area as the aircraft flies over. According to [4][5][6], the familiarity with the environment also has an influence on the POD. To help resolve this question, the next sub-section presents a short analysis. DRDC Valcartier TM

46 3.6.2 Familiarity with the environment The winners for each year of competition and the region where the competition took place are compared in Table 11. The aim of this comparison is to verify if the familiarity with the environment has an influence on PODs. Table 11: Winner versus the region where the SAREX event was held WINNER TEAMS PARTICIPING YEAR 442 Sqn (Comox, BC) Sqn (Trenton, ON) Sqn (Edmonton, AB) Sqn (Greenwood, NS) Sqn (Greenwood, NS) HOSTED BY... IN 442 in Comox, BC 424 in Trenton, ON 435 and 440 in Edmonton, AB 442 in Comox, BC 424 in Trenton, ON 442 (Comox, BC) and 413 (Greenwood, NS) in Comox, BC 444 Sqn (Cold Lake, AB) CSS ( ) in PEI 424 Sqn (Trenton, ON) CSS (439) in North Bay, ON As shown in Table 11, for SAREX events 1991 and 1993, the winner and the hosting teams were different. In 2005, no team was coming from Prince Edward Island so this event could not be considered in this evaluation. Finally, for 2006, there was no squadron based in North Bay so we could not consider this event in the evaluation as well. Thus, six events remain, for each of them, there were six teams participating and for four of the cases, the winner was from the region where the event was held. Therefore, the objective of the demonstration is to determine the probability that home teams won four of the six events. First, the probability that one team won one event was 1/6 and that one team looses was 5/6. Since we want to determine the chances that home teams won four of the six events and considering that the order (win and loose) of the events does not matter, Equation (1) extracted from [18] gives the number of possible combinations: 30 DRDC Valcartier TM

47 C r n n! = (1) r!( n r)! Where n corresponds to the number of events (n=6); and r corresponds to the number of events won by home teams (r=4). Thus, the home team had 15 different possibilities to win four times on six. For each possibility, Equation (2) provides the chances of winning four times on six: = = (2) The probability that the home team won four times is thus given by Equation (3): = (3) The result means that for the six events, there was only 0.8% of chance that four home teams won the SAREX search event. Since it happened, it is possible to conclude with very high confidence that the familiarity with the environment has direct influence on the POD Number of targets The approach developed to determine the number of targets necessary to demonstrate with a certain level of statistical confidence an increase in POD was explained in [2][3]. Figure 13 shows a comparison between the results obtained using the SAREX 2006 data and past SAREX 1988 to 1991, 1993, and 2004 to 2006 search events data (which provide more data to compile). Figure 13: Detection rates required to demonstrate improvements with 95% confidence using SAREX 2006 and past SAREX search events data DRDC Valcartier TM

48 The POD for the passive targets of SAREX 2006 was The POD for all passive targets in past SAREX events was Using those PODs and using the Bayesian/Monte-Carlo approach [2][3], it was possible to build the curves in Figure 13. As mentioned in [2], the trend is not continuous because of the limited number of discrete values that are suitable to calculate the detection rates. As explained in [2], the Figure 13 informs that if AIMS is expected, for example, to have a POD for passive targets of 75%, the SAREX search event would be appropriate to show its superiority if at least four passive targets were employed (using the curve of SAREX 2006 in this example). In the case where we might not know the POD expected for passive targets using AIMS but that we know the number of passive targets, we can deduce the minimum detection rate required. Nevertheless, Figure 13 is only an indication of the detection rates required versus the number of targets. First, discrete values are used to compile the data which makes the curve irregular; second, the results from one event can only be compared to the results obtained with the AIMS system at this same event when the conditions for comparison will be the same and finally, we don t know yet the POD for passive targets at SAREX 2009 where AIMS will be participating [2][3]. So, it is not recommended to use the curve obtained with all past SAREX data in Figure 13. Few conclusions can be drawn from the analysis of the PODs of SAREX 2006 data. First, we need to consider that PODs are not the same for passive and cooperative targets and unfortunately, the sample size of passive targets is usually smaller than that of cooperative ones. To resolve this problem, in future SAREX events, it will be important to increase the number of passive targets. Therefore, since the type of targets and the environment where they are installed change very often, it is highly recommended to define a number of targets that will be used each year during the SAREX and to locate them in similar environments. Second, the distribution of targets along the path or in the search area could be studied to see if it has an influence on the PODs. For example, if many targets are near each other and are located at one of the search corners, the SAR techs/spotters might be too busy to see them all. On the other hand, if they are uniformly distributed or spaced, it might be easier to see them. In any case, the locations where the targets are placed depend usually on the space available for the helicopter to land and on the background required. Third, as Fournier said in his document [19], the sequence of passive and cooperative targets could be studied. For example, is it better to place one passive target followed by a cooperative target, which would imply an equal number of each type of targets, or would it be better to do the selection randomly? The last suggestion is to involve a scientist in the selection of targets (for example, to choose the type of targets, the number of passive versus cooperative targets) and the positioning of these targets in the search area (for example, background contrast, type of search and position in the search area). Finally, a very important aspect to consider when comparing the results obtained with the use of AIMS is the fact that only two or three people will be onboard the Twin Otter, including the pilot and the operator using the system, as opposed to a SAREX team where there are at least two spotters on board. 32 DRDC Valcartier TM

49 4 SAREX 2007 Event The OR analyst supporting the AIMS project was also actively involved in SAREX 2007 held in Goose Bay. Throughout the year, a close collaboration was established with the SAR community [20]. Thus, the first part of this section explains this collaboration and the second part suggests a list of data to be collected at the next SAREX. 4.1 Collaboration with SAREX community Close collaboration in the selection of permanent targets was established with the SAR community [20]. As a result, seven permanent targets were chosen and will be used at each future SAREX search event. Annex E contains the letter of authorization for this project. On one hand, this will allow the evaluation of the effectiveness of spotters on the same basis each year and the assessment of the effectiveness of training. On the other hand, these new targets will also be useful to evaluate new techniques and equipments like AIMS. Therefore, these added targets will offer a good opportunity for the flight trials with the AIMS system to compare its performance to that of unassisted spotters. The choice of the targets is based on actual search missions. The following targets were chosen to be part of the set of reference targets for SAREX 2007 [20], but depending on the material availability and other unpredictable events, targets could be slightly modified at each SAREX search event: TARGET 1: Panel with letters on it (for example: SOS) with a reasonable height. TARGET 2: Intact fuselage with identification through trees (target smaller than a real fuselage but will have the same coating as a real one; target has to be small since it is carried by helicopter). TARGET 3: Fuselage ruined/burned (it will be a matte black frame with pipes, maybe one or two small panels to represent a seat). It happened in the past that a crash caused only a flash of fire, leaving no trace of the accident except a bundle of pipes. TARGET 4: Bundle of small targets (parachute, dummy, reflective tape, icebox) dispersed in either a swamp area, open area, or where trees are cut (to simulate the action of wild animals dispersing objects). TARGET 5: Floating debris (for example, floating life preserver, reflective strip with letters, geometric shapes). TARGET 6: Object under the surface of the water (panel with reflective strips submerged at 1 foot or 2 feet with 4 life buoys to maintain its level of floatability). TARGET 7: Drowned person (a person in t-shirt and jeans floating in the water). DRDC Valcartier TM

50 4.2 Data collection guidelines For future SAREX, it is recommended that data collection include the following: Joint Rescue Coordination Center (JRCC) tasking (an example is provided in Annex B). The sheet results from the SAREX search events (including the description of the search, flight sheets, scoring, maps of the search, etc.). It is not necessary to collect the CASARA data since the system is actually designated for military aircraft, only SAREX data will be used in a future comparison with the AIMS system. More information on the targets (description, position and environment setting). Getting pictures of the targets would help to decide whether it is a passive or a cooperative target. Information related to the environment in general (mountainous region, colour of the trees, level of human activity, etc.). The rules and regulations books, the participant book and each day to day schedule of the events. Weather reports. Moreover, at this event, the OR analyst should be sure that a GPS device is on board each aircraft (see CASARA chief judge) and that the tracks of each flight are well identified when the results are recorded. Finally, it is important to mention that since SAR techs/spotters will probably learn and pass on the information concerning the types of permanent targets from SAREX to SAREX, it might be possible that it increases their chances of finding these targets from year to year. This should be considered when analysing the results at SAREX Consequently, it is recommended that the approach of permanent targets be re-evaluated after SAREX DRDC Valcartier TM

51 5 Scenarios for the flight trials Due to the fact that SAREX teams fly neither under bad weather nor at night, it will be more difficult to demonstrate the performance of AIMS in those conditions. So, in order to be able to evaluate the performance of AIMS in all situations, we need to demonstrate its usefulness at night, during foul weather conditions and for different missions. To help in the selection of the scenarios, a general approach, defining the objective of the mission, the choice of the target and the environment conditions (defined into four sub-groups), is proposed. For the evaluation of the AIMS system, the objectives of the mission are Surveillance and Reconnaissance or Search and Rescue. The types of target should be related to the mission, it could be for examples an intact fuselage, a burned fuselage, a ship in distress or lost hunters. Finally, the environment could be composed of four sub-groups: 1) season, 2) time of the day or night, 3) type of terrain (over the ground, the sea, in mountainous region, open area, leafed, conifers, tundra, etc.) and 4) meteorological conditions (fog, rain, snow, etc.). Examples of scenarios that could be used to evaluate the AIMS system are provided below: SCENARIO 1: Surveillance and Reconnaissance of a boat entering Canadian waters. The identification of the boat is needed during the night. 1. Objective: Surveillance and Reconnaissance of a boat entering Canadian waters 2. Type of target: 50 feet boat 3. Environment: a. Season: Winter b. Time of the day or night: 22h00 c. Type of terrain: Sea d. Meteorological conditions: Snowing SCENARIO 2: Two canoeists on an expedition on the Ashuapmushuan River. It is asked to start the search in the night and to continue during the next day or until they find them. 1. Objective: Search and Rescue 2. Type of target: Two people, floating debris (for example, canoe, paddles, life jacket and camping equipment) 3. Environment: a. Season: Summer DRDC Valcartier TM

52 b. Time of the day or night: 19h00 (sunset time) c. Type of terrain: River, shore and woods. d. Meteorological conditions: Raining SCENARIO 3: A white and red helicopter crashed in a mountainous region of British Columbia. The two pilots and their passenger are reported missing. The search will start during the day and will continue during the night. 1. Objective: Search and Rescue 2. Type of target: 212 Helicopter 3. Environment: a. Season: Autumn b. Time of the day or night: 8h00 c. Type of terrain: Mountainous region d. Meteorological conditions: Sunny These scenarios are provided as examples and would probably change depending on the number of flight trials to be held, on the weather conditions during the flight trials and on the region where the flight trials will be performed. More information will be needed to establish a more detailed plan for flight trials with AIMS. 36 DRDC Valcartier TM

53 6 Conclusion The most important parameter in evaluating the performance of sensing equipment like AIMS in SAR searches is its ability to find the search object as quickly as possible. The probability of finding survivors after a crash is reduced by half every period of 8 hours [21]. To effectively assess this performance, criteria have been developed to evaluate the AIMS system, the operator and the performance of AIMS with the operator. This last criterion however do not identify whether the results are mostly attributed to the performance of AIMS and/or to the performance of the operator. The evaluation criteria contain a list of measures of effectiveness (MOEs) and specifications. The AIMS MOEs are summarized below: 1. The alignment error between the actual location of the target and the one given by the tracking device. 2. The maximum time that the sensor is able to stay on the target. 3. The average time it takes for the sensor to recover the target when loosing it. 4. The time from fly-over to detection. 5. The time necessary to plan an optimal search. 6. The number of hours of experience as a SAR tech/spotter (spotting with naked eyes) and as an operator using the sensor suite. 7. The maximum time that the sensor suite can be used without significant operator fatigue. 8. The maximum flight velocity for the operator to be able to analyse effectively the data in real time. 9. The precision of the geo-location system. 10. The average time required between the detection and the identification/recognition of a target. 11. The number of call-arounds. 12. The average number of false positives per unit of area. 13. The probability of detection. 14. The maximum detection range of the sensor suite. This list results from MOEs previously published in [9], from modifications done to some of them and from additional MOEs proposed in this document. Also, in this memorandum other criteria were mentioned and considered as assets for a SAR sensor suite. They are: DRDC Valcartier TM

54 15. The presence of a capability to store captured images. 16. The ability of the sensor suite to see colours and identify shapes. 17. The ability or effectiveness of the sensor suite to remain effective during different visibility and illumination conditions. 18. The presence of a reasonable image transmission capability. In addition to the MOEs and specifications listed above, this memorandum provided a summary on how to collect and analyse the data, and suggested types of flight trials to collect them. Data collected at the SAREX 2006 search event were analysed and increased the sample size already obtained from past SAREX events. It was shown that flight trials during SAREX can possibly demonstrate the performance of the AIMS system. However, some factors will have to be considered before a conclusion is drawn from the comparison between the performance of AIMS versus that of unassisted spotters at the SAREX 2009: 1. The aircraft on which the system will be installed is a Twin Otter and due to the lack of space in the aircraft, only the pilot, the technician and maybe one spotter will be on board. To make a fair comparison between a standard military flight at the SAREX 2009 and another flight with AIMS, the number of spotters on board should be the same. 2. Data will be collected for only one flight using the AIMS system and they will be compared to the data collected during five or six SAREX flights. Therefore, the data collected by the SAREX teams when combined altogether are more statistically valid than those collected by AIMS. 3. No trial is planned during night time or in bad weather conditions and it is in those situations that the AIMS system should be the most useful. 4. Usually, the majority of the targets are cooperative whereas the AIMS system has sensors like a thermal imager and an active imaging device which would be very helpful to find passive targets. Despite all these factors, if the AIMS system demonstrates equal or better performance than the search with spotters, it will demonstrate its usefulness for the Canadian Forces (CF). On the other hand, if the system has a bad performance, it will not be possible to conclude on the utility of the system for the CF. In that case, other flight trials will be needed to conclude on the usefulness of the system for the CF. A strong collaboration was established between the AIMS project and the SAR community. One of the results of this collaboration is that seven permanent targets were chosen and will be installed in all future SAREX search events. This new set of targets will help evaluate the effectiveness of spotters on the same basis each year, will be a training tool for SAR techs and will be useful to evaluate new techniques and/or equipment. Thus, these additional targets will offer a good opportunity for the flight trials with AIMS to compare its performance to the one of unassisted spotters with a more stable base. However, since SAR techs/spotters will probably 38 DRDC Valcartier TM

55 learn and pass on information concerning the permanent targets, it might be possible that this will influence their chances of finding these targets. This fact should be considered when analysing the results at SAREX Consequently, the approach of permanent targets should be reevaluated after SAREX The authors suggest that the OR analyst continues to be involved in the SAREX and continues to collect data until the flight trials with AIMS. Also, a suggestion to the SAREX committee could be to introduce a new facultative search at night time with NVGs. To do that, a target could be installed in a specific area and given this area, the time taken by each team to find this target could be measured. Finally, it is suggested that the OR analyst develop new MOEs to evaluate the knowledge and the effectiveness of the operator using the AIMS system. In the future, these MOEs could help the scientific team on the AIMS project to develop a training program for operators. DRDC Valcartier TM

56 References... [1] Larochelle, V. (2007), AIMS TD - Advanced Integrated Multi-sensing Surveillance Technology Demonstration, Defence R&D Canada Valcartier. [2] Vincent, E. (2006), Searching performance at the 2005 National SAREX, (DRDC Valcartier TM ) Defence R&D Canada Valcartier. [3] Vincent, E. (2005), Using SAREX Search Events to measure searching performance, (DRDC Valcartier TN ) Defence R&D Canada Valcartier. [4] Taylor, I. W. and Mack, I. C. (1988), A study of probability of detection using the 1988 search and rescue exercise, ATGOR Staff Note 6/88, Operational Research Branch, Air Transport Group Headquarters, CFB Trenton. [5] Vigneault, M. and Young, P. J. (1989), A study of detection capability using the 1989 search and rescue exercise, ATGOR Staff Note 7/89, Operational Research Branch, Air Transport Group Headquarters, CFB Trenton. [6] Vigneault, M. and Frank, G. (1990), Probability of detection analysis from SAREX 1990, ATGOR Internal Working Paper 2/90, Operational Research Branch, Air Transport Group Headquarters, CFB Trenton. [7] Vigneault, M. and Gammon, M. (1992), A study of probability of detection using the 1991 search and rescue exercise, ATGOR Working Paper 10/92, Operational Research Branch, Air Transport Group Headquarters, CFB Trenton. [8] Christopher, G. and Fournier, P. (1993), Probability of detection in SAREX 93, ATGOR Working Paper 15/93, Operational Research Branch, Air Transport Group Headquarters, CFB Trenton. [9] Vincent, E. (2006), Measures of effectiveness of airborne search and rescue imaging sensors, (DRDC Valcartier TM ) Defence R&D Canada Valcartier. [10] (2000), National Search and Rescue Manual, B-GA /FP-001 DFO 5449, Original May [11] Vincent, E. (2006), CEFCOM, Private communications. [12] National SAREX planning Committee (2006), SAREX Rules & Regulations Book, North Bay, SAREX [13] Bischoff, P. (Capt.) (2006), 1 Canadian Air Division Headquarters, Private communications. 40 DRDC Valcartier TM

57 [14] Copyright 2005 Satellite Signals Ltd, Degrees, Minutes, Seconds to Decimal Degrees calculator (online), Johnston E., (Access date: ) [15] This lat, long, bearing and range calculator is copyright 1999, 2004 Satellite Signals Ltd, Great circle azimuth bearing and range calculator (with magnetic north) (online), Johnston E., (Access date: ) [16] Taylor, D. E. (2006), CASARA Chief Judge, Private communications. [17] Des Newman s OziExplorer GPS Mapping Software, Newman, D. and L., (Access date: ) [18] Hayslett, H. T. (1968), Statistics Made Simple, Made Simple Books, Published by Doubleday, p. 51. [19] Fournier, P. (1994), Proposal for the experimental design of the search event in SAREX 94, Unpublished, Defence R&D Canada Valcartier. [20] Hotton, A. (Sgt) (2006), 14 Wing, Private communications. [21] Larochelle, V., Mathieu, P., Simard, J.-R. (1999), Two generations of Canadian Active Imaging Systems: ALBEDOS and ELVISS, SPIE vol3698: Infrared technology and application XXV, paper 31. [22] A3 Disp op SAR (2006), Guide de planification de l exercice national de recherche et sauvetage, BPR : A3 Disp op SAR/QG 1 DAC (1re Division aérienne du Canada), version 1.0. DRDC Valcartier TM

58 This page intentionally left blank. 42 DRDC Valcartier TM

59 Annex A SAREX Events description Usual SAREX events [12][22] are explained below : Bell Ringer: This is an activity which is not scored in the overall score and is restricted to people older than 40 years. The jumper has to carry all his equipment from the landing point to the bell. The winner is the one who took the least time from the landing point to ring the bell. Parachute accuracy event: When landing on the ground, the feet of the jumpers have to land as close as possible to the centre of a 10 meter radius circle. If the landing is done outside the 10 m circle, the distance recorded will be 10 m. Medical event: Each team is composed of three SAR techs that are evaluated according to their capability to triage, evaluate, treat, and deliver all survivors to a medical facility according to the Search and Rescue Pre-Hospital Protocols and Procedures. Search event (including CASARA aircraft): During this event, each team receives a map of the zone to search and a mission order. They are tasked to find targets along a given path in a given time and to record them. After the search, they are tasked to locate an ELT signal. In this event, the localization of the targets is evaluated. Rescue event: In a 45 minute period; the team is tasked to drop survival equipment in a predetermined location. The distance is measured from the initial impact point of each survival equipment pack to the center of the target. The 2 nd part of the rescue event is composed of the following steps: 1) parachutists land in the zone, 2) the rescue team has to bring all the rescue equipment to a designated location X, 3) they have to get an object at another location given a GPS and a set of coordinates and finally, 4) they have to come back to the location X with this object using the GPS. Maintenance event: Five areas are covered by this event: 1) Servicing/Snag, 2) Refuelling, 3) Safety, 4) Crew Professionalism and 5) Maintenance Scenario. Helo event (Helicopter Accuracy event): Different manoeuvres have to be made and are scored: 1) navigate an obstacle course, 2) conduct precision hoists and 3) conduct a precision landing. Ground Search event DRDC Valcartier TM

60 Annex B Search Event Scenario 2006 The search event scenario 2006 (detailed in B.1 and B.2) was provided by [13]: B.1 Search Event Scenario Two 212 helicopters from the company Extreme Adventures, departed North Bay to drop off numerous adventure training participants along a predetermined route NW of North Bay. The coordinates for the route flown are; Start point N W TP1 N W TP2 N W TP3 N W TP4 N W Exit N W Thirty minutes after the last group of 2 were dropped off, a distress call was received by cell phone from an adventure training group member. Due to poor communications, (limited cell phone coverage in the area) very little information could be received. The only words extracted from the short call were, serious injury ; Shortly after drop off ; Need assistance. Considering the limited time from drop off to injury, it expected that the subject individual (s) will be within close proximity of route flown by the helicopters while inserting team members. A total of 8 members (age 19-32) were dropped off. These teams were participating in different events including mountain biking, BASE jumping, canoeing and extreme hiking. Teams were dropped off at locations sporadically along the route picked for their type of adventure training. B.2 JRCC Tasking Proceed to Start point and conduct a Track crawl at 1000 and 2 along planned route. Report all sightings of a suspicious nature. Require Lat/Long, and accurate description of any sightings. 44 DRDC Valcartier TM

61 Annex C Matlab Program The Matlab program used to determine the confidence rate that a particular method is better than another one was provided by [11]: function c = bayesian_bernouilli(n1,k1,n2,k2,i,it) % i = number of intervals in posterior distr of p % it = number of monte carlo iterations ps = zeros(2,i); y = k1+k2; n = n1+n2; for j=1:i end ps(1,j) = j/i-1/(2*i); ps(2,j) = ps(1,j)^(y-0.5)*(1-ps(1,j))^(n-y-0.5); s = sum(ps(2,1:i)); ps(2,1:i) = ps(2,1:i)/s; for j=2:i end; ps(2,j) = ps(2,j)+ps(2,j-1); % ps(2,1:i) is now posterior distribution of p % monte carlo trial rand('state',sum(100*clock)); p = rand(1,it); pp = zeros(2,it); d = k2/n2-k1/n1; for j=1:it DRDC Valcartier TM

62 k = 1; while(p(j)>ps(2,k)) k = k+1; end pp(1,j) = ps(1,k); pp(2,j) = 0; for kk2=0:n2 kk1 = ceil(n1*(kk2/n2+d)); ss = 0; while (kk1 <= n1) ss = ss+nchoosek(n1,kk1)*pp(1,j)^kk1*(1-pp(1,j))^(n1-kk1); kk1 = kk1+1; end pp(2,j) = pp(2,j)+ss*nchoosek(n2,kk2)*pp(1,j)^kk2*(1-pp(1,j))^(n2-kk2); end end c = mean(pp(2,1:it)); 46 DRDC Valcartier TM

63 Annex D Examples of GPS tracks Maps of the GPS tracks of SAREX 2006 (as shown below) were provided by [16]. Figure B-1 shows the GPS track on the search map for the team Janke. Figure B-1: GPS track of the team Janke DRDC Valcartier TM

64 Figure B-2 shows the GPS track on the search map for the team Duffy. Figure B-2: GPS track of the team Duffy 48 DRDC Valcartier TM

65 Figure B-3 shows the GPS track on the search map for the team CSS - Siket Figure B-3: GPS track of the team CSS - Siket SAREX 5 DRDC Valcartier TM