INTERIM TECHNICAL REPORT

Transcription

1 INTERIM TECHNICAL REPORT Detection and Discrimination in One-Pass Using the OPTEMA Towed-Array ESTCP Project MR Jonathan Miller, Inc. NOVEMBER 2014 Distribution Statement A

2 REPORT DOCUMENTATION PAGE Form Approved OMB No Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing this collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and Reports ( ), 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to any penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number. PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS. 1. REPORT DATE (DD-MM-YYYY) TITLE AND SUBTITLE 2. REPORT TYPE Technical Report Detection and Discrimination in One-Pass Using the OPTEMA Towed Array: APG Standardized UXO Demonstration Site Technical Report 3. DATES COVERED (From - To) Dec Sept a. CONTRACT NUMBER W912HQ-12-C b. GRANT NUMBER 5c. PROGRAM ELEMENT NUMBER 6. AUTHOR(S) Miller, Jonathan S., Keranen, Joe, Schultz, Gregory, Shubitidze, Fridon, Inc. 5d. PROJECT NUMBER 5e. TASK NUMBER 5f. WORK UNIT NUMBER 7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION REPORT NUMBER, Inc. 115 Etna Rd., Building 3, Suite 1 Lebanon, NH SPONSORING / MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR S ACRONYM(S) ESTCP Environmental Security Technology Certification Program 4800 Mark Center Drive, Suite 17D SPONSOR/MONITOR S REPORT Alexandria, VA NUMBER(S) 12. DISTRIBUTION / AVAILABILITY STATEMENT Approved for public release; distribution is unlimited. 13. SUPPLEMENTARY NOTES 14. ABSTRACT The primary objective of this demonstration was to assess the feasibility of acquiring classification quality data during dynamic Electromagnetic Induction (EMI) surveys of Munitions Response Sites (MRS). The process of obtaining both detection and classification data during a single dynamic survey is known as the one-pass or dynamic classification method of data collection. By demonstrating the dynamic classification methodology using the One Pass Time-domain EMI Array (OPTEMA) to survey areas of the Aberdeen Proving Ground (APG) Standardized UXO Technology Demonstration Site, we acquired a baseline data set that allowed us to quantify the detection and classification performance enabled by this methodology with performance metrics (e.g., probability of detection, probability of false alarm, etc.) corresponding to a standard set of test items. The intent of this demonstration was to determine if the dynamic classification approach can enable detection and classification performance comparable to that of the existing set of cued classification sensors. 15. SUBJECT TERMS Dynamic classification, Munitions response, Electromagnetic induction 16. SECURITY CLASSIFICATION OF: 17. LIMITATION OF ABSTRACT a. REPORT UU b. ABSTRACT UU c. THIS PAGE UU 18. NUMBER OF PAGES 19a. NAME OF RESPONSIBLE PERSON Jonathan Miller UU 75 19b. TELEPHONE NUMBER (include area code) Standard Form 298 (Rev. 8-98) Prescribed by ANSI Std. Z39.18

3 CONTENTS CONTENTS... i FIGURES... iii ACRONYMS... viii EXECUTIVE SUMMARY... ix 1.0 INTRODUCTION BACKGROUND OBJECTIVE OF THE DEMONSTRATION REGULATORY DRIVERS TECHNOLOGY TECHNOLOGY DESCRIPTION TECHNOLOGY DEVELOPMENT ADVANTAGES AND LIMITATIONS OF THE TECHNOLOGY PERFORMANCE OBJECTIVES OBJECTIVE: DETECTION OF ALL TARGETS OF INTEREST OBJECTIVE: MAXIMIZE FALSE POSITIVE REJECTION RATE (FPR) OBJECTIVE: MAXIMIZE CORRECT CLASSIFICATION OF TOIS OBJECTIVE: ACCURATE AND PRECISE TARGET LOCATION ESTIMATION OBJECTIVE: PRODUCTION RATE SITE DESCRIPTION SITE SELECTION TEST DESIGN CONCEPTUAL EXPERIMENTAL DESIGN SITE PREPARATION SYSTEM SPECIFICATION CALIBRATION ACTIVITIES DATA COLLECTION PROCEDURES VALIDATION DATA ANALYSIS AND PRODUCTS PREPROCESSING DETECTION PROCESSING CLASSIFICATION PROCESSING CLASSIFICATION ANALYSIS PERFORMANCE ASSESSMENT OBJECTIVE: DETECTION OF ALL TARGETS OF INTEREST OBJECTIVE: MAXIMIZE FALSE POSITIVE REJECTION OBJECTIVE: MAXIMIZE CORRECT CLASSIFICATION OF TOIS OBJECTIVE: ACCURATE AND PRECISE TARGET LOCATION ESTIMATION APG Demonstration Report i September 2014

4 7.5 OBJECTIVE: PRODUCTION RATE QUALITATIVE PERFORMANCE OBJECTIVES COST ASSESSMENT COST MODEL COST DRIVERS IMPLEMENTATION ISSUES REFERENCES Appendix A: Points of Contact Appendix B: ATC Report APG Demonstration Report ii September 2014

5 FIGURES Figure 1. The OPTEMA sensor electronics include Transmitter/Receiver boards, analog-todigital data acquisition hardware, embedded controller, inverter, and operator PC Figure 2. Map view diagram of the OPTEMA sensor head configuration. Horizontal axis (x - and y -) transmitters are electrically connected in pairs (x - with x - and y - with y -) to provide two orthogonal excitation axes. The larger base vertical axis (z-) transmitter provides the third excitation axis. Blue squares represent the location of the 3-axis receivers Figure 3. OPTEMA sensor head assembly on tow frame (left) and complete system in operation (right) Figure 4. OPTEMA library data for an 81mm mortar. Magenta lines correspond to principal polarizabilities recovered from static data files associated with 6 different target orientations/standoffs. Dark grey lines correspond to the averaged library polarizabilities.. 6 Figure 5. Library polarizabilities recovered from OPTEMA data (dark grey lines) and MetalMapper data (light grey lines). Polarizabilities correspond to a 37mm projectile (left) and a small ISO (right). The slight difference in late time decays between the OPTEMA and MetalMapper polarizabilities corresponding to the ISO is likely due to variations in material permeability (different ISO s were used) Figure 6. OPTEMA system in dynamic survey mode Figure 7. OPTEMA Z- component (Transmit Z-, Receive Z-) detection map generated from test lane data. The map is generated by summing the values of the time gates between 400 and 500 microseconds in each Z- component data channel. This lane contains three targets (ISO medium, 81 mm mortar, and 37mm projectile) all buried at 15 cm depth. ROIs corresponding to each target are circled in red Figure 8. Polarizabilities recovered from inversion of the first ROI. Blue, red, and green lines represent the primary, secondary, and tertiary (respectively) polarizabilities recovered from the ROI dynamic data. Grey lines represent library polarizabilities generated from static calibration data. The results indicate a good match to the 37mm library Figure 9. Polarizabilities recovered from inversion of the second ROI. Blue, red, and green lines represent the primary, secondary, and tertiary (respectively) polarizabilities recovered from the ROI dynamic data. Grey lines represent library polarizabilities generated from static calibration data. The results indicate a good match to the 81mm library Figure 10. Polarizabilities recovered from inversion of the third ROI. Blue, red, and green lines represent the primary, secondary, and tertiary (respectively) polarizabilities recovered from the ROI dynamic data. Grey lines represent library polarizabilities generated from static calibration data. The results indicate a good match to the ISO medium library Figure 11. Diagram of APG UXO Technology Demonstration Site showing areas surveyed. The OPTEMA demonstration was conducted in the calibration grid, blind grid, and indirect fire areas APG Demonstration Report iii September 2014

6 Figure 12. OPTEMA survey in the indirect fire area. Excessive levels of standing water due to recent rains and spring thaw produced several exclusion areas. Overall, approximately 70% coverage was achieved in the indirect fire area Figure 13. OPTEMA transmitter layout and dimensions Figure 14. OPTEMA sensor head receiver cube layout. Receiver cubes are 10cm Figure 15. OPTEMA tow sled Figure 16. Waveform corresponding to 3 OPTEMA data blocks. Each block comprises one sequence of Transmit-Z, Transmit-Y, or Transmit-X. The number of base waveform repeats for each transmitter cycle depends on the decay period selected. This example shows an 8.3ms decay period. This period produces only 1 repeat for each transmitter cycle. If stacking is increased, this complete sequence (three blocks) is repeated and the result is averaged Figure 17. Example of instrument noise standard deviation for seven receiver cubes. The red line represents a baseline quality objective threshold. The magenta, cyan and blue lines correspond to the z-, y-, and x- component receiver data Figure 18. Calibration ball spike test. A calibration ball is placed over each receiver cube in a repeatable location to identify any inconsistencies in data channel output Figure 19. Calibration ball response in principal Tx-Rx pairings for seven receiver cubes. The red line indicates the reference measurement; the blue line indicates the measured response Figure 20. Polarizabilities recovered from daily IVS data matched to a library reference. Red, black and blue lines: primary, secondary and tertiary recovered polarizabilities. Grey lines: reference polarizabilities. IVS items shown here are medium ISOs. All fits are within 95% match Figure 21. OPTEMA data map ( microsecond time gate sum of Transmit Z-/Receive Z- channels) of the calibration grid Figure 22. Example X- (left), Y- (center), and Z- (right) detection data maps. This example is for a 37 mm projectile buried at 15 cm depth lying horizontally across the data collection swath Figure 23. Optimal target locations. Data are selected for soundings acquired while the sensor array was positioned such that the target was located beneath one of the highlighted regions. These regions correspond to areas of greater orthogonality between transmitter fields Figure 24. TOP LEFT: OPTEMA DGM Map showing several anomalies in a portion of the survey area. The anomaly of interest is circled in black with the model-based estimated location shown as the magenta dot. TOP RIGHT: Library matching for six different UXO targets. Soundings corresponding to the anomaly were selected for inversion and the resulting nine sets of polarizabilities (each set comprises a primary, secondary, and tertiary polarizability) are plotted in magenta against the TOI library polarizabilities (shown in dark grey). The polarizabilities obtained from these consecutive soundings are almost identical and show a clear match to the 105mm Indirect Fire munition type. BOTTOM: Estimated model parameters corresponding to each sounding. Parameters are highly consistent for APG Demonstration Report iv September 2014

7 each sounding, indicating a high confidence decision can be made. The Northing and Easting errors for each sounding are within +/- 3cm of the mean estimated location (shown as the magenta dot in the DGM map). Depth estimates are consistent to within +/- 2cm and orientation estimates are consistent within +/- 6 degrees Figure 25. Low SNR source. Each polarizability (magenta curves) recovered from this particular anomaly source is noisy as a result of the object s depth; however, plotting all sets of polarizabilities recovered from soundings within the ROI against the relevant libraries (dark grey curves) makes it fairly easy to identify a match to the 25mm TOI Figure 26. Example APG scoring template Figure 27. OPTEMA ROC curves for blind grid performance. The orange dashed line shows probability of detection vs. probability of false positive for ranking assignments based purely on sensor response amplitude (response). The blue solid line shows probability of detection vs. probability of false positive for ranking assignments based on classification decisions (discrimination). The black solid line indicates our stop dig threshold (the point on our list at which we changed assignment from ordnance to clutter). The black dashed line indicates the sensor noise threshold (the threshold below which we did not assign a detection). These ROC curves indicate that we can achieve detection of 100% of the ordnance with 85% clutter rejection. For this analysis, our operating threshold was set with maximum (100%) efficiency (i.e., the point where maximum clutter rejection can be achieved with no decrease in detection) Figure 28. OPTEMA ROC curves for open field indirect fire performance. The orange dashed line shows probability of detection vs. probability of false positive for ranking assignments based purely on sensor response amplitude (response). The blue solid line shows probability of detection vs. probability of false positive for ranking assignments based on classification decisions (discrimination). The black solid line indicates our stop dig threshold (the point on our list at which we changed assignment from ordnance to clutter). The black dashed line indicates the sensor noise threshold (the threshold below which we did not assign a detection). These ROC curves indicate that we can achieve detection of 99% of the ordnance with 83% clutter rejection (based on 100% efficiency operating threshold). For this analysis, our operating threshold was set with 99% efficiency Figure 29. LEFT: OPTEMA DGM data map showing the anomaly of interest circled in black. Using the 2-D data, it is difficult to resolve the location of the sources; however, classification analysis of the data reveals two distinct sources. RIGHT: Polarizabilities (red and magenta curves) generated from classification analysis of the data plotted against library polarizabilities (dark grey curves). Soundings from within the ROI (circled in black) are inverted for classification features. One set of soundings clearly indicates the presence of an 81mm TOI (red polarizability curves corresponding to a source location indicated by the red dot on the map) while another set of soundings indicates a large piece of clutter (magenta polarizability curves corresponding to a source location indicated by the magenta dot on the map) Figure 30. LEFT: DGM map showing the ROI circled in black. This ROI corresponds to a cluster of targets. Using the threshold target picking algorithm, the peak detector generates eight potential source locations in the ROI. If a follow-up cued survey were used, it would entail soundings at each of these eight locations, requiring several minutes to complete. APG Demonstration Report v September 2014

8 RIGHT: Polarizabilities (red, blue, and magenta curves) generated from classification analysis of the data plotted against library polarizabilities (dark grey curves). The polarizabilities are divided into three groups (red, blue, magenta), each group corresponding to the location of a distinct clutter object in the DGM map (red, blue, magenta dots) APG Demonstration Report vi September 2014

9 TABLES Table 1. OPTEMA static data acquisition parameters Table 2. OPTEMA default dynamic data acquisition parameter settings Table 3. Performance Objectives Table 4. Schedule of APG field activities Table 5. OPTEMA Performance Summary Table 6. Mean location error and standard deviation for target location estimates Table 7. Cost requirements for OPTEMA survey APG Demonstration Report vii September 2014

10 ACRONYMS AOL Advanced Ordnance Locator APG Aberdeen Proving Ground ATC Aberdeen Test Center DAQ Data Acquisition system DGM Digital Geophysical Mapping DGPS Differential GPS EMI Electromagnetic Induction FPR False Positive Rejection IVS Instrument Verification Strip MEC Munitions and Explosives of Concern MLE Mean Location Error MR Munitions Response MRS Munitions Response Site NI National Instruments OPTEMA One Pass Time domain EMI Array P cc Probability of correct classification P d Probability of TOI detection QC Quality Control ROC Receiver Operating Characteristic ROI Region of Interest RTK Real Time Kinematic SDV Standard Deviation SNR Signal to Noise Ratio TEMTADS Time domain EMI Towed Array Detection System TOI Target of Interest UTM Universal Transverse Mercator UXO Unexploded Ordnance WRT YPG Yuma Proving Ground APG Demonstration Report viii September 2014

11 EXECUTIVE SUMMARY The primary objective of this demonstration was to assess the feasibility of acquiring classification quality data during dynamic Electromagnetic Induction (EMI) surveys of Munitions Response Sites (MRS). The process of obtaining both detection and classification data during a single dynamic survey is known as the one-pass or dynamic classification method of data collection. By demonstrating the dynamic classification methodology using the One Pass Time-domain EMI Array (OPTEMA) to survey areas of the Aberdeen Proving Ground (APG) Standardized UXO Technology Demonstration Site, we acquired a baseline data set that allowed us to quantify the detection and classification performance enabled by this methodology with performance metrics (e.g., probability of detection, probability of false alarm, etc.) corresponding to a standard set of test items. Collecting OPTEMA data in the calibration, blind grid, and indirect fire areas of the APG UXO site provided a benchmark to gauge system performance as these areas have served as proving grounds for a large number of UXO classification sensors. The intent of this demonstration was to determine if the dynamic classification approach can enable detection and classification performance comparable to that of the existing set of cued classification sensors. If so, this performance would represent a potentially significant improvement in operational efficiency for classification surveys. Overall, the objectives for the demonstration were met. Independent scoring of the demonstration results indicated that the system enabled detection of 100% of the ordnance in the blind grid with a false positive rejection rate of 85%, and detection of 99% of the ordnance in the open field indirect fire area with a false positive rejection rate of 83%. These results were achieved with a production rate of 0.5acre/hr in the blind grid and 0.3acre/hr in the open field. Production rate in the open field was reduced considerably due to the excessive levels of standing water throughout the site. Saturated ground and deep water holes caused several delays due to tow vehicle immobilization and maneuvering of the sensor array around several of the water holes. Additionally, these site conditions resulted in total survey coverage of approximately 70% of the indirect fire area. During the analysis of our demonstration data, the benefit of being able to directly correlate classification features with 2-D map features was exceedingly evident. Using the same high resolution data for mapping and classification makes it possible to reference classification features to a specific set of coordinates on the 2-D anomaly map. This correlation enables improved characterization of the 2-D target space in high anomaly density areas, which are particularly challenging for cued sensors due to the high probability of cued sensor placement errors and the subsequent necessity for multiple recollects. Ultimately the ability to use one data set for target detection and classification could lead to higher confidence in classification decisions for these high density survey areas. APG Demonstration Report ix September 2014

12 1.0 INTRODUCTION This project was undertaken by, Inc. (WRT) to assess the feasibility of acquiring classification quality data during dynamic Electromagnetic Induction (EMI) surveys of Munitions Response Sites (MRS). The process of obtaining both detection and classification data during a single dynamic survey is known as the one-pass or dynamic classification method of data collection. Currently, standard practice for classification surveys incorporates a two-step data collection process where a preliminary detection-level survey is performed as part of the Digital Geophysical Mapping (DGM) operations. The DGM data serve as a basis for identifying anomalies (i.e., target picking) that are subsequently interrogated during a secondary survey by static (or cued) EMI measurements. Commercial sensors such as the Geometrics MetalMapper are configured for cued surveys and, accordingly, have demonstrated significant success when applied in this two-step process. While this cued mode is extremely effective for distinguishing Munitions and Explosives of Concern (MEC) items from non-hazardous objects, it has two significant drawbacks. First, incorporating a secondary cued EMI survey creates additional costs. While the cued survey is more efficient than excavation of all anomalies, it is typically more time consuming than the preliminary DGM survey and thus adds significantly to the survey costs allocated for the project. Second, because the DGM and cued portions of the survey apply different sensors (e.g., Geonics EM61 and MetalMapper), there is often a disconnect between the target picking process and the cued survey process. The lower spatial and temporal resolution of the DGM sensor can lead to misguided target picking in high density areas. The one-pass method offers improvements over the cued method in regards to these limitations. Detection and classification achieved from a single EMI survey is efficient and could significantly reduce the costs associated with the burden of an additional cued survey. Furthermore, using high resolution classification sensors for detection and identification of anomalies could ultimately provide better initial characterization of the target space than that typically afforded by the standard DGM sensors. This improved characterization could lead to better classification results in high density areas. In conducting this project, our primary objective is to determine whether it is possible, using a one-pass survey mode, to achieve classification results comparable to those achieved during cued surveys. Because of the anticipated lower operating costs associated with dynamic classification surveys, comparable classification quality would indicate that one-pass survey modes may offer a better solution than cued surveys for many projects. 1.1 BACKGROUND The practical basis for munitions classification is founded on the development of advanced EMI sensors that provide high spatial and temporal resolution data. The data produced by these sensors enable the application of advanced physical models to extract useful classification parameters that correspond to physical properties of the object. The initial intent of these classification technologies was to provide a means for inserting additional steps into the existing data collection and processing workflows for Munitions Response (MR) projects without altering the established protocols. Accordingly, the cued survey process was developed as an add-on to the existing geophysical survey workflow for such projects. By relying on the standard DGM survey data for target picking, advanced sensors could be incorporated in cued mode into the existing process with minimal impact to the overall flow. This process has demonstrated APG Demonstration Report 1 September 2014

13 significant success for discriminating Targets of Interest (TOI) from clutter at demonstration and production sites [1]. With the gradual trend towards acceptance of classification technologies in the production environment, the possibility now exists for shifting the focus of classification technology development from improved performance to improved efficiency and feasibility. One-pass detection and classification may provide the greatest return on investment for future classification technology development. One-pass surveys could effectively reduce excavation costs without increasing geophysical survey costs to the extent required for cued sensor data collection. While one-pass survey costs may be somewhat higher than those of a standard DGM survey as a result of slightly lower survey speeds, this increase would be more than offset by eliminating the costs associated with the follow-up cued survey. Given the recent successes of cued EMI sensors and the potential benefits of dynamic classification, several demonstrations have been performed to evaluate the effectiveness of applying cued EMI sensors, such as the TEMTADS 2x2 variant and MetalMapper advanced sensors, in a dynamic mode. We developed the One-Pass Time-Domain EMI Array (OPTEMA) for the specific purpose of acquiring classification-level data in a dynamic survey mode rather than a cued survey mode. Consequently, the OPTEMA comprises certain features that are designed to produce high quality classification data during dynamic surveys. Details of these design elements are presented in Section OBJECTIVE OF THE DEMONSTRATION The primary objective of this demonstration was to quantify the detection and classification performance of the OPTEMA sensor. By demonstrating the system at the Aberdeen Proving Ground (APG) Standardized UXO Technology Demonstration Site, we acquired a baseline data set that enabled us to quantify the detection and classification performance of the OPTEMA with performance metrics (e.g., probability of detection, probability of false alarm, etc.) corresponding to a standard set of test items. Collecting OPTEMA data in the calibration, blind grid, and indirect fire areas of the APG UXO site provided a benchmark to gauge system performance as these areas have served as proving grounds for a large number of UXO classification sensors. The intent of this demonstration was to determine if the OPTEMA can provide, in dynamic operation mode, detection and classification performance comparable to that of the existing set of cued classification sensors. If so, this performance would represent a potentially significant improvement in operational efficiency for classification surveys. 1.3 REGULATORY DRIVERS Demonstration of this technology will elucidate potential cost savings to munitions remediation projects as a result of decreased time and labor devoted to classification surveys. DoD directives for munitions response projects now include scope for classification technologies at a number of sites. We expect that over the next 5-10 years, the list of sites amenable to classification practices will increase significantly. Consequently, any technologies that streamline operations associated with munitions classification could have a large impact on the DoD s ability to effectively implement these technologies. By replacing, to the extent possible, the DGM/cued survey sequence with a one-pass detection and classification survey, it may be possible to extend the feasibility of applying classification to a broader range of sites. APG Demonstration Report 2 September 2014

14 2.0 TECHNOLOGY The OPTEMA comprises an array of multi-directional transmitters and receivers that are optimally configured to provide good EMI characterization across the entire sensor swath. This capability is the basis for effective dynamic classification since sensor position during dynamic surveys is based on survey transects rather than on a priori target location. Consequently, it is likely that a large number of targets will be located at some lateral offset relative to the array center during a dynamic survey. Details of the OPTEMA design and operation are presented in the following subsections. 2.1 TECHNOLOGY DESCRIPTION The OPTEMA sensor is built around the G&G Sciences National Instruments (NI)-based data acquisition framework. These data acquisition components are similar to those incorporated in the first generation commercial MetalMapper systems. Data acquisition hardware is housed in a National Instruments PXI-1042 chassis and includes an NI PXI-8108 embedded controller (Windows OS) and six 8-channel 16-bit NI PXI-6143 A/D cards. Intermediate hardware is housed in an external module and includes the transmitter controller and power distribution board, and three 16-channel receiver boards. These components along with a 2000 Watt inverter are contained in a ruggedized vehicle-mounted chassis (Figure 1) and compose the OPTEMA sensor electronics. Figure 1. The OPTEMA sensor electronics include Transmitter/Receiver boards, analog-to-digital data acquisition hardware, embedded controller, inverter, and operator PC. APG Demonstration Report 3 September 2014

15 One of the key differences between the OPTEMA and MetalMapper data acquisition components is the effective doubling in number of receiver channels (OPTEMA provides 42 receiver channels compared to the MetalMapper s 21 channels). The increase in receiver channels led to the addition of three PXI-6143 cards in the hardware design. Additionally, a modified version of the MetalMapper data acquisition software (EM3DAcquire) is needed to manage the additional data channels. The OPTEMA sensor head comprises five transmitters and fourteen 3-axis receivers across a 1.8 meter sensor swath (Figure 2). This design ensures that three orthogonal magnetic fields are produced at any across track location. The distribution of the 14 receivers also ensures that fields scattered by any target located across the sensor swath will be characterized sufficiently to invert the data and constrain target classification parameters. Figure 2. Map view diagram of the OPTEMA sensor head configuration. Horizontal axis (x - and y -) transmitters are electrically connected in pairs (x - with x - and y - with y -) to provide two orthogonal excitation axes. The larger base vertical axis (z-) transmitter provides the third excitation axis. Blue squares represent the location of the 3-axis receivers. The transmitter coils include four horizontal axis transmitters (x - and y - axes) and one large vertical axis transmitter (z- axis). The horizontal axis transmitters are connected in series pairs (i.e., x - with x - and y - with y -) to provide two effective orthogonal excitation axes. The horizontal axis transmitters and triaxial receivers (aka, receiver 'cubes') share the same reference coordinate frame (primed) that is rotated 45 degrees from the principal coordinate system (i.e., referenced to the direction of travel). Sensor array position and orientation data are provided by a Trimble RTK DGPS R8 receiver and a Microstrain 3DM-GX3-25 inertial sensor, respectively. Position and orientation sensors are mounted on the sensor tow frame, which provides undercarriage protection and mobility for the sensor head. Pictures of the complete sensor head and tow assembly are shown in Figure 3. APG Demonstration Report 4 September 2014

16 Figure 3. OPTEMA sensor head assembly on tow frame (left) and complete system in operation (right). Each OPTEMA transmitter produces an exponential current ramp that is rapidly terminated to generate a strong electromotive force in nearby targets. The receivers measure the decay of secondary magnetic field contributions over time gates spanning a period from 24 microseconds to up to approximately 8 milliseconds. The combination of three effective transmitters (including horizontal axis transmitters in pairs) and fourteen 3-axis receivers provides 126 data channels. The magnetic field decays measured by the receivers usually include contributions from system components (e.g., supporting hardware, survey vehicle, etc.). To maximize sensitivity to anomalies, a background response is typically acquired to establish these intrinsic sources, and later subtracted from subsequent data sets to isolate the anomaly response. In addition to a background subtraction, a transmitter current normalization is applied to all data files to account for any small discrepancies in current between the three transmitters. The OPTEMA uses a 50% duty cycle 60 Hz sub-harmonic as the base waveform. Data blocks (of duration BlockT) are built from a specified number of repeats (nrepeats) of this base waveform. The base waveform comprises a single period of a 50% duty cycle square wave voltage. During the voltage-on periods, the current in the energized transmitter coil ramps exponentially according to the coil time constant parameters. During the voltage-off periods, the receivers measure the ambient magnetic field decays. These decays are averaged with the appropriate sign changes for the positive and negative half cycles of the base period. Further signal averaging is applied for each repeat of the base waveform within the data block. In addition to the base repeats, the operator can select a number of blocks to stack (nstk) or average, in order to improve signal-to-noise ratio. Thus, the OPTEMA provides noise reduction through the inherent base repeat averaging and the explicit stack averaging. Understanding the effect of these data acquisition parameters on detection and classification performance is a key component of this demonstration. The OPTEMA can be operated in either static or dynamic mode. As with cued EMI sensors, the static mode is ideal for maximizing signal-to-noise ratio (SNR). Because there are no restrictions on sample rate when the sensor is operated in static mode, data stacking can be maximized to reduce noise. Increasing the number of samples stacked significantly decreases the sample rate. Thus, for dynamic mode where higher sample rates are required to maximize data density, stacking must be reduced as much as APG Demonstration Report 5 September 2014

17 possible while allowing the sensor to maintain adequate SNR for detection and classification. Table 1 provides an example of typical data acquisition parameters applied during static operation of the OPTEMA. Table 1. OPTEMA static data acquisition parameters. Mode Hold-Off Time (µs) BlockT (s) nrpts Window Width (%) Static Notes: µs = microsecond mv = millivolts nrpts = number of repeats nstks = number of stacks % = percent s = second nstks RxMode TxMode Decay Decimated For the OPTEMA system, the static mode is used primarily for collection of reference library data. Library data sets can be acquired with targets placed directly beneath the sensor (e.g., test pit data collection) or with targets placed above the sensor (e.g., test jig data collection). For library static mode, stacks are increased to 10 samples to maximize SNR. Targets are placed in several orientations and at several depths/heights to ensure recovery of consistent polarizability features for all potential target encounters. Once a sufficient sample of orientations/depths is achieved, polarizability features recovered from inversion of each static file are averaged to produce a library set of polarizabilities for the target. Figure 4 shows an example of OPTEMA library features corresponding to an 81mm mortar. ZYX Figure 4. OPTEMA library data for an 81mm mortar. Magenta lines correspond to principal polarizabilities recovered from static data files associated with 6 different target orientations/standoffs. Dark grey lines correspond to the averaged library polarizabilities. Polarizabilities recovered from inversion of OPTEMA data are comparable to features recovered from other cued EMI sensors such as the MetalMapper. Figure 5 shows an example of library data collected by the OPTEMA and MetalMapper systems. APG Demonstration Report 6 September 2014

18 Figure 5. Library polarizabilities recovered from OPTEMA data (dark grey lines) and MetalMapper data (light grey lines). Polarizabilities correspond to a 37mm projectile (left) and a small ISO (right). The slight difference in late time decays between the OPTEMA and MetalMapper polarizabilities corresponding to the ISO is likely due to variations in material permeability (different ISO s were used). With the exception of the static mode used for library data collection and sensor calibration, the primary data collection mode for the OPTEMA is dynamic. In dynamic mode the OPTEMA provides both detection and classification information. Several data acquisition parameter options are possible for dynamic survey mode. The target survey speed is approximately 0.5 m/s. Because the nominal data block size (a single Transmit-Z, Transmit-Y, or Transmit-X sequence) is 33 ms, this advance rate results in 1 OPTEMA data sample (the combined Transmit- Z, Transmit-Y, Transmit-X sequence) every 5 cm (assuming only 1 sample per stack). If multiple stacks are used, the sampling resolution decreases (for example, 3 stacks produces 1 sample every 15 cm). Within each data block, multiple waveform cycles may be averaged depending on the decay window selected. For an 8.3 ms decay period, only a single bipolar waveform is acquired for each transmitter cycle within the data block. If the decay time is changed to 2.8 ms, three waveform periods are acquired for each transmitter block, thus producing an inherent stacking of 3 samples (i.e., nrpt=3). The primary tradeoff to consider when selecting dynamic data acquisition parameters is between data SNR and data density (i.e., spatial resolution). Increasing the number of stacks will provide higher data SNR, but will compromise spatial sampling resolution both of which are factors that influence the quality of classification parameters. Another consideration is the tradeoff associated with decay period. Inherent stacking (nrpts), and therefore data SNR, can be increased by shortening the decay period from 8.3 ms (1 effective stack) to 2.8 ms (3 effective stacks); however, the shorter decay period limits the ability to extract late time parameters from the data, features that are particularly important for classifying larger or thick-walled objects. Selection of optimal dynamic survey parameters represents a key component of this demonstration. Based on preliminary performance tests, we have settled on four sets of default data acquisition parameters for OPTEMA surveys. Table 2 lists these parameter settings. APG Demonstration Report 7 September 2014

19 Table 2. OPTEMA default dynamic data acquisition parameter settings. Setting Decay (ms) nstk nrpt Total Stacks Sample Resolution (cm) Once dynamic acquisition parameters are set, the OPTEMA is ready for operation. Line files generated for the survey area are loaded into the EM3DAcquire program. These files contain the coordinates of pre-determined transects within the survey area. EM3DAcquire provides line following based on the sensor head GPS and IMU data. Using the line following display, the OPTEMA is aligned with the starting coordinates of the selected transect. Once the sensor is sufficiently aligned (i.e., vehicle and sled heading is approximately in-line with transect), the EM3DAcquire start acquisition button is selected and the program will start recording dynamic data. Figure 6 shows an image of the OPTEMA collecting data along a transect in our test field. Figure 6. OPTEMA system in dynamic survey mode. After completing a transect (or transects) the OPTEMA data can be stitched together to form a detection map. Several options exist for detection. Early, middle, or late time gates within the Z-, Y-, or X- component data can be used to generate a detection map. Subsequently, Regions of Interest (ROIs) are selected from the map based on a detection threshold value (determined from noise floor standard deviation). Figure 7 shows a detection map generated from data collected over one of our test lanes. To generate detection ROIs, we sum the values of the time gates between 400 and 500 microseconds in the Transmit Z-/Receive Z- data channels. APG Demonstration Report 8 September 2014

20 Figure 7. OPTEMA Z- component (Transmit Z-, Receive Z-) detection map generated from test lane data. The map is generated by summing the values of the time gates between 400 and 500 microseconds in each Z- component data channel. This lane contains three targets (ISO medium, 81 mm mortar, and 37mm projectile) all buried at 15 cm depth. ROIs corresponding to each target are circled in red. Once threshold analysis is complete; classification is performed on each ROI extracted from the data grid. Each ROI comprises a number of effective static soundings. Depending on the target size, target depth, and data sample resolution a typical ROI will comprise useful soundings. Thus, a very large volume of data exists for classification. There are several options for selecting data to classify. Inversion may be performed on each individual sounding in the ROI or on a subset of soundings within the ROI and the resulting classification features can be averaged to provide one set of polarizabilities for the ROI. Alternatively, data from multiple soundings within the ROI can be aggregated and an inversion can be performed on the resulting composite data set. This approach has the benefit of providing a data set that provides optimal constraint on the inversion; however, its effectiveness depends heavily on the accuracy of the position and orientation data. The relative (i.e., point-to-point) position and orientation of the sensor head for each sounding must be incorporated in the forward model to produce effective results. Any errors in the relative position for each sounding can lead to a poor model fit when the inversion is applied. Based on our testing to date, we have found two methods that produce good results. Both methods require selecting a subset of soundings to use for inversion. Soundings are selected based on the number of receiver channels in each sounding that exceed an SNR threshold and the across-track position of the anomaly within the ROI (across-track position of the anomaly determines which soundings provide the best orthogonal magnetic field excitation). Typically, 3-10 soundings within each ROI will meet these criteria. Once the subset of soundings is selected, our first method is to invert the data in each sounding separately and average the results. The second method involves aggregating data from each sounding in the subset and APG Demonstration Report 9 September 2014

21 inverting the composite data set. Figure 8, Figure 9, and Figure 10 show the results of applying the first method (inversion on an individual basis and subsequent averaging of the polarizabilities) to the three ROIs shown in Figure 7. Figure 8. Polarizabilities recovered from inversion of the first ROI. Blue, red, and green lines represent the primary, secondary, and tertiary (respectively) polarizabilities recovered from the ROI dynamic data. Grey lines represent library polarizabilities generated from static calibration data. The results indicate a good match to the 37mm library. Figure 9. Polarizabilities recovered from inversion of the second ROI. Blue, red, and green lines represent the primary, secondary, and tertiary (respectively) polarizabilities recovered from the ROI dynamic data. Grey lines represent library polarizabilities generated from static calibration data. The results indicate a good match to the 81mm library. APG Demonstration Report 10 September 2014

22 Figure 10. Polarizabilities recovered from inversion of the third ROI. Blue, red, and green lines represent the primary, secondary, and tertiary (respectively) polarizabilities recovered from the ROI dynamic data. Grey lines represent library polarizabilities generated from static calibration data. The results indicate a good match to the ISO medium library. Another key component of this demonstration is identifying the optimal methods for selecting and inverting dynamic data. Prior to the demonstration, we applied the aforementioned methods successfully on test data acquired; however, applying these methods to a larger data set acquired during the demonstration that includes more target diversity will further elucidate any refinements to these methodologies. 2.2 TECHNOLOGY DEVELOPMENT Development of the OPTEMA technology has progressed through two stages. Under project MM-0908, the initial design concept for a one-pass detection and classification system was evaluated to identify optimal hardware components. Initially, this system concept was based on modifications to the Time domain ElectroMagnetic Towed Array Detection System (TEMTADS) hardware, which comprised mono-loop (i.e., single-axis) transmitter and receiver coils. During this preliminary phase of development, early proof-of-concept tests in conjunction with modeling results indicated that it would be beneficial to use three-axis components instead of the TEMTADS coils. Consequently, a first prototype OPTEMA was developed using a modified version of the Advanced Ordnance Locator (AOL) system architecture. A key feature of this first generation system was the incorporation of the Novatel SPAN, which integrates RTK GPS with a high performance Inertial Navigation System (INS) to produce extremely high precision/accuracy position and orientation data. The initial development stage (project MM-0908) culminated with a demonstration of the first generation OPTEMA prototype at the Yuma Proving Ground (YPG) Standardized UXO Demonstration Site [2]. Under the current demonstration project (MR ), we have performed several additional modifications to the earlier prototype. These advances are related primarily to improvements in APG Demonstration Report 11 September 2014

23 design feasibility and functionality for production operations. Specifically, we redesigned the sensor head to cover a wider swath. This feature required the addition of 5 receiver cubes (from 9 to 14) and 2 transmitter coils (from 3 to 5) as well as the complete reconfiguration of the sensor head layout. The sensor head design now provides thorough multi-axis illumination and enables high spatial resolution sampling across the entire 1.8m sensor swath. The increase in receivers and transmitters also required several modifications to the data acquisition system, which included the addition of several A/D boards as well as a new version of the data acquisition software. Another key difference in this next generation system is the absence of the Novatel SPAN. While the SPAN provided very high quality position/orientation data, it proved to be cost prohibitive from a feasibility standpoint. Accordingly, this current stage of development has focused on reducing the classification performance dependency on extremely high accuracy position/orientation data such that standard RTK differential GPS combined with a low cost inertial sensor will suffice for position/orientation requirements. We have tested and implemented the aforementioned data processing methods to create an approach that is not reliant on the highest quality position/orientation data. A timeline summarizing OPTEMA system development is provided: - Initial Design Concept 2009 (MM-0908): modeling, simulation, and benchtop data collection to identify optimal hardware components; modified design concept from TEMTADS variant to AOL variant; - Positioning Requirements Assessment 2009 (MM-0908): proof-of-concept data collection to assess the necessity for high quality position/orientation data; resulted in the incorporation of the Novatel SPAN into the first generation prototype design; - First Generation System Demonstration 2010/2011 (MM-0908): initial development stage culminated with a demonstration of the AOL-based system at YPG; system included the high performance SPAN during this demonstration; - Next Generation Design Evaluation 2012 (MR-1225): design and fabrication of custom sensor head components; integration of components in new configuration; integration, testing, and development of new data acquisition hardware and software; - Data Processing Refinement 2013/2014 (MR-1225): evaluated testbed data to refine classification processing methods that would reduce reliance on position and orientation data; developed software processing modules that incorporated these methods for processing production level data; - Next Generation System Demonstration 2014 (MR-1225): performed dynamic classification surveys at APG demonstration site to gauge system performance enhancements; system included hardware components described in the previous subsection. 2.3 ADVANTAGES AND LIMITATIONS OF THE TECHNOLOGY The primary advantage of the OPTEMA technology over existing cued EMI sensors is the ability to acquire effective classification data in dynamic mode. This capability means that the OPTEMA delivers detection and classification in a DGM survey mode, thus eliminating the need for a secondary cued survey. By removing the cued survey from the workflow, the one-pass APG Demonstration Report 12 September 2014

24 method not only improves the overall efficiency of the geophysical survey process; it also removes the ambiguity associated with reconciling high resolution (spatial and temporal) cued EMI data with lower resolution DGM data. During the analysis of our demonstration data, the benefit of being able to directly correlate classification features with 2-D map features was exceedingly evident. Using the same high resolution data for mapping and classification makes it possible to reference classification features to a specific set of coordinates on the 2-D anomaly map. This correlation enables improved characterization of the 2-D target space in high anomaly density areas, which are particularly challenging for cued sensors due to the high probability of cued sensor placement errors and the subsequent necessity for multiple recollects. Ultimately the ability to use one data set for target detection and classification could lead to higher confidence in classification decisions for these high density survey areas. The main limitations of the current technology are associated with the reduced data SNR compared to that of cued sensors such as the MetalMapper. There are two reasons for this reduction in SNR. The first is due to the data acquisition parameter constraints imposed by the dynamic survey mode (i.e., reduced stacking, positioning ambiguity, etc.). These constraints are fundamental to dynamic surveys and will need to be overcome through optimal selection of data acquisition parameters and data inversion methodologies as described in the previous section. The second cause of reduced SNR is due to hardware constraints associated with the current sensor design. Because the OPTEMA uses the MetalMapper data acquisition architecture, the system is required to operate within the limits imposed by these electronics. These limits include most notably the transmitter current and kickback voltage limits. The OPTEMA provides an almost effective doubling in coverage area compared to that provided by the MetalMapper (1.8 meter swath vs. 1.0 meter swath). This larger swath is produced by a larger effective total transmitter area; however, due to the kickback voltage limits associated with the MetalMapper electronics, these transmitters do not carry the same current levels as those produced by the MetalMapper transmitters. Specifically, the OPTEMA transmitters operate at approximately 2/3 the current level of the MetalMapper transmitters. This reduced current translates to sub-optimal data SNR. While the results of our APG demonstration indicate excellent detection performance, we believe data SNR could be further improved prior to any future surveys through a modification to the transmitter driver components. APG Demonstration Report 13 September 2014

25 3.0 PERFORMANCE OBJECTIVES Performance objectives for the proposed demonstration are summarized in Table 3. Table 3. Performance Objectives. Performance Data Success Results Achieved Metric Objective Required Criteria Quantitative Performance Objectives Blind Grid Open Field Dynamic OPTEMA data Probability of Ranked P Detection of detection (P d ) anomaly list d >=0.98 for all TOIs all TOIs of all seeded Aberdeen encountered TOIs Test Center (ATC) scoring report Yes P d =1.0 (100% TOI identified) Yes P d =0.99 (99% TOI identified) Maximize False Positive Rejection (FPR) Rate at maximum P d value Maximize correct classification of TOIs Accurate estimation of target locations Number of non-tois correctly classified as non-tois out of total number of non-tois Number of TOIs classified as correct type out of total number of TOIs encountered Northing, Easting, and depth mean location error (MLE) Dynamic OPTEMA data Ranked anomaly list ATC scoring report Dynamic OPTEMA data Ranked anomaly list ATC scoring report Dynamic OPTEMA data Estimated target locations ATC scoring report FPR>=0.60 for all TOIs detected (i.e., at least 60% digs saved with maximum detection of TOIs) P cc >=0.90 (i.e., 90% or greater correct classification) N, E, depth MLE <=.10m Yes FPR=0.85 (85% clutter rejection) Yes P cc =0.93 (93% of TOI correctly classified according to type) Yes Depth MLE=.032 (Northing and Easting MLE N/A for grid) Yes FPR=0.83 (83% clutter rejection) Yes P cc =0.90 (90% of TOI correctly classified according to type) Yes Depth MLE=.01; Northing MLE=.01; Easting MLE=-.02 APG Demonstration Report 14 September 2014

26 Precise estimation of target locations Production rate Northing, Easting, and depth error standard deviation (SDV) Effective area surveyed per hour of operation Qualitative Performance Objectives Ease of use Dynamic OPTEMA data Estimated target locations ATC scoring report Field logs, data file time stamps Operator feedback regarding ease of operation (e.g., navigation, speed control, etc.) N, E, depth SDV <=.10m >=0.5acre/hr Yes Depth SDV=.053 (Northing and Easting SDV N/A for grid) Yes 0.50acre/hr Yes Yes Depth=.07; Northing=.06; Easting=-.07 No 0.30 acre/hr Yes 3.1 OBJECTIVE: DETECTION OF ALL TARGETS OF INTEREST Our objective is to detect all seeded TOIs encountered for data collected in the blind grid and indirect fire areas. This test establishes the OPTEMA s ability to function as a detection sensor Data Requirements Detections were identified from the OPTEMA dynamic survey data acquired in the aforementioned areas. We applied a threshold based on noise floor standard deviation to identify ROIs for further classification. Data collected in the calibration grid were used to establish the optimal threshold and subset of data channels for detection (e.g., Transmit Z Receive Z, Transmit Y Receive Y, etc.). We used ground truth data for the calibration grid to adjust detection parameters. Analysis of blind data sets required ATC scoring to determine detection performance. A detection was considered valid if the ground truth TOI position fell within a 0.5m radius of the detection coordinates Metric This objective applies a Probability of Detection (P d ) metric to assess performance. P d values for each area were generated by taking the ratio of the total number of TOIs detected to the total number of TOIs encountered by the OPTEMA Success Criteria A P d >=0.98 is considered successful. APG Demonstration Report 15 September 2014

27 3.1.4 Results Achieved We achieved a P d =1.0 (100% TOIs detected) in the blind grid and a P d =0.99 (99% TOIs detected) in the indirect fire area. The objective was met for both areas. 3.2 OBJECTIVE: MAXIMIZE FALSE POSITIVE REJECTION RATE (FPR) Our objective is to maximize rejection of false positives (i.e., unnecessary digs) while maintaining correct identification of all TOIs detected (i.e., with P d value maximized) Data Requirements We applied classification to all ROIs generated from the detections identified in the dynamic survey data. Each ROI contained a subset of soundings from the dynamic data that corresponded to the anomaly source. Based on the results of inverting these data, we generated a ranked list of likely TOIs. We placed highest confidence TOIs at the top of the list and highest confidence non-tois (i.e., clutter) at the bottom of the list. Our objective was to rank the list such that the last TOI identified on the list would be ranked above at least 60% of the non-tois. This list was submitted to ATC for ground truth scoring Metric This objective applies a False Positive Rejection (FPR) rate. FPR values for each area were generated by taking the ratio of the number of non-tois ranked below the last TOI on the list out of the total number of non-tois on the list Success Criteria An FPR value >=0.60 (60% clutter rejection) is considered successful Results Achieved We achieved an FPR=0.85 (85% clutter rejection) in the blind grid and an FPR=0.83 (83% clutter rejection) in the indirect fire area. The objective was met for both areas. 3.3 OBJECTIVE: MAXIMIZE CORRECT CLASSIFICATION OF TOIS Our objective is to correctly classify TOIs by type Data Requirements We applied classification to all ROIs generated from the detections identified in the dynamic survey data. Each ROI contained a subset of soundings from the dynamic data that corresponded to an anomaly source. Based on the results of inverting these data, we generated polarizability curves corresponding to each anomaly source. We compared these polarizabilities to those of known TOIs in each area to classify each ranked TOI by type. These classifications were recorded on the ranked anomaly list submitted to ATC for ground truth scoring Metric This objective applies a Probability of Correct Classification (P cc ) metric. P cc values were generated for each area by taking the ratio of the number of TOIs correctly classified by type out of the total number of TOIs encountered Success Criteria A P cc >=0.90 (90% correct classification) is considered successful Results Achieved APG Demonstration Report 16 September 2014

28 We achieved a P cc =0.93 (93% correct classification) in the blind grid and a P cc =0.90 (90% correct classification) in the indirect fire area. The objective was met for both areas. 3.4 OBJECTIVE: ACCURATE AND PRECISE TARGET LOCATION ESTIMATION Our objective is to provide accurate and precise estimates of TOI x, y, and z locations (i.e., Easting, Northing, and depth positions) Data Requirements We applied classification to all ROIs generated from the detections identified in the dynamic survey data. Each ROI contained a subset of soundings from the dynamic data that corresponded to an anomaly source. Based on the results of inverting these data, we extracted position and orientation estimates for each TOI. These position and orientation estimates were recorded on the ranked anomaly list submitted to ATC for ground truth scoring Metric Mean Location Error (MLE) and location error Standard Deviation (SDV) were used as metrics to gauge TOI localization accuracy and precision. The MLE value was calculated by taking the mean error value of the Northing, Easting, and depth location estimates for all TOIs. The SDV value was calculated by taking the standard deviation for each of these errors Success Criteria An MLE<=.10m and SDV<=.10m is considered successful Results Achieved We achieved a depth MLE=.032m with SDV=.053m in the blind grid (Northing and Easting metrics are not applicable for the grid area). We achieved a depth MLE=.01m with SDV=.07m, an Easting MLE=-.02m with SDV=.06m, and a Northing MLE=.01m with SDV=.07m. The objective was met for both areas. 3.5 OBJECTIVE: PRODUCTION RATE Production rate is a useful indicator of the overall efficiency of the one-pass method. It provides a quantifiable metric to be used for comparison to combined production rates associated with DGM surveys and cued surveys Data Requirements To determine production rate, we used the detection maps to identify the total survey area covered by the OPTEMA system. We then consulted field logs and file time stamps to determine an effective survey time. Production time included time required to collect data along each transect, time required to maneuver the vehicle around obstacles and turn the vehicle around at the end of each line, and time required for regular equipment checks (e.g., battery voltage check, operation parameter adjustments, etc.). To the extent possible, we did not include time spent on inadvertent system down-time associated with delays not related to standard production operations; however, it proved challenging to isolate these instances in the indirect fire survey data. Excessive amounts of standing water in the indirect fire area required careful repositioning of the array around deep water holes. In several instances, the vehicle got stuck in deep holes and required several minutes to rectify. As a result, data collection along almost every transect in the indirect fire area was affected by unusually challenging conditions. This led to a lower overall production rate for the indirect fire area. APG Demonstration Report 17 September 2014

29 3.5.2 Metric Production rate was calculated based on the average area surveyed per hour Success Criteria A production rate of 0.5acre/hr or greater is considered successful Results Achieved We achieved a production rate of 0.50acre/hr in the blind grid and a production rate of 0.30acre/hr in the indirect fire area. The objective was met for the blind grid, but was not met for the indirect fire area. The aforementioned excessive levels of standing water in the indirect fire area led to the lower than expected production rate for this area. APG Demonstration Report 18 September 2014

30 4.0 SITE DESCRIPTION This demonstration was conducted at the Standardized UXO Technology Demonstration Site located at the Aberdeen Proving Ground in Aberdeen, MD. The site comprises several test areas, most of which are located in open, relatively flat, grassy areas. The open field has several lowlying areas, which tend to accumulate high levels of standing water during certain times of year. 4.1 SITE SELECTION This site was selected for our demonstration since it serves as one of the two principal proving grounds (along with YPG) for emerging munitions classifications sensors. Consequently, results generated from data collected at this site provide good quantification of the relative performances of various sensor technologies. Because OPTEMA is a large towed array designed for maximum efficiency in open field areas, we selected areas of the site that were amenable to towed array operations. We collected data in three areas at the UXO Test Site: 1) the calibration grid; 2) the blind grid; and 3) the indirect fire area. These areas provided a good sample of target types and emplacement depths for gauging sensor detection and classification performance. Figure 11 provides a diagram that depicts the site layout. Figure 11. Diagram of APG UXO Technology Demonstration Site showing areas surveyed. The OPTEMA demonstration was conducted in the calibration grid, blind grid, and indirect fire areas. APG Demonstration Report 19 September 2014

31 During our two weeks at the site, we experienced several significant rain events. Those events, combined with the recent spring thaw meant that water levels were very high, particularly in several portions of the indirect fire area (Figure 12). Consequently, there were several large exclusion areas that we were not able to survey. Overall, we achieved approximately 70% coverage of the indirect fire area. Figure 12. OPTEMA survey in the indirect fire area. Excessive levels of standing water due to recent rains and spring thaw produced several exclusion areas. Overall, approximately 70% coverage was achieved in the indirect fire area. APG Demonstration Report 20 September 2014

32 5.0 TEST DESIGN The field component of the demonstration was designed to provide the data required for evaluating the performance objectives described in section 3. Details of the field test are described in the following subsections. 5.1 CONCEPTUAL EXPERIMENTAL DESIGN The field test comprised dynamic OPTEMA surveys conducted in the calibration grid, blind grid, and indirect fire areas of the APG UXO Technology Demonstration Site. Initial test activities were focused on the calibration grid. These initial test activities included: static data collection over calibration items to generate polarizability libraries for items likely to be encountered in the blind areas; establishment of an Instrument Verification Strip (IVS) from a subset of targets in the calibration grid; and dynamic surveys over the complete calibration grid using the 4 sets of data acquisition parameters listed in Table 2. Once initial system verification was completed, we conducted the remainder of the test activities in the blind grid and indirect fire areas of the site. We spent longer than initially anticipated in the calibration area due to factors related to the untested survey vehicle. During the period leading up to the demonstration, we had conducted tests at our facility using an available pickup truck; however, this vehicle was not acceptable for the demonstration due to site restrictions. Consequently, we rented a 4x4 off-road utility vehicle that was delivered to the site upon our arrival. The hitch on the vehicle was lower than anticipated causing the leading edge of the tow sled to drag on the ground. Initially, we performed calibration activities with this configuration; however, the wet ground caused the sled to dig in and create added resistance, which proved too much for the vehicle s traction and resulted in several instances where the vehicle got stuck in the mud. We resolved this problem by modifying the boom on the tow sled to lift the sled higher and closer to the anticipated operational height. This on-site modification resolved the problem, but delayed blind area data collection. Another factor that contributed to the extension of the calibration period was the appearance of sporadic and very high amplitude noise spikes in the data. This noise had never occurred during any of our pre-demonstration tests. Initially, we thought it might be a failure of one of the board components in the receiver electronics; however, replacing with spares did not resolve the issue. Ultimately, the noise was traced to an inverter used to power the data acquisition laptop. Although, the laptop had no power connections to the OPTEMA DAQ, the Ethernet cable was enough to couple the noise. We resolved the issue by replacing the laptop batteries with spares throughout the day instead of utilizing vehicle-supplied power for the inverter and laptop. Once we resolved the noise and tow issues, we continued test activities in the blind areas. The remaining blind test surveys were conducted efficiently with the exception of some delays in the indirect fire area caused by excessive levels of standing water. Our tow vehicle got stuck on several occasions and we frequently had to reposition the array around deep water holes. Table 4 provides a summary of our field activity schedule. APG Demonstration Report 21 September 2014

33 Table 4. Schedule of APG field activities. 5.2 SITE PREPARATION Because this demonstration was conducted at one of the Standardized UXO Technology Demonstration Sites, no special preparation was required. 5.3 SYSTEM SPECIFICATION The OPTEMA comprises five transmitter coils and fourteen 3-axis receiver cubes. The five transmitters include one large base Z-directed (vertical axis) transmitter that surrounds two pairs of X - and Y - directed (horizontal axis) transmitters. The Z-transmitter dimensions are 1.8m x 1.0m and the X - and Y - (primed coordinates denote 45 degree rotation to primary reference frame) transmitters are 1.22m x 1.0m. Figure 13 shows a diagram of the transmitter layout and dimensions. Figure 13. OPTEMA transmitter layout and dimensions. The receivers are the standard G&G Sciences 10cm cubes developed for advanced EMI sensors (Figure 14). These cubes are rotated 45 degrees to the main reference frame, in-line with the horizontal axis transmitters. Figure 14 shows the layout of the receiver cubes within the OPTEMA sensor head. APG Demonstration Report 22 September 2014

34 Figure 14. OPTEMA sensor head receiver cube layout. Receiver cubes are 10cm. The OPTEMA sensor head is mounted in a non-metallic tow sled. The sled features a wooden tow boom, a protective skid plate, and low pressure pneumatic tires. The tires provide shock absorption, reduce wear on the sled frame, and provide lower tow resistance. In flat terrain, the sled rolls on the aft-mounted wheels; however, the skid plate provides additional support in more challenging terrain conditions. The tow bar puts the leading edge of the OPTEMA sensor head approximately three meters behind the hitch point on the vehicle. Figure 15 shows a picture of the sensor head mounted in the tow sled. APG Demonstration Report 23 September 2014

35 Figure 15. OPTEMA tow sled. The OPTEMA incorporates a Trimble Real Time Kinematic Differential GPS for sensor head position data. A Trimble R8 receiver is mounted directly above the center of the sensor head. Inertial measurements are provided by a Microstrain 3DM-GX3-25 orientation sensor. The Euler angle outputs provide pitch, roll, and yaw measurements for the OPTEMA sensor head. The IMU is co-located with the GPS receiver. OPTEMA sensor electronics include the transmitter and receiver modules, data acquisition computer, and power inverter. Electronics are mounted in a 2 x 2 x 3 (LxWxH) enclosure that can be fastened to the bed of any tow vehicle. A pair of deep cycle marine batteries provide +/- 12V for the transmitter and receiver modules. A single deep cycle battery provides 12V for the inverter, which supplies power for the data acquisition computer. An operator PC can be connected to the data acquisition computer via remote desktop through an Ethernet cable. The transmitter controller provides switching at 60 Hz subharmonics at 50% duty cycle. This design produces a bipolar square waveform with a user selected period. The waveform period determines the decay window measured by the receivers. Typically, 2.8ms or 8.3ms decay windows are used for dynamic survey mode (see Table 2 for reference). The number of waveform repeats in each data block (a single Transmit-Z, Transmit-Y, or Transmit-X sequence) is a function of the waveform period selected. Figure 16 shows an example of a sequence of 3 data blocks corresponding to an 8.3ms decay period. APG Demonstration Report 24 September 2014

36 Figure 16. Waveform corresponding to 3 OPTEMA data blocks. Each block comprises one sequence of Transmit-Z, Transmit-Y, or Transmit-X. The number of base waveform repeats for each transmitter cycle depends on the decay period selected. This example shows an 8.3ms decay period. This period produces only 1 repeat for each transmitter cycle. If stacking is increased, this complete sequence (three blocks) is repeated and the result is averaged. For the demonstration we used the data acquisition parameters shown in Table CALIBRATION ACTIVITIES Calibration activities at the demonstration site consisted of both initial and daily calibration routines. We performed initial system verification and baseline measurements in the calibration grid. Initial calibration activities included basic instrument verification measurements such as noise and static (spike) tests. These instrument verification measurements were conducted with the OPTEMA system located in a clean (anomaly free) area near the calibration grid. The noise test established a baseline noise floor for the OPTEMA system. For the noise test, the system was stationed over the cleared location and set to run in a continuous (dynamic) mode where each measurement data block was recorded as a separate data point (i.e., stacking set to 1). Typically, N=30 or more data-points were collected. To calculate the OPTEMA noise floor, the standard deviation of the N data points of each time-channel for each of the 126 transmitterreceiver combinations was calculated. This process helps to identify any significant deviations in sensor noise on a day-to-day basis. Significant deviations in data channel noise may be indicative of a hardware fault. An example of instrument noise standard deviation is shown in Figure 17. This test was performed on an initial basis for baseline reference, and subsequently on a daily basis for verification. During our blind area test activities, no significant deviations in sensor noise were noted. APG Demonstration Report 25 September 2014

37 Figure 17. Example of instrument noise standard deviation for seven receiver cubes. The red line represents a baseline quality objective threshold. The magenta, cyan and blue lines correspond to the z-, y-, and x- component receiver data. Static, or spike, tests were used to verify consistency in data channel output on a daily basis. Static tests were performed with the OPTEMA stationed in a clean area near the calibration grid. After setting the OPTEMA data acquisition parameters to static mode (Table 1), a calibration ball was placed in a test jig centered over each receiver cube (Figure 18). This test provided another measurement of instrument consistency. The calibration ball response in each principal data channel (i.e., Transmit-Z/Receive-Z, Transmit-Y/Receive-Y, Transmit-X/Receive-X) should be repeatable within 10% deviation on each test (some deviation may be caused by small jig placement inconsistencies). Any significant deviations in the calibration ball response may be indicative of hardware faults. An example of the principal data channel calibration ball response is shown in Figure 19. This test was performed on an initial basis for baseline reference, and subsequently on a daily basis for verification. During our blind test activities, only one instance occurred where a significant deviation in spike amplitude was noted. This deviation occurred as a result of a broken receiver channel connector in the DAQ. The strain relief for our cables had slipped causing one of the receiver cables to pull APG Demonstration Report 26 September 2014

38 the connector and break one of the solder joints. This malfunction affected 7 data files in the indirect fire area, which were subsequently recollected. Figure 18. Calibration ball spike test. A calibration ball is placed over each receiver cube in a repeatable location to identify any inconsistencies in data channel output. Figure 19. Calibration ball response in principal Tx-Rx pairings for seven receiver cubes. The red line indicates the reference measurement; the blue line indicates the measured response. APG Demonstration Report 27 September 2014

39 Initial calibration activities also included static measurements over calibration grid items. These static data were used to generate polarizability libraries for TOIs in the blind grid and indirect fire areas. For each TOI type in the calibration grid, we acquired 3-4 static measurements with each X -/Y - pair of transmitters approximately centered over the target. By centering each transmitter pair over the target, we were able to measure the response across the complete sensor swath; ensuring consistent results were achieved using any subset of data channels within the array. Reference calibration grid polarizabilities were also used on a daily basis for instrument verification. We collected static data over a subset of targets in the calibration grid (3-4 targets) regularly to ensure that recovered polarizabilities were consistent on a day-to-day basis. These targets served as an IVS. Polarizabilities recovered from IVS data should match the reference libraries with 95% fit for static measurements collected with each X -/Y - transmitter pair centered over the target. Additionally, estimated target location coordinates recovered from inversion of the IVS data should be within 15cm of the ground truth coordinates. Thus, this test served as a measurement of instrument consistency as well as model consistency. Figure 20 shows an example of IVS data matched to library polarizabilities IVS 1: Horiz. X-track z=6" 10 2 IVS 2: Horiz. X-track z=6" 10 2 IVS 3: Horiz. X-track z=6" IVS 4: Vertical z=12" IVS 5: Vertical z=12" IVS 6: Vertical z= Figure 20. Polarizabilities recovered from daily IVS data matched to a library reference. Red, black and blue lines: primary, secondary and tertiary recovered polarizabilities. Grey lines: reference polarizabilities. IVS items shown here are medium ISOs. All fits are within 95% match. Instrument verification measurements were conducted on a twice daily (morning and afternoon) basis to ensure consistency throughout each day of operation. During our blind test activities, all objectives were met for the location and fit accuracy metrics. APG Demonstration Report 28 September 2014

40 5.5 DATA COLLECTION PROCEDURES Throughout the demonstration, we performed OPTEMA data collection procedures in both static and dynamic modes. The static data collection mode was used for initial library data collection and daily instrument verification tests. Dynamic mode was used for surveys conducted in the calibration grid, blind grid, and indirect fire areas. Table 1 and Table 2 (section 2.1) show the default static and dynamic mode acquisition parameter settings, respectively. We performed static measurements only in the calibration grid. These measurements required an initial background measurement acquired in a clean area (anomaly free) of the calibration grid. This measurement was then used for background subtraction from subsequent static measurements made in the grid. We used static measurements for library data collection, IVS measurements, and daily spike tests. Static measurements over targets were made with both pairs of horizontal axis transmitters centered over the target to verify measurement consistency across the entire swath of the array. We performed dynamic measurements as part of calibration grid, blind grid, and indirect fire area surveys. We conducted an initial dynamic survey in the calibration grid. This survey comprised full sensor coverage of the grid four times using each of the four default settings listed in Table 2. Figure 21 shows an anomaly map generated from full coverage OPTEMA data collected in the calibration grid. Figure 21. OPTEMA data map ( microsecond time gate sum of Transmit Z-/Receive Z- channels) of the calibration grid. After collecting these calibration grid data, we performed an in-field analysis of the data corresponding to each parameter setting to down-select two optimal settings for the site. This decision was based on library fits for the polarizabilities recovered from each dynamic data set. We selected settings 1 and 3 in Table 2 (nstk = 1 for the 8.3ms and 2.8ms decays, respectively). These settings provided a down track sample resolution of approximately 5cm. Our intent was APG Demonstration Report 29 September 2014

41 for the 2.8ms decay to provide slightly better SNR due to the increase in the number of repeats in the data block. We performed dynamic surveys in the blind grid using each of the two downselected parameter settings; however, further analysis indicated that the 8.3ms decay setting provided better information for classification (even at the slightly elevated noise levels). Therefore, we selected only the 8.3ms setting (setting 1 in Table 2) for the open field survey. Background measurements were not required for dynamic data collection. Because dynamic surveys cover both anomalies and background regions, background measurements can be pulled directly from the dynamic data to be used for subtraction from the ROIs. Navigation for both static and dynamic measurements was provided by the EM3DAcquire navigation interface. We generated line segment files (for dynamic measurements) and cued files (for static measurements) prior to conducting surveys in the UXO Test Site areas. Ground truth for the calibration grid was used to generate cues for each of the items in the calibration grid. Grid coordinates for all three areas were required to generate transect lines in each survey area. The format for each cue is a standard longitude, latitude coordinate in decimal degrees. The format for line segments is the longitude, latitude of each line endpoint in decimal degrees. We used a 1.2m spacing for transect lines in dynamic survey areas. This provided a 33% overlap for adjacent transects. Dynamic data files can be quite large. Depending on the acquisition parameters selected, the OPTEMA generates data samples at a rate of 5-10 Hz. This can produce as much as 250MB for 10 minutes of operation. To accommodate the large volume of data, we used an external drive to store data and periodically transfer data from the data acquisition hard drive to ensure that sufficient memory was available for the duration of dynamic survey operations. We performed quality checks of all data collected on a daily basis. Specifically, we looked for discrepancies in receiver channel output that would indicate sensor faults. In addition to the aforementioned instrument verification activities, we also performed visual quality control for all dynamic data by spot-checking samples along each data transect. Our daily quality checks identified two instances of incomplete line collections in the blind grid caused by operator error. These lines were subsequently recollected VALIDATION Upon completion of the UXO Technology Demonstration Site data collection, we generated ranked anomaly lists using the standard ATC scoring formats. Ranked lists for each scoring area included confidence rankings of anomalies in the blind areas with highest confidence TOI rankings at the top of the list and highest confidence non-toi rankings at the bottom of the list. The ranked lists also included the estimated location, depth, and orientation for anomalies identified as TOIs and the estimated location for anomalies identified as clutter. These lists were submitted to ATC for scoring. To ensure results for the blind data analysis were not influenced by prior knowledge of the survey area; analysts who had never performed any previous analysis of APG blind areas conducted the analysis of the OPTEMA data set. APG Demonstration Report 30 September 2014

42 6.0 DATA ANALYSIS AND PRODUCTS OPTEMA data analysis includes preprocessing, detection processing, and classification processing and analysis stages. 6.1 PREPROCESSING Preprocessing steps are similar for both static and dynamic data. A transmitter current normalization is performed on all OPTEMA data files by dividing all data channel values by the peak current value corresponding to the appropriate transmitter (e.g., Transmit-Y/Receive-Z data channels are divided by the peak Y-transmitter current). This process ensures that only the size and number of windings (both of which are constant) for each transmitter affect the response measured. Thus, no discrepancies in transmitter current (which may vary) will influence the data. Current normalization is performed once for a static data file and is repeated for each sounding in a dynamic file. After each data file is current-normalized, a background subtraction step is performed by subtracting the value of each data channel in a background data file from the corresponding data channel value in a data file. Background data are collected with the OPTEMA sensor stationed over a clean (anomaly free) area of the site. Background subtraction is performed once for a static data file and is repeated for each sounding in a dynamic file. For dynamic files, background measurements can also be pulled directly from the data. This process requires identification of anomaly free locations in the survey area. Identification of these regions is performed during the detection processing stage. 6.2 DETECTION PROCESSING Detection maps are created by processing dynamic data files. Detection processing requires data gridding for each transmit/receive pairing. GPS latitude/longitude values are converted to UTM coordinates (Easting / Northing). Each sounding in a dynamic file is associated with a GPS Easting and Northing coordinate and a measurement of sensor head pitch, roll, and yaw angles. Receiver cube positions for each sounding are calculated by applying an Euler transformation to the vector from the center of the GPS antenna to the center of each receiver cube. This transformation creates an Easting/Northing position for each receiver cube for each sounding. For each principal transmit/receive pairing (i.e., Transmit-Z/Receive-Z, Transmit-Y/Receive-Y, Transmit-X/Receive-X), a data value is generated for each receiver cube by summing the values of the middle time gates in the corresponding data channel. These data values are then mapped to a 2-D space using the receiver cube locations. Finally, 2-D interpolation is applied to generate the final uniformly-spaced X-, Y-, and Z- data maps. Figure 22 shows the X-, Y-, and Z- data maps for a 37 mm projectile buried at 15 cm depth. APG Demonstration Report 31 September 2014

43 Figure 22. Example X- (left), Y- (center), and Z- (right) detection data maps. This example is for a 37 mm projectile buried at 15 cm depth lying horizontally across the data collection swath. The detection algorithm consists of two steps: 1) apply a threshold based on the data noise floor standard deviation to create areas containing high SNR responses, and 2) in each high SNR area apply a peak detection algorithm to create a unique detection and ROI for each peak found. Step 2 splits large high response areas into multiple detections and is an important step in high clutter areas or when a small anomaly is in close proximity to a large anomaly. For each detection a radius is applied to identify the ROI surrounding each peak. The radius size is based on the local gradient associated with the peak and the number of peaks associated with an anomaly (1 peak for Z-data; 2 peaks for X- and Y-data). Finally, across track and along track indices are generated for each alarm in an ROI. These indices correspond to the receiver cube and sounding number associated with each alarm. Each ROI is saved as a data volume (number of soundings by number of time gates by number of data channels) in.mat format. Alarm indices and UTM coordinates are saved as part of the data structure as well. 6.3 CLASSIFICATION PROCESSING Following detection processing, all ROIs are further processed for classification analysis. The first step in classification processing is to improve the relative point-to-point positional accuracy within each ROI. Because the inversion algorithm relies on relative, not absolute, positioning between consecutive soundings removing small absolute positioning errors associated with GPS accuracy can enhance the classification performance. An initial filtering step is performed by comparing the calculated point-to-point GPS heading to the value calculated by averaging consecutive IMU heading measurements. The calculated GPS point-to-point distance traveled is projected onto the IMU heading vector to establish the relative position traveled between consecutive soundings. These relative positions are then assigned to each sounding within the detection ROI. Position vectors corresponding to the center of the OPTEMA array at each sounding location within the ROI are generated using the relative point-to-point positions. These relative position vectors are referenced to the position of the OPTEMA during the peak response measurement (i.e., peak detection position of the OPTEMA center is [0 0 0] in [x y z]). APG Demonstration Report 32 September 2014

44 A second background subtraction is also performed. A background measurement outside each ROI is selected and subtracted from all soundings within the ROI to further remove any localized background response. Based on the across track and along track indices for each ROI alarm, a subset of soundings is selected from the ROI dynamic data for inversion. Optimal soundings are those corresponding to data collected while the target was beneath one of the horizontal axis transmitters (Figure 23). These data ensure the best orthogonal illumination of the target and provide better constraint on the inversion. Figure 23. Optimal target locations. Data are selected for soundings acquired while the sensor array was positioned such that the target was located beneath one of the highlighted regions. These regions correspond to areas of greater orthogonality between transmitter fields. Two methods of inversion are applied to the selected data within each ROI as described in section 2.1. The first method is to invert the data in each sounding separately and compile all sets of polarizabilities in a group for further analysis. The second method involves aggregating data from each sounding in the subset and inverting the composite data. This method applies the relative position vectors to assign position coordinates to each data point in the aggregated data matrix. Finally, data are fit to a dipole-based forward model using a least squares inversion. The forward model accounts for the sensor array geometry, the position of the array relative to the target, and the target physical features, which are modeled as a set of orthogonal magnetic dipoles (i.e., polarizability tensor). Once model parameters are chosen to minimize the error between the data and the model output, the resulting target polarizabilities are selected for feature classification. Starting parameters for the inversion, such as target position, are chosen based on the detection parameters (i.e., across track and along track indices for each alarm). Additionally, these detection parameters may be used to identify potential multi-object scenarios. For example, if an ROI includes two unique alarms a two-source forward model will be applied to the data inversion. 6.4 CLASSIFICATION ANALYSIS APG Demonstration Report 33 September 2014

45 For the APG data set, we determined the optimal classification approach was to invert each sounding in the ROI on an individual basis. Using this approach, we treated each sounding in the ROI as an effective cued measurement. Thus, each sounding provided a unique measurement at a point along the survey transect. The benefit of this approach was that it enabled recovery of several sets of polarizabilities corresponding to each anomaly source in the survey area. Consequently, we were able to improve confidence in our classification decision by using the statistical variability of the various model parameters (e.g., polarizabilities, depth, location, orientation, etc.) corresponding to each source. Highly consistent model parameters were a good indication that the source was well characterized. Figure 24 shows an example of the repeatability of these model parameters for a particular source. Figure 24. TOP LEFT: OPTEMA DGM Map showing several anomalies in a portion of the survey area. The anomaly of interest is circled in black with the model-based estimated location shown as the magenta dot. TOP RIGHT: Library matching for six different UXO targets. Soundings corresponding to the anomaly were selected for inversion and the resulting nine sets of polarizabilities (each set comprises a primary, secondary, and tertiary polarizability) are plotted in magenta against the TOI library polarizabilities (shown in dark grey). The polarizabilities obtained from these consecutive soundings are almost identical and show a clear match to the 105mm Indirect Fire munition type. BOTTOM: Estimated model parameters corresponding to each sounding. Parameters are highly consistent for each sounding, indicating a high confidence decision can be made. The Northing and Easting errors for each sounding are within +/- 3cm of the mean estimated location (shown as the magenta dot in the DGM map). Depth estimates are consistent to within +/- 2cm and orientation estimates are consistent within +/- 6 degrees. APG Demonstration Report 34 September 2014

46 After inverting the data corresponding to each ROI in the blind survey areas, we analyzed the inverted model parameters to determine a classification decision for each anomaly source. We assigned TOI (ordnance) rankings to anomalies based on polarizability fits to TOI libraries. We used TOI libraries for ordnance types known to be located in each area (25mm, 37mm, 60mm, 81mm, 105mm, and 105mm artillery for the blind grid; 60mm, 81mm, and 105mm for the indirect fire area). Highest confidence TOI rankings were assigned to anomalies that provided good fits (>90% match) for all three polarizabilities. Lower confidence TOI rankings were assigned to anomalies that provide good fits (>90% match) for the primary polarizability only. We also identified a subset of anomalies that produced low SNR data. We used point-to-point jitter for the mid-time channels ( ms) in the primary polarizability to identify lower SNR sources. These were flagged for review by the analyst during the final QC stage. We assigned non-toi rankings to anomalies based on symmetry features (e.g., ratio of secondary to tertiary polarizabilities), decay features (e.g., ratio of late time to early time for primary polarizability), and size features (e.g., primary polarizability sum of all time gates). Lowest confidence non-toi rankings were assigned to anomalies showing good symmetry, large size, and long decay features. Highest confidence non-toi rankings were assigned to anomalies showing poor symmetry, small size, and short decay features. After assigning an initial ranking value to each anomaly, we performed a final QC of the list during which time the analyst reviewed the low SNR cases. For these cases, the statistical variability of the model parameters was particularly useful for assigning a final classification decision. Specifically, the repeatability of the polarizabilities enabled better characterization of the noisy cases. Figure 25 shows an example of noisy polarizabilities associated with a deep 25mm. Although each set of polarizabilities is relatively noisy, when viewed together against a library reference it is easy to identify a match to the 25mm TOI. Figure 25. Low SNR source. Each polarizability (magenta curves) recovered from this particular anomaly source is noisy as a result of the object s depth; however, plotting all sets of polarizabilities recovered from soundings within the ROI against the relevant libraries (dark grey curves) makes it fairly easy to identify a match to the 25mm TOI. APG Demonstration Report 35 September 2014

47 After completing the QC of each classification decision, we assigned a final ranking value to each anomaly according to the standardized UXO Test Site scoring template ( We assigned decision values for each anomaly in three stages: 1) response stage; 2) discrimination stage; and 3) classification stage. The response stage corresponds to the basic detection capabilities of the sensor. Values assigned in the response stage represented the sensor output response amplitude. Values above our predefined amplitude threshold indicated detections. The discrimination stage indicates whether an anomaly is likely ordnance (O) or clutter (C). Confidence values assigned during this stage indicate the probability that an anomaly is ordnance (higher numbers correspond to high confidence ordnance is present; low numbers correspond to high confidence ordnance is not present). These discrimination confidence values determined the overall ranking for the entire list. During the discrimination stage we selected an operating threshold indicating our effective stop dig point (i.e., all ordnance identified with maximum clutter left in the ground). This operating threshold was the point on the list at which we changed our discrimination assignment from O to C. During the classification stage we indicated the predicted ordnance caliber and type for each anomaly identified as potential ordnance. For the grid data, we also indicated whether the grid contained no anomaly (i.e., blank B). Finally, we assigned estimated depth and orientation (i.e., Euler angle) values to items identified as ordnance. Figure 26 shows an example of the scoring format used. Figure 26. Example APG scoring template. We generated separate lists for the grid and open field (indirect fire) areas and submitted the final lists to ATC for scoring. ATC used these lists to score our detection, discrimination, and classification performance for each blind survey area. APG Demonstration Report 36 September 2014

48 7.0 PERFORMANCE ASSESSMENT ATC-generated Receiver Operating Characteristic (ROC) curves for the two blind survey areas are presented in Figure 27 and Figure 28. These curves show the OPTEMA detection (response) and discrimination performance achieved in both the blind grid area and the indirect fire area. Figure 27. OPTEMA ROC curves for blind grid performance. The orange dashed line shows probability of detection vs. probability of false positive for ranking assignments based purely on sensor response amplitude (response). The blue solid line shows probability of detection vs. probability of false positive for ranking assignments based on classification decisions (discrimination). The black solid line indicates our stop dig threshold (the point on our list at which we changed assignment from ordnance to clutter). The black dashed line indicates the sensor noise threshold (the threshold below which we did not assign a detection). These ROC curves indicate that we can achieve detection of 100% of the ordnance with 85% clutter rejection. For this analysis, our operating threshold was set with maximum (100%) efficiency (i.e., the point where maximum clutter rejection can be achieved with no decrease in detection). APG Demonstration Report 37 September 2014

49 Figure 28. OPTEMA ROC curves for open field indirect fire performance. The orange dashed line shows probability of detection vs. probability of false positive for ranking assignments based purely on sensor response amplitude (response). The blue solid line shows probability of detection vs. probability of false positive for ranking assignments based on classification decisions (discrimination). The black solid line indicates our stop dig threshold (the point on our list at which we changed assignment from ordnance to clutter). The black dashed line indicates the sensor noise threshold (the threshold below which we did not assign a detection). These ROC curves indicate that we can achieve detection of 99% of the ordnance with 83% clutter rejection (based on 100% efficiency operating threshold). For this analysis, our operating threshold was set with 99% efficiency. Overall, these results indicate that we can achieve detection of 100% of the ordnance in the blind grid while rejecting 85% of the clutter (false positives) and we can achieve detection of 99% of the ordnance in the indirect fire area while rejecting 83% of the clutter (false positives). For the blind grid analysis we selected our operating threshold at 100% efficiency (i.e., the point where maximum clutter rejection can be achieved with no loss in detection). For the indirect fire area we selected our operating threshold at 99% efficiency (slightly too aggressive). The performance results for 100% efficiency thresholds are shown in Table 5. APG Demonstration Report 38 September 2014

50 Table 5. OPTEMA Performance Summary SURVEY AREA PERCENTAGE ORDNANCE DETECTED CLUTTER REJECTION ACHIEVED Blind Grid 100% 85% Indirect Fire Area 99% 83% Details of the specific performance objectives are presented in the following subsections. Further information regarding the ATC analysis is located in Appendix B. 7.1 OBJECTIVE: DETECTION OF ALL TARGETS OF INTEREST The ATC scores show that the OPTEMA detected all ordnance in the blind grid and 99% of the ordnance in the indirect fire area surveyed (approximately 70% of the total test area was covered including low, medium, and high density portions of the site). Further analysis of the results indicates that the targets missed in the indirect fire area were two 105mm projectiles buried in the 8D to 12D (8 x diameter to 12 x diameter) depth range. We applied the same detection threshold to both the blind grid and indirect fire areas. This threshold was selected based on the calibration activities, and was set well below the threshold required to detect all 105mm TOIs in the calibration area. We achieved our objective of 98% or better detection for both areas. While we did not achieve 100% detection in the open field indirect fire area, review of other UXO Technology Demonstration Site reports indicates that the OPTEMA detection performance compares favorably to those of other UXO technologies demonstrated in the indirect fire area [3, 4, 5]. Because ground truth is not available for this area, we can speculate that this site offers significant detection challenges for the demonstration technologies. 7.2 OBJECTIVE: MAXIMIZE FALSE POSITIVE REJECTION Applying the 100% efficiency threshold, the OPTEMA enabled rejection of 85% and 83% of the clutter in the blind grid and indirect fire areas, respectively. We detected a total of 92% of the clutter in the blind grid and 86% of the clutter in the indirect fire area. We achieved our objective of 60% or greater clutter rejection in both areas. 7.3 OBJECTIVE: MAXIMIZE CORRECT CLASSIFICATION OF TOIS Overall, we achieved 93% correct classification by type of the ordnance in the blind grid and 90% correct classification by type of the ordnance in the indirect fire area. The most challenging targets for classification by type were the 105mm (73%) in the blind grid and the 81mm (85%) in the indirect fire area. The 105mm presented a challenge in the blind grid because it produced polarizabilities similar to those of the 105mm artillery, which was also present in the blind grid. For lower SNR cases, it was difficult to distinguish the two types. Similarly, for very low SNR cases in the open field it was difficult to distinguish the 81mm from the 60mm. 7.4 OBJECTIVE: ACCURATE AND PRECISE TARGET LOCATION ESTIMATION Our estimated target locations were based on the average values recovered from all soundings in the ROI. Using the mean value proved to be an effective method to localize the each source. Mean error values and standard deviation are reported in Table 6 for each localization parameter (i.e., Northing, Easting, depth). For each of these parameters, we were within the objective values of 10cm for both the mean and standard deviation. APG Demonstration Report 39 September 2014

51 Table 6. Mean location error and standard deviation for target location estimates. BLIND GRID INDIRECT FIRE Mean (cm) Standard Deviation (cm) Mean (cm) Standard Deviation (cm) Northing N/A N/A Easting N/A N/A Depth OBJECTIVE: PRODUCTION RATE Our objective production rate was 0.5acre/hr or greater for both areas. This rate is based on an advance rate of 2km/h and 33% overlap in sensor coverage for adjacent transects. Our objective rate also factors in time for vehicle maneuvers at the end of each line and periodic equipment checks. We calculated production rate using the timestamps embedded in the data files. For an uninterrupted data collection period (i.e., one in which the only stoppages were for routine actions such as data downloads, sensor calibration, file transfers, or acquisition parameter adjustments), we calculated the timestamp difference between the first and last files collected. We then determined the total area surveyed using the 2-D data maps associated with those files. For the blind grid, we achieved a production rate of 0.5acre/hr, which met the performance objective. Our total time in the blind grid included two full surveys of the area as well as a few recollected lines due to data acquisition errors. We collected two full surveys in order to determine whether measuring a shorter decay period would significantly improve SNR. For the second survey, we changed the decay time measured from 8.3ms to 2.8ms. The shorter decay meant that we could average more samples in each transmit cycle to reduce noise. After analyzing the data, we determined that the 8.3ms decay data was slightly noisier, but overall provided significantly more information for classification. We chose to stay with the 8.3ms setting for the indirect fire survey. We expected that the production rate for the indirect fire area would be slightly higher than for the blind grid because the transects in the grid are relatively short compared to those in the open field. Thus, a larger percentage of the total survey time is spent maneuvering the vehicle at the end of each line in the grid survey. We found, however, that the benefits of running longer lines in the open field were overcome by the challenges associated with the wet ground conditions. During our post-survey analysis of the data file time stamps, it was difficult to isolate the down time related to the challenging conditions. Almost every transect run in the indirect fire area encountered delays caused by the wet conditions; either because the vehicle got stuck or the array had to be repositioned around deep water holes. Consequently, our production rate in the indirect fire area suffered. In these conditions we achieved a production rate of 0.3acre/hr, which did not meet our 0.5acre/hr objective; however, we are confident that under better ground conditions an open field survey could achieve a production rate closer to 0.6 or 0.7acre/hr. Additionally, lessons learned during operation of the platform in the difficult conditions encountered in the open area will increase efficiency in the future if the platform is deployed in similar conditions. APG Demonstration Report 40 September 2014

52 7.6 QUALITATIVE PERFORMANCE OBJECTIVES From a user s perspective, dynamic classification survey procedures using a tow sensor such as the OPTEMA are very similar to those of standard DGM surveys. Consequently, many of the sensor positioning requirements that create challenges during cued surveys are not an issue. The main operational requirements for the OPTEMA include basic line following in the survey area such that complete coverage is achieved and periodic (i.e., beginning and end of the day) calibration routines to ensure instrument functionality. Some of the more onerous tasks associated with cued surveys, such as frequent background data collection and extensive maneuvering to center the sensor over a set of 2-D coordinates are not required. From this perspective, the OPTEMA provides a straightforward solution for classification-level surveys and offers distinct advantages over the cued approach. The greatest operational challenge we faced during the demonstration was posed by the site conditions. The excessive levels of standing water in the open field led to frequent stops to dislodge the vehicle and reposition the array, slowing production significantly. While these conditions were problematic, we do not believe these challenges were indicative of any system design flaws. These conditions would likely be difficult for any vehicle-towed system. These challenges, therefore, represent technology-level rather than system-specific limitations. Aside from the aforementioned adverse site condition areas, the OPTEMA surveys were conducted with a large degree of efficiency. All personnel operating the OPTEMA also had considerable experience conducting cued surveys. Based on the operator feedback from our demonstration, it is clear that the dynamic surveys are more intuitive and straightforward to conduct than typical cued sensor surveys. In addition to ease of operation, one of the greatest advantages of the dynamic survey approach over the cued approach is the ability to correlate classification features with 2-D map features. During our post-survey classification analysis, we identified several instances where data collected in high density areas created some ambiguity in the 2-D data map; however, once the classification features were extracted from the inversion, the source locations were easily resolved. High density areas are often problematic for cued sensors. Because the initial DGM survey is usually conducted with a low resolution sensor, it can be difficult to identify the optimal location for sensor placement during the follow-up cued survey. The OPTEMA surveys clearly demonstrated the advantages of conducting both the mapping and classification stages from one data set. Figure 29 shows an example of two objects in the survey area that were close enough to produce overlapping responses. Using the detection algorithm alone it was difficult to separate the anomaly into separate responses. In this case, if a cued sensor were to follow a standard DGM sensor, the target picking analysis might direct the cued sensor to a location between the two targets and fail to characterize them individually. Using the OPTEMA data, however, it was possible to separate soundings associated with one object from soundings associated with the adjacent object during the classification analysis. In this case, one group of soundings clearly indicated a TOI (81mm) while the other group of soundings indicated a large piece of clutter. Because both objects were of comparable size and emplaced at similar depths in close proximity, it would be difficult to distinguish their locations for a follow-up survey using the standard cued approach. APG Demonstration Report 41 September 2014

53 Figure 29. LEFT: OPTEMA DGM data map showing the anomaly of interest circled in black. Using the 2-D data, it is difficult to resolve the location of the sources; however, classification analysis of the data reveals two distinct sources. RIGHT: Polarizabilities (red and magenta curves) generated from classification analysis of the data plotted against library polarizabilities (dark grey curves). Soundings from within the ROI (circled in black) are inverted for classification features. One set of soundings clearly indicates the presence of an 81mm TOI (red polarizability curves corresponding to a source location indicated by the red dot on the map) while another set of soundings indicates a large piece of clutter (magenta polarizability curves corresponding to a source location indicated by the magenta dot on the map). Another example of the improved mapping/classification feature correlation is presented in Figure 30, which shows data acquired over an area containing a cluster of objects. In this case, the peak detector identifies eight potential sources. If a follow-up cued sensor were used here, the cued sensor would need to be moved around to eight different locations, requiring several minutes to complete. Using the OPTEMA data for classification, however, it is apparent that there are only three significant sources in the area, all of them similar clutter objects. Figure 30. LEFT: DGM map showing the ROI circled in black. This ROI corresponds to a cluster of targets. Using the threshold target picking algorithm, the peak detector generates eight potential source locations in the ROI. If a follow-up cued survey were used, it would entail soundings at each of these eight locations, requiring several minutes to complete. RIGHT: Polarizabilities (red, blue, and magenta curves) generated from classification analysis of the data plotted against library polarizabilities (dark grey curves). The polarizabilities are divided into three groups (red, blue, magenta), each group corresponding to the location of a distinct clutter object in the DGM map (red, blue, magenta dots). APG Demonstration Report 42 September 2014