APPENDIX: ESTCP UXO DISCRIMINATION STUDY

Transcription

1 SERDP SON NUMBER: MMSON-08-01: ADVANCED DISCRIMINATION OF MILITARY MUNITIONS EXPLOITING DATA FROM THE ESTCP DISCRIMINATION PILOT STUDY APPENDIX: ESTCP UXO DISCRIMINATION STUDY 1. Introduction 1.1 Background In 2003, the Defense Science Board observed: The problem is that instruments that can detect the buried UXOs also detect numerous scrap metal objects and other artifacts, which leads to an enormous amount of expensive digging. Typically 100 holes may be dug before a real UXO is unearthed! The Task Force assessment is that much of this wasteful digging can be eliminated by the use of more advanced technology instruments that exploit modern digital processing and advanced multi-mode sensors to achieve an improved level of discrimination of scrap from UXOs. [1] Significant progress has been made in discrimination technology. To date, these technologies have primarily been tested at constructed test sites, with only limited application at live sites. The routine implementation of discrimination technologies will require demonstrations at real UXO sites under real world conditions. Any attempt to declare detected anomalies to be harmless and requiring no further investigation will require demonstration to regulators and project managers of not only individual technologies, but an entire decision making process. The ESTCP UXO discrimination study will be the first phase in what is expected to be a continuing effort that will span several years. The data gathered during these studies will also serve as a resource for continuing and new research into advanced discrimination techniques. The first ESTCP discrimination study is currently being planned. Data will be available from this study prior to the initiation of work to be selected under the FY 2008 SERDP Statement of Need Advanced Discrimination of Military Munitions Exploiting Data From The ESTCP Discrimination Pilot Study. The information provided below is intended to provide prospective SERDP proposers sufficient understanding of the ESTCP Pilot Demonstration to develop a proposal for such work. 1.2 Objective of the ESTCP Discrimination Demonstration 1. Test and validate detection and discrimination capabilities of currently available and emerging technologies on real sites under operational conditions. 2. Investigate in cooperation with regulators and program managers how discrimination technologies can be implemented in cleanup operations. Within each of these two overarching objectives, there are several sub-objectives. 1

2 Technical objectives of the Study Test and evaluate the capabilities of various discrimination processes which each consist of a selected sensor instrument, a survey mode, and a software based processing step. Compare these advanced methods to existing practices and validate the pilot technologies for the following: o Detection of UXOs o Identification of features that can help distinguish scrap and other clutter from UXO o Reduction of false alarms (items that could be safely left in the ground that are incorrectly classified as UXO) while maintaining Pds acceptable to all o Quantify the cost and time impact of advanced methods on the overall cleanup process as compared to existing practices. Understand the applicability and limitations of the pilot technologies in the context of project objectives, site characteristics, and suspected ordnance contamination Identify sources of uncertainty in the discrimination process and quantify their impact to support decision making, including issues such as impact of data quality due to how data is collected Explore how the dig-no dig decision process can be made. Examine potential QA/QC processes for discrimination Collect high-quality, well documented data to support the next generation of signal processing research Develop an objective methodology for field testing of discrimination systems. 2

3 2. Demonstration Design 2.1 Overall The ESTCP pilot program will explore four basic approaches to UXO detection, and discrimination. We will gather data to characterize the cost and performance of each of the following. The steps that will be included are pre-survey activities; survey; target selection/discrimination analysis; reacquisition and digging. Methods MAG AND FLAG AND DIG: The model for this is the traditional M&F approach, where every detection is dug. DGM AND DIG: The model for this is the limited use of digital data, as it is currently practiced on many sites. The geophysical map will be used to select targets, but no additional processing will be performed. All targets above a predetermined threshold, set by acceptable performance on the GPO, will be dug. DGM, CHARACTERIZE AND DIG: The model for this is the more sophisticated use of the geophysical data. For each target that is picked, the geophysical data will be analyzed to estimate the size and depth of the target. This information will be used to guide digging. We will attempt to measure the effect that guidance from this process has on the reacquisition and digging costs. Targets below a certain size threshold will be filtered out. DGM, CHARACTERIZE, DISCRIMINATE AND DIG: The model here is the full implementation of post-detection signal processing. As in the above, the geophysical data will be analyzed for estimation of target parameters, including not only size and depth, but also other features (e.g., polarizability and material properties) needed for the discrimination algorithms being tested. These properties will feed a signal processing stage to make a determination of the likelihood that the anomaly detected is an object of interest. The target parameters again, will be used to guide the reacquisition and digging. The overall ESTCP discrimination pilot study will consist of several combinations of data collection platforms and analysis approaches. These systems will collect data over a geophysical prove out (GPO) and the live ste study area. The sensor collections are summarized below. MAG AND FLAG The objective of the Mag and Flag survey is to provide a point of comparison on detection. A mag and flag contractor will be retained to perform an industry standard operation on the study area. The contractor will be instructed to survey the area to the industry standard based on Corps of Engineers Standard Operating Procedures and will be provided with descriptions of the targets 3

4 of interest. The team will survey the GPO and be given feedback on their detection of targets of interest. Then the team will be permitted to mag and dig a 100 X100 grid for training. They will then search the demonstration study area and place flags at anomaly locations. The flags will be surveyed by a registered surveyor and then removed. At the conclusion of the project, all flagged locations will be reacquired and dug. Because of the potential for mag and flag to produce an extremely large number of targets, the digging may be limited to 100% of the flagged targets in a few grids. DATA COLLECTION DIGITAL GEOPHYSICAL MAPPING The following sensors will be used for the data collection: In a survey mode these sensors will cover 100% of the site: o EM61 Mk II: Data will be collected by an existing COE contractor using a standard cart platform. The main selection criterion will be performance in the top band at the standardized test sites. Data requirements will be specified in the data collection plan and based on reliable detection of the targets of interest in the GPO. Typical industry standard equipment and procedures for navigation, geolocation (GPS), and data recording and reduction will be used o EM61 Mk II MTADS array: This platform is intended to provide the best quality data obtainable using EM61 technology. Data will be taken in the MTADS standard configuration using cm-level GPS and an IMU for geolocation. This configuration also provides platform orientation. o Magnetometer MTADS array: This platform is intended to provide the best quality data obtainable using Cs vapor technology. Data will be taken in the MTADS standard configuration using cm-level GPS for geolocation. o GEMTADS: This platform incorporates an array of frequency domain GEM sensor on a towed platform. Data will be taken in the MTADS standard configuration using cm-level GPS for geolocation. This configuration also provides platform orientation. In a cued mode the following sensors will investigate on the order of 100s of targets. Targets will be selected by the program office team to capture a range of SNR, sizes and depths based on the survey data. o EM63: The EM63 is a 26-channel time domain EMI instrument. It will be deployed on an air-suspension cart to gather cued data over a precise grid above some 100s of targets. It will use Robotic Total Station (laser) and IMU positioning. o Hand Held GEM-3: Will be used in a cued mode to gather data over a precise grid above some 100s of targets. The geolocation system for this sensor is TBD, awaiting demonstrator input. 4

5 The following sensor will operate in both a survey and cued mode: o LBL BUD (Berkeley UXO Discriminator): This is a developmental portable Active Electromagnetic (AEM) system that comprises three orthogonal transmitters for target illumination, and eight pairs of differenced receivers for response recording. It measures the entire decay curve up to ~1.4 ms at each point, although only data after 140 µs are used for interpretation. It will cover a subsection of the site, to be determined based on its productivity in upcoming tests. A portion of the LBL BUD data will be collected in a cued mode and the remainder in a survey mode. The cued mode data will be collected on all targets selected for GEM3 and EM63 analysis. BUD will collect data over approximately 5 acres of the site using its intended survey/self-cue operation. VALIDATION A master validation list will be produced by the ESTCP Program Office as the union of the data from the three complete surveys (EM61 MTADS, Magnetometer MTADS and GEMTADS). Duplicate detections will be eliminated where obvious. Where there is doubt, potential multiple detections of the same object will be retained and the uncertainty resolved in the dig process. All targets from all dig lists will be dug. If resources constrain, digs will at a minimum be performed on 100% of a portion of the site. Additional validation based on sampling of the complete dig lists may also be performed, with emphasis on validating high confidence non ordnance targets. The allocation of validation resources will be decided upon consultation with the Advisory Group. TESTING Blind testing under the ESTCP pilot will take place on the bulk of the site, and include all targets. The processors will be provided all geophysics data for the site, but not the truth data for the testing areas. Following submission of prioritized dig lists from all the signal processing groups, the validation data for the entire site will be released. This will include: o Dig information for every contact, at least for a sub set of the site (2000 anomalies) o Record of the location, size, depth, description This validation data will be used to provide an initial score for each approach for the purposes of a blind demonstration. This scoring will be done by the Institute for Defense Analyses (IDA) according to the criteria below. PRODUCTS FROM ESTCP DEMONSTRATORS At a minimum: o Ranked dig list for each sensor/processing combination o Derived parameters for each target from each method o A measure of the probability of ordnance likeness. o Cut-off threshold or thresholds for "no-dig" decision point. 5

6 o Cost and production rate information RANKED DIG LISTS AND THRESHOLDS The main concern of the regulatory community regarding discrimination is the high confidence determination that an item may be safely left in the ground. Dig lists will be ranked to reflect this. A schematic is shown below. The top item in the list should be that which you are most certain does NOT need to be dug up (shown in green). The bottom items should be those that you are most certain are munitions and must be dug (shown in red). A threshold should be set at the point beyond which you would recommend digging all targets, either because you are certain they are ordnance or because a high confidence determination cannot be made (heavy black divided). Two other bands should be specified indicating (1) the range of targets where the SNR, data quality or other factors prevent any meaningful analysis (shown in grey), and (2) the range of targets where the data can be fit in a meaningful way, but the derived parameters do not permit a conclusion (shown in yellow). These represent two levels of guessing. Rank Comment 1 2 High confidence NOT ordnance (no dig) 3 Can t make a decision (dig) Can t analyze (dig) Can t make a decision (dig) High confidence ordnance (dig) Selecting the Test Site Ultimately, multiple live site demonstrations will be conducted. Initially: Artillery or Mortar Target o One type or limited types of ordnance o Simple clutter environment 6

7 o Benign geology o Live ordnance used o Benign topography and vegetation to allow good data collection Basic Parameters o Approximately acres o Density commensurate with mostly isolated anomalies per acre o ~ anomalies in study area o Site to be seeded with like targets 2.3 Pre-Demonstration Testing and Analysis Predemonstration activities will include: Civil survey to establish cm-level control points to be used for all emplacement, data collection and validation activities. Magnetometer survey of approximately 50 acres, to be used to guide selection of the acre demonstration site, as well as to guide the emplacement of seed targets (discussed below). To preserve the integrity of a blind demonstration, this survey will be conducted by personnel not involved in the discrimination processing demonstration. Based on the magnetometer survey, one 100 X100 grid will be dug, in the manner generally used in EE/CAs. This information will be used to guide the seeding and to provide site-specific information to the demonstrators. This plot will be located nearer to the target center in a higher density area than is planned for the study area so that better statistics regarding type, quantity and depth distribution may be obtained. Magnetometer survey of nearby areas to guide selection of a site for the Geophysical Prove Out (GPO). Clearance of the GPO site and emplacement of GPO targets. Surface clearance of demonstration area, if needed. 2.4 Sensor Calibration Targets The calibration targets will be surveyed morning and evening of each data collection day to verify system operation. The target emplacement parameters are shown in Table 2.1. Data will be digitally recorded, checked for appropriate signal strength and noise levels immediately, and inverted in post processing to verify consistency of parameter estimation. Calibration targets will be separated by a distance sufficient to ensure signals are not overlapping. Locations will be surveyed in using the on-site civil survey control points. 7

8 Table 2.1. Calibration Targets (example, will include all ordnance of interest once the site is finalized) Item Depth Orientation Site UXO Type 1 Site UXO Type 2 5 X diameter 11 X diameter 5 X diameter 11 X diameter NS, EW NS, EW NS, EW NS, EW Copper ring Y inch diameter ferrous sphere 6 inches 5 X diameter 11 X diameter 2.5 Geophysical Prove Out (GPO) A geophysical prove out will be established to verify detection thresholds for all instruments.. The intent of the GPO is to verify that the targets of interest are detected at the depths of interest under site specific conditions at the selected threshold. The thresholds to be used to select the target list for each sensor on the field site will be based on GPO results. No attempt will be made to detect lower signal targets than those determined to be of interest prior to the study. All subsequent discrimination analysis will be performed from a target list determined by application of the threshold established in the GPO analysis. The location of the GPO will be based on the pre-demonstration magnetometer survey. A relatively anomaly-free area will be chosen and it will be cleaned prior to emplacement. It will be approximately 1 acre in size. The GPO will contain targets, distributed as shown in Table 2.2. The burial depths will be biased in the direction of deeper depths. Several of the items of each type will be buried in the 3x-5x range and a portion in the very near surface. Targets will be separated by a minimum of 6 m in all directions. Target locations will not be on a regular grid and will not be known by demonstrators. Locations will be surveyed relative to a cm-level marker on the site. Depth and location information will be collected for the nose, tail and center of each item. Center locations will be used for all analyses. Prior to conducting any field work, each data collection demonstrator must show that all targets of interest (i.e., up to depths of 11 times their diameters) are detected using the procedures that will be used in the field. Targets should be detected at sufficient SNR in the GPO to ensure reliable detection in the field. 8

9 Table 2.2. GPO Targets (example, will include all ordnance of interest once the site is finalized) Munition type Quantity Depth Range Orientations Site UXO, Type 1 TBD Flush buried to 11X Random orientations, predominantly from +/- 45 from horizontal, few vertical targets Site UXO, Type 2 TBD Flush buried to 11X As above The GPO data will be used to set appropriate detection thresholds as indicated in Figure 1. The lines indicate the predicted magnetic signal strength for the most and least favorable orientations of the object of interest. The geophysical system will collect data over the GPO plot. The demonstrator will select all detectable targets. The detections corresponding to the item of interest will be plotted as shown by the black symbols, to demonstrate that the system is working properly (i.e., that the measured signal strengths will fall between the two bounds). The minimum signal strength at the maximum depth of interest will be used to set the threshold for target detection with an appropriate safety margin. Initial analyses will be performed with a factor of two margin, which will be adjusted if necessary for example, in the event that all seeded targets are not detected in the survey data. 9

10 10000 most favorable orientation least favorable orientation Peak Anomaly (nt) APG Noise 1 Pueblo Noise Depth (cm) Figure 1. Signal versus depth plot for most and least favorable orientations. 2.6 Seeding The demonstration area will be seeded with targets comparable to the munitions found at the site. To the extent available, recovered munitions will be used for the seed targets. The exact x, y location, depth to the center of the target and orientation will be recorded for each emplaced item. Objects will not be emplaced at depths in excess of 11 times their diameter, the de facto expectation for detectability with modern geophysical equipment. The depth distributions will mimic the depths observed in the predemonstration grid. The objective is to emplace the seeded targets so that they are representative of what would be realistic conditions at the site. As such, unless the observed depth distribution in the dug grid dictate, there are no plans to seed targets just above the threshold SNR or pose other concocted challenges. The target seeding plan is in Table

11 Table 2.3. Blind Seeded Targets Munition type Quantity Depth Range Orientations Site UXO, Type 1 TBD Flush buried to 11X Site UXO, Type 2 TBD Flush buried to 11X Random orientations, predominantly from +/- 45 from horizontal, few vertical targets As above 11

12 3. ESTCP Performance Assessment 3.1 Performance Criteria and Confirmation Methods The program performance criteria for this demonstration are detailed in Table 3-1. A TARGET OF INTEREST shall be defined to include Intact munitions, both HE and practice Sizeable pieces of the munitions of interest, on the order of half a round Items that look like munitions (i.e., pipes of similar size) ITEMS THAT MAY BE SAFELY LEFT IN THE GROUND shall include HE fragments, single fins, cultural debris and geology. In the event that the site chosen contains small arms rounds such as 50-cal bullets, these shall also be treated as items that may be left in the ground. HAZARDOUS COMPONENTS OF MUNITIONS such as fuzes, spotting charges, bursters, and the like will be treated separately. Once a site has been selected and the investigation grid dug, specific examples of targets of interest and items that may be safely left in the ground will be provided. Both photos and descriptions will be prepared. The focus will be on identifying items that may be safely left in the ground. For the purposes of the ESTCP demonstration, the main failure is misclassifying a target of interest as an item that can be left in the ground. Incorrect identification of a clutter item as UXO will be noted and may be addressed in post demonstration failure analysis. Individual targets will be scored as indicated in the following table. These labels will then be used to calculate the rolled up performance measures below. 12

13 Table 3-1 Scoring Criteria Demonstrator Says Target of Interest Not Target of Interest Can t tell Truth UXO or Intact Training Munition Munition debris (sizeable pieces of munitions - on the order of half a round) Pass Fail Type I Pass Pass Fail Type I Pass Frag/small scrap Fail Type II Pass Fail Type II Non-munitions debris Fail Type II Pass Fail Type II Noise Fail Type II Pass Fail Type II Geology Fail Type II Pass Fail Type II Type I = False Negative; Type II = False Positive The performance of the individual technologies and their combined analyses will be measured against the criteria listed in Table 3-1. All demonstrator dig lists will be scored against the emplaced and recovered targets by IDA, which will produce a separate scoring document describing the procedures to be used in scoring the detection and discrimination phases of the test. An example ROC curve is shown in Figure 2, with the areas of interest for the analysis indicated. The scoring will be done in two phases. The first detection phase will assess detection of all targets of interest (i.e., above the SNR associated with acceptable performance on the GPO as discussed above). The percent of seeded items detected will be used as QC for this step. The goal of the detection step is only to identify candidate targets with appropriate signal strength, so no penalty is associated with the inclusion of items that are not ordnance objects of interest. These items will simply be tallied for use as the possible number of clutter items that could be eliminated by correct classification in the next step. 13

14 The detection step for each of the survey systems will be performed by the program office team. The contractor will pick the cart data. The threshold for this step will be selected as described above, consistent with detecting the weakest signal from an item of interest with an added safety margin. These target lists will be used as the basis in the subsequent discrimination scoring. A check will be made to ensure that all seeded targets are captured by this method. Approximately 2000 targets per list are expected. The lists will not necessarily be identical, as the intent is to retain the targets that are unique to each list and investigate the strengths and weaknesses of each data set. Analysis approaches using joint or cooperative inversions will evaluate all the targets that appear in any of the relevant lists. The intent of this demonstration is to assess performance against individual targets. All targets selected will be isolated anomalies to the best of our analyst s ability to distinguish them. Clusters will be masked out and may be studied in subsequent analyses. The discrimination phase will order the dig lists by likelihood of ordnance or clutter. The demonstrator will specify a threshold or thresholds beyond which all objects are predicted to be nonordnance. The demonstrator may indicate a portion of the dig list as cannot classify. The Pdisc calculated for this step will use the total number of UXO detected in the prior step by the sensors used as the possible number of correctly discriminated targets (i.e., you can t be penalized for not discriminating something that was never detected.) Note, however, that cooperative or joint inversion algorithms utilizing multiple sensor data will be scored on the union of the detection lists of the sensors employed. The Pfa will be measured relative to the number of clutter items identified in the detection step. 14

15 1 Probability of Detection Point beyond which all detections are clutter Demonstrator Selected No-Dig Probability of False Alarm Figure 2. Example ROC curve 15

16 Table 3-2. Confirmation Methods to Be Employed Performance Metric Pdet (emplaced) Pdet (recovered ord) FA count for the detection step Pdisc (emplaced) Pdisc (recovered) Pfa at 100% Pdisc Threshold ID ROC Accuracy of Parameter Estimation (Examples) Applicable Technologies All Survey All All All All All All All All Description Number of emplaced objects on dig list/number of emplaced objects Number of recovered ordnance on dig list/number of recovered ordnance Number of picks on the dig list not ordnance; to be used as the denominator for the Pfa below in the discrimination step Number of emplaced objects above threshold on discrimination list/number of emplaced objects detected Number of recovered ordnance above threshold on discrimination list/number of recovered ordnance detected Number of FA above threshold/number of FA in entire dig list Ability to select the appropriate stop dig point(s) Comparison over the entire operating space β X, Y Z size Confirmation Method Comparison to seeded items Validation Digging Validation Digging Comparison to seeded items Validation Digging Validation Digging Gains over chance diagonal Comparison to truth data 16

17 5. References 1. Report of the Defense Science Board Task Force on Unexploded Ordnance, December 2003, Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics, Washington, D.C , 17

18 List of Acronyms 11X BUD DGM DSB EE/CA EMI ESTCP FAR GPO GPS HE IDA LBL M&F MR MTADS Pdet Pdisc Pfa QA QC ROC SNR UXO 11 times the object s diameter Berkeley UXO Discriminator Digital Geophysical Mapping Defense Science Board Engineering Evaluation/Cost Analysis Electromagnetic Induction Environmental Security Technology Certification Program False Alarm Rate Geophysical Prove Out Global Positioning System High Explosive Institute for Defense Analyses Lawrence Berkeley Laboratory Mag and Flag Munitions Response Multisensor Towed Array Detection System Probability of Detection Probability of Correct Discrimination Probability of False Alarm Quality Assurance Quality Control Receiver Operating Characteristic Signal to Noise Ratio Unexploded Ordnance 18

19 GLOSSARY Discrimination Detection False Alarm (Detection) False Alarm (Discrimination) Target of Interest Characterization Classification Determination that a detected object is (1) high confidence clutter (and need not be dug), or (2) ordnance or unknown (and must be removed) Determination of the presence of a target, typically by observation of a signal level crossing a threshold set to limit the probability that a crossing would be caused by noise or interference. Declaration of a target that is actually caused by noise, interference, or geology Declaration of an item that could safely be left in the ground as ordnance or unknown For this study, defined to be intact munitions, both HE and practice; sizeable pieces of munitions (on the order of half a round); and items that look like munitions (e.g., pipes of similar size) Determination of parameters that are intrinsic to a target and can be used to make a discrimination decision. Formally, determination that an object belongs to a particular class of ordnance (i.e., is a 155 as opposed to an 81). Classification, by its formal definition, will not be explored in this study. Instead, in this document we will use the term classification as a synonym for discrimination. 19