Introduction to Classification Methods for Military Munitions Response Projects. Herb Nelson

Transcription

1 Introduction to Classification Methods for Military Munitions Response Projects Herb Nelson 1

2 Objective of the Course Provide a tutorial on the sensors, methods, and status of the classification of military munitions using geophysical methods Advanced processing of data collected with existing commercial instruments Promising results from emerging optimized systems Introduction 2 SERDP and ESTCP have supported a number of investigators over the years who have developed processing approaches to extract target-specific attributes from data collected by commercial geophysical sensors, and demonstrated advanced sensors designed with the munitions response problem in mind. These research efforts have resulted in an impressive ability to classify the source of geophysical anomalies as targets-of-interest or non-hazardous items under simple conditions with the promise of expansion to a wider range of conditions as the newest sensors mature. This course is intended as a tutorial on these classifications methods. We begin with a brief introduction to some of the terminology and concepts that will be used, introduce the basics of the two primary geophysical instruments used in munitions response, discuss the methods used for classification and illustrate them with two case studies, preview the next generation of EM sensors emerging from the research program, and conclude with a brief summary and presentation of a idealized cost model for classification. 2

3 Presenters Dr. Steve Billings (Sky Research) Dr. Thomas Bell (SAIC) Dr. Dean Keiswetter (SAIC) Introduction 3 The success of this course is due to the hard work of our three primary presenters. Dr. Steve Billings from Sky Research will introduce the terminology to be used and discuss the concepts of magnetics. Dr. Tom Bell from SAIC will discuss the basics of EM sensors and later in the course present examples of the capabilities of the emerging EM sensors. Dr. Dean Keiswetter of SAIC will discuss the methods used for classification and follow with two case studies that illustrate the performance that can be achieved using commercially-available sensors. 3

4 The Munitions Problem There are over 3,000 sites suspected of contamination with military munitions They comprise 10s of millions of acres The current annual cleanup effort is on the order of 1% of the projected total cost To make real progress on this problem, we need a better approach Introduction 4 There are a very large number of sites in the US suspected of being contaminated with military munitions but the remediation budget each year represents only about 1% of the multi-billion dollar projected total remediation cost. This leads to remediation projects having planned completion dates late in this century. Given budget realities, the only way to accelerate this effort is to develop methods to accomplish more remediation with the available funding. 4

5 Cost - $B Munitions Response Cost Breakout 5 0 Site Assessment Indirect Cost Direct Cost Survey and Mapping Vegetation Removal Scrap Metal Removal UXO Removal & Disposal Defense Science Board Task Force on UXO Introduction 5 This chart, from the 2003 report of the Defense Science Board Task Force on UXO [ shows us one approach to the savings we seek. On a typical munitions clean-up project, an overwhelming fraction of the money is spent removing non-hazardous items from the site. If a method can be devised to identify these non-hazardous items and remove them with fewer safety precautions or leave them in the ground, this money could be transferred to other projects. 5

6 Classification Classification offers the chance to divide anomalies into those caused by targets of interest and those caused by other things Recognize that current field methods involve implicit discrimination Mag & Flag instrument sensitivity setting and human interpretation Digital Geophysics threshold selection; what is a target? Our goal is a principled, data based approach to classify targets as non hazardous or targets of interest Introduction 6 As we saw in the last slide, classification (sorting the sources of geophysical anomalies into targets-of-interest and non-hazardous items) holds the promise of real cost savings. Many stakeholders, however, are leery of applying classification methods at their site. It is important that they recognize that classification is being performed implicitly now. In an analog geophysical survey (often termed Mag & Flag), the operator decides on the instrument sensitivity to select and what level of response to call a hit. Neither of these choices can be revisited after the survey is complete. Even if digital geophysical mapping techniques are used, the data analyst makes decisions about what to call an anomaly, often on-the-fly and without defined procedures. What we seek is a principled, data-driven approach to classification. This involves data collection and analysis methods as well development of a process in which all stakeholders can have confidence. 6

7 This Page Has Been Intentionally Left Blank

8 Terminology and Concepts Stephen Billings 1

9 Acronyms and Definitions Munitions and Explosives of Concern (MEC) Military Munitions - Unexploded Ordnance UXO - Discarded Military Munitions DMM Munitions Constituents MC Geophysical sensors detect metal (UXO, DMM and other metallic items) and cannot directly detect munitions constituents Terminology and Concepts 2 Munitions and Explosives of Concern (MEC). This term, which distinguishes specific categories of military Munitions that may pose unique explosives safety risks, means: (A) Unexploded Ordnance (UXO), as defined in 10 U.S.C (e) (9); (B) Discarded military munitions (DMM), as defined in 10 U.S.C (e) (2); or (C) Munitions constituents (e.g., TNT, RDX) present in high enough concentrations to pose an explosive hazard. Unexploded Ordnance (UXO). Military munitions that: (A) Have been primed, fused, armed, or otherwise prepared for action; (B) Have been fired, dropped, launched, projected, or placed in such a manner as to constitute a hazard to operations, installations, personnel, or material; and (C) Remains unexploded whether by malfunction, design, or any other cause. (10 U.S.C. 101(e)(5) Discarded Military Munitions (DMM). Military munitions that have been abandoned without proper disposal or removed from storage in a military magazine or other storage area for the purpose of disposal. Munitions Constituents (MC). Any materials originating from unexploded ordnance, discarded military munitions, or other military munitions, including explosive and nonexplosive materials, and emission, degradation, or breakdown elements of such ordnance or munitions. (10 U.S.C (e) (4)) 2

10 Acronyms and Definitions Target of Interest (TOI): Military munitions, explosive fragments, fuzes, items that give a sensor response indistinguishable from military munitions Non TOI: Fragments, clutter, cultural items, etc. that are not hazardous Parameter Estimation: The extraction or estimation of parameters that represent some useful attributes of the buried object. Also referred to as parameter extraction and geophysical inversion. Terminology and Concepts 3

11 Digital geophysics Requires a geophysical sensor system (based on either magnetometry or electromagnetic induction ) A positioning device (e.g. Global Positioning System, GPS) A computer for digital data acquisition Magnetometer Electromagnetic sensor Location device (e.g. GPS) Geophysical sensor Digital data acquisition Terminology and Concepts 4 The three essentials elements of a digital geophysical mapping system are illustrated on this slide. Note that, in addition to the location device, measurement of sensor orientation can prove useful. 4

12 Standard processing stream The standard processing stream for detection and classification of munitions using geophysical data 1. Data Collection 2. Parameter Estimation (Target Attributes) 3. Classification data Non-munitions Munitions Parameters Terminology and Concepts 5 Schematic showing the standard process flow of a digital geophysical survey. The data are collected and captured by a data logger. After the survey is finished, the data are typically transferred to another computer where initial data processing and then parameter estimation are performed. The parameters are then used to make classification decisions. 5

13 Collect Data Terminology and Concepts 6 An example of this process flow taken from the seafood industry. The task is to automatically sort fish as either salmon or sea bass. The first step is to collect data. In this case, the sensor of choice is a camera that captures a digital photo of the fish to be classified. 6

14 Extract Parameters Terminology and Concepts 6 Step 2 is to extract some parameters from the data. One obvious parameter is the length of the fish. 7

15 Extract Parameters Terminology and Concepts 6a Another parameter that might be of value is the width. 8

16 Extract Parameters Terminology and Concepts 6b Or the overall coloration of the fish. 9

17 Extract Parameters Terminology and Concepts 6c Or the number and placement of fins. 10

18 Classify Based on the Extracted Parameters Terminology and Concepts 7 Using the extracted parameters, we will attempt to classify this fish. Here we see that while salmon are, on average, shorter than sea bass there is significant overlap between the two classes so this is not the best parameter to choose for classification 11

19 Classify Based on the Extracted Parameters Terminology and Concepts 7a Now let s try lightness. There is less overlap between the classes using this parameter so we are making progress. 12

20 Classify Based on the Extracted Parameters Terminology and Concepts 7b Finally, we see that using both lightness and width we have defined a space that allows us to reliably separate salmon from sea-bass. 13

21 Data and target attributes Parameter extraction methods are used to estimate the attributes of a buried object from the measured data Data profiles Anomaly in geophysical data caused by buried object Ground-surface Buried Item The buried item is parameterized by a set of magnetic or electromagnetic target attributes that indirectly reflect the physical characteristics of the object Terminology and Concepts 8 Schematic of magnetometer data collection. Data are collected along nominally straight pathways at a desired lane spacing. Each black dot shows the location of a magnetic measurement with the magnetic data shown in red. Positive values lie above the measurement plane, with negative values below. 14

22 Classification vs identification Classification: Distinguish Military Munitions (the Targets of Interest or TOI) from shrapnel, range scrap, cultural debris etc (non TOI) Identification: Determine the type of Military Munition Classification Identification Target of Interest Non Targets of Interest Terminology and Concepts 9 In this context, we seek to classify items as Targets of Interest (UXO or DMM) or Non-Targets of Interest (all non-hazardous items on the site). There may be value in identifying individual sub-classes (e.g. 81-mm mortars) but that is not our primary objective. 15

23 Receiver Operating Characteristic (ROC) Curve Used to characterize the performance of a munitions classification strategy Proportion of Targets of interest Can t analyze Classification 100% Detection ~260 False Positives Detection Only 100% Detection ~430 False Positives Desired Performance 100% Detection No False Positives Terminology and Concepts Non-Targets of interest 11 We measure the performance of a classification effort using what is known as a Receiver Operating Characteristics (ROC) curve. It is retrospective in that it requires all ground truth. This curve was taken from a study at Camp Sibert in which all contacts were dug for learning purposes. Basically, it is a plot comparing the number of actual non-toi objects versus TOI if our prioritized dig list were excavated in order. Desired Performance Ideally, we would like perfect classification. If we could achieve this, the curve would rise straight to this location: 100% TOI recovered and 0% non-toi. Detection Only This is the other extreme. Here, we ignore classification altogether and simply excavate all detected targets. In the end, we have 100% of the TOI recovered (approximately 150 munitions), but we also have removed 100% of the non-toi which for this site was approximately 600 non-hazardous objects. Classification Here, we evaluate the classification results. This is the point at which the demonstrator drew the threshold between high-confidence non-toi and everything else. As shown here, 100% of the munitions were recovered but only 200 of the non-hazardous non-toi recovered. This is a huge success. Can t Analyze This portion of the curve is reserved for those anomalies that were thrown into the Can t Analyze category. Because no useful information regarding the nature of these targets can be extracted from the measured data, they must be treated as potential targets of interest. 16

24 Signal to Noise Ratio (SNR) EM61 (gate 3) rms noise = 0.25 mv S max = 38.3 mv, SNR = 153 S max = 4.5 mv, SNR = 18 S max = 0.75 mv, SNR = 3 Terminology and Concepts 12 Signal to noise ratio (SNR), as the name suggests, is the ratio of the amplitude of the signal relative to the amplitude of the noise. Thus SNR can be varied by changing the characteristics of either the signal or noise. The root-mean-square (rms) noise is 0.25 mv in each of these three panels. The maximum amplitude of the signal varies from 38.3 mv in the top panel to 0.75 mv in the bottom panel. The SNR decreases with decreasing signal amplitude from 153 (signal amplitude 153 times larger than the rms noise) in the top-panel down to 3 in the bottom panel (signal amplitude 3 times larger than the rms noise). Detection of the signal with SNR = 3 would be very difficult, whereas at the higher signal levels detection is easy. 17

25 This Page Has Been Intentionally Left Blank

26 Magnetics: Fundamentals and Parameter Extraction Stephen Billings Magnetic module outline Magnetics fundamentals Sensor systems Data examples and demo Parameter extraction Concepts Real-world examples Classification Using the parameters to make discrimination decisions Magnetics 2 1

27 Standard processing stream The standard processing stream for detection and classification of UXO using geophysical data 1. Data Collection 2. Parameter Estimation (Target Attributes) 3. Classification data Non-UXO UXO Parameters Magnetics 3 Schematic showing the standard process flow of a digital geophysical survey. The data are collected and captured by a data logger. After the survey is finished, the data are typically transferred to another computer where initial data processing and then parameter estimation are performed. The parameters are then used to make classification decisions. 2

28 Detection of metal with a magnetometer Most ordnance contain ferrous metal Ferrous metal causes a distortion of the Earth s magnetic field No object Ferrous ordnance Magnetics 4 The Earth has a magnetic field whose direction and magnitude varies across the surface of the Earth. Within most of North America, the magnetic field lies within 10 degrees of true-north and is oriented at about 65 degrees from horizontal. Magnetic field-lines are straight when there is no metallic object present. When a ferrous object is introduced, the field lines become distorted and are essentially attracted into the object. 3

29 Total-field magnetometers Measure the total magnetic field (generally in the direction of the ambient field) In Northern hemisphere, positive lobe to south and negative lobe to the North Ferrous ordnance Object only Ambient field South North Magnetics 5 The distortion of the magnetic field caused by a compact object like a UXO can be approximately modeled as a magnetic dipole (essentially a bar-magnet with a north and south pole on either end). The magnetic field lines leave one end of the object (the south-pole) and wrap around and re-enter the object at the other end (the North pole). Most magnetometers in use today measure the total magnetic field. The Earth s field is large (around 50,000 nt) compared to the distortions caused by buried metal (typically 1 to 1,000 nt). By subtracting off the Earth s field, we see that the total-field anomaly caused by a buried object is positive when the field from the object is in the same direction as the earth s magnetic field and negative when it is in the opposite direction. In the Northern hemisphere this causes a positive lobe to the south and a negative lobe to the north of the buried item. 4

30 Data collection systems Single-sensor G858 Portable quad-sensor array Cart-based quad-sensor array Towed-array with 8 sensors (MTADS) Magnetics 6 Examples of magnetometer data collection systems. Each of the systems has one or more magnetic sensors, a positioning system (either Global Positioning System or Robotic Total Station) and a data logger for digital capture of the sensor and position data. 5

31 What are data Data profiles Sensor locations Burial depth Survey height Measurement plane Ground-surface Magnetics Ordnance Item 7 Schematic of magnetometer data collection. Data are collected along nominally straight pathways at a desired lane spacing. Each black dot shows the location of a magnetic measurement with the magnetic data shown in red. Positive values lie above the measurement plane, with negative values below. 6

32 Data collection Magnetics 8 Screen shot of the data collection animation 7

33 Data collection Magnetics 8a Screen shot of the data collection animation 8

34 Data collection Magnetics 8b Screen shot of the data collection animation, showing a gridded image of the data in plan view. The magnetic data are plotted using the color-scheme shown by the color-bar at right. For instance, any regions where the magnetic field is 70 nt are colored red, while regions with -10 nt are shown in blue. The anomalous field is plotted here; the Earth s field has been subtracted from the measurements. 9

35 Real time demonstration Object attributes Map view Survey attributes Perspective view Profile view Magnetics 9 Screen-capture of the set-up used for real-time demonstration of magnetic data. 10

36 Effect of item depth 20 cm 60 cm 40 cm 80 cm Magnetics 9a As the depth of the item increases, the amplitude decreases and the distance between the positive and negative peaks increases. The amplitude decreases as the third-power of distance away from the sensor. The effect of increasing the depth by 10 cm while keeping the sensors the same distance from the ground is the same as increasing the sensor distance by 10 cm without changing the burial depth. 11

37 Effect of item size 50 mm 100 mm 75 mm 125 mm Magnetics 9b As the size of the item increases so does the magnitude of the magnetic anomaly that it creates. The amplitude increase is proportional to the diameter of the object. Thus, if the object diameter is doubled, then so is the magnitude of the anomaly. 12

38 Effect of item orientation Dip 0; Azimuth 0 Dip 0; Azimuth 20 Dip 65; Azimuth 0 Dip 0; Azimuth 40 Magnetics 9c The magnitude and shape of a magnetic anomaly depends on the orientation of the buried object. The amplitude is largest when the long axis of the object is aligned along the direction of the Earth s magnetic field and is smallest when oriented perpendicular to that direction. 13

39 Signal to noise ratio Depth 0.4 m Noise 0 nt Depth 0.4 m Noise 4 nt Depth 0.4 Noise 2 nt Depth 0.2 m Noise 4 nt Magnetics 9d Signal to noise ratio (SNR), as the name suggests, is the ratio of the amplitude of the signal relative to the amplitude of the noise. Thus SNR can be varied by changing the characteristics of either the signal or noise. For a given munition at a given depth and orientation, the SNR decreases as the noise measured by the sensor increases. The lower the SNR the lower the detection probability and the harder it becomes to extract parameters that reflect the intrinsic attributes of the buried object. At a fixed noise level, the SNR is also decreased by increasing the depth of the object. 14

40 Lane spacing 25 cm 75 cm 50 cm 100 cm Magnetics 9e The distance between adjacent sensor paths (the lane spacing) needs to be small enough to capture the full character of an anomaly. In this example, information on the shape of the anomaly is lost as the lane-spacing is increased from 25 cm to 75 cm. 15

41 Position error 0 cm 4 cm 2 cm 6 cm Magnetics 9f Error in the position of the sensor when it takes a magnetic measurement distorts the measured magnetic anomaly. 16

42 Data collection summary Item attributes impact the shape, size and amplitude of the anomalous magnetic field: Depth; Orientation; Size of UXO Sensor attributes that effect the quality of the data Sensor noise Line spacing Positional error Sensor height above ground (and any variation) Magnetics 10 17

43 Detection performance Detection performance is dependent on Object size Noise Data density 4.2 mortar Magnetics 11 This slide shows how the amplitude (measured as the difference between the positive and negative parts of an anomaly) of a 4.2 mortar varies as the depth below the surface is increased. There is a significant difference between the least favorable (mortar at right-angles to the Earth s magnetic field) and most favorable (mortar parallel to the Earth s field) orientation. Also shown are the amplitudes of magnetic anomalies observed over 100 different 4.2 mortars that were seeded at Camp Sibert, Alabama. The magnetic sensor was 30 cm above the ground. An example noise floor of 10 nt is marked on the graph. 18

44 Detection performance A smaller 60 mm mortar has a reduced detection depth 4.2 mortar 60 mm mortar Magnetics 12 The smaller the object, the smaller the anomaly amplitude and hence the shallower the detection depth. This is evident in the above plot where the 60 mm anomaly amplitude intersects the notional 10 nt noise line at shallower depths than the 4.2 mortar. At more favorable orientations, the anomaly amplitude remains above the noise floor to greater depths. 19

45 Target picking processing flow Magnetometer data are collected along survey lines Geophysicist reviews and processes the profile data Bad data are rejected (e.g. out of range) Filters are applied to suppress diurnal changes in the magnetic field and longer wavelength features due to geology Data are generally gridded to produce an image of the magnetic data Regions of anomalous response are selected as potential metallic targets Magnetics 13 The above represents the standard processing flow used for digital geophysics. 20

46 Anomaly identification Total-field data from Montana Raw total field data from a 100 m by 100 m area at Chevallier Ranch Montana Magnetic field (nt) Northing (m) Easting (m) Magnetics 14 This slide shows a three-dimensional perspective view of the magnetic field over a 100 m by 100 m area of the Chevallier Ranch site in Montana. Sub-surface metallic objects cause localized distortions in the measured magnetic field. More extensive, largely linear features in the magnetic data are caused by variations in the magnetite content of the underlying geology (trending roughly east-west), or by magnetite transported along drainage channels (the north-south features). 21

47 Anomaly identification Total-field data from Montana Magnetic data after application of filters Magnetic field (nt) Northing (m) Easting (m) Magnetics 15 An appropriately tuned high-pass filter (which passes the shorter spatial scale [higher spatial frequency] target response while removing the longer spatial scale [lower spatial frequency] background interference) can be used to suppress the effect of the longer-wavelength geological features while accentuating the localized anomalies caused by buried metal. 22

48 Anomaly identification Plan-view Magnetics 16 This is a plan-view of the same image as the last slide. 23

49 Anomaly identification Targets picked Magnetics 17 The locations of potential ordnance items are either selected manually or by automatic target selection methods. 24

50 Examples of good data High SNR 210 nt Medium SNR 22 nt Easting (m) Easting (m) Medium SNR Low SNR Northing (m) Northing (m) Northing (m) -140 nt -21 nt 8.5 nt 4.3 nt Northing (m) -7.5 nt -2.7 nt Magnetics Easting (m) Easting (m) 18 The four anomalies shown in this slide were obtained by the MTADS magnetometer array at Camp Sibert, AL. Each anomaly has dense data coverage (the black dots) and no obvious distortions caused by positional or other errors in the data. Notice the apparent striping caused by background geology in the image on the lower right. 25

51 Examples of poorer data Data gap, low SNR 4.2 nt Data gap 18 nt Easting (m) Interference with geology -1.3 nt 4.2 nt Easting (m) Geology or sensor lag? -8 nt 14 nt Northing (m) Northing (m) Northing (m) Northing (m) Magnetics Easting (m) -1.4 nt Easting (m) -10 nt 19 The anomalies shown in this slide were collected by a man-portable magnetometer array at Chevallier Ranch, MT, under more challenging conditions than those at Camp SIbert. The top two anomalies suffer from data gaps caused by variations in the lane-spacing as the operator avoids small bushes at the site (the black dots mark the sensor locations). The bottom two anomalies are distorted either by geology or by positional inconsistencies between adjacent traverses over the anomaly. 26

52 Magnetic module outline Magnetics fundamentals Sensor systems Data examples and demo Parameter extraction Concepts Real-world examples Classification Using the parameters to make discrimination decisions Magnetics 20 27

53 Parameter extraction The data that are measured are an indirect indicator of what is buried under the ground Inversion or parameter extraction is used to estimate the parameters of an underlying model that encapsulates some useful attributes of the buried object Model Parameters: m Forward Operator d =g [m] Sensor data: d Magnetics m=g -1 [d ] Inverse Operator 21 The data themselves do not directly tell us if the underlying object is a munition or something non-hazardous like shrapnel, range scrap or cultural debris. The objective of parameter extraction is to estimate the parameters of an underlying model that encapsulates some useful attributes of the buried object. The forward problem involves estimating the magnetic anomaly caused by an object with particular attributes. The parameter extraction, or inverse, operation is more difficult and involves estimating the object attributes from the measured data. 28

54 Real-time demonstration of parameter extraction Observed data Azimuth wrong Adjust azimuth Magnetics 22 Screen-shots of a real-time demonstration of parameter extraction. The observed data are shown on the left. The data that would be produced by an initial guess at the underlying target attributes are shown in the center. They do not provide a good match to the data. The parameter extraction method (in this case we use physical intuition) adjusts the azimuth so that the orientation of the modeled anomaly now looks correct (the size and shape of the anomaly are still wrong at this stage). 29

55 Real-time demonstration of parameter extraction Observed data Depth wrong Adjust depth Magnetics 22a The modeled data in the middle panel produce a smaller anomaly than what was observed. The modeled data agree much better with the observed data after the target is pushed deeper. The anomaly amplitude still doesn t agree. 30

56 Real-time demonstration of parameter extraction Observed data Size wrong Adjust size Magnetics 22b The depth and orientation are correct but the size is wrong. The amplitudes of the observed and modeled data match closely after increasing the size of the target. 31

57 Demonstration of parameter extraction Observed data Predicted data Extracted parameters Magnetics 22c The parameters that provide the best-match to the observed data reflect our best estimate of the target attributes of the underlying object. 32

58 Parameter extraction The demonstration we have just seen described one method of parameter extraction Search by trial and error with a visual assessment of what model fits the best In practice, highly efficient automated parameter extraction techniques based on non linear least squares are used The objective is to minimize the difference between the actual and predicted data Magnetics 23 33

59 The model We model the response of buried items by a dipole (equivalent to a bar magnet): Position Depth Orientation Size N ORIENTATION S DEPTH Position Magnetics 24 A magnetic dipole is used as the underlying model for parameter extraction from magnetic data. The dipole is equivalent to a bar-magnet, whose lateral position, depth, orientation and magnitude need to be estimated. 34

60 Parameter extraction DATA MODEL Magnetics Residuals Easting = 0.10 m Northing = 0.21 m Depth = 0.46 m Moment = 0.20 Am 2 Azimuth = 31.5 o Dip = o Fit quality = 0.98 Parameters of interest 25 Example of a dipole model fit to magnetic data collected at Chevallier Ranch, MT. The panel shows the observed data (top left), modeled data (top-right), residuals (which are the difference between observed and modeled data, bottom-left) and extracted parameters (bottom right). While the estimated lateral position and depth are important, they don t tell us anything substantial about the possible identify of the object. The moment, azimuth and dip, which encapsulate the size and orientation of the underlying dipole, provide information on the likelihood that the underlying object is a munition. The parameter extraction technique returns an estimate of the fit quality which is an indicator of the reliability of the parameter estimates. If the fit-quality is low, then there is considerable uncertainty in the values of the underlying target attributes. In this case the fit-quality is high. 35

61 Parameter extraction DATA MODEL Magnetics Residuals Easting = m Northing = 0.16 m Depth = 0.26 m Moment = Am 2 Azimuth = 37 o Dip = 28.8 o Fit quality = 0.95 Parameters of interest 26 Another parameter extraction example from Chevallier Ranch. In this case, the moment is about 1/10 th the size of the previous example. 36

62 Can t analyze DATA MODEL Could be 2 anomalies or a mismatch in position on adjacent passes Magnetics Residuals Easting (m) Easting = 0.20 m Northing = For some m anomalies the data are not of sufficient Depth = 0.26 m quality to support reliable Moment = parameter 0.05 Am 2 extraction. We Azimuth = refer 95.5 o to these as can t Dip = -13 o analyze anomalies Fit quality = Example of an anomaly with low fit quality. In this case, we can t rely on the extracted parameters and would place the anomaly in a can t analyze category. In the absence of further information, this anomaly would need to be treated as a potential target-of-interest. 37

63 Magnetic module outline Magnetics fundamentals Sensor systems Data examples and demo Parameter extraction Concepts Real-world examples Classification Using the parameters to make discrimination decisions Magnetics 28 38

64 Classification The objective of a UXO remediation strategy is to produce a prioritized dig list with an indication of how many items have to be excavated as potential UXO The topic will be covered in detail by Dean Keiswetter in a later module. Here we just provide an example of ranking the digsheet based on the size of the moment Magnetics 29 39

65 Example: Camp Sibert Reject as much clutter as possible Without leaving any 4.2inch mortars unearthed. Magnetics Target of Interest Non Targets of Interest 30 The example that follows comes from Camp Sibert, AL where the objective was to recover all 4.2 mortars while leaving as much clutter in the ground as possible. 40

66 Size versus amplitude Ranking by amplitude results in significant overlap between all classes Overlap is reduced considerably when ranking by size of object Histogram based on amplitude Histogram based on size Magnetics 31 The bar-chart at left provides histograms of the amplitude response (difference between positive and negative lobes of the measure anomaly) from (1) Shrapnel and cultural debris (or junk) (2) Base-plates (3) Partial rounds; and (4) Intact 4.2 inch mortars Most of the shrapnel and debris have low anomaly amplitudes. The 4.2 mortars tend to have higher amplitudes, but there is a considerable range in values. To recover all 4.2 mortars would require digging up almost all of the clutter items. The bar-chart at right shows histograms of dipole moments obtained through parameter extraction. The smaller items (shrapnel, base-plates) tend to have small moments and can largely be distinguished from the larger 4.2 mortars which have larger estimated moments. Many clutter items could be left if the ground if the digsheet were prioritized based on the size of the moment. 41

67 Summary 1 Ferrous ordnance and non ordnance distort the Earth s magnetic field Cesium vapor total field magnetometers are used extensively in ordnance detection applications Magnetic anomalies depend on the size, shape, orientation and depth of the buried object Survey parameters such as sensor height, sensor noise levels, position errors and lane spacing effect the quality of the collected magnetic data Magnetics 32 Summary 2 Parameter extraction routines are used to estimate the attributes (size, orientation, depth) of a detected buried object The extracted parameters are used to create a prioritized dig list Magnetic data are largely immune to sensor orientation, can be rapidly collected and are highly sensitive to the depth of the buried item Magnetic data can be adversely affected by geology, only return an approximate estimate of the object s size and can t be used to (uniquely) determine the object s shape. Magnetics 33 42

68 This Page Has Been Intentionally Left Blank

69 Electromagnetics (EM): Fundamentals and Parameter Extraction Thomas Bell EM Module Outline EM Fundamentals Basic principles of electromagnetic induction Sensors, EM signals & noise Classification Estimating target attributes (features) from EM data Feature based classification Data quality requirements & current technology limitations EMI 2 1

70 EM Fundamentals EMI 3 This sequence of slides shows the fundamental concepts involved in EM measurements, starting with a picture of a typical EM sensor being pulled across a field. 2

71 EM Fundamentals EMI 3a The basic elements of an EM sensor are a transmit coil and a receive coil shown by the loops above the ground surface. A current pulse running through the transmit coil creates the primary EM field, illustrated by the arrows flowing along field lines shown in red. This pulse excites the munitions item under the sensor. 3

72 EM Fundamentals EMI 3b Changes in the primary field set up eddy currents in the object, shown schematically by the green arrows seeming to flow around the buried munitions item. 4

73 EM Fundamentals EMI 3c The eddy currents produce a secondary or induced EM field emanating from the object. This field can be represented by an induced dipole at the object's location. The strength and orientation of the dipole moment are determined by the primary field at the object and physical properties of the object such as its size and shape, as well as its orientation. 5

74 EM Fundamentals EMI 3d The induced field is measured by the receive coil, the output signal being proportional to the rate of change of the EM flux through the receive coil. 6

75 EM Fundamentals EMI 3e There are two basic types of EM sensor: continuous wave (frequency domain), and pulsed wave (time domain). Frequency domain sensors transmit a continuous waveform, while time domain sensors transmit a sequence of EM pulses. The pulsed sensor is the most commonly used configuration because it allows the eddy current response to be measured when the primary field is not changing and is no longer overwhelming the signal due to the induced field. The two plots show typical transmit and receive waveforms for a pulsed EM sensor and identify the three stages of the EM measurement process. (1) The object is magnetized only during the transmit pulse. (2) The eddy currents are excited in the target when the pulse abruptly ends. (3) The EM response is measured during the eddy current decay after the primary field pulse ends. This measured decay contains the information that is used to classify the target. 7

76 Basic EM Concepts 1. The primary field magnetizes the buried object (similar to magnetics) 2. Abrupt change in the primary field excites eddy currents in the object. 3. Eddy currents diffuse throughout the object and decay (basic EM response which applies to all metal objects) EMI 4 Review. A typical EM sensor measures the EM field associated with the decay of eddy currents in metal objects near the sensor. The eddy current decay occurs after the current pulse in the transmitter loop is completed, and hence after the response directly caused by any magnetization of material near the sensor. Unlike magnetometers, EM sensors respond to all types of metal objects, not just ferromagnetic ones. 8

77 EM Survey EMI 5 This sequence shows the measured EM response as a sensor travels back and forth along survey lines over an object. The black dots correspond to the measurement positions and the red points represent the measured response at those points. The measured signal corresponds to the induced field at a fixed time during the eddy current decay cycle. 9

78 EM Survey EMI 5a Data are collected along a sequence of survey lines over the object. 10

79 EM Survey EMI 5b The data are mapped to a grid and displayed using a color scheme where the response to the object shows up as an isolated feature or "anomaly" above the background level. 11

80 EM vs. Magnetics EM Magnetics EMI EM eddy current response is less sensitive to interference from geology than magnetic response Time decay of eddy current response provides classification information not available with magnetic sensors 6 This slide shows color maps of data from EM and magnetic surveys over the same area. Anomalies due to buried objects show up as red or purple regions in the EM data map, and paired red and blue regions in the magnetic data map. The magnetic data map shows much more background structure than does the EM data map. Because they measure the eddy current response after the magnetization phase, EM sensors are less sensitive to interference from geologic structures than magnetic sensors. The time decay of the eddy current response provides classification information that is not available with magnetic sensors. 12

81 EM Classification Time decay of eddy current response Determined by physical properties of object. Varies differently with sensor/object geometry for different objects. Early time eddy currents at surface, response set by objectʹs size and shape. Late time eddy currents diffused through object, response set by thickness of material. EMI 7 EM classification is based on properties of the eddy current decay which are determined by physical properties of the object being measured with the EM sensor. The graph on the right is a blowup of the eddy current decay illustrated previously. Early in the response the eddy currents are confined near the surface of the object, and the characteristics of the measured signal reflect the size and shape of the object. As time progresses, the eddy currents diffuse into the object until at late times the response is determined primarily by the thickness of the material that the object is made of. A key factor exploited in the classification process is that the induced field due to the eddy current decay varies in a systematic way as the sensor/object geometry is changed. 13

82 Processing Stream Stages in the EM classification process: 1. Data Collection 3. Classification 2. Signal Attribute Extraction EMI 8 There are three stages in the classification process. They are illustrated schematically in this slide. The first stage is data collection over an object. The picture shows data being collected with a Geonics EM61 metal detector, which is a widely available EM sensor. In the second stage, attributes or features of the EM response that relate to physical properties of the object are extracted from the data by fitting to a physics-based model. Finally, the object is classified by deciding whether the set of attributes is more like those typical of munitions items or those typical of clutter items. 14

83 Commercial EM Sensors EMI 10 There is a wide variety of commercially available EM sensors. A few are illustrated on this slide. Geonics EM61 metal detectors (shown in the upper two pictures) are the most commonly used EM sensors in UXO work. The lower set of pictures shows, from left to right, a Geophex frequency domain sensor, the Geonics EM63 sensor (which measures a larger portion on the eddy current decay cycle than the EM61), and a Minelab sensor which has less sensitivity to magnetic soils than most other EM sensors. 15

84 Raw EM Data The Geonics EM61 metal detector is the most commonly used EM sensor for UXO detection Signal strength depends on size, shape, orientation and depth of buried object to be detected EM61 output (channel 2 of 4) as sensor is wheeled over a 3 Stokes mortar buried 80 cm below ground level EMI 11 This slide shows the EM61 sensor and a sample of EM61 data collected as the sensor was pulled over a buried munitions item. The EM61 has four signal channels. The first three sample the eddy current decay measured by the lower rectangular coil at three different times. The fourth channel can sample either the upper coil response in the third time gate or the lower coil response at a fourth time. The peak signal strength in any channel depends in a predictable way on the size, shape and depth of the buried object. 16

85 Signal Strength vs. Depth Signal strength decays as sixth power of depth (vs. third power for magnetics) Peak signal is strongest for UXO if the item is aligned vertically, weakest if it is lying flat EMI 12 The graph on this slide is a typical plot showing measurements of an object's signal strength (peak signal over the object as shown by the blue bar on the EM61 signal trace reproduced from the previous slide) as a function of the depth of the object. The EM signal decays more rapidly with depth than the corresponding magnetic signal (sixth power vs. third power, respectively). The dashed lines on the graph show the bounding curves for maximum and minimum signal strengths corresponding to vertical and horizontal target orientations. The strength of the signal due to an object depends not only on the object's depth, but also on the orientation of the object. This is reflected in the scatter of the measurements at a fixed depth. For horizontal coplanar coils (like the EM61) the peak signal is strongest when the object is oriented with its long axis vertical, and weakest when it is horizontal. 17

86 Detection Limits Detectability limited by sensor and background noise EM61 (gate 3) rms noise = 0.25 mv 75 mm projectile, 35 cm deep (30 dip) 37 mm projectile, 35 cm deep (30 dip) 20 mm projectile, 35 cm deep (30 dip) EMI 13 Whether or not a given object will be detected depends not only on how strong a signal it creates, but also on the level of noise in the measurements. This slide shows profiles of noise measured by an EM61 with added signals that would be created by three different munitions types. The root mean square (rms) noise level is 0.25 mv, which is relatively benign. The top plot includes the signal due to a 75 mm projectile buried 35 cm below the ground surface, tilted down 30º from horizontal. Signal plus noise is shown in red, with the noise-only profile overlaid in black. The signal completely dominates the plot. The middle plot is for a 37 mm projectile at the same depth and orientation. Here the signal is about ten times weaker (note the change in scale on the vertical axis), but still much stronger than the noise. At this scale the noise fluctuations are clearly visible. The bottom plot is for a 20 mm projectile at the same depth and orientation. The signal is weaker still (the vertical scale is further magnified in this plot), and is now obscured by the noise fluctuations. 18

87 Signal to Noise Ratio (SNR) Require SNR (S max / rms noise) > 5 6 for reliable detection EM61 (gate 3) rms noise = 0.25 mv S max = 38.3 mv, SNR = 153 S max = 4.5 mv, SNR = 18 S max = 0.75 mv, SNR = 3 EMI 14 The signal-to-noise ratio is a simple measure of the relative strengths of the signal and the noise. It is equal to the ratio of the maximum signal strength to the standard deviation of the noise (root mean square or rms noise). This slide repeats the profiles from the previous slide and shows the corresponding SNR values for the different targets. For the 75 mm projectile, the peak signal is 38.3 mv. The rms noise is 0.25 mv, so the SNR is 153. The object is clearly detectable. For the 37 mm projectile, the signal is 4.5 mv and the SNR is 18. It is still readily detectable. The 20 mm projectile is not detectable in this noise. Its peak signal is 0.75 mv and its SNR is 3. Typically, we will need a signal to noise ratio greater than 5 or 6 in order for the object to be reliably detected. 19

88 Noise Sources Electronic noise in sensor Ambient electromagnetic noise (natural or man made) Sensor bouncing motion relative to ground & geomagnetic field Geologic structure Metallic debris in ground Operator related noise Improper/careless operation of sensor (loose cables & connections, metal stuck in wheels, etc.) Metal carried by field personnel Processing Artifacts EMI 15 Any fluctuations in the sensor output when there is no target along the path of the sensor represent noise. There are a variety of different sources of noise. The noise can be inherently time-dependent, as in the case of noise from the sensor electronics or atmospheric noise associated with lightning, nearby transmission lines, etc. Or it can arise from the temporal evolution of space/time processes. Noise is introduced when the sensor moves up and down near the ground so that the measured EM soil response fluctuates as the sensor moves along. Similarly, irregular movement of the sensor can introduce noise due to changing flux of the earth's magnetic field through the receive coil. Spatial variations in the geologic response can introduce additional noise as the sensor is moved over the ground. Generally, the largest noise contributions come from metallic debris in the ground, operator-related noise and data artifacts that can be inadvertently introduced during processing. Operator-related noise can result from improper or careless operation of the sensor (loose cables and connections, small metal pieces stuck in the wheels, metal carried by the operator, etc.) 20

89 Comparative Noise Levels ESTCP noise study at Blossom Point, MD test site Effects of platform motion (controlled tests, relatively benign site) EMI Channel 4 = upper coil, gate 3 16 This slide shows rms EM61 noise levels measured with three different sensor configurations: a wheeled EM61 pushed by the operator, a trailer-mounted EM61 towed behind a vehicle, and a stationary EM61. Each plot shows noise levels for the four EM61 channels measured at different locations at the test site. The moving platforms (left and middle) show significantly higher noise levels than that recorded while the sensor is stationary, and the vehicle-towed sensor shows lower noise levels than the operator-pushed sensor. Recall that the noise profiles used in the previous SNR illustrations were for channel 3 and had an rms level of 0.25 mv. This is comparable to the vehicle-towed noise for channel 3. 21

90 Target Attributes 1. Data Collection 3. Classification 2. Signal Attribute Extraction EMI 17 Once the EM data have been collected, anomalies possibly due to buried objects are identified and the corresponding data are processed to extract target attributes or features that can be associated with physical attributes of the object. Generally speaking, target attributes are the basic building blocks of the object's EM response. They depend only on what the object is, not where it is or how it is oriented. The blue boxes in the plot in the middle of the slide show the four EM61 measurement time windows, superimposed on the full decay curve for a test object. For singlechannel EM61 data, the basic target attributes are a set of three parameters that can be used to calculate the EM61 response in that channel at any location above the object. 22

91 Processing EM Data Dipole response model is used to interpret EM data collected over an object Induced dipole response (M) to primary field (H 0 ) is expressed as vector sum of responses in the object s three principal axis directions. Principal axis responses are dipole moment components (M i ) EMI 18 EM data collected over an object are interpreted using a dipole response model similar to the dipole moment representation used with magnetometer data. The figure on the left at the bottom of this slide shows a munitions item with the primary field lines superimposed in red. The eddy current decay response to the primary field can be represented by an induced, time-dependent dipole at the center of the target. The dipole moment is expressed as a vector sum of component responses along each of the object's three principal axes, as shown schematically at the bottom right with the induced field lines superimposed in blue. For the symmetric object shown, the two smaller responses are equal and are designated as M 2. The measured signal is the sum of the principal axis responses multiplied by the corresponding primary field components. The principal axis response functions are sometimes called principal axis polarizabilities, and represent how the eddy currents decay when the object is excited in each of the principal directions. 23

92 The set of three principal axis responses (polarizabilities) constitutes the basic EM signature of an object. EM Signature Typical UXO items are symmetric, so two of the principal axis responses are the same. Irregular clutter items typically have three different response functions. EMI 105 mm projectile tractor muffler 19 The set of three principal axis responses constitutes the basic EM signature of an object. This is shown in the two figures at the bottom of the slide. Each figure shows a picture of an object (a 105 mm projectile on the left and a comparably sized muffler on the right), a set of arrows representing the three principal axes of the objects, and a set of response functions color coded by principal axis. All three principal axis responses are shown in each plot, with the one corresponding to the principal axis arrow colored accordingly and the others shown in gray. Two of the principal axis responses for the munitions item are the same because of the object's axial symmetry. All three of the principal axis responses are different for the muffler. This is expected for irregularly shaped clutter items. 24

93 Sampling the Full EM Response EMI 20 In order to be able to determine the principal axis responses, the object must be excited and measured over a range of directions. This sequence illustrates how data collected at different locations above an object can accomplish this. In the first slide the object (e.g. the 81 mm mortar shown in the picture) is directly under the sensor, and we measure its response to a vertical field. This is shown by the blue curve in the graph at the right. 25

94 Sampling the Full EM Response EMI 20a When the sensor is moved off to the side, the primary field at the object changes direction and we measure a different response. Now, the original response from above the object is the gray curve, while the new response from off to the side is shown by the new blue curve. 26

95 Sampling the Full EM Response EMI 20b If we take measurements from enough different locations, we can generate a complete set of different decay curves that can then be used to determine the principal axis response functions. 27

96 Sensor/Object Geometry Classification exploits eddy current decays observed from a variety of excitation directions EMI An object directly underneath the sensor is excited with a vertical primary field If the sensor moves off to the side then the object is excited by a horizontal primary field 21 Summarizing, classification exploits eddy currents observed from a variety of excitation directions. An object directly underneath the sensor is excited with a vertical primary field, as shown in the diagram on the left. If the sensor moves off to the side, then the object is excited with a horizontal primary field, as shown in the diagram on the right. By measuring the response from enough different locations we can collect the data needed to calculate the principal axis response functions. 28

97 Inversion of EM Data The dipole response model is used to extract the principal axis polarizabilities by inverting EM data collected over a buried object. The EM response must be measured from enough different directions to adequately sample the principal axis responses (e.g. with accurately mapped EM survey data) The object s location, depth & orientation are determined as part of inversion { V(t) = μ IC C B(t) } 1 0 R Inverse operation T EMI Data from different sensor/object geometries Principal axis polarizabilities 22 The dipole response model is used to extract the principal axis response functions by inverting EM data collected over a buried object. Inverting the data simply means using the dipole response model to work backwards from the measured data to the model parameters (the object's location and depth, the orientations of its principal axes, and its principal axis response functions) needed to reproduce the data. The process is illustrated schematically in the flow diagram on this slide. The sheaf of graphs on the left represents the set of induced field decay curves measured at various locations over the object. The equation in the middle represents the inversion process that operates on the data collected over the target. The graph on the right shows the principal axis response functions determined by the inversion process. To begin with, we have only the data collected over the target and the corresponding locations where the measurements were taken. In order for the inversion to produce accurate results, the locations where the measurements were taken must be known very accurately. The location and depth of the object are unknown, as are the orientations of the object's principal axes and the principal axis response functions. These are all determined by inverting the data. The inversion proceeds by systematically varying possible values for the object's location, depth, etc. and using the dipole response model to calculate the expected responses at the measurement locations. As the parameters (location, depth, orientation, and principal axis responses) are systematically varied, the inversion compares the set of calculated responses with the measured responses until the best match is found. 29

98 Classification 1. Data Collection 3. Classification 2. Signal Attribute Extraction EMI 23 The final stage is classifying the object as a munition or clutter on the basis of target attributes that have been extracted from the EM data. Depending on the particular EM sensor that is used to collect the data and on the detailed workings of the classifier, the target attributes could be the complete set of principal axis response functions, principal axis responses averaged over some time window, a set of parameter values that describe significant aspects of the basic EM response, etc. The figure used to illustrate classification is a scatter plot of single-channel EM61 principal axis response values for various munitions and clutter items. 30

99 Conventional EM Technology EM61 measures eddy current decay averaged over four time windows or gates after primary field cutoff Limited decay curve classification capability Can provide some size/shape/depth information EM61 EMI 24 The EM61 is widely used in munitions detection work. The picture on the right shows an EM61 in the field. The blue boxes in the plot in the middle of the slide show the four EM61 measurement time windows, superimposed on a blowup of the decay curve shown in the plot at the left. Because the EM61 only measures EM response averaged over four time windows or time gates, it has diminished decay curve classification capability. However, it can still provide size and shape information useful for classification. 31

100 Size/Shape Classification Single EM61 channel inversion The object s features are related to the polarizabilities (βs) Size correlates with the net polarizability (sum of βs) Target shape is reflected in the relative sizes of the three βs UXO should have one large (β 1 ) and two smaller, equal (β 2,3 ) polarizabilities clutter often has three distinct βs EMI 25 This slide shows the basic idea of classification using single-channel EM61 data. The data were collected using the handheld version of the EM61 (EM61-HH). Each symbol in the plot on the right corresponds to first time gate EM61 data collected over an object that have been inverted using the dipole response model to determine the object's three principal axis response coefficients. These coefficients are the polarizabilities averaged over the first time window measured by the EM61. They are sometimes referred to as "betas". The sum of the three betas (net polarizability) is a measure of the object's size, and is roughly proportional to the volume of the object. The shape of the object determines the relative magnitudes of the three betas. Munitions items will generally have one large and two smaller, equal betas, while irregular clutter items can have three different betas. The plot on the right shows betas calculated for various munitions and clutter items. The munitions are plotted in blue and the clutter in red. For each set of betas, the dominant (largest) polarizability is plotted along the horizontal axis, and the two secondary polarizabilities are plotted as a vertical line running upwards from the smallest beta to the middle one. There are six sets of betas calculated from different data sets for each munitions item. They all overlay nicely, and all show the property that the secondary (smaller) betas are roughly equal. The magnitude of the polarizaility correlates with munition size, which ranges from 20 mm caliber to 155 mm caliber. The clutter items (range scrap from Aberdeen Proving Ground) scatter throughout the munitions items and show varying degrees of shape irregularity as evidenced by the various lengths of the secondary beta lines. 32

101 Processing EM Data EM61 EMI 26 With support from ESTCP, procedures for extracting target attributes from EM61 data have been implemented in the Geosoft Oasis montaj geophysical data mapping and analysis software suite. This sequence shows the processing flow. First, EM61 survey data are gridded and mapped, as shown in the first slide. 33

102 Processing EM Data EMI 26a The operator interactively selects an anomaly from the survey map for analysis. This is shown here by the box around one of the anomalies. 34

103 Processing EM Data EMI 26b The software performs a model match to the anomaly data and displays the dipole fit alongside the measured anomaly data. In this case, the dipole model does a good job reproducing the data. The target parameters (location, depth, principal axis orientations and polarizabilities) are calculated, along with the fit coherence, which is a measure of how well the model reproduces the data. The fit coherence is equal to the square of the correlation coefficient between model fit and data. 35

104 Anomaly Report Sheet Gridded Data Model Fit Single channel EM61 inversion Badlands Bombing Range, Cuny Table, Target 3 Target Attributes Profiles thru center with fit EMI 27 An anomaly report sheet is prepared for each anomaly that is analyzed. An example from the Badlands Bombing Range is shown on this slide. The report sheet shows side-by-side maps of the gridded data with the anomaly highlighted and the model fit to the anomaly. The locations of the actual measurements are shown as black dots on the anomaly map. Profiles of the four EM61 data channels along the survey line passing closest to the center of the anomaly are plotted at the bottom of the page. The fit results (target parameters and fit quality) are also listed on the sheet. 36

105 Survey Data Quality Good survey data Good survey quality control is important for classification. Poor survey data EMI 28 Good survey quality control is essential for reliable classification. This slide compares two different EM61 surveys over the same object. The anomaly map and profiles through the center of the anomaly are shown for each. The survey data illustrated on the top line is noticeably better than that shown on the bottom. In the lower survey, the anomaly shape is distorted, and the profiles are clearly skewed. The dipole inversion relies on subtle features of the anomaly shape, and can end up converging on an erroneous set of target attributes that reproduce the distorted data. 37

106 Data Requirements Inversion of EM data requires accurately mapped survey data or data collected using a grid template to control positioning Also requires measurements of sensor orientation and vertical position Data density and spatial extent must adequately sample the principal axis polarizabilities No overlapping signals from nearby objects SNR needed for classification > SNR needed to detect object EMI 29 Because the dipole inversion seeks to faithfully reproduce the shape of the EM anomaly, it is very sensitive to data distortions. It requires accurately mapped survey data or data collected using a grid template to control positioning of the sensor. The orientation and vertical position of the sensor have to be measured or constrained. There cannot be any interference due to signal overlap from nearby objects. Significantly, the signal-to-noise ratio (SNR) required for classification is much higher than that needed to detect an object. Finally, the data density along survey lines, the line spacing, and the spatial extent of the data patch selected for inversion must adequately sample the principal axis polarizabilities. 38

107 Model Match Fit Quality Fit quality is determined by the mismatch between the data and dipole model fit to the data Reflects ability of data quality to support inversion and estimation of target attributes for classification Well resolved anomaly with good dipole fit quality (5% fit error) EM61 (gate 1) data over 2.75 rocket warhead at Badlands Bombing Range EMI 30 The dipole fit quality is determined by the mismatch between the data and the dipole model fit to the data. It reflects on whether or not the data quality is adequate to support reliable inversion and estimation of target attributes for use in target classification. The mismatch (fit error) is related to the fit coherence introduced a couple of slides back. The fit coherence is equal to one minus the square of the fit error. The plots on this slide show an anomaly due to a 2.75 inch rocket warhead and the dipole fit to the anomaly. This is very good data. The anomaly is well resolved. Several survey lines cross it, and there are a good number of data samples over the anomaly along each line. The dipole fit quality is good, with only a 5% fit error (fit coherence = 0.998). 39

108 Parameter Extraction Issues Line spacing does not provide adequate sampling of polarizabilities Overlapping signals Weak signal (low SNR) EMI 31 This slide shows some examples of poorer data. The survey line spacing does not provide adequate sampling of the principal axis responses for the anomaly on the left. Only two lines cross the anomaly. The anomaly show in the center is actually a couple of overlapping anomalies. These cannot be sorted out using currently available processing techniques. The anomaly on the right has a signal-to-noise ratio that is too low to support reliable inversion. 40

109 SNR Requirements Reliable estimation of target attributes (polarizabilities) requires very high quality data with dipole fit error less than 5 10% SNR approaching 100 is required for classification, compared to ~5 for detection EMI 32 The plot on this slide shows the results of dipole inversions on channel one data from a relatively high quality EM61 survey on a test field. The dipole fit error for each of the inverted anomalies is plotted as a function of the signal-to-noise ratio (SNR) of the anomaly. In general, the fit error decreases with increasing SNR. Experience has shown that reliable estimation of target attributes (first time gate principal axis response coefficients) requires very high quality data with dipole fit errors less than 5-10%. The plot shows that dipole fits errors in this range are generally not achieved if the SNR is less than about 100. This is much, much larger than the SNR of about 5 or 6 needed to reliably detect the object. 41

110 Positioning Errors Errors in recorded sensor locations corresponding to EM data can substantially increase dipole fit error It is very hard to maintain survey geolocation at the accuracy level required to support reliable classification EMI 33 Errors in the recorded sensor locations corresponding to EM measurements act like noise in the data, and can have a significant impact on the dipole fit error. The basic fit error vs. SNR plot on this slide is reproduced from the previous slide. The solid line shows the relationship between dipole fit error and SNR that would be expected if the recorded sensor positions were correct. The dashed lines show how the relationship is affected by increasing rms positioning errors. Once the errors in the recorded sensor locations get larger than a few cm, the dipole fit errors even for very high SNR targets will be large enough to compromise the accuracy of target attributes extracted from the data. 42

111 Inversion Failures For some objects, inversion of the EM data fails to even produce interpretable results (ʺcanʹt analyzeʺ) Generally due to low SNR, positioning errors and/or interference from signals due to nearby objects. Results from EM61 survey at Camp Sibert Classification Study EMI 34 For some objects, inversion of the EM data fails to even produce interpretable results. We refer to these cases as "can't analyze". Since we cannot classify them, anomalies that fall into the "can't analyze" category must be treated as indicating possible munitions items. Most anomalies that fall in this category have low SNR, significant positioning errors and/or interference from signals due to nearby objects. The plot is a histogram of the distribution of SNR values for EM61 anomalies analyzed for the ESTCP Camp Sibert Classification Study. The blue line shows the distribution for all of the anomalies, and the grayed area shows the distribution for the subset that were declared "can't analyze". It shows that a significant fraction of the weaker anomalies could not be properly inverted and analyzed. 43

112 Technology Improvements Classification performance of conventional technology (e.g. EM61) is limited by two primary factors The eddy current decay cycle is not fully captured Multi cm positioning errors inherent to field survey work compromise the accuracy of dipole inversion and estimation of target attributes New UXO specific technologies which avoid these problems are being developed and tested under SERDP and ESTCP EMI 35 Classification performance using data collected with conventional technology such as the EM61 is limited by two primary factors: (1) the eddy current decay cycle is not fully captured, and (2) cm-level positioning errors inherent to field survey work compromise the accuracy of dipole inversion and estimation of target attributes. SERDP and ESTCP are developing and testing new UXO-specific technologies which avoid these problems and should provide significantly improved discrimination performance. These emerging technologies will be described in a later section of this course. 44

113 This Page Has Been Intentionally Left Blank

114 Classification Dean Keiswetter Classification, as used here, is a process that results in a decision. The decision we are trying to make is whether or not the subject anomaly could possibly be caused by a target of interest. In this module, we will introduce basic concepts and approaches used during the classification process. We will briefly touch upon: 1. The attributes that form the basis of the decision, 2. Approaches for ranking the anomalies, 3. Methods for selecting decision thresholds, 4. The product of the classification process a prioritized dig list 5. Evaluating performance, and finally 6. A model for implementing the classification process. 1

115 Classification Introduction At this stage in the process, we have derived target attributes from the measured data that describe the source for each and every anomaly Our task is to use these attributes to identify those anomalies, if any, that cannot possibly be targets ofinterest (TOI) at our site In other words, we need a principled process that results in a decision is the source of the subject anomaly hazardous or not? Classification 2 In the previous modules, we have learned how to derive attributes from the measured field data that describe the source object. Here, we are interested in using these attributes to make a decision. Our goal is to identify those anomalies, if any, that cannot possibly be caused by targets of interest at the site in question. To the extent that we can uniquely identify the non-toi (or clutter), we can effect the remediation process and save money, time, and effort. Ideally, the decision process should be transparent, based on quantitative attributes, and principled. 2

116 Classification Classification Example Remember the goal: identify anomalies that are NOT TOI Assume attributes for a site with a single munitions item General Process: 1) Visualize attributes 2) Obtain labels (e.g., ground truth information) 3) Establish boundaries this is the classification piece It can be this easy if the features are separable Attribute UXO TOI Half Shell Munition Debris Cultural Attribute 1 TOI space Non TOI space 3 Let s use a simple example to illustrate the process in very general terms. Here we assume that there is only a single target of interest. Geophysical data have been acquired and inverted and we have derived two attributes. It doesn t matter at this time if we started with magnetometry or EMI data. All that matters is that we have used phenomenological models to derive target attributes. The general process is to (1) first visualize the attributes, (2) obtain labels (viz., ground truth descriptions regarding the physical nature of each source item), and (3) establish boundaries, in parameter space, that segment the TOI from the non- TOI. The third step in the generalized process mentioned above is the classification piece and is the focus of this module. Before diving into details, let s review our objective in terms of what the classification process will produce. 3

117 Classification Product Prioritized Dig List Rank based on the probability that the anomaly is non TOI Divided into 4 categories - High confidence non TOI - Can t make a decision - High confidence TOI - Can t analyze Non-TOI TOI Prioritized Dig List Classification 4 This slide presents a prioritized dig list. This is the product that we are after. It ranks anomalies based upon the probability that the source is a non-toi and explicitly incorporates uncertainties. It does this by having 4 categories. There is one category for non-toi, one for TOI, one for Can t Decide, and one for Can t Analyze. As a geophysical services community, this is where our job ends. Our task is to provide as much information to the stakeholders as possible regarding each and every anomaly. This includes attributes describing the source object and classification results based on information available at the time. The clients and stakeholders, in turn, decide how to act upon the information. 4

118 How do we classify? Visually, we use physical attributes Size Symmetry Shape Classification 5 Okay let s begin by discussing how we might perform the classification process if we could visually see the objects. This is something we can all relate to. We might use size if appropriate. We might use shape if appropriate. We might even use axial symmetry to group the objects. 5

119 How do we classify? Unfortunately, we cannot visually inspect buried objects We have to utilize attributes derived from geophysical data: Size, Symmetry, Shape Decay Rate EMI & Mag Early Time Eddy currents at surface Response reflects target size and shape 60mm Polarizability Late Time Eddy currents diffused thru target Decay rates determined by thickness of target material Classification Time (ms) 6 Unfortunately, we cannot visually inspect buried objects. We have to use other attributes -- attributes that can be derived from the available data. Earlier, we learned that both magnetic and EMI data can be inverted to obtain model parameters, or attributes as used here, that are intrinsic to the target. Intrinsic means that the attributes depend on the physical nature of the object itself, not on its surrounding (like location or orientation). The principle axis polarization data shown in this example illustrate the information available to us for making the decisions. There are three polarizations, one along the longitudinal axis and two for the transverse axes. They are functions of time and provide information regarding the target s size and shape, as well as wall thickness. Attributes appropriate for classification can be derived from either magnetic or EMI data. 6

120 Which Attributes are Important? Answer: any attribute or set of attributes that are stable and provide separation between TOI and non TOI. They should be intrinsic to the target (not orientation or location) Intrinsic attributes include: Size - Magnetic Moment - Principal axis polarizations (EMI) Symmetry Shape Decay rate Two basic approaches to making a decision based on the attributes Rule based Statistical Classifier Classification 7 Which attributes are important? Answer any or all of them, as long as they are stable and can be used to differentiate TOI from at least some of the non-toi. The attributes should relate to the target itself. Orientation with respect to the earth s field or spatial location, for example, are not good indicators of munitions. The intrinsic attributes that we have access to include Size of the target Symmetry of the target Shape of the target, and Decay rate. As discussed earlier, target size attributes can be derived from either magnetic or EMI data. Target attributes relating to axial symmetry, shape, and EMI decay rate can be derived from EMI data only. Next, let s discuss two fundamentally different approaches to making the decision based on the selected attributes. 7

121 Rule-based Decisions Rule based classifiers base decisions on formal rules. Uses classical IF (condition) THEN (consequence) logic. For example: IF the size based attribute > 1, THEN TOI. IF the size based attribute > 1 AND the decay based attribute is > 100, THEN TOI. Can be used to bound decision. For example: IF (regardless of other attributes) the depth > 1m, THEN non TOI (viz., not interested in deeply buried objects) IF (regardless of other attributes) the size is less than 0.02, THEN non TOI (viz., not interested in small objects). Rules can be combined to form decision trees. Classification 8 One approach for making the classification decision is to use a Rule-based method. This method uses classical IF (condition) THEN (consequence) logic. Rules can be used to identify TOI directly. Rules can be used to identify certain non-toi directly. Rules can be used to bound decisions. Rules can be combined to form decision trees. Rule-based decisions are easy to comprehend when a limited number of attributes are used. 8

122 Rule-based Decisions Use simple relationships and rules to make a decision Remember the Goal: Identify anomalies that are NOT TOI { IF (condition) THEN (consequence)} Decay based Attribute Classification Target Size based Attribute 9 Rule-based decisions use simple relationships and rules to make a decision. Here we have some example attributes. The scatter plot shows an attribute related to target-size on the x-axis and an attribute related to the target s decay rate on the y-axis. Colored symbols are used to show the labels or ground truth information. In this case, the red plus signs are used for UXO. The remaining colors and symbols identify various types of non-toi. Our objective is to develop rules that allow us to segment the feature space such that each and every anomaly is ranked as either a TOI or not and provide some level of confidence or measure of certainty. 9

123 Setting the Threshold for a Rule-based Decision Remember the Goal: Identify anomalies that are NOT TOI Decision Thresholds Based on both Attributes (conservative) Decay based Attribute Classification Target Size based Attribute Decision Thresholds Based on both Attributes (aggressive) Decision Thresholds Based on Decay based Attribute Decision Thresholds Based on Size based Attribute 10 One approach might be to base the decision solely on the target-size-based attribute. As you can see, given this distribution of attributes, this would result in a good but perhaps not optimal result. The blue line separates most of the TOI from the majority of the non-toi. As shown here, however, a number of the larger fragments [namely the Partials (rounds that have been split open but not fragmented)] would be mis-classified. Another approach might be to base the decision solely on the decay-based attribute (green line). As before, this would have some classification success but is not optimal If we combine the two rules, we can achieve a much better result. Here (small square), the decision space is aggressively chosen to include all TOI. There isn t much play allowed for unanticipated variability. Alternatively, (large square) we could select a larger section of the feature space. This later approach provides significantly more buffer to handle TOI variances that are not represented in the training data. 10

124 Statistical Classifiers Statistical classifiers are computer algorithms that make use of one or more attributes to make a quantitative decision. They statistically characterize attributes and create group associations use training data to attach labels (viz., ground truth information) to the groups provide explicit probabilities accommodate many attributes and data dimensions Classification 11 Statistical classifiers are computer algorithms that consider multiple variables to make a quantitative assessment of the likelihood that a signal corresponds to a target of interest. They statistically characterize the attributes to create group associations. Unlike the rule-based scenario, statistical classifiers partition the feature space automatically based on the nature of the attributes and the training data provided. They do not make yes or no decisions per se. Uncertainties and measures of confidence are inherent to the decision process. In fact, they provide explicit probabilities. Probabilities range from 0 to 1. Simply put, the result from a statistical classifier may be phrased as the probability that this anomaly is caused by a TOI is less than 0.1 Additionally, due to the digital nature of the process, statistical classifiers easily accommodate numerous attributes and associated data dimensions. If presented with many attributes, they can be used to automatically select which attributes provide optimum results. 11

125 Statistical Classifiers Classifier performance depends greatly on the characteristics of the data to be classified There is no single classifier that works best on all given problems There are many statistical classification schemes a few common classifiers include: Support Vector Machines k Nearest Neighbors Generalized Likelihood Ratio Test (a Bayesian approach) Probabilistic Neural Networks Classification 12 Having made the statements on the previous slide, a little caution is warranted Domain knowledge is certainly required. Without a fair amount of knowledge regarding the specific classifier being used, it is easy to misinterpret and/or overstate the results. 12

126 Statistical Classifiers Remember the Goal: Identify anomalies that are NOT TOI Use computer algorithms to make a decision Numerically classified based on quantitative attributes and on training data 2 Obtain labels Submit data/labels to classifier & return probabilities Classification TOI UXO Half Shell Munition Debris Cultural Decay-based Attribute Size-based Attribute 13 Statistical Classifiers are computer algorithms that statistically characterize the attributes and create group associations. Again, our objective is to segment the feature space such that each and every anomaly is ranked as either a TOI or not and provide some level of confidence or measure of certainty. Consider these example data. As before, the scatterplot shows an attribute related to target size on the x-axis and an attribute related to the target s decay rate on the y-axis. Colored symbols are used to show the labels or ground truth information. In this case, black diamonds are used for a single TOI. The remaining colors and symbols identify various types of non-toi. The next step is submit the attributes and associated labels to the classifier for training. After training, the classifier returns a probability for each anomaly. These are plotted according to the grey scale on the right. The shaded contours identify areas in which the probabilities change rapidly. In this simple case with two attributes and a single item of interest, it is easy to see that the results appear reasonable. 13

127 Setting the Threshold for a Statistical Classifier Remember the Goal: Identify anomalies that are NOT TOI Classifiers return probabilities 2 Probability Non TOI > 0.9 Principle Threshold: 1. Boundary between high confidence non TOI and everything else 2. Chosen to exclude all TOI 3. Adjusted to account for observed variability Decay-based Attribute 1 0 Probability Non TOI < 0.1 TOI UXO Half Shell Munition Debris Cultural Size-based Attribute Classification 14 Given the classifier results, lets move on to setting thresholds. Instead of setting the threshold(s) at a specific attribute level as was done in the rule-based approach we can use the probabilities provided by the statistical classifier. The primary threshold that we are after is the boundary between high-confidence non-toi and everything else. As before, we start by identifying a boundary that includes all of the TOI. The green line shown here identifies the region for which probability of being non- TOI is less than 0.1. The anomalies within the area bounded by this line are unlikely to be to clutter. As you can see, all of the UXO or the TOI are within this boundary. The next step is to adjust our probability threshold to account for unexpected variability in the TOI distribution. The red line shows the boundary for non-toi probabilities of 0.9. Any anomaly outside this boundary is very likely to be non-toi. This is the primary threshold the boundary between high confidence non-toi and everything else - that was actually used. 14

128 High-confidence & TOI Threshold The non TOI threshold is chosen to exclude all labeled TOI. It is, therefore, driven by the observed variability within the TOI class Dependent on the data available for training It can and should be updated during excavations (and the unknowns reprioritized based on the new information). Unknown targets are classified and ranked based on their relationship to this threshold Classification 15 As shown in these two data examples, the non-toi threshold was chosen to exclude all TOI and account for potential variability within the TOI class. Because of this, the decision boundary is very dependent on the data available for training and the intra-class variance of the extracted attributes for the TOI. As the dig program proceeds, additional ground truth information will become available. This important information can and should be used to retrain, reclassify, and reprioritize the dig list. 15

129 Prioritized Dig List Rankings for Rule based classifier are based on the distance, in parameter space, from the TOI boundary Rankings for Statistical classifier are based on the probability of belonging to a the non TOI class Non-TOI TOI Prioritized Dig List Classification 16 Here we see the prioritized dig list again. This is, again, our ultimate product. For Rule-based methods, the anomalies are ranked based on the distance, in parameter space, from the observed TOI boundary. For Statistical classifiers, the anomalies are based on the probability of belonging to the non-toi class. Next, let s take a look at one approach for expressing the results. 16

130 Expressing the Results Receiver Operating Characteristic Curve Retrospective (only exist if all targets removed) TOI (normalized) Sensor B Sensor A Classification 100% TOI recovered ~200 non-toi removed Detection Only 100% TOI recovered ~600 non-toi removed Desired Performance 100% TOI recovered 0 non-toi removed Can t Analyze Classification non-toi 17 Here we have plotted what is known as a Receiver Operating Characteristics (ROC) curve. It is retrospective in that it requires all ground truth. These curves were taken from a study at Camp Sibert in which all contacts were dug for learning purposes. Basically, it is a plot comparing the number of actual TOI objects versus non-toi if our prioritized dig list were excavated in reverse order. The colors of the points plotted represent the classification categories used for the prioritized dig list. Of specific importance are those in Green, because this identifies anomalies that were classified high-confidence non-toi. Desired Performance Ideally, we would like perfect classification. If we could achieve this, the curve would rise straight to this location: 100% TOI recovered and 0% non-toi. Detection Only this is the other extreme. Here, we ignore classification altogether and simply excavate all detected targets. In the end, we have 100% of the TOI recovered (approximately 150 munitions), but we also have removed 100% of the non-toi which for this site was approximately 600 non-hazardous objects. Classification Here, we evaluate the classification results. This is the point at which the demonstrator drew the threshold between high-confidence non-toi and everything else. As shown here, 100% of the munitions were recovered but only 200 of the non-hazardous non-toi recovered. This is a huge success. Can t Analyze This portion of the curve is reserved for those anomalies that were thrown into the Can t Analyze category. Because no useful information regarding the nature of these targets can be extracted from the measured data, they must be treated as potential targets of interest. 17

131 Practical Model for the Classification Process Need prospective rather than retrospective approach Real world challenges include: How to proceed with the dig program? - dig all items not classified high confidence non TOI How to decide when to stop digging? - accept the analysts threshold and stop - increase confidence by Seeding the site Remediate 100% or a number of grids Excavate a percentage of the non TOI class Classification 18 The ROC is good for learning, but here we consider a practical model for the classification process. Questions commonly encountered in real-world settings include 1. How to proceed with the dig program 2. How do we decide to stop digging? When can we trust the classification? With regard to the first question the first step is to recognize that all of the targets not classified as high-confidence non-toi must be dug. This naturally leads to the second question when to stop? Here we have some options. The first option is to simply accept the analysts threshold and stop. There are, however, a few steps that can be easily implemented to increase confidence along the way. These include Seeding the site (recommended) Digging 100% of a number of the grids Digging a percentage of the non-toi class, especially those that cluster in feature space. 18

132 The Classification Process is not static The initial ranking was trained using labeled data available at the beginning of the project Each dig provides additional ground truth Classifier should be retrained on a regular basis As we gain information, it may be possible to redraw the decision boundaries and exclude more non TOI Classification Probability (TOI) Decay Attribute 1 0 Can t analyze Target Size Attribute Initial High confidence TOI Can t Decide High confidence Non-TOI 19 As discussed before, the classification process is not static. The initial rankings and decision thresholds were based on data and labels available at the time. Often, these can be limited in number and diversity. As ground truth information is revealed, the classification process should be rerun to take into account this new information. The cartoon on the right shows an attribute scatter plot on the top and the corresponding TOI-probability versus classification category on the bottom. 19

133 The Classification Process is not static The initial ranking was trained using labeled data available at the beginning of the project Each dig provides additional ground truth Classifier should be retrained on a regular basis As we gain information, it may be possible to redraw the decision boundaries and exclude more non TOI Classification Probability (TOI) 1 0 Decay Attribute Can t analyze High confidence TOI Target Size Attribute Revised Can t Decide High confidence Non-TOI 19a Perhaps with additional data and more ground truth, the classification categories could change as shown here. Basically, the new information has allowed the analyst to establish tighter thresholds and increase the number of high-confidence non-toi declarations. This is a cartoon, and actual results may trend in the opposite direction for a given site. In other words, the site may become more and more complex as more ground truth is revealed. The point remains, however, that ground truth information acquired over time throughout the digging phase should be incorporated into the classification process. 20

134 Summary Thoughts for Real World Sites The goal of classification is a decision Classification schemes allow us to identify anomalies that cannot be caused by the site specific TOI in a principled manner The classification process must be transparent Attributes should be: (i) stable, (ii) consistent, and (iii) show separation between TOI and non TOI classes The value depends on the items defined to be TOI and non TOI, the observed variance within the TOI classes, and the risk tolerance of the stakeholders Classification 20 21

135 This Page Has Been Intentionally Left Blank

136 Case Histories Dean Keiswetter In this module, we will present two successful classification projects. As we go through the projects, we will revisit many of the topics presented earlier. Specifically, we will discuss the 1. objectives what are we looking for and what can be left behind safely? 2. which data types and methods of collection were used, 3. which attributes were used as the decision basis, 4. how were the attributes used to make a decision, and 5. after digging all anomalies, how would we have performed if our classification scheme had been implemented. 1

137 Former Camp Sibert, Gadsden, Alabama Target of Interest: 4.2-inch mortars Non-Target of Interest: Native munitions debris Historical Use: Training center and airfield for chemical warfare Example of: Physics-based discrimination using magnetic and electromagnetic survey data Case Histories 2 The first case history concerns work conducted at Former Camp Sibert, located near Gadsden Alabama during This site was the first of a series of Large Scale Classification Demonstrations being conducted by ESTCP. There are two broad overarching objectives. They are to (1) assess performance of existing and emerging technologies, and (2) investigate the decision making process in cooperation with the regulatory community. The Large Scale Classification program is a multi-year effort involving multiple sites of varying complexity Camp Sibert was selected as the initial site because it has a single target of interest with benign topography and vegetation issues. It was used in the early 1940 s as a training center and airfield for the simulation of chemical air attacks against troops. The site was closed in 1945 and is no longer in active use by the military. The Target of Interest (TOI) at Camp Sibert was a 4.2in mortar. The non-toi items consist of native munitions debris and cultural clutter. 2

138 CAMP SIBERT - Objective Reject as much clutter as possible Without leaving any 4.2inch mortars unearthed. Target of Interest Non Targets of Interest Case Histories 3 Here we show photographs of the TOI and non-toi objects. The 4.2 Inch (107mm) mortar was a US rifled mortar used during the Second World War and the Korean War. It is 4.2 inches in diameter and roughly 1.3ft long. It is a large piece of steel. Examples of the non-toi at Camp Sibert are also shown here. Representatives of the large, medium and small munitions debris are shown along with some agricultural debris and miscellaneous nails. The objective was to reject as much non-toi as possible without misclassifying any 4.2inch mortars. Misclassifying a 4.2inch mortar would result in a false negative and represents a classification failure. 3

139 CAMP SIBERT - Data EM61 MK2 Sensor Data Case Histories 4 Multiple data sets were acquired at Camp Sibert, but for the purposes of this brief we primarily focus on the reconnaissance electromagnetic induction (EMI) and magnetometer data. On the right, we show EM61 MK2 data from Camp Sibert. This is a typical data view. It is a plan map where the colors indicated the magnitude of the EMI measurement at that location. The circles represent anomalies that were selected for analysis and classification by the ESTCP program office. As you can see, the background values are small and consistent across the site. These are clean data. The data were acquired by the Naval Research Laboratory s (NRL) Multi-sensor Towed Array Detection System (MTADS) EM61 MK2 array. The geophysical data are georeferenced using differential GPS. In addition, the attitude of the sensor is recorded using an auxiliary inertial measurement unit. Data were acquired along survey lines spaced 0.5m apart. Along the survey lines, data were collected every 20cm or so. 4

140 CAMP SIBERT - Data Magnetic Sensor Data Case Histories 4a Here, we see magnetic data. As discussed earlier, magnetic anomalies have positive and negative lobes. Again, the circles represent targets that were selected for analysis. The data were acquired using the MTADS magnetometer array, shown on the left. Magnetic data were collected along survey lines spaced 0.25m apart. Along the survey lines, data were collected every 15cm or so. 5

141 CAMP SIBERT Target Selection EM61 MK2 Anomaly selection set at 50% of the smallest expected signal amplitude for the 4.2inch mortar at maximum penetration depth (11x diameter or ~1.2m) All seeded targets were detected The number of total anomalies depends on the sensor and survey combination Case Histories Sensor Magnetometer Array EM61 MK2 Array EM61 MK2 Cart Peak Signal (mv) Anomaly Detection Threshold 6 nt 25 mv 19 mv, sum of three gates x depth GPO RMS Noise Anomalies on Master List Depth (m) most favorable orientation least favorable orientation test pit measurements GPO items threshold Seed Targets Detected Individual anomalies were selected by first determining the smallest expected signal amplitude for the 4.2inch mortar at is least favorable position (viz., maximum depth and horizontal). The target selection threshold was then set to be equal to 50% of this minimum target response. This was done for each sensor independently. First, note that all of the seeded items were detected by each sensor (light yellow). Note also that the number of anomalies is different for each sensor (light blue). Almost 970 anomalies were declared in the magnetometer data compared to only 633 in the EM61 Cart data a difference of over 330! All of the anomalies were excavated, carefully documented, and archived. 6

142 CAMP SIBERT Dig List & Scoring Anomaly list ranked from highest confidence not munitions to highest confidence munitions 100 TOI Non-TOI % mortars correctly identified Threshold high confidence munitions can't decide - munitions like can't decide - clutter like high confidence not munitions Case Histories number Number of false of non TOI positives Classification performance evaluated by comparing the number of false alarms versus the percentage mortars correctly identified 6 For the Sibert Demonstration, we used the Prioritized Dig List and Scoring protocols discussed in the Classification brief. Data analysts from multiple organization used inversion schemes presented earlier to characterize and classify each anomaly. The anomalies were ranked from high confidence non-toi to Can t make a Decision to high confidence TOI. If reliable attributes could not be extracted from the anomaly data due perhaps to sensor glitches or data gaps the anomalies were categorized as Can t Analyze. Classification performance is evaluated by plotting the number of non-toi recovered versus the number of TOI recovered if the targets were excavated in reverse order; that is, the anomalies labeled can t analyze are investigated first, followed by high-confidence TOI, followed by can t decide, and then highconfidence non-toi. 7

143 CAMP SIBERT Magnetic Size Attribute Histogram of the magnetometer derived size attribute and analysis results. 100 Number clutter mortars Target "Size" from Mag Analysis % mortars correctly identified Threshold high confidence munitions can't decide - munitions like can't decide - clutter like high confidence not munitions number Number of false of non TOI positives Magnetic Data: All munitions were correctly classified by most vendors 40 70% of the non hazardous objects correctly classified Case Histories 7 First, let s look at a target size attributes. Here, we show a histogram of the magnetometer derived target size attribute on the left. For illustration purposes, we show only two classes. Red identifies the 4.2inch mortars, and Green identifies everything else. Clearly, a significant fraction of the clutter is smaller than the 4.2inch mortars so this is a useful attribute. The ROC curve on the right shows classification results using this magnetometer derived size estimate. These are terrific results by any measure. In hindsight, we now know that there were over 700 anomalies caused by non-toi objects in addition to the roughly 120 TOI in the test set. If classification had not been used, all of the objects would have to be recovered. Using the classification scheme and associated prioritized dig list, we see that only 70 or so non-toi objects would have been removed in the process of recovering all of the ordnance in addition to the 100 or so anomalies marked can t analyze. 8

144 140 CAMP SIBERT EMI Size Attribute Histogram of the EMI derived size attribute and comparison of EM61 MK2 cart and array data analysis results. 100 Number clutter mortars % mortars correctly identified cart array 0 EMI Size-based Feature Case Histories number Number of false of non TOI positives EM61 MK2 Data: All munitions were correctly classified by most vendors 40 50% of the non hazardous objects correctly classified 8 Next, let s look at EMI target size attributes. As before, the histogram on the left clearly shows that a significant number of the non-toi can be eliminated using only this one attribute. The classification performance of this attribute is also very good. Using this attribute alone, a significant number of the non-toi, on the order of 40% to 50%, were successfully classified as high-confidence non-toi without a single false negative. 9

145 CAMP SIBERT Target size vs. peak signal Normalized number of TOI Prioritized by Target Size Estimate Prioritized by decreasing Signal Strength Case Histories Number of non-toi 9 Based on the mix of TOI and non-toi at this sight, it was clear that the size attributes would probably work well and we have just seen that they did. Here we compare performance results for two prioritization schemes. The first is based on the derived target-size attribute. The second is ordered by decreasing signal strength. Signal strength isn t entirely unreasonable because, as we ve seen, signal strength and target size are directly proportional. Depth of burial and target orientation however, also effect the measured signal strength and can confuse the issues. As expected, the list that is prioritized by decreasing signal strength starts out strong. This is seen in the rapidly rising portion of the black curve. It doesn t finish as strong, however. The last 4.2inch mortar isn t recovered until 404 non-toi were needlessly removed. The list prioritized by the EMI-derived target-size attribute, in comparison, doesn t start strong but does finish strong. Using this attribute for classification purposes, all of the 4.2inch mortars would have been recovered while only extracting 165 non- TOI. This includes the roughly 100 targets categorized as Can t Analyze. 10

146 decay rate wall thickness Case Histories CAMP SIBERT Multiple EMI Attributes Combining multiple EMI attributes, namely size and decay rate, improved classification performance for this demonstrator empty hole 4.2" mortar partial baseplate munitions debris cultural clutter response amplitude size % mortars correctly identified number Number of false of non TOI positives Full Classifier Size Only 10 We have observed good classification performance using a single, simple attribute at this site. Here, we show that the classification performance can be improved by intelligently combining multiple EMI attributes; namely, size and decay rates. 11

147 Case Histories CAMP SIBERT Advanced EM Sensor The Berkeley UXO Discriminator (BUD) was deployed over a portion of the entire site and produced a nearperfect performance. Starting from the munitions side, the initial 56 anomalies were munitions. The next 6 anomalies were false positives. The remaining 203 anomalies true negatives. % mortars correctly identified High Confidence Not Munition High Confidence Munition number Number of false of non TOI positives 11 So far, we have focused on EM and magnetic sensors that are commercially available. In addition to commercially existing sensors, a few emerging sensors were also demonstrated as part of the Camp Sibert Classification Study. Shown here are results for the Berkeley UXO Discriminator (BUD) sensor, which was deployed over a portion of the site. The BUD is a recently developed EMI sensor that consists of multiple coils arranged in a fixed geometry. It will be discussed later in the Emerging Technologies segment. As you can see by the ROC curve, the results are nearly perfect. The classification was based on attributes related to target size and decay rate. First, there were no Can t Analyze or Can t Decide declarations. Of the 62 high confidence TOI declarations, the initial 56 targets were munitions, and the remaining 6 were false positives. Of the 203 high confidence non-toi declarations, all were true negatives. 12

148 CAMP SIBERT Program Conclusions Demonstration of successful classification at a simple site The data were carefully collected and then analyzed using physics based analysis techniques The target selection criteria were based on the minimum expected signal strength for the TOI A number of the data+analysis combinations correctly classified all munitions Well over 50% of the detected clutter items were routinely eliminated with high confidence The BUD sensor produced near perfect results Case Histories 12 13

149 Remington Woods Bridgeport & Stratford, Connecticut ` Target of Interest: Non-Target of Interest: Historical Use: Example of: 37mm to 75mm projectiles Industrial and agricultural clutter Former munitions testing ground Classification approach for a site that is surrounded by residential housing and wooded Case Histories 13 Moving on to the second case history This site is a former munitions testing ground straddling the towns of Stratford and Bridgeport CT. It is a Brownfield redevelopment site known as Remington Woods. As you can see from the photograph, it is a more challenging environment than Camp Sibert. Generally it is wooded. Where the trees thin, it is vegetated or rocky. The targets of interest at this site include 37mm to 75mm projectiles (105mm are rumored but no real evidence to support the claim has been found). The clutter at the site is generally related to agricultural or industrial activities. 14

150 Remington Woods - Overview 422 acre former munitions testing site owned by DuPont Safety concerns dictate remote excavation and blast shield containment Process is time consuming and costly. Excavation and blast containment costs for a four acre parcel donated for highway improvements was roughly $1,000/target Site littered with metal clutter from 100 s of years of use Case Histories 14 Remington Woods is owned by DuPont and occupies 422 acres. Dupont and URS Corporation have been systematically clearing the site in phases since The property is surrounded by residences. Safety concerns dictate remote excavation and blast shield containment. Because of this, the remedial process is slow and costly. 15

151 Remington Woods - Process Site Clearance 1: Historical review 2: Surface contact removal thorough sweep of area using Schonstedt to detect and remove surface contacts to a depth of three inches 3: EM 61 Mk II survey to identify contacts Prioritize contacts based on survey data analysis: (1) possible UXO, (2) uncertain / more info needed, (3) high confidence not UXO 4: Cued identification Reacquire category 1 and 2 contacts, collect and process high resolution EM 61 HH data for target classification On basis of EM61 HH processing, reassign reacquired contacts to category 1, 2 or 3 5: Remove final category 1 and 2 targets using remote excavation with blast shield. Case Histories 15 Because the wooded terrain present problems for georeferencing the geophysical sensors data, a phased approach has been adopted. The geophysical phase of the clearance begins with an EM61 reconnaissance survey to identify anomalies. The EM61 survey used dead-reckoning techniques for spatial registration information. These data undergo a screening process that looks for rapidly decaying signals to identify thin walled items. Following the standard production survey, all anomalies that were not rejected based upon decay rate are resurveyed using an EM61HH sensor using a gridded template. The EM61HH data are characterized and classified using the methods discussed earlier today. The objective is to uniquely identify as much of the anthropic clutter as possible without leaving behind any TOI. At this site, the prioritized dig list allows the stakeholder the option of using a variety of remedial procedures thereby saving money in aggregate. 16

152 Remington Woods - Data Collection Identify potential hazards using reconnaissance survey data, collect additional data over targets using a template, extract attributes & classify Case Histories 0.25m 16 Upper left photograph This photograph is the EM61 sensor during the reconnaissance survey. The resulting data are used to locate anomalies for further investigation. Upper right photograph This is a photograph of the EM61HH sensor in operation using a gridded template. In this mode, a wooden template is used to precisely position the sensor at 0.15m intervals. 17

153 Remington Woods - Classification Approach Small to medium caliber projectiles Classification Approach 1. Signal decay rate over four time gates of EM61 MkII can be used to identify thin walled (<1/16 inch) scrap metal and wire 2. High resolution EM61 HH data grid over contact inverted to estimate target size and shape and match to possible UXO items Industrial/agricultural clutter Case Histories 17 Small to medium sized projectiles are the targets of interest at this site. Small to medium industrial and agricultural clutter represent the vast majority of the non-toi objects recovered to date. Due to its 100-year long history, a number of horseshoes are recovered. Almost without exception, the non-toi items are shallowly buried. As alluded to earlier, the classification approach has two steps. First, signal decay rates over the four time gates of the EM61 MKII sensor are used to detect and eliminate thin-walled objects. Next, EM61HH data are acquired using a gridded template over the remaining anomalies. Using a template, the measurements are precisely located with respect to each other but not referenced in any other way. Collecting data using a gridded template allows us to minimize spatial registration problems. The EM61HH data are then analyzed to extract attributes and classify. 18

154 Late/Early Decay Remington Woods Classification Step 1 Training Data Screening Sheet metal from Survey Data EMI signal decays more quickly in sheet metal scrap than in UXO Target identified as scrap metal if ratio of EM61 Mk II signal in last time gate to signal in first time gate is less than 1/10 Case Histories Since 2002, the percent rejected varies across the site: average 36% (ranges from 8% to 56%) 18 This slide summarizes the screening process applied to the reconnaissance EM61 data. The graph plots the Late to Early Decay rate as measured for various test items of varying wall thickness. Thin-walled objects are characterized by a rapid decay. The ratio of late to early responses, therefore is small. Objects with wall thicknesses of 2mm or more, as is the case with TOI at this site, decay more slowly and therefore generate a larger ratio. Since, 2002, this approach has allowed us to reject an average of 36% of the total anomalies. 19

155 Remington Woods Classification Step 2 Classification based on size and symmetry attributes EM61 HH grid data inverted for polarizability Attributes Symmetry based Attribute Unknown Size Symmetry Size based Attribute Case Histories 19 The plot on the right presents the decision space. The plotted values are the principle axis polarizations. On the x-axis, we plot an attribute related to the targets size. On the y-axis, we plot an attribute related to the inverted axial symmetry. Specifically, we use the secondary and tertiary polarizations. A target s size is reflected in this feature space by where it lies along a diagonal. Axially asymmetry is indicated by the vertical line length. The classification process is based on these attributes. Contrary to Sibert where there was a single TOI, at this site there multiple targets of interest. To derive a classification decision, we essentially evaluate the distance of the attributes for the unknown target to that observed for our training data. 20

156 Remington Woods Performance Summary Active Program since acres surveyed to date 12,700+ contacts in EM61 reconnaissance survey ~4,100 EM61 contacts ruled out (decay based) ~3,600 EM61HH contacts ruled out (size and symmetry) ~3,400 eliminated during cued collection (surface clutter found) Net result: ~1,600 contacts not classified as high confident non TOI Number of UXO found: 13 (no classification failures) Client estimated cost savings on excavation and blast containment in 2003 was ~$1M Case Histories 20 This project has been active since 2002 and has been quite successful. Roughly half of the total acreage has been surveyed and remediated. In total, more than 12,700 anomalies were identified. Of these, ~4,100 were ruled out during the screening process. Approximately 3,600 of the remaining anomalies were ruled out based on the classification analysis of the EM61HH data. Another 3,400 anomalies were eliminated because surface items were discovered during collection of the cued data. The net result all but 1,600 anomalies were eliminated and do not require expensive intrusive actions. DuPont remediate s a significant fraction of the eliminated, high confidence non- TOIs. A total of 13 munitions have been recovered with no classification failures. Cost savings, which are realized by changing the remediation process according to the final classification, are significant. They approached $1M in the last time cost savings were estimated by DuPont. 21

157 Classification in Action October 28, 2008 Only 462 of the contacts were possible unexploded ordnance, though, and are being excavated 10% of the remaining objects will be excavated to verify that they are what we say they are Case Histories 21 This article was recently published in a local Connecticut newspaper. It summarizes the project and illustrates the public nature of this type of program. Inherent in the text is the acceptance of the adopted classification process 22

158 This Page Has Been Intentionally Left Blank

159 Future Sensors: System Design and Classification Implications Thomas Bell Outline Electromagnetic Induction (EM) fundamentals Limitations of current commercial technology Next generation SERDP/ESTCP technology UXO specific features Examples Future Sensors 2 1

160 EM Fundamentals (A) Abrupt change in primary field excites eddy currents in buried object. (B) Eddy currents diffuse throughout the object and decay. Details depend on the size, shape and composition of the object. Future Sensors 3 A typical EM sensor measures the EM field associated with the decay of eddy currents induced in metal objects near the sensor. The eddy current decay occurs after the current pulse in the transmitter loop is completed, and hence after the response directly caused by any magnetization of material near the sensor. Unlike magnetometers, EM sensors respond to all types of metal objects, not just ferromagnetic ones. 2

161 Commercial EM Technology Widely used for detection surveys. Single axis coil sensor requires spatially mapped data to determine target parameters for classification. DGM survey towed array grid/template Future Sensors 4 The most widely used EM sensor for munitions detection is the Geonics EM61. The EM61 is a single-axis coil sensor and so data collected with the EM61 must be spatially mapped before it can be processed to determine target parameters for use in classification. The pictures show the EM61 in various modes of operation: on a gridded template to collect precisely positioned data over a previously located target, in the conventional wheeled mode for digital geophysical mapping (DGM) survey work, and as a vehicle-towed sensor array. The inset plot next to the standard wheeled EM61 shows a sample of survey data. The measurements are indicated by color coded dots. The background level is blue green and the red area shows signals due to a buried piece of metal. 3

162 Factors Affecting Performance Limited capability for target classification in survey mode Analog smoothing distorts signal shape Limited decay time coverage Centimeter level sensor positioning uncertainty degrades target parameter estimates Towed arrays have limited target illumination with transmitters operated simultaneously. Reduced data rate otherwise. Some success for cued target ID when used to collect static data with grid template EM63 (extended decay time coverage) had very good classification performance in cued template mode in Camp Sibert Classification Study Future Sensors 5 The EM61 has limited capability for target classification when used in the survey mode: smoothing that occurs during data acquisition to improve the signal-to-noise ratio distorts the signal shape, there is limited decay time coverage, and centimeterlevel positioning uncertainty degrades target parameter estimates. Towed arrays can improve the positioning accuracy. However, if the different transmitters are pulsed synchronously then the primary fields merge together and do not excite the target from different directions, while if they pulse sequentially then the data rate is reduced. The EM61 has had some success when used for target classification in a cued identification mode, collecting precisely positioned static data on a gridded template placed over the target. The Geonics EM63 is similar to the EM61, but has extended decay time coverage. It had very good classification performance when it was used in a cued template mode in the ESTCP Camp Sibert Classification Study. 4

163 New EM Technology New UXO specific EM technologies are being developed and tested under SERDP & ESTCP All digital electronics, measuring complete eddy current decay cycle Multi axis target excitation and observation for complete interrogation of principal axis polarizabilities. Future Sensors 6 SERDP and ESTCP are developing and testing new munitions-specific technologies which avoid these problems and should provide significantly improved discrimination performance. Several of the new systems are shown in the pictures. They have all digital, programmable electronics and are capable of measuring the complete eddy current decay cycle. They provide multi-axis target excitation and observation for complete interrogation of the principal axis response functions. 5

164 Classification Process The new EM technologies provide improvements at each stage in the classification process 1. Data Collection 2. Signal Attribute Extraction 3. Classification Future Sensors 7 There are three stages in the classification process. They are illustrated schematically in this slide. The first stage is data collection over an object, illustrated on the left by a picture of one of the new sensor systems stationed over a target to collect data. In the second stage, attributes or features of the EM response that relate to physical properties of the object are extracted from the data. The center figure shows a principal axis decay curve extracted from data collected over an object, along with a parametric fit to the curve using a physics-based response model. Finally, the object is classified by deciding whether the set of attributes is more like those typical of munitions or those typical of clutter items. The illustration on the right is a scatter plot two of the target attributes extracted from measurements of various mortar fragments (blue symbols) and intact mortars (red symbols). 6

165 Data Collection In order to observe the complete EM response pattern the object must be excited and measured from all directions. The new technologies accomplish this with multi axis coil sensors or single axis coil arrays. Multi axis coil array Single axis planar array Future Sensors 8 In order to adequately sample the complete EM response pattern the object must be excited and observed from all directions. The new technologies accomplish this with multi-axis coil sensors like that illustrated in the drawing on the left, or with single-axis coil arrays like that shown on the right. 7

166 Target Attribute Estimation Goal is to excite and measure object from all directions. Then the fundamental response functions (principal axis polarizabilities) can be extracted by inverting the set of EM data using the dipole response model. { V(t) = μ IC C B(t) } 1 0 R T Future Sensors 9 If the target is excited and measured from a broad range of angles, then the fundamental response functions (principal axis polarizabilities) can be extracted by using a standard dipole response model to invert the set of EM measurements. The process is illustrated schematically in the flow diagram on this slide. The sheaf of graphs on the left represents the set of measured induced field decay curves. The equation in the middle represents the inversion process that operates on the data collected over the target. The graph on the right shows the principal axis response functions determined by the inversion process. 8

167 Classification Time decays of the three principal axis polarizabilities are the EM signature of an object and are the basis for classification. Future Sensors 10 The set of three principal axis responses constitutes the basic EM signature of an object and is the basis for classification. The two figures at the bottom of this slide show the basic idea. Each is a plot of principal axis response curves extracted from data collected over the object shown in the corresponding inset picture: a 105 mm projectile on the left and a comparably sized tractor muffler on the right. At early times the responses are similar, but they evolve differently at later times. Two of the principal axis responses for the munitions item are the same because of the object's axial symmetry. All three of the principal axis responses are different for the muffler. This is expected for irregularly shaped clutter items. 9

168 New Technologies Multi axis coil sytems Berkeley UXO discriminator (BUD) Metal Mapper USGS ALLTEM Single axis coil arrays NRL TEM array ALLTEM BUD Metal Mapper Future Sensors TEM Array 11 New EM systems that have undergone testing in ESTCP are illustrated on this slide. The Berkeley UXO Discriminator (BUD), the USGS ALLTEM and the Geometrics Metal Mapper are all multi-axis coil systems, while the NRL time domain EM (TEM) system is a planar array of single axis coils. 10

169 Berkeley UXO Discriminator Multiaxis coil system Operates in survey mode (detection only) and cued (discrimination) mode. In survey mode, once a target is detected, stop and switch to discrimination mode. Excellent performance in Camp Sibert Classification Pilot Program (416 anomalies: 100% 10.8% PFA) Future Sensors 12 The Berkeley UXO Discriminator (BUD) is shown in the picture on the left at the bottom of this slide. It is a multi-axis coil system. The coil configuration is shown in the diagram on the right. It can operate in a survey mode for detection only. Once a target is detected, the operator stops and switches over to the discrimination mode to collect multi-axis data for classification. The BUD had excellent performance at the ESTCP Camp Sibert Classification Study. Using data collected at 416 anomalies, the BUD correctly identified all munitions items as munitions, and incorrectly identified only ~10% of the clutter items as munitions. 11

170 BUD Examples from Camp Sibert Future Sensors 13 This slide shows examples of principal axis response functions calculated from BUD data collected at Camp Sibert. The graph on the left shows the three polarizabilities for an intact 4.2 inch mortar (shown in the bottom left picture), while the graph on the right shows the polarizabilities for the large mortar fragment shown in the picture bottom right picture. Both objects were correctly classified based on the calculated polarizabilities. 12

171 Metal Mapper Geometrics, G&G Sciences and Snyder Geoscience Multi axis Tx coils, multiple small 3 axis Rx Survey (detection/location) and cued static classification modes APG Standardized Test Site demo September ʹ08 Future Sensors 14 This slide shows the Metal Mapper system developed by Geometrics in collaboration with G&G Sciences and Snyder Geoscience. The picture shows the Metal Mapper being towed over the Yuma Proving Ground test field. The diagram on the right shows the multi-axis coil configuration. Like the BUD, the Metal Mapper can operate in either a detection/location survey mode or a static classification mode. It underwent testing at the Aberdeen Test Site in September

172 MM Examples from YPG Cal Future Sensors 15 This slide shows examples of principal axis response functions calculated from Metal Mapper data collected in the Calibration Area at Yuma Proving Ground. The graph at the upper left shows the three polarizabilities for a steel sphere. All are equal as expected. The upper right and lower left figures are for a 105 mm HEAT round and a 60 mm mortar, respectively. Both show the expected behavior for munitions wherein the two weaker polarizabilities are equal due to the axial symmetry of the objects. The lower right figure is for a fragment from an exploded 155 mm projectile. The three distinct polarizabilities are consistent with the irregular shape of the munitions fragment. 14

173 ALLTEM USGS and Colorado School of Mines Multi axis Tx/Rx coils, sampled at ~3 Hz in survey mode Triangular Tx waveform includes on time induced field response 2005 & 06 tests at YPG Future Sensors 16 ALLTEM is a collaborative development of the US Geological Survey (USGS) and the Colorado School of Mines. It is unique among the new SERDP/ESTCP systems in that it uses a continuous triangular waveform, and therefore measures the magnetic response while the primary field is exciting the target in addition to the eddy current response. The ALLTEM is shown behind a tow vehicle in the picture on the left, and in an expanded view on the right, showing the locations of the various transmit and receive coils. It can operate in a survey mode with limited response sampling for use in classification, and has undergone testing at Yuma Proving Ground. 15

174 ALLTEM Examples from YPG Future Sensors 17 This slide shows examples of ALLTEM data. The graph in the upper right shows one cycle of the ALLTEM waveform (blue line) along with responses to a MK 118 rockeye (red line) and a 60 mm mortar (green line). The rockeye (bottom picture) is aluminum. Being nonmagnetic, it has a distinctively different response than does the steel 60 mm mortar pictured above the rockeye. The plot on the left is a survey map of the Calibration Area at Yuma Proving Ground that was created using gridded data (at one point in the waveform cycle) collected with the ALLTEM. The system is showing a good, strong response from all of the calibration targets. 16

175 NRL TEM Array NRL, G&G Sciences, Nova Research and SAIC 2D array of 25 time domain EMI sensors, decay times from 0.04 to 25 msec APG Standardized Test Site demo June ʹ08, > 200 targets/day Future Sensors 18 The time domain electromagnetic (TEM) array developed by the Naval Research Laboratory (NRL) in collaboration with G&G Sciences, Nova Research and Science Applications International Corporation (SAIC) is shown on this slide. The photograph at the lower left shows the array behind the NRL tow vehicle, and the drawing at the right shows the layout of the array elements. There are 25 transmit/receive coil pairs. The TEM array can measure induced eddy current decay from 0.04 msec to beyond 25 msec after the primary field pulse. It performed very well in a demonstration at the Aberdeen Proving Ground Standardized Test Site in June The system operates in a cued interrogation mode, collecting data while parked stationary over a target. At Aberdeen it was able to interrogate over 200 targets per day. 17