Quality Management for Advanced Classification David Wright Senior Munitions Response Geophysicist CH2M HILL
Goals of Presentation Define Quality Management, Quality Assurance, and Quality Control in the context of Advanced Classification Present current state of the art with examples Discuss direction of current development and associated challenges. 2
Quality Management Objectives Ensure the quality of the data being collected Ensure that the collected data and derived products support conclusions based upon these results Document the QC/QA findings so that the client can be assured of the quality of the results 3
Quality Management Quality Management: Program activities involved with all aspects of quality Quality Assurance: Audit project activities against planning documents Quality Control: Audit project data against planned or standard measurement quality objectives Quality Management Personnel Requirements Qualifications Training Quality Assurance Audit against planning docs 3 rd party data verification /validation Validate results Quality Control Verify data Validate data Project Planning QAPPs Work Plans MQOs System Maintenance Hardware upgrades Software upgrades SOPs 4
Recent Demonstrations Different Sites Different Sensor Technologies Different Analysts/ Approaches 5
Quality Control Verify and Validate: Data Acquisition Cued measurements Dynamic measurements Data Modeling Single-target or multi-target inversion to find model(s) that best fit observed data Output = features intrinsic to the target Classification Use features to sort targets into a prioritized list 6
Advanced Sensors Sensor Tx/ target orientation Tx/ Rx combinations Time gates Data pts/ cued target EM61 Mk2 Dynamic* Single 4 324* TEMTADS 2x2 Dynamic 4 Tx x 12 Rx = 48 100 4,800 MetalMapper Dynamic 3 Tx x 21 Rx = 63 40 2,500 * requires multiple positions (9x9 pt sample grid) 1 2 4 3 7
Sensor Function QC Attempts to use inversion results over known targets (IVS) for field QC often fail. Inversions tend to average out errors Inversions do not always exercise all sensor components. Single coil (Rx2Z) failure example Field QC inversion results for a medium ISO using MetalMapper sensor missing RX coil #7 Medium ISO 10 2 10 0 8 10-2 0.849 0.0005 0.001 0.005
Betas Betas Sensor Function QC TEMTADS Single coil (Rx2Z) failure example (20 mm): Center of Array Coil 1 Coil 2 Coil 3 Coil 4 9
Advanced Sensor Function QC TX1 TX2 TX3 TX4 Rx4X Rx4Y Rx4Z Rx3X Rx3Y Rx3Z Tx/Rx1 Tx/Rx2 Tx/Rx3 Tx4/Rx4 Rx2X Rx2Y Rx2Z Monostatic Terms X Rx1X 1 2 Y Rx1Y Rx1Z 4 3 Z 10
blank 213am 213pm 214am 215am 215pm 217am 217pm 218am 218pm 219am 219pm 220am 220pm 224am 224pm Sensor Function Failure Example Single Single Multiple QC reports should be: Easy to generate on a daily basis Designed to identify problems in a timely manner Standardized for easy interpretation and evaluation 75 50 25 0-25 -50-75 TEMTADS 2x2 Function Test New Boston AFS (Cued) Tx0_Rx0X Tx0_Rx0Y Tx0_Rx0Z Tx1_Rx1X Tx1_Rx1Y Tx1_Rx1Z Tx2_Rx2X Tx2_Rx2Y Tx2_Rx2Z Tx3_Rx3X Tx3_Rx3Y Tx3_Rx3Z 11
Background Removal QC Background corrections applied to remove: 1. Soil response 2. Zero level drift Therefore background measurements must: 1. Be taken in similar soils to that of the target (minimize the spatial offset) 2. Be taken frequently (minimize temporal offset) 3. Be free of signal due to metal sources 12
Background Removal QC Dynamic surveys have the advantage of minimizing the temporal and spatial separation between the background measurement and the target measurement. Dynamic background locations 13
Background Removal QC Static Backgrounds are temporally and spatially offset so will inject some error Static background location 14
Data Modeling Principal Axes Polarizabilities (betas) + Refined Position /Attitude estimates 15
Data Modeling QC 1 Polarizability (m 3 /A) 0.1 0.01? 0.001 0.1 1 10 Time (ms) The intrinsic parameters are representative of the target if: 1. The modeled data match the observed data 2. The target was energized along all three principal axes 16
Data Modeling QC Fit coherence metric indicates how well the model fits the observed data Fit coh =.994 17
Data Modeling QC Fit coherence metric indicates how well the model fits the observed data Fit coh =.791 18
Data Modeling QC Targets outside of the sweet spot may not have been energized along all three axes Fit position offset > 40 cm indicate the results may be unreliable 1 2 1 2 Plan View 4 3 4 3 Side View 40 cm 40 cm 19
Classification State of art is converging to Library Matching Library Match Feature Space 20
Classification Library matching is the primary approach to classification, but there is considerable variability in how it is implemented: Some art still used optimize the approach for some sites Man-in-the-loop decisions are made on difficult targets more experience = greater success What is more important to the client, performance or transparency? 21
Classification QC Poor performances attributable to lack of: training, SOP s and proper QC/QA experienced analyst inexperienced analyst Variability Due to Analysts / Approach 22
Variability Due to Analysts Success Requires: Training Experience Quality Control/Assurance 23
Quality Management Personnel Requirements Qualifications Training Project Planning QAPPs Site considerations System Maintenance Hardware software upgrades SOPs 24
Personnel Requirements NAOC working group recommendations: Munitions Response Advanced Classification (AC) Geophysical Qualifications Position Experience 1 Education Training 2 Certifications 40 hours in applicable DGM Software, including Documented experience o Geosoft (or equivalent) applying advanced Degree in: o Geosoft QA QC tools (or classification 3 geophysics, equivalent) No applicable 1 year experience analyzing geology, or 16 hours of training on AC method certifications DGM data for MR projects a closely related and mode of operation being at present 24 hours AC analysis under scientific or implemented the direct supervision of a engineering field 5 8 hours AC analysis refresher qualified AC Data Analyst training, if no AC analysis has occurred in past year AC Data Analyst Lead NOTES: 1. Experience qualifications follow individuals. Companies can acquire experience through joint ventures, teaming arrangements, or mentor-protégé relationships in addition to self performance. 2. Training could be from hardware/software manufacturers, USACE or other agencies responsible for executing AC projects, ESTCP, or equivalent in-house training as appropriate. 3. Experience could be from a Standardized Test Site, ESTCP demonstration project, pilot study, characterization project, response action, remedial action, or an equivalent project or standardized test data set. Recommended minimum experience consists of AC analysis on two projects or AC analysis of 2,500 targets using the proposed project methodology. 4. Experience could be from a Standardized Test Site, ESTCP demonstration project, pilot study, characterization project, response action, remedial action, or an equivalent project. 5. Or has documented AC experience prior to 2014. 25
Personnel Requirements NAOC working group recommendations: Munitions Response Advanced Classification (AC) Geophysical Qualifications Position Experience 1 Education Training 2 Certifications Documented experience Degree in: 8 hours training on each advanced collecting AC data 4 geophysics, EMI sensor and mode of operation 1 year experience collecting geology, or being utilized No applicable DGM data for MR projects a closely related 8 hours AC data collection refresher certifications 24 hours in-field AC data scientific or training, if no AC data collection has at present collection under the direct engineering field 5 occurred in past year supervision of a qualified AC Instrument Operator AC Data Collection Lead NOTES: 1. Experience qualifications follow individuals. Companies can acquire experience through joint ventures, teaming arrangements, or mentor-protégé relationships in addition to self performance. 2. Training could be from hardware/software manufacturers, USACE or other agencies responsible for executing AC projects, ESTCP, or equivalent in-house training as appropriate. 3. Experience could be from a Standardized Test Site, ESTCP demonstration project, pilot study, characterization project, response action, remedial action, or an equivalent project or standardized test data set. Recommended minimum experience consists of AC analysis on two projects or AC analysis of 2,500 targets using the proposed project methodology. 4. Experience could be from a Standardized Test Site, ESTCP demonstration project, pilot study, characterization project, response action, remedial action, or an equivalent project. 5. Or has documented AC experience prior to 2014. 26
Personnel Requirements NAOC working group recommendations: Munitions Response Advanced Classification (AC) Geophysical Qualifications Position Experience 1 Education Training 2 Certifications Documented experience Degree in: 8 hours training on each advanced collecting AC data 4 geophysics, EMI sensor and mode of operation 1 year experience collecting geology, or being utilized No applicable DGM data for MR projects a closely related 8 hours AC data collection refresher certifications 24 hours in-field AC data scientific or training, if no AC data collection has at present collection under the direct engineering field 5 occurred in past year supervision of a qualified AC Instrument Operator AC Data Collection Lead NOTES: 1. Experience qualifications follow individuals. Companies can acquire experience through joint ventures, teaming arrangements, or mentor-protégé relationships in addition to self performance. 2. Training could be from hardware/software manufacturers, USACE or other agencies responsible for executing AC projects, ESTCP, or equivalent in-house training as appropriate. 3. Experience could be from a Standardized Test Site, ESTCP demonstration project, pilot study, characterization project, response action, remedial action, or an equivalent project or standardized test data set. Recommended minimum experience consists of AC analysis on two projects or AC analysis of 2,500 targets using the proposed project methodology. 4. Experience could be from a Standardized Test Site, ESTCP demonstration project, pilot study, characterization project, response action, remedial action, or an equivalent project. 5. Or has documented AC experience prior to 2014. 27
Personnel Requirements NAOC working group recommendations: Munitions Response Advanced Classification (AC) Geophysical Qualifications Position Experience 1 Education Training 2 Certifications Documented experience Degree in: 8 hours training on each advanced collecting AC data 4 geophysics, EMI sensor and mode of operation 1 year experience collecting geology, or being utilized No applicable DGM data for MR projects a closely related 8 hours AC data collection refresher certifications 24 hours in-field AC data scientific or training, if no AC data collection has at present collection under the direct engineering field 5 occurred in past year supervision of a qualified AC Instrument Operator AC Data Collection Lead NOTES: 1. Experience qualifications follow individuals. Companies can acquire experience through joint ventures, teaming arrangements, or mentor-protégé relationships in addition to self performance. 2. Training could be from hardware/software manufacturers, USACE or other agencies responsible for executing AC projects, ESTCP, or equivalent in-house training as appropriate. 3. Experience could be from a Standardized Test Site, ESTCP demonstration project, pilot study, characterization project, response action, remedial action, or an equivalent project or standardized test data set. Recommended minimum experience consists of AC analysis on two projects or AC analysis of 2,500 targets using the proposed project methodology. 4. Experience could be from a Standardized Test Site, ESTCP demonstration project, pilot study, characterization project, response action, remedial action, or an equivalent project. 5. Or has documented AC experience prior to 2014. Bachelor of Science degree or demonstrated equivalent proficiency with scientific concepts 28
Project Planning Quality Assurance Project Plan (QAPP) QAPP: a formal document describing in comprehensive detail the necessary quality assurance (QA), quality control (QC), and other technical activities that must be implemented to ensure that the results of the work performed will satisfy the stated performance criteria EPA web site
Project Planning: Site Considerations Soil composition / geology Types and variability of anticipated TOI s, % of TOI Density distribution of metal Pilot Studies/Cost Benefit Analysis to manage expectations 30
ssed TOI models 400 300 passed TOI models 200 Site 10-2 -2 100 Conditions: Local Geology 10 10 2 10 0 10 0 L2 L3 6 8 ssed models ssed TOI models rary 6 8 ssed models ssed TOI models 0-5 0 5 10 Polarizability quality 500 400 300 200 100 Polarizability 0-5 0 5 10 Quality 400 300 200 100 Median = 1.73 / 6.78 Polarizability quality Median = 0.726 / 3.48 passed models passed TOI models passed models passed TOI models 10-4 Local geology Feature Extraction Classification (small ISO) Results Pole Mtn (resistive) Beale (conductive) 10 2 10 0 10-2 10-2 10-4 10-4 10 2 10 2 10 0 10-2 0.0005 0.001 0.005 Ltot L1 L2 L3 Polarizabilities for ISO seeds L1 L2 L3 0.0005 0.001 0.005 0.0005 0.001 0.005 L1 Ltot L2 L3 L1 L2 L3 6 8 0-5 0 5 10 Polarizability quality Median = 0.601 / 3.15 10-4 -4 0.0005 0.001 0.005 L1 Ltot L2 L3 ssed models ssed TOI models 400 passed models passed TOI models 10 2 10 2 L1 L2
Site Conditions: TOI Types 20mm Projectiles Excluded 20mm Projectiles Included Results from Pilot Study at the Former VNTR 32
System Maintenance Sensor redesign/upgrades Data format moving to Hierarchical Data Format (HDF)5. Field-worthiness improvements Software upgrades Take advantage of HDF5 format Streamlining to create a standard workflow Emphasis on QC tools 33
Standard Operating Procedures 1 Assemble the MetalMapper System and Verify Correct Operation 2 Test Sensor and System at the IVS 3 Production Area Seeding 4 Collect Dynamic Data Using the MetalMapper Sensor 5 Preprocess Dynamic Data and Identify Anomalies 6 Collect Static Background Measurements 7 Collect Cued Target Measurements 8 Verify Usability of Advanced Sensor Data 9 Background Correct Cued Anomaly Data 10 Invert anomaly data to extract source parameters 11 Compare extracted parameters to MEC signatures in the data library 12 Develop prioritized dig list using library matching and other factors 13 Verify recovered objects are compatible with advanced classification predictions 14 Develop verification sampling dig list and perform verification sampling 34
Quality Assurance Challenges: Library maintenance Classification approach Operator threshold and validation of results 35
TOI Library What items are in the library? How can we ensure that the library is appropriate for the sensor configuration being used? How do we handle the risk of unexpected munitions types that are not in the library? 36
Objective vs Subjective Analysis Ranking should be performed using quantitative, objective, defensible criteria. Subjective decisions? Similar to today s standard DGM practice of having the analyst finalize the dig list by adding and/or removing targets based upon their expert judgment 37
Subjective Demotions Subjective decisions to move a target from the dig to no-dig category should be strongly discouraged 38
Subjective Promotions Subjective decisions to move a target from the no-dig to dig category are allowed but: They must be identified as subjective promotions If a QC or QA seed is found in this manner, a root cause analysis must be performed 39
Analysts Threshold Who should ultimately make the stop dig decision? Other factors such as remediation budget, end use, exposure pathways are beyond the purview of the analyst. 40
Classification Validation Can the results act as a QC tool? Analyst Threshold +10% +20% 41
Final Thoughts Advanced Classification has been shown to provide significant efficiencies in risk reduction at military munitions response sites Quality Management is critical to the successful implementation of Advanced Classification 42
Comments/Questions? Contact: David Wright David.Wright@CH2M.com (919) 520 8673 43