Criteria And Methodology For Evaluating Water Distribution Network Early Warning Systems For Use On Military Bases Dan Kroll Hach Co. NDIA-E2S2 Denver, CO
The threat to our water is real! Number of different sensors and algorithms are on the market Some sold as packages and some as individual components. How can we evaluate?
Criteria for making determinations. Dual Use Value Detection Class Requirements Coverage Characteristics Operational Characteristics Performance Characteristics ROC Curves
Dual Use Value: How do we get an ROI (Return on Investment) Optimization of daily operations within system monitored Provide alarms for operational events Replace grab sampling for compliance Document system operation anomalies to assist with planning activities (e.g. pump or line replacement) Consumer confidence through continuous quality documentation.
Detection Class Requirements Detect to treat: Near 100% confident. Slow high cost. Detect to protect: High confidence. Response fast. Cost medium. Detect to warn: Lower confidence. Response fast. Cost low. An EWS needs to be rapid and have low cost for wide coverage. Assume detect to warn.
Coverage Characteristics ( Cost) May not be a factor for high value installations or icon facilities. Large coverage areas like cities may be a factor. Other constraints on costs like infrastructure.
Coverage Characteristics (Area of protection) Function of hydraulics Not all ideal deployment sites will be able to be utilized due to logistics and system requirements Site noise Protection of all not possible
Coverage Characteristics (Communication) Multiple points require communications Human interaction probably necessary to take action. EWS structured for secure communication with security.
Operational Characteristics Crisis critical systems must be easy to operate Intuitive user interface Minimally skilled operators. (Ease of use)
Operational Characteristics (Automated) Must operate w/o presence of a human under normal circumstances Service or maintenance interval
Operational Characteristics No long data gaps Max on order of a minutes Longer response times unduly endanger customers. (Continuous)
Operational Characteristics (Reliable) A non-working system is an opportunity for exploitation
Operational Characteristics Amortized cost per day similar to a comparable grab sample regime. Need to take into account better picture of system. (Cost effective)
Performance Characteristics Many and various contaminants Different characteristics Multivariate systems (Detection spectrum)
Performance Characteristics (Rapid) Response time = doses delivered Matter of minutes not hours
Performance Characteristics (Specific) Specific vs. general analyzers Orthogonal sensors to be favored
Performance Characteristics (Reproducible) Must be reproducible to be trustworthy Should be demonstrated on actual agents
Performance Characteristics (Low false negatives/positives) Some systems blind to some classes Multiple sensors more difficult to fool Random noise and insufficient information Site specific Alarm threshold Many methods inferential
Performance Characteristics (Qualitative) Contaminants named if possible with degree of confidence General category also useful and may be preferable
Performance Characteristics (Quantitative) Quantity helpful for treatment and clean-up
Performance Characteristics (Sensitive) Must be sensitive to harmful concentrations. MDLs can be misleading. Often site specific. ROC curves classic and modified
Classic ROC Curves FPR Vs1-FNR
Hit Rate Classic ROC is plotted parametrically from FAR as a function of trigger level versus Hit Rate as a function of trigger level. May not be best for continuous measurements. ROC for Sodium Fluoroacetate at 0.25, 0.5 and 1 % LD50 120 100 80 60 40 20 1 pct.5 pct.25 pct 0 0 50 100 150 False Alarm Rate
Alternate Method May provide more useful information to operators that need to set a threshold level for alarms FAR is expressed as Mean Time Between False Alarms versus Trigger Threshold. The Hit Rate function is translated into the amount of agent expressed as a % of the LD-50 for a 70kg male.
Trigger Amount %LD50 Allow user to select Trigger Threshold based on desired trigger sensitivity vs. acceptable time between false alarm due to noise. The blue markers are points of different Trigger Threshold.. ROC curve for Cyanide 0.3 0.25 0.2 0.15 0.1 0.05 0 0 5 10 15 20 25 Mean Time Betw een False Positives - Months
To plot the classical ROC curves we need False Alarm Rate vs. Trigger Threshold Hit Rate vs. Trigger Threshold To plot the alternate form of ROC curve we need Mean Time Between False Alarm vs. Trigger Threshold Trigger Amount (%LD-50) vs. Trigger Threshold
HR Vs. Trigger Threshold Curve The HR curve for classic ROC format is derived from a Monte Carlo simulation with a run of 1000 trials for each point on the HR curve. Agent is selected which defines the LD-50 Lab test data for that agent at known conc. are entered into the spread sheet. (5baseline data deviations) Process water + measurement noise for each parameter are defined
Use standard deviation for 4970 of the 5000 values during 30 minute run time were calculated of each parameter and the typical standard deviation values used to define the measurement + noise statistics. The resolution of the measurement devices is defined A trigger Threshold level and a Dose value are entered.
The spreadsheet then calculates 1000 independent points where the process has been dosed and noise added via an independent random number generator. The noise figures are max values observed during 30 minute intervals with no agents or upsets The probability density function for each parameter noise component was selected pessimistically to be a flat distribution between extreme values of the noise component. (worst case) Measurement values are rounded to represent quantization noise found in analog to digital conversion electronics of the system using pessimistic values
P( Trigger ) % Vector values processed by Trigger Algorithm to produce a signal for each vector. Values compared to Trigger Threshold values in Mote Carlo simulation to obtain number of hits per 1000 trials. Number per 1000 is converted to Hit Rate %. Repeated for different trigger levels. P( Trigger ) vs Threshold for 0.7% LD50 Nicotine 120 100 80 60 40 20 0 0 0.5 1 1.5 2 2.5 Threshold
FAR vs. Trigger Threshold FAR generated from 5,000 data points taken at a local water plant during a period when operation was normal and no upsets were present. The data for the five sensed parameters ( ph, conductivity, free chlorine, turbidity and Total Organic Carbon ) that are input to the Event Monitor were processed by the Trigger Algorithm to derive 5,000 points of Trigger Signal versus time.. The mathematical model for the FAR is: FAR = 100*EXP( -13.4945*Trigger Level )
Hit Rate With Hit Rate and FAR a ROC curve can be calculated ROC curve for NaFAc 120 100 80 60 40 1 pct.5 pct.25 pct 20 0 0 20 40 60 80 100 120 False Alarm Rate
MTBFA The FAR can be transformed into a Mean Time Between False Alarm function by including the sampling frequency of the continuous monitor. MTBFA = duration of monitoring period ( time ) number of hits in the period = duration of monitoring period ( time ). probability of hit*duration of monitoring period*sampling frequency( samples/time) = 1/ probability of hit*sampling frequency( samples/time ) For the Event Monitor, sampling frequency is once per minute, so MTBFA ( minutes ) = 1/probability of hit
Trigger Threshold For the modified ROC curve the trigger threshold is set at a selected value. Dose is adjusted to achieve 100% hit rate. A new trigger value is selected and process is repeated defining the Dose vs. Trigger threshold curve. MDL %LD50 RICIN vs Threshold 70 60 50 40 30 20 10 0 0 0.5 1 1.5 Percent LD50
Trigger Amount % LD50 The Dose vs. Threshold Curve is plotted parametrically with the MTBFA vs. Threshold curve to generate a modified ROC format. ROC curve for Ricin in 1 ppm FAC 70 60 50 40 30 20 10 0 0 5 10 15 20 25 Mean Time Between False Positives - Months
Scaled values Input Five Parameter Signals What is our vision? Output Single Signal Baseline Data 2.50 2.00 1.50 1.00 0.50 0.00 0.00 100.00 200.00 300.00 400.00 500.00 600.00 700.00 800.00 900.00 1000.00 Time Turb ph cond Chlorine TOC Analyze Software Trigger Signal 1% LD50 Cyanide 2 1.8 Alarm 1.6 1.4 Alarm Name 1.2 Statistics 1 0.8 0.6 0.4 0.2 0 0 200 400 600 800 1000 Learning The Event Monitor analyzes plant data, alarms on significant deviations from baseline, reports the Event Name if found, and learns the event fingerprint if not already in the Plant Event Library. 1 0.6 0.2-0.2-0.6-1 ph Cond Turb Chlor TOC Event Fingerprint Plant Event Library
Deployed at 2 Military and 1 Civilian Site to Collect Data for Evaluation Civilian site has quiescent water quality with little variability (Easy) Fort has less control with some variability (Medium) Base has extreme variability in a number of parameters especially ph and Chlorine (Difficult)
Months of data from each site. 9 continuous representative days selected for each site to do analysis. Parameter signals for all sensors characterized over the data set and imported into the ROC spreadsheets. Included variation in water plus noise and sensor errors
Weeks MTBA vs Threshold 20 15 10 BASE FORT 5 0 0.5 0.7 0.9 1.1 1.3 1.5 Threshold
% LD 50 Monte Carlo Simulation With 10,000 Samples of Random Noise Detection Concentration versus Threshold 2.5 2 1.5 1 0.5 0 0 1 2 3 4 Threshold
% LD50 Modified ROC ROC Curve - Cyanide @ Base 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0 20 40 60 80 Mean Time Between Alarms - Weeks
MDL % LD50 Aldicarb ROC Curves - BASE and FORT 3.5 3 2.5 2 1.5 1 0.5 0 0 20 40 60 80 MTBA - Weeks BASE FORT
% Probability Detection rate versus Concentration for Aldicarb at 2 sites. Hit Rate for Alarm - Aldicarb 120 100 80 60 40 BASE City 20 0 0 0.5 1 1.5 % LD 50
Monte Carlo vs. CSV File Dosing CASE ROC MDL CSV MDL Difference City - Aldicarb 0.81 0.779 0.031 City - Cyanide 0.523 0.499 0.024 City - Oxamyl 0.219 0.217 0.002 Base - Aldicarb 0.974 0.971 0.003 Base - Cyanide 0.609 0.498 0.111 Base - Oxamyl 0.258 0.229 0.029 Fort - Aldicarb 0.972 0.959 0.013 Fort - Cyanide 0.6 0.527 0.073 Fort - Oxamyl 0.257 0.228 0.029
Minutes Detection Time Analysis Detection Time vs Injection Peak Value 14 13 12 11 10 9 8 7 40 Minute 20 Minute 6 0 2 4 6 8 10 12 Peak Trigger Value
Other Observations Dual Use: Detected a number of non security problems including dead ends pumping problems and contamination with aviation fuel. Could detect acceptably low levels in a matter of minutes Coverage adequate Communications simple and secure
Conclusion Water distribution system early warning systems are shown to be effective means of enhancing the security of base water supplies and the new ROC method is shown to be an effective way of validating these system with a tool that can be useful to operators in running and tuning thee systems.
Questions?