Criteria And Methodology For Evaluating Water Distribution Network Early Warning Systems For Use On Military Bases

Transcription

1 Criteria And Methodology For Evaluating Water Distribution Network Early Warning Systems For Use On Military Bases Dan Kroll Hach Co. NDIA-E2S2 Denver, CO

2 The threat to our water is real! Number of different sensors and algorithms are on the market Some sold as packages and some as individual components. How can we evaluate?

3 Criteria for making determinations. Dual Use Value Detection Class Requirements Coverage Characteristics Operational Characteristics Performance Characteristics ROC Curves

4 Dual Use Value: How do we get an ROI (Return on Investment) Optimization of daily operations within system monitored Provide alarms for operational events Replace grab sampling for compliance Document system operation anomalies to assist with planning activities (e.g. pump or line replacement) Consumer confidence through continuous quality documentation.

5 Detection Class Requirements Detect to treat: Near 100% confident. Slow high cost. Detect to protect: High confidence. Response fast. Cost medium. Detect to warn: Lower confidence. Response fast. Cost low. An EWS needs to be rapid and have low cost for wide coverage. Assume detect to warn.

6 Coverage Characteristics ( Cost) May not be a factor for high value installations or icon facilities. Large coverage areas like cities may be a factor. Other constraints on costs like infrastructure.

7 Coverage Characteristics (Area of protection) Function of hydraulics Not all ideal deployment sites will be able to be utilized due to logistics and system requirements Site noise Protection of all not possible

8 Coverage Characteristics (Communication) Multiple points require communications Human interaction probably necessary to take action. EWS structured for secure communication with security.

9 Operational Characteristics Crisis critical systems must be easy to operate Intuitive user interface Minimally skilled operators. (Ease of use)

10 Operational Characteristics (Automated) Must operate w/o presence of a human under normal circumstances Service or maintenance interval

11 Operational Characteristics No long data gaps Max on order of a minutes Longer response times unduly endanger customers. (Continuous)

12 Operational Characteristics (Reliable) A non-working system is an opportunity for exploitation

13 Operational Characteristics Amortized cost per day similar to a comparable grab sample regime. Need to take into account better picture of system. (Cost effective)

14 Performance Characteristics Many and various contaminants Different characteristics Multivariate systems (Detection spectrum)

15 Performance Characteristics (Rapid) Response time = doses delivered Matter of minutes not hours

16 Performance Characteristics (Specific) Specific vs. general analyzers Orthogonal sensors to be favored

17 Performance Characteristics (Reproducible) Must be reproducible to be trustworthy Should be demonstrated on actual agents

18 Performance Characteristics (Low false negatives/positives) Some systems blind to some classes Multiple sensors more difficult to fool Random noise and insufficient information Site specific Alarm threshold Many methods inferential

19 Performance Characteristics (Qualitative) Contaminants named if possible with degree of confidence General category also useful and may be preferable

20 Performance Characteristics (Quantitative) Quantity helpful for treatment and clean-up

21 Performance Characteristics (Sensitive) Must be sensitive to harmful concentrations. MDLs can be misleading. Often site specific. ROC curves classic and modified

22 Classic ROC Curves FPR Vs1-FNR

23 Hit Rate Classic ROC is plotted parametrically from FAR as a function of trigger level versus Hit Rate as a function of trigger level. May not be best for continuous measurements. ROC for Sodium Fluoroacetate at 0.25, 0.5 and 1 % LD pct.5 pct.25 pct False Alarm Rate

24 Alternate Method May provide more useful information to operators that need to set a threshold level for alarms FAR is expressed as Mean Time Between False Alarms versus Trigger Threshold. The Hit Rate function is translated into the amount of agent expressed as a % of the LD-50 for a 70kg male.

25 Trigger Amount %LD50 Allow user to select Trigger Threshold based on desired trigger sensitivity vs. acceptable time between false alarm due to noise. The blue markers are points of different Trigger Threshold.. ROC curve for Cyanide Mean Time Betw een False Positives - Months

26 To plot the classical ROC curves we need False Alarm Rate vs. Trigger Threshold Hit Rate vs. Trigger Threshold To plot the alternate form of ROC curve we need Mean Time Between False Alarm vs. Trigger Threshold Trigger Amount (%LD-50) vs. Trigger Threshold

27 HR Vs. Trigger Threshold Curve The HR curve for classic ROC format is derived from a Monte Carlo simulation with a run of 1000 trials for each point on the HR curve. Agent is selected which defines the LD-50 Lab test data for that agent at known conc. are entered into the spread sheet. (5baseline data deviations) Process water + measurement noise for each parameter are defined

28 Use standard deviation for 4970 of the 5000 values during 30 minute run time were calculated of each parameter and the typical standard deviation values used to define the measurement + noise statistics. The resolution of the measurement devices is defined A trigger Threshold level and a Dose value are entered.

29 The spreadsheet then calculates 1000 independent points where the process has been dosed and noise added via an independent random number generator. The noise figures are max values observed during 30 minute intervals with no agents or upsets The probability density function for each parameter noise component was selected pessimistically to be a flat distribution between extreme values of the noise component. (worst case) Measurement values are rounded to represent quantization noise found in analog to digital conversion electronics of the system using pessimistic values

30 P( Trigger ) % Vector values processed by Trigger Algorithm to produce a signal for each vector. Values compared to Trigger Threshold values in Mote Carlo simulation to obtain number of hits per 1000 trials. Number per 1000 is converted to Hit Rate %. Repeated for different trigger levels. P( Trigger ) vs Threshold for 0.7% LD50 Nicotine Threshold

31 FAR vs. Trigger Threshold FAR generated from 5,000 data points taken at a local water plant during a period when operation was normal and no upsets were present. The data for the five sensed parameters ( ph, conductivity, free chlorine, turbidity and Total Organic Carbon ) that are input to the Event Monitor were processed by the Trigger Algorithm to derive 5,000 points of Trigger Signal versus time.. The mathematical model for the FAR is: FAR = 100*EXP( *Trigger Level )

32 Hit Rate With Hit Rate and FAR a ROC curve can be calculated ROC curve for NaFAc pct.5 pct.25 pct False Alarm Rate

33 MTBFA The FAR can be transformed into a Mean Time Between False Alarm function by including the sampling frequency of the continuous monitor. MTBFA = duration of monitoring period ( time ) number of hits in the period = duration of monitoring period ( time ). probability of hit*duration of monitoring period*sampling frequency( samples/time) = 1/ probability of hit*sampling frequency( samples/time ) For the Event Monitor, sampling frequency is once per minute, so MTBFA ( minutes ) = 1/probability of hit

34 Trigger Threshold For the modified ROC curve the trigger threshold is set at a selected value. Dose is adjusted to achieve 100% hit rate. A new trigger value is selected and process is repeated defining the Dose vs. Trigger threshold curve. MDL %LD50 RICIN vs Threshold Percent LD50

35 Trigger Amount % LD50 The Dose vs. Threshold Curve is plotted parametrically with the MTBFA vs. Threshold curve to generate a modified ROC format. ROC curve for Ricin in 1 ppm FAC Mean Time Between False Positives - Months

36 Scaled values Input Five Parameter Signals What is our vision? Output Single Signal Baseline Data Time Turb ph cond Chlorine TOC Analyze Software Trigger Signal 1% LD50 Cyanide Alarm Alarm Name 1.2 Statistics Learning The Event Monitor analyzes plant data, alarms on significant deviations from baseline, reports the Event Name if found, and learns the event fingerprint if not already in the Plant Event Library ph Cond Turb Chlor TOC Event Fingerprint Plant Event Library

37 Deployed at 2 Military and 1 Civilian Site to Collect Data for Evaluation Civilian site has quiescent water quality with little variability (Easy) Fort has less control with some variability (Medium) Base has extreme variability in a number of parameters especially ph and Chlorine (Difficult)

38 Months of data from each site. 9 continuous representative days selected for each site to do analysis. Parameter signals for all sensors characterized over the data set and imported into the ROC spreadsheets. Included variation in water plus noise and sensor errors

39 Weeks MTBA vs Threshold BASE FORT Threshold

40 % LD 50 Monte Carlo Simulation With 10,000 Samples of Random Noise Detection Concentration versus Threshold Threshold

41 % LD50 Modified ROC ROC Curve - Base Mean Time Between Alarms - Weeks

42 MDL % LD50 Aldicarb ROC Curves - BASE and FORT MTBA - Weeks BASE FORT

43 % Probability Detection rate versus Concentration for Aldicarb at 2 sites. Hit Rate for Alarm - Aldicarb BASE City % LD 50

44 Monte Carlo vs. CSV File Dosing CASE ROC MDL CSV MDL Difference City - Aldicarb City - Cyanide City - Oxamyl Base - Aldicarb Base - Cyanide Base - Oxamyl Fort - Aldicarb Fort - Cyanide Fort - Oxamyl

45 Minutes Detection Time Analysis Detection Time vs Injection Peak Value Minute 20 Minute Peak Trigger Value

46 Other Observations Dual Use: Detected a number of non security problems including dead ends pumping problems and contamination with aviation fuel. Could detect acceptably low levels in a matter of minutes Coverage adequate Communications simple and secure

47 Conclusion Water distribution system early warning systems are shown to be effective means of enhancing the security of base water supplies and the new ROC method is shown to be an effective way of validating these system with a tool that can be useful to operators in running and tuning thee systems.

48 Questions?