Optmzng a System of Threshold-based Sensors wth Applcaton to Bosurvellance Ronald D. Frcker, Jr. Thrd Annual Quanttatve Methods n Defense and Natonal Securty Conference May 28, 2008
What s Bosurvellance? Homeland Securty Presdental Drectve HSPD-21 (October 18, 2007): The term bosurvellance means the process of actve datagatherng of bosphere data n order to acheve early warnng of health threats, early detecton of health events, and overall stuatonal awareness of dsease actvty. [1] The Secretary of Health and Human Servces shall establsh an operatonal natonal epdemologc survellance system for human health... [1] Epdemologc survellance: survellance usng health-related data that precede dagnoss and sgnal a suffcent probablty of a case or an outbreak to warrant further publc health response. [2] [1] www.whtehouse.gov/news/releases/2007/10/20071018-10.html [2] CDC (www.cdc.gov/epo/dphs/syndromc.htm, accessed 5/29/07) 2
An Exstng System: BoSense
Thnk of It Lke a Large System of Sensors Hosptal X Hosptal W Hosptal Y Hosptal V Hosptal Z Issue: False alarms a serous problem most health montors learned to gnore alarms trggered by ther system. Ths s due to the excessve false alarm rate that s typcal of most systems - there s nearly an alarm every day! [1] [1] https://wk.crg.washngton.edu/pub/bn/vew/isds/survellancesystemsinpractce 4
The Problem n Summary Goal: Early detecton of dsease outbreak and/or boterrorsm Issue: Currently detecton thresholds set navely Equally for all sensors Ignores dfferental probablty of attack Result: Hgh false alarm rates Loss of credblty 5
Formal Descrpton of the System Let X t denote the output from sensor at tme t, =1,,n, t=1,2, Each sensor / locaton has a probablty of outbreak / attack: p,..., 1 p, 1 n p If no event of nterest anywhere n the network, X t ~F 0 for all and t If an event of nterest occurs at tme t, X t ~F 1 for exactly one A sgnal s generated at tme t* when X h for one or more t * 6
Idea of Threshold Detecton Dstrbuton of Background Dsease Incdence (f 0 ) Dstrbuton of Background Incdence and Attack/Outbreak (f 1 ) Probablty of a true sgnal: f 1( x ) dx 1 F 1( h ) xh Probablty of a false sgnal: f x dx F h x h 0 0 ( ) 1 ( ) h 7
Pr(sgnal attack) It s All About Choosng Thresholds For each sensor, choce of h s compromse between probablty of true and false sgnals ROC Curve No Attack/ Outbreak Dstrbuton Threshold (h) Pr(sgnal no attack) Attack/ Outbreak Dstrbuton 8
Mathematcal Formulaton of the Problem It s smple to wrte out: Pr(detecton) Pr(sgnal attack) Pr(attack) E(# false sgnals) Pr(sgnal no attack) Express t as an NLP optmzaton problem: max 1 F ( h ) h s.t. 1 F ( h) 0 1 p 9
Some Assumptons Sensors are spatally ndependent Montorng standardzed resduals from an adaptve regresson model Model accounts for (and removes) systematc effects n the data Result: Reasonable to assume F 0 =N(0,1) An attack wll result n a 2-sgma ncrease n the mean of the resduals Result: F 1 =N(2,1) Then, NLP s: mn ( h 2) p h s.t. ( h) n 10
Ten Sensor Example 11
Smplfyng to a One-dmensonal Optmzaton Problem System of n hosptals (sensors) means optmzaton has n free parameters Hard for to solve for large systems Can smplfy to one-parameter problem: Theorem: For F 0 =N(0,1) and F 1 =N(g,1), the optmzaton smplfes to fndng m to satsfy n 1 m ln( p ) n, 1 g and the optmal thresholds are then 1 h m ln( p ). g 12
Consder (Hypothetcal) System to Montor 200 Largest Ctes n US Assume probablty of attack s proportonal to the populaton n a cty: p m / m 13
Optmal Soluton for 200 Ctes Assume 2σ magntude event Constrant of 1 false sgnal system-wde / day Populaton Pr(attack) Threshold Pr(sgnal attack) Pr(sgnal no attack) Result: Pr(sgnal attack) = 0.388 Naïve result: Pr(sgnal attack) = 0.283 14
P d False Alarm Trade-Off () 15
Choosng g and Optmal probablty of detecton for varous choces of g and Choce of depends on avalable resources Settng g s subjectve: what sze mean ncrease mportant to detect? 16
Senstvty Analyses Optmal probablty of detecton Actual probablty of detecton 17
Optmzng a County-level System 18
Thresholds as a Functon of Probablty of Attack 19
Take-Aways BoSense and other bosurvellance systems performance can be mproved now at no cost Approach allows for customzaton E.g., ncrease n probablty of detecton at ndvdual locaton or add addtonal constrant to mnmze false sgnals Apples to other sensor system applcatons: Port survellance, radaton/chem detecton systems, etc. Detals n Frcker and Banschbach (2007) 20
Future Research Drectons Assess data fuson technques for use when multple sensors n each regon I.e., relax sensor (spatal) ndependence assumpton Generalze from threshold detecton methods to other methods that use hstorcal nformaton I.e., relax temporal ndependence assumpton 21
Selected References Background Informaton: Frcker, R.D., Jr., and H. Rolka, Protectng Aganst Bologcal Terrorsm: Statstcal Issues n Electronc Bosurvellance, Chance, 91, pp. 4-13, 2006 Frcker, R.D., Jr., Syndromc Survellance, Encyclopeda of Quanttatve Rsk Assessment (to appear). Selected Research: Frcker, R.D., Jr., and D. Banschbach, Optmzng a System of Threshold Detecton Sensors, n submsson to Operatons Research. Frcker, R.D., Jr., and J.T. Chang, A Spato-temporal Method for Real-tme Bosurvellance, Qualty Engneerng, (to appear). Frcker, R.D., Jr., Kntt, M.C., and C.X. Hu, Comparng Drectonally Senstve MCUSUM and MEWMA Procedures wth Applcaton to Bosurvellance, Qualty Engneerng (to appear). Joner, M.D., Jr., Woodall, W.H., Reynolds, M.R., Jr., and R.D. Frcker, Jr., A One-Sded MEWMA Chart for Health Survellance, Qualty and Relablty Engneerng Internatonal (to appear). Frcker, R.D., Jr., Hegler, B.L., and D.A Dunfee, Assessng the Performance of the Early Aberraton Reportng System (EARS) Syndromc Survellance Algorthms, Statstcs n Medcne, 2008. Frcker, R.D., Jr., Drectonally Senstve Multvarate Statstcal Process Control Methods wth Applcaton to Syndromc Survellance, Advances n Dsease Survellance, 3:1, 2007. 22