Dynamic Data-Driven Adaptive Sampling and Monitoring of Big Spatial-Temporal Data Streams for Real-Time Solar Flare Detection Dr. Kaibo Liu Department of Industrial and Systems Engineering University of Wisconsin-Madison Date: 8/8/2017 1
Outline Motivation State of the art Proposed DDDAS framework Data-Driven Dynamic Sampling Strategy Case study Conclusion 2
Motivation With the advancement of sensing technique and data collection capability, Big Data Streams have become widely available in many DoD applications. This provides an unprecedented opportunity to gain system-wide situational awareness through real-time anomaly detection and fault localization. The emerging NASA Solar Dynamics Observatory (SDO) continuously monitors the dynamic solar activities for 24 hours/7 days a week Solar flare detection generate a high resolution image every 0.75 second produce 1.5 TB big data per day 3 Source: NASA
Applications and Importance The solar flare activities have a close relationship with Air Force equipment and applications. significantly affect Earth s ionosphere, causing hourslong disruptions in radio communications affect GPS receivers and satellites, making it very difficult for search and rescue in a war zone lead to failures in large-scale power-grid with cascading effects Real-time detection system for the solar flare by exploiting the Big Data Streams of solar images is highly desired. 4
Challenges Big Data Streams place critical requirements and resources constraints for data communication and processing in real time Send only 6 images back every minute for real-time analysis given transmissions rate 130 million bits/second The occurrence of solar flare is naturally complicated (depends on the cycle and the inherent dynamics and randomness of solar activities) sparse (with a small signal-to-noise ratio (SNR)) transient (only lasts for minutes and hard to predict) Currently, there is a lack of efficient online monitoring scheme tailored to these unique characteristics. 5
Proposed DDDAS Framework Original Solar Image Update Model Updated Solar Image A dynamically updated spatial-temporal statistical model fully characterize the changing background (a) Applications Sample data SPC Chart t (b) Applications modeling DDDAS Update sampling Framework Dynamic Sampling A dynamic sampling algorithm that actively decides which data streams to observe given the resources constraints 6 (d) Mathematical and statistical algorithms Update SPC (c) Application measurement systems and methods t A scalable and robust SPC to effectively combine the information from significant data streams to produce an overall global monitoring system
State of the art and innovative idea Existing approaches to process monitoring focus on fixed sub-region (rigid spatial domain) assume that the locations of anomaly event are known fail to capture the dynamic features of solar flare events sample whole image at fixed frequency (rigid temporal domain) result in a large detection delay or miss the event Adaptive sampling strategy : require large amount of historical information Top-r based Adaptive Sampling (TRAS) at DDDAS 2016 Innovative idea: blue: sampled data streams red: anomaly regions black: overlapping random sampling fixed sampling 7
Problem formulation and objective p = m n data streams At time t, Y t = Y 1,t,, Y p,t When t < τ, the process is in control, Y k,t ~ i.i.d N(0,1) When t > τ, the process is out-ofcontrol, the mean of data streams in an unknown region C shifts to δ: Y k,t ~ i.i.d N δχ C x k, 1 m Y 1,t Y 2,t C n Y k,t Y p,t At each time t, only q out of p variables can be observed. kth data stream Location x k Observation Y k,t Change point τ Clustered shift Limited resources Goal: Based on dynamic observations in real time, actively decide which data stream to observe at the next time for quick detection of anomaly event while still maintaining a system-wide false alarm rate. 8
General strategy 1 Use local statistics To propose a scalable monitoring scheme, we define two local statistics W 1 (2) kt, W kt for each stream to describe the likelihood of upward or downward shift n Y 1,t Y 2,t m Y k,t Local statistics (1) : the likelihood of upward shift W kt W kt (2) : the likelihood of downward shift Y p,t 9
General strategy 2 Allocate observed data streams adaptively Wide search strategy In-control: observed data streams should be placed evenly to observe the shifted data stream rapidly. Deep search strategy Out-of-control: observed data streams should accumulate in suspected region. To combine these two strategies, we separate the observable variables into two groups: q W,t W-observations and q D,t D-observations. The proportion of q W,t and q D,t is adjusted adaptively. 10
General strategy 2 Allocate observed data streams adaptively Combination of two strategies q W,t = 6 W-observations q D,t = 4 D-observations To combine these two strategies, we separate the observable variables into two groups: q W,t W-observations and q D,t D-observations. The proportion of q W,t and q D,t is adjusted adaptively. 11
The proposed Spatial Adaptive Sampling and Monitoring (SASAM) algorithm Begin Three Questions q D,1 = 0 q W,t = q q D,t Allocate W-observations [i] [i] How to allocate W-observations? t t + 1 Update local statistics and charting statistics [ii] Charting statistics > H? Determine q D,t+1 ; Allocate D-observations for the next period [iii] N Y Alarm [ii] How to update local statistics? [iii] How to determine the number and location of D-observations? 12
[i] Allocate W-observations Objective: select O W,t = q W,t observations with good space filling property with diversity from the previous selection Method Generate I candidate Latin hyper-cube designs. For each i = 1,, I, calculate An example, O W,t =10 m 1,i = min x k O W,t x j O W,t d x k, x j [measuring space filling property] m 2,i = min x k O W,t x j O W,t 1 d x k, x j [measuring diversity from the previous design] Pick the candidate selection with the maximum m i = m 1,i + m 2,i index. 13
[ii] Update Local statistics Observations at time t: Y j,t, j O W,t O D,t Conventional CUSUM chart: (1) 1 W k,t = Wk,t 1 (2) 2 W k,t = Wk,t 1 + u min Y k,t u 2 min 2 + u min Y k,t u 2 min 2 the kth local statistics indicating +/- shift + + 14
[ii] Update Local statistics Observations at time t: Y j,t, j O W,t O D,t Conventional CUSUM chart: (1) 1 W k,t = Wk,t 1 (2) 2 W k,t = Wk,t 1 + u min Y k,t u 2 min 2 + u min Y k,t u 2 min 2 the kth local statistics indicating +/- shift + + With Spatial consideration: (1) 1 W k,t = Wk,t 1 + K h x k x j j O W,t O D,t K h ( ): influence function u min Y k,t 2 u min 2 2 (1) 1 u min W k,t = Wk,t 1 + K h x k x j u min Y k,t 2 j O W,t O D,t + Clustered shift Nearby data streams have similar likelihood of shift + (1) (2) W k,t = max W k,t, Wk,t Charting statistic: S t = max W k,t 1 k p Describe the likelihood that the most suspected data stream has a shift. 15
[iii] Allocate D-observations at next time point Where to allocate? Nearest neighbors of the most suspected stream local stats. Most suspected data stream 16
[iii] Allocate D-observations Where to allocate? Nearest neighbors of the most suspected stream How many? More D-observations for higher chance of shift Similar to variable sample size charts S t local stats. # of D-obs for time t + 1 q D,t+1 = f θ (S t ) The charting statistic An increasing function described by some parameter θ. We apply the linear function. Most suspected data stream q D,t+1 = 6 Li and Qiu (2014) used this technique on variable sample interval chart 17
Summary Begin q D,1 = 0 Generate a number of candidate samples, and select the best one in terms of space filling property. q W,t = q q D,t Allocate W-observations [i] Update local statistics with all observations of nearby variables using a kernel function. t t + 1 Update local statistics and charting statistics [ii] Charting statistics > H? Y Alarm New D-observations are selected around the most suspected stream; the number is determined by the charting statistics. Determine q D,t+1 ; Allocate D-observations for the next period [iii] N 18
A real case study - solar flare detection Dataset: p = 232 292 = 67744 dimensional variables; q = 500 observable pixels (~0.7% available); One solar flare starts from frame 87. Both charts are adjusted to have the same IC-ARL. Results: TRAS triggers alarm at t = 95. SASAM triggers an alarm at t = 91. TRAS SASAM 19
Video clip 20
Video clip 21
Result Observable stream Top-10 observable stream in TRAS method TRAS SASAM Time 91 The SASAM triggers alarm Time 95 The TRAS triggers alarm 22
Summary of the proposed sampling strategy A systematic adaptive sampling strategy is proposed for real-time monitoring of Big Data streams with dynamically selected partial information. Adaptability: Integrate two sampling strategies (wide or deep) Quickly detect a wide range of possible changes with no prior knowledge of the potential anomaly events by adaptively adjusting to the event locations; Actively select the data streams to observe from the whole streaming data to maximize the sensitivity for anomaly detection with consideration of resource constraints. 23
Proposed DDDAS new ideas Applications modeling Application measurement systems and methods Mathematical and statistical algorithms Objectives Existing Approaches Proposed New Methodology Either only capture spatial or temporal characteristics Establish a spatialtemporal statistical model for capturing the changing baseline Establish an effective sampling strategy to decide which data streams to observe Establish a scalable and robust SPC scheme to maximize change detection capability Extended models are rigid and limited to specific applications Require full observations Sampling over fixed subregions (rigid spatial domain) or whole data frames (rigid temporal domain) Search for all possible fault scenarios Require all historical data Assume potential fault scenarios are known High computational costs Require full observations Capture both domains by integration of graphical models with matrix factorization Generic transfer learning framework for adaptive learning Allow dynamic partial observations Dynamic sampling partial data streams over the spatial domain at each acquisition time based on resources constraints Automatically identify and localize fault scenarios Require current observations and a summary statistic Assume potential fault scenarios are unknown Only linear complexity in the number of data streams Allow dynamic partial observations 24
Conclusion and Impact to Air Force It is critically important for the Air Force to make rapid decisions in a battlefield based on Big Data continuously collected from massive sensors in real time. (a) Intrusion detection (b) unmanned vehicle surveillance (c) Cybersecurity 25 Early detection and localization of these anomaly events will enhance system-wide situational awareness to support warfighters/military operations, prevent damages, reduce cost, improve efficiency, and save billions of lives. Snapshots of the temperature profile from Climate Institute at ORNL
Thank you for coming! Questions? 26