PMU Big Data Analysis Based on the SPARK Machine Learning Framework

Transcription

1 PNNL-SA PMU Big Data Analysis Based on the SPARK Machine Learning Framework Pavel Etingov WECC Joint Synchronized Information Subcommittee meeting May , Salt Lake City, UT May 18,

2 Project team PNNL Jason Hou Dimitri Zarzhitsky Heng Wang Huiying Ren Pavel Etingov Partners LANL LBNL Columbia University Jacques De Chalendar (summer intern, Stanford University) May 18,

3 Project background Active deployment of phasor measurement units (PMUs), smart meters, and other measurement devices dramatically increased the size of data collected by electrical utilities. The volume of the collected information is continuing to grow that makes it very difficult to process it, run analysis and extract insights The collected data enables many insights about power system state and dynamic behavior Extracting this information can help: Increase situation awareness. Detect events in the system (e.g. under frequency or voltage events). Detect abnormalities. Help validate the system models. Detect equipment malfunction. May 18,

4 Project goals Develop a framework for PMU big data analysis Event detection Abnormalities detection Improved situational awareness System identification (learning system dynamic behavior) Advanced visualization Framework is based on the cloud technology and distributed computing: PNNL institutional cloud system or Microsoft Azure Apache SPARK for distributed big data analysis and Machine Learning (ML) May 18,

5 PNNL cluster infrastructure PNNL cloud is based on OpenStack (a free and opensource software platform for cloud computing) Cloudera Apache Hadoop Distribution: Apache Spark (an open source cluster computing framework) Apache Hive (a data warehouse infrastructure built on top of Hadoop for providing data summarization, query, and analysis) HBase (an open source, non-relational, distributed database) May 18,

6 Apache Spark Large scale parallel data processing framework Extremely powerful (up to 100x faster than Hadoop) Large datasets distributed across multiple nodes within a computer cluster Support real time data stream Built-in Machine Learning library Support different languages (Scala, Java, Python, R) Support different data sources (SQL, Hive, HBase, Cassandra, Oracle, etc) Open source and free Available through public cloud services (Amazon AWS, Microsoft Azure, IBM, etc) and through new PNNL institutional cloud system. May 18,

7 Spark research cluster based on PNNL cloud Current configuration 20 nodes RAM 320 Gb Recently upgraded to Spark 2.1 Cluster will be upgraded to 1 Tb RAM May 18,

8 Cloud based ML-PMU Framework PMU data stream SPARK cluster PySpark pdat extract PySpark Event detection PNNL EIOC PDAT Historical data.csv,.xml DB (HDFS) SPARK (head node) PySpark interface Node 1 Node 2 Node X Python Python Python User Input Display EIOC - Electricity Infrastructure Operations Center HDFS- Hadoop Distributed File System WEB based GUI May 18,

9 PMU data stream PNNL receives PMU data stream from Bonneville Power Administration 12 PMUs Multiple channels (Voltage and Current Phasors, Frequency, ROCOF) PMU Data stored in PDAT format PDAT format developed by BPA Based on IEEE Std. C Binary files Each file contains 1 minute of data One file ~ 5 MB Data frame organization defined by IEEE C May 18,

10 SPARK machine learning built in capabilities SPARK includes two ML libraries: Mllib (contains the original API built on top of RDDs*) ML (provides higher-level API built on top of Data Frames) Python or R libraries for ML can be deployed and used on single nodes *Resilient Distributed Dataset May 18,

11 Ongoing work Python (PySpark) modules: PDAT data extraction Data cleaning Bad data Missing points Outliers Event detection Frequency events Voltage events Features extraction and analysis Wavelet K-mean Clustering FFT Screenshot examples (Jupiter notebooks integration with PySpark) May 18,

12 PDAT data extraction Read information from PDAT and creates SPARK data frames Store information in Hive tables Implemented in PySpark that allows parallel processing of multiple PDAT files PNNL EIOC PMU data stream PDAT HDFS DB (HDFS) PDAT extract Spark Data frames Hive Tables Significantly increased performance To read information for 1 hour takes about 20 seconds (20 nodes cluster) May 18,

13 PMU Anomaly Detection Multi-resolution analysis (MRA) Signals are transformed into timefrequency domain Position and magnitude of anomalies are localized at orthonormal scales Wavelet series expansion jk x x k j/2 j, ( ) 2 (2 ) jk x x k j/2 j, ( ) 2 (2 ) f ( x) f ( x) f ( x) a d 1/2 ( x) h ( n)2 (2 x n) n 1/2 ( x) h ( n)2 (2 x n) n f ( x) c ( k) ( x) d ( k) ( x) j0 j0, k j j, k k j j0 k Lowpass to get approximation(a) Highpass to get details(d) MRA decomposition tree Extracted the detailed wavelet coefficients at the first 3 resolution levels which are adequate given the event duration. Identified anomaly candidates at multiresolution levels with moving-window-based outlier detection. Prescreened the anomaly candidates with a minimum duration of 3 seconds. Anomaly Detection & Classification Framework May 18,

14 Anomaly Scoring and Verification The anomaly score matrices were calculated across 12 PMUs at multiresolution levels for each PMU attribute. Frequency event detection and classification False alarm (bad data) weak spatiotemporal correlation (weak continuity, weak correlation across units, but note local event is possible) Real event strong correlation across units, relatively weak temporal correlation with a bandwidth of about 5~20seconds Clustering and Principal Component analysis(pca) were applied to obtain better understanding of the anomaly May 18, events (a) (b) (c) (d) (a) (b) (c) (d) PMU frequency attribute with observed real event marked in red; MRA wavelet coefficient at D1; MRA wavelet coefficient at D2; MRA wavelet coefficient at D3. The PMU anomaly candidates marked in red at each multiresolution level.

15 Examples of Voltage Detected Anomalies An example of detected PMU voltage magnitude anomaly where the PMUs have consistent behaviors and strong crosscorrelations. An example of localized PMU voltage magnitude anomaly where PMU4 has different behavior compared to other PMUs, and the cross-corrections are relatively weak. May 18,

16 Examples of Anomalies Classification (a) (b) (c) Classification results using (a) all 4 attributes; (b) 3 attributes (Magnitude, Frequency and angle difference); (c) 2 attributes (Magnitude and Frequency) The recorded real frequency events were marked in cyan (a) Time(sec) Time(sec) Time(sec) (b) (c) PCA results using (a) all 4 attributes; (b) 3 attributes (Magnitude, Frequency and angle difference); (c) 2 attributes (Magnitude and Frequency) The recorded real frequency events were marked in cyan May 18,

17 Preliminary Results Spark cluster for ML and PMU (big data) analysis was deployed. It is based on the PNNL institution cloud system PMU data has been collecting in PDAT format (PMU data stream from PBA to PNNL EIOC) Methodology for event detection based on wavelet analysis has been developed Python (PySpark) modules are under development May 18,