ADVANCES IN BIG DATA AND EXTREME SCALE COMPUTING ( BDEC ) William M. Tang

Size: px

Start display at page:

Download "ADVANCES IN BIG DATA AND EXTREME SCALE COMPUTING ( BDEC ) William M. Tang"

Alexandra Robertson
5 years ago
Views:

1 ADVANCES IN BIG DATA AND EXTREME SCALE COMPUTING ( BDEC ) William M. Tang Princeton Institute for Computational Science & Engineering (PICSciE) and Intel Parallel Computing Center (IPCC) Princeton University, Princeton, New Jersey Shanghai Jiao Tong University HPC Center and NVIDIA Center of Excellence Shanghai, China March 6, 2017

2 INTRODUCTION BDEC Challenges & Opportunities: Big Data Analytics & Statistical Machine Learning (ML)/Deep Learning (DL) Extreme Computing Discovery Science enabled by Extreme Scale Computing Big Data Goal à delivery of big-data-driven statistically-based ML/DL discoveryscience-capable predictive software Extreme Computing Goal à delivery of hypothesis-based discovery-science-capable software with good performance scaling, while demonstrating viable metrics on top supercomputing systems worldwide including portability, time to solution, & associated energy to solution Convergence Goal for BDEC à delivery of roadmap that plans for how exciting rapid advances in Big Data analytics and ML/DL predictive capabilities can intersect, influence, and potentially change national & international plans currently being laid out for achieving exascale computing.

Performance Development of HPC over the Last 24 Years from the Top500 11E+09 Eflop/s 100 Pflop/s 100000000 567 PFlop/s 93 PFlop/s 10 Pflop/s 10000000 1000000 1 Pflop/s 100 Tflop/s 100000 10 Tflop/s

3 Performance Development of HPC over the Last 24 Years from the Top500 11E+09 Eflop/s 100 Pflop/s PFlop/s 93 PFlop/s 10 Pflop/s Pflop/s 100 Tflop/s Tflop/s SUM N=1 286 TFlop/s 1 Tflop/s Gflop/s Gflop/s 10 1 Gflop/s Mflop/s TFlop/s 59.7 GFlop/s 400 MFlop/s N=

5 Applications Impact è Actual value of extreme Scale HPC to scientific domain applications & industry Context: US NATIONAL STRATEGIC COMPUTING INITIATIVE Practical Considerations: Better Buy-in from Science & Industry requires: - Moving beyond voracious (more of same - just bigger & faster) to transformational (achievement of major new levels of scientific understanding) - Improving experimental validation and verification to enhance realistic predictive capability of both hypothesis-driven and big-data-driven statistical approaches - Deliver software engineering tools to improve time to solution and energy to solution - David Keyes: (2015) à Billions of $ of scientific software worldwide hangs in the balance until better algorithms arrive to span the architecture-applications gap. Associated Challenges: - Hardware complexity: Heterogeneous multicore; gpu+cpu è Summit; mic+cpu è Aurora - Software challenges: Rewriting code focused on data locality Applications Imperative: Accountability aspect à Need to provide specific examples of impactful scientific and mission advances enabled by progress from terascale to petascale to today s multi-petascale HPC capabilities 5

6 CNN s MOONSHOTS for 21 st CENTURY HOSTED by FAREED ZAKARIA Five segments (broadcast in Spring, 2015 on CNN) exploring exciting futuristic endeavors in science & technology in the 21 st century (1) Human Mission to Mars (2) 3D Printing of a Human Heart (3) Creating a Star on Earth: Quest for Fusion Energy (4) Hypersonic Aviation (5) Mapping the Human Brain CNN Moonshots Series: Creating a Star on Earth à takes a fascinating look at how harnessing the energy of nuclear fusion reactions may create a virtually limitless energy source.

HPC SCIENCE APPLICATION DOMAIN: MAGNETIC FUSION ENERGY (MFE) plasma magnets magnetic field Tokamak Device ITER ~$25B facility located in France & involving 7 governments representing over half of

7 HPC SCIENCE APPLICATION DOMAIN: MAGNETIC FUSION ENERGY (MFE) plasma magnets magnetic field Tokamak Device ITER ~$25B facility located in France & involving 7 governments representing over half of world s population à dramatic next-step for Magnetic Fusion Energy (MFE) producing a sustained burning plasma -- Today: 10 MW(th) for 1 second with gain ~1 -- ITER: 500 MW(th) for >400 seconds with gain >10

8 BIG DATA ML/DL APPLICATION EXAMPLE: FUSION ENERGY SCIENCE Collaborative contributions from J. Kates-Harbeck, K. Felker, M. Parsons, E. Feibush, M. Churchill, Alexey Svyatkovskly, PPPL/Princeton University Most critical problem for MFE: avoid/mitigate large-scale major disruptions Approach: Use of big-data-driven statistical/machine-learning (ML) predictions for the occurrence of disruptions in EUROFUSION facility Joint European Torus (JET) Current Status: ~ 7 years of R&D results (led by JET) using Support Vector Machine (SVM) ML on zero-d time trace data executed on modern clusters yielding ~ reported success rates ranging from 80 up to 90% for JET, BUT > 95% with false alarm rate < 3% actually needed for ITER (Reference P. DeVries, et al., June 2015) Princeton Team Goals include: (i) improve physics fidelity via development of new ML multi-d, time-dependent software including better classifiers; (ii) develop portable (cross-machine) predictive software beyond JET to other devices and eventually ITER; and (iii) enhance execution speed of disruption analysis for very large datasets à development & deployment of advanced ML software via Support Vector Machine, Deep Learning/Recurrent Neural Networks,...

9 CLASSIFICATION Binary Classification Problem: Shots are Disruptive or Non-Disruptive Supervised ML techniques: Physics domain scientists combine knowledge base of observationally validated information with advanced statistical/ml predictive methods. Shots can be labeled D/ND retrospectively. Machine Learning (ML) Methods Engaged: Basic SVM approach initiated by JET team leading to APODIS software and later to DPFD PPPL; and New Deep Learning Recurrent Neural Net (DRNN) PPPL Approach: (i) examine appropriately normalized data; (ii) use training set to generate model; (iii) use trained model to classify new samples New multi-d data analysis will require new signal representations

10 Machine Learning Workflow Identify Signals Classifiers Feature Extraction Zero-D Signals Multi-D Signals Train Model with filters for missing/bad signals Use Model for Prediction Support Vector Machine (SVM) Deep Learning/ Recurrent Neural Nets Moving forward, PPPL team will focus on Deep Learning multi-dimensional (instead of present focus on zero-d time trace) signals e.g. radial temperature profile Many interesting possibilities for more informative, physics-motivated classifers with associated features Effective feature extraction is key challenge cannot simply feed raw data

11 JET Disruption Data # Shots Disruptive Nondisruptive Totals Carbon Wall Beryllium Wall (ILW) Totals JET produces ~ Terabyte (TB) of data per day Sample 7 Signals of zero-d time traces (07) Data Size (GB) Plasma Current 1.8 Mode Lock Amplitude 1.8 Plasma Density 7.8 Radiated Power 30.0 Total Input Power 3.0 d/dt Stored Diamagnetic Energy 2.9 Plasma Internal Inductance 3.0 ~55 GB data collected from each JET shot Well over 350 TB total amount with multidimensional data yet to be analyzed

12 Challenges & Opportunities Signal Normalization & Outlier Detection Put all signals on a reasonable numerical scale ~ O(1) Rescale signals from different machines such that the same meaning of the signal on the various machines gets mapped to the same numerical value after rescaling How? Physics-based (e.g. density divided by Greenwald Density) Data-based (e.g. all signals are divided by their standard deviation) Challenge: need to try many approaches and choose the best. Need rapid training time to iterate

13 DEEP LEARNING RECURRENT NEURAL NETS (RNN) APPROACH Julian Kates-Harbeck, CSGF Fellow from Harvard U. Rapid development of new GPU-compatible predictive software with results benchmarked vs. those from SVM analysis Very Promising Approach to Analysis of Higher Dimensional Signals via Deep Learning RNN 1D: (i) radial temperature profiles; (ii) density profiles; & (iii) radiation profiles Goals: -- Capture more physics and improve predictive capability -- Efficiently address challenges of more data and longer training time modern HPC training (e.g., via GPUs & MPI) progressing rapidly! -- Demonstrate how Neural Networks can extract salient features from higher-d data dimensional data automatically -- Demonstrate improvements in accuracy of ML predictions including harvesting new physical insights in timely way

Deep Recurrent Neural Networks (RNNs): Basic DescripNon Deep Hierarchical representation of complex data, building up salient features automatically Obviating the need for hand tuning, feature

14 Deep Recurrent Neural Networks (RNNs): Basic DescripNon Deep Hierarchical representation of complex data, building up salient features automatically Obviating the need for hand tuning, feature engineering, and feature selection Recurrent Natural notion of time and memory At every timestep, output depends on Last Internal state s(t-1) Recurrence! Current input x(t) The internal state can act as memory and accumulate information of what has happened in the past Internal State ( memory/ context ) Image adapted from: colah.github.io

15 New FRNN ( Fusion Recurrent Neural Net ) Code Performance Performance Tradeoff: Tune True Positives (good: correctly caught disruption) vs. False Positives (bad: safe shot incorrectly labeled disruptive). RNN Data: Testing 1200 shots from Jet ILW campaigns (C28-C30) True Posi8ves: 93.5% False Posi8ves: 7.5% True Posi8ves: 90.0% False Posi8ves: 5.0% All shots used, no signal filtering or removal of shots Jet SVM work: 990 shots from same campaigns Filtering of signals, ad hoc removal of shots with abnormal signals TP 90%, FP 1% Vega, Jesús, et al. "Results of the JET real-time disruption predictor in the ITER-like wall campaigns." Fusion Engineering and Design 88.6 (2013):

16 RNNs: HPC InnovaNons Engaged GPU training Neural networks use dense tensor manipulations, efficient use of GPU FLOPS Over 10x speedup better than multicore node training (CPU s) Distributed Training via MPI Linear scaling: Key benchmark of time to accuracy : we can train a model that achieves the same results nearly N times faster with N GPUs Scalable to 100s or 1000s of GPUs/ Leadership Class Facilities TBs of data or more Example: Best model training time on full dataset (~40GB, 4500 shots) of 0D signals training SVM (JET) : > 24hrs RNN ( 20 GPU s) : ~40min

17 RunNme

18 CURRENT PERSPECTIVE Forecasting disruptions using machine learning is an important application of a general idea: à Use multi outcome prediction to distinguish disruption types/scenarios à Eventually move from prediction to active control (including Reinforcement learning and Synthetic diagnostics) à Large and diverse data sets require us to build scalable systems to take advantage of leadership class computing facilities

19 Big Data Machine Learning Summary Fusion Energy Mission: -- Accelerate demonstration of the scientific & technical feasiblity of delivering Fusion Power -- Most critical associated problem is to avoid/mitigate large-scale major disruptions. ML Relevance to HPC: -- Rapid Advances on development of predictive methods via large-data-driven machinelearning statistical methods -- Approaches: (1) SVM s); and (2) Deep Learning/Recurrent Neural Nets (RNNs) -- Signifiance: Exciting alternative predictive approach to hypothesis-driven/first principles exascale predictive methods -- Complementarity: Exascale HPC needed to introduce/establish Supervised ML Classifiers with associated features Associated Challenge: Improvements over zero-d SVM-based machine-learning needed to achieve > 95% success rate, <5% false positives at least 30 ms before disruptions -- with portability of software to ITER via enhanced physics fidelity (capturing multi-d) with improvement in execution time enabled by access to advanced HPC hardware (e.g., large GPU systems).

20 HPC Focus in Current Presentation HPC Performance Scalability and Portability in an exascale-relevant grand challenge application domain (Fusion Energy Science) Emphasis à Rapid advances in Particle-in-Cell operations via scalable scientific software for extreme scale applications with FES as illustrative application domain Task à Deployment of innovative algorithms utilizing MPI & OpenMP, CUDA, and OpenACC within modern code that delivers new scientific insights on world-class systems à currently: Mira; Sequoia; K-Computer; Titan; Piz Daint; Blue Waters; Stampede;TH-2 & in near future on: Summit (via CAAR), Aurora (via ALCF ESP), Cori (NERSC)i, Stampede-II (TACC), Tsubame 3.0, (Japan) -----> TaihuLight (#1 Rated Supercomputer worldwide at Wuxi Supercomputing Center)) à Impressive recent progress on deployment of highly portable GTC-P PIC code on TaihuLight enabled by strong collaborative R&D between SJTU (HPC Center) and Princeton U (PICSciE) Reference - Stephen Wang, et al. best paper award at HPC China 2016

zeta 131 million grid points, 30 billion

21 Mathematics: 5D Gyrokinetic Vlasov-Poisson Equations Numerical Approach: Gyrokinetic Particle-in-Cell (PIC) Method theta 3D Torus theta radial zeta 131 million grid points, 30 billion particles, 10 thousand time steps Objective à Develop efficient numerical tool to realistically simulate turbulence and associated transport in magnetically-confined plasmas (e.g., tokamaks ) using high end supercomputers

22 Picture of Particle-in-Cell Method Charged particles sample distribution function Interactions occur on a grid with the forces determined by gradient of electrostatic potential (calculated from deposited charges) Grid resolution dictated by Debye length ( finite-sized particles) up to gyro-radius scale Specific PIC Operations: SCATTER, or deposit, charges as nearest neighbors on the grid Solve Poisson Equation for potential GATHER forces (gradient of potential) on each particle Move particles (PUSH) Repeat

23 ILLUSTRATION OF GTC-P CODE PORTABILITY Tianhe (51%) Titan (Cray XK7) (88%) Sequoia (BGQ) (100%) K-machine (41%) Mira (BGQ) (100%) Stampede 4096 (64%) Used nodes Unused nodes Piz Daint (Cray XC30) 4096 (78%) Peak performance (in peta-flops) Broad range of leading multi-pf supercomputers worldwide Percentage indicates fraction of overall nodes currently utilized for GTC-P experiments NOTE: Results in this figure are only for CPU nodes on Stampede and TH-2

24 Gyrokinetic PIC Code: six major subroutines à provides focus for Computer Science performance modeling Shift Push Charge Field Smooth Poisson Charge: particle to grid interpolation (SCATTER) Smooth/Poisson/Field: grid work (local stencil) Push: grid to particle interpolation (GATHER) update position and velocity Shift: in distributed memory environment, exchange particles among processors

25 wall clock time per ion step (s) smooth field poisson charge sort shift(pcie) shift push Mira Titan Piz Daint Titan Piz Daint CPU-only CPU+GPU Operational breakdown of time per step when using 80M grid points, 8B ions, and 8B kinetic electrons on 4K nodes of Mira, Titan, and Piz Daint.

ENERGY TO SOLUTION ESTIMATES (for Mira, Titan, and Piz Daint) Energy per ion time step (KWh) by each system/platform for the weakscaling, kinetic electron studies using 4K nodes.

26 ENERGY TO SOLUTION ESTIMATES (for Mira, Titan, and Piz Daint) Energy per ion time step (KWh) by each system/platform for the weakscaling, kinetic electron studies using 4K nodes. (Watts/node) * (#nodes) * (seconds per step) * (1KW/1000W) * (1hr/3600s) Power/Energy estimates obtained from system instrumentation including compute nodes, network, blades, AC to DC conversion, etc.

27 PERFORMANCE PORTABILITY vs. CODING COSTS (for kinetic electron simulations) Number of Lines of Code (LOC) modified provides quantitative measure of Level of Effort made to port and optimize GTC-P code to a specific architecture. -- considered pushe and sorte operations in GTC-P code -- speed-up measures: à GPU: single-node Kepler vs. single Sandybridge node à Xeon-Phi: single MIC vs. two Sandybridge nodes

28 APPLIED MATH LOCALITY CHALLENGE: GEOMETRIC HAMILTONIAN APPROACH TO SOLVING GENERALIZED VLASOV-MAXWELL EQUATIONS Hamiltonian à Lagrangian à Action à Variational Optimization à Discretized Symplectic Orbits for Particle Motion I. Ultrahigh Performance 3-Dimensional Electromagnetic Relativistic Kinetic Plasma Simulation -- Kevin J. Bowers, et al., Phys. Plasmas 15, (2008) è Basic foundation for symplectic integration of particle orbits in electromagnetic fields without frequency ordering constraints è Foundational approach for present-day simulations of laser-plasma interactions on modern supercomputing systems VPIC (data locality focus) major success story on Roadrunner (1st PF system) è Limited applicabiity with respect to size of simulation region and geometric complexity II. Geometric Gyrokinetic Theory for Edge Plasmas Hong Qin, et al., Phys. Plasmas 14, (2007) è Basic foundation for symplectic integration of particle orbits in electromagnetic lowfrequency plasma following GK ordering è Still outstanding challenge: Address reformulation of non-local Poisson Equations structure for electromagnetic field solve

29 Future Implications Demands for increased physics fidelity: Ø ITER-scale runs at the spatial resolution and temporal duration required, including complete electron dynamics Ø Capabilities to encompass electromagnetic physics, including faster and more portable multi-grid Poisson solvers (e.g., Challenges & Promise for performance modeling optimizations for PIC codes in general Ø Ø Asymptotically decreasing local memory with highly localized communication networks Addressing OpenMP4.5 (IPCC focus) & Open ACC2.0 (TaihuLight!) challenges Node and Network architecture Ø Ø PIC simulations with flop:byte ~ 1 require high on-node memory bandwidth For kinetic electron dynamics, inter-node communication begin dominating execution time. à network performance and software implications (e.g., MPI libraries) play significant role for the overall PIC code performance. Energy-efficient scientific computing Ø Ø Today, most computer centers provide little or no information on energy and power to the users at the end of an application Reporting energy by components (memory, processor, network, storage, etc) would enable scientists and vendors to help co-design their application to avoid energy hotspots and produce more energy-efficient systems.

30 SUMMARY I. PRESENTATION FOCUS: HPC Performance Scalability and Portability in a representative application domain à Illustration of domain application that delivers discovery science with good performance scaling, while also helping provide viable metrics on top supercomputing systems such as portability, time to solution, & associated energy to solution II. HPC APPLICATION DOMAIN: Fusion Energy Science References: (i) Scientific Discovery in Fusion Plasma Turbulence Extreme Scale; W. Tang, B. Wang, S. Ethier, Computing in Science and Engineering (CiSE), vol. 16. Issue 5, pp.44-52, 2014; (ii) SC 16 Technical Paper III. CURRENT PROGRESS: Deployment of innovative algorithms: MPI & OpenMP, CUDA, with active OpenACC and OpenMP4.5 R&D within modern code that delivers new scientific insights on world-class systems à currently: Mira; Sequoia; K-Computer; Titan; Piz Daint; Blue Waters; Stampede;TH-2; Tsubame 2.5; & in future on: Sunway TaihuLight, Cori, Stampede-II, Tsubame 3.0, Summit (via CAAR), Aurora (via ESP),. IV. FUTURE CHALLENGES: need algorithmic & solver advances further improving data-locality -- enabled by Applied Mathematics in an interdisciplinary Co-Design type environment together with Computer Science & Extreme-Scale HPC Domain Applications

Extreme Scale Computational Science Challenges in Fusion Energy Research

Extreme Scale Computational Science Challenges in Fusion Energy Research William M. Tang Princeton University, Plasma Physics Laboratory Princeton, NJ USA International Advanced Research 2012 Workshop