Computational Simulations of The World s Biggest Eye on GPUs

Transcription

1 Computational Simulations of The World s Biggest Eye on GPUs Hatem Ltaief Extreme Computing Research Center King Abdullah University of Science and Technology, Saudi Arabia NVIDIA GTC at San Jose, CA April 7, 06 H. Ltaief MOAO on Multiple GPUs / 6

2 Outline Motivations:Task-based Programming Model Application: The European Extremely Large Telescope Parallel Implementation 4 Performance Results 5 Conclusion H. Ltaief MOAO on Multiple GPUs / 6

3 Acknowledgments Students/Collaborators/Support Extreme Computing Research KAUST A. Charara and D. Keyes L Observatoire de Paris, LESIA R. Dembet, N. Doucet, E. Gendron, D. Gratadour, C. Morely, A. Sevin and F. Vidal Innovative Computing UTK PLASMA/MAGMA Team Barcelona Supercomputing Center, Spain R. Badia, P. Bellens, X. Martorell, G. Miranda and S. Zhuang INRIA/INP Bordeaux, France E. Agullo, M. Faverge, F. Pruvost, M. Sergent and S. Thibault KAUST IT Research Computing support NVIDIA hardware donations (can we get more? ) Funded partially by the French National Center for Scientific Research (CNRS, 06) For free H. Ltaief MOAO on Multiple GPUs / 6

4 Task Model Outline Motivations:Task-based Programming Model Application: The European Extremely Large Telescope Parallel Implementation 4 Performance Results 5 Conclusion H. Ltaief MOAO on Multiple GPUs 4 / 6

5 Task Model One of The Possible Solutions for Exascale Computing Fine-granularity computation Asynchronous execution High concurrency Runtime system for separation of concerns Productivity with abstraction Performance/debugging tools Popular: available from OpenMP.0 H. Ltaief MOAO on Multiple GPUs 5 / 6

6 Task Model General Procedure Taskify the application. This may require: Implementing a new algorithm from scratch Performing more flops at the end of the day Increasing code size Schedule the generated tasks. This may require a runtime system featuring: Static/dynamic scheduling Shared/Distributed memory systems NUMA-aware Heterogeneous architecture (x86+accelerators) H. Ltaief MOAO on Multiple GPUs 6 / 6

7 Task Model Many Success Stories Dense Linear Algebra (PLASMA/MAGMA/PaRSEC, FLAME). Fast Multipole Method (Agullo et al., SIAM SISC 04). Sparse iterative solvers (Ghysels et al., SIAM SISC 0). Parareal (Elwasif et al., JCP 0). At Petascale, the task-based Uintah framework (Meng et al. SC ) handled various sparse applications: Fluid-structure interaction with adaptive mesh refinement. Radiation modeling through raytracing. Turbulent combustion on a fixed-mesh requiring large-scale. linear solves. H. Ltaief MOAO on Multiple GPUs 7 / 6

8 E-ELT Outline Motivations:Task-based Programming Model Application: The European Extremely Large Telescope Parallel Implementation 4 Performance Results 5 Conclusion H. Ltaief MOAO on Multiple GPUs 8 / 6

9 E-ELT The Top 0 (present and future) Ground-based Telescopes H. Ltaief MOAO on Multiple GPUs 9 / 6

10 E-ELT The Top 0 (present and future) Ground-based Telescopes Rank Name Location Diameter Cost Year 0 Large Synoptic Survey Telescope (LSST) Chile 8.4m 450 million 04 9 South African Large Telescope (SALT) South Africa 9.m 6 million Keck USA 0m 00 million Gran Telescopio Canarias (GTC) Spain 0.4m 0 million Aricebo Observatory Puerto Rico 05m 9. million 96 H. Ltaief MOAO on Multiple GPUs 9 / 6

11 E-ELT The Top 0 (present and future) Ground-based Telescopes Rank Name Location Diameter Cost Year 0 Large Synoptic Survey Telescope (LSST) Chile 8.4m 450 million 04 9 South African Large Telescope (SALT) South Africa 9.m 6 million Keck USA 0m 00 million Gran Telescopio Canarias (GTC) Spain 0.4m 0 million Aricebo Observatory Puerto Rico 05m 9. million 96 5 Atacama Large Millimeter Array (ALMA) Chile m.4 billion 0 4 Giant Magellan Telescope (GMT) Chile 4.5m. billion 04 Thirty Meter Telescope (TMT) USA 0m.4 billion 00 Square Kilometer Array (SKA) Australia 90m billion 00 European Extremely Large Telescope (E-ELT) Chile 9m. billion 04 H. Ltaief MOAO on Multiple GPUs 9 / 6

12 E-ELT The Top 0 (present and future) Ground-based Telescopes Rank Name Location Diameter Cost Year 0 Large Synoptic Survey Telescope (LSST) Chile 8.4m 450 million 04 9 South African Large Telescope (SALT) South Africa 9.m 6 million Keck USA 0m 00 million Gran Telescopio Canarias (GTC) Spain 0.4m 0 million Aricebo Observatory Puerto Rico 05m 9. million 96 5 Atacama Large Millimeter Array (ALMA) Chile m.4 billion 0 4 Giant Magellan Telescope (GMT) Chile 4.5m. billion 04 Thirty Meter Telescope (TMT) USA 0m.4 billion 00 Square Kilometer Array (SKA) Australia 90m billion 00 European Extremely Large Telescope (E-ELT) Chile 9m. billion 04 Consortium: multiple nation initiatives Src: H. Ltaief MOAO on Multiple GPUs 9 / 6

13 E-ELT The World s Biggest Eye on The Sky Credits: ESO ( H. Ltaief MOAO on Multiple GPUs 0 / 6

14 E-ELT The World s Biggest Eye on The Sky Credits: ESO ( H. Ltaief MOAO on Multiple GPUs / 6

15 E-ELT The largest optical/near-infrared telescope in the world. A highest priority in ground-based astronomy. It weighs about 700 tons. The main mirror diameter is 9m. Location in Chile, South America. E-ELT does not exist yet, expected early 00s (delayed: 04). H. Ltaief MOAO on Multiple GPUs / 6

16 E-ELT Multi-object Adaptive Optics (MOAO) Probably the most challenging embedded instrument in the E-ELT. Observe/understand the evolution of a number of the most distant galaxies in parallel. Capable of exploiting the Field of View (FoV) of 7 to 0 arcminutes. It is used on telescopes to compensate, in real-time, for the effect of atmospheric turbulence, providing a significant improvement in resolution and in return, to adjust the deformable mirrors that compensate for these distortions before sending the light to the science instrument. H. Ltaief MOAO on Multiple GPUs / 6

17 E-ELT Multi-object Adaptive Optics (MOAO) Good news! H. Ltaief MOAO on Multiple GPUs 4 / 6

18 E-ELT Multi-object Adaptive Optics (MOAO) Good news! Extremely compute intensive. H. Ltaief MOAO on Multiple GPUs 4 / 6

19 E-ELT Global Workflow Chart ToR computation System Parameters Cmm Ctm ToR R Turbulence Parameters Cmm Ctm Observing sequence Ctt BLAS Cee BLAS Cvv Intersample H. Ltaief MOAO on Multiple GPUs 5 / 6

20 E-ELT Global Workflow Chart ToR computation System Parameters Cmm Ctm ToR R Turbulence Parameters Cmm Ctm Observing sequence Ctt BLAS Cee BLAS Cvv Intersample H. Ltaief MOAO on Multiple GPUs 6 / 6

21 E-ELT Global Workflow Chart ToR computation System Parameters Cmm Ctm ToR R Turbulence Parameters Cmm Ctm Observing sequence Ctt BLAS Cee BLAS Cvv Intersample H. Ltaief MOAO on Multiple GPUs 7 / 6

22 Parallel Implementation Outline Motivations:Task-based Programming Model Application: The European Extremely Large Telescope Parallel Implementation 4 Performance Results 5 Conclusion H. Ltaief MOAO on Multiple GPUs 8 / 6

23 Parallel Implementation Blocked Algorithms: Fork-Join Paradigm H. Ltaief MOAO on Multiple GPUs 9 / 6

24 Parallel Implementation LAPACK/MAGMA: Blocked Algorithms Principles: Panel-Update Sequence Transformations are blocked/accumulated within the Panel (Level BLAS) Transformations applied at once on the trailing submatrix (Level BLAS) Parallelism hidden inside the BLAS Fork-join Model A broken model! H. Ltaief MOAO on Multiple GPUs 0 / 6

25 Parallel Implementation Tile Data Layout Format LAPACK: column-major format PLASMA: tile format H. Ltaief MOAO on Multiple GPUs / 6

26 Parallel Implementation PLASMA: Tile Algorithms PLASMA: Parallel Linear Algebra for Scalable Multi-core Architectures = Break the bulk synchronous programming model Parallelism is brought to the fore May require the redesign of linear algebra algorithms Tile data layout translation Remove unnecessary synchronization points between Panel-Update sequences DAG execution where nodes represent tasks and edges define dependencies between them Default dynamic runtime system environment QUARK (but could use StarPU, PaRSEC, OmpSs, Uintah etc.) H. Ltaief MOAO on Multiple GPUs / 6

27 Parallel Implementation The Nanos++ infrastructure H. Ltaief MOAO on Multiple GPUs / 6

28 Parallel Implementation :4 :5 : 4: 5: 6: 7: 8:4 9:7 0: :9 :0 :0 4:7 5: 6:9 7: 8:6 9:9 0:0 :0 :7 : 4:9 5: 6:4 7:7 8:4 9:4 0: : :4 :6 4:4 Figure: DAG representation of the overall MOAO framework execution for the calculation of three ToR (i.e., three iterations of the outer loop) and a single observing sequence (i.e., one iteration in the inner loop) on a two-by-two tile matrix. H. Ltaief MOAO on Multiple GPUs 4 / 6 Directed Acyclic Graph for MOAO

29 Parallel Implementation Figure: DAG representation of the overall MOAO framework execution for the calculation of three ToR (i.e., three iterations of the outer loop) and a single observing sequence (i.e., one iteration in the inner loop) on a two-by-two tile matrix. H. Ltaief MOAO on Multiple GPUs 5 / 6 Directed Acyclic Graph for MOAO

30 Parallel Implementation Zooming in... H. Ltaief MOAO on Multiple GPUs 6 / 6

31 Performance Results Outline Motivations:Task-based Programming Model Application: The European Extremely Large Telescope Parallel Implementation 4 Performance Results 5 Conclusion H. Ltaief MOAO on Multiple GPUs 7 / 6

32 Performance Results Global Workflow Chart ToR computation System Parameters Cmm Ctm ToR R Turbulence Parameters Cmm Ctm Observing sequence Ctt BLAS Cee BLAS Cvv Intersample H. Ltaief MOAO on Multiple GPUs 8 / 6

33 Performance Results Environment Settings Hardware and software description: GPU system Intel(R) Xeon(R) CPU E5-670 (6 SDB cores) 8 NVIDIA Kepler K40 GPUs 64 GB of DDR4 main memory 64-bit instruction set with AVX.0 Software PLASMA v.6 OmpSs dynamic runtime system (mcc v.99.7 and nanox 0.9a) GCC compiler suite Intel MKL BLAS v5 H. Ltaief MOAO on Multiple GPUs 9 / 6

34 Performance Results MOAO performance in Tflop/s on 6 SDB cores + 8 K40 GPUs Tflops/s Async - 8GPUs + 6 cores Sync - 8GPUs + 6 cores Async - 4GPUs + 6 cores Sync - 4GPUs + 6 cores Async - GPUs + 6 cores Sync - GPUs + 6 cores Async - GPU + 6 cores Sync - GPU + 6 cores Total # of Measurements H. Ltaief MOAO on Multiple GPUs 0 / 6

35 Performance Results MOAO performance in seconds on 6 SDB cores + 8 K40 GPUs Time (s) Sync - GPU + 6 cores Async - GPU + 6 cores Sync - GPUs + 6 cores Async - GPUs + 6 cores Sync - 4GPUs + 6 cores Async - 4GPUs + 6 cores Sync - 8GPUs + 6 cores Async - 8GPUs + 6 cores Total # of Measurements H. Ltaief MOAO on Multiple GPUs / 6

36 Performance Results Performance scalability on 6 SDB cores + 8 K40 GPUs 4.5 8GPUs + 6 cores 4GPUs + 6 cores GPUs + 6 cores GPU + 6 cores.5 Speedup Total # of Measurements H. Ltaief MOAO on Multiple GPUs / 6

37 Performance Results Tracing on 6 SDB + 8 GPUs (a) Synchronous MOAO. (b) Asynchronous MOAO. H. Ltaief MOAO on Multiple GPUs / 6

38 Conclusion Outline Motivations:Task-based Programming Model Application: The European Extremely Large Telescope Parallel Implementation 4 Performance Results 5 Conclusion H. Ltaief MOAO on Multiple GPUs 4 / 6

39 Conclusion Summary Simulation almost real-time: 6min to simulate 50 PSFs. Capable of simulating hours of observations in few seconds. most efficient near real-time MOAO simulation frameworks available for the computational astronomy Efficient Task-based programming model Pipelining computational stages Leverage to other architectures e.g., Intel Xeon Phi, AMD APUs, ARM processors + GPUs (energy savings) H. Ltaief MOAO on Multiple GPUs 5 / 6

40 Conclusion Thank You! H. Ltaief MOAO on Multiple GPUs 6 / 6