Image-Domain Gridding on Accelerators

Transcription

1 Netherlands Institute for Radio Astronomy Image-Domain Gridding on Accelerators Bram Veenboer Monday 26th March, 2018, GPU Technology Conference 2018, San Jose, USA ASTRON is part of the Netherlands Organisation for Scientific Research (NWO)

2 Introduction to radio astronomy Observe the sky at radio wavelengths: Size of the telescope is proportional to the wavelength e.g. Hubble Space telescope: 1 um, 2.4 m Same resolution for 1 mm requires 2 km dish! Image credits: NRAO 1

3 Radio telescope: astronomical interferometer Array of seperate telescopes: interferometer Create an image: interferometry Combine the signals for pairs of telescopes Resolution similar to one very large dish 2

4 Interferometry theory Sparse sampling of the uv-plane : Every baseline samples the uv-plane: visibility and uvw-coordinate Orientation of baseline also determines orientation in the uv-plane A sample V (u, v) is the 2D FT of the brightness on the sky B(l, m) Apply Non-uniform Fourier transform to get sky image from uv data 3

5 Sampling using 2 antennas Every sample (in the uv-domain) shows up as a waveform in the image Image credits: NRAO 4

6 Sampling using 4 antennas Every baseline (pair of two antennas) adds information to the image 5

7 Sampling using 16 antennas (compact) Using 16 antennas, the (artificial) source in the center of the image becomes visibile 6

8 Sampling using 16 antennas (extended) Longer baselines (larger antenna spacings) increase resolution of the image 7

9 Sampling using 32 antennas, for 8 hours Sampling for an extended period of time increases signal to noise 8

10 Creating a sky-image Imager: visibilities gridder regular grid 2D FFT sky-image incoming radio waves ionosphere receiver correlator imager sky-model baseline (pair of receivers) visibilities I calibration sky-image imaging Correlator: combine signal into visibility (with associated uvw-coordinate ) 9

11 Gridding using AW-projection (and W-stacking) W-term: correct for curvature of the earth A-term: correct for direction-dependent effects 10

12 W-projection gridding and Image-Domain gridding grid: W-projection gridding using convolution kernels visibilities visibilities gridder kernel image subgrids gridder kernel FFT time Fourier subgrids visibility: convolution: updated pixel: channels Image-Domain gridding using subgrids Fourier grid For more details: Image-Domain Gridding on Graphics Processors, Bram Veenboer, Matthias Petschow and John. W Romein, IPDPS 2017 adder Fourier grid 11

13 Square Kilometre Array SKA1 Low, Australia SKA1 Mid, Africa Data rates up to visibilities/second 12

14 Results: runtime/throughput Runtime: time spend in one imaging cycle: gridding, fft and degridding Throughput: number of visibilities processed per second Haswell KNL Haswell KNL gridding degridding Pascal Vega gridder subgrid-ifft adder grid-fft splitter subgrid-fft degridder Runtime [seconds] Pascal Vega Throughput [MVisibilities/s] Most time spent in gridder/degridder GPUs perform > order of magnitude better than CPU and Xeon Phi Very similar throughput for gridding and degridding 13

15 Roofline analysis: overview Performance [TOp/s] 10 1 gridder degridder degridder Vega Pascal KNL Haswell gridder 0.1 Haswell KNL Pascal Vega Operational intensity [Op/Byte] 14

16 Throughput limitation: host-device transfers PCIe: 12 GB/s vs. NVLINK: 68 GB/s Fast interconnect needed to keep GPU computing. 15

17 Roofline analysis: overview Performance [TOp/s] 10 1 gridder degridder degridder Vega Pascal KNL Haswell gridder 0.1 Haswell KNL Pascal Vega Operational intensity [Op/Byte] 16

18 Inner loop for (de)gridder kernel: instruction mix Many fused-multiply-add (FMA) and one sine/cosine computation: 1 α =... ; 2 3 for c=1,..., C do // channel 4 Φ = cos (α) + i sin (α); 5 6 Re(pix 11) += Re(vis 11[c]) Re(Φ[c]); 7 Im(pix 11) += Re(vis 11[c]) Im(Φ[c]); 8 Re(pix 11) = Im(vis 11[c]) Im(Φ[c]); 9 Im(pix 11) += Im(vis 11[c]) Re(Φ[c]); // [... same for pix 12, pix 21 and pix 22] 12 end FMA: peak performance on all architectures sine/cosine: poor performance on Intel architectures 17

19 Roofline analysis: instruction mix Performance [TOp/s] 10 1 gridder degridder degridder Vega Pascal KNL Haswell gridder 0.1 Haswell KNL Pascal Vega Operational intensity [Op/Byte] 18

20 Roofline analysis: shared memory 10 degridder gridder Pascal Vega Performance [TOp/s] Pascal Vega Operational intensity [Op/Byte] 19

21 Results: energy consumption/efficiency Haswell Haswell gridder degridder KNL KNL Pascal Vega gridder subgrid-ifft adder grid-fft splitter subgrid-fft degridder host Energy consumption [kj] Pascal Vega Energy efficiency [GFlop/W] Most energy spent in gridder/degridder GPUs perform > order of magnitude better than CPU and Xeon Phi 20

22 Results: comparison with AW-projection Throughput [Visibilities/s] IDG Pascal WPG Pascal AWPG Pascal W-kernel size N W IDG outperforms W-projection, while it also corrects for the challenging A-terms 21

23 Results: creating very large images (gpu-only) 240 Throughput [MVisibilities/s] GPU only, gridding GPU only, degridding Size [pixels 2 ] The size of the image is restricted by the amount of GPU device memory 22

24 Results: creating very large images (gpu+cpu) 240 Throughput [MVisibilities/s] Hybrid, gridding Hybrid, degridding GPU only, gridding GPU only, degridding Size [pixels 2 ] The adder kernel is executed by the host CPU 23

25 Results: creating very large images (Unified Memory) 240 tiling in adder/splitter Throughput [MVisibilities/s] Unified, gridding Unified, degridding Hybrid, gridding Hybrid, degridding GPU only, gridding GPU only, degridding Size [pixels 2 ] Unified memory (and tiling) enables the GPU to create very large images 24

26 Image-Domain Gridding for the Square Kilometre Array SKA-1 Low SKA-1 Mid # receivers # baselines 13, # channels 65, # polarizations 4 4 integration time 0.9 (s) 0.14 (s) data rate 8.3 (GVis/s) 9.53 (GVis/s) Imaging data rate: 200 GVis/s Compute: 50 PFlop/s (DP) Power budget: 1 MW (De)gridding: 60% IDG on Tesla V100/NVLINK: 0.26 GVis/s per GPU 770 V100s required Required compute: PFlop/s 15 PFlop/s available Power budget: KW 600 KW available 25

27 Summary High-performance gridding and degridding, including AW-term correction GPUs much faster and more (energy)-efficient than CPUs and Xeon Phi On GPUs, IDG outperforms AW-projection IDG is able to make very large images (using Unified Memory) Most challenging sub-parts of imaging for SKA is solved! More details: Image-Domain Gridding on Graphics Processors, Bram Veenboer, Matthias Petschow and John. W Romein, IPDPS 2017 Source available at: 26