Signal Processing on GPUs for Radio Telescopes

Transcription

1 Signal Processing on GPUs for Radio Telescopes John W. Romein Netherlands Institute for Radio Astronomy (ASTRON) Dwingeloo, the Netherlands 1

2 Overview radio telescopes motivation processing pipelines signal-processing algorithms filter, correlator, beam forming, etc. performance 2

3 The LOFAR Radio Telescope 3

4 The LOFAR Radio Telescope largest low-frequency telescope no dishes distributed sensor network ~ receivers 4

5 LOFAR: A Software Telescope all-sky view antennas new science opportunities different observation modes require flexibility digitally steered concurrent observations supercomputer real time standard imaging pulsar survey known pulsar epoch of re-ionization transients ultra-high energy particles... 5

6 LOFAR SuperTerp 6

7 What's Next? The Square Kilometre Array world-wide effort unprecedented size 3,000 dishes + aperture array South Africa + Australia : 10% SKA : 100% SKA exascale computing; petascale I/O O(104) O(105) > LOFAR many challenges! 7

8 Our Efforts build/maintain/operate current telescopes (LOFAR, WSRT) research for SKA 8

9 Motivation 1) bring GPU technology to LOFAR 2) do accelerator research for the SKA 9

10 ASTRON/IBM Dome research technologies to develop the SKA green computing nano-photonics data & streaming P1 algorithms & machines P2 nano-photonics P3 access patterns P4 microservers P5 accelerators P6 compressed sampling P7 real-time communication 10

11 Accelerator Research accelerators for (radio-astronomical) signal processing GPUs, Xeon Phi, BG/Q, FPGAs,... fundamental understanding of accelerators which properties make architecture (in)efficient? I/O-compute balance? energy efficiency? programmability? architecture-(in)dependent optimizations? devise new algorithms? generic approach to program accelerators, or ad-hoc? applications: 1) LOFAR 2) SKA 11

12 LOFAR Data Processing developing new GPU-based system 12

13 Correlator Processing Overview 13

14 More Detail complex piece of software! several pipelines 14

15 Implement GPU Kernels killing two birds 1) develop new LOFAR correlator 2) code base for accelerator research 15

16 Why We Use OpenCL OpenCL advantages disadvantages vendor independent poor library support (e.g., FFT) GPU: runtime compilation cannot use all GPU features (e.g., GPUdirect) GPU: float8, float16, swizzling CPU: nice C++ interface (exceptions) float2 input_samples[nr_stations][nr_channels][nr_times][nr_polarizations]; float16 sums, weights; sums += weights.s abcdef012 * sample; 16

17 Why We Are Disappointed s l OpenCL advantages disadvantages o o t t vendor independent poor library support (e.g., FFT)en m p GPU: runtime compilation cannot use all GPU features o (e.g., GPUdirect) l e ev d GPU: float8, float16, swizzling n i t CPU: nice C++ interface (exceptions) por p u s L C n e p O d e p p float2 input_samples[nr_stations][nr_channels][nr_times][nr_polarizations]; r o r e l d i f A o I r D p I l V a float16 weights; N sums, u s i v sums += weights.s abcdef012 * sample; 17

18 GPUs FirePro S10000: driver/vbios/hardware (?) issues cannot always overlap compute PCIe I/O Xeon Phi: immature results NVIDIA Tesla K10 AMD FirePro S10000 Intel Xeon Phi (ECC off) FPUs SP TFLOPS GB/s W

19 From Feasibility Study to Production Software concentrated mostly on GPU kernels CPU parts: now/later 19

20 Implemented Kernels most important: filter correlator beam former 20

21 Processing Pipelines imaging mode sky images pulsar modes in development UHEP mode detect Ultra-High Energy Particles experimental 21

22 Processing Pipelines imaging mode sky images pulsar modes in development UHEP mode detect Ultra-High Energy Particles experimental 22

23 Processing Pipelines imaging mode sky images pulsar modes in development UHEP mode detect Ultra-High Energy Particles experimental 23

24 Imaging (Correlator) Pipeline 24

25 Imaging (Correlator) Pipeline four GPU kernels 25

26 Poly-Phase Filter (PPF) Bank splits frequency band into channels like prism trades time for frequency resolution 26

27 Poly-Phase Filter (PPF) Bank FIR filter + FFT 27

28 1) Finite Impulse Response (FIR) Filter history & weights (in registers) no physical shift reorder output for next kernel uncoalesced writes many FMAs operational intensity = 6.4 FLOPs / byte 28

29 FIR Filter Performance FirePro S10000 Tesla K GB/s TFLOPS 1.5 FirePro S10000 Tesla K #stations #stations 60 K10: stations small #threads 29 80

30 2) FFT 1D complex complex typically: tweaked Apple FFT library 30

31 2) FFT Performance FirePro S10000 Tesla K GB/s TFLOPS 1.5 FirePro S10000 Tesla K #stations #stations 60 memory I/O bound 31 80

32 3) Phase & BandPass Correction combined kernel: clock correction delay compensation bandpass correction transpose reduces #device memory accesses 32

33 3a) Clock Correction stations: shared clock corrects cable length errors merge with next step (phase delay) 33

34 3b) Delay Compensation (a.k.a. Tracking) track observed source delay telescope data delay varies due to earth rotation shift samples remainder: rotate phase (= cmul) 0.75 FLOPs / byte 34

35 3c) BandPass Correction powers in channels unequal artifact from station processing multiply by channel-dependent weight FLOPs / byte power frequency channel

36 Phase & BandPass Correction FirePro S10000 Tesla K GB/s TFLOPS 1.5 FirePro S10000 Tesla K #stations FLOPs / byte #stations 60 memory I/O bound 36 80

37 4) Correlator Kernel multiply samples from each station pair integrate ~1s 37

38 4) Correlation Triangle multiply samples from each station pair integrate ~1s 38

39 4) Correlator Implementation global memory local memory 1 thread per station pair (dual pol) 1 FLOP / byte (local mem) 39

40 4) Optimized Correlator Implementation increase register reuse 1 thread: 2x2 stations (dual pol) 2 FLOPs / byte (local mem) also: 3x3, 4x4 40

41 Correlator Register Usage #accumulator registers 1x1 8 2x2 32 3x3 72 max. registers/thread Tesla K10 FirePro S x3 on K10: excessive spilling 41

42 Correlator #Threads x1 2x2 3x3 #threads 768 max #threads Tesla K10 FirePro S , #stations cannot use all threads in last warp too few threads low occupancy too many threads multiple passes multiple thread blocks 42

43 1x1 vs. 2x2 and 3x3 (FirePro S10000) 2 1x1 2x2 3x3 300 GB/s TFLOPS 1.5 1x1 2x2 3x #stations #stations 60 use whichever performs best 43 80

44 Correlator Performance FirePro S10000 Tesla K GB/s TFLOPS 1.5 FirePro S10000 Tesla K compute bound 40 #stations #stations 60 S10000: multiple passes 44 80

45 Combined Pipeline #pragma omp parallel num_threads(2 * nrgpus) { cl::commandqueue queue(...); cl::buffer input(...), output(...),...; 2 queues/gpu: overlap I/O & computations 1 host thread/queue: easy model #pragma omp for schedule(dynamic) for (int subband = 0; subband < nrsubbands; subband ++) { receive(input, subband); queue.enqueuewritebuffer(input,...); queue.enqueuendrangekernel(fir_filter,...); queue.enqueuendrangekernel(fft,...); queue.enqueuendrangekernel(delay_bandpass,...); queue.enqueuendrangekernel(correlator,...); queue.enqueuereadbuffer(output,...); send(output, subband); } } 45

46 Correlator Pipeline Tesla K10 Performance Breakdown most challenging observation mode 8-bit samples 488 subbands (95.3 MHz) correlations most expensive need ~11 Tesla K10s? Correlate Phase & BandPass Correction FFT FIR filter #stations #GPUs 10

47 Large #Stations LOFAR SKA #stations #correlations ~80 ~12,960 3,000 #threads #thread blocks (K10) ~ ,000,000 1,125,000 1,000 different approach LOFAR SKA 47

48 Large #Stations: Another Strategy 192x192 stations (squares + rectangles) 2x2 stations (dual pol) 1 thread read samples stations 32x32 stations (16x16 threads) 32 FLOPs / byte 48

49 Large #Stations: Correlator Performance FirePro S10000 Tesla K GB/s TFLOPS #stations FirePro S10000 Tesla K #stations 640 to do: Tesla K20X possibly faster

50 Pulsar Pipelines 50

51 Pulsar Pipelines 1) search unknown pulsars 2) observe known pulsar many narrow beams credit: Jason Hessels 51

52 Pulsar Pipelines in development 52

53 Coherent Beam Forming add stations > sensitivity weighed compensate Δt change phase different Δt different beam many beams from same input CV beam = W beam, stat S stat stat 53

54 Coherent Beam Forming Implementation global memory local memory each thread: 1 beam, 1 polarization station-dependent weights in registers 2 passes of 24 stations 48 stations, 128 beams 14.2 FLOPs / byte CV beam = W beam, stat S stat stat 54

55 Coherent Beam Forming Performance 2.5 FirePro S10000 Tesla K10 FirePro S10000 Tesla K GB/s TFLOPS #beams stations #beams 96 sawtooth caused by unused threads

56 Ultra-High Energy Particle (UHEP) Pipeline 56

57 UHEP Pipeline store raw antenna voltages in 1.3s circular buffer create ~50 beams on moon detects ev particle collisions trigger in one/few adjacent beams: freeze/dump captured antenna voltages within 1.3s??? highly experimental pipeline Image Courtesy L. Viatour 57

58 UHEP Pipeline create ~50 beams on moon inverse filter for high time resolution trigger peak detection anti-coincidence check freeze raw antenna data buffer max 1.3s latency! 58

59 UHEP Pipeline create ~50 beams on moon inverse filter for high time resolution trigger peak detection anti-coincidence check freeze raw antenna data buffer max 1.3s latency! 59

60 UHEP Pipeline create ~50 beams on moon inverse filter for high time resolution trigger peak detection anti-coincidence check freeze raw antenna data buffer max 1.3s latency! 60

61 UHEP Pipeline create ~50 beams on moon inverse filter for high time resolution trigger peak detection anti-coincidence check freeze raw antenna data buffer max 1.3s latency! 61

62 UHEP Performance Breakdown Tesla K10 FirePro S Tesla K10 FirePro S GB/s TFLOP/s Beam Forming Transpose Inverse FIR Inverse FFT Trigger 0 Beam Forming Transpose Inverse FIR Inverse FFT 48 stations, 64 beams 62 Trigger

63 UHEP Performance Breakdown Input (Beam Former Weights) Input (Samples)? Trigger Inv. FFT Inv. FIR Transpose Beam Forming 60 ms ms data, 48 stations, 488 subbands most time spent in beam former I/O overlap latency 100 ms 0 Tesla K10 FirePro S

64 A Wild Idea 64

65 OpenCL on Top of CUDA Driver API fool CUDA RTS CPU: implement our own platform (ICD) OpenCL library calls CUDA Driver API Calls limited subset (proof of concept) GPU: use OpenCL PTX compiler (clc/clang/llvm) efficient does not support full language can use: visual profiler cufft, GPUdirect 65

66 OpenCL on Top of CUDA Driver API 66

67 Future Work 67

68 To Do (LOFAR Correlator) GPU kernels: dedispersion, flagging pulsar pipelines CPU code: work distribution network reordering (FDR IB) >240 Gb/s optimizations monitoring & control... 68

69 To Do (SKA Research) Xeon Phi OpenCL on FPGA energy efficiency fully understand all results 69

70 Conclusions many signal-processing GPU kernels new LOFAR correlator research for SKA OpenCL: vendor independent, elegant, but poor support NVIDIA FirePro S10000 faster then Tesla K10, but immature driver high efficiency on most important kernels 70

71 Thanks support: Intel, NVIDIA grants from Dutch national & province governments LOFAR GPU Correlator team: Alexander van Amesfoort, Wouter Klijn, Marcel Loose, Jan David Mol 71