A Scalable Computer Architecture for

Transcription

1 A Scalable Computer Architecture for On-line Pulsar Search on the SKA - Draft Version - G. Knittel, A. Horneffer MPI for Radio Astronomy Bonn with help from: M. Kramer, B. Klein, R. Eatough

2 GPU-Based Pulsar Timing FFT DeDisp IFFT Full Stokes Folding GPU DISK GPU User Mem DMA Fast Bit Reversal mory Syst tem Me CPU From ADC NIC

3 GPU-Processing Processing FFT Coherent Dedispersion DM IFFT Full Stokes Parameters Folding Pulse Period

4 Performance: 610M Samples/s (2 GPUs) GPU-Based Pulsar Timing

5 Pulsar Search FFT Coherent Dedispersion DM IFFT Full Stokes Parameters Folding Pulse Period

6 Pulsar Search FFT Coherent Dedispersion Trial DM Loop IFFT Stokes I Parameter Folding Trial Pulse Period

7 Binary Systems Search FFT Coherent Dedispersion Trial DM Loop IFFT Stokes I Parameter Folding Trial Pulse Period, Orbital Parameters

8 Pulsar Search Idea: Pulsar Search by Massively-Parallel Folding (in Time Domain)

9 Pulsar Search Idea: Pulsar Search by Massively-Parallel Folding Binary Systems: Make Length of Phase Bins variable (similar to Time Sequence Resampling)

10 Pulsar Search Idea: Pulsar Search by Massively-Parallel Folding Binary Systems: Make Length of Phase Bins variable High-Dimensional Search Space: Complete Coverage not possible.

11 Pulsar Search Frequency Domain: Pulsar Search by Massively-Parallel Harmonic Summation Binary Systems: Process Range of neighboring Frequency Bins Use same Hardware!

12 Pulsar Search Frequency Domain: Pulsar Search by Massively-Parallel Harmonic Summation Binary Systems: Process Range of neighboring Frequency Bins Use same Hardware! Not completely worked out yet.

13 Pulsar Search on the SKA Add Coherent Beamforming

14 Pulsar Search on the SKA Polyphase Filterbank FFT Coherent Beamforming Coherent Dedispersion Polyphase Filterbank FFT Coherent Beamforming Coherent Dedispersion IFFT Stokes I Parameter Folding Power Spectrum Harmonic Sum

15 Pulsar Search on the SKA Polyphase Filterbank FFT Coherent Beamforming Coherent Dedispersion IFFT Stokes I Parameter Folding FPGA CPU CPU GPU GPU GPU ASIC Polyphase Filterbank FFT Coherent Beamforming Coherent Dedispersion Power Spectrum Harmonic Sum

16 Mode of Operation (Time Domain) Telescope 0 Telescope 1 Telescope 127

17 Parallelization: Timeslicing Telescope 0 Telescope 1 Telescope 127 Timeslice to Timeslice to Timeslice to Rank 0 Rank 1 Rank 0

18 Parallelization Telescope 0 Telescope 1 Telescope 127 Required Processing Time defines Number of Ranks

19 Parallelization Data from all Telescopes Chain Network Rank 0 Rank 1 Rank 2 Rank 63 Architecturescales endlessly

20 Parallelization Telescope n Polyphase Filterbank 16 Subbands Compute Node 15 Compute Node 1 Compute Node 0

21 Parallelization One Rank Compute Node 0 Data Capture Phase Compute Node 1 Compute Node 15 8 Telescopes each

22 Parallelization One Rank Compute Node 0 Filtering and Subband Distribution Compute Node 1 Ring Network Compute Node 15

23 Coherent Beamforming Compute Node k Local Mem FPGA Polyphase Filterbank, Data Exchange System Mem Beam 0 Subb k Beam 1 Subb k Beam 2 Subb k Beam n Subb k Tel 0 Subb k Tel 1 Subb k Tel 2 Subb k Tel 127 Subb k FFT, Coherent Beamforming CPU CPU

24 Coherent Beamforming on CPUs Performance using AVX: 128 Input Spectra, single-precision float, 256k Elements 128 Beams of same Size: 2.1s per 3.5GHz (prel. Results)

25 GPU-Processing Processing Video Video DeDisp, IFFT, SI Mem Mem System Mem GPU0 GPUn Local Mem FPGA Beam 0 Subb k Beam 1 Subb k Beam 2 Subb k Beam n Subb k Tel 0 Subb k Tel 1 Subb k Tel 2 Subb k Compute Node k FFT, Coherent Beamforming CPU CPU Polyphase Tl127 Tel Subb bbk Filterbank, Data Exchange

26 GPU-Processing Processing Performance: 2 Spectra, horz/vert, single-precision float, 4M Elements Total Power Time Sequence of same Size: Total Power Time Sequence of same Size: 5ms

27 GPU-Processing Processing How to output the Results to the ASICs?

28 GPU-Processing Processing How to output the Results to the ASICs? All PCIe-Slots are already taken (GPUs, FPGAs)

29 GPU-Processing Processing How to output the Results to the ASICs? All PCIe-Slots are already taken (GPUs, FPGAs) Write to Screen Buffer to be output via Write to Screen Buffer, to be output via Monitor Cable

30 GPU-Processing Processing Mini DisplayPort Gbit/s ~ 70 Gbit/s Equiv. 1 PCIe x16 Slot

31 Does it work? GPU-Processing Processing

32 GPU-Processing Processing Does it work? Yes, but... GPU Kernel Screen Via DVI: 2.7 Gbit/s (Video)

33 Massively-Parallel Folding Local Mem Local Mem SI Time Sequence ASIC0 ASICn Monitor Cable Video Mem Massively Parallel Folding GPU0

34 Massively-Parallel Folding Compute Node 0 Compute Node 1 ASIC PC Up to 16 Monitor Cables Compute Node 15 GPU PC

35 Folding Time Domain Hypothetical Pulse Period P time Dt t S lit P l Detects Solitary Pulsars having P + small P

36 Folding Acceleration Search Hypothetical Acceleration Variable Bin Length (# of Samples per Bin) time Equiv. to Time Sequence Resampling

37 Harmonic Summation f 2 f 3 f 4 f f 0 2f 0 3f 0 4f 0 freq Dt Detects t Solitary Pulsars between f 0 and f 0 + f f 0 f 0 + f

38 Harmonic Sum - Acceleration Search f 2 f 3 f 4 f f 0 2f 0 3f 0 4f 0 freq Dt Detects t Binary Systems with max. Acceleration f f 0 f 0 + f

39 Folding Processor Broadcast Bus SI Time Sequence or Power Spectrum Accumulator Programmable Set of Counters and Incrementers Memory 64 x 32 bits To / from local Memory

40 Pulsar Detector ASIC Folding Processors

41 ASIC Network Ring Network Rank 0 Rank 1 Rank 2 Rank 63 ASIC PC

42 RFI Mitigation Subband-relative: Accumulation is per Subband Beam-relative

43 The Pulsar Search Machine 128 Telescopes 128 Beams 100 DMs Orbits 64x hypothetical Pulsars

44 The Pulsar Search Machine PC Cluster Switch-less Design, helps Scalability GPU-PC + ASIC-PC = Compute Node 64 Ranks of 16 Compute Nodes 2048 PCs, 4096 CPUs, 8192 GPUs, ASICS 32.5 M

45 Thanks!

46 The Pulsar Search Machine

47 Folding Processor

48 Costs

49 Supercomputer Costs 2005 Sandia National Laboratories Red Storm: $90 million Los Alamos National Laboratory ASCI Q: $215 million Earth Simulator Center, Japan: $250 million IBM Blue Gene/L: $290 million various (unreliable) Internet Sources