Software Correlators for Dish and Sparse Aperture Arrays of the SKA Phase I

Transcription

1 Software Correlators for Dsh and Sparse Aperture Arrays of the SKA Phase I Jongsoo Km Korea Astronomy and Space Scence Insttute Collaborators: Paul Alexander (Unv. of Cambrdge) Andrew Faulkner (Unv. of Cambrdge)

2 Correlators for Rado Interferometry ASIC (Applcaton-Specfc Integrated Crcut) FPGA (Feld-Programmable Gate Arrays) Software (hgh level-languages, e.g., C/C++) Rapd development Expandablty

3 Current Status of SC LBA (Australan Long Baselne Array) 8 antennas (Parkes, 22-64m, GHz) DFX software correlator (2006; Deller et al. 2007, 2011) VLBA (Very Long Baselne Array) 10 antennas (25m, 330MHz - 86GHz) DFX MPIfR (the Max Planck Insttute for Radoastronomy) Mark4 DFX

4 Current Status of SC (cont.) GMRT (Gant Metrewave Rado Telescope) 30 antennas (45m, 50MHz-1.5GHz), 32MHz ASIC software correlator (Roy et al. 2010) LOFAR (Low Frequency Array) LBA (Low Band Antennae) 10-90MHz HBA (Hgh Band Antennae) MHz IBM BlueGene/P: software correlaton

5 CoDR for SKA Phase I, Memo 125 Key Scences: H I and Pulsars Sparse Aperture Array MHz, A/T sys = 2000m 2 /K, L max =100Km Dsh Array GHz, A/T sys =1000m 2 /K, m dshes sngle-pxel feeds, L max =100Km Constructon: Budget: 350M Euros

6 SKA Phase I: Prelmnary System, Dsh array m dshes, Memo 130 Bandwdth: 0.55GHz ( GHz), and 1.0GHz ( GHz) Dual polarzatons Bts per sample: 4 bts

7 SKA Phase I: Prelmnary System, Memo 130 Sparse Aperture array 50 statons Bandwdth: 380 MHz ( MHz) Number of beam: 480 (160) Dual polarzatons Bts per sample: 4 bts

8 Correlaton Theorem, FX-correlator R ( f ) r ( t)e 2 ft dt F-step (FT): ~log 2 (N c ) operatons per sample r( t) r ( ) d R ( f) R * ( f) j j X-step (CMAC): ~ N operatons per sample

9 FLOPS of the X-step n FX correlator 4 N ( N 1) 8 N 2 b B Hz [FLOPS] 16N B GHz * * * * 4 s from R Rj, R Lj, L Rj, L Lj 8 s from 4 multplcatons and 4 addtons: R R * j ( a b )( a j b j ) ( a a N(N+1)/2 s the number of auto- and crosscorrelatons wth antenna (staton) N Dsh array (N=250, B = 1 GHz, N b =1) 16x250 2 GFLOPS = 1 PFLOPS Sparse AA (N=50, B=380MHz, N b =160) 16x50 2 x160x0.38 GFLOPS = 2.43 PFLOPS j bb j 2 N b ) ( b a j a b j [GFLOPS] )

10 top500

11 Desgn goals Connect antennas and computer nodes wth smple network topology Use future technology development of HPC clusters Smplfy programmng

12 Data Rate per Dsh Pure data: 2 (pol) x 4 (bt/sample) x 2 (Nyqust) x 1GHz (BW) =16Gb/s Encodng overhead 20% (8b/10b; PCIe 2.0) 1.5% (128b/130b; PCIe 3.0) UDP (User Data protocol) overhead < 1% (28 bytes for headers / 65,507 bytes data length) Oversample overhead ~ 20% (memo 130)

13 CoDR of a Software Correlator for the dsh array 250 dshes 100 Gb/s Ethernet 250 nodes CPUs+(GPUs) CPUs+(GPUs) CPUs+(GPUs) CPUs+(GPUs) CPUs+(GPUs) Requred BW>32Gb/s >4 TFLOPS 2x4x2x1GHz =16Gb/s 2 pols, 4bt samplng, Nyqust, BW CPUs+(GPUs)

14 Communcaton between Computer Nodes I 1 after FFT Nc Node 1 FFT 2x4x2xB 2x8x2xB Node 2 FFT 2 Nc

15 Communcaton between Computer Nodes II 1 after communcaton Nc/2 Node 1 FFT 1 2 2x4x2xB 2x8x2xB 2 Node 2 FFT Nc/2 1 2

16 All-to-All Communcaton between Computer Nodes III 1 2 2x4x2xB 2x8x2xB Node 1 FFT Node 2 FFT Nc/4 after communcaton N>>1 BW(nterconnect) =2 (2x4x2xB) =32B 3 Node 2 FFT Nc/4 1 Node FFT 3 4

17 Data Rate per Staton Pure data: 2 (pol) x 4 (bt/sample) x 2 (Nyqust) x 0.38GHz (BW) x 160 (beams) =972.8 Gb/s Encodng overhead 20% (8b/10b; PCIe 2.0) 1.5% (128b/130b; PCIe 3.0) UDP (User Data protocol) overhead < 1% (28 bytes for headers / 65,507 bytes data length) Oversample overhead ~ 20% (memo 130)

18 CoDR of a Software Correlator for the sparse AA 50 statons staton staton staton staton staton 100 Gb/s Ethernets 16 subclusters subcluster subcluster subcluster subcluster subcluster staton 60Gb/s x 16 >150 TFLOPS subcluster

19 Channelzed data n a staton Nchan subcluster1 subcluster2 subcluster beams Nchan/16

20 Connectvty between statons and nodes n a subcluster 50 statons 100 Gb/s Ethernets staton 50 nodes node subcluster1 staton staton staton staton node node node node Requred BW >60Gb/s >3 TFLOPS staton ~ 60Gb/s node

21 Cost and Power Estmates of SCs Dsh Array # of nodes Cost per node [keuros] Cost of IB per port [keuros] Power per node [kw] Total cost [M Euros] Total power [MW] Sparse AA Total

22 AI (Arthmetc Intensty) Defnton: number of operatons (flops) per byte AI = 8flops/16bytes (R,R j ) = 0.5 R R * j ( a b )( a AI = 32 flops/32bytes (R, L, R j, L j ) = 1.0 for 1x1 tle AI = 2.4 for 3x2 tles j b j ) ( a a ) ( b a Snce AIs are small numbers, correlaton calculatons are bounded by the memory bandwdth. Performance: AI x memory BW (=102GB/s) j bb j Wayth+ (2009), van Neuwpoort+ (2009) j a b j )

23 Performance of Tesla C1060 as a functon of AI Performance s, ndeed, memory-bounded. Maxmum performance s about 1/3 of the peak performance.

24 Tesla Roadmap

25 Conclusons Connectvty: one 100GE connecton / dsh GE connectons / staton Performance One cluster wth 1 PFLOPS for the dsh array 16 clusters, each cluster wth 150 GFLOPS, for the sparse AA Cost and Power 6 M Euros, 1 MW

26 AI for host-devce and host-host AI = N+1 FLOP/byte N x 16 bytes (R,L) = 16 N byte 4x8xN(N+1)/2 FLOP PCI bus bandwdth PCI-e 2.0: 8.0GB/s (15 Jan. 2007) PCI-e 3.0: 16.0GB/s (2Q 2010, 2011) PCI-e 4.0: 32.0GB/s (201?) PCI-e 5.0: 64.0GB/s (201?) Performance [GFLOPS] = PCI BW [GB/s] x AI [FLOP/B]

27 Measured bandwdth of host-to-devce (Tesla C1060) ~70% of PCI -e2 bandwdth

28 Expected performance bounded by the BWs of PCI bus and nterconnect

29 Power usage and Costng Computer nodes 1.4 KW, 4 K Euro for each server ncludng 2x0.236 KW (2 GPUs) 0.4 MW, 1.2 M Euro for 300 servers Network Swtches 3.8 KW for IB (40Gb) 328 ports

30 Technology Development n Q 2010, (2011): PCI-e 3 rd Generaton 2Q (Aprl) 2010: Nvda Ferm (512 cores, L1,L2 cache) (March 29) 2010: AMD 12 core Opteron 6100 processor (March 30) 2010: Intel 8 core Nehaem-EX Xeon processor

31 Data Flows Node Interconnect BW: 40Gb/s (Infnband) CPU, FFT 4 8bts bt/sample GPU, CMAC threads memory CPU, FFT 4 8bts bt/sample GPU, CMAC threads memory GPU Memory BW: 102GB/s (C1060) PCI-e BW: 8GB/s (gen2 16x)

32 Software Correlator wth current technology 10 Gb/s Ethernet 60 nodes CPUs+GPU CPUs+GPU CPUs+GPU CPUs+GPU CPUs+GPU Infnband 40 Gb/s 0.3TFLOPS 2x4x2x0.25GHz =4Gb/s 2 pols, 4bt samplng, Nyqust, BW CPUs+GPU

33 Infnband Roadmap (IBTA)

34 Software Correlator for Phase I SKA ( 2018) 100 Gb/s Ethernet 300 nodes CPUs+(GPUs) CPUs+(GPUs) CPUs+(GPUs) CPUs+(GPUs) CPUs+(GPUs) Requred BW>512Gb/s =64GB/s >20 TFLOPS 2x4x2x4GHz =64Gb/s 2 pols, 4bt samplng, Nyqust, BW CPUs+(GPUs)