Development of a parallel, tree-based neighbour-search algorithm

Transcription

1 Mitglied der Helmholtz-Gemeinschaft Development of a parallel, tree-based neighbour-search algorithm for the tree-code PEPC Andreas Breslau

2 Outline 1 Motivation 2 Short introduction to tree-codes 3 The tree-based neighbour search 4 Validation 5 Benchmarking 6 Summary and Outlook Andreas Breslau Slide 2

3 Outline 1 Motivation 2 Short introduction to tree-codes 3 The tree-based neighbour search 4 Validation 5 Benchmarking 6 Summary and Outlook Andreas Breslau Slide 3

4 Motivation for simulations In astrophysics the usual time-scales are much bigger than in other fields of physics Andreas Breslau Slide 4

5 Motivation for simulations In astrophysics the usual time-scales are much bigger than in other fields of physics Formation of a star 10 mio years Andreas Breslau Slide 4

6 Motivation for simulations In astrophysics the usual time-scales are much bigger than in other fields of physics Formation of a star 10 mio years Revolution of the sun around the center of the Milky Way 200 mio years Andreas Breslau Slide 4

7 Motivation for simulations In astrophysics the usual time-scales are much bigger than in other fields of physics Formation of a star 10 mio years Revolution of the sun around the center of the Milky Way 200 mio years dynamical timescale of superclusters few billion years Andreas Breslau Slide 4

8 Motivation for simulations In astrophysics the usual time-scales are much bigger than in other fields of physics Formation of a star 10 mio years Revolution of the sun around the center of the Milky Way 200 mio years dynamical timescale of superclusters few billion years Objects of interest (stars, starclusters, galaxies) are very big Andreas Breslau Slide 4

9 Motivation for simulations In astrophysics the usual time-scales are much bigger than in other fields of physics Formation of a star 10 mio years Revolution of the sun around the center of the Milky Way 200 mio years dynamical timescale of superclusters few billion years Objects of interest (stars, starclusters, galaxies) are very big that makes laboratory experiments difficult Andreas Breslau Slide 4

10 Motivation for simulations Apart from observation, simulations are the only way to test theories Andreas Breslau Slide 5

11 Motivation for neighbour search Often huge amounts of self-gravitating matter are simulated Andreas Breslau Slide 6

12 Motivation for neighbour search Often huge amounts of self-gravitating matter are simulated e.g. starclusters can be simulated as n bodies (only attracting forces) [NGC 3603 from hubblesite.org] Andreas Breslau Slide 6

13 Motivation for neighbour search Often huge amounts of self-gravitating matter are simulated e.g. starclusters can be simulated as n bodies (only attracting forces) gravitation only first approximation (radiation, magnetism) Andreas Breslau Slide 6

14 Motivation for neighbour search Often huge amounts of self-gravitating matter are simulated e.g. starclusters can be simulated as n bodies (only attracting forces) [Carina Nebula from hubblesite.org] gravitation only first approximation (radiation, magnetism) repulsing force from pressure-gradient for gas Andreas Breslau Slide 6

15 Motivation for neighbour search To simulate self-gravitating gas: simulate gravitational force as usual Andreas Breslau Slide 7

16 Motivation for neighbour search To simulate self-gravitating gas: simulate gravitational force as usual add thermodynamic forces from fluid simulation Andreas Breslau Slide 7

17 Motivation for neighbour search fluid-codes based on a fixed mesh would waste resources computing empty regions [Kelager, M., 2006] Andreas Breslau Slide 8

18 Motivation for neighbour search fluid-codes based on a fixed mesh would waste resources computing empty regions often matter is highly clustered within the simulation box [ Andreas Breslau Slide 8

19 Motivation for neighbour search fluid-codes based on a fixed mesh would waste resources computing empty regions often matter is highly clustered within the simulation box use Smoothed Particle Hydrodynamics (SPH) [Kelager, M., 2006] Andreas Breslau Slide 8

20 Motivation for neighbour search mesh-based fluid codes compute thermodynamic properties (temperature, density, pressure) locally using input from neighboring cells Andreas Breslau Slide 9

21 Motivation for neighbour search mesh-based fluid codes compute thermodynamic properties (temperature, density, pressure) locally using input from neighboring cells In SPH, fluid properties are computed from averages over neighboring particles Andreas Breslau Slide 9

22 Motivation for neighbour search mesh-based fluid codes compute thermodynamic properties (temperature, density, pressure) locally using input from neighboring cells In SPH, fluid properties are computed from averages over neighboring particles need to know the next neighbours of a particle Andreas Breslau Slide 9

23 Outline 1 Motivation 2 Short introduction to tree-codes 3 The tree-based neighbour search 4 Validation 5 Benchmarking 6 Summary and Outlook Andreas Breslau Slide 10

24 The n-body problem n bodies interacting with each other ( n(n 1) interactions ) Andreas Breslau Slide 11

25 The n-body problem n bodies interacting with each other ( n(n 1) interactions ) 3 particles, 3 forces Andreas Breslau Slide 11

26 The n-body problem n bodies interacting with each other ( n(n 1) interactions ) 5 particles, 20 forces Andreas Breslau Slide 11

27 The n-body problem n bodies interacting with each other ( n(n 1) interactions ) 8 particles, 56 forces Andreas Breslau Slide 11

28 The n-body problem n bodies interacting with each other ( n(n 1) interactions ) runtime O(n 2 ) 8 particles, 56 forces Andreas Breslau Slide 11

29 The n-body problem n bodies interacting with each other ( n(n 1) interactions ) runtime O(n 2 ) bad computation time for big systems 8 particles, 56 forces Andreas Breslau Slide 11

30 A better solution Andreas Breslau Slide 12

31 A better solution Use a tree Andreas Breslau Slide 12

32 A better solution Use a tree [openclipart.org] Andreas Breslau Slide 12

33 A better solution Use a tree [openclipart.org] Andreas Breslau Slide 12

34 A better solution Use a tree a b c d e f g [openclipart.org] h i j k l m n o Andreas Breslau Slide 12

35 Using a tree Andreas Breslau Slide 13

36 Using a tree Andreas Breslau Slide 13

37 Using a tree Andreas Breslau Slide 13

38 Using a tree Andreas Breslau Slide 13

39 Using a tree Andreas Breslau Slide 13

40 Using a tree Andreas Breslau Slide 13

41 Using a tree Andreas Breslau Slide 13

42 Direct summation vs. tree-code direct: 19 tree: Andreas Breslau Slide 14

43 Parallelization of tree-codes Andreas Breslau Slide 15

44 Parallelization of tree-codes Andreas Breslau Slide 15

45 Parallelization of tree-codes Andreas Breslau Slide 15

46 Outline 1 Motivation 2 Short introduction to tree-codes 3 The tree-based neighbour search 4 Validation 5 Benchmarking 6 Summary and Outlook Andreas Breslau Slide 16

47 The search algorithm the search algorithm was integrated in the tree-code PEPC written by Dr. Paul Gibbon in Andreas Breslau Slide 17

48 The search algorithm the search algorithm was integrated in the tree-code PEPC written by Dr. Paul Gibbon in 2003 PEPC is a tree-code following the tradition started in 1986 by Barnes and Hut Andreas Breslau Slide 17

49 The search algorithm the search algorithm was integrated in the tree-code PEPC written by Dr. Paul Gibbon in 2003 PEPC is a tree-code following the tradition started in 1986 by Barnes and Hut It uses a Hashed Oct-tree as described by Warren and Salmon (1993) Andreas Breslau Slide 17

50 The search algorithm the search algorithm was integrated in the tree-code PEPC written by Dr. Paul Gibbon in 2003 PEPC is a tree-code following the tradition started in 1986 by Barnes and Hut It uses a Hashed Oct-tree as described by Warren and Salmon (1993) the search algorithm also follows an idea of Warren and Salmon Andreas Breslau Slide 17

51 The search algorithm while there are p a r t i c l e s w ith less than N_nn found next neighbours s e a r c h _ n e i g h b o u r s _ o f _ p a r t i c l e _ i ( r _ i ) i f found n e x t neighbours < N_nn increase r _ i f o r t h i s p a r t i c l e put p a r t i c l e on l i s t to search neighbours again end end Andreas Breslau Slide 18

52 The search algorithm s e a r c h _ n e i g h b o u r s _ o f _ p a r t i c l e _ i ( r _ i ) { } walk through t r e e from r o o t to leaves 1) p a r t i c l e s w i t h i n r _ i put on next neighbour l i s t 2) ignore nodes / p a r t i c l e s outside r _ i 3) resolve nodes with overlap with r _ i end Andreas Breslau Slide 19

53 Tree based neighbour search Andreas Breslau Slide 20

54 Tree based neighbour search Andreas Breslau Slide 20

55 Tree based neighbour search Andreas Breslau Slide 20

56 Tree based neighbour search Andreas Breslau Slide 20

57 Tree based neighbour search Andreas Breslau Slide 20

58 Tree based neighbour search Andreas Breslau Slide 20

59 Tree based neighbour search Andreas Breslau Slide 20

60 Tree based neighbour search Andreas Breslau Slide 20

61 Tree based neighbour search Andreas Breslau Slide 20

62 Tree based neighbour search Andreas Breslau Slide 20

63 Tree based neighbour search Andreas Breslau Slide 20

64 Tree based neighbour search Andreas Breslau Slide 20

65 Tree based neighbour search Andreas Breslau Slide 20

66 Tree based neighbour search Andreas Breslau Slide 20

67 Tree based neighbour search Andreas Breslau Slide 20

68 Tree based neighbour search Andreas Breslau Slide 20

69 Outline 1 Motivation 2 Short introduction to tree-codes 3 The tree-based neighbour search 4 Validation 5 Benchmarking 6 Summary and Outlook Andreas Breslau Slide 21

70 Validation check found neighbours manually with plots Andreas Breslau Slide 22

71 Validation check found neighbours manually with plots write validation tool Andreas Breslau Slide 22

72 Validation with plots Andreas Breslau Slide 23

73 Validation with plots Andreas Breslau Slide 23

74 Validation with plots Andreas Breslau Slide 23

75 Validation with plots Andreas Breslau Slide 23

76 Validation with validation tool PEPC IDs, coordinates, momentum,... coordinates of next neighbours perl script Andreas Breslau Slide 24

77 Validation Andreas Breslau Slide 25

78 Validation manual checking plots proofed that the algorithm works correct for 2D Andreas Breslau Slide 25

79 Validation manual checking plots proofed that the algorithm works correct for 2D the validation tool proofed that it works correct for 3D Andreas Breslau Slide 25

80 Outline 1 Motivation 2 Short introduction to tree-codes 3 The tree-based neighbour search 4 Validation 5 Benchmarking 6 Summary and Outlook Andreas Breslau Slide 26

81 An estimation Andreas Breslau Slide 27

82 An estimation Andreas Breslau Slide 27

83 An estimation Andreas Breslau Slide 27

84 An estimation Andreas Breslau Slide 27

85 An estimation for the network and memory usage N, N nn, p ρ = N V, N nn = 4 3 πr3 search ρ, V p = 4 3 πr3 domain N fetch = = 3 27 N nnn 2 p πρ [ (R + r) 3 R 3] + 27 N2 nn N p + N 3 nn O(N nn ), O(N 2/3 ), O(p 2/3 ) Andreas Breslau Slide 28

86 An estimation for the network and memory usage N p, N nn, p O(N nn ), O(N 2/3 p ) N fetch = 3 27N nn N 2 p + 27N2 nn N p + N 3 nn Andreas Breslau Slide 29

87 An estimation for the network and memory usage N nn N p N fetch N fetch [%] Andreas Breslau Slide 30

88 Juropa 2208 compute nodes Compute node: 2 Intel Xeon quad-core processors at 2.93 GHz Total cores: Overall peak performance: 207 Teraflops Main memory: 24 GB per node / TB total Networks: Infiniband Fat Tree [ Andreas Breslau Slide 31

89 Weak scaling on Juropa 50 next neighbours, particles per process, relative 1 nn walk nn fetch rest 0.8 relative runtime number of nodes Andreas Breslau Slide 32

90 N nn scaling on Juropa 10 nodes, particles per process, absolute absolute runtime [s] nn walk nn fetch rest x 0.76 x number of next neighbours Andreas Breslau Slide 33

91 Strong scaling on Juropa 50 next neighbours, 12 mio particles, absolute, logscale 20 nn walk nn fetch rest *x *x *x absolute runtime [s] number of nodes Andreas Breslau Slide 34

92 N scaling on Juropa 50 next neighbours, 30 nodes, relative 1 nn walk nn fetch rest 0.8 relative runtime number of particles [mio] Andreas Breslau Slide 35

93 Outline 1 Motivation 2 Short introduction to tree-codes 3 The tree-based neighbour search 4 Validation 5 Benchmarking 6 Summary and Outlook Andreas Breslau Slide 36

94 Summary Andreas Breslau Slide 37

95 Summary Parallel tree-based neighbour-search successfully implemented Andreas Breslau Slide 37

96 Summary Parallel tree-based neighbour-search successfully implemented Validation tool implemented (further versions can easily be tested) Andreas Breslau Slide 37

97 Summary Parallel tree-based neighbour-search successfully implemented Validation tool implemented (further versions can easily be tested) Weak and strong scaling at least as good as gravitational force computation Andreas Breslau Slide 37

98 Summary Parallel tree-based neighbour-search successfully implemented Validation tool implemented (further versions can easily be tested) Weak and strong scaling at least as good as gravitational force computation Overhead currently 60% total iteration time, but Andreas Breslau Slide 37

99 Optimization needed [Xprofiler screen-shot] Andreas Breslau Slide 38

100 Outlook find out, what uses so much time Andreas Breslau Slide 39

101 Outlook find out, what uses so much time domain decomposition balanced with nn search work load Andreas Breslau Slide 39

102 Outlook find out, what uses so much time domain decomposition balanced with nn search work load compute nn lists only every t timesteps Andreas Breslau Slide 39

103 Outlook find out, what uses so much time domain decomposition balanced with nn search work load compute nn lists only every t timesteps neighbour search for symmetric SPH Andreas Breslau Slide 39

104 Andreas Breslau Slide 40

105 Thank you for your attention. Any Questions? Andreas Breslau Slide 40

106 References Barnes, J., Hut, P., 1986: A hierarchical O(N log N) force-calculation algorithm Gibbon, P., et. al., 2010: Progress in Mesh-Free Plasma Simulation With Parallel Tree Codes Kelager, M., 2006: Lagrangian Fluid Dynamics Using Smoothed Particle Hydrodynamics Warren, M. S., Salmon, J. K., 1993: A parallel hashed oct-tree n-body algorithm Warren, M. S., Salmon, J. K., 1995: A portable parallel particle program Some images used in this talk are intellectual property of other authors and may not be distributed or reused without their explicit approval Andreas Breslau Slide 41

107 Appendix More scaling plots About the speaker Andreas Breslau Slide 42

108 Weak scaling on Juropa 50 next neighbours, particles per process, absolute absolute runtime [s] nn walk nn fetch rest number of nodes Andreas Breslau Slide 43

109 Weak scaling on Juropa 50 next neighbours, particles per process, absolute, logscale 100 absolute runtime [s] 10 1 nn walk nn fetch rest 12.93*x *x *x number of nodes Andreas Breslau Slide 44

110 nn scaling on Juropa 10 nodes, particles per process, relative 1 nn walk nn fetch rest 0.8 relative runtime number of next neighbours Andreas Breslau Slide 45

111 nn scaling on Juropa 10 nodes, particles per process, absolute, logscale absolute runtime [s] nn walk nn fetch rest 0.24*x *x * x number of next neighbours Andreas Breslau Slide 46

112 Strong scaling on Juropa 50 next neighbours, 12 mio particles, relative 1 nn walk nn fetch rest 0.8 relative runtime number of nodes Andreas Breslau Slide 47

113 Strong scaling on Juropa 50 next neighbours, 12 mio particles, absolute nn walk nn fetch rest 14 absolute runtime [s] number of nodes Andreas Breslau Slide 48

114 N scaling on Juropa 50 next neighbours, 30 nodes, absolute absolute runtime [s] nn walk nn fetch rest number of particles [mio] Andreas Breslau Slide 49

115 N scaling on Juropa 50 next neighbours, 30 nodes, absolute, logscale 100 absolute runtime [s] 10 nn walk nn fetch rest 0.37*x *x *x number of particles [mio] Andreas Breslau Slide 50

116 About the speaker Andreas Breslau JÜLICH SUPERCOMPUTING CENTRE (JSC) Forschungszentrum Jülich GmbH Andreas Breslau Slide 51