Development of a High-Order Discontinuous Galerkin Fluid Solver Within SU2

Transcription

1 Development of a Hgh-Order Dscontnuous Galern Flud Solver Wthn SU2 Edwn van der Wede Department of Mechancal Engneerng Unversty of Twente Thomas D. Economon, Juan J. Alonso, Jae hwan Cho, Carlos da Slva Department of Aeronautcs and Astronautcs Unversty of Stanford Dheevatsa Mudgere, Alexander Henece, Gaurav Bansal Intel Corporaton Ramesh Balarshnan Argonne Natonal Laboratory

2 Next thngs to be done Outlne Motvaton for hgh order schemes Pros and cons of dfferent dscretzaton technques DG-FEM, the basc prncples Implementaton n SU2 Performance optmzaton Hgh order grd generaton Prelmnary results

3 Why do we need hgh order schemes? 2 nd order schemes have been extremely successful, certanly for RANS applcatons

4 But For some applcatons 2 nd order accuracy may not be enough Examples Wae and vortex flows Nose predcton LES/DNS

5 ALCF Theta Early Scence Program Collaboraton Stanford Argonne Scale Resolvng Smulatons of Wnd Turbnes wth SU2 Horns Rev wnd farm (Denmar) Waes must be resolved Ideally suted for hgh order schemes

6 Spatal dscretzaton methods for hyperbolc systems (most commonly used) Fnte Dfference Method (FDM) Fnte Volume Method (FVM) (Dscontnuous Galern) Fnte Element Method (DG-FEM)

7 Pros and cons of the methods : Not suted : Suted : Suted, possbly wth modfcatons, but not the most natural (or effcent) choce Taen from Hesthaven and Warburton, nd order: FVM s the best choce Hgh order: DG-FEM s the best choce

8 Hyperbolc system of PDE s Wea formulaton 0 x F t U p m V m V m V N m d F n dv x F dv t U, 1,, 0 j N j j x U x U p 1 : Element Nodal DG-FEM: the basc prncples (1)

9 Nodal DG-FEM: the basc prncples (2) Contrbuton from the contour ntegral Soluton at the nterfaces s multply defned and dscontnuous V F n d F, m V U L U R n m d Remann problem: Any approxmate Remann solver can be used => stablzes the dscretzaton 1 st order DG-FEM equals 1 st order FVM!!!

10 Nodal DG-FEM: the basc prncples (3) Nonlnear equatons and/or curved elements Integrals must be computed wth hgh order quadrature rules to avod alasng. Expensve!!! Dffuson problems, 2 nd dervatves Dscontnuous bass functons not suted => must be repared Even more expensve!!! However: most operatons are local to an element. Extremely well-suted for modern computer hardware

11 Implementaton n SU2 (1) together wth Tom Framewor of SU2 s very flexble => hgh level data structures can be used for any solver, so also DG-FEM Input parameter structure can be reused entrely FVM output functonalty could be reused (after some modfcatons) Stll a lot of wor was requred for other low level functons Parttonng of the grd (element wse) Preprocessng s completely dfferent from FVM Standard elements and standard orentaton of elements are ntroduced (see next slde) Spatal dscretzaton s completely new

12 Implementaton n SU2 (2) Trcy part: Relatve orentaton of elements Needed for the dscretzaton of the vscous terms 2D hybrd grd: 37 dfferent cases 3D hybrd grd: 248 dfferent cases Soluton: Renumber connectvty of adjacent elements for each face. Smlar to the CGNS transformaton matrx between blocs. => One general mplementaton for the contour ntegral

13 Performance optmzaton (1) Collaboraton wth Intel Target archtectures: Intel Knghts Landng (MIC archtecture) Intel Xeon Hybrd MPI/OpenMP parallelzaton MPI Doman decomposton (one halo layer of elements) Overlap computaton and communcaton Use of persstent communcaton OpenMP Am: Parallelzaton on the for-loop level (suffcent for tests on Xeon) Current data structures are desgned for ths approach Optmzed BLAS/LAPACK/LIBXSMM functons must be used to get good performance for matrx multplcatons Large contguous chuns of memory for data storage (not the case for the FVM solver)

14 Performance optmzaton (2) Motvaton for hybrd parallelzaton approach Flat MPI does not seem to wor too well on KNL Prelmnary results on Theta (Argonne) Elem/Core K 4.88K 19.5K 39.1K DOF/Core 4.27K 8.55K 14.2K 42.7K 0.17M 0.68M 1.37M Effcency(%) N/A

15 Hgh order grd generaton (1) Hgh order elements must be curved near sold wall boundares

16 Hgh order grd generaton (2) Two possbltes Truly hgh order grd generaton Post processng of lnear grds => seems to be the preferred choce Not a lot of actvty from commercal vendors Centaursoft: Quadratc elements va post processng Pontwse n collaboraton wth several unverstes (ncludng us) Open source software Gmsh A lot more effort should be put nto hgh order grd generaton n order not to become a show stopper!!

17 Hgh order grd generaton wthn SU2 Collaboraton wth Pontwse Post processng of lnear grds (qualty lnear grd s crucal) Geometry ernel of Pontwse (GE Lte) s used for projecton onto the CAD geometry Correcton due to surface projecton s propagated nto the doman Heurstc algorthm (only for structured normal drecton): very fast SU2 mesh deformaton algorthm: wors for all element types Wors very well for not too complcated geometres To be tested for really complcated cases (problems expected)

18 Prelmnary results (p = 4, trangles, nvscd) Vsualzaton va lnear sub-elements, not hgh order

19 Prelmnary results, nvscd Vsualzaton va lnear sub-elements, not hgh order p = 3, tets p = 4, trangles

20 Prelmnary results (p = 4, quads/trangles, vscous) Vsualzaton va lnear sub-elements, not hgh order

21 Next thngs to be done Fnalze debuggng vscous terms (almost ready) Thorough V&V of the mplementaton Bnary, parallel IO Shoc capturng Tme accurate local tme steppng LES models and boundary condtons, ncludng wall models Performance optmzaton, ncludng OpenMP parallelzaton Grd moton, sldng mesh nterfaces Grd sequencng, multgrd Hgh order grd generaton