The Practical Aspects of Computational Science and Engineering

Transcription

1 Computational Science and Engineering Frontiers in Computing and Data Science Michigan State University, 2-4 Oct 2016 Dr. Douglass Post, Associate Director for CREATE DISTRIBUTION STATEMENT A. Approved for Public Release. Distribution is unlimited. ITL-14-15, ITL-14-14, ITL-15-62, ITL-16-14

2 HPCMP Ecosystem Results Acquisition Science and Engineering Technology Decision Support DoD DoD Supercomputing Resource Centers (DSRCs) Acquisition Engineering U.S. Air Force Research Laboratory DSRC A technology-led, innovation-focused program committed to extending HPC to address the DoD s most significant challenges Decision Support Acquisition Engineering Networking and Security Defense Research & Engineering Network (DREN) Decision Support Acquisition Engineering Software Applications Core Software Decision Test and Support Evaluation U.S. Army Research Laboratory DSRC Computational Environments Maui High Performance Computing Center DSRC U.S. Army Engineer Research and Development Center DSRC Computer Network Defense, Security R&D, and Security Integration Education and Training Decision Acquisition Support Engineering U.S. Navy DSRC HPC User Support Page-2

3 BLUF-Bottom Line Up Front We live at the dawn of a New Era Today, for first time in history, we can accurately predict the future The performance of major physical systems using physics-based HPC computational tools Challenge is to develop and deploy the software Based on an High Performance Computing Ecosystem With virtual instead of physical Research and Engineering Facilities Paradigm shift with many sociological as well as technical challenges Page-3

4 HPC Ecosystem Becoming Ubiquitous Tool for Research, Prediction, Testing and Design Computational modeling is part of experimental and theoretical work in almost all the hard sciences and engineering, and social sciences as well Political Science Agent-based simulation graph Hurricane Sandy Gene Sequencing in Paleontology Predicted Path Calculated Tire Performance Hydroplane test rig Large Hadron Collider Higgs Boson Tracks Calculated Tokamak 3-D Turbulent Flow Observed Path Page-4

5 HPC is the Enabling Technology increase in computer power since 1945 is enabling us to: Utilize accurate solution methods Include all the effects we know to be important multi-physics Model a complete system Complete parameter surveys and analyze data from experiments with usable turn-around times In ~ 10 years, workstations will be as powerful as today s high performance computers Software applications capable of exploiting this computer power are the missing link! Performance (GFLOPs/sec) Computing Power For The World's Fastest Computer Floating-Point Operations/sec High Performance Computers Year Moore s Law Workstation Performance Cores Page-5

6 Experimental Facilities Have Grown from 1900 to Now 60 inch cyclotron at UC Berkeley CERN, Large Hadron Collider Wright Brothers Wind-Tunnel Simulator Small Large Machines Teams Cost Lifetime Complexity Data NASA Wind-Tunnel Simulator Page-6

7 Computational and Experimental Facilities Today Two Types of Facilities 1. Experimental test and research facilities 2. Virtual ecosystems for research and testing Virtual facilities include codes, computers, networks, etc. Like experimental facilities, virtual facilities require sustainment and modernization support Research Facilities Very Large Array Radio Telescopes Test Facilities Douglas XSB2D-1 airplane in NASA 40- by 80-foot tunnel Kestrel F-18 Simulation Supernova Simulation-FLASH Page-7

8 Application Paradigm is Changing In the past, code developers were often the major users SMEs had detailed understanding of the code s strengths and weaknesses Codes not designed for general use (focused V&V, hard to use, minimal documentation, ) Now, many SMEs use codes developed and supported by others Examples: Chemistry and Materials, Fluid Mechanics, Structural Mechanics, Climate and Weather, e.g., GAMESS, NWCHEM, OVERFLOW, NASTRAN, WRF, FLASH, ISV codes, SMEs less aware of code strengths and weaknesses 1960s today 1990s today Developers Users Developers Users Page-8

9 Science-based Code Productivity History Codes developed & used by researchers are generally successful Criterion for success is published research results, metrics: published papers Development of science-based codes for use by others takes a long time ( ~10 years) and a large multi-disciplinary team (~5 to 20), longer and larger than pure research codes E.g., NWCHEM, GAMESS, NNSA Design Codes, ISVs; Case studies (DARPA HPCS Phase II) Codes developed for use by others have more stringent requirements for success Useful for others for successful research or product design, measures: level of adoption, impact, Need adoption into customer community work flow, measures: level of adoption, impact, Higher level of software quality, user support, documentation,. Very important for success Evidence is that successful development rate for such codes is much lower Relatively few research codes graduate to community codes, measures: survival rate, # of users Many contractor-built large-scale complex software projects failed (FBI, NSA, FAA, ) at the $100M to $2B level Many ISV products are not profitable Page-9

10 HPCMP Ecosystem: It Takes a Village Results Acquisition Science and Engineering Technology Decision Support DoD DoD Supercomputing Resource Centers (DSRCs) Acquisition Engineering U.S. Air Force Research Laboratory DSRC A technology-led, innovation-focused program committed to extending HPC to address the DoD s most significant challenges Decision Support Acquisition Engineering Networking and Security Defense Research & Engineering Network (DREN) Decision Support Acquisition Engineering Software Applications Core Software Decision Test and Support Evaluation U.S. Army Research Laboratory DSRC Computational Environments Maui High Performance Computing Center DSRC U.S. Army Engineer Research and Development Center DSRC Computer Network Defense, Security R&D, and Security Integration Education and Training Decision Acquisition Support Engineering U.S. Navy DSRC HPC User Support Page-10

11 Development and Sustainment of Virtual Research and Test Facilities Involve Immense Challenges* Scientific and Engineering: Trade-off many different strongly interacting effects across many orders-ofmagnitude of multiple time- and distance-scales Verify and Validate highly-complex applications Wrong results lead to bad science and defective designs Develop problem generation methods required for larger, more complex problems Analyze and visualize larger, more complex datasets - era of Big Data Project: Evolve from small teams to large, multi-disciplinary and multi-institutional code development teams Achieve long-term, stable support for teams and provide customer support Programming: Changing computer architectures every 3-4 years Develop and run-codes for computers that will be 10 2 to 10 4 faster with 10 2 to 10 3 times more processors and greater memory architecture complexity than today Achieve efficient performance for next-generation computers Develop codes and run massively-parallel applications with relatively immature tools Present research communities have limited experience developing engineering applications additional challenges *c.f. The Opportunities, Challenges and Risks of High Performance Computing in, D.E. Post, R.P. Kendall and R.F. Lucas, Advances in Computers, Quality Software Development, 66, ( 2006), M. Zelkowitz, Ed., Academic Press pp Page-11

12 Examples of Success V&V on SE CREATE, GAMESS, FLASH, DoD Computational Research and Engineering Acquisition Tools and Environments (CREATE) 12-year, $300M program to develop design tools for DoD Air Vehicle, Ship and RF Antenna Major Weapon Systems, initiated in 2005, funding started in FY separate distributed code teams Multi-physics, multi-scale, HPC software After 8 years of development, software being used by ~ 130 DoD Acquisition Engineering Organizations (government and defense industry) GAMESS Community Chemistry Code led by Iowa State (Mark Gordon) Iowa State (Mark Gordon), ~ 10 staff on site, 150 contributors, ~6000 research papers written based on GAMESS use, distributed funding FLASH Astrophysics (Super-Nova) Code (U. of Chi.) Funded by DOE NNSA ASCI Alliance and NSF Community code widely used by astrophysical and HEDP community Metrics: # of users, impact, publications, longevity, financial support, Page-12

13 CREATE: Suite of Physics-based HPC Tools for the design and analysis of DoD platforms: Air Vehicles (AV) - Air Force, Army & Navy Concept design, High-Fidelity Fixed-Wing and Rotary-Wing Ships - Navy Concept design, Shock and Live-Fire Vulnerability, Hydrodynamics Radio Frequency (RF) Antennas - Air Force, Army & Navy RF Antenna electromagnetics & integration with platforms Ground Vehicles (GV) - Army, Marine Corps Design and evaluation of tactical ground vehicles Mesh and Geometry (MG) Generation Rapid generation of geometry representations and meshes F-35 CH-47 CREATE tools support all stages of acquisition from rapid earlystage design to full life-cycle sustainment and modifications Page-13

14 Existing DoD Paradigm (design, build, test, fix ) Requirements Design Scale-Model Tests Full-Scale Tests Production Goodyear Innovation Engine (design, virtual test, fix, build, deploy) Competitive Advantage Reduced product development time from 3+ years to ~9 months or less Cut prototype build & test costs by 62% Innovation engine new products from 10/year to over 60/year Design & Mesh Computational Prototype Design iterations Analyze & Test Computational Prototype Build & Test Physical Product Flight Radial Market Loren Miller, Simulation-Based Engineering for Industrial Competitive Advantage, computing in Science and Engineering (2010), May/June, pp Page-14

15 CREATE 6 Projects: 11 Multi-Physics Software Tools Ships - CREATE-Ships Rapid Ship Design Environment (RSDE) - Rapid Design and Synthesis Capability Navy Enhanced Sierra Mechanics (NESM) - Ship Shock and Shock Damage Assessment NAVYFOAM - Ship Hydrodynamics - Predicts hydrodynamic performance Integrated Hydro Design Environment (IHDE) - Facilitates access to naval design tools Air Vehicles - CREATE-AV DaVinci - Rapid conceptual design Kestrel - High-fidelity, full-vehicle, multi-physics analysis tool for fixed-wing aircraft Helios - High-fidelity, full-vehicle, multi-physics analysis tool for rotary-wing aircraft RF Antenna - CREATE-RF SENTRi - Electromagnetics antenna design integrated with platforms Ground Vehicles - CREATE-GV Mercury - High-fidelity, multi-physics simulation tool for vehicle systems and components Mobility Analysis Tool (MAT) - Analysis tool to evaluate ground vehicle performance metrics Meshing and Geometry - CREATE-MG Capstone - Components for generating geometries and meshes needed for analysis HPC Portal - Secure access to computers through a browser Page-15

16 Kestrel: A CREATE Example HPCMP CREATE TM AV Kestrel v5.x Computational Tool for Full Aircraft Design Analysis and Testing via high-fidelity, multi-disciplinary, physics-based simulation Key Disciplines (available now in Kestrel v5) Aerodynamics (Navier-Stokes solvers and full suite of BC s & turbulence models) Structural Dynamics (Modal models or FEA for aero-structure interaction) Flight Control Systems (Control surface movement deforming geometry or overset) Propulsion (Engine cycle-decks for propulsion effects, or direct engine simulation including inlet and rotating machinery, nozzle, and moving walls) Use-Cases Enabled Materially contribute to the design of next-generation aeronautical weapon systems. Verify design prior to key decision points (and prior to fabrication of test articles or full-scale prototypes) Plan/rehearse wind-tunnel and full-scale flight tests (more bang per test dollar) Evaluate planned (or potential) operational use scenarios Perform flight certifications (e.g., airworthiness, flight envelope expansion, mishap investigation, etc.) Generate response surfaces usable in DaVinci, flight-simulators, and other environments that require real-time access to performance data Page-16

17 Performed by AF AEDC using HPCMP CREATE TM A-10 Engine Inlet Flow Distortion Context The A-10 System Program Office (SPO) is exploring enhancements to the aircraft s current inboard leading-edge slat system. The A-10 mission requires operation at high angles-ofattack and high sideslip, which increases the likelihood of engine inlet flow distortion. Objective Apply HPCMP CREATE-AV TM Kestrel to simulate the A-10 with baseline and alternative wing leading-edge configurations to assess potential for improvements realizable from the planned enhancement. Explore wind-tunnel scale and fullscale conditions with both flow-through nacelles and the integrated TF34-GE D engine model. Jason Klepper (AF/AEDC) AEDC provided the A-10 SPO with engine inlet distortion data associated with various wing leading-edge designs. Analysis of the simulation results have identified the contributing sources of engine inlet distortion that could not be determined from wind-tunnel data alone. The A-10 enhancement program is still in progress. AEDC engineers have demonstrated that HPCMP CREATE TM -AV Kestrel is a valuable tool in design validation testing. Page-17

18 CREATE Organization Embedded with the Services Aligns customer and developer views and communications Embedding enables direct Service input and collaboration Aids adoption of tools HPCMP Director Dr. David Horner CREATE Program Assoc. Dir. Dr. Douglass Post Official HPCMP Advisory Panel Distributed, Multi-Organizational, Multi-Institutional Program Ships Project Project Manager HPCMP, Lorton NavyFOAM NSWC, Carderock Integrated Hydro Design Environment (IHDE) NSWC, Carderock Navy Enhanced Sierra Mechanics (NESM) NSWC, Carderock Rapid Ship Design Environment (RSDE) NSWC, Carderock Air Vehicles Project Project Manager HPCMP, Lorton Kestrel 46 th Test Wing, Eglin AFB Quality Assurance NAVAIR, Patuxent River Helios Army ADD, Ames DaVinci HPCMP, WPAFB RF Antennas Project Project Manager (SENTRI) Sensors Directorate, AFRL, WPAFB Mesh & Geometry Project Project Manager Navy NRL (Capstone) Ground Vehicles Project Project Manager ERDC Mobility Analysis (MAT) ERDC Mercury TARDEC Also senior oversight by Board of Directors for each project (AV, Ships, RF, MG, GV) composed of senior Service acquisition technical staff (SES, Flag Level, O-6) Page-18

19 Distributed Teams Can Work But Need Communication, Team meetings,! AV Ships RF GV MG HPCMP AMES U of Wy ETI CSU SPAWAR SNL SSI AFRL SNL SNL Maui HPCC UTx U of Mich U of Mich Mich Penn State TARDEC State UIUC AFLCMC AFRL MSU ERDC AEDC 46 th Test Wing MIT CERDEC NAVAIR HPCMP Carderock NAVSEA Indian Head ONR NRL Page-19

20 Flexible, Agile Software Engineering Software built by government-led teams of 5 to 10 staff Technical team and team leader embedded in customer institutions Optimal balance of team agility, structured process, and accountability Highly Disciplined, but Agile Software Development Processes Strong emphasis on software quality and accountability Supportive code development environment - computational clusters, central servers and code repository, dedicated high performance computers... Annual releases of each product following a roadmap Increased capability annually Extensive beta-tests of each release Rigorous V&V process Improved scalability for massively-parallel computers Improved usability Responsive to evolving requirements Extensive documentation Page-20

21 Annual CREATE Product Release Cadence 17 June 2016 Fiscal Year FY2011 FY2012 FY2013 FY2014 FY2015 FY2016 FY2017* Quarter AV-DaVinci AV-Helios AV-Kestrel MG-Capstone RF-SENTRi Ships-IHDE Ships- NavyFoam Ships-NESM 1 Ships-RSDE Approximately every year, a fully-tested upgraded code with the new features identified in the roadmap is released DP Revised: 6/4/ Planned Page-21

22 CREATE has Defined Core Software Engineering Practices for DoD Physics-based HPC Engineering Software Applications Development Team 1. Lean (<10), close-knit development teams led by technical experts 2. Transparency in development across CREATE projects. Customer Focus 3. Oversight by senior stakeholder and user representatives 4. Pilots to solicit customer reaction and feedback 5. Frequent reporting to stakeholders Technical Maturity 6. Proven technologies and customer-defined use-cases 7. VVUQ in alignment with NRC (NAS/NAE) best-practices for scientific codes Development Methods 8. Milestone-driven workflow management with agile flexible workflow execution and annual releases 9. Configuration management 10. Code builds based on tests 11. Adequate code documentation Requirements Definition 12. Reliance on prototypes and use-cases to define requirements Page-22

23 DDG-1000 CVN-78 Class Ohio SSBN Replacement LX(R) NAVAIR Aerostar & Raven UAVs F/A-18 E/F/G E-2D ARMY/USMC NAVSEA AFLCMC About 130 DoD Acquisition Engineering Organizations (Government and Industry) Now using CREATE Codes UH-60 CH-47 (ACRB) Guided Airdrop (RDECOM) V-22 F-15 SA/DB-110 Strategic Airlift CP&A A-10 B-52 Page-23

24 At Last Count, 130 Organizations Use CREATE Tools HCPMP CREATE TM Ships: (38) Allion Corporation, Cardinal Engineering, DRS Corporation, DYNAFLOW Corp, General Dynamics/Electric Boat Division, Hi-Test Laboratory, Northrop Grumman Corp Undersea Systems, Classified Program NSWC Carderock Code 65, Classified Program NSWC Carderock Code 66, Naval Underwater Warfare Center, Sandia National Laboratories, Weidlinger & Associates, General Dynamics Land Systems, Hydromechanics Division Naval Surface Warfare Center Carderock Division, Bath Iron Works (shipyard), BMT-Syntek, Bollinger (shipyard), Booz Allen Hamilton, CSC (NAVSEA/PEO engineering contractor), DRS (NAVSEA/PEO engineering contractor), Gibbs and Cox (NAVSEA/PEO engineering contractor), HII (Newport News and Pascagoula shipyards), Lockheed Martin, NASSCO (shipyard), NAVFAC (Naval Facilities), Northrup Grumman, Office of Naval Intelligence, University of Michigan, US Army Corps of Engineers, US Coast Guard, MIT-Department of Naval Architecture, NSWC Carderock Division, Center for Innovative Ship Design, US Coast Guard and Coast Guard Academy, Texas A&M, Naval Postgraduate School, U. of Washington, Virginia Tech, Georgia Tech, HPCMP CREATE AV: (32) AFLCMC/EN, AFLCMC/XZ, AFAEDC, AFSEO, AF Edwards, AF Hill, AF Holloman, AFRL, NAVAIR/4.3, NAVAIR/4.10, NAVAIR/Carderock, Army/ADD (Moffett Field), Army/AED (Redstone Arsenal - Aviation), Army/SSDD (Redstone Arsenal - Missiles), Army Research Laboratory (ARL), Army/Nadick Soldier Systems Center), AF Academy (USAFA), AF Institute of Technology (AFIT), USNA, GaTech, BYU, NASA ARC, Boeing Philadelphia/Mesa (Helicopters), Boeing St Louis (Fixed- Wing), Lockheed-Martin, Northrop-Grumman, Raytheon, Sikorsky, Bell Helicopters, Textron, Karem Aircraft, Inc, Mercer Engineering, and Bihrle Applied Research Company CREATE RF: (55) 57th Intelligence Squadron, 96 Test Wing SK/SKI, AFLCMC, AFRL, AFSEO, Air Force Institute of Technology, Air Force Research Laboratory (Munitions Directorate), Airborne Threat Simulation Organization (ATSO), AMRDEC-RDMR-SSM-G/Signature Solutions, Inc., Army Research Laboratory, Ball Aerospace & Technologies, Boeing, CERDEC, Cobham Defense Electronics, CSCF, DeposiTech, EP Analytics, Filius, General Atomics Aeronautical Systems, Georgia Tech Research Institute, Global Analytics, Harris Corporation, IERUS Technologies, Leidos, Lockheed Martin Aeronautics, MITRE, NASA Langley Research Center, NASIC/ACNS, NAVAIR, Naval Research Lab, NAWCAD Lakehurst, Northrop Grumman Corporation, NSWC Crane Division, NSWC Dahlgren, Pacific Defense Solutions LLC, Raytheon Company, Raytheon Missile Systems, Raytheon, Space & Airborne Systems, Riverside Research, Rolls-Royce North American Technologies, Signature Solutions, Inc, Sikorsky Aircraft Corporation, Space Command-DoD, Synclesis, Systems Engineering Group, TechFlow, TEDT-WSV-ED, MITRE Corporation, U.S Army CERDEC, U.S. Army Aviation & Missile RDEC, University of Dayton Research Institute, US Army ARDEC, FPAT, METC (RDAR-MEF-E), UTC Pratt & Whitney, Vehicle Technology Directorate, Army Research Lab, Vencore MG: (5 CREATE AV, Ships and RF) Navy Research Lab (NRL)/ Low-Frequency Broadband (LFBB) Program, NRL/ Strategic Environmental Research and Development Program (SERDP), NRL/ Jet-noise reduction program, Engineering Research and Design Center(ERDC)-CREEL: Unattended Ground Sensors Programs, ERDC-ITL: Terrain Modeling Page-24

25 BLUF-Bottom Line Up Front We live at the dawn of a New Era Today, for first time in history, we can accurately predict the future The performance of major physical systems using physics-based HPC computational tools Challenge is to develop and deploy the software Based on an High Performance Computing Ecosystem With virtual instead of physical Research and Engineering Facilities Paradigm shift with many sociological as well as technical challenges Page-25

26 Questions and Actions Prediction is very difficult, especially if it's about the future." attributed to Niels Bohr 1. What are the advantages of making accurate predictions of the performance of major mechanical systems with physics-based HPC software applications? 2. What challenges do we need to address to be able to make accurate predictions of such systems with physics-based HPC software applications? 3. What do we need to do to overcome those challenges? 4. What are the practical challenges of accurately predicting the future performance of other complex phenomena (weather, climate, biological, material, cultures, etc.)? Page-26

27 Questions? Page-27

28 References 1. Assessing the Reliability of Complex Models: Mathematical and Statistical Foundations of Verification, Validation, and Uncertainty Quantification, National Academy of Sciences, CREATE: Software Engineering Applications for the Design and Analysis of Air Vehicles, Naval Vessels, and Radio Frequency Antennas, D. Post et al, Computing in Science and Engineering, (2016) 18 (pp.14-24) 3. A Fixed-Wing Aircraft Simulation Tool for Improving DoD Acquisition Efficiency, S. Morton and D. McDaniel, Computing in Science and Engineering, (2016) 18 (pp.25-31) 4. HPCMP CREATE-AV Quality Assurance: Lessons Learned by Validating and Supporting Computation-Based Engineering Software, B. Hallissy, et al, Computing in Science and Engineering, (2016) 18 (pp.52-62) 5. A Risk-Based, Practice-Centered Approach to Project Management for HPCMP CREATE, R. Kendall, et al, Computing in Science and Engineering, (2016) 18 (pp.40-51) 6. Risk-Based Software Development Practices for CREATE Multiphysics HPC Software Applications, R. Kendall, et al, Computing in Science and Engineering, (2016) 18 (pp.2-13) Page-28

29 Questions and Actions Prediction is very difficult, especially if it's about the future." attributed to Niels Bohr What are the advantages of being able to make accurate performance predictions of major mechanical systems with physics-based HPC software applications? What challenges are involved in this? What do we need to do to overcome those challenges? What are the challenges of accurately predicting the future performance of other complex phenomena (weather, climate, biological systems, materials, cultures, etc.)? How likely is it that we will succeed with any of these systems. Page-29