NAMD: Innovation Beyond Petascale SC15 Workshop on Producing High Performance and Sustainable Software for Molecular Simulation

Size: px

Start display at page:

Download "NAMD: Innovation Beyond Petascale SC15 Workshop on Producing High Performance and Sustainable Software for Molecular Simulation"

Bernice Paul
5 years ago
Views:

1 NAMD: Innovation Beyond Petascale SC15 Workshop on Producing High Performance and Sustainable Software for Molecular Simulation James Phillips Beckman Institute, University of Illinois

NAMD Mission Statement: Practical Supercomputing for Biomedical Research 77,000 users

23,000 have downloaded more than one version. 5000 citations of NAMD reference papers.

Desktops and laptops setup and testing Hands-On Workshops Linux clusters affordable

petaflop/s performance GPUs from desktop to supercomputer User knowledge is preserved

2 NAMD Mission Statement: Practical Supercomputing for Biomedical Research 77,000 users can t all be computer experts. 18% are NIH-funded; many in other countries. 23,000 have downloaded more than one version citations of NAMD reference papers. One program available on all platforms. Desktops and laptops setup and testing Hands-On Workshops Linux clusters affordable local workhorses Supercomputers free allocations on XSEDE Blue Waters sustained petaflop/s performance GPUs from desktop to supercomputer User knowledge is preserved across platforms. No change in input or output files. Run any simulation on any number of cores. Available free of charge to all. Oak Ridge TITAN

3 10 8 A brief history of NAMD (and VMD) HIV capsid Number of atoms Lysozyme ApoA1 ATP Synthase Ribosome STMV

Titan XK7 (GTC15) 0.5 Edison XC30 (SC14) Blue Waters XE6 (SC14) 0.

4 64 32 NAMD Runs Large Petascale Simulations Well 21M atoms Performance (ns per day) (2fs timestep) Number of Nodes 224M atoms Blue Waters XK7 (GTC15) Titan XK7 (GTC15) 0.5 Edison XC30 (SC14) Blue Waters XE6 (SC14) Influenza, 210M atoms Amaro Lab, UCSD Topologyaware scheduler

5 A Sampling of Petascale Projects Using NAMD Chemosensory Array Chromatophore Technology Research Center for Macromolecular Modeling and Bioinformatics Rabbit Hemorrhagic Disease Rous Sarcoma VirusBiomedical Beckman Institute, University of Illinois at Urbana-Champaign - SC15 MolSim HIV

available only in person at the Beckman Institute are now

6 Remote Visualization Enables Petascale Anywhere High-end visualization and analysis workstations previously available only in person at the Beckman Institute are now accessible via remote visualization. Compressed Video 1 Gigabit Network

7 NAMD is based on Charm++ Complete info at charmplusplus.org and charm.cs.illinois.edu

8 Charm++ As Used in NAMD Parallel C++ with data driven objects. Object groups: Global object with a representative on each PE. Asynchronous method invocation. Prioritized scheduling of messages/execution. Measurement-based load balancing. Portable messaging layer.

9 NAMD 2 Hybrid Decomposition Kale et al., J. Comp. Phys. 151: , Spatially decompose data and communication. Separate but related work decomposition. Compute objects facilitate iterative, measurement-based load balancing system.

10 NAMD Overlapping Execution Phillips et al., SC2002. Objects are assigned to processors and queued as data arrives.

11 Coordinated Message Priorities Computes enqueued as messages Priorities based on critical path Earlier step before later step Earlier PME stages before later stages Computes for remote patches before local Modules must be coordinated! All priorities defined in Priorities.h Diagnose and confirm using Projections

12 Startup Phases In parallel after reading input data on pe 0 Used to load data, construct and link objects Phases separated by quiescence detections Simplest way to ensure side effects done Allows diagnosis of: What operation is crashing or hanging What operation is using memory or time

13 A Few Quick NAMD/Charm++ Tips DO NOT use the MPI network layer Low-level verbs, gni, pami layers exist because they are faster Leverage MPI startup via charmrun ++mpiexec See also ++scalable-start, ++remote-shell, ++runscript DO use SMP builds for larger simulations Reduced memory usage and often faster Trade-off: communication thread not available for work Major direction of future optimization and tuning DO set processor affinity explicitly For example: ++ppn 7 +commap 0,8 +pemap 1-7,9-15 Cray by default tends to lock all threads onto same core

14 NAMD for Large Systems Per-node memory usage Exploit redundant structure Pre-compressed static data Distributed per-atom data Special memopt build Not all features supported NAMD-only file formats May change between versions No VMD reader/writer - ever I/O performance Data is relatively small Parallelized POSIX I/O Performance is just OK New Charm++ I/O library being co-developed Parallelize load balancer Local load balancing only

15 Distribution and Licensing Binaries and source code Charm++ included Annual releases Nightly builds Registration required Public CVS access available Installed on supercomputers No redistribution Citation required Registration required Use for any purpose Combine up to 10% of source with at least 50% original code without restriction VMD plugins use BSD license

16 Special Licenses Are Possible

17 Support and Training Public mailing list Other scientists know best Archived and searchable Social conventions apply Bug report s Personal support Driving projects New capabilities Tutorials and Case Studies Written by scientists Focus on science problems Hands-on workshops Taught by scientists Several per year Various locations Requires only laptop

18 Software Engineering Charm++: Git revision control Redmine issue tracking Gerrit code review Nightly build & test NAMD: CVS revision control Manual tracking & review Nightly build & release Automated build & install Configuration Management Latest builds for users Older builds preserved Modules in use logged Charm++ ships with NAMD Verification and Validation New tests for new features Library of historical tests User reports of crashes Internal checksums

19 Every Run is a Test Internal consistency checks are critical Relatively few users, fewer developers Variety of use cases and feature sets runs are unique Configuration changes are rampant on supercomputers Testing at scale is expensive leverage production runs Crashes are annoying but harmless Goal is to avoid generating bad science Users do not read warning messages! They barely read FATAL ERROR messages on stderr. Exit on cases that risk incorrect results, link to web page.

20 Development Process/Philosophy Five-year funding cycle Code, science, publish, proposal Evolutionary development Fully functional code at all times No stable/development branches Large changes by refactoring only Simplify don t manage Separation of responsibilities Alignment of incentives Low coupling between people No code without an eager user No single-user features No schedules, no promises No design/code documentation Source code must be discoverable Use sandboxes to hide complexity Priorities and opportunities Enabling new science Supporting outside developers

21 Collaborative Driving Projects Nearly every TCBG collaboration relies on NAMD. High-end simulations push scaling efforts. Try to anticipate needs: Million-atom virus just worked in Innovative simulations generate feature requests: What is science goal? Existing features usable? Find a scalable method. Make it general purpose.

Tcl Scripting Enables Portable Innovation Top-level protocols: Minimize, heat, equilibrate Simulated annealing Replica exchange (originally via sockets) Long-range forces on selected atoms Torques

22 Tcl Scripting Enables Portable Innovation Top-level protocols: Minimize, heat, equilibrate Simulated annealing Replica exchange (originally via sockets) Long-range forces on selected atoms Torques and other steering forces Adaptive bias free energy perturbation Coupling to external coarse-grain model Special boundary forces Applies potentially to every atom Several optimizations for efficiency Shrinking phantom pore for DNA

23 Why NAMD and VMD Use Tcl History: Programs are ~20 years old. Maturity: Package management, portable. Stability: Interfaces haven t changed. Flexibility: Encapsulates mini-languages. Approachability: Looks like a simple scripting language, doesn t scare non-programmers.

Replica Exchange Enables Advanced Sampling

order parameters Outward θ 7 θ 1 Free Energy

Reaction Path Sampling Inward GlpT transporter in

Tajkhorshid, Mechanistic picture for conformational

24 Replica Exchange Enables Advanced Sampling Bias-Exchange Umbrella Sampling on quaternion-based order parameters Outward θ 7 θ 1 Free Energy (kcal/mol) IF OF Reaction Path (θ 1 +θ 7 ) Efficient Reaction Path Sampling Inward GlpT transporter in membrane M. Moradi and E. Tajkhorshid, Mechanistic picture for conformational transition of a membrane transporter at atomic resolution. PNAS, 110:18916 (2013). θ 7 θ 1 12 replicas

Multi-Copy Algorithm Sophistication Increases 30#r#x#20#ns#

200#replicas#(2D)#x#5#ns# 50#replicas#x#20#ns# 30#r#x#20#ns#

25 Multi-Copy Algorithm Sophistication Increases 30#r#x#20#ns# 30#r#x#20#ns# 12#replicas#x#40#ns#(H1/H7)# 50#replicas#x#20#ns#(10#Hs)# 150#replicas## 12#replicas#x#40#ns#(H1/H7)# 24#replicas#x#20#ns#(H1/H7)# 200#replicas#(2D)#x#5#ns# 50#replicas#x#20#ns# 30#r#x#20#ns# 30#r#x#20#ns# 30#r#x#20#ns# Mahmoud Moradi, bias-exchange umbrella sampling simulations of GlpT!

26 NAMD Multi-Copy Capabilities Evolve New: Workflow algorithms Multiple in-memory checkpoints with inter-replica access Global work queue, dependency-driven execution First use: Milestoning (Lane Votapka, Amaro Lab, UCSD) Improved: Replica exchange simulations Enhanced Tcl-based scripting of collective variables Multiplexing replicas on smaller number of parallel partitions Available in NAMD 2.11 First beta release November 10, 2015

Looking Forward NERSC Cori / Argonne Theta (2016) Knight s Landing (KNL) Xeon Phi Single-socket nodes, Cray Aries network Theta Early Science Project: Free Energy Landscapes of Membrane Transport

27 Looking Forward NERSC Cori / Argonne Theta (2016) Knight s Landing (KNL) Xeon Phi Single-socket nodes, Cray Aries network Theta Early Science Project: Free Energy Landscapes of Membrane Transport Proteins Oak Ridge Summit (2018) IBM Power 9 CPUs + NVIDIA Volta GPUs 3,400 fat nodes, dual-rail InfiniBand network CAAR Project Molecular Machinery of the Brain Argonne Aurora (2018) Knight s Hill (KNH) Xeon Phi Synaptic vesicle and presynaptic membrane

28 MIC Vectorization Options Intrinsics what we have Written by David Kunzman (formerly) of Intel Currently in production Compiler what we want #pragma [omp] simd assert In 2014 ~20% slower with refactored kernel Missed AoS optimization now available in compiler ISPC our backup plan Similar to CUDA, consider if compiler fails

NIH Biomedical Technology Research Center for Macromolecular Modeling and Bioinformatics Developers of the widely used computational biology software VMD and NAMD 290,000 registered

0 score (NIH) research projects include: virus capsids, ribosome, photosynthesis, protein folding, membrane reshaping, animal magnetoreception Achievements Built on People 5 faculty

29 NIH Biomedical Technology Research Center for Macromolecular Modeling and Bioinformatics Developers of the widely used computational biology software VMD and NAMD 290,000 registered VMD users 77,000 registered NAMD users 600 publications (since 1972) over 54,000 citations Renewed with 10.0 score (NIH) research projects include: virus capsids, ribosome, photosynthesis, protein folding, membrane reshaping, animal magnetoreception Achievements Built on People 5 faculty members 8 developers 1 systems administrator 17 postdocs 46 graduate students 3 administrative staff Beckman Institute, University ofluthey-schulten, Illinois at Urbana-ChampaignStone, - Tajkorshid, Schulten, Phillips, Kale, Mallon

NRC Workshop on NASA s Modeling, Simulation, and Information Systems and Processing Technology

NRC Workshop on NASA s Modeling, Simulation, and Information Systems and Processing Technology Bronson Messer Director of Science National Center for Computational Sciences & Senior R&D Staff Oak Ridge