Vampir Getting Started. Holger Brunst March 4th 2008

Transcription

1 Vampir Getting Started Holger Brunst March 4th 2008

2 What is Vampir? Program Monitoring, Visualization, and Analysis 1. Step: VampirTrace monitors your program s runtime behavior Event-based Produces detailed event traces rather than tabular profiles Information is recorded in OpenTraceFormat (OTF) 2. Step: Vampir visualizes the program s event trace graphically Zooming Timeline charts Statistics and profiles

3 Vampir History PARvis at Research Center Jülich Vampir at Research Center Jülich Vampir at ZIH, Technische Universität Dresden was commercially available via Pallas GmbH, later via Intel Vampir was developed by ZIH, Dresden Successor: VampirServer (VNG) client/server architecture distributed event storage improved scalability works on clusters and SMPs

4 Vampir Versions Vampir + VampirTrace 3.0 Vampir + VampirTrace 4.0 / Intel Trace Analyzer Intel Trace Collector 4.0 Vampir 5.2 Intel Trace Analyzer 7.1 VampirTrace VampirServer 1.8 Intel Trace Collector 7.1

5 Monitoring: VampirTrace Open source program monitor Available from Technische Universität Dresden, ZIH Google for tu dresden and vampirtrace Recorded events Function entry/exit MPI and OpenMP events Hardware/software performance counters (e.g. PAPI) OS events: Process creation, resource management Collected event properties Time stamp Location (process / thread / MPI) MPI specifics like message size, etc.

6 Performance Data: OTF Trace Format Open source trace file format (OTF) Available from Technische Universität Dresden, ZIH Google for tu dresden and otf Includes powerful libotf to be used in custom applications Human readable Fast searching and indexing Snapshots On-the-fly compression Actively developed In cooperation with the University of Oregon (TAU) and Lawrence Livermore National Laboratory VI-HPS

7 Visualization: Vampir & VampirServer Vampir: All-in-one performance visualization (small problem sizes) VampirServer: Distributed high-end performance visualization Client/server architecture Parallel event processing Runs on a (part of a) production environment No need to transfer huge traces Parallel I/O VampirServer (Browser) Lightweight client on local workstation Outer appearance identical to Vampir Receives visual content only Already adapted to display resolution (but no images) Moderate network bandwidth and latency requirements Scales to trace data volumes >100 GB

8 Event-based Performance Analysis 1 CPU CPU Application Monitor Application Monitor Trace Data Time Performance Visualization Enable Scalability? 2 CPU Application Monitor 3 4 CPU CPU... Application Monitor Application Monitor Trace Data 10,000 CPU Application Monitor

9 SCALABLE TOOL ARCHITECTURE Performance Visualization Trace Data Part 1 Trace Data Part 2 Trace Data Trace Data Part 3... Trace Data Part m Parallel Data Analysis Server Worker 1 Worker 2 Worker 3... Worker n Boss

10 Vampir Displays: Global Process Timeline Shows timed app. events: processes functions resp. function groups messages collective ops I/O activities Context information on click Supports zooming (horizontal & vertical) Can control reference time interval of other displays

11 Vampir Displays: Global Process Timeline Global Process Timeline with Thumbnail View

12 Vampir Displays: Global Process Timeline Zoom

13 Vampir Displays: Global Process Timeline Zoom Further

14 Vampir Displays: Process Timeline Timeline for single processes Unfolded call stack in vertical dimension Shows: functions/function groups messages, collective communication I/O activities performance counter values Supports zooming Context display

15 Vampir Displays: Process Timeline + MFLOPS Single Process Timeline with MFLOPS Graph

16 Vampir Displays: Process Timeline + Counters Single Process Timeline with MFLOPS and Instructions per Second Graph

17 Vampir Displays: Summary Timeline Histogram of program activities (Imagine a 90º rotated profile/summary chart over time) Shows the number of processes doing a certain activity

18 Vampir Displays: Summary Timeline Zoom

19 Vampir Displays: Summary Chart Statistical overview over functions or groups of functions Modes: Bar chart, pie chart, table Global (for all processes) or local (for a single process) Exclusive time, inclusive time, occurrences Absolute or logarithmic scale Absolute values or percentages Supports zooming Automatically adjusts to the current timeline interval

20 Vampir Displays: Summary Chart Grouped / Comprehensive Function Statistics

21 Vampir Displays: Summary Chart Pie and Table Representations

22 Vampir Displays: Message Statistics Sender/receiver matrix Zooming Displayed message properties: length, rate duration count Optionally as: minimum, maximum, average, sum Message length histogram

23 Vampir Displays: Message Statistics Message Statistics with Thumbnail

24 Vampir Displays: Message Statistics Zoomed Message Statistics

25 Vampir Displays: Call Tree Shows function call hierarchy Provides caller/callee relationship Invocation count Min/max runtime Level folding Searching

26 Vampir Displays: Call Tree

27 Vampir Displays: Global Filters Ignore certain items globally Processes/threads or groups of them Messages by communicator or by tag Collective operations by communicator or by type I/O events by communicator by file by read/write access

28 Vampir Displays: Event Filters Event Filter Dialog

29 Vampir Displays: Process Filter

30 Performance Flaws in Communication Communication as such ( computation (domination over Late sender, late receiver Point-to-point messages instead of collective communication Unmatched messages Overcharge of MPI s buffers ( bandwidth ) Bursts of large messages ( latency ) Frequent short messages ( barrier ) Unnecessary synchronization

31 Further Performance Flaws Memory bound computation inefficient L1/L2/L3 cache usage TLB misses detectable via HW performance counters I/O bound computation slow input/output sequential I/O on single process I/O load imbalance Exception handling

32 Performance Effects of Tracing Measurement overhead can be grave for tiny function calls solve with selective instrumentation Long, asynchronous trace buffer flushes Too many concurrent counters more data Heisenbugs

33 Examples (1) Idle OpenMP Threads

34 Examples (2) Unbalanced Computation/Communication

35 Examples (3) Low FP rate due to heavy FP exceptions

36 Examples (4) Performance Visualization of Cell/B.E.

37 Examples (5) Cell/B.E. with DMA

38 Examples (6) Thread Creation and Load Balancing Issues

39 Examples (7) Improved Thread Creation

40 Vampir at VI-HPS: VampirTrace Monitor (1) Type: module load vampir Supports OpenMP and MPI VampirTrace provides compiler wrappers for automatic program instrumentation: vtcc, vtcxx, vtf77, vtf90 Supported compilers: GNU (gcc, g++, gfortran, g95) Intel version 9.x (icc, icpc, ifort) Intel version 10.0 (icc, icpc, ifort) Portland Group (PGI) (pgcc, pgcc, pgf90, pgf77) SUN Fortran 90 (cc, CC, f90) IBM (xlcc, xlcc, xlf90) NEC SX (sxcc, sxc++, sxf90)

41 Vampir at VI-HPS: VampirTrace Monitor (2) Replace cc by vtcc in your makefile and you are set Alternative instrumentation types: Manual instrumentation API DynInst binary instrumentation Performance Counters PAPI processor counters I/O bandwidth tracking Memory allocation counters User defined counters Example: smg2000

42 Vampir at VI-HPS: VampirServer (1) Logon Xeon Cluster ssh -Y Load VampirServer module: module load vampir-server Launch VampirServer daemon: vngd-start.sh -p 300?? Results in: Launching VampirServer Version on 5 MPI processes Found license file: /opt/vampir-server/1.8.0/lic.dat Running... Server listens on: linuxhtc02.rz.rwth-aachen.de:300??

43 Vampir at VI-HPS: VampirServer (2) Results in: Launching VampirServer Version on 5 MPI processes Found license file: /opt/vampirserver/1.8.0/lic.dat Running... Server listens on: node02:30034 Launch visualization client in separate shell with: module load vampir-server vng a localhost p 300??

44 Thank You!

45 References A. Knüpfer, H. Brunst, and W. E. Nagel: High Performance Trace Visualization. In Proc. of 13th Euromicro Conference on Parallel, Distributed and Network-based Processing, Lugano, Switzerland, February H. Brunst, D. Kranzlmüller, W. E. Nagel: Tools for Scalable Parallel Program Analysis - Vampir NG and DeWiz. Distributed and Parallel Systems, Cluster and Grid Computing, Springer, Kluwer Intl. Series in Engineering and Computer Science, Vol 777, Budapest, Hungary, 2004 H. Brunst, H.-Ch. Hoppe, W. E. Nagel, and M. Winkler: Performance Optimization for Large Scale Computing: The Scalable VAMPIR Approach. Proc. of ICCS 2001, San Francisco, CA, USA, May ,Springer LNCS 2074, pp , 2001 Vampir User Guide (Available in download archive)