Models for On-the-Fly Compensation of Measurement Overhead in Parallel Performance Profiling

Transcription

1 Models for On-he-Fly Compensaion of Measuremen Overhead in Parallel Performance Profiling Allen D. Malony and ameer. hende Performance search Laboraory Deparmen of Compuer and Informaion cience Universiy of Oregon, Eugene, OR, UA Absrac. Performance profiling generaes measuremen overhead during parallel program execuion. Measuremen overhead, in urn, inroduces inrusion in a program s runime performance behavior. Inrusion can be miigaed by conrolling insrumenaion degree, allowing a radeoff of accuracy for deail. Alernaively, he accuracy in profile resuls can be improved by reducing he inrusion error due o measuremen overhead. Models for compensaion of measuremen overhead in parallel performance profiling are described. An approach based on raional reconsrucion is used o undersand properies of compensaion soluions for differen parallel scenarios. From his analysis, a general algorihm for on-he-fly overhead assessmen and compensaion is derived. Keywords: Performance measuremen and analysis, parallel compuing, profiling, inrusion, overhead compensaion. 1 Inroducion In parallel profiling, performance measuremens are made during program execuion. There is an overhead associaed wih performance measuremen since exra code is being execued and hardware resources (processor, memory, nework) consumed. When performance overhead affecs he program execuion, we speak of performance (measuremen) inrusion. Performance inrusion, no maer how small, can resul in performance perurbaion [7] where he program s measured performance behavior is differen from is unmeasured performance. Whereas performance perurbaion is difficul o assess, performance inrusion can be quanified by differen merics, he mos imporan of which is dilaion in program execuion ime. This ype of inrusion is ofen repored as a percenage slowdown of oal execuion ime, bu he inrusion effecs hemselves will be disribued hroughou he profile resuls. Any performance profiling echnique, be i based on saisical profiling mehods (e.g., see [4, 14]) or measured profiling mehods (e.g., see [2, 9]), will encouner measuremen overhead and will also have limiaions on wha performance phenomena can and canno be observed [7]. Unil here is a sysemaic basis for J.C. Cunha and P.D. Medeiros (Eds.): Euro-Par 2005, LNC 3648, pp , c pringer-verlag Berlin Heidelberg 2005

2 Models for On-he-Fly Compensaion of Measuremen Overhead 73 judging he validiy of differing profiling echniques, i is more producive o focus on hose challenges ha a profiling mehod faces o improve he accuracy of is measuremen. In his regard, we pose he quesion wheher i is possible o compensae for measuremen overhead in performance profiling. Wha we mean by his is o quanify measuremen overhead and remove he overhead from profile calculaions. (I is imporan o noe we are no suggesing ha by doing so we are correcing he effecs of overhead on inrusion and perurbaion.) Because performance overhead occurs in boh measured and saisical profiling, overhead compensaion is an imporan opic of sudy. In our Euro-Par 2004 paper [8], we presened overhead compensaion echniques ha were implemened in he TAU performance sysem [9] and demonsraed wih he NA parallel benchmarks for boh fla and callpah profile analysis. While our resuls showed improvemen in NA profiling accuracy, as measured by he error in oal execuion ime compared o a non-insrumened run, he compensaion models were deficien for parallel execuion due o heir inabiliy o accoun for inerprocess ineracions and dependencies. The conribuion of his paper is he modeling of performance overhead compensaion in parallel profiling and he design of on-he-fly algorihms based on hese models ha migh be implemened in pracical profiling ools. ecion 2 briefly describes he basic models from [8] and how hey fail. We discuss he issues ha arise wih overhead inerdependency in parallel execuion. In ecion 3, we follow a sraegy o model parallel overhead compensaion for message-based parallel programs based on a raional reconsrucion of compensaion soluions for specific parallel case sudies. From he raionally reconsruced models, a general on-he-fly algorihm for overhead analysis and compensaion is derived. Conclusions and fuure work are given in ecion 4. 2 Basic Models for Overhead Compensaion In our earlier work [8], we developed echniques for quanifying he overhead of performance profile measuremens and correcing he profiling resuls o compensae for he measuremen error inroduced. This work was done for wo ypes of profiles: fla profiles and profiles of rouine calling pahs. The echniques were implemened in he TAUprofiling sysem [9] and demonsraed on he NA parallel benchmarks. However, he models we developed were based on a local perspecive of how measuremen overhead impaced he program s execuion. Profiling measuremens are, ypically, performed for each program hread of execuion. (Here we use he erm hread in a general sense. hared memory hreads and disribued memory processes equally apply.) By a local perspecive we mean one ha only regards he overhead impac on he process (hread) where he profile measuremen was made and overhead incurred. Consider a message passing parallel program composed of muliple processes. Mos profiling ools would produce a separae profile for each process, showing how ime was spen in is measured evens. Because he profile measuremens are made locally o a process, i is reasonable, as a firs sep, o compensae

3 74 Allen D. Malony and ameer. hende for measuremen overhead in he process-local profiles only. Our original models do jus ha. They accouned for he measuremen overhead generaed during TAUprofiling for each program process (hread) and all is measured evens, and hen removed he overhead from he inclusive and exclusive performance resuls calculaed during online profiling analysis. The compensaion algorihm correced he measuremen error in he process profiles in he sense ha he local overhead was no included in he local profile resuls. The models we developed are necessary for compensaing measuremen inrusion in parallel compuaions, bu hey are no sufficien. Depending on he applicaion s parallel execuion behavior, i is possible, even likely, ha inrusion effecs due o measuremen overhead seen on differen processes will be inerdependen. We use he erm inrusion specifically here o poin ou ha alhough measuremen overhead occurs locally, is inrusion can have non-local effecs. As a resul, parallel overhead compensaion is more complex. In conras wih our pas research on performance perurbaion analysis [10 12], here we do no wan o resor o pos-morem parallel race analysis. The problem of overhead compensaion in parallel profiling using only profile measuremens (no racing) has no been addressed before. Cerainly, we can learn from echniques for race-based perurbaion analysis [13], bu because we mus perform overhead compensaion on-he-fly, he uiliy of hese algorihms will be consrained o deerminisic parallel execuion, for he same reasons discussed in [7, 13]. A a minimum, algorihms for on-he-fly overhead compensaion in parallel profiling mus uilize a measuremen infrasrucure ha conveys informaion beween processes a runime. I is imporan o noe his is no required for race-based perurbaion analysis (since he analysis is offline) and i is wha makes compensaion in profiling a unique problem. Techniques similar o hose used in PHOTON [15] and CCIFT [1] o embed overhead informaion in MPI messages may aid in he developmen of such measuremen infrasrucure. However, we firs need o undersand how local measuremen overhead affecs global performance inrusion so ha we can consruc compensaion models and use hose models o develop online algorihms. 3 Models of Parallel Overhead Compensaion To address he problem of overhead compensaion in parallel execuion, we mus develop models ha describe he effec of measuremen overhead on execuion inrusion. From hese models we can gain insigh in how he profiling overheads can hen be compensaed. However, unlike sequenial compuaion, he models mus idenify and describe aspecs of parallel ineracion ha may cause differen inrusion behavior and, hus, lead o differen mehods for compensaion. We know ha he mehods will involve he communicaion of informaion beween parallel hreads of execuion a he ime of heir ineracion. To be more specific, we will consider parallel compensaion in message passing compuaion. The parallel overhead compensaion models we presen below allow for informaion abou execuion delay o be passed beween processes during message

4 Models for On-he-Fly Compensaion of Measuremen Overhead 75 communicaion. The goal is o deermine exacly wha informaion needs o be shared and how his informaion is o be used in compensaion analysis. The modeling mehodology we develop exends o shared memory parallel compuing, bu he case for shared memory will no be presened here. The approach we follow below consrucs an undersanding of he parallel compensaion problem from firs principles. We firs look a only wo processes and hen hree processes. From his in-deph sudy, our hope is o gain modeling and analyses undersanding ha can exend o he general case. We will follow a sraegy of raional reconsrucion where we ake scenario measuremen cases and reconsruc an acual execuion as if he measuremen overhead were no presen. From wha we learn, we hen derive a model ha works for ha case and look for consisen properies across he models o formulae a general algorihm for overhead compensaion. The deails of overhead removal in he profile calculaion are described in our earlier paper [8]. The focus below is on deermining he acual overhead value o be removed for each process. These wo operaions ogeher consiue overhead compensaion. 3.1 Two Process Parallel Models The simples parallel compuaion involves only wo processes which exchange messages during execuion. Measuremen-based profiling will inroduce overhead and inrusion local o each process ha carries beween he processes as hey inerac. To model he inrusion and deermine wha informaion mus be shared for overhead compensaion, we consider he following wo-process scenarios: One send Two sends Handshake General Process sends one message o process sends wo messages o sends one message o, hen sends one message o General message send and receive For each scenario, we enumerae all possible cases for overhead relaions beween he processes (wha is called he measured execuion model) and for each case derive a represenaion of he execuion wih he overhead removed (wha is called he approximaed execuion model). We deermine he overhead-free approximaion using a raional reconsrucion of he acual even imings wih he measuremen overhead removed. Boh models are presened in diagrammaic form. In addiional, we presen expressions ha relae he overhead, waiing, and iming parameers from he measured execuion o hose correced parameers in he approximaed execuion. I is imporan o keep in mind ha he goal is o learn from he raional reconsrucion of he approximaed execuion how profile compensaion is o be done in he oher scenarios, especially he general case. For space reasons, we consider only he One end and General scenarios in his paper. cenario: One end. Consider a single message sen beween wo processes, and. Figure 1 shows he wo possible cases, disinguishing which process

5 76 Allen D. Malony and ameer. hende Case 1 Case 2 Measured Execuion (= x1) o2 w Measured Execuion (=x1) o2 w Approximaed Execuion x1 x2 >= o2 + w o2 = o2 + w w = 0 x2 = min(, o2+w) = o2+w Approximaed Execuion w x1 x2 < o2 + w o2 = o2 ( o2 if >o2) w = w + (o2 ) x2 = min(, o2+w) = Fig. 1. Two-Process, One-end Models and Analysis (Case: 1, 2). has accumulaed more overhead up unil he ime of he message communicaion. Execuion ime advances from lef o righ and shown on he imelines are send evens () and receive evens (, receive begin;, receive end). The overhead on is and he overhead on is o2. The overhead is shown as a blocked region immediaely before he or evens o easily see is size in he figure, bu i is acually spread ou across he preceding imeline where profiled evens occur. Also designaed is he waiing ime (w) beween and, assuming waiing ime can be measured by he profiling sysem. Case 1 occurs when s overhead is greaer han or equal o s overhead plus he waiing ime ( o2 + w). A raional reconsrucion of he approximaed execuion deermines ha would no have waied for he message (i.e., would occur earlier han ). Hence, he approximaed waiing ime (designaed as w ) should be zero, as seen in he approximaed execuion imeline. Of course, he problem is ha has already waied in he measured execuion for he message o be received. In order for o know s message would have arrived earlier, mus communicae his informaion. Clearly, he informaion is exacly he value, s overhead. This is indicaed in he figure by agging he message communicaion arrow wih his value. Wih s overhead informaion, can deermine wha o do abou he waiing ime. The waiing ime has already been measured and mus be correcly accouned. If he approximaed waiing is adjused o zero, where should he elapsed ime represened by w go? If he profiling overhead is o be correcly compensaed, he measured waiing ime mus be aribued o s approximaed overhead (o2 = o2 + w)! This is ineresing because i shows how he naive overhead compensaion can lead o errors wihou conveyance of delay

6 Models for On-he-Fly Compensaion of Measuremen Overhead 77 informaion beween sender and receiver. I is also imporan o noe ha canno be moved back any furher in he approximaed execuion. This suggess ha he only correcion we can ever make in he receiver is in he waiing ime. The overhead value sen by wih he message conveys o he informaion his message was delayed being sen by amoun of ime or his message would have been sen ime unis earlier. We conend ha his is exacly he informaion needed by o correcly adjus is profiling merics (i.e., compensae for overhead in parallel execuion). We refer o he value sen by as delay and will assign he designaor x o represen is modeling and analysis ha follows. For insance, s delay is given by x1. In boh cases, x1 =, bu i is no always rue ha delay will be equal o accumulaed overhead, as we will see. Now an ineresing quesion arises. How much earlier would fuure evens on process 2 occur in he approximaed execuion afer he message from has been received? In general, each process will mainain a delay value (xi for process Pi) for i o include in is nex send message o ell he receiving process how much earlier he message would have been sen. In he approximaed execuion, for denoaional purposes, we show he x1 and x2 values for and as shaded regions afer he las evens, and, respecively. We also show an expression for he calculaion of x2 for his case. Moving on o he second case, he overhead and waiing ime in is greaer han wha repors (i.e., < o2 + w). Raionally, his means ha happens afer in he approximaed execuion. Wha is he effec on w, he approximaed waiing ime? I is ineresing o see ha w can increase or decrease, depending on he relaion of o o2. (member, is he same as x1 in hese cases.) However, he occurrence of is cerainly dependen on and, hus, x2 will be enirely deermined by (and, in fac, equal o) x1. General cenario. The goal of he wo process models is o enumerae he possible cases arising from send/receive message communicaion. From hese cases, we can raionally reconsruc he approximaed execuion o deermine how overhead, waiing, and delay imes are o be adjused. From his reconsrucion, we can derive expressions for overhead analysis and correcion. The similariy in he case resuls leads us o propose a general scenario for wo processes. This scenario considers an arbirary message send on one process and corresponding message receive on he oher process. Thus, his is a generalizaion of he One end scenario above. However, we now use he delay values x1 and x2 insead of he and o2 overheads in he analysis. The expressions for he wo cases are given below (refer o Figure 1): Case 1 Case 2 x1 >= x2 + w x1 < x2 + w o2 = o2 + w o2 = o2 - (x1-x2 if x1>x2) w = 0 w = w + (x2-x1) x1 = x1 x1 = x1 x2 = min(x1, x2+w) = x2 + w x2 = min(x1, x2+w) = x1

7 78 Allen D. Malony and ameer. hende The imporance of he general scenario is he case analysis showing how he delay values are updaed and wha informaion is shared beween processes during message communicaion. (Keep in mind ha we are arbirarily designaing as he sender and as he receiver. The analysis also applies when is he receiver and he sender, wih appropriae reversals of noaion in he expressions.) Noice ha he overhead values (no shown) and o2 are accumulaed overheads. The o2 value is updaed here o accoun for waiing ime processing, bu whenever any new measuremen overhead occurs on or, he accumulaed overheads and o2 mus be updaed accordingly. imilarly, any new measuremen overhead mus also be added o he delay values x1 or x2. Jus o be clear, i is he overhead values ha are being removed during he profiling calculaions. Thus, we wan hese overhead o be accuraely accouned. The conclusion of he wo process modeling is ha we can handle he parallel overhead compensaion for ALL wo-process scenarios by applying he general analysis described above on a message-by-message analysis, mainaining he overhead and delay values as he online analysis proceeds. 3.2 Three Process Parallel Models The quesion a his poin is wheher ha conclusion applies o hree or more processes. Tha is, can he general wo-process analysis be applied on a messageby-message basis o all send/receive messages beween any wo processes in a muli-process compuaion and, more imporanly, give he desired overhead compensaion resul? We look a wo scenarios wih hree processes o ge a sense of he answer. These scenarios are: Pipeline Two ceive Process sends a message o, hen sends o P3 Process and P3 sends a message each o process We argue ha hese wo scenarios are enough o elucidae all similar cases regardless of he number of processes. Again, we follow a raional reconsrucion approach o deermine approximaed execuions and hen derive expressions for updaing overhead, waiing ime, and delay variables o mach he reconsruced execuions. Only he Two ceive scenarios is described in deail in his paper. cenario: Two ceive. When more han wo processes are communicaing, i is no hard o find a scenario ha raises unpleasan issues in our abiliy o correc overhead inrusion under a differen se of receive assumpions. These issues are brough on by he effec of inrusion on message sequencing. The Two ceive scenario exposes he problem. Here one process,, receives messages from wo oher processes. There are four cases o consider depending on he relaives sizes of overheads and waiing imes. Figures 2 and 3 show wo of he cases. For simpliciy, we reurn o looking only a he firs messages being sen and received on each process, and consider he iniial overheads (no he delays values) in he analysis.

8 Models for On-he-Fly Compensaion of Measuremen Overhead 79 P3 Measured Execuion o2 w y o2a P3 Firs message >= o2 + w (x1 = ) o2 = o2 + w w = 0 x2 = min(, o2+w) = o2+w = o2 econd message >= x2 + o2a + y (x3 = ) o2 = o2 + o2a + y y = 0 x2 = min(, x2+o2a+y) = x2+o2a+y = o2 P3 Approximaed Execuion x2 y o2a x2 Fig. 2. Three-Process, Two ceive Models and Analysis (Case 1). In Figure 2, a wo-par approximaed execuion is shown, wih par one (op) giving he sae afer he firs message is processed and par wo (boom) showing he resul afer he second message is processed. The analysis follows he approach we used before, wih new waiing values (w and y ) being calculaed and s delay value (x2) updaed. In his case, no waiing ime would have occurred, and no adjusmen o waiing ime is necessary. Oherwise, nohing paricularly srange sands ou in he approximaed resul. Wha would be a surprising resul? If he overhead analysis resuled in a reordering of send evens in ime, beween he measured execuion and he approximaed execuion, hen here would be concerns of performance perurbaion. In Figure 3, we see he send evens changing order in ime in he approximaed execuion, wih P3 s send aking place before s send. As wih he oher cases, our analysis reflecs a message-by-message processing algorihm. In he raional reconsrucion, we assume he message communicaion is explici and pairs a paricular sender and receiver. Under his assumpion, he order of messages received by mus be mainained in he approximaed execuion. In his case, is he ime reordering of send messages in Figure 3 a problem? In fac, no. I is cerainly possible ha a process () will firs receive a message from a process () sen afer anoher process (P3) sends a message o he receiving process. This jus reflecs he sric order of receives. However, if we consider receive operaions ha can mach any send, he send reordering exposes a problem wih overhead compensaion, since he message from P3 should have been received firs in he real execuion. The applicaion of our overhead compensaion models o programs using receive operaions ha can mach any send message resuls in profile analysis consrained o message orderings as hey are observed in he measured execuion. These message orderings are affeced by inrusion and, hus, may no be he

9 80 Allen D. Malony and ameer. hende P3 Measured Execuion o2 w y o2a P3 Approximaed Execuion w x2 y o2a Firs message < o2 + w (x1 = ) o2 = o2 w = w + (o2 ) x2 = min(, o2+w) = econd message >= x2 + o2a + y (x3 = ) o2 = o2 + o2a + y y = 0 x2 = min(, x2+o2a+y) = x2+o2a+y P3 w x2 Fig. 3. Three-Process, Two ceive Models and Analysis (Case: 2). message orderings ha occur in he absence of measuremen. However, while i is acually possible o deec reordering occurrences (i.e., measured versus approximaed orderings), i is no possible o correc for reordering during online overhead analysis and compensaion. Why? There are wo reasons. Firs, our analysis is unable o deermine if i is correc o associae a receive even wih a differen send even. Tha is, he performance analysis does no know wha ype of receive is being performed, one ha is for a specific sender or one ha can accep any sender. econd, even if we know he ype of receive operaion, i is no possible o know wheher changing receive order will affec fuure receive evens. Therefore, he models mus, in general, enforce message receive ordering. 3.3 Modeling ummary and General Algorihm Our above modeling and analysis of measuremen overhead in parallel message passing programs has produced hree imporan oucomes. Firs, he raional reconsrucions of he measuremen scenarios and he analysis of he approximaed execuions has resuled in a robus procedure for message-by-message overhead compensaion analysis in parallel profiling. I updaes correcly waiing imes associaed wih message processing and calculaes per process values ha capure online he amoun a process has been effecively delayed due o measuremen overhead and is effecs. From his overhead compensaion basis, he parallel profiling operaions used o updae inclusive and exclusive performance can be performed. econd, his analysis requires ALL send messages o be augmened wih he delay value of he sender process a he ime he message is sen. This informaion is necessary for he receiving process o apply he analysis procedures. Third, approximaion models based on receive ype can resul in more accurae overhead handling and profile resuls, bu he accuracy gains are anicipaed o be minor compared o he processing complexiy involved.

10 Models for On-he-Fly Compensaion of Measuremen Overhead 81 We argue ha general overhead scenarios for message passing compuaions can all be addressed from wha we learned in he wo- and hree-process modeling above. A general algorihm for overhead compensaion effecively applies he Two-Process, General modeling and analysis on a message-by-message basis. The algorihm is composed of hree pars: Updaing of local overhead and delay as a resul of local profile measuremens. Updaing of local overhead and delay as a resul of messages received and heir repored delay. Transmission of local delay when a process sends a message. If he ransmission of he delays values can be suppored, i should be possible o incorporae his overhead compensaion algorihm in a parallel profiling sysem such as TAU[9]. 4 Conclusion and Fuure Work Profiling is an imporan echnique for he performance analysis of parallel applicaions. However, he measuremen overhead incurred during profiling can cause inrusions in he parallel performance behavior. Generally speaking, he greaer he measuremen overhead, he greaer he chance he measuremen will resul in performance inrusion. Thus, here is fundamenal radeoff in profiling mehodology concerning he need for measuremen deail (as deermined by number of evens and frequency of occurrence) versus he desired accuracy of profiling resuls. We argue ha wihou an undersanding of how inrusion affecs performance behavior and wihou a way o adjus for inrusion effecs in profiling calculaions, he accuracy of he profiling resuls is uncerain. Mos parallel profiling ools quanify inrusion as a percenage slowdown in he whole execuion and regard his as an implici measure of profiling goodness. This is unsaisfacory since i assumes overhead is evenly disribued across all hreads of execuion and all profiling resuls are uniformly affeced. Our early work in parallel perurbaion analysis [11 13] demonsraed he abiliy o rack performance inrusion and remove is effecs in performance analysis resuls. However, here we had he luxury of a fully qualified even race which included synchronizaion evens ha exposed dependen operaion. This allowed us o recover execuion sequences and derive performance resuls for an approximaed uninsrumened execuion. While he same perurbaion heory applies, when profiling measuremens are used, he analysis mus be performed online. This paper conribues models for measuremen overhead compensaion derived from a raional reconsrucion of fundamenal parallel profiling scenarios. Using hese models we described a general on-he-fly algorihm ha can be used for message passing parallel programs. The errors encounered in our earlier work on he NA parallel benchmarks, resuling from our simpler overhead and compensaion models, should now be reduced. However, implemening his algorihms requires he abiliy o piggyback delay values on send messages and

11 82 Allen D. Malony and ameer. hende o process he delay values a he receiver. We are currenly developing a MPI wrapper library o suppor delay piggybacking ha we can use o validae our approach. Our implemenaion is inended o be porable o all MPI implemenaions and will no require ransmission of muliple messages. This scheme will be incorporaed in he TAU performance sysem. ferences 1. G. Bronevesky, D. Marques, K. Pingali, and P. odghill, Auomaed Applicaion-level Checkpoining of MPI Programs, Principles and Pracice of Parallel Programming (PPoPP), L. De Rose, The Hardware Performance Monior Toolki, Euro-Par Conference, A. Fago and J. de Kergommeaux, ysems Assessmen of he Overhead of Tracing Parallel Programs, Euromicro Workshop on Parallel and Disribued Processing, pp , Graham, P. Kessler, and M. McKusick, gprof: A Call Graph Execuion Profiler, IGPLAN ymposium on Compiler Consrucion, pp , June R. Hall, Call Pah Profiling, Inernaional Conference on ofware Engineering, pp , D. Kranzlmüller, R. ussner, and C. chaubschläger, Monior Overhead Measuremen wih KaMPI, EuroPVM/MPI Conference, LNC 1697, pp , A. Malony, Performance Observabiliy, Ph.D. hesis, Universiy of Illinois, Urbana-Champaign, A. Malony and. hende, Overhead Compensaion in Performance Profiling, Euro-Par Conference, LNC 3149, pringer, pp , A. Malony, e al., Advances in he TAU Performance ysem, In V. Geov, M. Gernd, A. Hoisie, A. Malony, B. Miller (eds.), Performance Analysis and Grid Compuing, Kluwer, Norwell, MA, pp , A. Malony, D. ed, and H. Wijshoff, Performance Measuremen Inrusion and Perurbaion Analysis, IEEE Transacions on Parallel and Disribued ysems, 3(4): , July A. Malony and D. ed, Models for Performance Perurbaion Analysis, ACM/ONR Workshop on Parallel and Disribued Debugging, pp. 1 12, May A. Malony, Even Based Performance Perurbaion: A Case udy, Principles and Pracices of Parallel Programming (PPoPP), pp , April arukkai and A. Malony, Perurbaion Analysis of High-Level Insrumenaion for PMD Programs, Principles and Pracices of Parallel Programming (PPoPP), pp , May Unix Programmer s Manual, prof command, ecion 1, Bell Laboraories, Murray Hill, NJ, January J. Veer, Dynamic aisical Profiling of Communicaion Aciviy in Disribued Applicaions, ACM IGMETRIC Join Inernaional Conference on Measuremen and Modeling of Compuer ysems, ACM, 2002.