Implicit Coscheduling:

Transcription

1 Implicit Cscheduling: Crdinated Scheduling with Implicit Infrmatin in Distributed Systems Andrea Carl Apraci-Dusseau Reprt N. UCBèCSD December 998 Cmputer Science Divisin èeecsè University f Califrnia Berkeley, Califrnia 94720

2 Implicit Cscheduling: Crdinated Scheduling with Implicit Infrmatin in Distributed Systems by Andrea Carl Arpaci-Dusseau B.S. ècarnegie Melln Universityè 99 M.S. èuniversity f Califrnia, Berkeleyè 994 A dissertatin submitted in partial satisfactin f the requirements fr the degree f Dctr f Philsphy in Cmputer Science in the GRADUATE DIVISION f the UNIVERSITY f CALIFORNIA at BERKELEY Cmmittee in charge: Prfessr David Culler, Chair Prfessr Katherine Yelick Prfessr David Freedman 998

3 The dissertatin f Andrea Carl Arpaci-Dusseau is apprved: Chair Date Date Date University f Califrnia at Berkeley 998

4 Implicit Cscheduling: Crdinated Scheduling with Implicit Infrmatin in Distributed Systems Cpyright 998 by Andrea Carl Arpaci-Dusseau

5 Abstract Implicit Cscheduling: Crdinated Scheduling with Implicit Infrmatin in Distributed Systems by Andrea Carl Arpaci-Dusseau Dctr f Philsphy in Cmputer Science University f Califrnia at Berkeley Prfessr David Culler, Chair In this thesis, we frmalize the cncept f an implicitly-cntrlled system, als referred t as an implicit system. In an implicit system, cperating cmpnents d nt explicitly cntact ther cmpnents fr cntrl r state infrmatin; instead, cmpnents infer remte state by bserving naturally-ccurring lcal events and their crrespnding implicit infrmatin, i.e., infrmatin available utside f a deæned interface. Many systems, particularly in distributed and netwrked envirnments, have leveraged implicit cntrl t simplify the implementatin f services with autnmus cmpnents. T cncretely demnstrate the advantages f implicit cntrl, we prpse and implement implicit cscheduling, an algrithm fr dynamically crdinating the time-sharing f cmmunicating prcesses acrss distributed machines. Crdinated scheduling, required fr cmmunicating prcesses t leverage the recent perfrmance imprvements f switchbased netwrks and lw verhead prtcls, has traditinally been achieved with explicit cscheduling. Hwever, implementatins f explicit cscheduling ften suæer frm multiple failure pints, high cntext-switch verheads, and pr interactin with client-server, interactive, and IèO-intensive jbs. Implicit cscheduling supprts nt nly traditinal parallel applicatins n Netwrks f Wrkstatins, but als general-purpse wrklads. By bserving and reacting t implicit infrmatin èe.g., the rund-trip time f request-respnse messagesè, prcesses acrss the system make independent decisins that crdinate their scheduling in a fair and eæcient manner. The principle cmpnent f implicit cscheduling is cnditinal tw-phase waiting, a generalizatin f traditinal tw-phase waiting in which spin-time is nly partially determined befre the prcess begins waiting and may be cnditinally increased depending upn events that ccur while the prcess spins. A secnd imprtant cmpnent is a fair and preemptive lcal perating system scheduler. With simple mdels and analysis, we derive the apprpriate baseline and cnditinal spin amunts fr the waiting algrithm as a functin f system parameters. We shw thrugh simulatin and an implementatin n a cluster f 32 wrkstatins that implicit cscheduling eæciently and fairly handles cmpeting applicatins with a wide range f cmmunicatin characteristics. We predict that mst well-behaved parallel applicatins will perfrm within 5è f ideal explicit cscheduling.

6 Prfessr David Culler Dissertatin Cmmittee Chair 2

7 T a nine-year-ld girl iii

8 iv Cntents List f Figures List f Tables viii xi Intrductin. Mtivatin ::::::::::::::::::::::::::::::::::::.2 Scheduling Requirements :::::::::::::::::::::::::::: 2.2. Prblem Statement :::::::::::::::::::::::::::: 3.3 Previus Appraches ::::::::::::::::::::::::::::::: 4.4 Implicit Cscheduling :::::::::::::::::::::::::::::: 4.4. Lcal Scheduler :::::::::::::::::::::::::::::: Cnditinal Tw-Phase Waiting :::::::::::::::::::: 6.5 Organizatin ::::::::::::::::::::::::::::::::::: 7 2 Implicit Systems 8 2. Cnstructing Traditinal System Services ::::::::::::::::::: Cntrl Structure ::::::::::::::::::::::::::::: Obtaining Infrmatin :::::::::::::::::::::::::: Implicitly-Cntrlled Systems :::::::::::::::::::::::::: Traditinal Surces f Infrmatin ::::::::::::::::::: Implicit Infrmatin ::::::::::::::::::::::::::: 2.3 Examples ::::::::::::::::::::::::::::::::::::: Single-Prcessr Systems :::::::::::::::::::::::: Multiprcessr Systems ::::::::::::::::::::::::: Netwrked Systems :::::::::::::::::::::::::::: Summary ::::::::::::::::::::::::::::::::::::: 7 3 Scheduling Backgrund 9 3. Evlutin f Clusters ::::::::::::::::::::::::::::::: Requirements ::::::::::::::::::::::::::::::::::: Cluster Scheduling :::::::::::::::::::::::::::::::: Lcal Scheduling ::::::::::::::::::::::::::::: Explicit Cscheduling :::::::::::::::::::::::::: Dynamic Cscheduling :::::::::::::::::::::::::: 34

9 v Fair Allcatin acrss a Shared Cluster :::::::::::::::: Summary ::::::::::::::::::::::::::::::::::::: 35 4 Implicit Cscheduling Framewrk Cmpnents f System :::::::::::::::::::::::::::::: Machine Architecture :::::::::::::::::::::::::: Message-Layer :::::::::::::::::::::::::::::: User Prcesses :::::::::::::::::::::::::::::: Applicatin Wrklad :::::::::::::::::::::::::: Operating System Scheduler ::::::::::::::::::::::: Cmpnents f Implicit Cscheduling ::::::::::::::::::::: Interactin between Prcesses and the Scheduler ::::::::::: Interactin between Cmmunicating Prcesses :::::::::::: 43 5 Lcal Scheduler Requirements ::::::::::::::::::::::::::::::::::: Preemptin :::::::::::::::::::::::::::::::: Amrtized Cntext-Switches :::::::::::::::::::::: Fair Cst-Mdel ::::::::::::::::::::::::::::: Multilevel Feedback Queue Schedulers ::::::::::::::::::::: Overview f Slaris Time-Sharing Scheduler :::::::::::::: Preemptin :::::::::::::::::::::::::::::::: Amrtized Cntext-Switches :::::::::::::::::::::: Fair Cst-Mdel ::::::::::::::::::::::::::::: Prprtinal-Share Schedulers :::::::::::::::::::::::::: Overview f Stride Schedulers :::::::::::::::::::::: Preemptin :::::::::::::::::::::::::::::::: Amrtized Cntext-Switches :::::::::::::::::::::: Fair Cst-Mdel ::::::::::::::::::::::::::::: Scheduling in the Cluster :::::::::::::::::::::::: Summary ::::::::::::::::::::::::::::::::::::: 6 6 Cst-Beneæt Analysis f Waiting Cnditinal Tw-Phase Waiting ::::::::::::::::::::::::: Maintaining Crdinatin with Destinatins :::::::::::::::::: Request-Respnse :::::::::::::::::::::::::::: One-Way Requests :::::::::::::::::::::::::::: All-t-all Synchrnizatin :::::::::::::::::::::::: Discussin ::::::::::::::::::::::::::::::::: Maintaining Crdinatin with Senders :::::::::::::::::::: Incming Request-Respnse ::::::::::::::::::::::: Incming One-way Requests ::::::::::::::::::::::: Incming Synchrnizatin Messages :::::::::::::::::: Discussin ::::::::::::::::::::::::::::::::: Optimizatins fr Lng Waiting Times ::::::::::::::::::::: 77

10 vi 6.4. Netwrk Latency ::::::::::::::::::::::::::::: Lad-Imbalance :::::::::::::::::::::::::::::: Summary ::::::::::::::::::::::::::::::::::::: 83 7 Simulatin Envirnment Machine Architecture :::::::::::::::::::::::::::::: Message Layer :::::::::::::::::::::::::::::::::: User Prcesses :::::::::::::::::::::::::::::::::: Cmmunicatin Primitives ::::::::::::::::::::::: Waiting Algrithm :::::::::::::::::::::::::::: Applicatin Wrklad :::::::::::::::::::::::::::::: Operating System Scheduler ::::::::::::::::::::::::::: Class Independent Functinality :::::::::::::::::::: Scheduling Classes :::::::::::::::::::::::::::: 97 8 Simulatin Results Scheduling Crdinatin ::::::::::::::::::::::::::::: Sensitivity t Netwrk Latency ::::::::::::::::::::: Sensitivity twrklad Parameters :::::::::::::::::: Discussin ::::::::::::::::::::::::::::::::: Baseline Spin ::::::::::::::::::::::::::::::::::: Bulk-Synchrnus Cmmunicatin ::::::::::::::::::: Cntinuus-Cmmunicatin Wrklads :::::::::::::::: Discussin ::::::::::::::::::::::::::::::::: Cnditinal Spin ::::::::::::::::::::::::::::::::: Bulk-Synchrnus Wrklads :::::::::::::::::::::: Cntinuus-Cmmunicatin Wrklads :::::::::::::::: Discussin ::::::::::::::::::::::::::::::::: Lad-Imbalance :::::::::::::::::::::::::::::::::: Bulk-Synchrnus Wrklads :::::::::::::::::::::: Cntinuus-Cmmunicatin Wrklads :::::::::::::::: Apprximating Lad-Imbalance ::::::::::::::::::::: Discussin ::::::::::::::::::::::::::::::::: Lcal Scheduler :::::::::::::::::::::::::::::::::: Bulk-Synchrnus Wrklads :::::::::::::::::::::: Cntinuus-Cmmunicatin Wrklads :::::::::::::::: Discussin ::::::::::::::::::::::::::::::::: Summary ::::::::::::::::::::::::::::::::::::: 37 9 Prttype Implementatin System Architecture ::::::::::::::::::::::::::::::: Message Layer :::::::::::::::::::::::::::::::::: User Prcesses :::::::::::::::::::::::::::::::::: Cmmunicatin Primitives ::::::::::::::::::::::: Waiting Algrithm :::::::::::::::::::::::::::: 42

11 vii 9.4 Applicatin Wrklad :::::::::::::::::::::::::::::: Synthetic Applicatins :::::::::::::::::::::::::: Real Split-C Applicatins :::::::::::::::::::::::: Operating System Scheduler ::::::::::::::::::::::::::: Lcal Schedulers ::::::::::::::::::::::::::::: Explicit Cscheduling :::::::::::::::::::::::::: Jb Placement :::::::::::::::::::::::::::::: 54 0 Implementatin Study Veriæcatin f Simulatins :::::::::::::::::::::::::::: Baseline Spin ::::::::::::::::::::::::::::::: Cnditinal Spin ::::::::::::::::::::::::::::: Lcal Scheduler :::::::::::::::::::::::::::::: Range f Wrklads ::::::::::::::::::::::::::::::: Additinal Cmmunicatin Primitives ::::::::::::::::: Real Applicatins ::::::::::::::::::::::::::::: Jb Scalability :::::::::::::::::::::::::::::: Wrkstatin Scalability ::::::::::::::::::::::::: Jb Placement :::::::::::::::::::::::::::::: Summary ::::::::::::::::::::::::::::::::::::: Implementatin Perfrmance :::::::::::::::::::::: Cmparisn t Simulatin Predictins ::::::::::::::::: Advice fr Prgrammers ::::::::::::::::::::::::: 83 Cnclusins 84. Summary ::::::::::::::::::::::::::::::::::::: 84.. Cnditinal Tw-Phase Waiting :::::::::::::::::::: Lcal Operating System Scheduler ::::::::::::::::::: Perfrmance ::::::::::::::::::::::::::::::: 85.2 Future Wrk ::::::::::::::::::::::::::::::::::: Implementatin Issues :::::::::::::::::::::::::: Prgramming Mdel ::::::::::::::::::::::::::: Wrklads ::::::::::::::::::::::::::::::::: Jb Allcatin and Placement ::::::::::::::::::::: Theretical Analysis ::::::::::::::::::::::::::: Implicit Systems ::::::::::::::::::::::::::::: 92 Bibligraphy 93

12 viii List f Figures 2. Mdel f an Implicitly-Cntrlled System. ::::::::::::::::::: 3. Lcal Scheduling. ::::::::::::::::::::::::::::::::: Simulated Perfrmance f Lcal Scheduling versus Cscheduling. :::::: Explicit Cscheduling Matrix. :::::::::::::::::::::::::: Impact f Interactive Prcesses n Parallel Jbs. ::::::::::::::: Transfer f Infrmatin in Implicit Cscheduling. ::::::::::::::: Measured Fairness with Slaris Time-Sharing Scheduler. ::::::::::: Basic Stride Scheduling. ::::::::::::::::::::::::::::: Stride-Scheduling with System Credit Extensin. ::::::::::::::: Stride-Scheduling with Lan & Brrw Extensin. :::::::::::::: Fairness in Cluster with Independent Stride Schedulers. ::::::::::: Fairness in Cluster with Cperating Stride Schedulers. ::::::::::: Measured Fairness with Ticket Server and Stride Scheduler. ::::::::: Time fr Request-Respnse Message with Remte Prcess Scheduled. : : : Time fr Request-Respnse Message with Triggering f Remte Prcess. :: Time fr All-t-all Synchrnizatin with Prcesses Scheduled. :::::::: Time fr All-t-all Synchrnizatin with Triggering f Rt Prcess. :::: Cst f Incming Request-Respnse Messages if Lcal Prcess Spin-Waits. : Cst f Incming Request-Respnse Messages if Lcal Prcess Blcks. : : : Cst f Incming One-Way Requests if Lcal Prcess Spin-Waits. :::::: Cst f One-Way Requests if Lcal Prcess Blcks. :::::::::::::: Cst with Netwrk Latency when Prcesses are Uncrdinated and Blck. : Cst with Netwrk Latency when Prcesses are Crdinated and Spin-Wait Mdel f Bulk-Synchrnus Synthetic Parallel Applicatins. ::::::::: Characteristics f Bulk-Synchrnus Benchmarks. ::::::::::::::: Mdel f Cntinuus-Cmmunicatin Synthetic Parallel Applicatins. : : : Characteristics f Cntinuus-Cmmunicatin Benchmarks. ::::::::: Impact f Latency and Cntext-Switch Time n Cntinuus-Cmmunicatin Wrklads. :::::::::::::::::::::::::::::::::::: 0

13 ix 8.2 Impact f Latency and Cntext-Switch Time n Bulk-Synchrnus Wrklads Perfrmance f Immediate Blcking fr Bulk-Synchrnus Prgrams. :::: Perfrmance f Immediate Blcking fr Cntinuus-Cmmunicatin Prgrams Sensitivity t Read Baseline Spin fr Bulk-Synchrnus Prgrams. ::::: Sensitivity t Barrier Baseline Spin fr Bulk-Synchrnus Prgrams. :::: Sensitivity t Baseline Spin fr Cntinuus-Cmmunicatin Prgrams. : : : 8.8 Sensitivity t Cnditinal Spin fr Cntinuus-Cmmunicatin Prgrams. : Clseups f Sensitivity t Cnditinal Spin Amunt èw = 50çsè. :::::: Clseups f Sensitivity t Cnditinal Spin Amunt èw = 200çsè. ::::: 6 8. Sensitivity t Cnditinal Spin fr Cntinuus-Cmmunicatin Prgrams with Frequent Barriers. ::::::::::::::::::::::::::::: Sensitivity t Baseline Spin fr Bulk-Synchrnus Prgrams with Lad-Imbalance èw =50çsè. ::::::::::::::::::::::::::::::::::: Sensitivity t Baseline Spin fr Bulk-Synchrnus Prgrams with Lad-Imbalance èw = 200çsè. ::::::::::::::::::::::::::::::::::: Sensitivity t Baseline Spin fr Cntinuus-Cmmunicatin Prgrams with Lad-Imbalance èw = 50çsè. :::::::::::::::::::::::::: Sensitivity t Baseline Spin fr Bulk-Synchrnus Prgrams with Lad-Imbalance èw = 200çsè. ::::::::::::::::::::::::::::::::::: Perfrmance with Glbal Apprximatin f Lad-Imbalance fr Bulk-Synchrnus Prgrams. ::::::::::::::::::::::::::::::::::::: Fairness fr Bulk-Synchrnus Prgrams with Identical Placement è3 Jbsè Fairness fr Bulk-Synchrnus Prgrams with Identical Placement. ::::: Fairness fr Bulk-Synchrnus Prgrams with Randm Placement. ::::: Fairness fr Cntinuus-Cmmunicatin Prgrams èg = 00msè. :::::: Fairness fr Cntinuus-Cmmunicatin Prgrams èg = sè. ::::::::: Internals f Ultra Wrkstatin. :::::::::::::::::::::::: Netwrk Tplgy fr Cluster f 32 Wrkstatins. :::::::::::::: Micrbenchmark Results fr Read Baseline Spin. ::::::::::::::: Micrbenchmark Results fr Linear Barrier Baseline Spin. :::::::::: Cmmunicatin Characteristics f Radix. ::::::::::::::::::: Cmmunicatin Characteristics f EM3D. ::::::::::::::::::: Cmmunicatin Characteristics f Radix:Small. :::::::::::::::: Cmmunicatin Characteristics f EM3D:Small. ::::::::::::::: 5 0. Sensitivity t Baseline Spin fr Bulk-Synchrnus Prgrams. :::::::: Perfrmance n Bulk-Synchrnus Wrklads. ::::::::::::::::: Sensitivity twaiting Algrithm fr Cntinuus-Cmmunicatin Prgrams Perfrmance f Cntinuus-Cmmunicatin Prgrams with Request-Respnse Messages. ::::::::::::::::::::::::::::::::::::: Fairness with Bulk-Synchrnus Prgrams. :::::::::::::::::: Fairness with Cntinuus-Cmmunicatin Prgrams. ::::::::::::: Perfrmance f Cntinuus-Cmmunicatin Prgrams with One-Way Requests Perfrmance f Cntinuus-Cmmunicatin with Bulk Messages. :::::: 7

14 0.9 Perfrmance f Real Split-C Applicatins. ::::::::::::::::::: Jb Scalability with Bulk-Synchrnus Prgrams. ::::::::::::::: 74 0.Jb Scalability with Cntinuus-Cmmunicatin Prgrams. ::::::::: Wrkstatin Scalability. ::::::::::::::::::::::::::::: Sensitivity t Jb Placement. :::::::::::::::::::::::::: 80 x

15 xi List f Tables 4. System Parameters. ::::::::::::::::::::::::::::::: Wrklad Parameters. :::::::::::::::::::::::::::::: 4 5. Slaris 2.6 Time-Sharing Default Dispatch Table. ::::::::::::::: Cnditinal Tw-Phase Waiting Parameters fr Implicit Cscheduling. : : : Measured Netwrk Latency. ::::::::::::::::::::::::::: System Parameters in Simulatins. ::::::::::::::::::::::: Cnditinal Tw-Phase Waiting Parameters fr Simulatin. ::::::::: Measured Cntext-Switch Cst. :::::::::::::::::::::::: System Parameters in Implementatin. ::::::::::::::::::::: Cnditinal Tw-Phase Waiting Parameters fr Implementatin. :::::: Cmmunicatin Characteristics f Benchmark Applicatins. ::::::::: 48

16 xii Acknwledgements Iwuld like t begin by thanking my advisr, Prfessr David Culler, whm I greatly admire and respect. Receiving a cmplement frm David is a reward in itself. David is an inspiring persn t wrk with í always enthusiastic and ready t lk at anther set f graphs and hypthesize abut what is happening. I appreciate that whenever I ask him an ambiguus questin, he assumes I am asking the deeper questin èwhether r nt that is the caseè. Despite all f my previus cmplaints and gripes, I am grateful fr the experience f being a part f the Netwrk f Wrkstatins prject. In additin t my advisr, I thank Prfessrs Tm Andersn and David Pattersn fr leading ur large grup. I culd nt have implemented any f this wrk withut the GLUnix and AM-II sftware prvided by Dug Ghrmley and Alan Mainwaring, respectively. I thank them and all f the students fr the infrastructure they prvided, as well as the general shared experience. I ænd it surprising hw much I already miss the meetings and the retreats. I'd als like t thank my cmmittee members fr their feedback n this dissertatin. Prfessrs Kathy Yelick, Tm Andersn, and David Freedman gave mevaluable feedback and great perspectives n this dcument, as well as previus related papers. Thanks t Prfessr Je Hellerstein fr general advice and encuragement. I thank the lder graduate students in David's grup, Thrsten vn Eicken, Klaus Schauser, and Seth Gldstein fr being inspiring in their accmplishments, if nt a bit intimidating. But, mst f all, I thank them fr acquiring the crner windw æce back in Evans Hall. Thanks as well t Steve Lumetta, Alan Mainwaring, and Girija Narlikar, fr sharing æce space, spare change, technical discussins, and ccasinal wrries. The students and faculty wh participated in David Culler's System f System graduate class in the Fall f 997 gave meinteresting perspectives n implicit systems and helped me t think abut the general issues. In particular, I remember useful cmments made by Armand Fx, Steve Gribble, David Wagner, and Prfessr Jseph Hellerstein. I still think very fndly back tmy undergraduate days at Carnegie Melln University. In thse days, I spent a great deal f time and energy questining whether r nt Iwanted t remain in the Cmputer Engineering department. Bb Barker and Prfessrs Bill Birmingham, Daniel Siewirek, Jhn Shen, and Jim Hburg all gave me encuragement andèr research pprtunities at key decisin pints. I dubt they knw hwmuch their actins inæuenced the directin f my life. Thanks t Vanessa Karubian fr giving me the pprtunity tvlunteer at the Cmputer Club at Hawthrne Elementary Schl. She has a gift fr bth making things happen and sharing the credit with thers. Thanks t the furth and æfth graders fr reminding me f the imprtance f attitude and f hw t ænd signiæcant meaning in wrk. I am grateful t my parents fr the envirnment they raised me in. Ever since my dad tk me t his wrk and let me play Adventure while drinking ht chclate, the path has been set. I thank my parents fr managing t simultaneusly cnvey that they will always lve me n matter what I d and that they are s prud f my accmplishments. I will always remember hw happy my mm sunded when I tld her I had been accepted at Berkeley and then seeing her seven years later wiping tears frm her eyes at my graduatin.

17 Finally, there are n suæcient wrds fr me t express my indebtedness t my husband, Remzi. Hw many years will it take me t understand? S slwly I realize hw prfundly lucky I am t be with yu and t wrk with yu. T brrw frm Gethe, yu are my true measure f all that is gd: supprt, faith, and, mst f all, lve. xiii

18 Chapter Intrductin Building system services in a distributed envirnment is a diæcult task; the primary challenge is that the cmpnents implementing the service must knw the current state f the entire system in rder t act in a cperative manner. Since the distributed cmpnents d nt inherently have this glbal knwledge, they usually explicitly query a set f remte cmpnents t gather this infrmatin. Unfrtunately, adding mre cmmunicatin t the distributed system can cause several prblems: the transfer f cntrl infrmatin may be cstly r cnsume critical resurces, the infrmatin may be ut-f-date by the time it is received, the desired interface may nt be available, r errrs may ccur while the infrmatin is being transmitted. The thesis f this wrk is that implicit cntrl simpliæes the cnstructin f distributed system services; with implicit cntrl, cmpnents neither query remte cmpnents fr infrmatin nr explicitly cntrl remte cmpnents in their actins. Instead, autnmus cmpnents infer remte state by bserving naturally-ccurring lcal events and their crrespnding implicit infrmatin, i.e., infrmatin available utside f a deæned interface. An example f a naturally-ccurring event is the arrival f a data request frm a remte nde; the crrespnding implicit infrmatin is the arrival rate f the incming requests. Thus, an implicit system cntains n additinal cmmunicatin beynd that which is inherent in cnstructing the service. This chapter presents a high-level synpsis f this thesis. The ærst sectin describes the requirements fr services in a distributed system, in particular a scheduler fr cmmunicating prcesses in a Netwrk f Wrkstatins. This is fllwed by a brief summary f related wrk. The third sectin gives a descriptin f the cmpnents and cntributins f implicit cscheduling. The chapter cncludes with an verview f the rganizatin f the remainder f this thesis.. Mtivatin The imprtance and the prevalence f distributed systems is cntinuing t grw. One emerging platfrm f particular imprtance are Netwrks f Wrkstatins ènowsè r clusters, which are tightly-cupled cllectins f cmmdity wrkstatins èr persnal cmputersè cnnected with a switch-based, high-bandwidth netwrk. NOWs represent a

19 2 cst-eæective way f building high-perfrmance, scalable servers ë4, 9, 37, 6, 75, 43, 54, 75ë. Unlike previus distributed systems which supprted nly lng-running, sequential applicatins ë04, 78ë, mdern clusters can eæciently supprt a variety f jb classes. First, due t the advent f lw-latency, high-bandwidth, switch-based netwrks ë2, 2ë and lw-verhead message prtcls ë, 0, 36, 65, 66ë, NOWs can execute æne-grain parallel applicatins ë36, 6ë. Secnd, clusters can imprve the client-server applicatins f traditinal distributed systems èe.g., naming, lcking, and æle servicesè by updating these services t mdern cmmunicatin prtcls ë5ë. Third, acting as a shared server acrss many cmpeting users, NOWs can time-share the interactive, develpmental jbs fund in general-purpse wrklads ë7ë. Finally, clusters cnstitute an excellent platfrm fr applicatins with large IèO demands ë9, 43ë..2 Scheduling Requirements Due t the presence f this wide range f jb classes, scheduling and resurce allcatin is a much mre diæcult prblem. In additin t achieving high thrughput fr lng-running applicatins, the perating system scheduler in the cluster must prvide the fllwing criteria. æ Crdinated Scheduling: Cmmunicating prcesses frm the same parallel jb must be scheduled simultaneusly acrss a set f wrkstatins t achieve eæcient perfrmance. Only if prcesses are crdinated can cmmunicatin prceed at the rate f the underlying hardware withut incurring a cntext-switch. æ Dynamic Identiæcatin: Cmmunicating prcesses must be identiæed at run-time t supprt client-server applicatins and prcesses that cmmunicate with a wide set f prcesses ver their life-time. æ Fast Respnse Time: Interactive jbs must receive quick respnse time, while negligibly harming the thrughput f lng-running parallel jbs. æ Eæcient Resurce Usage: Jbs that are waiting fr events èe.g., disk r user IèOè shuld relinquish the prcessr s that they d nt waste resurces. æ Fair Allcatin: Since the clustered envirnment is shared acrss many cmpeting users each running jbs with diæerent characteristics, the amunt f resurces allcated t each user shuld be independent f the number f jbs each user runs and the cmmunicatin behavir f thse jbs. Further, all system services shuld have the fllwing structural qualities. æ Autnmus: Each nde shuld maintain cntrl ver its wn actins, participating in the system because it is in its best interest t d s, nt by impsitin; participating in the cluster shuld nt degrade the abslute perfrmance f the single nde.

20 3 æ Recnægurable: Wrkstatins shuld be able t jin and leave the cluster dynamically withut restarting the system service. æ Reliable: The service shuld tlerate the failure f any nde f the system..2. Prblem Statement In this thesis, we investigate the design and implementatin f crdinated scheduling as an implicitly-cntrlled distributed service. Traditinally, crdinated time-sharing f cperating prcesses acrss the ndes f a cluster has required explicit cntrl; fr example, in explicit cscheduling ë34ë, r gang-scheduling, a set f master cmpnents determines a glbal schedule f prcesses ver time, and explicitly cmmunicates this schedule t the participating lcal schedulers. With implicit cscheduling, cperating prcesses achieve crdinated scheduling withut explicit cntrl r additinal cmmunicatin. The primary gal f implicit cscheduling and f this thesis is btaining fair, crdinated scheduling fr cmmunicating prcesses in traditinal parallel applicatins. We believe that this is the ærst and mst diæcult step in cnstructing a system that can supprt mre cmplete wrklads. Therefre, while implicit cscheduling has been designed t supprt the additinal needs f a general-purpse wrklad with client-server, interactive, and IèO-bund applicatins, we have nt yet evaluated such wrklads. We refer t a set f cmmunicating prcesses as a jb. This jb may be a dynamic cllectin f cmmunicating prcesses èe.g., a server and multiple clientsè r a predeæned, static cllectin èe.g., a traditinal parallel applicatinè. The set f prcesses within a jb are cperating prcesses, while the thers in the system are cmpeting prcesses. The scheduling f parallel jbs in a multiprgrammed envirnment has lng been an active area f research. The prblem is typically decmpsed int tw steps: allcatin, where individual prcesses are placed n prcessrs, and dispatching, where thse prcesses are scheduled ver time. The allcatin step fr parallel jbs has been investigated in detail ë3, 33, 66, 70, 0, 8, 9, 20, 26, 39, 49ë. Given the ppular single-prgrammultiple-data èspmdè parallel prgramming mdel, it has generally been fund that the respnse time and thrughput f the wrklad are best when cmpeting prcesses frm diæerent jbs can share the same prcessr ë35, 53, 82, 05, 44, 48, 73ë. In this wrk, we fcus n the secnd step f dispatching the cmmunicating prcesses ver time. We cnsider the prblem after the parallel jbs have been divided int a æxed number f cmmunicating prcesses and after thse prcesses have been placed n a set f prcessrs. We evaluate allcatins where nly cmpeting prcesses, and nt cperating prcesses, share the same prcessr. Further, we assume that the allcatin step has accunted fr all memry cnsideratins. Whether r nt prcesses migrate between prcessrs ver time is rthgnal t ur wrk. Fr ur prgramming mdel, we assume that cperating SPMD prcesses cmmunicate with lw-level messages, such as thse deæned by the Active Message mdel ë65ë. We assume that the receiving prcess must be scheduled t handle a message and t return a respnse. We cnsider applicatins written with request-respnse and ne-way cmmunicatin peratins and with barrier synchrnizatin peratins. Finally, we assume that we have n cntrl ver the cmmunicatin r synchrnizatin perfrmed by the applicatins.

21 4.3 Previus Appraches Previus wrk in this area f time-sharing parallel wrklads has examined tw distinct appraches: lcal scheduling and explicit cscheduling. Each has signiæcant weaknesses that render it unæt fr the NOW envirnment and wrklad. The mst straight-frward apprach fr time-sharing jbs is fr the perating system scheduler n each wrkstatin t schedule its allcated prcesses independently. While this simple apprach flcal scheduling has the desired structural qualities f a distributed service èi.e., autnmy, recnægurability, and reliabilityè, it fails the mst imprtant criteria: perfrmance. Because cmmunicating prcesses are nt scheduled in a crdinated fashin, each machine spends a signiæcant prtin f its time cntext-switching between prcesses; the resulting perfrmance fr æne-grain parallel applicatins is nt acceptable. Over the years, numerus researchers have fund that the slwdwn f lcal scheduling may be rders f magnitude wrse than ideal fr frequently cmmunicating prcesses ë7, 35, 55, 56, 70, 99ë. T reslve the perfrmance ineæciencies f lcal scheduling, cscheduling ë34ë r gang scheduling, explicitly schedules the cperating prcesses frm a single jb simultaneusly acrss prcessrs. A strict rund-rbin glbal schedule f prcesses is cnstructed, and a glbal cntext-switch is perfrmed acrss all schedulers at the same time. With cscheduling, each jb is given the impressin that it is running n a dedicated machine. We refer t this traditinal frm f cscheduling as explicit cscheduling. Hwever, explicit cscheduling als fails t meet many f the requirements f ur distributed system service. Straight-frward implementatins are neither scalable nr reliable; hierarchical cnstructins can remve single pints-f-failure and scalability bttlenecks, but nly with increased implementatin cmplexity ë54ë. In all cases, cmpnents must act as slaves, rather than in an autnmus manner. Diæculties als ccur when using explicit cscheduling with a mixed, general-purpse wrklad: explicit cscheduling requires that the schedule f cmmunicating prcesses be precmputed and the strict rund-rbin schedule perfrms prly with interactive jbs and with jbs initiating IèO ë7, 0, 49, 00ë..4 Implicit Cscheduling T meet the requirements f a time-shared scheduling apprach fr general-purpse wrklads n netwrks f wrkstatins, we have develped implicit cscheduling. Implicit cscheduling requires n additinal cmmunicatin between cmpnents, either t determine remte state r t direct ther cmpnents in their actins. Instead, each prcess determines fr itself when it is beneæcial t be scheduled with implicit infrmatin and shares this decisin with the lcal perating system scheduler. With implicit cscheduling, it is advantageus fr a prcess t remain scheduled when it is crdinated with remte, cperating prcesses. A prcess determines whether scheduling is crdinated with tw pieces f lcally-visible implicit infrmatin. The ærst piece f implicit infrmatin is the bserved rund-trip time f request-respnse messages naturally ccurring within the applicatin; in general, receiving a fast respnse t a request message implies that the remte destinatin prcess is scheduled, whereas receiving a slw respnse implies that the remte prcess is nt scheduled. The secnd piece f implicit

22 5 infrmatin is the arrival rate f incming messages; a message arrival implies that the remte sending prcess is scheduled. After a prcess has determined whether scheduling is crdinated and whether it shuld remain scheduled, it infrms the lcal perating system scheduler. If it is beneæcial t the jb as a whle fr the prcess t be scheduled, then the prcess simply remains runnable. If it is nt beneæcial, then the prcess vluntarily relinquishes the prcessr and sleeps until it is s. Thus, n changes are required t the interface f the lcal scheduler. The lcal scheduler can thus remain tuned fr sequential wrklads cntaining interactive and IèO-bund jbs. By cnstructin, implicit cscheduling has the three structural prperties desired fr any distributed system service: autnmy, recnægurability, and reliability. Because the cmpnents running n each wrkstatin in the cluster act autnmusly and rely n nly implicit infrmatin, ndes can enter and exit the system at any time èwhether due t chice r t failureè, withut impacting the service n ther ndes. Achieving distributed crdinatin with implicit cscheduling requires twkey cmpnents. First, the perating system scheduler running n each machine in the cluster must prvide a fair cst-mdel t user prcesses; this cst-mdel allws each prcess t calculate whether it shuld cmpete fr the CPU at the current time. Secnd, each prcess must use cnditinal tw-phase waiting when dependent nanevent frm a remte prcess èe.g., when the prcess must wait until it receives a respnse t a previus request befre it can cntinue cmputingè. We nw describe this tw cmpnents in mre detail..4. Lcal Scheduler The ærst key t implicit cscheduling is the behavir f the lcal perating system scheduler. T btain gd wrklad thrughput, fair allcatin acrss jbs cmmunicating at diæerent rates, and fair allcatin acrss users running diæerent numbers f jbs, the perating system scheduler must have the fllwing three prperties: æ Preemptin. Due t the interdependencies f cmmunicating prcesses acrss machines, the scheduler must be willing t preempt a prcess t prevent deadlck; a simple rund-rbin scheduler is suæcient. Perfrmance may be als imprved if the scheduler immediately dispatches prcesses when they becme runnable due t message arrivals. æ Amrtized Cntext-Switch. T amrtize the cst f btaining crdinatin acrss machines, the duratin f a time-slice shuld be lng relative t a cntext-switch. æ Fair Cst-Mdel. The lcal scheduler shuld exprt a well-deæned cst-mdel s that prcesses use the CPU nly when it is beneæcial fr them t d s. This als allws higher-level plicies t prvide fair allcatins t cmpeting users acrss the cluster. In this thesis, we describe and implement a lcal scheduler that achieves these three prperties. Our apprach is based n extensins t a stride scheduler, a prprtinal-share scheduler intrduced by Waldspurger ë68ë, in which resurces are allcated t a prcess

23 6 in prprtin t its number f tickets. The primary diæerence in ur scheduler is that prcesses receive the same amunt f resurces regardless f their cmputatinal behavir èi.e., when they run versus when they sleepè. This is accmplished by giving exhaustible tickets ë7ë t prcesses that relinquished the prcessr in the past and did nt receive their prprtinal-share. With mre tickets, these prcesses are then scheduled mre frequently in the future; thus, prcesses that cmmunicate èand subsequently sleepè at diæerent rates are scheduled fairly..4.2 Cnditinal Tw-Phase Waiting The secnd key t implicit cscheduling is the behavir f prcesses that are waiting fr cmmunicatin and synchrnizatin t cmplete. Prcesses can achieve crdinated scheduling by simply deciding t spin r t blck when waiting fr an peratin; T determine the crrect actin, each prcess applies its knwledge f the current scheduling state f the cluster t the scheduler's cst mdel. Cnditinal tw-phase waiting incrprates this knwledge. In this dissertatin, we intrduce cnditinal tw-phase waiting as a generalizatin f tw-phase waiting ë34ë. In the ærst phase f tw-phase waiting, the prcess spins fr sme baseline amunt f time while waiting fr the desired event; if the event des nt ccur, then, in the secnd phase, the prcess vluntarily relinquishes the prcessr and blcks. Unlike traditinal tw-phase waiting where the spin-time is determined befre the prcess begins waiting, with cnditinal waiting the prcess may dynamically increase its waiting time, depending upn bserved events. Gd perfrmance with implicit cscheduling requires picking the crrect amunt f baseline and cnditinal spin-time in the ærst phase f the waiting algrithm. In previus research, the desired spin-time fr tw-phase waiting has been calculated with cmpetitive analysis, assuming that waiting times are chsen by an adversary ë85, 86, 02ë. Hwever, in ur envirnment, it is nt apprpriate t assume that waiting times are adversarial. Instead, prcesses can inæuence future waiting times advantageusly by acting in a cperative manner. We have fund that there are three factrs t cnsider in cnditinal waiting. æ Baseline Spin: Prcesses shuld wait fr the expected wrst-case cmpletin time f the peratin that ccurs when the arrival f a message triggers the scheduling f a remte prcess. Spinning lng enugh t maintain crdinatin increases the likelihd that future waiting times will als be lw, which imprves cmmunicatin and synchrnizatin perfrmance. æ Cnditinal Spin: Prcesses shuld spin lnger when receiving messages. At sme threshld incming message rate, it is beneæcial fr the waiting prcess t remain scheduled, even thugh it is nt making frward prgress, because it is handling incming messages and allwing remte cperating prcesses t make prgress. æ Lng Waiting Times: In sme cases, by relinquishing the prcessr rather than waiting fr slw peratins t cmplete, a prcess may achieve better perfrmance with implicit cscheduling than is pssible with explicit cscheduling. These cases

24 7 ccur when netwrk latency r the internal lad-imbalance f the applicatin is high relative t the cst f a cntext switch..5 Organizatin In the fllwing tw chapters, we cver the backgrund material fr this thesis. Speciæcally, in Chapter 2 we describe the philsphy f implicit systems and give examples f implicitly-cntrlled systems and implicit infrmatin in ther areas. In Chapter 3, we discuss previus research in scheduling cmmunicating prcesses in multiprgrammed distributed and parallel systems. In the next three chapters, we discuss implicit cscheduling in detail. In Chapter 4 we describe ur mdel f the system and the available implicit infrmatin. We discuss the requirements f the lcal perating system scheduler in Chapter 5 and describe extensins t an existing scheduler with the desired prperties. In Chapter 6 we present the tw-phase waiting algrithm emplyed by prcesses when they cmmunicate and derive the desired amunt f spin-time. In the ænal set f chapters we cver the perfrmance f implicit cscheduling fr a variety f wrklads and envirnments. After describing ur simulatin envirnment in Chapter 7, we present ur simulatin results in Chapter 8. In Chapter 9 we describe ur implementatin n the U.C. Berkeley NOW cluster; we present measurements f ur system in Chapter 0. We discuss ur cnclusins and extensins fr future research in Chapter.

25 8 Chapter 2 Implicit Systems ëwhere there's smke, there's ære." í English Prverb Mst interesting systems cntain multiple cmpnents that interact with ne anther t implement a cmmn service. T interact in a cperative manner, each cmpnent usually requires infrmatin abut the state and behavir f ther cmpnents. Cnstructing such systems is ften quite cmplicated, due t the cmpnents' lack f perfect knwledge f remte cmpnents. The idea behind an implicitly-cntrlled system is that rather than explicitly cntacting ther cmpnents t btain infrmatin, cmpnents infer the state f ther cmpnents frm lcal bservatins f naturally-ccurring events. Rather than bey cmmands frm a cmpnent with cmplete cntrl ver the system, each cmpnent reacts independently t the frces that are inæuencing it. In this chapter, we describe the prperties and utility f implicit infrmatin fr building cmplex services, especially services in distributed systems. We begin by deæning the cllectin f cmpnents in a system, their cmmn structure, and the interfaces fr exchanging infrmatin between cmpnents. We then describe implicitly-cntrlled systems and implicit infrmatin. We cnclude by citing examples f systems that are implicitly cntrlled r leverage implicit infrmatin. 2. Cnstructing Traditinal System Services Hardware and sftware systems are cllectins f cmpnents, r mdules, that interact with ne anther t implement a cmmn set f services. Cmplex systems are divided int multiple, interacting cmpnents fr a variety f reasns. First, mdules can hide unnecessary details, simplifying the cnceptualizatin and maintenance f the verall system ë37ë. Secnd, cmpnents encapsulate a particular implementatin, allwing designers t ptimize fr that implementatin while preserving cmpatibility when underlying assumptins change. Finally, physical limitatins may restrict the size f an individual cmpnent; fr example, in hardware systems, nly a certain number f transistrs may æt n a single chip.

26 9 Cmpnents exist at many diæerent levels in bth hardware and sftware systems. Fr example, in a hardware system, each chip can be a cnsidered an individual cmpnent. Multiple chips can als be jined tgether t frm higher-level cmpnents, such asthe CPU and memry subsystems within a wrkstatin. Finally, each wrkstatin can frm a cmpnent in a parallel r distributed system. Likewise, in a sftware system, each prcedure can be cnsidered a cmpnent. Multiple prcedures can be jined tgether t frm an applicatin r a library. The user applicatin, run-time libraries, and perating system are all individual cmpnents n a single wrkstatin, yet the cllectin f sftware n each nde can act as a single cmpnent in a distributed system. In this dissertatin, we fcus n distributed sftware systems cntaining large numbers f similar cmpnents. 2.. Cntrl Structure Cperating cmpnents implement a cmmn service by cmmunicating with ne anther. Cmmunicatin can be divided int data and cntrl traæc ë88, 64ë. Data traæc is that which is inherently required t implement the service, even if implemented by a single cmpnent. Cntrl traæc represents all additinal traæc, such as that required t btain the internal state f anther cmpnent r t direct anther cmpnent in its actins. In general, the cmpnents that cntrl the decisins f the system determine the structure f the system. The cntrl and cmmunicatin structure f cmpnents can be classiæed int tw main grups: master-slave and peer-t-peer. With the master-slave structure, cmpnents behaving as masters directly cmmand the actins f ther cmpnents, the slaves. The mst straight-frward design is centralized, where a single master cntrls all slaves. Hwever, a centralized master can have severe perfrmance and reliability restrictins in a distributed system; therefre, a number f variatins exist. First, the design can be mdiæed slightly such that there is a parallel master, a grup f cmpnents acting as a uniæed master. Secnd, in a hierarchical design, a cmpnent may be bth a slave t higher-level ndes and a master t lwer-level ndes. While these mre cmplicated structures can imprve perfrmance and reliability, they may still break dwn at sme scale; fr very large-scale systems, r when the state f the system is changing at such a rapid pace that a master cannt determine quickly enugh what is best fr the entire system, an alternative is needed. In a peer-t-peer design, each autnmus cmpnent is respnsible fr ptimizing its speciæc part f the prblem. This design is inherently mre scalable and reliable than master-slave appraches. The primary challenge f a peer-t-peer design is fr each cmpnent t btain an accurate view f the state f the glbal system. We nw discuss traditinal ways in which cmpnents gather such infrmatin Obtaining Infrmatin Regardless f the structure f the system, t gather infrmatin, cmpnents usually cmmunicate thrugh well-deæned explicit interfaces. Fr example, a lcal cmpnent

27 0 sends a message t a remte cmpnent, querying it abut a particular variable. Hwever, a number f drawbacks exist t this explicit apprach. First, nt all relevant infrmatin may be btainable frm the interface. In sftware systems, whether infrmatin is available depends primarily upn whether the cmpnent designer anticipated this requirement and prvided the interface. Hwever, in hardware systems, the amunt f infrmatin that can be transferred between cmpnents has physical limitatins as well, related t the number f available data lines and pins. Secnd, each interface must have an assciated errr mdel. Cmpnents must be able t handle all pssible errrs when cmmunicating with anther cmpnent. While this can be a diæcult prblem in sequential systems, it can be an unslvable prblem in a distributed envirnment due t the ptential failure f bth remte cmpnents and the cmmunicatin mechanisms themselves in unexpected ways at unpredictable times. Thus, dealing with errrs adds a signiæcant burden n the prgrammer. Third, the cst f accessing the interface may be prhibitive. Obtaining lcal infrmatin in a sequential sftware system may invlve nly a nminal cst, such as executing a library functin r calling int the perating system; hwever, in any physically distributed system, there will be a latency assciated with accessing remte cmpnents. The verhead f btaining the infrmatin in this case may exceed its beneæt. Furth, btaining the cntrl infrmatin may cnsume resurces that are required fr the data transfers f the service. This is primarily a factr in netwrked systems, where the cntrl messages cnsume available bandwidth. We desire an apprach that des nt cntend fr the very resurces that it is trying t cntrl r allcate. Finally, the infrmatin frm the interface may be ut-f-date by the time it is received. Once again, this is primarily a cnsideratin in distributed systems. If there is a delay between the time that infrmatin is sent frm a remte surce and the time that infrmatin is lcally accessible, then the infrmatin may have changed in the interim. Maintaining infrmatin cnsistency is especially diæcult when clients must crdinate infrmatin frm multiple remte sites. 2.2 Implicitly-Cntrlled Systems Due t the cst and cmplexity f explicitly cntacting ther cmpnents fr cntrl infrmatin, we prpse that cmpnents shuld nly cmmunicate the necessary data infrmatin and that cntrl infrmatin shuld be exclusively inferred frm naturallyccurring lcal events. A system which infers all f its cntrl in such a manner is an implicitly-cntrlled system, r mre succinctly, animplicit system. While the data cmmunicatin may be structured in either a master-slave r a peer-t-peer fashin, we believe that an implicit system is mre naturally suited t a peer-t-peer design. We nw discuss hw cmpnents can btain cntrl infrmatin withut explicitly cntacting ther cmpnents Traditinal Surces f Infrmatin A large bdy f theretical wrk has examined the prblem f inferring infrmatin and making decisins in cmplex systems. Sme research has fcused n systems where

28 Cmpnent Data in Lcal State Data ut Implicit Inf Data Decisin Prcedure Figure 2.: Mdel f an Implicitly-Cntrlled System. Cmpnents that are implicitly-cntrlled d nt receive explicit cntrl messages frm ther cmpnents in the system. Instead, cmpnents infer cntrl infrmatin frm naturally-ccurring data transfers. n cmmunicatin ccurs between cmpnents í an extreme example f an implicitlycntrlled system. Fr example, game thery fcuses n predicting the actins f adversaries in the absence f any cmmunicatin ë29ë. Similarly, decisin thery prvides a mdel fr ptimizing the utility f decisins based n uncertain infrmatin that has been quantiæed with a prbability measure ë5ë. Finally, team decisin thery cmbines the previus tw appraches, stressing the distributed nature f the decisin makers ë4ë. Cmpnents can als infer the state f remte cmpnents if they have cmmn knwledge ë7ë. Cmmn knwledge is the strngest frm f knwledge in a distributed system and cnsists f thse facts that all cmpnents knw, and that all cmpnents knw that all cmpnents knw, ad inænitum. Such knwledge can be practically btained in distributed systems nly when the facts are static and predeæned, fr example, when cmpnents are required t cnfrm t a standard r are knwn t be identical t ne anther. Other research, mst ntably distributed prblem slving èdpsè within artiæcial intelligence, assumes multiple autnmus agents are slving a single prblem in a cperative but decentralized fashin. Much f the wrk in this æeld is directly relevant t implicit systems, such as mdeling the levels f knwledge f distributed agents ë7, 7, 59ë and understanding f the utility f infrmatin as it ages ë3, 38ë Implicit Infrmatin Implicit infrmatin is a pwerful additinal surce f knwledge in implicitlycntrlled systems. A lcal cmpnent can infer the state f remte cmpnents frm a cmbinatin f arriving data traæc and implicit infrmatin. Implicit infrmatin is available frm bserving characteristics f lcal events that are utside f their deæned interface. The bserved lcal events must ccur naturally as part f the system; thus, implicit infrmatin frm remte cmpnents cannt be requested n demand. An example f a naturally-ccurring lcal event is the arrival f a data request frm a remte nde. Exam-

29 2 ples f implicit infrmatin that can be bserved abut such anevent are its cmpletin time and the rate f incming requests. Figure 2. diagrams an implicit system leveraging implicit infrmatin. Implicit infrmatin may beinteresting fr its wn sake, with n additinal infrmatin. Hwever, we believe implicit infrmatin is mst useful when it is cmbined with ther knwledge; fr example, knwledge f hw the remte cmpnents behave. This cmbinatin may allw a cmpnent t infer the state in which the remte cmpnent must reside t have prduced this event. Implicit infrmatin diæers frm hints in that a hint can be incrrect ë93ë. Implicit infrmatin itself is never wrng; the implicit infrmatin carried by the naturally-ccurring event is smething directly bservable. Hwever, cmpnents may still derive incrrect inferences frm implicit infrmatin. Therefre, just as a hint must be crrect the majrity f the time in rder t be useful, s must the inference frm the implicit infrmatin. Implicit infrmatin als diæers frm piggy-backed infrmatin: it must ccur naturally as part f the data transfer. Systems that react t implicit infrmatin are naturally adaptive t current cnditins. We nte that if a cmpnent knws that ther cmpnents react in a certain manner t implicit infrmatin, the cmpnent may manipulate implicit infrmatin t prduce a desired respnse. Fr example, in Sectin 2.3.3, we examine a prpsal fr netwrk ruters that drp packets t manipulate the cngestin mechanisms within implicitly-cntrlled TCP endpints. Fr a system t rely entirely n implicit infrmatin, the interesting state r behavir in remte cmpnents must be bservable in lcal events. When it is nt the case that critical events can be inferred with implicit infrmatin, then explicit cntrl messages are required. Furthermre, t make cnclusive inferences frm nly implicit infrmatin, the lcal bservatins must ccur simultaneusly with the change in remte state and leave n rm fr misinterpretatin. Therefre, when implicit infrmatin is nly crrelated with interesting remte events, but nt an entirely reliable predictr, implicit infrmatin is mre successful in imprving perfrmance than in guaranteeing crrectness. T reiterate, in an implicit system, the nly cmmunicatin acrss cmpnents is the data mvement that is inherent t the service; an implicit system des nt need t leverage implicit infrmatin, it simply must nt explicitly cmmunicate cntrl messages acrss cmpnents. Alternatively, inanexplicit system, sme cmmunicatin ccurs strictly t cnvey cntrl infrmatin acrss cmpnents í either t infrm r query cmpnents f remte state r t direct cmpnents in their actins. An explicit system may be adaptive r may leverage implicit infrmatin t imprve perfrmance, but, by deænitin, requires explicit cntrl messages. 2.3 Examples Many systems exist in cmputer science that either are implicitly-cntrlled r leverage implicit infrmatin; hwever, they have tended t d s in an ad hc manner. One f the gals f this thesis is t explre the cmmn characteristics f such systems and t develp a systematic methdlgy fr building implicit systems. In this sectin, we