Scalable, Distributed, Dynamic Resource Management for the ARMS Distributed Real-Time Embedded System

Transcription

1 Scalable, Dtrbuted, Dynamc Reource Management for the ARMS Dtrbuted Real-Tme Embedded Sytem Kurt Rohloff, Yarom Gabay, Janmng Ye and Rchard Schantz BBN Technologe Cambrdge, MA, USA {krohloff, ygabay, ye, Abtract We preent a calable, herarchcal control ytem for the dynamc reource management of a dtrbuted real-tme embedded (DRE) ytem. Th DRE npred by the DARPA Adaptve and Reflectve Mddleware Sytem (ARMS) program. The goal of the control ytem to multaneouly manage multple reource and QoS concern ung a utlty-drven approach for decon makng and performance evaluaton. At each level of the control herarchy there are multple local controller whch autonomouly make decon to optmze ther local utlty. The controller n the herarchy can ue dfferent, localzed reource control algorthm and the ytem uer can tune the operaton of the local controller. We dcu how the electon of local control algorthm affect the behavor of the overall ytem. The control ytem degned to be ealy adaptable to other mult-tered DRE ytem. 1. Introducton Large dtrbuted real-tme embedded ytem are often degned wth tatc reource management tratege talored for pecfc goal or mon. Thee rgd reource allocaton tratege are ncapable of adaptng to changng ytem goal, reource level and operatng envronment. Th nablty to adapt can caue DRE ytem to fal to meet end-to-end qualty of ervce (QoS) requrement when condton change /07/$20.00 IEEE We preent a herarchcal control ytem for the dynamc reource management of herarchcal DRE ytem that capable of multaneouly managng multple reource and QoS concern. Dynamc reource management ha the capablty to acheve much hgher performance n a contraned reource ytem than tatc reource management approache. The DRE applcaton area npred by the DARPA Adaptve and Reflectve Mddleware Sytem (ARMS) program n conuncton wth Raytheon and Lockheed Martn. The ARMS ytem can be decompoed nto multple mon and mon can be decompoed nto multple ubmon called applcaton trng or trng. A mult-tered behavoral herarchy uch a th a common apect of many DRE ytem. A key element of the control ytem we are degnng for thee DRE ytem a utltydrven approach for decon-makng and performance evaluaton wth repect to reource allocaton n the controller. Utlty computed for each element n the ytem herarchy (trng, mon, ytem) and a meaure of that element ablty to perform t dered tak. The allocaton of ytem reource dynamcally managed to locally maxmze utlty at each level of the ytem herarchy wth ndvdual controller deployed for the whole ytem and all mon and t trng. The general phloophy for the control ytem a bottom-up approach to dynamc reource management. At the lowet level, controller perform fat, frequent, local tunng of ytem behavor, whle at the hghet level, controller perform le frequent, but more aggreve control acton. Th bottom-up calable control approach can be ealy appled to other herarchcal DRE ytem. Other

2 dynamc control approache to meet QoS requrement are n [2, 3]. The next ecton decrbe key element of the mult-tered herarchcal ytem we are controllng. The control obectve and utlty meaure for dynamc reource management are outlned n Secton 3. The control archtecture and the algorthm ued by the controller are dcued n Secton 4. We dcu the behavoral effect of ung the varou control algorthm n Secton 5. We conclude the paper and dcu everal avenue of future work n Secton Sytem Archtecture Properte of DRE ytem can be undertood va apect of both ther reource and applcaton runnng on thoe reource. The reource apect of DRE ytem nclude the computaton and communcaton reource of the ytem. The computatonal reource are a et of general purpoe computer hot. The communcaton reource n the ytem are the communcaton lnk formed between varou hot n the ytem and the attrbute of thee lnk uch a bandwdth, maxmum delay and operatng mode. Hot are aumed to be grouped nto pool or cluter of computng reource baed on ther phycal locaton. Pool are managed ndependently of one another by local pool manager. Communcaton between hot n the ame pool aumed to be generally nexpenve, whle hot n a pool hare lmted communcaton gateway to hot n other pool. Therefore, communcaton between hot parttoned nto ntra- and nter-pool Communcaton Lnk communcaton. A dagram of the ytem reource nteracton can be een n Fgure 1. Software applcaton are deployed onto the computatonal reource and can be vewed at multple level of abtracton. At the lowet level of abtracton, applcaton run on hot and perform work requrng certan computng reource. At the next hghet level of abtracton, an applcaton trng, or trng, a logcal equence of applcaton that equentally proce nformaton wth unque tartng and endng applcaton. Strng are generally deployed acro multple hot and pool, o trng controller need to manage nter- and ntra-pool communcaton. Strng generally perform work ubect to end-to-endqos requrement. Two or more trng may hare an applcaton. At the penultmate level of abtracton, a mon a group of trng that cooperate to acheve common goal. At the hghet level of abtracton, the ytem ncorporate all runnng mon and reource thoe mon have acce to. A chematc of the ytem-montrng decompoton can be een n Fgure 2. The ARMS ytem ha oftware component called the Infratructure Allocator (IA) that allocate applcaton to hot and a Bandwdth Broker (BB) that allocate bandwdth on ntra- and nter-pool communcaton lnk. The IA actuate the control acton of the control ytem by (re)allocatng computatonal reource, and the BB actuate the control acton of the control ytem by (re)allocatng communcaton reource. It aumed that the DRE ytem condered n th paper have mlar oftware component to actuate control acton. It alo aumed that ytem tatu meaurement nformaton hared by a dtrbuted publh-and-ubcrbe ervce called RSS (Reource Statu Servce). RSS allow Sytem Pool wth hot Mon Mon Mon Strng Strng Strng Fgure 1: Reource of DRE Sytem Fgure 2: Software Herarchy of DRE Sytem

3 ytem component to mprove effcency by not havng to redundantly gather and dtrbute tatu nformaton ndependently([4]). 3. Reource Management Obectve The herarchcal control ytem ue a et of utlty functon to evaluate the performance of trng and mon n the ytem agant current reource allocaton. The control ytem alo ue the utlty etmaton functon to etmate the derablty of varou control acton wth repect to the future performance and utlty of the ytem. The control ytem chooe control acton that would reult n a hgher level of etmated utlty. If the ytem ha enough unued ytem reource, the ytem could allocate reource to prevouly undeployed mon or applcaton trng to boot t overall utlty and performance. Converely, f reource contenton were to occur due to an over-deployment of mon (pobly due to reource falure among other poble caue), then the performance and utlty of the deployed mon would drop. A change n reource avalablty ndcate that the controller may need to adut reource uage to attempt to maxmze utlty due to the current operatng condton. We ue a et of herarchcal utlty functon to meaure the performance of the ytem that follow the ytem-mon-trng herarchy outlned n the ntroducton. Utlty functon are defned for the ytem, each of the mon and each of the applcaton trng that meaure the performance of thee entte under current reource allocaton. More formally, at a gven tme t: U t the utlty of ytem performance. U m t the utlty of mon. U t the utlty of trng of mon. We defne the ytem-level utlty, U t to be a weghted um of the mon-level utlte: = M m m U t w = U t 0 m The weght factor w a meaure of the relatve mportance of mon. Smlarly, the mon ablty to complete t requred tak depend on the ablty of t trng to complete ther dered tak, o mon utlty U m t a weghted um of the mon trng-level utlte: S S t w U t m U = = 0 The weght factor relatve mportance of trng S w a meaure of the of mon. The utlty of trng from mon depend on the tmelne, qualty, and throughput of nformaton proceed by the trng. Thee factor are an ndcaton of how well the trng can proce and tranmt nformaton. Tmelne ( T t ) a meaure of the applcaton trng ablty to meet end-toend real-tme requrement. Qualty ( q t ) a meaure of how ueful the nformaton proceed by an applcaton trng. Throughput ( Th t ) the rate at whch nformaton to be proceed ent to the trng. The mappng of the T t, q t and Th t to the utlty of the trng may vary from trng to trng, o we defne U t to be computed by a generc functon F (,, ) : U t = F ( T t, q t, Th t ). The utlty U t perodcally computed by t trng controller and publhed on RSS o that the hgher level controller can compute U m t and U t. Expreon for tmelne, qualty, and throughput are applcaton dependent. 4. Control Archtecture In order to herarchcally allocate reource n the ytem to maxmze ytem utlty, controller are deployed wth one controller for every trng (called the trng controller), one controller for every mon (called the Mon Controller or MC ) and one ytem controller (called the Mult-Mon Coordnator or MMC). A dagram of the ytem-mon-trng herarchy can be een n Fgure 3. At the top of the dagram, the MMC control the gro allocaton of reource to the mon. At the next level down, the local MC coordnate the local deployment of trng whch conume the local allocaton of reource. At the lowet level, the trng controller, fat local tunng of the local reource uage.

4 Strng Controller All of the controller n the herarchy communcate wth ther parent and/or chldren to facltate tradeoff between local run-tme utlte and reource allocaton among control layer n the bottom-up control degn. The controller nteract wth each other through drect communcaton, but the controller receve nformaton about ytem reource tatu or performance through RSS. The low level controller are generally fat and reponve, whle the hgh level controller have the ablty to take more aggreve control acton. Hgher level control acton are more nvave, o the hgher level controller are degned take more tme to better etmate whch of ther control acton wll maxmze ther local utlty. Local controller n th degn attempt to greedly mantan ther local utlty and the bottom-up control phloophy lmt local, fat utlty gan that are potentally detrmental to the overall ytem utlty. 5. Local Control Algorthm We now dcu the operaton of the ndvdual controller at each level of the control herarchy. 5.1 Strng Controller MMC MC MC MC Strng Controller Strng Controller Fgure 3: Control Herarchy of the DRE Sytem Strng controller perform fat low-level tunng of qualty and throughput n order to mantan ther local trng utlty. A drop n trng utlty could be caued by ether reource contenton or falure, but on reource falure, the trng controller expected to receve notfcaton about the falure from RSS. In the abence of a notfcaton from RSS ndcatng otherwe, the trng controller aume drop n utlty are caued by reource contenton. Generally the qualty ( q t ) and the throughput ( Th t ) of an applcaton trng can be drectly controlled by the trng controller by adutng applcaton n a trng, but the tmelne ( T t ) cannot. However, the tmelne of a trng can be nfluenced by tunng the qualty and throughput of nformaton proceed by a trng. When a trng controller oberve that U t gnfcantly below t meaured baelne, the trng controller attempt to decreae the trng qualty and throughput. Any oberved mprovement n tmelne by decreang qualty and throughput wll not be ntantaneou, o ncremental decreae are made n both qualty and throughput on the utlty meaurement cycle. Qualty and throughput are contnually decremented untl a local maxmum of the meaured trng utlty U t found. If the local maxmum not uffcently cloe to the utlty baelne, the trng controller end a gnal to the mon controller that the mon controller hould attempt to releve the trng oberved reource contenton. It reman an open problem to determne how aggrevely the trng controller hould decrement qualty and throughput n attempt to mantan ther local utlty. 5.2 Mon Controller When gven acce to an amount of reource, an MC decde whch of t mon trng hould be deployed ung thoe reource to maxmze the mon utlty. We have degned an ARMS mon controller that operate wth two algorthmc component. A chematc of the mon controller nternal operatonal logc een n Fgure 4. One mon control algorthmc component, called the trng electon logc, determne whch trng to deploy/kll/redeploy baed on the trng mportance to the mon, the amount of reource the mon allowed to ue and the trng current deployment tatu. The econd algorthmc component, called the trng bndng logc, elect the reource that deployed trng hould ue baed on how much reource are avalable to the mon. The trng bndng logc nterface wth the IA to determne whch reource are free and trng hould be deployed on.

5 Informaton on Strng Value, Reource Uage Selecton Method Strng Statu Informaton Strng Deployment & Kllng Requet Reource Statu Strng Selecton Logc Strng Bndng Logc Strng Deployment Plan Fgure 4: Mon Controller Logc Schematc Strng Deployment Operaton The trng electon logc operate n repone to partal ytem falure and uer nput to enure that mportance revaluaton are followed through the mon trng deployment. We have teted varou algorthm for the trng electon logc. Thee algorthm nclude mportance-baed greedy orderng, reource-effcency-baed greedy orderng and dynamc programmng. When electng whch trng to deploy, the amount of reource the mon allowed to ue ntended to be ued a an nput from the MMC o that the MMC can drect the overall dvon of reource to the mon. Note that ntead of beng allocatng pecfc reource, the mon are gven nput a to how much reource they are allowed to ue. 5.3 Mult-Mon Coordnator The MMC perform the gro-level allocaton of reource between the mon. Note that rather than gvng the mon controller acce to pecfc reource, the MMC gve the mon controller the rght to ue an amount of reource. When dvdng the avalable ytem reource up amongt the mon, the MMC predct what utlty the ytem would attan from allocatng varou amount of reource to the mon. To do th, the MMC receve a lookup table from each mon controller that map an approxmaton of the um of the mportance value of trng the mon controller could deploy for ther mon f gven the ablty to ue varou level of reource. The lookup table are generated and ent to the MMC by every mon controller at ntalzaton and are baed on uer-commanded mon goal. The lookup table are alo ntended to be updated regularly whenever a mon receve a command drectve to refne t local behavor baed on the relatve mportance value of the mon and t trng. Fgure 5 contan a chematc of MMC operaton whch ndcate that the lookup table of R V Mon Reource Allocaton MMC Reource avalablty R V MC MC MC Strng deployment Sytem Reource Pool Fgure 5: MMC Concept of Operaton

6 mon reource-value mappng. When the lookup table are ent to the mon controller, the reource level lted n the lookup table are quantzed baed on the level of qualty of ervce provded by the mon for deployng group of trng. Becaue the deployment of the mon mot mportant/crtcal trng neceary for mnmal mon operaton, the lowet quantzaton level of the lookup table correpond to the reource neceary to deploy ut the mot mportant/crtcal trng for each of the mon. The other quantzaton level n the mon lookup table are dependent upon the context of the mon and could be ued to tune the operaton of the MMC when dvdng ytem reource among the mon. When the MMC ha the lookup table from the mon controller and gven nformaton about the avalablty of reource n the ytem, the MMC need to decde how much reource hould be provded to every mon controller n order to guarantee all crtcal trng can be run and to maxmze the total value of all trng that can be deployed. The problem of the MMC allocatng reource to the mon can be formalzed a a multple-choce knapack problem [1]. The MMC could ue any number of algorthm to compute the mot effcent dvon of reource baed on nformaton from the lookup table and reource effcency. We found the dynamc programmng algorthm to be very effectve and effcent conderng the relatvely mall number of mon that we are ung. Once the MMC compute the dvon of the reource amongt the mon, the MMC communcate to the mon controller how much of the computaton reource they are each allowed to ue. Th communcaton ndcated n the chematc of the MMC operaton n Fgure 5. When the mon controller have nformaton about how many reource they are allowed to ue, they decde whch of ther trng to deploy n order to maxmze local utlty. The mon controller receve no nformaton about the allocaton of reource to other mon. Therefore, on the occurrence of gnfcant ytem event uch a partal ytem falure or mpotance revaluaton, the mon controller make all of ther local reource allocaton decon under the aumpton that ther total reource allocaton han t changed unle they receve updated nformaton from the MMC. 6. Sytem Smulaton We developed a large-cale, hghly confgurable Matlab/Smulnk model of the ARMS mult-mon ytem to obectvely compare the utlty-meaured performance of the ytem ung the dynamc reource controller to a baelne ytem where reource are tatcally allocated at ntalzaton. For our mulaton experment, we confgured the model to cont of three mon wth 100 trng each that can be deployed on 5 pool wth nter-pool lnk reource. When the mon controller perform trng deployment operaton, there a confgurable actuaton delay between the tme the mon controller end the actuaton gnal untl the tme the trng become operatonal whch we approxmated a 0.1ec. In the mulaton model, the operatng condton of the trng are hghly confgurable. The computatonal and communcaton requrement of the trng can be cutomzed to model varou mon cenaro a long a there are at leat two applcaton n every trng. The uer-agned mportance value of the trng are alo confgurable and can be ued a expermental parameter n mulaton. The amount of reource avalable to the mult-mon ytem can alo be aduted n the mulaton model. In partcular, the number of pool and how many applcaton can be run n each pool and be ndvdually aduted along wth the amount of nter-pool bandwdth avalable to the mon trng n the ytem nter-pool communcaton lnk. It not neceary that the pool and lnk have homogenou reource confguraton. We mulate partal ytem falure n real-tme n the model by removng all of a pool node to model a complete pool falure. Ung the large-cale Matlab/Smulnk model of the ARMS ytem, we generated 100 expermental trng deployment cenaro contng of 3 mon of 100 trng, each wth randomly choen applcaton length unformly dtrbuted between 2 and 11. Inter-applcaton bandwdth requrement were randomly choen to be ether 1 or 2 megabt per econd. The 100 trng were randomly agned nteger mportance value wth a unform random dtrbuton between 1 and 10, ncluve. To generate the lookup table generated by the mon controller and ent to the MMC, we randomly grouped the mon trng et nto 10 quantzaton level.

7 Performance Rato % 50% 100% Reource Defcency Fgure 6: Rato of Dynamc to Statc Utlty Performance v. Reource Defcency For each cenaro, the ytem had fve operatonal pool at ntalzaton wth uffcent reource deploy all trng. The pool were allocated computaton reource uch that after the falure of a pecfc pool, the mon controller would caue the mon to have only 80% of the reource requred to deploy all trng. The falure of two pool would caue the ytem to have 60% of the reource to deploy all trng, the falure of three pool would caue the ytem to have 40% of the reource to deploy all trng and the falure of four pool would caue the ytem to have 20% of the reource to deploy all trng. The Matlab/Smulnk mulaton were run uch that the MMC and mon controller were gven uffcent tme to deploy all trng after ntalzaton. After ntalzaton wa completed, a et of pool wa faled, and the mon controller wa allowed to complete t recovery operaton n repone to the pool falure. We recorded the utlty attaned by the mon controller mmedately before the falure and after falure recovery operaton completed. We alo collected data on the performance of the ytem f a dynamc reource controller wa ued where reource are allocated at ntalzaton and then no change are made n the reource allocaton. Th tatc reource allocaton trategy wa the baelne ytem n the ARMS program. Fgure 6 contan a graph that demontrate how the rato of performance for ytem ung the tatc and dynamc MMC vary wth reource defcency. A can be een from the graph, a reource defcency ncreae, the dynamc MMC able to acheve over 2x performance gan over the tatc MMC. 7. Concluon We have preented a utlty drven herarchcal controller degn for mult-tered DRE ytem to dynamcally manage ytem reource. Although only three level of abtracton are condered here for the DRE and control ytem, our degn ealy caled to any number of control level where low level controller take fat, lmted acton and hgh level controller take low, more nvave acton. Acknowledgement Th work wa upported by the Defene Reearch Proect Agency (DARPA) under contract NBCHC Approved for publc releae. Dtrbuton unlmted. Bblography [1] K. Dudz k and S. Walukewcz. Exact method for the knapack problem and t generalzaton, European Journal of Operaton Reearch, 28, pg 3-21, 1987 [2] C. Lu, J. Stankovc, S. Son, and G. Tao. Feedback control real-tme chedulng: Framework, modelng, and algorthm. Real-Tme Sytem, 23(1 2):85 126, July [3] H. Wu, B. Ravndran, E. Jenen, and P. L. Tme/utlty functon decompoton technque for utlty accrual chedulng algorthm n real-tme dtrbuted ytem. IEEE Tranacton on Computer, 54(9): , [4] J. Znky, J. Loyall, and R. Schapro. Runtme performance modelng and meaurement of adaptve dtrbuted obect applcaton. In Proceedng of Internatonal Sympoum on Dtrbuted Obect and Applcaton (DOA), Irvne, CA, 2002.