Energy-Efficient Resource Utilization for Heterogeneous Embedded Computing Systems

Transcription

1 58 IEEE TRANSACTIONS ON COMPUTERS, VOL. 66, NO. 9, SEPTEMBER 7 Energy-Effcent Reource Utlzton for Heterogeneou Embedded Computng Sytem Jng Hung, Renf L, Senor Member, IEEE, Jyo An, Member, IEEE, Derrck Ntlh, Fn Yng, nd Keqn L, Fellow, IEEE Abtrct In th pper, the jont optmzton problem wth energy effcency nd effectve reource utlzton nvetgted for heterogeneou nd dtrbuted mult-core embedded ytem. The ytem model condered to be fully heterogeneou model, tht, ll node hve dfferent mxmum peed nd power conumpton level from the perpectve of hrdwre whle they cn employ dfferent chedulng trtege from the perpectve of pplcton. Snce the concerned problem by nture mult-contrned nd mult-vrble optmzton problem n whch cloed-form oluton cnnot be obtned, our m to propoe power llocton nd lod blncng trtegy bed on Lgrnge theory. Furthermore, when the problem cnnot be fully olved by Lgrnge pproch, dt fttng method employed to obtn core peed frt, nd then lod blncng chedule olved by Lgrnge method. Severl numercl exmple re gven to how the effectvene of the propoed method nd to demontrte the mpct of ech fctor to the preent optmzton ytem. Fnlly, multon nd prctcl evluton how tht the theoretcl reult re content wth the prctcl reult. To the bet of our knowledge, th the frt work tht combne lod blncng, energy effcency, hrdwre heterogenety nd pplcton heterogenety n heterogeneou nd dtrbuted embedded ytem. Index Term Embedded nd dtrbuted ytem, energy effcency, effectve reource utlzton, lod dtrbuton, power llocton, queueng model Ç INTRODUCTION. Motvton A typcl complex embedded ytem wll hve heterogeneou dtrbuted mult-core rchtecture tht cn repond to vrety of complcted computtonl requet t the pplcton level. It common for complex embedded ytem, uch utomotve electronc nd vonc ytem, to hve over 6 Electronc Control Unt (ECU)[3], wth ech ECU dedcted to hndlng numerou tk of dfferent ze nd level of urgency. A the complexty of embedded ytem contnue to ncree to meet the demnd of modern pplcton for ncreed computtonl power nd performnce, the need for energy effcency nd effectve reource utlzton wll become ncrengly gnfcnt. Current nd future embedded ytem mut be ble to gn generl tk to node n mnner tht J. Hung, R. L, J. An, D. Ntlh, nd F. Yng re wth the College of Computer Scence nd Electronc Engneerng of Hunn Unverty, Ntonl Supercomputng Center n Chngh, Key Lbortory for Embedded nd Network Computng of Hunn Provnce, Chngh 48, Chn. E-ml jngh@hnu.edu.cn, lrenf@vp.n.com, nbobcn@lyun. com, dbntlh@gml.com, yngfnf7@6.com. K. L wth the College of Computer Scence nd Electronc Engneerng of Hunn Unverty, Ntonl Supercomputng Center n Chngh, Key Lbortory for Embedded nd Network Computng of Hunn Provnce, Chngh 48, Chn Deprtment of Computer Scence, Stte Unverty of New York, New Pltz, NY 56. E-ml lk@newpltz.edu. Mnucrpt receved 6 Sept. 6 reved 8 Mr. 7 ccepted 5 Apr. 7. Dte of publcton Apr. 7 dte of current veron 5 Aug. 7. Recommended for cceptnce by A. Ykovlev. For nformton on obtnng reprnt of th rtcle, plee end e-ml to reprnt@eee.org, nd reference the Dgtl Object Identfer below. Dgtl Object Identfer no..9/tc mprove reource utlzton wthout ffectng dedcted tk. Power mut be llocted reonbly to ech node n order to cheve mnmum power uge by the ytem. Attnng optml llocton of tk nd power n dtrbuted ytem well-known mult-vrble optmzton problem. In lght of thee ue, the development of heterogeneou dtrbuted embedded ytem chllengng. In heterogeneou ytem, the rchtecture of ech node my dffer, o the chrctertc of node my vry. Ech node mght hve dfferent mxmum nd mnmum core peed, or dfferent power conumpton level [9]. The performnce of the overll ytem cn be nfluenced by ny node. Therefore, to cheve energy effcency n heterogeneou envronment, the chrctertc of ech node mut be condered crefully. From the pont of vew of dtrbuted ytem, ech node gned preloded dedcted tk, nd ech tk my hve dfferent tk rrvl rte nd tk ze. To cheve effectve utlzton of reource, dtrbuted ytem requre n effcent lod blncng lgorthm tht cn gn tk pproprtely to ech node. From the pont of vew of embedded ytem, dedcted tk executed on pecfed node re more mportnt or urgent thn generl tk. Moreover, ech cl of dedcted tk h dfferent degree of urgency. To utlze ll the vlble reource effcently, ech node hould be et wth n pproprte chedulng polcy correpondng to the degree of urgency of dedcted tk gned to t. From the pont of vew of the overll ytem, computng performnce vtl metrc when ytem Qulty of Servce (QoS) beng evluted. Thu, the QoS tll need to be gurnteed. Blncng ll of thee fctor chllenge for the development of heterogeneou dtrbuted nd embedded ytem ß 7 IEEE. Peronl ue permtted, but republcton/redtrbuton requre IEEE permon. See http// for more nformton.

2 HUANG ET AL. ENERGY-EFFICIENT RESOURCE UTILIZATION FOR HETEROGENEOUS EMBEDDED COMPUTING SYSTEMS 59 Dcplne On th node, the queueng prncple tht dedcted tk re lwy cheduled before generl tk. All tk re executed wthout nterrupton. We dentfy th dcplne, prortzed dedcted tk wthout preempton. Dcplne 3 Dedcted tk re lwy cheduled before generl tk on th node, wth preempton. We term th dcplne, prortzed dedcted tk wth preempton. Our m to fnd the mnmum overll power conumpton of the ytem, long wth the repone tme of generl tk, wthn n cceptble rnge. Our mjor contrbuton re follow Fg.. Sytem tructure. tht re both energy effcent nd mkng the bet ue of reource. Although there re mny tude of the dvere pect th problem, mot of the extng reerch don t conder thee fctor jontly. Therefore, t mportnt to tudy how energy effcency nd hgh reource utlzton cn be cheved together on heterogeneou nd dtrbuted embedded ytem.. Our Contrbuton In th pper, we tudy the problem of gnng et of generl tk to the computng node of computtonl heterogeneou dtrbuted embedded ytem, wheren ech node preloded wth dfferent number of dedcted tk, equpped wth DVFS feture. The tructure of the ytem hown n Fg.. A node cn be treted computtonl unt, whch my nclude proceor, memory etc. Chngng node from t leep tte to runnng tte tke long tme []. In embedded envronment, node my be gned mportnt tk tht cnnot be delyed. Conequently, we don t hve the opton to put n embedded node to leep, even f t core not workng. In our nvetgton, to blnce the power conumpton nd tme dely, we ume tht core contnue to run t low frequency even when t dle. Clerly, the power conumpton dffer when the core workng nd when t not workng. Therefore, the core cn be condered to hve two dtnct mode [5] Core buy-power The power conumpton of core when there re tk runnng on the core, the mjor power conumpton of core. Core dle-power The power conumpton of core when there no tk runnng. We vew ech node n M/M/ queueng model wth nfnte wtng queue cpcty [4], nd defne three queueng dcplne-dcplne, Dcplne, nd Dcplne 3- ech one of whch could be employed by ny node. The detl of the dcplne re follow Dcplne All generl tk nd dedcted tk on th node re cheduled on frt-come, frt-erved b, wthout prorty. We dentfy th dcplne, dedcted tk wthout prorty. To the bet of our knowledge, th work the frt tudy of the mnmum power conumpton problem n heterogeneou dtrbuted embedded ytem tht conder the lod dtrbuton n combnton wth the chrctertc, queung dcplne, nd dle peed of ech node. We propoe n lgorthm for fndng the optml lod dtrbuton nd power llocton cheme of the ytem, uch tht the overll power conumpton of the ytem mnmzed. We re the frt to tke the optml oluton trnng dt to ft the reltonhp between the tk ze nd core peed, nd then ue optml lod blncng to olve the problem when the problem cnnot be olved by Lgrngn ytem. Expermentl reult how th trtegy to be effcent. Bed on our lgorthm, we how the nfluence of dfferent prmeter on the optml power llocton nd lod dtrbuton. Thee prmeter nclude dle peed of core, well power conumpton exponent, preloded tk, queueng dcplne, nd number of node n the ytem. We provde numercl exmple to demontrte the effectvene of our lgorthm for ech prmeter. Furthermore, we gve n exmple where ll prmeter re dfferent. Smulton nd prctcl evluton how tht the theoretcl reult re content wth the prctcl reult. Our tudy focue on well-defned, mult-contrned, nd mult-vrble optmzton problem. The nvetgton n th pper h mde gnfcnt contrbuton to hghperformnce nd energy-effcent computng n modern heterogeneou nd dtrbuted embedded ytem. RELATED WORK Becue energy effcency prmry concern for embedded ytem, epeclly for ytem wth lmted power, th topc h been tuded extenvely, nd lrge body of lterture ext [], [3], [4], [5], [6]. In recent yer, upercomputer opertor lo hve pd conderble ttenton on energy effcency becue upercomputer hve very lrge power requrement. Whle upercomputer re focued on performnce ther mot gnfcnt metrc, the technque ued by embedded ytem to cheve energy effcency mlr to tht of upercomputer. Energy effcency bout mkng power conumpton proportonl to ytem utlzton [] n mnner tht decree unnecery energy lo. There re mny pproche to chevng power reducton. Mot

3 5 IEEE TRANSACTIONS ON COMPUTERS, VOL. 66, NO. 9, SEPTEMBER 7 commonly, dynmc voltge nd frequency clng (DVFS) [], [3] mplemented t the opertng ytem level to mnge power nd to regulte the frequency nd voltge of CPU. Generlly pekng, two DVFS technque ext for mult-core ytem One globl DVFS, whch cle the frequency nd voltge of ll the core multneouly, nd the other locl DVFS, whch regulte the frequency nd voltge on per core b [7]. Experment ndcted tht locl DVFS could cheve better performnce thn globl DVFS [8], [9], but t more complcted. The energy effcency of embedded ytem h been tuded by number of reercher. Becue the rchtecture nd pplcton for embedded ytem re qute dvere, reercher hve needed to etblh vrou theore to tudy the problem of energy effcency n thee dfferent ytem. In [6], the uthor nvetgted the trdeoff between nter-pplcton concurrency wth performnce nd power conumpton under vrou ytem confgurton. They propoed runtme optmzton pproch to cheve energy effcency, mplemented on rel pltform clled Odrod XU- 3. In [7], the mnmum energy conumpton w obtned bed on runnng model generted through regreon-bed lernng of energy/performnce trde-off between dfferent computng reource n the ytem. In [8], to upport pplcton qulty of ervce nd to ve energy, n energy-effcent oft rel-tme CPU cheduler for moble devce w propoed tht prmrly rn multmed pplcton. In ddton to embedded computng, energy effcency lo ply n mportnt role n cloud computng, whch mrked by huge nd ncreng power conumpton. The technque for chevng energy effcency ued n multcore embedded ytem nd cloud computng ytem re mlr. Therefore, they could lern from ech other. In [], the uthor ued DVFS nd worklod dependent dynmc power mngement to mprove ytem performnce nd to reduce energy conumpton. In [], bed on coopertve gme-theoretcl pproch nd DVFS technology, the uthor nvetgted the problem of lloctng tk onto computtonl grd, wth the m of mnmzng multneouly the energy conumpton nd the mkepn. In [], the uthor lo employed gme-theoretc pproch to tudy the problem of mnmzng energy conumpton n dtrbuted ytem. An effcent lod blncng trtegy key component to buldng out ny dtrbuted rchtecture. The complexte re reflected n the extenve body of lterture on the topc, exemplfed by the excellent reference collecton gven n [3]. The purpoe of lod blncng to gn tk pproprtely to node n term of the worklod nd computng power of ech node. In [5], reercher propoed fult tolernt, hybrd lod blncng trtegy for heterogeneou grd computng envronment. In [6], the uthor ddreed the problem of optml lod blncng of tk when power contrned. The queueng dcplne h lo been tuded wdely. In [4], two type of ce were condered, nmely, ytem wth nd wthout pecl tk. The uthor ddreed the problem of mnmzng the verge repone tme of generc tk. Both [7] nd [8] tuded optml lod dtrbuton n heterogeneou dtrbuted computer ytem wth both generc nd dedcted pplcton. In [7], ech node w modeled n M/G/ non-preemptve queung ytem, nd w ppled to everl type of dedcted tk, whle n [8], ech node w treted n M/M/ nonpreemptve queung ytem. The uthor of [9] umed tht ech node w preloded wth dedcted tk, nd three condton were tken nto ccount Dedcted tk wthout prorty, nd prortzed dedcted tk wth nd wthout preempton. Ech node w treted n M/G/ queueng ytem, nd the uthor focued on the problem of optml lod blncng of generl tk. In dtrbuted heterogeneou embedded ytem, n order to cheve energy effcency nd effectve utlzton of reource, t necery to conder the combnton of node heterogenety, pplcton urgency (prorty of tk, whch mght be dfferent for ech node), energy effcency, nd the dle CPU tte. To the bet of our knowledge, preent tude on lod blncng nd energy effcency hve not condered fully ll of thee fctor together. 3 SYSTEM MODEL ANDPROBLEM FORMULATION 3. Power Model The power dpton of n embedded proceor core mnly cont of three prt, nmely, dynmc, ttc, nd hort-crcut conumpton, mong whch dynmc power conumpton the domnnt component. The dynmc power conumpton cn be expreed by P ¼ kcv f where k n ctvty fctor, C the lodng cpctnce, V the upply voltge, nd f the clock frequency. Gven tht / f nd f / V, then P /, where round 3 []. For ee of dcuon, we model the power llocted to proceor core wth peed. The core buy-power dfferent from core dle-power. There re mpled energy-frequency nd frequency-performnce relton. In th pper, the performnce (peed) defned the number of ntructon core cn perform per econd (IPS). Therefor, the dynmc power when the core workng t frequency f nd the correpondng peed. When core not workng, becue there re no ntructon to perform, t npproprte to defne the core peed drectly. In tht ce, our reerch focue on the power conumpton rther thn core peed. Therefore, when the core dle, we ume the peed to be I, correpondng to frequency f, uch tht I equl the ctul power of the core,.e., I ¼ CV f. A proceor core tll conume ome mount of bc power P tht nclude ttc power dpton, hort crcut power dpton, nd other lekge nd wted power. Therefore, the power model cn be formulted P ¼ð ¼ r ¼ þ P Þr þð I þ P Þð r Þ þ ð r Þ I þ P b br þ e er þ b br þ e er! I þ P () 3. Queueng Model The queueng model ued to formulte nd tudy the problem of power llocton nd lod blncng n heterogeneou dtrbuted embedded envronment. Tkng n the number of heterogeneou embedded computng node

4 HUANG ET AL. ENERGY-EFFICIENT RESOURCE UTILIZATION FOR HETEROGENEOUS EMBEDDED COMPUTING SYSTEMS 5 TABLE Mthemtcl Notton n Th Pper Symbol Defnton The core peed of node v when t core buy I The core peed of node v when t core dle power conumpton exponent e Arrvl rte of dedcted tk to v b Arrvl rte of generl tk to v b ¼ e þ b ¼ b þ b þþ b n br Averge tk ze of generl tk er Averge tk ze of dedcted tk on v ex ¼ er = Averge executon tme of dedcted tk on v bx ¼ br= Averge executon. tme of generl tk on v br b bx ¼ b br er e ex ¼ e er. r ¼ br þ er Averge percentge of tme tht node v buy bt Averge repone tme of generl tk on v bt Acceptble tme of generl tk on ytem v v...v n (mply clled node), ech of whch h t own dedcted et of job tht follow Poon trem of tk wth rrvl rte e tht cn only be executed on t. There ext generl Poon trem of tk wth rrvl rte b tht need to be executed by beng plt nto n ubtrem b gned to ech node. Thu, ech node del wth combned trem of dedcted nd generl tk. The tk ze of dedcted nd generl tk re exponentl rndom vrble rd nd rg, repectvely, wth men er nd br, repectvely. Thu, the two type of men executon tme on node v re ex ¼ er = bx ¼ br=, repectvely. Snce the rrvl rte nd proceng rte of tk re ubject to Poon dtrbuton, we cn tret ech node n M/M/ queueng ytem. Prmeter ued re hown n Tble. To mntn the queue tedy, we ume tht r <, for ll n. 3.3 Problem Formulton We pecfy our mult-vrble optmzton problem follow gven n number of embedded node v v...v n, the rrvl rte e e... e n nd verge tk ze er er... er n of dedcted tk on ech node, the totl rrvl rte b nd verge tk ze br of generl tk, the dle-peed I I... In, be power upply P P...P n, queueng dcplne of ech node, nd the cceptble repone tme bt of generc tk, fnd the tk rrvl rte b b... b n nd core peed... n on ech node uch tht the power conumpton of the ytem P ¼ P n ¼ P mnmzed whle tfyng the followng contrnt b þ b þþ b n ¼ b () b b T þ b b T þþ b n b T n bt (3) 4 THE PROPOSED METHOD Ech node treted n M/M/ queung ytem nd h dfferent queung dcplne. Dfferent queung dcplne hve dfferent expreon of repone tme of generl tk. Thu, ll node re dvded nto three group ccordng to the queung dcplne. We ume tht group G nclude ll thoe node whoe queung dcplne dedcted tk wthout prorty, group G nclude ll thoe node whoe queung dcplne prortzed dedcted tk wthout preempton, nd group G 3 nclude ll thoe node whoe queung dcplne prortzed dedcted tk wth preempton. Let bt denote the repone tme of generc tk on node v. For node v belong to group G (v G ), we hve [4, p. 7] bt ¼ br þ b br þ e er b br e (4) er For node v belong to group G (v G ), we hve [4, p. 7] bt ¼ br b br þ þ e er e er e er b (5) br For node v belong to group G 3 (v G 3 ), we hve [4, p. 74] bt ¼ e br þ b br þ e! er er e er b (6) br Our objectve functon P b b... b n... n ¼ Xn ¼ ubject to nd b brþ e er X v G b T b þ X v j G þ b brþ e er b j T b j þ X v k G 3 b þ b þþ b n ¼ b!! I þ P b k b T k bt Snce the bckground of th problem cler, we cn ue Lgrnge ytem to olve our problem. We et c b b... b n... n! ¼ bt X X X (8) b bt b þ b bt þ b bt v G v G v G 3 nd b b... b n ¼ b þ b þþ b n b (9) two contrnt functon. Accordng to Lgrnge ytem, we hve tht, rp ¼ fr ð b b... b n Þ þ trcð b b... b n... b b... b b ¼ b b... b n... n þ b (7) ()

5 5 IEEE TRANSACTIONS ON COMPUTERS, VOL. 66, NO. 9, SEPTEMBER 7 for ll n, where f nd t re two Lgrnge b b... b n @c b b... b () n... n þ Bed on Equton (), we get br br I ¼ f t bt þ b! () where for ll v G, we bt br e er þ e er b ¼ e er b br for ll v G, we b ¼ br b br þ e er e er b br 3 þ e er e er b br for ll v G 3, we bt br e er þ e b ¼ e er b br e er Bed on Equton (), for ll v G, we cn get where b þ b b þ c ¼ (3) ¼ br R b ¼ e er brr c ¼ ðbr R Þ e er þ e er e er R ¼ f þ br I br t Solvng Equton (3), we cn obtn vffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff u b ¼ t br br ut brt þ e t er t (4) for ll v G mlrly, bed on Equton (), we cn get vffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff b ¼ t br t brt þ e u t er br d t =t þ br e (5) er for ll v G nd vffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff b ¼ e er brt þ e u t er t (6) br br d d for ll v G 3, where t ¼ e er d ¼ br I þ f br Bed on Equton (), we tke the prtl dervtve wth repect to, tht, t bt ¼ ð b brþ e er b brþ e er þ I where for ll v G we bt ¼ br b br þ e er b br e B þ b br e A er for ll v G we ¼ nd for ll v G 3 we hve b br þ e er e er b br e er e er b bt b br þ e er e er b B br e er b þ br A br e er (7) Through tkng the dervtve wth repect to b nd repectvely, we hve obtned the Equton (4), (5), (6) nd (7) for ll n. Bng on thee Equton, our problem modfed to fnd the pproprte f, t nd ech node peed to tfy the condton Equton () nd (3). Th well-defned mult-vrble optmzton problem whch dffcult to get cloed-form oluton epeclly tht dfferent queueng dcplne hve dfferent expreon of b bt.@. Thu, we hve to deve the numercl oluton. We conder 8 vffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff t br t brt þ e u t er t v G br d vffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff f ð f tþ ¼ t br t brt þ e >< u t er br d t =t þ br e v G (8) vffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff er t br brt þ e u t er t > v G 3 br nd where g ð b Þ¼ d b brþ e er b. Þ þ I t ¼ e er d ¼ br I þ f br (9)

6 HUANG ET AL. ENERGY-EFFICIENT RESOURCE UTILIZATION FOR HETEROGENEOUS EMBEDDED COMPUTING SYSTEMS 53 Snce b vewed functon of, f nd t, nd t treted functon of nd b, t need to fnd the domn defnton of functon f ð f tþ nd g ð b Þ. The tk rrvl rte b mut be lrger thn zero, nd r <. Hence, we hve 8 < b e er b br () br I þ f br > for ll n. In rel tuton of dtrbuted envronment, my be decml nd not the me for dfferent node, therefore, t mpoble to obtn cloed-form oluton of Equton (). We hll gve the numercl oluton n Secton 5. How to obtn the pproprte b,, f nd t bed on f ð f tþ nd g ð b Þ tht cn tfy contrnt condton Equton () nd (3) wll be ntroduced n Secton 5. We gve the dervton wth repect to of functon f ð f tþ nd g ð b Þ, whch wll be ued n Secton 5 where h ð Þ¼ t H t, nd f ð f tþ ¼ br h ð Þ p ffffffffffffffffffffffff br h ð Þ d h ð Þ¼ t brt þ H t f br ð Þ d for ll v G h ð Þ ¼ t H t d t þ tbr e, nd er BðH þ t br Þ þ t H ð h ð Þ ¼ t d t þ tbr e d t þ d Þt H er d t þ tbr e A er for ll v G hð Þ¼ H t, nd d for ll v G 3 nd We get d h ð ð Þ¼t br þ H Þd d H d t ¼ e er d ¼ br I þ f br H ¼ br e er þ e er d ¼ f br g ð b Þ¼ ð Þ þ I. b bt.@ D e er þ. b brþ b er þ ð Þð Þ 3 I b brþ e er 3 where bt.@ ¼ br D 3 t þ t b br t t b br b br b br þ e er þ b!! br t t b b br þ e er br t t b br t þ b B br t b A br for ll v G bt.@ ¼ br f D ð Þ b D þ t t b br for ll v G, where D ¼ f ð Þ b! br t b br þ e er b b br þ e! er t b 6t þ b br b br 6 br t t þ br f ð Þ b! br 3t t ¼ e er nd bt.@ ¼ br D 3 t þ t b br t t b br f ð Þbr b br þ e er þ f ð Þbr t t b br b br þ e er ð f ð Þbr Þt þ t b B br t 3 t b 3 A br wth t ¼ e er, for ll v G 3. Bed on bove equton, the next ecton wll ntroduce how to employ lgorthm to obtn the pproprte t, f nd of ech node tht tfy Equton () nd (3). 5 THE ALGORITHM We wll mplement the lgorthm to olve the preent multvrble optmzton problem. And, how to obtn the defnton domn of decrbed n Secton 5., whle Secton 5. ntroduce how to fnd the pproprte Lgrnge multpler f nd t bed on f ð Þ nd g ð f ð ÞÞ n Equton (8) nd (9) under the contrnt condton n () nd (3). Furthermore, nce tht the dfference between the preloded tk nd of ech node lrge, nd then t dffcult to ccomplh lod blncng only by the Lgrnge theory, Secton 5.3 wll olve the mult-vrble optmzton problem by combnng wth the Lgrnge method nd dt fttng technque.

7 54 IEEE TRANSACTIONS ON COMPUTERS, VOL. 66, NO. 9, SEPTEMBER 7 re ued to quckly fnd pont tfyng f ð Þ >. Then, tkng the vlue of to fnd the low bound nd upper bound b (See Algorthm ). The bnry erch wll be motly ued n our lgorthm. In order to vod repetedly ung lt of erch method, we defne t n Algorthm. Fg.. Severl exmple of F ð Þ. 5. Defnng the Serch Spce of A mentoned n Secton 4, we vew f ð f tþ nd g ð b Þ functon of, for ll n, the domn defnton of whch defned by Equton (). However, t mpoble to get cloed-form oluton wth regrd to for Equton (). We hve to deve numerc oluton. We conder where nd F ð Þ ¼ f ð f tþ ¼t F ð Þ 8 rffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff t brt þ e. er t d v G vffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff t brt þ e >< u t er t d t þ tbr e v G rffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff er brt þ e. > er t v G 3 t ¼ e er d ¼ br I þ f br Gven f nd t, functon F ð Þ for ll n hve the mlr chngng trend core peed chnge. Fg. how n exmple of F ð Þ. Aume F ð Þ nd t nterect t two pont ð F ð ÞÞ nd ð b F ð b ÞÞ. If b, then f ð Þ ele f ð Þ. The two vlue nd b re repectvely the lower bound nd upper bound of domn defnton of functon f ð Þ. In order to erch for the vlue of nd b,weneedpont whch tfe f ð Þ >. Accordng to the rule tht f b,thenf ð Þ ele f ð Þ, we cn repectvely employ bnry erch to fnd the vlue of between ½ e er Š nd the vlue of b between ½ mx Š, where mx repreent the oluton of br I þ f br ¼. Bng on the Lgrnge Men Vlue Theorem, there mut be pont between nd b tht mke F ð Þ¼ F ð b Þ F ð Þ b d ¼ t ð b Þ t ð Þ b ¼ When, we hve F ð Þ nd f ð Þ whle, we hve F ð Þ nd f ð Þ. To tke dvntge of th feture, Squeeze theorem nd bnry erch method Algorthm. bserch(vr lb ub crteron) Input vr lb ub crteron Output vr whle (ub lb > ") do vr ðub þ lbþ= 3 f (crteron) then 4 ub vr 5 ele 6 lb vr 7 end f 8 end whle 9 return vr. Algorthm. getdomnof ( e er br f t) Input e er br f t. Output lb, ub. lb e er ub MxS bserchð lb ub f þ br I br < Þ 3 mx lb e er 4 whle (f ð f tþ < ) do 5 f (f ð f tþ < ) then 6 lb 7 ele 8 ub 9 end f ðub þ lbþ= end whle lb e er ub 3 bserchð lb ub f ð f tþ > ÞÞ 4 lb lb ub mx 5 bserchð lb ub f ð f tþ < ÞÞ 6 ub 7 return ub, lb. 5. Serchng for Lgrnge Multpler Our trget to fnd the pproprte f, t nd ll of ( n), whch cn mke condton Equton () nd (3) be tfed bng on f ð Þ nd g ð f ð ÞÞ. Our trtegy tht by fxng Lgrnge multpler we try to erch for n pproprte vlue of the other Lgrnge multpler whch cn mke one contrnt condton (Equton () or (3) be tfed, then djutng the vlue of frt Lgrnge multpler, under the new vlue, we contnue to erch the correpondng vlue of the other Lgrnge multpler. The proce wll fnh only f the two pproprte Lgrnge multpler re found, or f the loop condton re volted. 5.. Serchng Lgrnge Multpler t Theorem. If there no dedcted tk on node ( e ¼ ), then the peed of the node ndependent of t, nd h the followng form I ð br Þ ¼ br f

8 HUANG ET AL. ENERGY-EFFICIENT RESOURCE UTILIZATION FOR HETEROGENEOUS EMBEDDED COMPUTING SYSTEMS 55 Proof. Tkng e ¼ nto f ð Þ bt.@, we cn get b ¼ br bt ¼ B where ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff! brt () I þ f br b br b br C b A br ¼ ð br þ br L Þbr ð br L Þbr B A ð br L Þbr ¼ br þ ð L Þð þ L Þ L ¼ br L ¼ I þ f br t ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff brt L ¼ I þ f br () Subttutng Equton () nd () nto Equton (7), we cn get tht, t ¼ br I þ f br ð Þ þ I t I þ f br ¼ br ð Þ þ I Bng on the bove equton, we obtn I ð br Þ ¼ br f nd the theorem proven. If there re dedcted tk on node, t very dffcult to drectly olve th problem by ung mthemtcl dervton, nd mpoble to get cloed-form oluton. Through obervng the form of g ð f ð ÞÞ, we notce tht f we et e ¼ (there re no dedcted job), then the form of g ð f ð ÞÞ wll be trnlted nto g ð b Þ¼. Þ þ I The repone tme T could be treted convex functon of. bt.@ wll decree ncree. n ncreng functon of, nd I decreng functon of, whch mple tht the bove equton hould decree ncree nd then ncree contnue to ncree. The feture of functon g ð f ð ÞÞ wll not be chnged even f e not equl to zero. Vrtully th feture cn be oberved from gret del of dt experment. Fg. 3 Fg. 3. Severl exmple of functon g ð b Þ. how n exmple of four node, nd ech node preloded wth dfferent mount of tk. Algorthm 3. fnd turnng Pont (f e er ) Input f e er. Output. lb ub getdomnof ð e er br f tþ bserchð lb ub g ðf ð Þ ÞÞ 3 return. Ech g ð b Þ h t mnmum vlue, menng tht there pont of peed tht mke g ð b Þ obtn the mnmum vlue. The cn be obtned by ung the dervtve of g ð b Þ wth repect to (See Algorthm 3). The Lgrnge multpler t hould hve the me vlue for ech g ð b Þ, where n. Thu, the low bound of g ð b Þ the mxmum vlue of ll mnmum vlue of g ð b Þ, nmely, the mxmum vlue of ll g ð f ð ÞÞ. From Fg. 3, we cn oberve tht for ech g ð b Þ there re two vlue of tht cn be mpped onto the me vlue of t, one locted t the left de of nd the other one locted t the rght de of. Notce tht for the me t, the dfference n peed of ech node wth dfferent preloded tk lrge, nd the chnge n vlue of functon g ð b Þ chnge drtc, f we tke the left de vlue the vlue of, th mple tht the left de vlue not utble vlue of. In fct, we hve tred to tke the left de vlue the vlue of, nd the reult bnorml. Thu, gven vlue of t, we dopt the rght de vlue correpondng to the t the vlue of. At the rght de of,echg ð b Þ monotone ncreng functon of. Hence, gven vlue of g ð b Þ, we cn mmedtely get the correpondng vlue of (See Algorthm 4). Algorthm 4. Clculte (t ) Input t. Output. lb ub ub bserchð lb ub g ð f ð ÞÞ < tþ 3 return. In order to tfy the condton n Equton () nd (3), we need to djut f nd t. Ech g ð b Þ treted functon of, nd tll repreent the vlue of t. Functon f ð Þ

9 56 IEEE TRANSACTIONS ON COMPUTERS, VOL. 66, NO. 9, SEPTEMBER 7 Fg. 4. The nd b chngng tendency t chnge. repreentng b lo contn t. Thu, the vlue of g ð b Þ hould equl the vlue of t ncluded n f ð Þ. Let g ð b Þ¼t, through Algorthm 4 we cn get the vlue of correpondng to the t. Our trget to get n pproprte t whch cn mke Equton (3) tfed. Thu, we hve to djut the vlue of t. By nlyzng Equton (7), we oberve tht the wll decree t ncree. In Fg. 4, we repectvely gve ere of nd correpondng b, whch re the oluton of g ð b Þ¼t when t et to dfferent vlue. In dtrbuted ytem, f the core peed of node reduced, then the node wll be gned wth leer tk, th led to the verge repone tme of generl tk on th node to reduce. Snce bt / b, b / nd / =t, then b bt / =t, th mple tht gven pproprte vlue of f, the bnry erch method could be employed to fnd the pproprte t tht cn be ued to get ll peed mkng the contrnt condton Equton (3) be tfed (ee Algorthm 5). Algorthm 5. ClculteAll Input f e... e n er... er n br I... In bt. Output... n. for ( n þ ) do fndturnngpont (f e er ) 3 end for 4 ub z lb 5 t bserchðt lb ub P n ¼ f ðclculte ðt Þ f tþ bt < bt Þ 6 return t... n. wll ncree monotonclly wth f. In term of the rule, our oluton to the problem of optml power llocton nd lod dtrbuton cn be decrbed follow 5.. Serchng Lgrnge Multpler f It cn be oberved from Equton (8) tht for ll n, f we reduce the vlue of f, then the vlue of f ð Þ wll decree. Thu, ctully, f ð Þ could be vewed ncreng functon of f. Snce gven n pproprte f, t tht mke condton Equton (3) be tfed could be obtned, the condton Equton () lo cn be met by djutng f. In fct, we hve the followng rule For ll olved n Algorthm 5 tht tfy Equton (3), b wll be ncreng monotonclly wth f, th ndcte tht P n ¼ b Step Gven f, ung Algorthm 5 to fnd the t tht equl to ll of g ð b Þ ( n), well cn mke Equton (3) be tfed. Step Bed on Step, djut f untl the condton Equton () tfed. Through the bove Step () nd () we cn fnd the optml oluton to our problem. However, Fg. ugget tht f the vlue of f become mller, the vlue of g ð b Þ wll become lrger, th men tht mll f wll be mtched wth lrge t. By obervng Equton (8), we know tht f ð Þ my be le thn zero when f excevely mll nd t too lrge. Under th tuton, there mght not ext uch common t tht mke ll tfy Equton (3), th men tht ung Step () nd () cnnot olve the current problem. Accordng to common t tht ext, we cn fnd threhold of f clled f B, when f f B there wll ext common t tht cn mke ll tfy Equton (3) ( n), whle f < f B there wll not ext common t. Snce f mtched wth b, there ext b B correpondng to the f B. When b b B, we cn fnd the optml oluton to our problem bng on Lgrnge ytem when b < b B, Lgrnge ytem cnnot be ued to olve the problem becue Lgrnge multpler cnnot be found. We cll the erchng proce for f B clbdoff, due to lmted pce, t moved to the upplementry mterl, whch cn be found on the Computer Socety Dgtl Lbrry t http// do.eeecomputerocety.org/.9/tc Algorthm 6 cn be employed to olve our problem under the tuton tht b > b B. How to del wth the tuton tht b < b B decrbed n next ecton. Algorthm 6. Cculte_P Input e... e n er... er n br bt. Output b... b n,... n f t. f B clbdoff lb f B 3 repet 4 f f 5... n clculteall () 6 untl b þ b þþ b n > b 7 ub f 8 whle (ub lb > ") do 9 f ðlb þ ubþ=... n clculteall () f ( b þ b þþ b n < ) b then lb f 3 ele 4 ub f 5 end f 6 end whle 7 return b... b n,... n f t. 5.3 Dt Fttng Notce tht the key fctor for olvng our problem how to determne the peed for ech node. In other word, lod blncng depend on power llocton. Therefore, the frt work to olve our problem hould be how to determne the core peed for ech node. We cnnot dopt Lgrnge ytem to obtn the optml core peed of ech node when b < b B, whle t ey for u to get lot of optml llocton dt when b b B. Snce the bckground of our problem cler, we nt tht there ext mppng reltonhp between rrve rte of generl tk b nd ech core peed. Snce lot of optml llocton dt cn be obtned by ung Lgrnge ytem when b b B, thee dt could be employed trnng dt to ft the reltonhp between rrve rte of generl tk b nd ech core peed. The detl re decrbed follow

10 HUANG ET AL. ENERGY-EFFICIENT RESOURCE UTILIZATION FOR HETEROGENEOUS EMBEDDED COMPUTING SYSTEMS 57 TABLE Numercl Dt n Secton 6. When Sytem Prorty Strtegy I Dedcted Job wthout Prorty b r P T TABLE 3 Numercl Dt n Secton 6. When Sytem Prorty Strtegy I Dedcted Job wthout Prorty b r P T Aume tht through Lgrnge ytem, we hve group of optml peed llocton dt pont ð b... n Þ ð b... n Þ,...,ð b... n Þ N. N the number of the dt pont, b dfferent for ech dt pont, nd b > b B for ll the N dt pont. We wnt to etmte ech core peed ( n) under the tuton tht rrve rte of generl tk b nd < b b B. We hll ft the dt ung polynoml functon of the form ¼ w þ w b þþwm b M ¼ XM k w k b where M the order of the polynoml, nd b k denote b red to the power of k. The polynoml coeffcent w w w M requre to be olved. We dopt root-menqure to ft the dt, whch cn mmedtely get the vlue of w w w M, nce ech of them h cloed form oluton. Once we get the polynoml coeffcent w w...w M, then we obtn the equvlent peed of node v, the remnng work to fnd the pproprte b b... b n, f nd t ubject to b þ b þþ b n ¼ b, nd b ð b bt þ b bt þþ b n bt n Þ bt Snce peed for ll node hve been obtned, the pproprte b b... b n, f nd t cn be obtned by removng the tep for erchng peed n Algorthm 6. Due to pce lmtton, the lgorthm regrdng how to olve the problem under the tuton tht < b b B moved nto the upplementry mterl, vlble onlne. For ytem under nly, we frt clculte the threhold b B nd f B, nd then determne whch method could be dopted to olve our problem. It worth notng tht lthough the reult olved by erchng method, ll method we dopted re bnry erch method. Moreover, the erch for nd b ( n) of ech node ndependent except for the hred Lgrnge multpler f nd t. Th mple tht our method cn explot dtrbuton nd prllelm to olve the problem when the ytem clblty fctor (n) lrge. 6 NUMERICAL EXAMPLES In th ecton, we demontrte number of numercl exmple. All prmeter n our exmple re for llutrton purpoe only, nd could be chnged to ny other rel vlue. In heterogeneou dtrbuted prllel computng envronment, ech prmeter of node cn hve n mpct on power llocton nd lod dtrbuton. We how thee mpct, nd um up the objectve lw oberved from our expermentl dt n the ltter prt of ech ecton. k¼ 6. The Impct of Idle Speed I In th ecton, we conder the mpct of dle peed I on power llocton nd lod dtrbuton. We conder group of n ¼ 7 embedded node. We ume tht e ¼ per econd, er ¼ 3 (gg ntructon), ¼ 7, P ¼ Wtt, for ll n. Further I ¼, I ¼ 4, I3 ¼ 6, I4 ¼ 8, I5 ¼, I6 ¼, I7 ¼ 4 IPS, b ¼ 6 per econd, br ¼ 3 (gg ntructon), nd bt ¼ 5 econd. We how the optml lod dtrbuton b b... b 7, the optml node peed... 7, the node utlzton r r... r 7, the node power conumpton P P...P n nd the verge generl tk repone tme T T...T 7. Reult hown n Tble re for ll node n the ytem employng the Dcplne, dedcted tk wthout prorty, nd the ytem power conumpton Wtt. The mlr reult cn be obtned when ytem queueng dcplne re et to Dcplne nd 3. Due to pce lmtton, they re moved to the upplementry mterl, vlble onlne. From th ecton, we cn oberve tht the ytem wll gn more tk to node wth hgher core dle-power, whch h phycl menng, nce the node conume more power when t dle, tryng to reduce t dle tme cn decree power lo. 6. The Impct of Power Conumpton Exponent In th ecton, we conder the mpct of on power llocton nd lod dtrbuton. We lo conder group of n ¼ 7 embedded node. We ume tht e ¼ per econd, er ¼ 3 (gg ntructon), I ¼ 3 IPS, P ¼ Wtt for ll n b ¼ 7 per econd. Further br ¼ 3 (gg ntructon), bt ¼ 6 econd, ¼ 6, ¼ 65, 3 ¼ 7, 4 ¼ 75, 5 ¼ 8, 6 ¼ 85, nd 7 ¼ 9. Reult hown n Tble 3 re for ll node n the ytem employng the Dcplne, nd the ytem power conumpton P ¼ Wtt. The reult for ll node employng Dcplne nd 3 re moved to upplementry mterl, vlble onlne. From th ecton, we cn oberve tht the ytem wll gn more tk to node wth mller vlue of, whch h phycl menng,.e., f node cpble of performng t the me cpcty of work other node, but conume le power, then gnng more tk to the node reonble. 6.3 The Effect of Dt Fttng In th ecton, we conder the ce tht ung Lgrnge ytem cnnot obtn the optml power llocton nd lod

11 58 IEEE TRANSACTIONS ON COMPUTERS, VOL. 66, NO. 9, SEPTEMBER 7 Fg. 6. Fttng veru GA. Fg. 5. Fttng reult. dtrbuton trtegy. We ft the reltonhp between the rrvl rte of generl tk nd core peed for ech node. The verge tk ze nd cceptble repone tme of generc tk re br ¼ 5 (gg ntructon) nd bt ¼ 5 econd, repectvely. The other prmeter re e ¼, ¼ 6, I ¼ 5 for ll 8,(er, er, er 3, er 4, er 5, er 6, er 7, er 8 ) (.,.3,.4,.,.3,.4,.,.4), node,, 3 employe Dcplne, node 4, 5, 6 employe Dcplne, nd node 7, 8 employe Dcplne 3. For ech node v, we elect 7 dt pont ( b nd correpondng ) from regon 3 b 4 trnng dt et, whch hve been olved wth Lgrnge ytem. Fg. 5 how the fttng reult when the order of polynoml M M ¼, M ¼ 3, M ¼ 4 nd M ¼ 5. Once the fttng functon obtned, we could obtn the peed mmedtely for ech node correpondng to gven b. Therefore, bed on, t ey for u to obtn the tk llocton b gned to ech node, well the power conumpton for the overll ytem. Tble 4 how the power conumpton correpondng to dfferent vlue of M nd b. The oluton reched by th fttng method mght not be n optml oluton. A we ll know, genetc lgorthm (GA) cn olve non-lner problem, nd cheve globl pproxmte optml oluton. In order to check the qulty of oluton, we compre our reult wth the oluton produced by the genetc lgorthm from the genetc lgorthm toolbox GAOT n MATLAB. We ue the me group ze nd nput prmeter Secton 6.6, nd et the order of the polynoml M ¼ 4. The reult of th compron re hown n Fg. 6. Menwhle, we compre the oluton provded by fttng method wth optml oluton. The reult pper n Fg. 7. From Fg. 6, we oberve tht our oluton, mot of the tme, re better thn the oluton provded by the GA lgorthm. Th reult mple tht the qulty of oluton obtned by our method good. From Fg. 7, we oberve tht the dfference between fttng oluton nd optml oluton re wthn. W. Th fndng mple tht the fttng oluton could replce the optml oluton to certn degree. The beneft of the fttng method tht t could reduce the erch proce for fndng ech core peed. Thee reult demontrte tht reltonhp ext between the totl tk rrvl rte nd the core peed the bgger the order of the polynoml M, the cloer t to the optml power llocton. Moreover, th work provde n mportnt nght. When the dfference between the preloded tk nd of ech node lrge, the worklod of the ytem reult n n mblnce between ech node pror to gnng generl tk to ech node. It dffcult to ccomplh lod blncng when the tk rrve rte b mll. Thu, under thee crcumtnce, t poble tht ung Lgrnge ytem cnnot olve the problem of optml power llocton nd lod blncng on the ytem. 6.4 The Impct of Preloded Tk Due to pce lmtton, thee dervton re moved to the upplementry mterl, vlble onlne. 6.5 The Impct of Queueng Dcplne Due to pce lmtton, thee dervton re moved to the upplementry mterl, vlble onlne. 6.6 The Stuton of Fully Heterogeneou Sytem Due to pce lmtton, th ecton moved to the upplementry mterl, vlble onlne. TABLE 4 Numercl Dt n Secton 6.3 M b ¼ 8 b ¼ 9 b ¼ b ¼ b ¼ W 6.46 W W 9.99 W 3.66 W W W W W W W W W W W W W W W W Fg. 7. Fttng veru optml.

12 HUANG ET AL. ENERGY-EFFICIENT RESOURCE UTILIZATION FOR HETEROGENEOUS EMBEDDED COMPUTING SYSTEMS 59 TABLE 5 Numercl Dt n Secton 7 b r P T The Impct of Sytem Sclblty Due to pce lmtton, th ecton moved to the upplementry mterl, vlble onlne. 6.8 Performnce Compron Due to pce lmtton, th ecton moved to the upplementry mterl, vlble onlne. 7 EXPERIMENT EVALUATION The reult hown n Secton 6 re theoretcl reult. In th ecton, we provde our fndng concernng the dfference between the theoretcl reult nd the reult obtned from expermentl evluton. In the evluton experment we conder the ytem contng of x node, the verge rrve rte nd tk ze of generl tk re b ¼ 78 nd br ¼ repectvely, the verge rrve rte nd tk ze of dedcted tk on ech node re e ¼ 6 6, nd er ¼ er ¼, er 3 ¼ er 4 ¼ 3, er 5 ¼ er 6 ¼ 45, repectvely, nd the bc power nd power conumpton exponent P ¼ 35W nd ¼ 3 ( 6) repectvely. The dle peed I ¼ 6. For th nvetgton, ll node employed Dcplne. Bed on thee prmeter, we obtned the optml power nd llocton of tk hown n Tble 5. Bed on Tble 5, the expermentl evluton dvded nto two prt follow. 7. Smulton Evluton The followng dcuon revew the dfference between the theoretcl vlue nd multon vlue obtned from the executon of n etblhed number of tk. The reult lted n Tble 5 theoretcl vlue. To nvetgte the dfference between the theoretcl vlue nd ctul multon vlue, we generted number of generl nd dedcted tk. The rrvl ntervl tme nd tk ze for generl tk re exponentl rndom vrble = b nd br repectvely for dedcted tk the rrvl ntervl tme nd tk ze re = e nd er repectvely. TABLE 6 Smulton Reult TT RTGT GN wtter P T 9, , ,8 3, , , ,98 3, , , ,59 58, , ,96. 48, 58, , ,6. 4,653, , , ,93, TABLE 7 Pltform Prmeter CPU OS Memory DVFS tool Cortex-A Debn G cpufreq Frequency Mx trnton ltency GHz m Once thee tme re etblhed, we chedule thee tk. The chedulng reult re hown n Tble 6, n whch TT repreent the totl tme, RTGT repreent the repone tme for ll of generl tk, GN repreent the number of generl tk, Wtter repreent the totl power cot, P ¼ wtter=tt repreent the verge power cot per econd, nd T ¼ RTGT=GN repreent the verge repone tme of generl tk. By comprng Tble 5 nd 6, we fnd tht there good greement between the theoretcl nd multon reult regrdng the verge repone tme of generl tk T nd power cot per econd P. 7. Prctcl Evluton Bed on Tble 5 nd the tk generted n Secton 7., we wll nvetgte the dfference between theoretcl reult nd prctcl reult on rel pltform contng of the x node (embedded bord) correpondng repectvely wth node mentoned n Secton 7, nd t detled prmeter re lted n Tble 7. The tetng proce dvded nto three tep. Step We need to tet the core peed (IPS) nd power when core opertng t vrou frequence. A progrm commonly cont of number of embly ntructon, uch JUMP, MOV, CMP, ADD, nd MUL. By recordng the ctul number of embly ntructon nd correpondng executon tme, we re ble to obtn the IPS. Power the product of current nd voltge. The bord voltge kept t 5 V n our experment. Notce tht the teted power not only nclude the proceor power, but lo the power of other component. Whle the power of the proceor dynmc, the power of the other component reltvely tble. Thu, we re ble to tret the core ttc power nd ll of the other component power the bc power P, whch could be obtned by ettng the core frequency to GHz. When the core dle, t core dle-power equl the node (embedded bord) power mnu P. In th experment, the frequency of the Cortex-A dul core CPU et t 336 MHz when the ttu of t core dle. In rel envronment, the power conumpton exponent uully defned 3.. Thu, the dle peed cn be clculted bed on the core dlepower nd. The dt obtned from tet re hown n Tble 8, where PB repreent the power of node when there re tk runnng, nd PI repreent the power ofq node ffffffffffffffffffffff tht PI P there no tk runnng, nd derved by I ¼. Step Snce tht the optml peed hown n Tble 5 computed theoretclly, n the ctul tet pltform, the rel core frequency need to be djuted to mp the correpondng core peed nto theory vlue. Bed on Tble 5 nd 8, we djut the core frequency n term of the mllet gp between

13 53 IEEE TRANSACTIONS ON COMPUTERS, VOL. 66, NO. 9, SEPTEMBER 7 the optml nd prctcl peed. For exmple, the cloet frequency tht we could get peed.667,.644, nd.58 Gg IPS.,.96, nd.864 GHz, repectvely. Therefore, ech node frequency djuted to correpondng level. Step 3 In the coure of experment, by recordng the rrvl, trt nd completon tme of ech tk, we cn clculte the totl tme (TT), executon tme (ET) nd repone tme of ll tk. Note tht Cortex-A nclude two core. The power conumpton per econde for node wth one core cn be clculted P ¼ P B P þ P TABLE 8 IPS nd Power freqency(ghz) (Gg IPS) PB(W) PI(W) I ET þ TT P I P þ P ðtt ET The P B nd P I re obtned from Tble 8. The TT, ET, RTGT (repone tme of ll generl tk), r, P, nd T re hown n Tble 9. From the Tble 5 nd 9, we cn fnd out tht the error of repone tme between optml nd prctcl reult re le thn.6 econd (3.6 percent), nd the error of power re le thn.4w (5 percent). We nlye tht the error between theoretcl vlue nd prctcl vlue re due to the followng reon. () The peed re mde lght djutment. () Some power my be gnored. (3) Speed or power tet proce my be uncertn. (4) Core envronment ext noe etc. In ummry, the experment how tht the preent theoretcl reult re bclly n lne wth the prctcl reult. 8 CONCLUSION In th pper, we hve tuded the jont optmzton problem of lod blncng nd power llocton n heterogeneou dtrbuted embedded ytem. From the perpectve of hrdwre, we pecfy tht ll node n the ytem re heterogeneou, wth ech node hvng dfferent mxmum peed nd power conumpton. We lo pecfy tht the prorty of ech tk dfferent on ech node, nd the peed of ech core dfferent from the perpectve of the pplcton. We propoe n effcent lgorthm to olve the jont optmzton problem ung Lgrnge method. When the problem could not be olved ung the Lgrnge method, we TABLE 9 Prctcl Reult TT ET RTGT GN r P T 9, , ,57. 38, ,37.7 3,8.3 33, , ,6.9 6, , , , , , , ,39. 44,86.5 7, , , ,87.4 7, Þ degn n lgorthm to determne the pproprte peed of ech core by ung fttng dt method to ft the reltonhp between tk rrvl rte nd core peed. Th pproch olve the problem. Extenve numercl exmple re gven to demontrte the mpct of ech fctor on the ytem. Furthermore, we employe both multon nd prctcl evluton to how tht preent theoretcl reult re content wth the prctcl reult. Th reerch mke n orgnl contrbuton to optml lod blncng nd power llocton wth performnce contrnt for multple embedded computng node n heterogeneou nd dtrbuted embedded ytem. ACKNOWLEDGMENTS The work reported n th pper w upported by Ntonl Key Reerch nd Development Pln of Chn Grnt No. 6YFB45, the Nturl Scence Foundton of Chn (Grnt No , 63795, 656). The correpondng uthor Renf L. REFERENCES [] J. Prk, D. Shn, N. Chng, nd M. Pedrm, Accurte modelng nd clculton of dely nd energy overhed of dynmc voltge clng n modern hgh-performnce mcroproceor, n Proc. ACM/ IEEE Int. Symp. Low-Power Electron. De., Aug., pp [] S. H. Km, et l., C-lock Energy effcent ynchronzton for embedded multcore ytem, IEEE Trn. Comput., vol. 63, no. 8, pp , Aug. 4. [3] E. Veg, A. Sntn, A. Fnc, R. Jnk, V. A. Pedron, nd L. S. Olver, Towrd n energy-effcent nomly-bed ntruon detecton engne for embedded ytem, IEEE Trn. Comput., vol. 66, no., pp , Jn. 7, do.9/tc [4] A. Munr, S. Rnk, nd A. Gordon-Ro, Hgh-performnce energy-effcent multcore embedded computng, IEEE Trn. Prllel Dtrb. Syt., vol. 3, no. 4, pp , Apr.. [5] W. Dlly, et l., Effcent embedded computng, IEEE Comput., vol. 4, no. 7, pp. 7 3, Jul. 8. [6] J. Blfour, Effcent embedded computng, Ph.D. dertton, EE Dept., Stnford Unv., Stnford, CA, USA, My. [7] G. Kornro, Mult-core embedded ytem. New York, NY, USA Tylor nd Frnc Group, CRC Pre,. [8] J. Donld nd M. Mrtono, Technque for multcore therml mngement Clfcton nd new explorton, n Proc. IEEE 33rd Int. Symp. Comput. Archt., Jun. 6, pp [9] R. Jyeeln nd T. Mtr, A hybrd locl-globl pproch for mult-core therml mngement, n Proc. IEEE/ACM Int. Conf. Comput.-Aded De., Nov. 9, pp [] K. L, Improvng multcore erver performnce nd reducng energy conumpton by worklod dependent dynmc power mngement, IEEE Trn. Cloud Comput., vol. 4, no., pp. 37, Apr. Jun. 6. [] S. U. Khn nd I. Ahmd, A coopertve gme theoretcl technque for jont optmzton of energy conumpton nd repone tme n computtonl grd, IEEE Trn. Prllel Dtrb. Syt., vol., no. 3, pp , Mr. 9. [] B. Yng, Z. L, S. Chen, T. Wng, nd K. L, Stckelberg gme pproch for energy-wre reource llocton n dt center, IEEE Trn. Prllel Dtrb. Syt., vol. 7, no., pp , Dec. 6, do.9/tpds [3] B. A. Shrz, A. R. Huron, nd K. M. Kv, Ed., Schedulng nd Lod Blncng n Prllel nd Dtrbuted Sytem. Lo Almto, CA, USA IEEE Comput. Soc. Pre, 995. [4] K. L, Optml lod dtrbuton for multple heterogeneou blde erver n cloud computng envronment, J. Grd Comput., vol., no., pp ,. [5] J. Blngmehwr nd N. Rju, A hybrd polcy for fult tolernt lod blncng n grd computng envronment, J. Netw. Comput. Appl., vol. 35, no., pp. 4 4,. [6] J. Co, K. L, nd I. Stojmenovc, Optml power llocton nd lod dtrbuton for multple heterogeneou multcore erver proceor cro cloud nd dt center, IEEE Trn. Comput., vol. 63, no., pp , Jn. 4.

14 HUANG ET AL. ENERGY-EFFICIENT RESOURCE UTILIZATION FOR HETEROGENEOUS EMBEDDED COMPUTING SYSTEMS 53 [7] K. W. Ro nd D. D. Yo, Optml lod blncng nd chedulng n dtrbuted computer ytem, J. ACM, vol. 38, no. 3, pp , 99. [8] F. Bonom nd A. Kumr, Adptve optml lod blncng n nonhomogeneou multerver ytem wth centrl job cheduler, IEEE Trn. Comput., vol. 39, no., pp. 3 5, Oct. 99. [9] K. L, Optml lod dtrbuton n nondedcted heterogeneou cluter nd grd computng envronment, J. Syt. Archt., vol. 54, no. /, pp. 3, 8. [] L. Brroo nd U. Holzle, The ce for energy-proportonl computng, IEEE Comput., vol. 4, no., pp , Dec. 7. [] B. Zh, D. Bluw, D. Sylveter, nd K. Flutner, Theoretcl nd prctcl lmt of dynmc voltge clng, n Proc. 4t Annu. De. Autom. Conf., 4, pp [] Q. Qu nd M. Pedrm, Dynmc power mngement contnuou-tme Mrkov decon procee, n Proc. 36th De. Autom. Conf., 999, pp [3] M. Weer, B. Welch, A. J. Demer, nd S. Shenker, Schedulng for reduced CPU energy, n Proc. Int. Symp. Opertng Syt. De. Implementton, 994, pp [4] A. O. Allen, Probblty, Stttc, nd Queueng Theory wth Computer Scence Applcton, nd ed. Boton, MA, USA Acdemc, 99. [5] M. J. Wlker, et l., Accurte nd tble run-tme power modelng for moble nd embedded CPU, IEEE Trn. Comput.-Aded De. Integr. Crcut Syt., vol. 36, no., pp. 6 9, Jn. 7. [6] A. Alud, R. Shfk, A. Rfev, F. X, S. Yng, nd A. Ykovlev, Power-wre performnce dptton of concurrent pplcton n heterogeneou mny-core ytem, n Proc. ACM Int. Symp. Low Power Electron. De., 6, pp [7] S. Yng, et l., Adptve energy mnmzton of embedded heterogeneou ytem ung regreon-bed lernng, n Proc. 5th Int. Workhop Power Tmng Modelng Optmzton Smul., 5, pp. 3. [8] W. Yun nd K. Nhrtedt, Energy-effcent oft rel-tme CPU chedulng for moble multmed ytem, n Proc. ACM Symp. Opertng Syt. Prncple, 3, pp [9] M. Gorczko, J. Lu, D. Lymberopoulo, S. Mtc, B. Prynth, nd F. Zho, Energy-optml oftwre prttonng n heterogeneou multproceor embedded ytem, n Proc. Degn Autom. Conf., 8, pp [3] G. Xe, G. Zeng, L. Lu, R. L, nd K. L, Hgh performnce reltme chedulng of multple mxed-crtclty functon n heterogeneou dtrbuted embedded ytem, J. Syt. Archt., vol. 7, pp. 3 4, 6. Jng Hung receved the MSc degree n computer cence nd technology from Hunn Unverty, Chngh, Chn, n 3. He currently workng towrd the PhD degree t Hunn Unverty. H reerch nteret nclude prllel computng, hgh-performnce computng, dtrbuted computng, energy-effcent computng, heterogeneou computng, cloud computng, nd mchne lernng. Renf L full profeor of computer cence nd electronc engneerng, nd the den of College of Computer Scence nd Electronc Engneerng, Hunn Unverty, Chn. He the drector of the Key Lbortory for Embedded nd Network Computng of Hunn Provnce, Chn. He lo n expert commttee member of Ntonl Supercomputng Center n Chngh, Chn. H mjor reerch nclude embedded ytem, dtrbuted ytem, nd cyber-phycl ytem. He enor member of the IEEE, nd the ACM. Jyo An receved the MSc degree n mthemtc from Xngtn Unverty, Chn, n 998 nd the PhD degree n mechncl engneerng from Hunn Unverty, Chn,. He w vtng cholr n the Deprtment of Appled Mthemtc, Unverty of Wterloo, Ontro, Cnd, from 3 to 4. Snce, he joned the College of Computer Scence nd Electronc Engneerng, Hunn Unverty, Chngh, Chn, where he currently profeor. H reerch nteret nclude cyber-phycl ytem (CPS), Tkg- Sugeno fuzzy ytem, prllel nd dtrbuted computng, nd computng ntellgence. He h publh more thn 5 pper n nterntonl nd dometc journl nd refereed conference pper. He n ctve revewer of nterntonl journl. He member of the IEEE nd the ACM, nd enor member of the CCF. Derrck Ntlh receved the BSc degree from Copperbelt Unverty (CBU), Zmb, n nd the MSc degree n nformton ytem engneerng from the Unverty of Mncheter, Unted Kngdom, n 5. He currently workng towrd the PhD degree n the College of Informton Scence nd Engneerng, Hunn Unverty, Chn. H reerch nteret nclude context wrene computng, mddlewre, prllel nd dtrbuted ytem, nd Internet of Thng (IoT) Fn Yng receved the MSc degree n computer cence nd technology from Hunn Unverty, Chngh, Chn, n. He w vt cholr wth Mchgn Stte Unverty, from 4-5. He currently workng towrd the PhD degree wth Hunn Unverty. H reerch nteret nclude cyber phycl ytem, embedded ytem, nd computer rchtecture. He member of Chn Computer Federton. Keqn L SUNY dtnguhed profeor of computer cence. H current reerch nteret nclude prllel computng nd hghperformnce computng, dtrbuted computng, energyeffcent computng nd communcton, heterogeneou computng ytem, cloud computng, bg dt computng, CPU-GPU hybrd nd coopertve computng, multcore computng, torge nd fle ytem, wrele communcton network, enor network, peer-to-peer fle hrng ytem, moble computng, ervce computng, Internet of thng, nd cyber-phycl ytem. He h publhed more thn 47 journl rtcle, book chpter, nd refereed conference pper, nd h receved everl bet pper wrd. He currently or h erved on the edtorl bord of the IEEE Trncton on Prllel nd Dtrbuted Sytem, the IEEE Trncton on Computer, the IEEE Trncton on Cloud Computng, the IEEE Trncton on Servce Computng, the IEEE Trncton on Sutnble Computng. He fellow of the IEEE. " For more nformton on th or ny other computng topc, plee vt our Dgtl Lbrry t