Ciy Universiy of ew York (CUY) CUY Academic Works Inernaional Conference on Hydroinformaics 8-1-2014 Conex-Aware Self-Organized Resource Allocaion In Inelligen Waer Informaics Kyung Sup Kwak Qinghai Yang Follow his and addiional works a: hp://academicworks.cuny.edu/cc_conf_hic Par of he Waer Resource Managemen Commons Recommended Ciaion Kwak, Kyung Sup and Yang, Qinghai, "Conex-Aware Self-Organized Resource Allocaion In Inelligen Waer Informaics" (2014). CUY Academic Works. hp://academicworks.cuny.edu/cc_conf_hic/446 This Presenaion is brough o you for free and open access by CUY Academic Works. I has been acceped for inclusion in Inernaional Conference on Hydroinformaics by an auhorized adminisraor of CUY Academic Works. For more informaion, please conac AcademicWorks@cuny.edu.
11 h Inernaional Conference on Hydroinformaics HIC 2014, ew York Ciy, USA COTEXT-AWARE SELF-ORGAIZED RESOURCE ALLOCATIO I ITELLIGET WATER IFORMATICS MEG QI 1,QIGHAI YAG 1, KYUG SUP KWAK 2 1 Sae Key Laboraory of IS, Xidian Universiy, China. Email:qhyang@xidian.edu.cn. 2 UWB Wireless Communicaions Research Cener, Inha Universiy, Korea. Email: kskwak@inha.ac.kr. In his paper, we invesigae he waer resource allocaion for smar waer grids (SWG) wih users conex informaion. A waer resource sharing scheme is proposed for efficien managing waer resources wih he aid of he inelligen waer informaics. A novel specral clusering algorihm is developed o classify end-users ino differen communiies wih respec o he end-users profiles. We characerize he dynamics of he SWG wih he Markov decision process (MDP), and an online Q-learning aided waer allocaion algorihm is conceived by virue of he MDP for adaping he dynamics of he SWG, hus improving he waer uiliy efficiency. umerical resuls demonsrae he advanages of he proposed scheme over he convenional saic schemes. I. ITRODUCTIO Severely aging infrasrucure of waer grid always leads o waer loss, waer hef as well as he loss in revenue of waer uiliies [1]. In order o improve he efficiency of he waer resource disribuion, he smar waer grid (SWG) has emerged as a highly efficien nex-generaion waer managemen sysem, which relies on he advanced informaion and communicaion echnologies o overcome he problems of he radiional waer resource managemen sysems [2]. Wih a combinaion of communicaion echnologies and waer resource managemen sysems, SWG helps ease he regional and/or emporal imbalance of waer resources by accuraely conrolling waer demand and supply in real-ime. In addiion, Endowing SWG wih self-organizing capabiliies is insrumenal in helping operaors perform smar operaions and mainenance. Inherenly, he accurae conrol and operaion of he waer disribuion relies on he conex informaion of he SWG, which generally demands he real-ime wo-way informaion ransmission, namely managemen-cener collecing informaion from end-users (e.g., meers deployed a facory, farmland, residens ec.) as well as i disseminaing signaling informaion o end-users. Specifically, he reliabiliy as well as he real-ime requiremen of informaion are crucial for efficien delivery of waer resources from he waer generaing unis o end-users [3]. The derimenal impac of equipmen failures, capaciy consrains, and unrespecable reducion of qualiies of waer resources, which cause he unbalance of waer resources allocaion, will be largely minimized by he effecive waer condiion monioring, diagnosics and opimizaion [4]. The inelligen waer monioring and conrol enabled by conex informaion and communicaion echnologies [5] have become essenial o realize he envisioned SWG. A wireless sensor neworks based waer-grid srucure was invesigaed in [6], where he small sensors are able o promply deec paricular evens or working condiions and hen o repor relaed informaion o he waer manage sysem. In [7], he smar wireless sensor sysem was currenly deployed in waer managemen sysems o keep rack of he informaion of he end user s waer consuming aciviies, preferences as well as locaions and ime. However, mos of he curren approaches canno mee he requiremen of real-ime waer supply for end-users. Wih he aid of conexual informaion such as end users waer consuming preferences, waer qualiies and he sae of he grid nework, he waer allocaion sysem is able o deliver more suiable waer o end-users in real ime. In radiional sysems, he waer allocaion canno
coincide wih he dynamic qualiy of waer service (QoWS) requiremens, which is naural in real siuaions. To he bes of our knowledge, few lieraures has ever addressed on he sysemaic research resulss of SWG by opimizing he waer resource allocaion wih he aid of he conex informaion of SWG. The ransmission of he conex informaion of SWG demands an efficien communicaion nework. Typically, mesh neworking ogeher wih sensor nodes is a cos effecive approach wih respec o is dynamic self-organizaion, self-healing, as well as self-configuraion and high scalabiliy services [7]. oe ha he advanced waer meering infrasrucures and informaion managemen sysems are he main sources of conex informaion. Reinforcemen learning (RL) approach describes a learning scenario, where an agen gradually improves is behavior by aking opimal acions in is environmen wih he reward for performing well or under he punishmen for failure [8]. The Q-learning, a good RL approach o realize self-organizaiowhich aemps o esimae he discounedd fuure coss while aking an acion in he agen s curren sae. The oupu of he Q-funcion is called Q-value. By exploring he waer grid environmen, he agens (waer consumer communiy) creae a able of Q-values for boh saes of he conex and poenial acions of waer allocaion. In his paper, we conceive a conex-aware dynamic waer resource allocaion scheme for SWGs. A generic inelligen waer disribuion framework is designed wih he combinaion of funcion allows he agen o build incremenally a Q-funcion, he informaion/communicion echnologies and waer grid. A novel specral clusering algorihm is developed o classify end-users ino differen communiies wih respec o he end-users profiles. The Markov decision process (MDP) is employed o characerize he dynamics of he SWG, and hen an online Q-learning aided waer allocaion algorihm is developed for adaping he dynamics of he SWG. Figure 1: Schemaic diagram of SWG II. COTEXT AWARE SWG MODEL A generic SWG sysem is illusraed in Fig. 1, where he SWG is designed and operaed wih he following funcions: o make good use of waer resources including rainwaer, reclaimed waer and seawaer; o disribue, manage and ranspor waer efficienly o easee he imbalance of waer resources; o monior he sabiliy of SWG in real ime by virue of an advanced sensor nework [3]. In he currenly exising waer resource managemen sysems, here is a large amoun of waer loss owing o frequen waer leakage. Furhermore, he uilizaion of waer resource is under a low efficiency and as well under a bad end-user s experience owing o he fac ha he diversiy of he end-users QoWS requiremens is seldom concerned. aurally, differen end-users in SWGs have differen ineress for waer resources, namely differen QoWS requiremens. In addiion, manual operaions in SWG would be very cosly. Therefore, he self-organizing funcions of he waer-resourcee managemen are endowed o adap he dynamics of SWG, whichh include he change of he supply of waer resources, he varying of
users ineress, as well as he change of he operaion scheduling/planing of SWG, ec. Figure 2: Framework of self-organized managemen sysem in SWG. Fig. 2 illusraes a ypical waer resource managemen framework, which consiss of waer resources module, self-organized managemen module and end-users module. Differen ypes of waer resources such as rainwaer, reclaimed waer and seawaer are associaed wih he waer resource modules, which are ypically deployed a he edge of he SWG. The end-users module represens he waer consumers such as residens, indusry and farmland and so on, which are clusered ino differen waer communiies by considering he similar waer consuming profiles of he end-users, e.g., he habi of waer usage, he locaion of end-users, he waer consuming QoWSs and quaniies, ec. In his conribuion, he user s conex informaion includes he informaionn of he user required QoWSs and he waer volume of he really-depleed. The opimal and dynamic waer resource allocaion is performed by he self-organized managemen module based on he informaion of users conex, he resources conex, as well as he SWG nework saes. Besides he measuring funcions, he deployed smar waer meers have he communicaion and conrol funcions, ha is hey may collec/repor he informaion (may be colleced by sensor nework) o he managemen cener, and as well hey may conrol/adjus he waer supply upon receiving he insrucions from he managemen cener. We lis he noaions in TABLE I. oaion c,g c,g Table 1 oaions Descripion Se of waer communiies Se of waer QoWS Se of waer resource ypes Se of waer resource conex informaion, e.g., he coss of producion and/or sorage Se of SWG sae informaion, e.g., he coss of operaion/mainenance of pipeline neworks Volume of resource g disribued o communiyc Volume of resource g really-depleed by communiy yc III. PROBLEM FORMULATIO In his secion, we shall formulae he problem of waer resource allocaion. We denoe = { g ( c ), c } wih g (c ) being he favorie waer ypes of communiy c. And, = { q ( ), ( ) (, )} wih q ( c, g ) being he QoWS level ha SWG provides waer ype g o communiy c. oe ha each ype of waer resources has been agged wih a specified QoWS. In addiion, each communiy may expec o consume muliple ypes of waer resources simulaneously, ha is each communiy has diverse QoWS
requiremens in one ime. W.o.l.g, we assume ha he acion of he waer resource allocaion is updaed in ime-slos. In he end of a ime-slo, he daa of he depleed volume, ( c, g ), which is measured by he waer meers, will be repored o he managemen cener. Since he daa of he allocaed volume { } a ime-slo is inherenly known a he cener, he difference of hese wo volumes can be direcly compued as = c c, g. In his conribuion, are ermed as he users conex informaion and insighfully, hey denoe he weighs of he links beween muliple resources and muliple communiies in a biparie graph. We furher assume ha he operaor has configured he buffering waer-pools for each communiy for leveraging he dynamics of he waer arrivals, which ypically leads o addiional coss. Therefore, we expec he really-consumed volume and he allocaed volume could be mached as beer as possible, even under he dynamics of boh he waer-resource supply and pipeline neworks. Considering he sysem uiliy in a long ime horizon, we define he averaged waer allocaion mismach as he objecive funcion, which is expressed as T 1 } = limsup ( w, n ), w, n, ( c, ). (1) { g T T =0 Our objecive is o minimize he mismach by allocaing differen ypes of waer resources for he end-users under he consrain of QoWS wih respec o he dynamics of he SWG. min { c }, (2), 0 c g c g s.. { c } { c }, ( c, g ), (3) q ( ), ( ) (, ), (4) where denoes he corresponding weigh, Eq. (3) consrains ha he averaged depleed volume canno exceed he allocaed volume, and Eq. (4) ensures o saisfy he QoWS requiremens. The solving of his problem is o find he opimal maching beween muliple waer resources and muliple waer communiies in a long-ime horizon. Generally, he end-users are locaed in a very broad area. Hence, even some of hem have he same profile and hen could be clusered ino one communiy, he number of communiies is ypically very large. This requires an efficien approach of communiy-clusering. Furhermore, he soluion of he problem should be adap o he dynamics of he SWG, which ypically involves in he highly-dimensional informaion, an online learning aided dimension-reduced dynamic programming approach is demanded herein. In a word, he managemen sysem should be self-organized o perform he opimal waer allocaion or he auonomic operaion/mainenance wih respec o he varying of differen ypes of informaion. IV. PROPOSED WATER RESOURCES ALLOCATIO SCHEME In his secion, he opimal waer resource allocaion is deermined wih he aid of conex informaion, while saisfying he QoWS of waer communiies. A. Clusering Communiies The end-users are clusered ino differen communiies based on he profiles of he end-users, which include he waer uilizaion habi, he geographic locaion of users, he waer volume and/or QoWS required by end-users. A specral clusering algorihm is developed wih he aid of specral graph heory, which has he advanage of clusering in he sample space of arbirary shape and shows good convergence o he global opimal soluion [9]. Our novel specral clusering mehod for clusering waer communiies is realized hrough consrucing adjacen
marix and gain marix (c.f. [9]). We assume ha he SWG consiss of end-users, which are classified ino K = waer communiies. The calculaion of he wo marices is given as follows: Adjacen Marix: Le denoe A as he adjacen marix in he nework ( A : ). If here is a connecion beween user i and j, which represens a common profile or ineres, hen i marked as A = 1, else A = 0. Furher defining he muual marix M, he elemen of M can be expressed as: = ika kj, where k =1 M A A and are he elemens of adjacen marix A, and furhermore A ika kj = 1 when user i and user j are boh have a connecion side wih user k, which represens he number of common neighborhoods (common ineress) beween user i and user j. Gain Marix: Le define l i and l j as he connecion degrees of user i and user j respecively, which denoe he numbers of he connecions beween he specified user and oher users in he nework. And furher define he number of common neighborhoods beween l random users pairs ( i, j) i l j as. We compue he gain funcion as: E ll ) = ( M, where A kj denoes he membership funcion in he communiy. If user i and j are in he same communiy, = 1; else, = 0. Concreely, E can be inerpreed as he difference beween he numbers of common neighborhood based on common ineress and ha based on random selecion. Then, we denoe he mark vecor of he communiy as (, ) s =, else S s s s. If user i belongs o he firs communiy, 1 = 1 2 1 s i = 1. We obain ha ( sis j 1) =, and hen 2 = 1 ll E ( M )( 1) 2 sis j. I can be expanded as 1 ll 1 ll E = ( M ) sis j ( M ), (5) 2 2 We only consider he coniguous iem for he communiies, namely he former iem, as 1 ll 1 ll ( M ) = ( ik kj ), 2 s s A A (6) 2 k =1 Finally, he gain marix C can be compued as d id j C = ( c ) = ( AikAkj ). Given he adjacen marix A and he gain k =1 marix C, he complicaed nework is firsly clusered ino wo communiies: calculae he principal eigenvecors of he larges eigenvalue of he gain marix C, and cluser he SWG nework ino wo waer communiies based on he symbols of he main elemens in he eigenvecors. Coninuously performing he same process wih his specral clusering mehod, each communiy may be divided ino differen smaller-sized communiies. The clusering process sops while deviaing he specified condiions. i
B. Q-Learning Aided Dynamic Resource Allocaion In his secion, he Q-learning aided self-organized algorihm is developed for waer resource allocaion. Afer he agen makes a decision based on he curren sae of he conex environmen, i receives he profi, eiher in posiive or negaive. If he profi of an acion is posiive, he probabiliy of his acion being seleced again is increased, oherwise i decreases. Considering ha he resource conex informaion w and he SWG sae n are varying beween differen waer allocaion ime-slos, we may informaion porray he solving of problem Eq.(2) wih an MDP approach. Le rewrie he consrained problem Eq. (2) ino an unconsrained MDP problem by virue of he Lagrange approach. We finally obain he corresponding value funcions { V ( f)} wih respec o saes {f }. WLOG., upon selecion of an acion, he agen should analyze he new sae ha i has ransied o. Mahemaically, he probabiliy of ransiion o sae f, saring from sae h can be expressed as P ( H = f H 1 = h, a 1)= P hf ( a 1), where a 1 is he acion aken a ime insance 1 H, he previous sae. corresponding o 1 Since he agen has no means of knowing if one acion seleced was good or no, a reward is required o leverage he measuremen. A posiive reward signifies a beneficial acion, while he negaive says ha i requires o furher ry oher poenial acions. While considering he long-erm reward, he value funcion can be formulaed by he Bellman funcion: V ( f)= R ( ( f)) P ( ) V ( h ) (7) h hf where V (f ) is he value funcion in sae f wih policy, is he discoun facor, and R ( ( f)) is he immediae reward, which has larger weigh han fuure rewards achieved when ransiing o he fuure sae h. The goal of RL is o find an opimal policy, which maximizes he oal discouned payoff V : V ( h )= max [ R ( h, a ) P ( a ) V ( f)]. (8) y fh The solving of he above problem requires he knowledge of he ransiion probabiliies, which are naurally unknown a prior. The mapping from saes o acions can be achieved over an infinie horizon of saes and acions wih he help of he Q-values. For each of he mapping, i associaed wih a Q-value: Q ( h, a )= R ( a ) P ( ( h )) Q ( f)]. (9) h f fh Given an opimal policy, he Q-values associaed wih Q are also maximized. Deermining he opimal policy is hus closely relaed o he deermining of he opimal Q-values. This can be realized a he agen hrough a recursive learning procedure. Q ( h, a )= Q ( f, a ) [ R max Q ( f, a ) Q ( h, a )] (10) a where is he learning rae and a is he seleced acion, which has caused he ransiion from he iniial sae h o he new sae f. The saes ha he agens use o characerize he environmen are given by I, QoWS and UCA :
QoWS 1, above 0, else hreshold, I 1, 0, for coverage no coverage (11) 1, UCA = 2, 3, high meduim low (12) where I is a binary indicaor of he resource coverage on he waer communiies (0 for no coverage, 1 for coverage) ), QoWS is a binary value illusraing if he waer communiy associaed wih a given resource is saisfied in QoWS (1 if he QoWS of he communiy c for he resource g is above he hreshold, else 0) and UCA is he waer volume level of he really-depleed, defined by {high-3, medium-2, low-1}. among differen communiies, where he QoWS requiremens are saisfied. The objecive of defining he saes in his way is o provide he idenical prioriy Again, he reward in Eq. (10) may be assigned in erms of he resource allocaion mismach: R = = c. (13) The Q-learning aided resource allocaion algorihm is deailed as: Sep1: Iniializaion, creae a Q-marix for each communiy wih he iniial elemens of Q-values seing o 0. Sep2: Compue I, QoWS and UCA, he value funcion and he reward R. Sep3: Compue he acion probabiliy, if 0.1, hen selec random acion; else selec acion corresponding o he maximum values of he Q-marix for he curren sae. Sep4: Compue he reward and evaluae new sae. Sep5: Updae he Q-marix. If convergs, ends he algorihm; else goes back o sep2; Figure 3: Volume percenage of differen resources 5 SIMULATIO RESULTS In his secion, we provide simulaion resuls for evaluaing he proposed waer allocaion schemes. Three ypes of conexual informaion are aken ino accoun, including he informaion of he allocaed volume, he QoWS, and he really-depleed volume. We assume here are 50 communiies in SWG, and 5 ypes of waer resources. In he Q-learning process,
we se he learning rae = 0.1. Fig. 3 depics he waer allocaion percenage beween differen communiies for a cerain ype of waer resource. We selec he scenario of 5 communiies as an example o illusrae he diversiy of he resource allocaion, while guaraneeing he communiy users QoWS requiremens. Fig. 4 illusraes he waer uilizaion efficiency. As expeced, he proposed scheme achieves higher efficiency over he random allocaion mehod. I implies ha he proposed dynamics allocaion approach has exploied he conex informaion for adaping he dynamics of he SWG, hus leading o an improved efficiency. Figure 4: Waer uilizaion efficiency 6 COCLUSIOS In his paper, we proposed a conex-awaree self-organized waer resource allocaion scheme for SWGs. A novel specral clusering approach was developed for clusering end-users ino waer communiies in erms of he end-users profiles. A Q-learning aided dynamic algorihm was proposed o deermine he opimal waer allocaion by virue of he MDP approach. Acknowledgemens: This research was suppored by a gran (12-TI-C01) from Advanced Waer Managemen Research Program funded by Minisry of Land, Infrasrucure and Transpor of Korean governmen and SF of China (61001127). REFERECES [1] B. David, C. T. Yin, ec., SMART-CITY: Problemaics, echniques and case sudies," Proc. of 8h In. Conf. on Compuing Tech. and Infor. Managemen (ICCM), pp. 168-174, Aug. 2012. [2] Y. He, Y. H. Liu, onineracive localizaion of wireless camera sensors wih mobile beacon," IEEE Trans. Mobile Compuing, vol. 12, no. 2, pp. 333-345, Feb. 2013. [3] M. I. Mohamed and W. Y. Wu, Power harvesing for smar Sensor neworks in monioring waer disribuion sysem," Proc. of IEEE Conf. on eworking, Sensing and Conrol, pp. 393-398, April 2011. [4] J. Hayes and K. Tong, A wireless sensor nework for monioring waer reamen," Proc. of IEEE Conf. on Sensor Technologies and Applicaions, pp. 514-519, Aug. 2007. [5] A. Osfeld, The bale of he waer sensor neworks: a design challenge for engineers and algorihms, J. of Waer Resources Planning and Managemen Division, ASCE, Vol. 134, no. 6, pp. 556-568, 2008. [6] M. Lin, Y. Wu and I. Wassell, Wireless Sensor ework: waer disribuion monioring sysem, Proc. IEEE Radio Wireless Symposium, pp. 775-778, 2008. [7] M. Mencarelli, M. Pizzichini and L. Gabrielli, Self-Powered sensor neworks for waer grids: challenges and preliminary evaluaions," IEEE J. Sel. Areas in Telecomm., pp.1-8, Oc. 2012. [8] F. L. Lewis and K. G. Vamvoudakis, Reinforcemen learning for parially observable dynamic processes: Adapive dynamic programming using measured oupu daa. IEEE Transacions on Sysems, Man, and Cyberneics, vol. 41, no. 1, pp. 14-25. 2011. [9] S. Z. iu, D. L. Wang, S. Feng, and G. Yu, An improved specral clusering algorihm for communiy discovery," In Proc of IEEE Hybrid Inelligen Sysems, pp. 262-267, Aug. 2009.