An Online Q-learning Based Multi-Agent LFC for a MultiArea Multi-Source Power System Including Distributed

Transcription

1 An Onlne Q-learnng Base Mult-Agent LFC for a MultArea Mult-Source Power System Inclung Dstrbute Energy Resources H. Shayegh*,**(C.A.) an A. Younes* Abstract: Ths paper presents an onlne two-stage Q-learnng base mult-agent (MA) controller for loa frequency control (LFC) n an nterconnecte mult-area mult-source power system ntegrate wth strbute energy resources (DERs). The propose control strategy conssts of two stages. The frst stage s employe a PID controller whch ts parameters are esgne usng sne cosne optmzaton (SCO) algorthm an are fxe. The secon one s a renforcement learnng (RL) base supplementary controller that has a flexble structure an mproves the output of the frst stage aaptvely base on the system ynamcal behavor. Due to the use of RL paragm ntegrate wth PID controller n ths strategy, t s calle RL-PID controller. The prmary motvaton for the ntegraton of RL technque wth PID controller s to make the exstng local controllers n the nustry compatble to reuce the control efforts an system costs. Ths novel control strategy combnes the avantages of the PID controller wth aaptve behavor of MA to acheve the esre level of robust performance uner fferent kn of uncertantes cause by stochastcally power generaton of DERs, plant operatonal conton changes, an physcal nonlneartes of the system. The suggeste ecentralze controller s compose of the autonomous ntellgent agents, who learn the optmal control polcy from nteracton wth the system. These agents upate ther knowlege about the system ynamcs contnuously to acheve a goo frequency oscllaton ampng uner varous severe sturbances wthout any knowlege of them. It leas to an aaptve control structure to solve LFC problem n the mult-source power system wth stochastc DERs. The results of RL-PID controller n comparson to the tratonal PID an fuzzy-pid controllers s verfe n a mult-area power system ntegrate wth DERs through some performance nces. eywors: Mult-Agent, Renforcement Learnng, DERs, LFC, RL-PID. Introucton T HE sze of the moern power systems s ncrease ue to the nterconnecton of varous energy resources to meet the ncreasng loa eman. Loa frequency control (LFC) of such power systems s more complex []. In aton, the amount of renewable Iranan Journal of Electrcal & Electronc Engneerng, 07. Paper frst receve 4 May 06 an accepte 05 November 07. * The authors are wth the Department of Electrcal Engneerng, Unversty of Mohaghegh Arabl, Arabl, Iran. E-mals: hshayegh@gmal.com an a.younes@eee.org. ** The author s wth the Centre of Excellence for Power System Automaton an Operaton, Department of Electrcal Engneerng, Iran Unversty of Scence an Technology, Tehran, Iran. Corresponng Author: H. Shayegh. energy proucton contnuously on the rse ue to serous ssues lke global warmng, weak transmsson lnes, an outate nfrastructure of the power systems. On the other han, nherent uncertantes n the output of the renewable energy resource such as photovoltac (PV) an wn turbne generator (WTG) makes that the effcent frequency controller esgnng be more ffcult. Moreover, an aaptve control strategy s requre for nterconnecte operaton moes [-]. Note that, the moern power system s subjecte to frequency an te-lne power flow oscllatons ue to hourly energy proucton of DERs an loa changngs [4]. Thus, f a sutable controller s not consere for provng a goo ampng, the oscllatons may persst for long tme, causng sntegraton of the system [5]. 85

2 Many researchers have employe varous kn of LFC strateges to mantan the frequency an te-lne power flow of the electrc network n ther corresponng values [6-5]. Among the consere LFC methos, tratonal controllers are the most wely use because of ther smplcty, easy realzaton, low cost, an sutable relablty [6, 8,, 6]. In aton, fferent optmzaton algorthms lke genetc algorthm, partcle swarm optmzaton, fferental search algorthm, an teachng-learnng base optmzaton algorthm [7] are use for optmzng the ynamc performance of the tratonal controllers. Snce the fxe gans of a tratonal controller are esgne uner a loang conton, t can t guarantee the performance of the power system n the other operatng contons. Ths weakness shoul be resolve n LFC of mult-area mult-source power systems especally when they are ntegrate wth the renewable energy resources (RERs) ue to nherent uncertantes n the output of ths type of energy sources. In the past several ecaes, fuzzy logc base controllers (FLCs) have been extensvely use to cope wth the power system uncertantes cause by changes n the system parameters an nterconnecton of stochastc DERs [,, 4]. However, several serous problems regarng FLCs are reporte n Refs. [, 7]: ) Robustness s often assume as a funamental property of FLCs, thus, t s not taken nto account urng the tunng proceure. In fact, that s false. ) Dfferent parts of the FLCs such as membershp functons (the numbers, lmts, an the functon types), fuzzy rules, an control gans shoul be optmze coornately to acheve the optmal performance of the FLCs, whch s tme-consumng an a teous work wth an enormous computatonal buren. In the recent years, fferent applcatons of the Qlearnng soluton of RL s reporte n the power systems [8-]. As reporte n Ref. [8], RL base methos can cope wth system nonlneartes an stochastc behavors base on the partal nformaton. They on t nee to any knowlege about the system ynamcs. These characterstcs are very useful for solvng the LFC problem n a mult-area mult-source power system wth RERs, to cope wth serous problems such as mensonalty an fferent kns of the uncertantes cause by system parameter changngs, physcal nonlneartes, an fckle output of the RERs. Snce RLbase control methos learn the close-loop control laws an upate them contnuously, they are known to be robust. Aaptve behavor s an mportant property for these controllers when the power system s face wth a stuaton, whch s not experence n ts synthess proceure. Because, RL paragm upates ts knowlege about the system ynamcs contnuously. Thus t can prove an aaptve ampng control sgnal. Furthermore, MA-base controllers can be use n combnaton wth tratonal control methos to mprove performances. Ahme et. al [] suggeste two stnct RL base AGC algorthms. The frst algorthm 86 works base on the area control error (ACE) sgnal, an the secon one s base on the restorng the loa generaton balance. Ths stuy has two funamental flaws, havng large overshoot an slow system response. These shortcomngs are ue to mproper selecton of the MA acton sets. The nherent complextes of the LFC emphasze on effectve control strategy selecton to amp the frequency an te-lne power flow oscllatons perfectly. Accorng to the applcablty of RL paragm as a supplementary controller, here, RL technque s employe to supervse a classc PID controller, whch s wely use n nustry, to esgn an aaptve controller for solvng LFC problem. The propose control strategy s compose of two stages. The frst stage s employe a PID controller whch ts gans are optmze wth the sne-cosne (SCO) algorthm. The secon stage s a supplementary control sgnal that s prove usng RL technque. The well-known Q-learnng approach s use for solvng RL n ths stuy. It shoul be mentone that the propose control strategy has two offlne an onlne moes. The ntellgent agents use the offlne moe for learnng the optmal control polcy of LFC task. In the onlne moe, agents optmally control the plant by makng the learne control laws an upate ther knowlege, too. The propose LFC scheme successfully s apple to a three-area mult-source power system nclung DERs an physcal nonlneartes such as tme elay (TD), generaton rate constrant (GRC), an governor ea ban (GDB). Tme oman performance of the RL-PID controller compare to PID an fuzzypid controllers. The prmary nvestgatons of the present work are: ) To suggest an aaptve structure base on the RL technque for supervsng the exstng PID controller for LFC task. ) To verfy the robustness of the RL-PID controller uner we changes n loang pattern of RERs an severe system parameter changes. ) To stuy the ynamc performance of the propose RL-PID controller n a mult-area mult-source power system nclung DERs an GRC, GDB an tme elay. v) To compare the performance of the suggeste RLPID controller wth the tratonal optmze PID an fuzzy-pid controllers by SCO algorthm the above power system. Power System Moellng for AGC Problem A three-area power system ntegrate wth PV, WTG, an DEG unts s utlze to verfy the ynamc performance of the propose RL-PID controller for solvng LFC problem. All areas have a thermal unt an a hyropower staton. Area, Area, an Area have PV, DEG, an WTG, respectvely. The smplfe representaton of the moern power system n a general form s shown n Fg..

3 Area # Area # Area # Fg. General scheme of the propose three-area mult-source power system. /Rhy apf- ACE st g - Tt s Controller Governor wth GDB TD Tme elay apf- Δf st rs st rh st gh apf- rt r s st r Turbne wth GRC Mechancal Hyraulc Governor /RPV AREA# ΔP /Rth TW s 0.5TW s t s Δf P TP s Power system Hyraulc Turbne wth GRC ΔP PV 8s 900 s 00s 500 PV /RDEG apf- DEG T DEG s DEG /Rwn apf- wn ( stw ) ( stw )( s ) ΔP wn Wn Fg. Block agram of the th area nclung thermal, hyropower an DESs... Transfer Functon of The Thermal an Hyro Power Statons Transfer functon moel of the area # of the power system nclung a thermal power plant wth a hyropower staton s shown n Fg. [4, 5]. As shown n ths fgure, TD, GDB, an GRC are consere as the system physcal lmtatons.. The Transfer Functon Moel of DERs The transfer functons of the PV an DEG are represente by Eqs. () an (), respectvely [6, 7]. 8s 900 s 00s 50 DEG P GDEG ( s) DEG stdeg f GPV ( s) () () 87

4 The ptch control hyraulc actuator transfer functon of WTG s gven by Eq. () [8, 9]: propose PID controller are tune usng the SCO algorthm []. Wn ( stw ) H ( s) U ( s) (Tk s stw )( s)... Sne Cosne Algorthm, an Overvew () snce Tk s smaller than Tw, t can be gnore. Thus, Eq. () can be rewrtten as Eq. (4). H ( s) Wn ( stw ) U ( s) ( stw )( s) (4). The Propose Controller A smple llustraton of the propose two-stage RLPID controller s shown n Fg.. In ths fgure, u s the control sgnal (area control error (ACE) n LFC problem), an y s the output of the controller. A stnct avantage of the propose controller s that we can apply the stage two for upatng the ynamc performance of the exstng PID controller n LFC task to acheve the esrable ynamc response characterstcs.. Frst Stage: The Classcal Controller The applcaton of the PID controller n the nustry s noteworthy. They are extensvely use n nustry ue to ther smplcty, easy to mplement an the aequate performance they prouce for a broa range of the process [0]. Conserng the avantages of the PID controller, t s a sutable choce for employng n the frst stage of the propose strategy. Eq. (5) shows the mathematcal formulaton of the conventonal PID controller. y p u ut t (5) The founaton of the SCO algorthm s bult on the trgonometrc sne an cosne functons. In general, the optmzaton process can be ve nto two phases, exploraton, an explotaton. In the frst phase, the algorthm ranomly combnes the ranom solutons to fn the best area of avalable solutons. But, n the secon phase, the changng n the obtane solutons s graual wth a lower egree of ranomness n comparson to the frst phase []. Eq. (6) s use for upatng the poston of the populaton nvuals n both phases. t t t X r sn(r ) r P X r4 0.5 X t t t t X r cos(r ) r P X r4 0.5 (6) where, X t s the poston of the menson n the teraton t, r-r4 are ranom numbers, an Pt s the poston of the target pont n teraton t of menson. The orentaton an amount of the movement are etermne usng parameters r an r, respectvely. The parameter r s a weght coeffcent whch ncreases (r>) or ecreases (r<) the effect of the target pont. SC algorthm s explane n etal n []. Fg. 4 shows the flowchart of the SCO algorthm... Optmzaton results Snce the power system uner stuy s not symmetrc, n each area one PID controller s consere. Each controller has three control gans. Thus, the propose optmzaton problem has nne parameters shoul be tune. The objectve functon, whch s use for optmzng the PID controllers, s gven by Eq. (7). 00 where, p,, an are proportonal, ntegral, an ervatve gans, respectvely. Also, u s the nput sgnal to the controller. Here, the controller gans of the J ITAE t ( F F F 0 (7) Pte Pte )t Start Calculatng the ftness functon for all members of the populaton Intalzaton Sne cosne algorthm u y PID controller Exportng the best solutons Yes Stop crtera reache? Frst stage Preprocessng No Upate... Q-learnng base Mult-Agent controller Target pont Secon stage The propose Two stage RL-PID controller Fg. Illustratve moel of the propose aaptve Q-learnng base PID controller. 88 r,r, r, an r 4 The poston of the nvuals, Eq.(0) Fg. 4 flowchart of the SCO algorthm.

5 Optmal tunng of the PID controllers s formulate as the followng constrant optmzaton problem an solve usng SCO algorthm. Table Optmal results of the PID controller parameters. Parameters p Value p p Mnmze J subject to: mn p,, mn,, mn,, p p p max p (8) max max In ths way, frst populaton an the maxmum number of teratons are consere as 0 an 50, respectvely. Upper an lower lmts of all parameters are consere equal to 0 an, respectvely. The optmzaton results are shown n Table.. Secon Stage: The Aaptve Renforcement Learnng Base Controller Renforcement learnng s an algorthmc approach to solve stochastc optmal control problems by tral-anerror [8]. It escrbes how one (or more than one) agent nteracts wth ts envronment to learn an optmal control polcy for satsfyng a pre-efne goal. Optmal control polcy means selectng the best acton among all actons that exst for each state of the agent n the envronment []... Q-Learnng Q-learnng s one of the most known soluton methos of the renforcement learnng whch wll be use n ths paper. Smple structure, nepenent of the moel of the system uner control, robustness aganst changes n the operatng pont an system uncertantes an aaptve behavor are the most mportant avantages of the Qlearnng base control methos [, 4]. Ths control metho can be utlze as a complementary controller for tratonal controllers [] to mprove ther performances. Q-learnng base renforcement learnng assumes the envronment (system uner control) s ve nto a fnte number of states s shown wth set {S}. Agent forms a matrx calle Q, whch has a value (ntally 0 ) for each set of acton-state pars an ncates the gooness of partcular acton n the corresponng state. In each tme step, agent calculates ts state st, an base on a efne strategy selects acton a among avalable actons of stat st {A}. Immeately after applyng the acton, the agent takes a rewar r from the envronment an calculates ts next state st. Then t upates the corresponng element of the Q matrx. The rewar r shows the amount of satsfacton from the acton a. Ths proceure wll contnue untl satsfacton of pre-efne goal. The purpose of the Qlearnng s to learn a strategy whch maps the states to actons to maxmze scounte long-term rewar []. Dscounte long-term rewar of the system s gven by Eq. (9). Rt k rt k (9) k 0 where, r s the rewar, γ s a number at the range 0 to an s calle scount factor. Ths coeffcent shows the mportance of the future rewars n ecson makng. Settng γ = 0 means, the future rewars are gnore n ecson makng an settng γ = means next rewars are consere n the ecson-makng process [4]. Q matrx s efne as: Q ( s, a ) E k rt k st s, at a k 0 (0) where, π, s, a, an r are the control polcy, current state, selecte acton, an the receve rewar, respectvely. In each tme step, Eq. (0) shoul be upate usng optmal Bellman equaton, whch s gven by Eq. (). Q rt max Q( st, at ) Q( st, at ) a () where, α s ϵ (0,) an s calle attenuaton factor an shows the real amount of error []. The flowchart of the propose Q-learnng metho s summarze n Fg. 5. It s event from Fg. 5 that after completng the learnng phase (offlne smulaton), the system wll be swtche to onlne smulaton... States, Actons, an Rewar Defnton To acheve the esre level of satsfactory from Qlearnng controller t s necessary to efne the set of the states, actons, an rewar functon carefully. In the followng, states, actons, an rewar functon efnton for AGC problem are escrbe n etal. A. States The prmary am of AGC s to balance the generaton an eman to reuce the frequency fluctuatons. In other wor, ths controller shoul amp the oscllatons of the frequency an te-lne power flow. Thus, f or 89

6 Q-learnng metho Offlne smulaton Upate the epso crtera Set all Q(s,a ) = 0 Memory Swtch to onlne smulaton Set values of α, γ, an ε Calculate current state Select an acton wth ε-reey metho. (0.0< ε <0.0) Apply the selecte acton to the envronment, then get the rewar an calculate the next state Yes Upate the Q matrx for the current state usng Eq. () Put the next state at the current state Defne set of states {S}, actons {A}, an rewar Epsoe crtera reache? Onlne smulaton No Calculate current state Yes Is the goal been acheve? ε-greey metho No generatng a ranom number n the range [0 ] Select the acton wth hghest value of Q n the coresponng state Yes Is > ε (0.< ε <0.6) No Select an acton ranomly Put the next state at the current state Upate the Q matrx for the current state usng: Q=Q Q OR Apply the selecte acton to the envronment, then get the rewar an calculate the next state () Fg. 5 Flowchart of the Q-learnng metho. Pte an or a combnaton of them can be use for state efnton. Here, f s ntene for ths purpose. In each tme step for calculaton of the current state of the agent of area, f an ts ervatve are use. The range of 0. to 0. n f s scretze to 50 equal segments. Eq. () calculates the state of the agent of area n each the tme step t. F,t () S,t ( F,t, ) t B. Actons Defnng the set of actons s very complex an mportant. For smplcty base on a tral-an-error metho an a set of three actons as Eq. () s consere for each state. Increasng the number of actons may mprove the performance of the propose controller but also, ncreases the tme of learnng 90 proceure. A { 0.00,0,0.00} () C. Rewar Functon Snce, the purpose of ths paper s to amp the frequency oscllatons, the rewar of the agent of area n tme t s consere as the evaton of f n area an all other areas. The rewar functon s calculate usng Eq. (4). Rewar,t ( t f [k ] k t (w j j ) t k t f j [k ] ) (4)

7 where, wj s the weght coeffcent of area j (j ) an s consere equal to 0.5 for areas that are rectly connecte to area an equal to 0. for the other areas. Also, ε, α, an γ are consere as 0.0, 0.05, an 0.98, respectvely. fferent realstc power system scenaros. The control parameters of the use fuzzy-pid controller are optmze usng SCO algorthm n the range [- ] an shown n Table. The etals of the employe fuzzypid controller are escrbe n [5]. 4 Smulaton Results 4. Scenaro The suggeste RL-PID controller s mplemente n a three-area mult-source power system. The moel of the system uner stuy has been evelope n MATLAB/SIMULIN envronment an the RL base supplementary control sgnal for LFC task has been prove usng a.m fle. Power system parameters are gven n Appenx A. The propose control strategy utlze for LFC n the above power system compare to the SCO-tune PID an fuzzy-pid controllers [5] n In ths scenaro, the ynamc performances of the LFC wth % step loa perturbaton (SLP) an 5% change n the tme constant of the governors n all areas, uner the acton of RL-PID, PID, an fuzzy-pid controllers are analyze. The fluctuatons n the frequency an telne power flow of the power system are represente n Fgs. 6 an 7. The actons were taken by the agents of each area are shown n Fg. 8. Table Optmal results of the fuzzy PID controller parameters. Area # Parameters 4 5 Value Area # Parameters 4 5 Value Area # Parameters 4 5 Value Fg. 6 Frequency evatons for % ncreasng n loa eman n scenaro. 9

8 Fg. 7 Te-lne power flow evatons for % ncreasng n loa eman n scenaro. Fg. 8 The actons of the agent of each area n scenaro. It s event from Fgs. 6 an 7 that the propose RLPID controller effectvely ampe the oscllatons faster than the tratonal PID an fuzzy-pid controllers. Accorng to Fg. 8, untl the frequency evatons n each area are n the normal state (± n ths paper), the RL controller s nactve an LFC task wll perform entrely by the PID controller. However, when the evatons of the frequency n each area go out of the normal state ue to any sturbance n any area, the RL controller proves the suffcent complement control sgnal an mproves the frequency oscllaton 9 ampng. It s event that when the evatons returne to the normal state the RL controller becomes nactve agan. 4. Scenaro To verfy the superorty of the propose aaptve LFC strategy, the ynamc response of the power system wth % SLP n Area, the stochastc output of WTG n area an PV n Area n aton to -5% change n Tr an r n all areas are plotte n Fgs. 9-.

9 Fg. 9 Frequency evatons n scenaro. Fg. 0 Te-lne power flow evatons n scenaro. Accorng to the Fgs. 9 an 0 t s clear that the superb ampng performance of RL-PID n term of settlng tme an overshoot of frequency responses an te-lne power evatons are better than other controllers. The complement ampng sgnals that are prove by the agent of each area are shown n Fg.. controller aganst loang conton changng, n aton to % SLP n area, the loang conton s change by 0% change n the DC gan of the power system (p) an -0% change n the tme constant of the power system (Tp) at a same tme. The ynamc performances of the LFC controllers are shown n Fgs. -4 n ths scenaro. 4. Scenaro To show the robustness of the suggeste RL-PID 9

10 Fg. Te-lne power flow evatons for % ncreasng n loa eman n scenaro. Fg. Frequency evatons n scenaro. Fgs. an show that the RL-PID controller s less senstve to loang conton varatons. Furthermore, to show the superorty of the propose RL-PID controller over the fuzzy-pid an PID controllers optmze by SCO algorthm, tme oman performance nces such overshoot(os), settlng tme (Ts), ITAE, an ISE are calculate an shown n Table. ITAE an ISE are calculate usng Eqs. (6) an (7) ITAE f [n ] (6) n 50 ISE 00 f [ n] (7) n As the results s prove that the propose controller s aaptve, smple an robust aganst parameter an loang conton changng.

11 Fg. Te-lne power flow evatons n scenaro. Fg. 4 The actons of the agent of each area n scenaro. From Table the mprovement n the relatve tme oman performance nces can be seen. Accorng to the results, we can see that oscllatons n area two of the power system are mnuscule. Ths can be ue to the presence of DEG n ths area. The propose control strategy has mprove the performance of the tratonal PID controller even better than fuzzy PID controller. Although the propose controller s enhance all characterstcs of the evatons n frequency an te- lne power flow, the largest mpact s on the settlng tme. Snce the RL-PID controller s base on screte tme smulaton, t s reasonable to have a lttle mpact on the overshoot/unershoot. Here, samplng tme consere as 0.05 secon. Reucng the samplng tme mproves the mpact of the propose controller on the overshoot/unershoot an ts overall performance, but ncreases the smulaton tme an tme to learnng proceure, too. 95

12 Table Comparson of tme oman performance nces for SC-PID, SC-FPID, an RL-PID controllers. Scenaro Overshoot (%) Settlng Tme (n) ITAE Δf Δf Δf Δf Δf Δf Δf PID FPID RL-PID Scenaro Overshoot (%) Settlng Tme (n) ITAE Δf Δf Δf Δf Δf Δf Δf PID > FPID RL-PID ISE Δf ISE Δf Conclusons Appenx A. Power System Parameters In the present work, a two-stage aaptve MA-base controller has been presente for LFC stuy of a threearea mult-source power system nclung stochastc DERs an system physcal nonlneartes. The suggeste control strategy s compose of a tratonal PID controller that s tune usng SCO algorthm an supervse by a supplementary control sgnal, whch s prove usng Q-learnng metho of RL paragm. The propose control scheme combnes the features of the tratonal PID controller lke smplcty, sutable relablty, an easy realzaton wth the mportant characterstcs of mult-agent systems such as aaptve behavor, nepenent from the system moel, an robustness aganst fferent kns of uncertantes to evelop an effectve controller for solvng the LFC problem n the moern power systems. Smulatons are carre out n two phases, offlne phase an onlne phase. The offlne phase s base on the exploraton an the ntellgent agents learn the optmal polcy of the LFC through nteractng wth the envronment. The secon phase s base on the explotaton an the autonomous agents optmally perform the LFC task base on the learne optmal control laws an upate ther knowlege. Eventually, the ynamc performance of the propose RL-PID controller s verfe compare to SCO tune PID an fuzzy-pid controllers n the stuy of the LFC of a three-area moern power system conserng system nonlneartes n some realstc scenaros. Accorng to the results, t s event that the ynamc performance of the suggeste RL-PID controller s superb compare to both SCO tune PID an fuzzy-pid controllers n terms of settlng tme, overshoot, ITAE, an ISE. Furthermore, t s foun that the RL-PID controller s more robust an stable aganst fferent kns of uncertantes ue to the stochastc output of DERs an changes n the plant parameters an the system loang conton. In the other han, the smple ea an structure of the propose control strategy bese ts amazng propertes make t a sutable choce to mplement n the real-tme applcatons. Data of The System 96 B = 0.45; B = B; B = B; ps = 0; ps = ps; ps = ps; Tps = 0; Tps = Tps; Tps = Tps; RTH =.4; RTH = RTH; RTH = RTH; RHY = RTH; RHY = RTH; RHY = RTH; Tt = 0.; Tt = Tt; Tt = Tt; Tt4 = Tt; Tsg = 0.08; Tsg = Tsg; Tsg = Tsg; Tsg4 = Tsg; r = 0.5; r = r; r = r; r4 = r; Tr = 0; Tr = Tr; Tr = Tr; T = 0.04; T = 0.04; Tgh = 48.7; Tgh = Tgh; Tgh = Tgh; Tw = ; Tw = Tw; Tw = Tw; Trs = 0.5; Trs = Trs; Trs = Trs; Trh = 0; Trh = Trh; Trh = Trh; a =-; a = -; Generaton Rate Constrant (GRC) an Tme Delay (TD) GRCth = 0.0; TD = 0.05; GRChy_ub =.7; GRChy_lb =.6; apf = 0.4; apf = 0.; apf = 0.; apf = 0.4; apf = 0.4; apf = 0.; apf = 0.4; apf = 0.; apf = 0.; PV pv =, Tpv =.8; Rpv =.4; Dsel Generator sel = 00-; Tesel = ; Rsel =.4; Wn Farm RWn =.4; Twn = 6; Twn = 0.04; wn =.5; wn =.4 References [] H. Shayegh, A. Younes, A robust screte fuzzypfuzzyifuzzyd loa frequency controller for mult-source power system n restructurng envronment, Journal of Operaton an Automaton n Power Engneerng, Vol. 5, No., pp. 6-74, 07.

13 [] P.. Hota an B. Mohanty, Automatc generaton control of mult source power generaton uner eregulate envronment, Internatonal Journal of Electrcal Power & Energy Systems, Vol. 75, pp. 05-4, 06. [] F. Daneshfar an E. Hossen, Loa-frequency control n a eregulate envronment base on bsecton search, Iranan Journal of Electrcal & Electronc Engneerng, Vol. 8, pp. 0-0, 0. [4] H. Rajab Mashha, S. M. Eslam, an H. Mor Shanech, Analyss of wn spee forecastng error effects on automatc generaton control performance, Iranan Journal of Electrcal & Electronc Engneerng, Vol. 0, pp. -9, 04. [5] O. Abena, N. Amja, A. Ghasem, H. Shayegh, Mult-stage fuzzy loa frequency control base on mult-objectve harmony search algorthm n eregulate envronment, Journal of Operaton an Automaton n Power Engneerng, Vol., No., pp. 6-7, 0. [6] A. hoabakhshan an R. Hooshman, A new PID controller esgn for automatc generaton control of hyro power systems, Internatonal Journal of Electrcal Power & Energy Systems, Vol., pp. 75-8, 00. [7] D. Guha, P.. Roy, an S. Banerjee, Stuy of fferental search algorthm base automatc generaton control of an nterconnecte thermalthermal system wth governor ea ban, Apple Soft Computng, Vol. 5, pp , 07. [8] N. Pathak, T. S. Bhatt, an A. Verma, New performance nces for the optmzaton of controller gans of automatc generaton control of an nterconnecte thermal power system, Sustanable Energy, Grs an Networks, Vol. 9, pp. 7-7, 07. [9] Y. Xu, F. L, Z. Jn, an C. Huang, Flatness-base aaptve control (FBAC) for STATCOM, Electrc Power Systems Research, Vol., pp , 05. [0] B.. Sahu, S. Pat, P.. Mohanty, an S. Pana, Teachng learnng base optmzaton algorthm base fuzzy-pid controller for automatc generaton control of mult-area power system, Apple Soft Computng, Vol. 7, pp , 05. [] M. H. hooban an T. Nknam, A new ntellgent onlne fuzzy tunng approach for mult-area loa frequency control: Self Aaptve Mofe Bat Algorthm, Internatonal Journal of Electrcal Power & Energy Systems, Vol. 7, pp. 54-6, 05. [] D.. Sahoo, R.. Sahu, G. T. C. Sekhar, an S. Pana, A novel mofe fferental evoluton algorthm optmze fuzzy proportonal ntegral ervatve controller for loa frequency control wth thyrstor controlle seres compensator, Journal of Electrcal Systems an Informaton Technology, 06. [] P. C. Prahan, R.. Sahu, an S. Pana, Frefly algorthm optmze fuzzy PID controller for AGC of mult-area mult-source power systems wth UPFC an SMES, Engneerng Scence an Technology, an Internatonal Journal, Vol. 9, pp. 8-54, 06. [4] D.. Chaturve, R. Umrao, an O. P. Malk, Aaptve polar fuzzy logc base loa frequency controller, Internatonal Journal of Electrcal Power & Energy Systems, Vol. 66, pp , 05. [5] H. Yousef, Aaptve fuzzy logc loa frequency control of mult-area power system, Internatonal Journal of Electrcal Power & Energy Systems, Vol. 68, pp , 05. [6] M. R. Sathya an M. Mohame Thameem Ansar, Desgn of bogeography optmzaton base ual moe gan scheulng of fractonal orer PI loa frequency controllers for mult source nterconnecte power systems, Internatonal Journal of Electrcal Power & Energy Systems, Vol. 8, pp. 64-8, // 06. [7] P. Alberto an A. Sala, Fuzzy logc controllers. Avantages an rawbacks., n XIII Congreso e la Asocaton Chlena e Controlo Automatco, 998, pp [8] D. Ernst, M. Glavc, an L. Wehenkel, Power systems stablty control: renforcement learnng framework, IEEE Transactons on Power Systems, Vol. 9, pp , 004. [9] T. Yu an W. G. Zhen, A renforcement learnng approach to power system stablzer, n Proc. of the IEEE Power & Energy Socety General Meetng, Calgary, AB, pp. -5, 009. [0] X. Zhang, T. Yu, B. Yang, an L. Cheng, Acceleratng bo-nspre optmzer wth transfer renforcement learnng for reactve power optmzaton, nowlege-base Systems, Vol. 6, pp. 6-8, 07. [] J. G. Vlachoganns an N. D. Hatzargyrou, Renforcement learnng for reactve power control, IEEE Transactons on Power Systems, Vol. 9, pp. 7-5, 004. [] V. Nanur an T.. Das, A Renforcement Learnng Moel to Assess Market Power Uner Aucton-Base Energy Prcng, IEEE Transactons on Power Systems, Vol., pp , 007. [] T. P. Imthas Ahame, P. S. Nagenra Rao, an P. S. Sastry, A renforcement learnng approach to automatc generaton control, Electrc Power Systems Research, Vol. 6, pp. 9-6, 00. [4] T. S. Gorrpotu, R.. Sahu, an S. Pana, AGC of a mult-area power system uner eregulate envronment usng reox flow batteres an nterlne power flow controller, Engneerng Scence an Technology, an Internatonal Journal, Vol. 8, pp ,

14 [5] S. R. hunta an S. Pana, Smulaton stuy for automatc generaton control of a mult-area power system by ANFIS approach, Apple Soft Computng, Vol., pp. -4, 0. [6] D. J. Lee an L. Wang, Small-sgnal stablty analyss of an autonomous hybr renewable energy power generaton/energy storage system Part I: Tme-oman smulatons, IEEE Transactons on Energy Converson, Vol., pp. -0, 008. [7] S. A. Je, S. H. Abbas, an F. Shabanna, Loa frequency control of two area nterconnecte power system (Desel Generator an Solar PV) wth PI an FGSPI controller, n The 6th CSI Internatonal Symposum on Artfcal Intellgence an Sgnal Processng (AISP 0), pp. 56-5, 0. [8] D. Das, S.. Atya, an D. P. othar, Dynamcs of esel an wn turbne generators on an solate power system, Internatonal Journal of Electrcal Power & Energy Systems, Vol., pp. 8-89, 999. [9] R.. Sahu, T. S. Gorrpotu, an S. Pana, Automatc generaton control of mult-area power systems wth verse energy sources usng teachng learnng base optmzaton algorthm, Engneerng Scence an Technology, an Internatonal Journal, Vol. 9, pp. -4, 06. [0] R. D. O. Perera, M. Verones, A. Vsol, J. E. normey-rco, an B. C. Torrco, Implementaton an test of a new autotunng metho for PID controllers of TITO processes, Control Engneerng Practce, Vol. 58, pp. 7-85, 07. [] S. Mrjall, SCA: a sne cosne algorthm for solvng optmzaton problems, nowlege-base Systems, Vol. 96, pp. 0-, 06. [] R. Ha an B. Jeyasurya, Renforcement learnng base real-tme we-area stablzng control agents to enhance power system stablty, IEEE Transactons on Smart Gr, Vol. 4, pp , 0. [] C. J. C. H. Watkns an P. Dayan, Techncal note: Q-Learnng, Machne Learnng, Vol. 8, pp [4] N. Vlasss, A Concse Introucton to Multagent Systems an Dstrbute Artfcal Intellgence, Morgan an Claypool Publshers, 007. [5] H. Shayegh, A. Younes, an Y. Hashem, Optmal esgn of a robust screte parallel FP FI FD controller for the automatc voltage regulator system, Internatonal Journal of Electrcal Power & Energy Systems, Vol. 67, pp , H. Shayegh receve the B.S. an M.S.E. egrees n Electrcal an Control Engneerng n 996 an 998, respectvely. He re ceve hs Ph.D. egree n Electrcal Engneerng from Iran Unversty of Scence an Technology, Tehran, Iran n 006. Currently, he s a full Professor n Techncal Engneerng Department of Unversty of Mohaghegh Arabl, Arabl, Iran. Hs research nterests are n the applcaton of robust control, artfcal ntellgence an heurstc optmzaton methos to power system control esgn, operaton an plannng an power system restructurng. He has authore an co-authore of 5 books n Electrcal Engneerng area all n Fars, one book an two book chapters n nternatonal publshers an more than 0 papers n nternatonal journals an conference proceengs. Also, he collaborates wth several nternatonal journals as revewer boars an works as etoral commttee of three nternatonal journals. He has serve on several other commttees an panels n governmental, nustral, an techncal conferences. He was selecte as stngushe researcher of the Unversty of Mohaghegh Arabl several tmes. In 007 an 00 he was also electe as stngushe researcher n engneerng fel n Arabl provnce of Iran. Furthermore, he has been nclue n the Thomson Reuters lst of the top one percent of most-cte techncal Engneerng scentsts n 05 an 06, respectvely. Also, he s a member of Iranan Assocaton of Electrcal an Electronc Engneers (IAEEE) an Senor member of IEEE. A. Younes receve B.S an M.S.E egrees both n Electrcal Engneerng from Faculty of Techncal Eng. Department of the Mohaghegh Arabl Unversty, Arabl, Iran n 0 an 05 respectvely. Currently He s a PhD. stuent n Techncal Eng. Department of the Unversty of Mohaghegh Arabl, Arabl, Iran. Hs areas of nterest are applcaton of artfcal ntellgence n power system automaton an control, applcaton of Renforcement Learnng to power system control, Fuzzy Systems, Heurstc optmzaton n power system control. He s a stuent member of Iranan Assocaton of Electrcal an Electronc Engneers (IAEEE) an IEEE.