Adaptive PID Controller Based on Reinforcement Learning for Wind Turbine Control

Transcription

1 Adaptive PID Controller Baed on Reinforcement Learning for Wind Turbine Control D. Arzaghi-Hari Engineering Dept. Ilamic Azad Univerity of Saveh Saveh, IRAN Abtract: - A elf tuning PID control trategy uing reinforcement learning i propoed in thi paper to deal with the control of wind energy converion ytem (WECS). Actor-Critic learning i ued to tune PID parameter in an adaptive way by taking advantage of the model-free and on-line learning propertie of reinforcement learning effectively. In order to reduce the demand of torage pace and to improve the learning efficiency, a ingle RBF neural network i ued to approximate the policy function of Actor and the value function of Critic imultaneouly. The input of RBF network are the ytem error, a well a the firt and the econd-order difference of error. The Actor can realize the mapping from the ytem tate to PID parameter, while the Critic evaluate the output of the Actor and produce TD error. Baed on TD error performance index and gradient decent method, the updating rule of RBF kernel function and network weight were given. Simulation reult how that the propoed controller i efficient for WECS and it i perfectly adaptable and trongly robut, which i better than that of a conventional PID controller. Key word: wind energy converion ytem, reinforcement learning; Actor-Critic learning; adaptive PID control; RBF network Introduction A a reult of increaing environmental concern, the impact of conventional electricity generation on the environment i being minimized and effort are made to generate electricity from renewable ource. The main advantage of electricity generation from renewable ource are the abence of harmful emiion and the infinite availability of the prime mover that i converted into electricity. One way of generating electricity from renewable ource i to ue wind turbine that convert the energy contained in flowing air into electricity. Variou electromechanical cheme for generating electricity from the wind have been uggeted, but the main drawback i that the reulting ytem i highly nonlinear, and thu a nonlinear control trategy i required to place the ytem in it optimal generation point. Different intelligent approache have uccefully been applied to identify and nonlinearly control the WECS and other plant. For intance, Kanello and Hatziargyriou [], Yong-tong and Cheng-zhi [] and Zhao-da et al [] have uggeted neural network a powerful building block for nonlinear control trategie. The mot famou topologie for thi purpoe are multilayer perceptron (MLP) and radial bai function (RBF) network [4]. Mayoky and Cancelo [5] propoed a neural-network-baed tructure for Wind turbine control that conit of two combined control action, a uperviory control and an RBF network-baed adaptive controller. Sedighizadeh et al [6,7,8] uggeted an adaptive controller uing neural network frame Morlet wavelet together with an adaptive PI controller uing RASP wavenet for Wind turbine control. In thi paper, the reinforcement learning i ued to deign of controller. Thi learning method unlike upervied learning of neural network adopt a trial and error mechanim exiting in human and animal learning. Thi method emphaize that an agent can learn to obtain a goal from interaction with the environment. At firt, a reinforcement learning agent exploit the environment actively and then evaluate the exploitation reult, baed on which controller i modified. It can realize unupervied on-line learning without a ytem model [9-0]. Actor-Critic learning propoed by Barto et al i one of the mot important reinforcement learning method, which provide a working method of finding the optimal action and the expected value imultaneouly []. Actor-Critic learning i widely ued in artificial intelligence, robot planning and control, optimization and cheduling field. Baed on thi analyi, in thi paper a new adaptive PID controller baed on reinforcement ISSN: ISBN:

2 learning for WECS control i propoed. PID parameter are tuned on-line and adaptively by uing the Actor-Critic learning method, which can olve the deficiency of realizing effective control for complex and time-varying ytem by conventional PID controller. The next ection preent detail of the wind energy converion ytem in thi imulation. Section decribe the adaptive network algorithmic implementation. Then, the ection 4 introduce controller deign tep. After that, the ection 5 preent the imulation reult and finally, the ection 6 explain concluion. Wind Energy Converion Sytem. Wind Turbine Characteritic Before dicuing the application of wind turbine for the generation of electrical power, the particular aerodynamic characteritic of windmill need to be analyzed. Here the mot common type of wind turbine, that i, the horizontal-axi type, i conidered. The output mechanical power available from a wind turbine i [5]. P = 0.5ρC p( V ) ω A () Where ρ i the air denity, A i the area wept by the blade, andvω i the wind peed. C p i called the power coefficient, and i given a a nonlinear function of the parameter λ λ = ωr / Vω () Where R i the radiu of the turbine and ω i the rotational peed.uually C P i approximated a C = + γλ P αλ + βλ,where α, β and γ are contructive parameter for a given turbine. Fig.. Power coefficient C p veru turbine peed [5] Fig. how typical C p veru turbine peed curve, with Vω a a parameter. It can be een that C P max, the maximum value for C P, i a contant for a given turbine. That value, when replaced in (), give the maximum output power for a given wind peed. Thi correpond to the optimal relationhip λ opt between ω and V ω. The torque developed by the windmill i C p Tl = 0.5ρ ( Vω ) πr λ () Fig. how the torque/peed curve of a typical wind turbine, withv ω a a parameter. Note that maximum generated power ( C P max ) point do not coincide with maximum developed torque point. Superimpoed to thoe curve i the curve of C P max. It can be een that the maximum C p (and thu the maximum generated power), and the maximum torque are not obtained at the ame peed. Optimal performance i achieved when the turbine operate at the C P max condition. Thi will be the control obective in the preent paper.. Induction Generator and Slip Power Recovery A wind technology progree, an increaing number of variable peed WECS cheme are propoed. An intereting configuration among them i the one that ue grid-connected double-output induction generator (DOIG) with lip energy recovery in rotor, hown in Fig. [8]. ISSN: ISBN:

3 RECTIFIER v a n ia GENERATOR i a r i a 5 v b v c 4 6 t i a FILTER TRANSFORMER Fig.. Torque/peed curve (olid) of a typical wind turbine. The curve of i alo plotted (dotted) [5]. C P max Slip power i inected to the AC line uing a combination of rectifier and inverter known a a tatic Kramer drive [5]. Change on the firing angle of the inverter can control the operation point of the generator, in order to develop a reitant torque that place the turbine in it optimum (maximum generation) point. Normally commutated inverter in DOIG demand ome reactive power. In addition, it recover active lip power to the upply. Conequently, the aborbed reactive power by whole ytem raie leading to a lower power factor of the drive. Alo, a rather large low-order harmonic i inected to the upply. The power factor of a converter can be improved uing a forced commutation method. The amplitude of the harmonic can alo be reduced [8]. The pule width modulation (PWM) technique i one of the mot effective method in achieving the above goal. Thi method of improving the power factor eliminate the low-order harmonic. However, the amplitude of the high-order harmonic i increaed, which can be eaily filtered. To obtain a convenient performance, a current ource type ix valve converter from inuoidal pule width modulation (SPWM) technique control with three-mode witching ignal i ued [8]. In the SPWM technique, by changing the index modulation (m), the pule width and the mean value of the inverter voltage are varied, thu the torque generated by DOIG i controlled. The torque developed by the generator/kramer drive combination i [4] INVERTER Figure. Baic Power Circuit of a DOIG V Req Tg = (4) Ω[( R + Req) + ( ω Ll + ωllr ) ] where Req = f (, m) ω = ( ) Ω and (5) R : Stator reitance; L l : Stator diperion inductance; L lr :Rotor diperion inductance; ω : Synchronou pulation; Ω : Synchronou machine rotational peed; m: index modulation (All value referred to the rotor ide).. Turbine/Generator Model The dominant dynamic of the whole ytem (turbine plu generator) are thoe related to the total moment of inertia. Thu ignoring torion in the haft, generator electric dynamic, and other higher order effect, the approximate ytem dynamic model i Jω = Tl ( ω, Vω ) Tg ( ω, m) (6) where J i the total moment of inertia. Regarding () and (4), ytem model become C ω = (0.5ρ( P )( V ) ω πr Tg ( ω, m) J λ (7) Where R eq depend nonlinearly on the index modulation according to (5). CP, λ, and V ω alo depend on ω in a nonlinear way (). Moreover, it i well known that certain generator parameter, uch a wound reitance, are trongly dependent on factor ISSN: ISBN:

4 uch a temperature and aging. Thu a nonlinear adaptive control trategy eem very attractive. It obective i to place the turbine in it maximum generation point, in depite of wind gut and generator parameter change. Thu the propoed control trategy, which conit of changing m to produce a generator torque ettle the turbine on the ω opt, T l(op point [5]. The general form of (7) i ω = h( ω, m), where h i a nonlinear function accounting for the turbine and generator characteritic. Adaptive PID Controller Baed on Reinforcement Learning. Controller architecture The tructure of an adaptive PID controller baed on Actor-Critic learning i illutrated in Fig. 4. It i baed on the deign idea of the incremental PID controller decribed by Eq. (8). u( = u( t ) + Δu( = u( t ) + K( x( = u( t ) + k ( x ( + k x ( + k x ( = (8) I P D u( t ) + k I ( e( + k PΔe( + k DΔ e( Where T x ( = [ x( t ), x (, x ( ] = [ e(, Δe(, Δ e ( ] ; e( = yd ( y(, Δe( = e( e ( t ) and Δ e( = e( e( t ) + e( t ) repreent the ytem output error, the firt-order difference of error and the econd-order difference of error repectively; K( = [ k I (, k P (, k D (] i a vector of PID parameter. r( e( x( r( K ( δ TD ( u( t ) Δu( Fig. 4 Self-adaptive PID controller baed on reinforcement learning V ( Z K( u( y( In Fig. 4, y( and y d ( are the deired and the actual ytem output repectively. The error e( i converted into a ytem tate vector x( by a tate converter, which i needed by the Actor-Critic learning part. There are three eential component of an Actor- Critic learning architecture, including an Actor, a Critic and a tochatic action modifier (SAM). The Actor i ued to etimate a policy function and realize the mapping from the current ytem tate vector to the recommended PID parameter K ( = [ k I (, k P (, k D (] that will not participate in the deign of the PID controller directly. The SAM i ued to generate tochatically the actual PID parameter K( according to the recommended PID parameter K ( uggeted by the Actor and the etimated ignalv ( from the Critic. The Critic receive a ytem tate vector and an external reinforcement ignal (i.e., immediate reward) r( from the environment and produce a TD error (i.e., internal reinforcement ignal) δ TD ( and an etimated value functionv (. δ TD ( i provided for the Actor and the Critic directly and i viewed a an important bai for updating parameter of the Actor and the Critic. V ( i end to the SAM and i ued to modify the output of the Actor. The effect of the ytem error and the change rate of error on control performance mut be conidered imultaneouly during the deign of the external reinforcement ignal r(. Therefore, r( i defined a r( = α re ( + βrec ( (9) where α and β are weighted coefficient, 0 e( ε r e ( = 0. 5 otherwie 0 e( e( t ) r ec ( = 0. 5 otherwie and ε i a tolerant error band.. Actor-Critic learning baed on RBF network The RBF network i a kind of multi-layer feedforward neural network. It ha the characteritic of a imple tructure, trong global approximation ability and a quick and eay training algorithm []. On the other hand, the input of the Actor and the Critic are both the ame tate vector derived from the environment and their mall difference i the difference in their output. Therefore, there i only one ISSN: ISBN:

5 RBF network, a hown in Fig. 5. It i ued to implement the policy function learning of the Actor and the value function learning of the Critic imultaneouly. That i, the Actor and the Critic can hare the input and the hidden layer of the RBF network. Thi working manner can decreae the demand for torage pace and avoid the repeated computation for the output of the hidden unit in order to improve the learning efficiency. The definite meaning of each layer i decribed a follow. e( Δe( Δ e( Φ ( Φ i ( Φ h ( w m v K P ( K I ( V ( K D ( Fig. 5 Actor-Critic learning baed on RBF network Layer : input layer. Each unit in thi layer denote a ytem tate variable x i where i i an input variable index. Input vector x( R i tranmitted to the next layer directly. Layer : hidden layer. The kernel function of the hidden unit of RBF network i adopted a a Gauian function. The output of the th hidden unit i x( μ ( Φ ( = exp( ), =,,..., h σ ( T [ μ, μ, μ ] Critic Actot (0) where μ = and σ are the center vector and the width calar of the th hidden unit repectively, h the number of hidden unit. Layer : output layer. The layer i made up of an Actor part and a Critic part. The m th output of the Actor part, K m ( and the value function of the Critic part, V ( are calculated a K ( = m = h = h w m ( Φ (, m =,, () V ( = v ( Φ ( () where wm denote the weight between the th hidden unit and the m th Actor unit, and v denote the weight between the th hidden unit and the ingle Critic unit. In order to olve the dilemma of exploration and exploitation, the output of the Actor part doe not pa to the PID controller directly. A Gauian noie term ηk i added to the recommended PID parameter K ( coming from the Actor [9], conequently the actual PID parameter K( are modified a Eq. (). The magnitude of the Gauian noie depend on V (. If V ( i large, ηk i mall, and vice vera. K( = K ( T ) +η k (0, σv ( ) () Whereσ V ( = + exp(v ( ) The feature of Actor-Critic learning i that the Actor learn the policy function and the Critic learn the value function uing the TD method imultaneouly []. The TD error δ TD ( i calculated by the temporal difference of the value function between ucceive tate in the tate tranition. δ TD ( = r( + γv ( t + ) V ( (4) where r( i the external reinforcement reward ignal, 0 < γ < denote the dicount factor that i ued to determine the proportion of the delay to the future reward. The TD error indicate, in fact, the goodne of the actual action. Therefore, the performance index function of ytem learning can be defined a follow. E( = δtd ( (5) Baed on the TD error performance index, the weight of Actor and Critic are updated according to the following equation through a gradient decent method and a chain rule. K m ( K m ( wm ( t + ) = wm ( + α AδTD ( Φ ( 6) σ V ( v ( t + ) = v ( + α CδTD ( Φ ( (7) where α A and α C are learning rate of Actor and Critic repectively. Becaue the Actor and the Critic hare the input and the hidden layer of RBF network, the center and the width of hidden unit need to be updated only once according to the following rule. xi ( μi ( μi ( t + ) = μi ( + ημδtd( v ( Φ ( (8) σ ( ISSN: ISBN:

6 x ( μ ( σ ( t + ) = σ ( + ησδtd( v ( Φ ( (9) σ ( where μ μ and μ σ are learning rate of center and width repectively. 4 Controller Deign Step The overall block diagram of controller i illutrated in fig. 6. The whole deign tep of the propoed adaptive PID controller can be decribed a follow. Step. Initializing parameter of Actor-Critic learning controller, including w m (0), v (0), μ i (0), σ (0), μ μ, μ σ, α C, α A, γ, ε,α and β. Step. Detecting the actual ytem output y(, calculating the ytem error e(, contituting ytem tate variable e (, Δe(t ) and Δ e(. Step. Receiving an immediate reward r( from Eq.(9). Step4. Calculating the Actor output K ( and the Critic value function V( from Eq. () and Eq.() at time t repectively. Step5. Calculating the actual PID parameter K( from Eq. () and conequently calculating the control output of PID controller u( from Eq. (8). Step 6. Applying u( to the controlled plant and oberving the ytem output y( t +) and the immediate reward r( t +) at the next ampling time. Step 7. Calculating the Actor output K ( t +) and the Critic value function V( t +) from Eq. () and Eq. () at time repectively. Step 8. Calculating the TD error δtd( from Eq. (8). Step9. Updating the weight of the Actor and the Critic from Eq. (6) and Eq. (7) repectively. Step0. Updating the center and the width of RBF kernel function according to Eq. (8) and Eq. (9) repectively. Step. Judging whether the control proce i finihed or not. If not, then t ( t +) and turn to Step. Fig. 6 Overall Controller block diagram 5 Simulation Reult Fig. 4 depict the block diagram of the adaptive PID Controller Baed on Reinforcement Learning for WECS control, while the dynamic of WECS i decribed by Eq. (7). For thi cae tudy, the deired ignal y d ( i optimal rotor peedω, actual output y( i rotor peedω and control opt ignal u( i index modulation (m). The optimum haft rotational peed ω opt i obtained, for each wind peed V ω, and ued a a reference for the cloed loop. Note that wind peed act alo a a perturbation on the turbine model. We applied the propoed adaptive PID controller and the conventional PID controller to track the optimal rotor peed ignal. Sampling period T = during the ISSN: ISBN:

7 imulation. PID parameter of the conventional PID controller are et off-line a kp = 5, ki = 5 and k D = 0 through the ue of the Ziegler-Nichol tuning rule. The correponding parameter for the adaptive PID controller are et a follow, α = 0.67, β = 0. 47, ε = 0.04, γ = 0. 9, α A = 0. 07, α C = 0.04, η μ = 0. 0 andη σ = The detailed imulation reult are hown in Fig. 7. The Simulation reult indicate that the propoed adaptive PID controller exhibit perfect control performance and adapt to the change of parameter of the WECS. Therefore, it ha the characteritic of being trongly robut and adaptable. Rotor peed(rad/ec) Rotor peed(rad/ec) Traditional PID Controller Time(ec) Propoed PID Controller 00 Refrence Etimated Refrence. Etimated Time(ec) Fig. 7. Simulation Reult [5] Mayoky, M. A., Cancelo, G. I. E., 999. Direct adaptive control of wind energy converion ytem uing gauian network. IEEE Tranaction on neural network, 0(4): [6] Kalantar, M., Sedighizadeh, M., 004. Adaptive Self Tuning Control of Wind Energy Converion Sytem Uing Morlet Mother Wavelet Bai Function Network. th Mediterranean IEEE Conference on Control and Automation MED 04, Kuadai, Turkey [7] Sedighizadeh, M., Kalantar, M., 004. Adaptive PID Control of Wind Energy Converion Sytem Uing RASP Mother Wavelet Bai Function Network. IEEE TENCON 004, Chiang Mai, Thailand. [8] Sedighizadeh, M., et al, 005. Nonlinear Model Identification and Control of Wind Turbine Uing Wavenet. Proceeding of the 005 IEEE Conference on Control Application Toronto, Canada, PP [9] WANG Xue-ong, CHENG Yu-hu, SUN Wei. A Propoal of Adaptive PID Controller Baed on Reinforcement LearningJ China Univ Mining & Technol 007, 7(): [0] Wang X S, Cheng Y H, Sun W. Q learning baed on elforganizing fuzzy radial bai function network. Lecture Note incomputer Science, 006, 97: [] Barto A G, Sutton R S, Anderon C W. Neuronlike adaptive element that can olve difficult learning control problem. IEEETranaction on Sytem, Man and Cybernetic, 98, (5): Concluion Simulation reult indicate that the propoed adaptive PID controller can realize table tracking control for WECS. It i trongly robut for ytem diturbance, which i better than that of a conventional PID controller. 7 REFERENCES [] Kanello, F.D., Hatziargyriou, N.D., 00. A new control cheme for variable peed wind turbine uing neural network. IEEE Power Engineering Society Winter Meeting, : [] You-tong, F., Cheng-zhi, F., 007. Single neuron network PI control of high reliability linear induction motor for Maglev. Journal of Zheiang Univerity SCIENCE A, 007, 8():408-4 [] Zhao-da, Y., Chong-guang, Z., Shi-chuan, S., Zhen-tao, L., Xizhen, W., 00. Application of neural network in the tudy of combution rate of natural ga/dieel dual fuel engine. Journal of Zheiang Univerity SCIENCE A, 00, 4():70-74 [4] Haykin, S., 994. Neural Network, A Comprehenive Foundation. New York: Macmillan, 994. ISSN: ISBN: