DUring the past decades, the process control techniques

Transcription

1 Parameter Optimization of PID Controllers by Reinforcement Learning X. Y. Shang, T. Y. Ji, Member, IEEE, M. S. Li, Member, IEEE, P. Z. Wu and Q. H. Wu, Fellow, IEEE Abstract This paper focuses on implementing a reinforcement learning algorithm for solving parameter optimization problems of Proportional Integral Derivative (PID) controllers. Function Optimization by Reinforcement Learning () remarkably outperforms a number of population-based intelligent algorithms when executed on benchmark functions in high-dimension circumstances. Therefore, this paper aims at examining the performance of when optimizing parameters of PID controllers in a low-dimension space. According to the experiment studies in this paper, is able to optimize the PID parameters with advantage over and in terms of convergence speed. Index Terms Parameter optimization, PID Controller, Reinforcement Learning. I. INTRODUCTION DUring the past decades, the process control techniques in industry have made great advancements. Numerous control methods, for instance, adaptive control, neural control, fuzzy control [1], etc., have been studied by researchers all over the world. PID controllers, a common feedback loop component in industrial control applications [2], have been widely used due to its simple structure and robust performance under a wide range of operating conditions. Thus, PID parameter optimization has long attracted much attention. Approaches to optimizing parameters of the PID controller can be divided into two main categories: conventional PID tuning methods and intelligent PID tuning methods [3]. Conventional approaches include Ziegler-Nichols [4] and the optimal PID parameter tuning algorithm based on integral square time error criterion (ISTE) [5]. The tuning process of traditional approaches is complex and it is hard to avoid oscillation and large overshoot, hence achieving the optimal parameters is quite difficult. Therefore, research focuses on various intelligent PID tuning methods [6] [7]. Genetic algorithms (s) [8], particle swarm optimization () [9] [1], fuzzy reasoning algorithm and neural networks can effectively overcome the drawbacks of previous techniques and enhance the performance of PID controllers. However, these existing intelligent algorithms also have a number of shortcomings. X. Y. Shang, T. Y. Ji and M. S. Li are with the School of Electric Power Engineering, South China University of Technology (SCUT), Guangzhou, 51641, P. R. China. P. Z. Wu is with the Shenzhen Institute of Advanced Technology (SIAT), Chinese Academy of Sciences, Institute of Biomedical and Health Engineering, Paul C. Lauterbur Research Center for Biomedical Imaging, Shenzhen, 51855, P. R. China. Q. H. Wu is with the Department of Electrical Engineering and Electronics, The University of Liverpool, Liverpool, L69 3GJ, U.K. and Distinguished Professor at SCUT. Corresponding author: Professor Q. H. Wu, Tel: ; Fax: ; q.h.wu@liv.ac.uk. The project is supported by Guangdong Innovative Research Team Program (No.211N ). For instance, has to deal with tedious coding process; and most of these algorithms are population-based search, thus they suffer from long computation time as well as slow convergence rate. Furthermore, there is no systemic method to choose parameters of fuzzy reasoning algorithm and determine the number of hidden layers, the amount of neurons and initial weights for neural networks. In the area of artificial intelligence, Reinforcement Learning (RL) is one of the typical learning patterns, which stems from the idea of trial-and-error learning, and focuses on goaldirected learning from the interaction between learners and the environment. After a learner senses the state of the environment, it will take certain action so as to achieve specified goals. In the meantime, to evaluate the action that has been taken in the situation, the learner receives a immediate reward from the environment and the action itself also in turn influences the environment, resulting in a new state. In order to solve the optimization problems concerned with high-dimensional complex functions, a novel algorithm, Function Optimization by Reinforcement Learning (), was first proposed in [11] and tested with a large set of standard benchmark functions which are given in [12]. This paper aims at exploring the implementation of on optimization problems of PID controllers, so as to find optimal parameters and enhance the performance of this classical control technique. In this paper, the parameter optimization of PID controllers using will be introduced in detail. The organization of the paper is as follows. Section II explains the close-loop control system, the PID controller and the objective function of the problem raised above. The procedure of is given in section III. In section IV, to verify the performance of, simulation studies are carried out and the results are compared with those of and. Finally, conclusions are drawn in section V. II. PID CONTROLLER r(t) e(t) C(s) u(t) G(s) y(t) _ Fig. 1. Block diagram of the close-loop control system A close-loop control system, with a PID controller and a controlled plant, is illustrated in Fig. 1. The input and the output of the whole control system are r(t) and y(t) /13/$ IEEE 77

2 respectively. The input signal serves as the reference for the output. The difference between the input and the output is defined as the tracking error e(t). This error signal is then sent to the PID controller C(s), whose transfer function is expressed in (1). Subsequently, the output of PID controller u(t) is sent to the plant G(s) and finally the plant generates the output y(t). In the control process, given the input signal (or the reference) and the plant, the PID controller will deal with the tracking error, so that the output of the system can approach the reference. The Laplace transfer function of the PID controller can be formulated as follows: 1 C(s) =K p K i s K ds (1) where K p, K i and K d are proportional, integral and derivation parameters of the PID controller respectively. These three parameters should be optimized and tuned to meet certain performance criteria. K p proportionally reflects the signal error; thus, once the error occurs, the proportional controller will work immediately to reduce it. The integral controller, K i, mainly reduces the steady component of the error, improving the smoothness of the system. The derivation controller, K d, can track the trend of the error and effectively reduce the dramatic decrease or increase of the error. Involving absolute error, control output and rise time, the objective function of the parameter optimization problem is formulated as: ( J = w1 e(t) w 2 u 2 (t) ) dt w 3 t u (2) where t u is rise time (the time period in which y(t) rises from 1% to 9% of the steady value) of the step response of the plant; w 1, w 2 and w 3 are weighting factors, respectively. To avoid a large overshoot, a penalty function is added to the performance criterion. If ey(t) < (set: ey(t) =y(t) y(t 1)), the objective function of optimization can be rewritten as: ( J = w1 e(t) w 2 u 2 (t)w 4 ey(t) ) dt w 3 t u (3) where w 4 is a weighting factor and w 4 w 1 in the parameter optimization process, as the effect of the overshoot ey(t) is much greater than the tracking error e(t) in terms of dynamic performance. III. PARAMETERS OPTIMIZATION OF PID CONTROLLER BY A. Dimensional states employed in this paper was first proposed in [11], which effectively solves the optimization problems of highdimensional functions. And in comparison with six other algorithms including,, evolutionary programming (EP), etc., shows its superior performance in terms of accuracy and computation efficiency. adopts a dimensional search strategy and a mechanism of dividing each dimension into cells so that a search action is taken via moving a state from one cell to another to avoid the problem of unmanageable memory of an infinite number of states. If it is an N-dimensional search problem, the state can be defined as X =[x 1,,x i,,x N ]. Therefore, for simplicity, when the jth cell of the ith dimension is being searched, the dimensional state and its state value are denoted as X(x i,j ) and V (X(x i,j )), respectively. To be more precise, V (X(x i,j )) represents the value function of the jth cell of the ith dimension intersecting the state vector, as long as x i,j is located in this cell. Then a possible subsequent path should be selected before a search action is taken, to estimate the potential of finding a better solution if searches down on this path. A path value at state x i,j on the lth (l =1, 2) path, denoted by L l (x i,j ), can be calculated by the value of k continuous states. For example, the path value for the left path is calculated using: L l (x i,j )=(1 λ 1 ) k 1 m=1 λ m 1 1 v m λ k 1 1 v k (4) where vm denotes the mth element of the vector reordered in descending k state values and k is a pre-set integer; λ 1 represents a weight introduced for the value of the state on the lth path, λ 1 subject to (1 λ k 2 1 ) λ k and k 1 m=1 λm 1 1 λ k 1 1 = 1, B. Immediate reward and state values The reasonable immediate reward of a possible search action can dramatically facilitate the search pattern, for it is beneficial to the determination of a best search direction. In order to minimize the objective function J which is given above, a proper reward can be obtained via the following rule: { 1 if J(X(xi,j )) J(X r(x(x i,j )) = best ) (5) otherwise where X best defines the last best solution and it will be updated during the search of a certain dimension. Subsequently by revising previous value with both discounted current reward and information on searching paths, the state value is calculated as: V (X(x i,j )) V (X(x i,j )) α[r(x(x i,j )) (1 λ 2 )L max (x i,j ) (6) λ 2 L min (x i,j ) V (X(x i,j ))] where L max (x i,j ) and L min (x i,j ) are two estimates of the right and left directions of the selected dimension, respectively, and L max (x i,j ) >L min (x i,j ); α is used for an incremental update of the value of a state and α<1. To make L max have a greater influence on the state value than that of L min, the parameter λ 2 should be given such that (1 λ 2 ) >λ 2. C. Taking a search action To take an action is to choose whether to follow the left path or the right path. Such a choice is based on the values obtained from (4). A probability is introduced and calculated as: e L l(x i,j)/τ P (L l (x i,j )) = 2 (7) s=1 els(xi,j)/τ where τ is temperature. A large value of τ indicates selecting two paths with the same probability. If the value of τ is /13/$ IEEE 78

3 small, there is a higher probability of choosing one path over the other. Therefore, this parameter is able to balance the exploration and exploitation. As for the step length following selected direction, it should not exceed the pre-set integer k. Hence, the step length parameter η is defined via η = ζk, where ζ is a random number, rounds the elements to the nearest integers towards minus infinity, and ζ (, 1]. D. Perturbation operation For the purpose of increasing the diversity of solutions and avoiding solutions being trapped in local optima, different perturbations are introduced to all dimensional states of X simultaneously, according to the following rule: X X Δ, Δ=[Δ 1,, Δ i,, Δ N ] (8) where Δ i =sign(κ)θ(x max,i x min,i ), and θ is a random variable which is selected from the range [,K/D]. The sign function is used to choose the moving direction of x i. Suppose x i is located in cell c i,j, then the input to the sign function is the subtraction of the two adjacent cell values of c i,j, that is, κ =(V (c i,j1 ) V (c i,j 1 )).Ifκ, sign(κ) =1; otherwise, sign(κ) = 1. From previous description, it can be seen that x i moves towards the direction in which the adjacent cell has a larger cell value. Upon completing the perturbation operation, X best is updated according to the following equation: { X if J(X) <J(Xbest ) X best (9) X best otherwise In the optimization process, a covariance matrix C is adopted to approximate inverse Hessian matrix. C guides the search according to the contour lines of the objective function and locates exact position of the optimum. Following (8), based on the mean of previous X best (denoted as m),anew step is taken by a perturbation that is generated via N(,C), i.e. the multivariate normal distribution with zero mean and covariance matrix C. The covariance matrix is updated as follows: T C (1 c cov )C c cov Y i Y i (1) where c cov is the learning rate of the covariance matrix, Y i =(X best m)/σ, and σ is the step size. It can be seen that covariance matrix C estimates the distribution of previous selected steps. The maximum likelihood of X best is added into the covariance matrix through the use of (1). According to this approach, the probability of generating the solutions around X best increases. When a series of previously selected favourable steps X best are used to update the covariance matrix, the process leads to an alignment of the search distribution N(,C) to the distribution of favourable steps. This distribution presents the information of correlation between variables in some sense. Given this distribution, sampling from C tends to produce successful steps. IV. SIMULATION STUDIES A. Simulation model In Fig. 2, a simulation model with a PID controller is established using Simulink, which is a platform for the modeling, simulating and analysing of dynamic systems. This close-loop control system aims at importing arguments (K p,k i,k d ) from the workspace and returning the results (e, u, y, t u ) back again for calculating objective functions after the simulation. A step signal is chosen as the reference; and the output of the plant can be observed through the scope. Based on the transfer function shown as (1), a PID controller is composed of three multipliers, one integral operator, one differential operator and one summator to deal with the error signal (e) by the input arguments (K p,k i,k d ). Then, PID controller s output (u) is sent to the plant and in turn the final outcome (y) is sent back to system s initial part to generate an error signal for the PID. Here, a second-order system is given and its transfer function is shown below: G(s) = 4 s 2 5s. (11) B. Parameter setting Parameters of and are set according to [2]. For, population size is set to 2 and inertia weight factor w is set according to (12). w = w max w max w min i (12) M The acceleration constants are c 1 = c 2 =2, and the number of iterations is G =1. As for, the population size is the same as that of, while the crossover rate and mute rate are.9 and.1, respectively. Parameters of are: k =4, λ 1 =.5, λ 2 =.25, α =1, τ =.2, D =1. In order to compare the performance of with and in optimizing the parameters of PID controllers, experimental studies are implemented on the same system. Specifically, if the system has no overshoot, the algorithms adopt the objective function (2), and their weights are w 1 =.999,w 2 =.1 and w 3 =2, respectively. If there exists an overshoot in the output, (3) is selected as the objective function and w 4 = 1. C. Results and analysis When simulations are performed using a step signal, the system with PID controllers optimized by three algorithms respectively gives step responses. The response curves and convergence performance of these three algorithms are illustrated in Fig. 3 and Fig. 4, respectively. TABLE I EXPERIMENT RESULTS OF THREE ALGORITHMS K p K i K d t u J best J best refers to the optimal objective function value obtained by certain algorithm As can be seen from Fig. 3, the PID controllers with optimal parameters obtained via the three algorithms respectively can effectively eliminate the fluctuation and the overshoot of the response curve before it levels off at the steady value /13/$ IEEE 79

4 e To Workspace u To Workspace1 Matrix Multiply y To Workspace2 Step _ Matrix Multiply1 Integrator Scope Matrix Multiply2 Derivative Add From Workspace From Workspace1 From Workspace2 Fig. 2. Simulation model of PID controller y(t).6 J best Time(s) Fig. 3. The step responses of system with optimal PID controllers Iteration Fig. 4. The convergence curves of the algorithms Meanwhile, the rise time can also be decreased, thus the response can soar to steady state once the step signal is given to the system. For a 3-dimension optimization problem,, and present equivalent performances on the output signal. In Fig. 3, the response curve of almost coincides with that of. turns out to rise faster but level off slightly slower when compared with the other two responses. As for the convergence performance, Fig. 4 shows the average outcomes of the three algorithms of 1 test runs. is able to find the best initial point but converges after the twentieth iteration. reaches the optimal value for the objective function in less than 5 iterations, however, its optimum is inferior to that of the other two algorithms. Although cannot begin with a satisfactory starting point, its objective function rapidly falls to the optimal value and converges around tenth iteration. Combined with the standard deviations of the three algorithms for each iteration point shown in Fig. 5, the following conclusions can be drawn: is the most steady one as its corresponding curve is lower than the others except for some initial points; while keeps the same fluctuation on each iteration point during 1 optimization courses. To be more specific, Table I lists all the optimization results of three algorithms, including optimal parameters for PID controllers, rise time and the optimal values of objective function /13/$ IEEE 8

5 Standard Deviation [11] Q.H. Wu and H.L. Liao. High-dimensional function optimisation by reinforcement learning. In Evolutionary Computation (CEC), 21 IEEE Congress on, pages 1 8, July. [12] X. Yao, Y. Liu, and G.M. Lin. Evolutionary programming made faster. Evolutionary Computation, IEEE Transactions on, 3(2):82 12, july Iteration Fig. 5. The standard deviations of the convergence curves V. CONCLUSION Recently, has proved its outstanding performance in high-dimensional function optimization problems. This paper mainly presents the implementation of on parameter optimization of PID controllers. Through adjusting the parameters of, this paper successfully optimizes the PID controller and the output of the control system is deemed to be satisfactory. Based on the simulation studies, the convergence rate of is slightly faster than that of and. Therefore, from our previous research and the study presented in this paper, it is obvious that is a promising optimization algorithm and deserves much more attention in the future. REFERENCES [1] Q.H. Xiao, D.Q. Zou, and P. Wei. Fuzzy adaptive pid control tank level. In Multimedia Communications (Mediacom), 21 International Conference on, pages , August. [2] C. Ou and W.X. Lin. Comparison between pso and ga for parameters optimization of pid controller. In Mechatronics and Automation, Proceedings of the 26 IEEE International Conference on, pages , June. [3] Y. Luo, J.T. Zhang, and X.X. Li. The optimization of pid controller parameters based on artificial fish swarm algorithm. In Automation and Logistics, 27 IEEE International Conference on, pages , August. [4] P. M. Meshram and R.G. Kanojiya. Tuning of pid controller using ziegler-nichols method for speed control of dc motor. In Advances in Engineering, Science and Management (ICAESM), 212 International Conference on, pages , March. [5] E.D. Bolat, K. Erkan, and S. Postalcioglu. Experimental autotuning pid control of temperature using microcontroller. In Computer as a Tool, 25. EUROCON 25.The International Conference on, volume 1, pages , November. [6] G. Reynoso-Meza, S. Garcia-Nieto, J. Sanchis, and F.X. Blasco. Controller tuning by means of multi-objective optimization algorithms: A global tuning framework. Control Systems Technology, IEEE Transactions on, 21(2): , 213. [7] E.B. Muhando, T. Senjyu, H. Kinjo, and T. Funabashi. Extending the modeling framework for wind generation systems: Rls-based paradigm for performance under high turbulence inflow. Energy Conversion, IEEE Transactions on, 24(1): , 29. [8] G.H. Lin and G.F. Liu. Tuning pid controller using adaptive genetic algorithms. In Computer Science and Education (ICCSE), 21 5th International Conference on, pages , August. [9] J. Kennedy and R.C. Eberhart. Particle swarm optimization. Proceedings of IEEE International Conference on Neural Networks, 4: , [1] M.M.R.A. Milani, T. Cavdar, and V.F. Aghjehkand. Particle swarm optimization x214; based determination of ziegler-nichols parameters for pid controller of brushless dc motors. In Innovations in Intelligent Systems and Applications (INISTA), 212 International Symposium on, pages 1 5, July /13/$ IEEE 81