Lake Level Prediction Using Artificial Neural Network with Adaptive Activation Function

Transcription

1 Lae Level Prediction Using Artificial eural etwor with Adaptive Activation Function Gülay TEZEL Selcu Unv. Engineering Fac. Computer Engineering Department Meral Büyüyıldız Selcu Unv. Engineering Fac. Civil Engineering Department Humar Kahramanlı Selcu Unv.Technology Fac. Computer Engineering Department Abstract This paper presents new neural networ structure with adaptive activation function (AAF) to determine lae level fluctuation. The proposed structure is based on the multi layer perceptron (MLP), the most common neural networ algorithm. The achievement of AAF was compared MLP using monthly observation data set consists of incoming and outgoing discharges evaporation, rainfall, in terms of common performance criteria such as MAE, MSE, RMSE and determination coefficient. In these structures, they are used adaptive activation functions with free parameters. As a result, the success of both algorithms was close to each other. However the stability of AAF was higher than MLP with regard to learning rate and the number of hidden neurons. Keywords MLP, adaptive activation function, lae level I. ITRODUCTIO The prediction of lae level is an important issue to plan, design and construct of lae shore structures and to manage of fresh water for the water supply purposes. However, there are many environmental factors affecting lae level such as incoming and outgoing discharges, direct and indirect runoffs from neighbor catchments, interactions between the lae and the low lying aquifers precipitations falling on the lae surface or lae watershed, groundwater exchange, evaporation from the lae surface, wind speed, humidity, and temperature. Sensitivity of lae levels to these factors might differ from one region to the other and the accurate measurement of them are often difficult and with a great amount of uncertainty [-3] It is necessary to develop models for simulation of the extreme or abnormal level variations in order to control future lae level changes. Level measurements or their future equally liely replicas obtained through a simulation model are a direct way of obtaining lae management decision variable. It is possible to setting up sophisticated models considering the hydrological and hydro-meteorological variables such as the rainfall, runoff, temperature and evaporation; however, it is economically preferable to use a model that simulates the level variations. Over the past few decades, the traditional predicting models, such as time series, regression analysis were commonly applied to predict hydrology and water resources system s parameters. Because of the complexity of these processes, it is hard to apply traditional least square regression and multiple regressions with high accuracy to deal with complex nonlinearity of the displacement-time series.[,,4,5] Recently, artificial intelligence (AI) techniques have been accepted as an appropriate tool for modeling complex nonlinear phenomena in modeling hydrologic processes, and have been applied to a range of different areas including rainfall-runoff, water quality, sedimentation and rainfall forecasting, leading to widening of their applications. Furthermore, eural networs (), one of the AI techniques have been successfully applied in a number of diverse fields including water resources in special. Furthermore, the common A algorithm in literature is Multi Layer Perceptron algorithm (MLP) applied to a range of different areas including, water level, forecasting, water quality, sedimentation and rainfall [,,4,6] In MLP, the weighted sum of each neuron inputs is computed and then it is applied to a non-linear function called activation functions [7, 8]. In general, the performance of MLP depends on the number of hidden layers, the number of hidden neurons, the learning algorithm and the type of activation function for each neuron [9].The commonly investigated activation functions in literature are sigmoid function, generalized sigmoid functions and the radial basis function, so on. These functions which all fixed and cannot be adjusted to adapt to different problems represent a relation between a single input, the weighted sum, and a single output, the neuron response. One common characteristic of these activation functions is that they are activation function is critical as the behavior performance of MLP depends on it[- ]. So far there have been limited studies with emphasis on setting a few free parameters in the activation function. In Liu[3], real variables, node offset( c) and slope of the sigmoid function(s) in sigmoid activation function were adjusted during learning process. Yu at al., established an adaptive activation function for MLP to solve -Parity and two spiral problems. Vecci at al.[4] and Solazzi and Uncini [8] studied with adaptive spline activation function neural networs. Xu and Zhang [8-], studied Adaptive Higher Feed-Forward eural etwors for financial analysis. In this paper, MLP and AAF model with adaptive activation function with free parameters are applied to forecast monthly lae level of Lae Beysehir using measured rainfall, ISB:

2 evaporation, inflow and outflow. The performances of models were compared using MAE, MSE, RMSE and R. II. DATA Lae Beysehir, a tectonic lae (with coordinates 37 o 45 orth, 3 o 36 East) is located in southwest Turey, within the Konya Closed Basin. The water level and the area of the lae change seasonally as well as from year to year, The observation data obtained from Beysehir Lae Station between the years 96 and 99 were used for experiments. The number of observation was 343. This data set has five parameters; water level, rainfall, evaporation, inflow and outflow. III. EURAL ETWORK WITH ADAPTIVE ACTIVATIO FUCTIO (AAF) The neural networ with adaptive activation function (AAF) considered here has three layers (an input layer, one hidden layer and output layer) lie as MLP. The net input of hidden and output layers is weighted sum of its inputs. It is used no activation function in the input neurons of input layer. Sigmoid activation function with fixed parameters is used in the output neurons of the output layer. But adaptive activation functions with free parameters are used as the activation function in the hidden nodes of the hidden layer. The structures in this study were implemented with MATLAB Release R9a software pacage. Whereas fixed activation function (Eq.) preferred in the hidden nodes of MLP, adaptive activation function defined in Eq.was used only in the hidden neurons of AAF model. In addition, the sigmoid function with fixed parameters (Eq.3) was selected as activation function in output layer of the AAF because the data was normalized in the range of [ ]. a ψ ( x) a Sin( b x) + () bx + e + x x e e ψ ( x) () + x x e + e ψ 3 ( x) + e x (3) Where a, a, b, b are real variables which will be tuned during training as weights between neurons. There are four free parameters (a,a,b and b) in the Equ..[9,-3] A. Learning Algorithm for AAF In this study, it was developed a learning algorithm which is not far from traditional bacpropagation algorithm. In the proposed structures, free parameters in adaptive activation functions are adjusted as weights between neurons with this learning algorithm based on the steepest descent rule. In bacpropagation algorithm, there are two phase: feedforward and error bacpropagation[5-4]. Firstly, all the weights and biases initialized to small real random values to the initial values in feedforward phase [5,6]. After initializing, it is presented training pair (input vector and corresponding desired responses) to the networ inputs. Each hidden unit sums its weighted signals (Eq.4) and then applies its activation function (ψ ) lie as Eq.5 to compute its output signal and applies fixed sigmoid activation function (Eq.3) in each output unit which sums its weighted signals to calculate the output signals of output in feeforward phase. The input of ith neuron in the th layer is defined as: [ w j,o j, (u ] + θ I (u) ) j (4) where j is neuron number in the layer (-), the value of output from ith neuron in the th layer for activation function; o ( u) ψ (I (u)) a Sin( b.i a (u)) + + e b.i (u) It is suggested using gradient descent to perform steepest descent in which the adjustment of weight is proportional the first derivative of the output function (Eq.6) in each neuron. Similarly, the adjustment of free parameters in each activation functions is proportional the first derivative of the output function in each neuron (Eq.7 -Eq. 9). The networ is training to minimize the error function by adjusting the weight and free parameters in the activation functions by using steepest descent rule. For an efficient learning algorithm, this method specifies how to reduce the mean squared error for all patterns through an adjustment of these free parameters simultaneously in bacpropagation phase. The mean squared error function in Eq. is sum of the squared error between the actual networ and the desired output for all input pattern [7-6]. r r w j, w j, + β (6) w r r θ θ + β θ j, (5) (7) r r a a + β (8) a b r r b + β b m E (d j (u) o j,l ) () j Where I (u) is the input of ith neuron in the th layer, w j, is the weight between jth neuron in the layer (-) and ith neuron in the layer, o (u) is the value of output from ith neuron in the th layer, θ is the threshold value of ith neuron in the th layer, β is learning rate, d j (u)nis the desired value of jth output neuron, m is otal number of neurons in the output layer, p is (9) ISB:

3 total number of neurons in the hidden layer, l is total number of networ layers, r is the iteration number, In this algorithm, the weights are updated after each training pattern is presented. An epoch is one cycle through the entire set of training vectors. At the end of the every epoch, free parameters of ψ (a, b, a, b) are adjusted as weights. After completing the training procedure of the neural networ, the weights of AAF are frozen and ready for use in the testing mode [5, 6]. IV. EXPERIMETAL STUDIES A. Performance Evaluation Criteria In this study, different performance criteria were used to compare the success of the models. The strength of the relationship between two variables is measured with R ; various types of information about the estimation capabilities of the model were provided by mean square error (MSE), root mean square error (RMSE) and mean absolute error (MAE); and the goodness-of-fit related to high level values, since MAE yields a more balanced perspective of the goodness-offit at moderate level, was measured by MSE and RMSE. Then, MSE, RMSE and coefficient of determination MAE are defined as: MAE RMSE i Yi observed Yi estimate i ( Yi observed Yi estimate ) () () MSE ( Yi observed Yi estimate ) (3) i in which is the number of data set, and Y i is the monthly lae level. B. Results Before developed model (AAF) and MLP were applied to the data set, the input and output values were normalized in the range of [, ] using Eq.4 in order to remove disparities related to the differing units of used to represent the parameters. X X min X norm (4) X X max min where x norm, x, x min, x max are the normalized value, observed value, minimum value, and maximum value, respectively. In this study, the monthly values for the variations of rainfall, evaporation, inflow and outflow discharges of Lae Beysehir between 96 and 99 were used to predict level changes of lae using MLP and AAF. Initially, the normalized data set, consists of 343 observation was randomly mixed to achieve robustness and effective models. Experimental trials for both models were evaluated in two steps. In the first step, the mixed data set was divided as training data set and test data set to obtain the best structures of MLP and AAF. Then the cross validation method was applied to the models obtained in the first step to prove robustness of these models. Because the aim of first step is to reach the best model, many trials were carried out to determine the most successful structures by changing numbers of hidden layer neurons, the values of learning rate and momentum coefficient. During the experiments, firstly, the number of hidden neurons was changed between 3-5with fixed learning rate and momentum coefficient. After the optimum hidden neurons were found, the values of learning rate and momentum coefficient were scanned within the range. and, at intervals of. to find the best structure of models. Fig. was depicting the alternation of MAE and RMSE values for test and training results of MLP and AAF according to the number of hidden neurons respectively. It was pointed out that MLP showed both highest error value and sharp fluctuation according to the number of hidden neurons. Also, Fig. showed that the alternation of error was less than MLP with regard to the learning rate for AAF. In addition, Fig. illustrates that minimum MSE, RMSE values for test and training data set were obtained with and hidden neurons for MLP and AAF, respectively. Moreover, the best values of learning rate and momentum coefficient were got as.4 and.9 for MLP and.7 and for AAF from Fig.. When R values in training scatter diagrams were compared, it was seen that training R value for MLP is bigger than AAF but test s R value for AAF is higher than MLP s value. Fig.3 and Fig.4 demonstrate that the performances of models were close to each other. Thus, the first step was finished and it was passed the next step using the best structures of MLP and AAF. Thus, the first step was finished and it was passed the next step using the best structures of MLP and AAF. In the second step of this study, the robustness of best structures was investigated using 5 fold cross-validation. After the experiments, the results of models obtained with cross validation were given in Table and Table for each fold and average of them According to the average values, MLP shows the best performance in the training phase whereas AAF is the best one in the test phase. Furthermore, both models illustrated the worst performance for test data set in the 3. Fold. As a result, it was seemed that achievement of with adaptive activation function was close to the success of with fixed activation function. ISB:

4 ,35,3 MLP,8 MLP_Training,5,,5,, The number of Hidden euron MAE_Train RMSE_Train MAE_Test RMSE_Test,6,4, y,98x R²,985 -,4 -,,,4,6,8 -, -,4,3,5,,5,, The number of Hidden euron AAF MAE_Train MAE_Test RMSE_Train RMSE_Test Figure. Comparision of MAE and RMSE values for training and test according to the number of hidden neurons for MLP and AAF,5,4,3,, -, -,3 -,4 MLP_Test y,97x R²,99 -,4 -, -,,,4,6 Figure 3. Scatter Diagrams for MLP for training and test results,6,4,,,8,6,4,,,,3,4,5,6,7,8,9 Learning Rate MAE_Train_MLP MAE_Test_MLP MAE_Train_AAF MAE_Test_AAF,8,6,4, AAF_Training y,98x R²,984 -,4 -,,,4,6,8 -, Figure. Comparision of MAE and RMSE values for training and test according to the learning rate for MLP and AAF -,4,5,4,3,, -, -,3 -,4 AAF_test y,964x R²,994 -,4 -, -,,,4,6 Fig.4 Scatter Diagrams for AAF for training and test results ISB:

5 Table. Training and Test performance criteria for each fold of MLP Fold umber MLP Mean.Fold.Fold 3.Fold 4.Fold 5.Fold with a high performance. However, the experiments demonstrated that the dependency of networ to the learning rate and the number of hidden neuron is less for AAF than MLP. This result showed the stability of AAF. Training MAE,345,349,799,84,44,8844 MSE,69,557,577,95,486,9 RMSE,43,365,46,3457,46,976 REFERECES Test R,94433,97687,97965,95966,983, MAE,569,55,75,35,763,3775 MSE,496,74,,99,4,57 RMSE,64,797,34936,779,,376 R,979757,984437,956476,98798,996588,9889 Table. Training and Test performance criteria for each fold of AAF AAF Training Test Fold umber.fold.fold 3.Fold 4.Fold 5.Fold Mean MAE,6384,673,9884,8495,5775,643 MSE,46,44,3,476,57,49 RMSE,493,599,495,8,54,77 R,984545,984,994,9834,9859,98486 MAE,338,636,556,7,97,53 MSE,588,36,68,34,8,75 RMSE,448,3674,3565,743,677,666 R,9777,97733,9565,98788,999,9747 C. Discussion and Conclusion In this study, neural networ with adaptive activation function is developed to estimate lae level of Beysehir Lae as well as MLP with fixed neural networ. Thus, the affect of adaptive activation function in hidden neurons was investigated. Fluctuating behavior of lae level is important for planning and designing lae coastal structures and water resources. There are many complex and interactive factors affecting the lae level. Therefore, modeling the lae level is a difficult problem for mathematical modeling techniques. In this study, monthly observation data set consists of incoming and outgoing discharges evaporation, rainfall, was used as input parameters to predict lae level. The performance of models were measured and evaluated with MAE, MSE, RMSE and determination coefficient (R ). The experiments showed that with adaptive activation function reached approximately close achievement with MLP. Generally both MLP and AAF could predict lae level [] M. Çimen, O. Kis Comparison of two different data-driven techniques in modeling lae level fluctuations in Turey, Journal of Hydrology, vol. 378, 9, pp [] M. Talebizadeh, A. Moridnejad, Uncertainty analysis for the forecast of lae level fluctuations using ensembles of A and AFIS models, Expert Systems with Applications, vol. 38,,pp [3] O Kis J. Shirib, B. ioofar, Forecasting daily lae levels using artificial intelligence approaches, Computers & Geosciences, vol. 4,, pp [4] G. Wenxian, W. Hongxiang, X. Jianxin, Z. Yunfeng, RBF eural etwor model based on Improved PSO for Predicting River Runoff,, International Conference on Intelligent Computation Technology and Automation. [5] L.V. Chinh, K. Hiramatsu, M. Harada, M. Mor Estimation of water levels in a main drainage canal in a flat low-lying agricultural area using artificial neural networ models, Agricultural Water Management, vol.96, 9, pp [6] A. Yarar, M. Onucyıldız,.K Copty, Modelling level change in laes using neuro-fuzzy and artificial neural etwors, Journal of Hydrology, vol. 365, 9,pp [7] C.C. Yu, Y.C. Tang, B.D. Liu, An adaptive Activation Function for Multilayer Feedforward eural etwors,, Proceeding of IEEE TECO. [8] M. Solazz A. Uncin Artificial eural etwors with Adaptive Multidimensional Spline Functions, eural networs, vol.7,, pp [9] S.,Xu, M. Zhang, Justification of A euron-adaptive Activation Function, proceeding of IEEE-IS-ES International Joint Conference on eural etwors, IJC, vol. 3, 4-4 July, pp [] S. Xu, M. Zhang, Adaptive Higher-Order Feedforward eural etwors, IJC '99 IEEE International Joint Conference on eural etwors, -6 July 999, vol., pp [] S. Xu, M. Zhang, A ovel Adaptive Activation Function, Proceedings IJC International Conference on eural etwors, 5-9 July,vol.4, pp , [] S. Xu, M. Zhang, Data Mining- An Adaptive eural etwor Model for Financial Analysis, ICITA 5, 5, IEEE. [3] T.I. Liu, On-line Sensing of Drill Wear Using eural etwor Approach, IEEE International Conference on eural etwors, vol., 993, pp [4] L. Vecc F.Piazza, Uncini A.:Learning and approximation capabilities of adaptive spline activation function neural etwors, eural etwors, Vol., 998, pp [5] L. Fauset, Fundamentals of neural networs: Architectures, Algorithms and Applications, Prentice Hall Inc. A simon&schuster Company, 994. [6] S. Hayin, eural etwors: A Comprehensive Foundation, ew Yor: Macmillan, 994. ISB: