A Quantitative Comparison of Different MLP Activation Functions in Classification
|
|
- Baldric Andrews
- 6 years ago
- Views:
Transcription
1 A Quantitative Comparison of Different MLP Activation Functions in Classification Emad A. M. Andrews Shenouda Department of Computer Science, University of Toronto, Toronto, ON, Canada Abstract. Multilayer perceptrons (MLP) has been proven to be very successful in many applications including classification. The activation function is the source of the MLP power. Careful selection of the activation function has a huge impact on the network performance. This paper gives a quantitative comparison of the four most commonly used activation functions, including the Gaussian RBF network, over ten real different datasets. Results show that the sigmoid activation function substantially outperforms the other activation functions. Also, using only the needed number of hidden units in the MLP, we improved its conversion time to be competitive with the RBF networks most of the time. 1 Introduction Introducing the back-propagation algorithm (BP) is a landmark in Neural Networks (NN) [5]. The earliest description of (BP) was presented by Werbos in his PhD dissertation in 1974 [1], but it did not gain much publicity until it has been independently rediscovered by Le Cun, Parker, Hinton and Williams [7]. MLP is perhaps the most famous implementation for the (BP). It is very successfully used in many applications in various domains such as prediction, function approximation and classification. For classification, MLP is considered a super-regression machine that can draw complicated decision borders between nonlinearly separable patterns [5]. The nonlinearity power of MLP is due to the fact that all its neurons use a nonlinear activation function to calculate their outputs. MLP with one hidden layer can form single convex decision regions, while adding more hidden layers can form arbitrary disjoint decision regions. In [6], Huang, et al showed that single hidden layer feedforward neural networks (SLFN s) with some unbounded activation function can also form disjoint decision regions with arbitrary shapes. If a linear activation function is used, the whole MLP will become a simple linear regression machine. Not only determining the decision borders, but the value of the activation function also determines the total signal strength the neuron will produce and receive. In turn, that will affect almost the all aspects of solving the problem in hand like: the quality of the network initial state, speed of conversion and the efficiency of the synaptic weights updates. As a result, a careful selection of the activation function has a huge impact on the MLP classification performance. In theory, the (BP) is universal in this matter, such J. Wang et al. (Eds.): ISNN 006, LNCS 3971, pp , 006. Springer-Verlag Berlin Heidelberg 006
2 850 E.A.M.A. Shenouda that any activation function can be used as long as it has a first derivative. Activation functions can be categorized into three basic families [15]: linear, e.g. Step-like functions, logistic, e.g. Sigmoidal and Radial Basis Functions (RBF), e.g. Gaussian. In spite of the importance of the activation function as an integral part of any feedforward network, it has not been well investigated in the NN literature. Although Michie et al showed that both MLP and RBF are computationally equivalent; they could not interpret their discrepancy in classification performance because MLP outperformed RBF but needed order of magnitude more training time []-[15]. In this paper, we compare the performance of the three families of functions with respect to classification over 10 different real datasets using both batch and online learning. With careful MLP pruning, our results showed that MLP with Sigmoid activation function is superior for classification with less or competitive tainting time to RBF. 1.1 Related Work The mathematical and theoretical foundation of various MLP activation functions, including RBF, can be found by Duch and Jankowski in [15]. In [14], the authors provided a comprehensive survey of different activations function. Others compared the performance of different activation functions in MLP. In [3], the authors provided a visual comparison for the speed of conversion of different linear and sigmoid functions. However, the comparison did not provide much information since one randomly generated easy dataset was used. Comparing the performance of MLP and RBF networks attracted the attention of more researchers. While the RBF outperformed the MLP for voice recognition in [10], other authors showed that MLP is superior to RBF in fault tolerance and resource allocation applications [8]-[9]. Various attempts to combine both MLP and RBF networks in one hybrid less complex and better performing networks exist. In [1], the authors showed a new MLP-RBF architecture that outperformed both of the individual networks in an equalizer signal application. In [4] the authors provided a unified framework for both MLP and RBF via a conic section transfer functions. For classification, the most comprehensive source of comparison is the Statlog report in []. The report presents the performance comparison between different NN and statistical algorithms for classification over different datasets. Both MLP and RBF did well most of the time. Results showed there is a huge discrepancy between RBF and MLP in terms of cross validation error. However, MLP outperformed RBF most of the time; RBF sometimes did not report valid results. The only problem with the MLP performance in this report is its training time which was order of magnitude greater than RBF networks training time. In [13], Wang and Huang systematically studied both the Sigmoidal and RBF activations functions in protein classification and reached the same conclusion as we did. Wang et al work is very significant due to the fact of using Extreme Learning Machines (ELM) instead of the (BP). Unlike BP which works only with differentiable activation functions, ELM can work with any nonlinear activation function; also, ELM does not need any control parameters.
3 A Quantitative Comparison of Different MLP Activation Functions in Classification 851 According to Wang et al, ELM achieved higher classification accuracy with up to 4 four orders of magnitude less training time compared to BP. In addition to confirming the fact that MLP with sigmoid activation function is superior for classification applications, our contribution in this paper is significantly pruning the MLP size to improve its training time to be less than the time needed for the RBF most of the time. Also, we showed the relation between the problem dimensionality and number of classes from one side and the performance of the activation function on the other side. We carefully observed the network initial state using each activation function and related it to the network ability of generalization. Activation Functions in Comparison Where n denotes the net-input to the hidden unit i and i o i denotes the unit output using the activation function: Form the linear family we have used the linear activation function such that for (a) and (b) as constants: o = an + b Derivative is: (a). i i (1) From the logistic family, we have used the two most commonly used functions: The Sigmoid function (the asymmetric logistic function [5]) : 1 o = i 1+ e ni Derivative is: oi(1 oi) () The hyperbolic tangent function (the symmetric logistic function [5] ): e ni e ni 1 e ni o = = i e n i e n i 1 e n Derivative is: (1 o ) + i i (3) Those functions are the most widely used in MLP implementation because their derivatives are easy to compute and can be expressed directly as a function of the net input. Also, their curve shape contains a linear region which makes them the most suitable functions for regularizing the network using weight decay [11]. From the RBF family, we use the Gaussian activation function: o 1 exp i n i m = i Derivative is: σ ( n ) oi i mi σ (4) Where m and σ are the center and the width, respectively. The reader is directed to [16] where there are more net input calculation and activation functions. 3 Datasets Selection We tried to cover all possible dataset criteria. Table 1 summarizes the datasets used.
4 85 E.A.M.A. Shenouda Table 1. Datasets characteristics Dataset Name Cases Dimension Classes % continuous valued features % nominal valued features 1 Optdigit No B_C_W Yes 3 Wpbc Yes 4 Wdbc No 5 Dermatology Yes 6 Car No 7 Monks No 8 Pima_diabates No 9 Sonar No 10 votes Yes Missing Values 4 Number of Hidden Unites and Centers of RBF To reduce MLP training time, all networks used in the experiments are of one hidden layer, which is the case by definition in RBF networks. For the MLP, network pruning is recommended by both Zurada and Haykin [17]-[5]. Instead of pruning only the weights, we started with number of hidden units equal to twice the number of the input features. Then, without affecting the CV error, we kept dividing this number by. By doing this, the number of hidden units in all MLP used was in the range between 4 and 5. From the 4 methods proposed in [5] to adapt RBF centers, we chose the closest to MLP implementation, where the number of hidden units is fixed and the centers are adapted via an unsupervised learning. The width of all functions are kept constant to be 1. Cross Validation (CV) is our main method for regularization and stopping. 5 Results Results for different activation functions performance via batch learning are tabulated in the Table. The analogous results for online learning are in Table 3: Where, A is the network initial state (starting MSE) after the first 50 epochs. B is the best MSE Table. Results for batch learning Sigmoid Tanh Linear Gaussian(RBF) DS A B C D A B C D A B C D A B C D
5 A Quantitative Comparison of Different MLP Activation Functions in Classification 853 Table 3. Results for the online learning DS Sigmoid Tanh Linear Gaussian(RBF) A B C D A B C D A B C D A B C D achieved in training, C is the MSE for testing, and D is the training time in seconds. (Note, cells A, B, C are divided by 10-3, bold figures represent best testing MSE). 6 Results Analysis We will provide result analysis with respect to MSE, training time needed and network initial state in both online and batch learning. Unlike the case in the StatLog report in [], we did not get huge discrepancy in performance between the MLP and RBF. In spite of the expected poor performance of the linear function compared to the other functions, it will get a special attention in the network initial state section. 6.1 Test MSE From the above tables and figures 1 and, we can observe that the sigmoidal activation function always achieved the best classification MSE in 9 of the 10 datasets. In dataset 4, the RBF got the best MSE in both online and batch learning, which still close to the MLP MSE. Also, it is natural for the tanh MSE to be the closest to the sigmoid MSE most of the time. Both functions belong to the logistic family and geometrically equivalent except for the there minimum values, which is the minimum saturation region. For the sigmoid, tanh and RBF, it is observable that: 1- Except for the dataset number 4 in online learning, the MSE for the three functions tends to increase or decrease together for each dataset. - The maximum value of the MSE for each dataset is the same or so close in both online and batch learning. 3- The sigmoid function experiences the least variance in terms of its MSE value in both online and batch learning. The difference between the average of the online test MSE and batch test MSE over the 10 datasets was , , and for the sigmoid, tanh and RBF respectively. That means the dataset characteristics like dimensionality, features type and number of classes desired do not have a remarkable effect of the sigmoid function ability to generalize.
6 854 E.A.M.A. Shenouda Results show that there is no clear or direct relation between the dataset dimensionality and the network ability to generalize for the three activations functions. For example, the three activation functions produced less MSE in dataset number 1 than what they produced for dataset number 6 which is less than 1/10 of the dimension, while the situation is reversed in case of datasets and 9. Fig. 1. Test (MSE), online learning Fig.. Test (MSE), batch learning 6. Training Time MLP with sigmoid activation did not only achieve the best MSE, but also needed the least training time in either online or batch leaning in all datasets. For example, for dataset 3, the sigmoid function consumed slightly more time than the RBF function using online learning, but it consumed less time for the same dataset in batch learning. The situation is reversed in the case of dataset 4. That means, by choosing the right training method and the only needed number of hidden units, we can reduce the MLP training time to be less or competitive to the RBF networks training time. Although the difference is minimal, the tanh function tended to consume less time than the sigmoid function in many cases, but this small difference in time should not give preference of the tanh over the sigmoid function because of the MSE of sigmoid is the best most of the time. Excluding the first and fourth datasets, the batch learning training time is less than the online learning training time, in average. Network initial state Fig. 3. Training time, online learning Fig. 4. Training Time, batch learning
7 A Quantitative Comparison of Different MLP Activation Functions in Classification 855 For the sigmoid, tanh and RBF we can observe the following: 1- Most of the time in both online and batch learning, the training time for the three functions tends to increase or decrease together for each dataset. - The sigmoid and tanh functions tend to experience less variance than the RBF networks in terms of time needed for each dataset. This means that the dimensionality of the dataset does not have a remarkable effect on the MLP time needed for training as long as there are enough examples. 3- On the other hand, results show that there is a strong relation between the number of classes and the time needed for training in both MLP and RBF. This is observable by inspecting datasets number 1 and 9. They have dimensionality of 64, 60 and number of classes of 10 and respectively. Although their dimensionalities are close and both have 100% continuous data type, the time needed for the first data set is order of magnitude more than the time needed for the 9 th dataset. The same relation holds for nominal data types as well in datasets 6 and The Network Initial State By network initial state we mean the MSE after 50 epochs of training. This section is a very important one as it carries, to some extend, the explanation of the previous analysis. Also, it clarifies the behavior of the linear activation function. Concerning the linear activation function, by observing the columns A and B in tables 1 and it would be clear that the final MSE after training is identical to the initial network state in 18 experiments out of 0, or 90% of the time. Also, in the remaining 10% the difference is almost negligible. This happens because the linear activation function turns the whole MLP into a normal linear regression machine, which does not tend to learn more by more iteration on the same data. However, it might improve its MSE when more examples are introduced. Fig. 5. Network initial state after 50 epochs, online learning Fig. 6. Network initial state after 50 epochs, batch learning Our results concerning this function comply with the results in [3] where the authors stated that they have experienced the same behavior form the step-like function after 10 epochs. That is why it would be misleading to compare the linear
8 856 E.A.M.A. Shenouda activation function training time with the other functions because it reaches its maximum generalization ability very quickly but always produces the highest MSE. Form the above tables and Figures 5 and 6, it is observable that the sigmoid activation function tends to produce the best network state after the first 50 epochs. The next best initial state was for the tanh while RBF and linear activation functions come last. The most important observation here is the analogy between the figures 5 and 1 and figures 6 and. The MSE values obtained are relatively symmetrical for each function in each dataset. That means whenever the function reaches a good initial state, it generalizes well and vice verse. So, observing the network initial state is an essential task to see how fast the MSE error drops and what value it will reach; this might be an indicator that the network parameters needs some modifications or the number of hidden units is not enough. Sometimes only re-randomizing the weights and neurons thresholds values will put the network in a better initial state. Also, observing the network initial state was very useful while performing MLP hidden units pruning to decrease time needed for training. 7 Concluding Remarks and Future Work We showed that the sigmoidal activation function is the best for classification applications in terms of time and MSE. We showed a relation between the dataset dimensionality, number of classes and time need for training. Also, the results indicated that the network initial state highly impacted the test MSE and consequently, the overall network ability for generalization. Still, more research can and should be done concerning MLP activation functions. It will be very interesting to use the EM algorithm to adapt the centers of RBF networks. Also, the performance of RBF with a huge fixed number of centers versus the ones with small but adapted centers is still an open issue. Haykin showed that they should perform the same, but he also showed that this contradicts with the NETtalk experiment where using unsupervised learning to adapt the centers of RBFN always resulted in a network with poor generalization than MLP, which was the opposite when supervised learning was used. Acknowledgement The author would like to thank Prof. Anthony J. Bonner for his valuable advices for the final preparation of this manuscript. References 1. Lu, B., Evans, B.L.: Channel Equalization by Feedforward Neural Networks. In: IEEE Int. Symposium on Circuits and Systems, Vol. 5. Orlando, FL (1999) Michie, D., Spiegelhalter, D.J., Taylor, C.C.: Machine Learning, Neural and Statistical Classification. Elis Horwood, London (1994) 3. Piekniewski, F., Rybicki, L.: Visual Comparison of Performance for Different Activation Functions in MLP Networks. In: IJCNN 004 & FUZZ-IEEE, Vol. 4. Budapest (004)
9 A Quantitative Comparison of Different MLP Activation Functions in Classification Dorffner, G.: A Unified Framework for MLPs and RBFNs: Introducing Conic Section Function Networks. Cybernetics and Systems 5(4) (1994) Haykin, S.: Neural Networks A Comprehensive Introduction. Prentice Hall, New Jersey (1999) 6. Huang, G., Chen, Y., Babri, H.A.: Classification Ability of Single Hidden Layer Feedforward Neural Networks. IEEE Transactions on Neural Networks 11(3) (000) Le Cun, Y., Touresky, D., Hinton G., Sejnowski, T.: A Theoretical Framework for Backpropagation. In: The Connectionist Models Summer School. (1988) Li, Y., Pont, M. J., Jones, N.B.: A Comparison of the Performance of Radial Basis Function and Multi-layer Perceptron Networks in Condition Monitoring and Fault Diagnosis. In: The International Conference on Condition Monitoring. Swansea (1999) Arahal, M.R., Camacho, E.F.: Application of the Ran Algorithm to the Problem of Short Term Load Forecasting. Technical Report, University of Sevilla, Sevilla (1996) 10. Finan, R.A., Sapeluk, A.T., Damper, R.I.: Comparison of Multilayer and Radial Basis Function Neural Networks for Text-Dependent Speaker Recognition. In: IEEE Int. Conf. on Neural Networks, Vol. 4. Washington DC (1996) Karkkainen, T.: MLP in Layer-Wise Form with Applications to Weight Decay. Neural Computation 14(6) (00) Werbos, P. J.: Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences. Doctoral Thesis, Applied Mathematics, Harvard University. Boston (1974) 13. Wang, D., Huang, G.: Protein Sequence Classification Using Extreme Learning Machine. In: IJCNN05, Vol. 3. Montréal (005) Duch, W., Jankowski, N.: Survey of Neural Transfer Functions. Neural Computing Surveys (1999) Duch, W., Jankowski, N.: Transfer functions: Hidden Possibilities for Better Neural Networks. In: 9 th European Symposium on Artificial Neural Network. Bruges (001) Hu, Y., Hwang, J.: Handbook of Neural Network Signal Processing. 3 rd edn. CRC-Press, Florida (00) 17. Zurada, J. M.: Introduction to Artificial Neural Systems. PWS Publishing, Boston (1999)
AN IMPROVED NEURAL NETWORK-BASED DECODER SCHEME FOR SYSTEMATIC CONVOLUTIONAL CODE. A Thesis by. Andrew J. Zerngast
AN IMPROVED NEURAL NETWORK-BASED DECODER SCHEME FOR SYSTEMATIC CONVOLUTIONAL CODE A Thesis by Andrew J. Zerngast Bachelor of Science, Wichita State University, 2008 Submitted to the Department of Electrical
More informationIBM SPSS Neural Networks
IBM Software IBM SPSS Neural Networks 20 IBM SPSS Neural Networks New tools for building predictive models Highlights Explore subtle or hidden patterns in your data. Build better-performing models No programming
More informationApproximation a One-Dimensional Functions by Using Multilayer Perceptron and Radial Basis Function Networks
Approximation a One-Dimensional Functions by Using Multilayer Perceptron and Radial Basis Function Networks Huda Dheyauldeen Najeeb Department of public relations College of Media, University of Al Iraqia,
More informationEnhanced MLP Input-Output Mapping for Degraded Pattern Recognition
Enhanced MLP Input-Output Mapping for Degraded Pattern Recognition Shigueo Nomura and José Ricardo Gonçalves Manzan Faculty of Electrical Engineering, Federal University of Uberlândia, Uberlândia, MG,
More informationUsing of Artificial Neural Networks to Recognize the Noisy Accidents Patterns of Nuclear Research Reactors
Int. J. Advanced Networking and Applications 1053 Using of Artificial Neural Networks to Recognize the Noisy Accidents Patterns of Nuclear Research Reactors Eng. Abdelfattah A. Ahmed Atomic Energy Authority,
More informationLesson 08. Convolutional Neural Network. Ing. Marek Hrúz, Ph.D. Katedra Kybernetiky Fakulta aplikovaných věd Západočeská univerzita v Plzni.
Lesson 08 Convolutional Neural Network Ing. Marek Hrúz, Ph.D. Katedra Kybernetiky Fakulta aplikovaných věd Západočeská univerzita v Plzni Lesson 08 Convolution we will consider 2D convolution the result
More informationNEURAL NETWORK BASED MAXIMUM POWER POINT TRACKING
NEURAL NETWORK BASED MAXIMUM POWER POINT TRACKING 3.1 Introduction This chapter introduces concept of neural networks, it also deals with a novel approach to track the maximum power continuously from PV
More informationComparison of MLP and RBF neural networks for Prediction of ECG Signals
124 Comparison of MLP and RBF neural networks for Prediction of ECG Signals Ali Sadr 1, Najmeh Mohsenifar 2, Raziyeh Sadat Okhovat 3 Department Of electrical engineering Iran University of Science and
More informationA COMPARISON OF ARTIFICIAL NEURAL NETWORKS AND OTHER STATISTICAL METHODS FOR ROTATING MACHINE
A COMPARISON OF ARTIFICIAL NEURAL NETWORKS AND OTHER STATISTICAL METHODS FOR ROTATING MACHINE CONDITION CLASSIFICATION A. C. McCormick and A. K. Nandi Abstract Statistical estimates of vibration signals
More informationNeural Filters: MLP VIS-A-VIS RBF Network
6th WSEAS International Conference on CIRCUITS, SYSTEMS, ELECTRONICS,CONTROL & SIGNAL PROCESSING, Cairo, Egypt, Dec 29-31, 2007 432 Neural Filters: MLP VIS-A-VIS RBF Network V. R. MANKAR, DR. A. A. GHATOL,
More informationGENERATION OF TANGENT HYPERBOLIC SIGMOID FUNCTION FOR MICROCONTROLLER BASED DIGITAL IMPLEMENTATIONS OF NEURAL NETWORKS
GENERATION OF TANGENT HYPERBOLIC SIGMOID FUNCTION FOR MICROCONTROLLER BASED DIGITAL IMPLEMENTATIONS OF NEURAL NETWORKS Mutlu Avcı, Tulay Yıldırım Yildiz Technical University Electronics and Communication
More informationUse of Neural Networks in Testing Analog to Digital Converters
Use of Neural s in Testing Analog to Digital Converters K. MOHAMMADI, S. J. SEYYED MAHDAVI Department of Electrical Engineering Iran University of Science and Technology Narmak, 6844, Tehran, Iran Abstract:
More informationDIAGNOSIS OF STATOR FAULT IN ASYNCHRONOUS MACHINE USING SOFT COMPUTING METHODS
DIAGNOSIS OF STATOR FAULT IN ASYNCHRONOUS MACHINE USING SOFT COMPUTING METHODS K. Vinoth Kumar 1, S. Suresh Kumar 2, A. Immanuel Selvakumar 1 and Vicky Jose 1 1 Department of EEE, School of Electrical
More informationTransient stability Assessment using Artificial Neural Network Considering Fault Location
Vol.6 No., 200 مجلد 6, العدد, 200 Proc. st International Conf. Energy, Power and Control Basrah University, Basrah, Iraq 0 Nov. to 2 Dec. 200 Transient stability Assessment using Artificial Neural Network
More informationNEURAL NETWORK DEMODULATOR FOR QUADRATURE AMPLITUDE MODULATION (QAM)
NEURAL NETWORK DEMODULATOR FOR QUADRATURE AMPLITUDE MODULATION (QAM) Ahmed Nasraden Milad M. Aziz M Rahmadwati Artificial neural network (ANN) is one of the most advanced technology fields, which allows
More informationDeep Neural Networks (2) Tanh & ReLU layers; Generalisation and Regularisation
Deep Neural Networks (2) Tanh & ReLU layers; Generalisation and Regularisation Steve Renals Machine Learning Practical MLP Lecture 4 9 October 2018 MLP Lecture 4 / 9 October 2018 Deep Neural Networks (2)
More information1 Introduction. w k x k (1.1)
Neural Smithing 1 Introduction Artificial neural networks are nonlinear mapping systems whose structure is loosely based on principles observed in the nervous systems of humans and animals. The major
More informationCurrent Harmonic Estimation in Power Transmission Lines Using Multi-layer Perceptron Learning Strategies
Journal of Electrical Engineering 5 (27) 29-23 doi:.7265/2328-2223/27.5. D DAVID PUBLISHING Current Harmonic Estimation in Power Transmission Lines Using Multi-layer Patrice Wira and Thien Minh Nguyen
More informationSonia Sharma ECE Department, University Institute of Engineering and Technology, MDU, Rohtak, India. Fig.1.Neuron and its connection
NEUROCOMPUTATION FOR MICROSTRIP ANTENNA Sonia Sharma ECE Department, University Institute of Engineering and Technology, MDU, Rohtak, India Abstract: A Neural Network is a powerful computational tool that
More informationNeural Labyrinth Robot Finding the Best Way in a Connectionist Fashion
Neural Labyrinth Robot Finding the Best Way in a Connectionist Fashion Marvin Oliver Schneider 1, João Luís Garcia Rosa 1 1 Mestrado em Sistemas de Computação Pontifícia Universidade Católica de Campinas
More informationCHAPTER 6 BACK PROPAGATED ARTIFICIAL NEURAL NETWORK TRAINED ARHF
95 CHAPTER 6 BACK PROPAGATED ARTIFICIAL NEURAL NETWORK TRAINED ARHF 6.1 INTRODUCTION An artificial neural network (ANN) is an information processing model that is inspired by biological nervous systems
More informationIntroduction to Machine Learning
Introduction to Machine Learning Deep Learning Barnabás Póczos Credits Many of the pictures, results, and other materials are taken from: Ruslan Salakhutdinov Joshua Bengio Geoffrey Hinton Yann LeCun 2
More informationReplacing Fuzzy Systems with Neural Networks
Replacing Fuzzy Systems with Neural Networks Tiantian Xie, Hao Yu, and Bogdan Wilamowski Auburn University, Alabama, USA, tzx@auburn.edu, hzy@auburn.edu, wilam@ieee.org Abstract. In this paper, a neural
More informationA Robust Footprint Detection Using Color Images and Neural Networks
A Robust Footprint Detection Using Color Images and Neural Networks Marco Mora 1 and Daniel Sbarbaro 2 1 Department of Computer Science, Catholic University of Maule, Casilla 617, Talca, Chile marco.mora@enseeiht.fr
More informationA New Localization Algorithm Based on Taylor Series Expansion for NLOS Environment
BULGARIAN ACADEMY OF SCIENCES CYBERNETICS AND INFORMATION TECHNOLOGIES Volume 16, No 5 Special Issue on Application of Advanced Computing and Simulation in Information Systems Sofia 016 Print ISSN: 1311-970;
More informationCHAPTER 4 LINK ADAPTATION USING NEURAL NETWORK
CHAPTER 4 LINK ADAPTATION USING NEURAL NETWORK 4.1 INTRODUCTION For accurate system level simulator performance, link level modeling and prediction [103] must be reliable and fast so as to improve the
More informationAnalysis of Learning Paradigms and Prediction Accuracy using Artificial Neural Network Models
Analysis of Learning Paradigms and Prediction Accuracy using Artificial Neural Network Models Poornashankar 1 and V.P. Pawar 2 Abstract: The proposed work is related to prediction of tumor growth through
More informationAn Hybrid MLP-SVM Handwritten Digit Recognizer
An Hybrid MLP-SVM Handwritten Digit Recognizer A. Bellili ½ ¾ M. Gilloux ¾ P. Gallinari ½ ½ LIP6, Université Pierre et Marie Curie ¾ La Poste 4, Place Jussieu 10, rue de l Ile Mabon, BP 86334 75252 Paris
More informationISSN: [Jha* et al., 5(12): December, 2016] Impact Factor: 4.116
IJESRT INTERNATIONAL JOURNAL OF ENGINEERING SCIENCES & RESEARCH TECHNOLOGY ANALYSIS OF DIRECTIVITY AND BANDWIDTH OF COAXIAL FEED SQUARE MICROSTRIP PATCH ANTENNA USING ARTIFICIAL NEURAL NETWORK Rohit Jha*,
More informationMultiple-Layer Networks. and. Backpropagation Algorithms
Multiple-Layer Networks and Algorithms Multiple-Layer Networks and Algorithms is the generalization of the Widrow-Hoff learning rule to multiple-layer networks and nonlinear differentiable transfer functions.
More informationMLP for Adaptive Postprocessing Block-Coded Images
1450 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 10, NO. 8, DECEMBER 2000 MLP for Adaptive Postprocessing Block-Coded Images Guoping Qiu, Member, IEEE Abstract A new technique
More informationARTIFICIAL NEURAL NETWORK BASED CLASSIFICATION FOR MONOBLOCK CENTRIFUGAL PUMP USING WAVELET ANALYSIS
International Journal of Mechanical Engineering and Technology (IJMET), ISSN 0976 6340(Print) ISSN 0976 6359(Online) Volume 1 Number 1, July - Aug (2010), pp. 28-37 IAEME, http://www.iaeme.com/ijmet.html
More informationPOLITEHNICA UNIVERSITY TIMISOARA
POLITEHNICA UNIVERSITY TIMISOARA ELECTRONICS AND TELECOMMUNICATIONS FACULTY NEURAL NETWORK APPLICATIONS FOR RADIOCOVERAGE STUDIES IN MOBILE COMMUNICATION SYSTEMS Ph. D. Thesis Eng. Ileana Popescu Supervisors:
More informationNeural Network based Digital Receiver for Radio Communications
Neural Network based Digital Receiver for Radio Communications G. LIODAKIS, D. ARVANITIS, and I.O. VARDIAMBASIS Microwave Communications & Electromagnetic Applications Laboratory, Department of Electronics,
More informationMINE 432 Industrial Automation and Robotics
MINE 432 Industrial Automation and Robotics Part 3, Lecture 5 Overview of Artificial Neural Networks A. Farzanegan (Visiting Associate Professor) Fall 2014 Norman B. Keevil Institute of Mining Engineering
More informationCC4.5: cost-sensitive decision tree pruning
Data Mining VI 239 CC4.5: cost-sensitive decision tree pruning J. Cai 1,J.Durkin 1 &Q.Cai 2 1 Department of Electrical and Computer Engineering, University of Akron, U.S.A. 2 Department of Electrical Engineering
More informationArtificial Neural Networks. Artificial Intelligence Santa Clara, 2016
Artificial Neural Networks Artificial Intelligence Santa Clara, 2016 Simulate the functioning of the brain Can simulate actual neurons: Computational neuroscience Can introduce simplified neurons: Neural
More informationThe Basic Kak Neural Network with Complex Inputs
The Basic Kak Neural Network with Complex Inputs Pritam Rajagopal The Kak family of neural networks [3-6,2] is able to learn patterns quickly, and this speed of learning can be a decisive advantage over
More informationThe Hamming Code Performance Analysis using RBF Neural Network
, 22-24 October, 2014, San Francisco, USA The Hamming Code Performance Analysis using RBF Neural Network Omid Haddadi, Zahra Abbasi, and Hossein TooToonchy, Member, IAENG Abstract In this paper the Hamming
More informationApplication of Feed-forward Artificial Neural Networks to the Identification of Defective Analog Integrated Circuits
eural Comput & Applic (2002)11:71 79 Ownership and Copyright 2002 Springer-Verlag London Limited Application of Feed-forward Artificial eural etworks to the Identification of Defective Analog Integrated
More informationA Neural Solution for Signal Detection In Non-Gaussian Noise
1 A Neural Solution for Signal Detection In Non-Gaussian Noise D G Khairnar, S N Merchant, U B Desai SPANN Laboratory Department of Electrical Engineering Indian Institute of Technology, Bombay, Mumbai-400
More informationIntroduction to Machine Learning
Introduction to Machine Learning Perceptron Barnabás Póczos Contents History of Artificial Neural Networks Definitions: Perceptron, Multi-Layer Perceptron Perceptron algorithm 2 Short History of Artificial
More informationComparative Analysis of Self Organizing Maps vs. Multilayer Perceptron Neural Networks for Short - Term Load Forecasting
Comparative Analysis of Self Organizing Maps vs Multilayer Perceptron Neural Networks for Short - Term Load Forecasting S Valero IEEE Member (1), J Aparicio (2), C Senabre (1), M Ortiz, IEEE Student Member
More informationNeural Model for Path Loss Prediction in Suburban Environment
Neural Model for Path Loss Prediction in Suburban Environment Ileana Popescu, Ioan Nafornita, Philip Constantinou 3, Athanasios Kanatas 3, Netarios Moraitis 3 University of Oradea, 5 Armatei Romane Str.,
More informationStock Price Prediction Using Multilayer Perceptron Neural Network by Monitoring Frog Leaping Algorithm
Stock Price Prediction Using Multilayer Perceptron Neural Network by Monitoring Frog Leaping Algorithm Ahdieh Rahimi Garakani Department of Computer South Tehran Branch Islamic Azad University Tehran,
More informationBackground Pixel Classification for Motion Detection in Video Image Sequences
Background Pixel Classification for Motion Detection in Video Image Sequences P. Gil-Jiménez, S. Maldonado-Bascón, R. Gil-Pita, and H. Gómez-Moreno Dpto. de Teoría de la señal y Comunicaciones. Universidad
More informationApplication of Multi Layer Perceptron (MLP) for Shower Size Prediction
Chapter 3 Application of Multi Layer Perceptron (MLP) for Shower Size Prediction 3.1 Basic considerations of the ANN Artificial Neural Network (ANN)s are non- parametric prediction tools that can be used
More informationNeural Network Classifier and Filtering for EEG Detection in Brain-Computer Interface Device
Neural Network Classifier and Filtering for EEG Detection in Brain-Computer Interface Device Mr. CHOI NANG SO Email: cnso@excite.com Prof. J GODFREY LUCAS Email: jglucas@optusnet.com.au SCHOOL OF MECHATRONICS,
More informationTransactions on Information and Communications Technologies vol 1, 1993 WIT Press, ISSN
Combining multi-layer perceptrons with heuristics for reliable control chart pattern classification D.T. Pham & E. Oztemel Intelligent Systems Research Laboratory, School of Electrical, Electronic and
More informationPractical Comparison of Results of Statistic Regression Analysis and Neural Network Regression Analysis
Practical Comparison of Results of Statistic Regression Analysis and Neural Network Regression Analysis Marek Vochozka Institute of Technology and Businesses in České Budějovice Abstract There are many
More informationAvailable online at ScienceDirect. Procedia Computer Science 85 (2016 )
Available online at www.sciencedirect.com ScienceDirect Procedia Computer Science 85 (2016 ) 263 270 International Conference on Computational Modeling and Security (CMS 2016) Proposing Solution to XOR
More informationNeural Network Modeling of Valve Stiction Dynamics
Proceedings of the World Congress on Engineering and Computer Science 7 WCECS 7, October 4-6, 7, San Francisco, USA Neural Network Modeling of Valve Stiction Dynamics H. Zabiri, Y. Samyudia, W. N. W. M.
More informationA Technique for Pulse RADAR Detection Using RRBF Neural Network
Proceedings of the World Congress on Engineering 22 Vol II WCE 22, July 4-6, 22, London, U.K. A Technique for Pulse RADAR Detection Using RRBF Neural Network Ajit Kumar Sahoo, Ganapati Panda and Babita
More informationLearning New Articulator Trajectories for a Speech Production Model using Artificial Neural Networks
Learning New Articulator Trajectories for a Speech Production Model using Artificial Neural Networks C. S. Blackburn and S. J. Young Cambridge University Engineering Department (CUED), England email: csb@eng.cam.ac.uk
More informationHarmonic detection by using different artificial neural network topologies
Harmonic detection by using different artificial neural network topologies J.L. Flores Garrido y P. Salmerón Revuelta Department of Electrical Engineering E. P. S., Huelva University Ctra de Palos de la
More informationMAGNT Research Report (ISSN ) Vol.6(1). PP , Controlling Cost and Time of Construction Projects Using Neural Network
Controlling Cost and Time of Construction Projects Using Neural Network Li Ping Lo Faculty of Computer Science and Engineering Beijing University China Abstract In order to achieve optimized management,
More informationMicroprocessor Implementation of Fuzzy Systems and Neural Networks Jeremy Binfet Micron Technology
Microprocessor Implementation of Fuy Systems and Neural Networks Jeremy Binfet Micron Technology jbinfet@micron.com Bogdan M. Wilamowski University of Idaho wilam@ieee.org Abstract Systems were implemented
More informationSystolic modular VLSI Architecture for Multi-Model Neural Network Implementation +
Systolic modular VLSI Architecture for Multi-Model Neural Network Implementation + J.M. Moreno *, J. Madrenas, J. Cabestany Departament d'enginyeria Electrònica Universitat Politècnica de Catalunya Barcelona,
More informationComputational Intelligence Introduction
Computational Intelligence Introduction Farzaneh Abdollahi Department of Electrical Engineering Amirkabir University of Technology Fall 2011 Farzaneh Abdollahi Neural Networks 1/21 Fuzzy Systems What are
More informationPID Controller Design Based on Radial Basis Function Neural Networks for the Steam Generator Level Control
BULGARIAN ACADEMY OF SCIENCES CYBERNETICS AND INFORMATION TECHNOLOGIES Volume 6 No 5 Special Issue on Application of Advanced Computing and Simulation in Information Systems Sofia 06 Print ISSN: 3-970;
More informationSensors & Transducers Published by IFSA Publishing, S. L.,
Sensors & Transducers Published by IFSA Publishing, S. L., 2017 http://www.sensorsportal.com Modeling Nanoscale FinFET Performance by a Neural Network Method 1 Jin He*, 1 Guoqing Hu, 1 Bing Xie, 1 Guangjin
More informationIDENTIFICATION OF POWER QUALITY PROBLEMS IN IEEE BUS SYSTEM BY USING NEURAL NETWORKS
Fourth International Conference on Control System and Power Electronics CSPE IDENTIFICATION OF POWER QUALITY PROBLEMS IN IEEE BUS SYSTEM BY USING NEURAL NETWORKS Mr. Devadasu * and Dr. M Sushama ** * Associate
More informationArtificial Intelligence Elman Backpropagation Computing Models for Predicting Shelf Life of. Processed Cheese
Vol.4/No.1 B (01) INTERNETWORKING INDONESIA JOURNAL 3 Artificial Intelligence Elman Backpropagation Computing Models for Predicting Shelf Life of Processed Cheese Sumit Goyal and Gyanendra Kumar Goyal
More informationGeometric Neurodynamical Classifiers Applied to Breast Cancer Detection. Tijana T. Ivancevic
Geometric Neurodynamical Classifiers Applied to Breast Cancer Detection Tijana T. Ivancevic Thesis submitted for the Degree of Doctor of Philosophy in Applied Mathematics at The University of Adelaide
More informationApplication of Generalised Regression Neural Networks in Lossless Data Compression
Application of Generalised Regression Neural Networks in Lossless Data Compression R. LOGESWARAN Centre for Multimedia Communications, Faculty of Engineering, Multimedia University, 63100 Cyberjaya MALAYSIA
More informationSegmentation of Fingerprint Images
Segmentation of Fingerprint Images Asker M. Bazen and Sabih H. Gerez University of Twente, Department of Electrical Engineering, Laboratory of Signals and Systems, P.O. box 217-75 AE Enschede - The Netherlands
More informationOn the Application of Artificial Neural Network in Analyzing and Studying Daily Loads of Jordan Power System Plant
UDC 004.725 On the Application of Artificial Neural Network in Analyzing and Studying Daily Loads of Jordan Power System Plant Salam A. Najim 1, Zakaria A. M. Al-Omari 2 and Samir M. Said 1 1 Faculty of
More informationA Novel Fuzzy Neural Network Based Distance Relaying Scheme
902 IEEE TRANSACTIONS ON POWER DELIVERY, VOL. 15, NO. 3, JULY 2000 A Novel Fuzzy Neural Network Based Distance Relaying Scheme P. K. Dash, A. K. Pradhan, and G. Panda Abstract This paper presents a new
More informationBEING wideband, chaotic signals are well suited for
680 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS II: EXPRESS BRIEFS, VOL. 51, NO. 12, DECEMBER 2004 Performance of Differential Chaos-Shift-Keying Digital Communication Systems Over a Multipath Fading Channel
More informationA Comparison of Particle Swarm Optimization and Gradient Descent in Training Wavelet Neural Network to Predict DGPS Corrections
Proceedings of the World Congress on Engineering and Computer Science 00 Vol I WCECS 00, October 0-, 00, San Francisco, USA A Comparison of Particle Swarm Optimization and Gradient Descent in Training
More informationSeveral Different Remote Sensing Image Classification Technology Analysis
Vol. 4, No. 5; October 2011 Several Different Remote Sensing Image Classification Technology Analysis Xiangwei Liu Foundation Department, PLA University of Foreign Languages, Luoyang 471003, China E-mail:
More informationArtificial Neural Network Engine: Parallel and Parameterized Architecture Implemented in FPGA
Artificial Neural Network Engine: Parallel and Parameterized Architecture Implemented in FPGA Milene Barbosa Carvalho 1, Alexandre Marques Amaral 1, Luiz Eduardo da Silva Ramos 1,2, Carlos Augusto Paiva
More informationPrediction of Cluster System Load Using Artificial Neural Networks
Prediction of Cluster System Load Using Artificial Neural Networks Y.S. Artamonov 1 1 Samara National Research University, 34 Moskovskoe Shosse, 443086, Samara, Russia Abstract Currently, a wide range
More informationFigure 1. Artificial Neural Network structure. B. Spiking Neural Networks Spiking Neural networks (SNNs) fall into the third generation of neural netw
Review Analysis of Pattern Recognition by Neural Network Soni Chaturvedi A.A.Khurshid Meftah Boudjelal Electronics & Comm Engg Electronics & Comm Engg Dept. of Computer Science P.I.E.T, Nagpur RCOEM, Nagpur
More informationApplication of Classifier Integration Model to Disturbance Classification in Electric Signals
Application of Classifier Integration Model to Disturbance Classification in Electric Signals Dong-Chul Park Abstract An efficient classifier scheme for classifying disturbances in electric signals using
More informationIJITKMI Volume 7 Number 2 Jan June 2014 pp (ISSN ) Impact of attribute selection on the accuracy of Multilayer Perceptron
Impact of attribute selection on the accuracy of Multilayer Perceptron Niket Kumar Choudhary 1, Yogita Shinde 2, Rajeswari Kannan 3, Vaithiyanathan Venkatraman 4 1,2 Dept. of Computer Engineering, Pimpri-Chinchwad
More informationAdaptive Multi-layer Neural Network Receiver Architectures for Pattern Classification of Respective Wavelet Images
Adaptive Multi-layer Neural Network Receiver Architectures for Pattern Classification of Respective Wavelet Images Pythagoras Karampiperis 1, and Nikos Manouselis 2 1 Dynamic Systems and Simulation Laboratory
More informationA linear Multi-Layer Perceptron for identifying harmonic contents of biomedical signals
A linear Multi-Layer Perceptron for identifying harmonic contents of biomedical signals Thien Minh Nguyen 1 and Patrice Wira 1 Université de Haute Alsace, Laboratoire MIPS, Mulhouse, France, {thien-minh.nguyen,
More informationA fast and accurate distance relaying scheme using an efficient radial basis function neural network
Electric Power Systems Research 60 (2001) 1 8 www.elsevier.com/locate/epsr A fast and accurate distance relaying scheme using an efficient radial basis function neural network A.K. Pradhan *, P.K. Dash,
More informationA study on the ability of Support Vector Regression and Neural Networks to Forecast Basic Time Series Patterns
A study on the ability of Support Vector Regression and Neural Networks to Forecast Basic Time Series Patterns Sven F. Crone', Jose Guajardo^, and Richard Weber^ Lancaster University, Department of Management
More informationFAULT DIAGNOSIS AND PERFORMANCE ASSESSMENT FOR A ROTARY ACTUATOR BASED ON NEURAL NETWORK OBSERVER
7 Journal of Marine Science and Technology, Vol., No., pp. 7-78 () DOI:.9/JMST-3 FAULT DIAGNOSIS AND PERFORMANCE ASSESSMENT FOR A ROTARY ACTUATOR BASED ON NEURAL NETWORK OBSERVER Jian Ma,, Xin Li,, Chen
More informationTiny ImageNet Challenge Investigating the Scaling of Inception Layers for Reduced Scale Classification Problems
Tiny ImageNet Challenge Investigating the Scaling of Inception Layers for Reduced Scale Classification Problems Emeric Stéphane Boigné eboigne@stanford.edu Jan Felix Heyse heyse@stanford.edu Abstract Scaling
More informationSynthesis of Fault Tolerant Neural Networks
Synthesis of Fault Tolerant Neural Networks Dhananjay S. Phatak and Elko Tchernev ABSTRACT This paper evaluates different strategies for enhancing (partial) fault tolerance (PFT) of feedforward artificial
More informationCHAPTER 6 ANFIS BASED NEURO-FUZZY CONTROLLER
143 CHAPTER 6 ANFIS BASED NEURO-FUZZY CONTROLLER 6.1 INTRODUCTION The quality of generated electricity in power system is dependent on the system output, which has to be of constant frequency and must
More informationChapter - 7. Adaptive Channel Equalization
Chapter - 7 Adaptive Channel Equalization Chapter - 7 Adaptive Channel Equalization 7.1 Introduction The transmission o f digital information over a communication channel causes Inter Symbol Interference
More informationBehavior Emergence in Autonomous Robot Control by Means of Feedforward and Recurrent Neural Networks
Behavior Emergence in Autonomous Robot Control by Means of Feedforward and Recurrent Neural Networks Stanislav Slušný, Petra Vidnerová, Roman Neruda Abstract We study the emergence of intelligent behavior
More informationThe Use of Neural Network to Recognize the Parts of the Computer Motherboard
Journal of Computer Sciences 1 (4 ): 477-481, 2005 ISSN 1549-3636 Science Publications, 2005 The Use of Neural Network to Recognize the Parts of the Computer Motherboard Abbas M. Ali, S.D.Gore and Musaab
More informationSupervised Versus Unsupervised Binary-Learning by Feedforward Neural Networks
Machine Learning, 42, 97 122, 2001 c 2001 Kluwer Academic Publishers. Manufactured in The Netherlands. Supervised Versus Unsupervised Binary-Learning by Feedforward Neural Networks NATHALIE JAPKOWICZ nat@site.uottawa.ca
More informationLecture 3 - Regression
Lecture 3 - Regression Instructor: Prof Ganesh Ramakrishnan July 25, 2016 1 / 30 The Simplest ML Problem: Least Square Regression Curve Fitting: Motivation Error measurement Minimizing Error Method of
More informationTCM-coded OFDM assisted by ANN in Wireless Channels
1 Aradhana Misra & 2 Kandarpa Kumar Sarma Dept. of Electronics and Communication Technology Gauhati University Guwahati-781014. Assam, India Email: aradhana66@yahoo.co.in, kandarpaks@gmail.com Abstract
More informationA Simple Design and Implementation of Reconfigurable Neural Networks
A Simple Design and Implementation of Reconfigurable Neural Networks Hazem M. El-Bakry, and Nikos Mastorakis Abstract There are some problems in hardware implementation of digital combinational circuits.
More informationStacking Ensemble for auto ml
Stacking Ensemble for auto ml Khai T. Ngo Thesis submitted to the Faculty of the Virginia Polytechnic Institute and State University in partial fulfillment of the requirements for the degree of Master
More informationCHAPTER 4 MONITORING OF POWER SYSTEM VOLTAGE STABILITY THROUGH ARTIFICIAL NEURAL NETWORK TECHNIQUE
53 CHAPTER 4 MONITORING OF POWER SYSTEM VOLTAGE STABILITY THROUGH ARTIFICIAL NEURAL NETWORK TECHNIQUE 4.1 INTRODUCTION Due to economic reasons arising out of deregulation and open market of electricity,
More informationForecasting Exchange Rates using Neural Neworks
International Journal of Information & Computation Technology. ISSN 0974-2239 Volume 6, Number 1 (2016), pp. 35-44 International Research Publications House http://www. irphouse.com Forecasting Exchange
More informationPartial Discharge Classification Using Novel Parameters and a Combined PCA and MLP Technique
Partial Discharge Classification Using Novel Parameters and a Combined PCA and MLP Technique C. Chang and Q. Su Center for Electrical Power Engineering Monash University, Clayton VIC 3168 Australia Abstract:
More informationConstant False Alarm Rate Detection of Radar Signals with Artificial Neural Networks
Högskolan i Skövde Department of Computer Science Constant False Alarm Rate Detection of Radar Signals with Artificial Neural Networks Mirko Kück mirko@ida.his.se Final 6 October, 1996 Submitted by Mirko
More informationStatistical Tests: More Complicated Discriminants
03/07/07 PHY310: Statistical Data Analysis 1 PHY310: Lecture 14 Statistical Tests: More Complicated Discriminants Road Map When the likelihood discriminant will fail The Multi Layer Perceptron discriminant
More informationJ. C. Brégains (Student Member, IEEE), and F. Ares (Senior Member, IEEE).
ANALYSIS, SYNTHESIS AND DIAGNOSTICS OF ANTENNA ARRAYS THROUGH COMPLEX-VALUED NEURAL NETWORKS. J. C. Brégains (Student Member, IEEE), and F. Ares (Senior Member, IEEE). Radiating Systems Group, Department
More informationENVIRONMENTALLY ADAPTIVE SONAR CONTROL IN A TACTICAL SETTING
ENVIRONMENTALLY ADAPTIVE SONAR CONTROL IN A TACTICAL SETTING WARREN L. J. FOX, MEGAN U. HAZEN, AND CHRIS J. EGGEN University of Washington, Applied Physics Laboratory, 13 NE 4th St., Seattle, WA 98, USA
More informationA comparative study of different feature sets for recognition of handwritten Arabic numerals using a Multi Layer Perceptron
Proc. National Conference on Recent Trends in Intelligent Computing (2006) 86-92 A comparative study of different feature sets for recognition of handwritten Arabic numerals using a Multi Layer Perceptron
More information