MINE 432 Industrial Automation and Robotics Part 3, Lecture 5 Overview of Artificial Neural Networks A. Farzanegan (Visiting Associate Professor) Fall 2014 Norman B. Keevil Institute of Mining Engineering
Today s Topics Introduction Biological neurons in humans Artificial Neuron Artificial Neural Networks Applications of ANNs MINE 432 - Industrial Automation and Robotics 2
Introduction
Introduction Artificial neural network (ANN) is a research subject in many scientific and engineering fields including machine learning, paralleled processing, modeling and simulation, image processing, etc. The idea of developing artificial neural networks was inspired by findings in neurosciences and biology about human brain and how it works MINE 432 - Industrial Automation and Robotics 4
Milestones in ANN Development 1943 McCulloch-Pitts neurons 1949 Hebb s law 1958 Perceptron (Rosenblatt) 1960 Adaline, better learning rule (Widrow, Huff) 1969 Limitations (Minsky, Papert) 1972 Kohonen nets, associative memory 1977 Brain State in a Box (Anderson) 1982 Hopfield net, constraint satisfaction 1985 ART (Carpenter, Grossfield) 1986 Backpropagation (Rumelhart, Hinton, McClelland) 1988 Neocognitron, character recognition (Fukushima) MINE 432 - Industrial Automation and Robotics 5
ANN as part of MATLAB Math, Statistics and Optimization Apps ANN Toolbox was added to MATLAB in 1992 MINE 432 - Industrial Automation and Robotics 6
Biological Neurons in Humans
Biological Neurons in Humans Most of neurons in human s body are located in brain. It is estimated that human s brain contains around 100 billions of interconnected neurons. Neurons can also be found in human spinal cord and peripheral nervous system. There are different types of neurons in humans including sensory neurons, brain and spinal cord neurons and motor neurons. MINE 432 - Industrial Automation and Robotics 8
Neurons Neurons gather and transmit electrochemical signals. They have the same characteristics and parts as other cells, but the electrochemical aspect lets them transmit signals over long distances (up to several feet or a few meters) and pass messages to each other. MINE 432 - Industrial Automation and Robotics 9
Neurons Network Neurons in brain are connected to each other at synapses. Signals are passed from one neuron to others through axons and terminal point synapse. MINE 432 - Industrial Automation and Robotics 10
Artificial Neurons
Artificial Neurons (Perceptron) An artificial neuron as the name suggests is artificial and process its input data much similar to a biological neuron such as human brain neuron cells. It is also known as perceptron. In brief, an artificial neuron is a piece of computer code which mathematically process numerical data and produces one or more output results. A biological neuron An artificial neuron MINE 432 - Industrial Automation and Robotics 12
Input Input Data Processing by An Artificial Neuron The weights represent how information being used by the network to solve a problem. x 1 w 1 x 2 x 3 w 2 w 3 z n wi x i 1 i ; y H ( z) Output y x n-1 x n w n-1 w n Activation function Weighted average MINE 432 - Industrial Automation and Robotics 13
Activation Functions Identity f(x) = x Binary step f(x) = 1 if x >= q f(x) = 0 otherwise Sigmoid f(x) = 1 / (1 + e -sx ) MINE 432 - Industrial Automation and Robotics 14
Artificial Neural Network (ANN)
Why Networks? A single disconnected artificial neuron cannot do much to solve real life problems Similar to human brain neurons, artificial neurons must be connected to each other in order to make a powerful computational network. A human brain can learn from new experiences and after learning can apply its knowledge to solve new problems. Similarly, an artificial neural network can be trained first to learn new rules or models from pre-defined examples, then it can be employed to solve new problems. ANN can learn supervised or unsupervised. MINE 432 - Industrial Automation and Robotics 16
Artificial Neural Network An artificial neural network is built by connecting several single artificial neurons in a specific structure to process input data. There are various types of ANN. The most classic one is a Multi-Layer Perceptron (MLP) in which a large number of artificial neurons or perceptron are highly interconnected in input, hidden and output layers. MINE 432 - Industrial Automation and Robotics 17
Types of ANNs Architecture Single layer feedforward Multilayer feedforward Recurrent Learning algorithm to determine connection weights Supervised Unsupervised Reinforcement Activation Function Function to compute output signal from input signal MINE 432 - Industrial Automation and Robotics 18
Training ANNs Given a set of input data, the goal of an ANN is to compute a correct output ( outputs). The output mainly depends on the weight of connections between artificial neurons and other internal mathematical parameters. x 1 x 2 x i w 1 w 2 y 1 w i If connection weights are not chosen accurately, then the ANN output will be incorrect or inaccurate. Therefore, an ANN needs to be trained and correct weights before it can be used for reliable predictions. x n w n y 2 MINE 432 - Industrial Automation and Robotics 19
Supervised Learning Learning is performed by presenting patterns with targets During learning, produced output is compared with the desired output The difference between both output is used to modify learning weights according to the learning algorithm Recognizing hand-written digits, pattern recognition, etc. Neural network models: perceptron, feed-forward, radial basis function, support vector machine. MINE 432 - Industrial Automation and Robotics 20
Unsupervised Learning Targets are not provided Appropriate for clustering task Find similar groups of documents in the web, content addressable memory, clustering. Neural network models: Kohonen, self organizing maps, Hopfield networks. MINE 432 - Industrial Automation and Robotics 21
Training ANN using Backpropagation The backpropagation algorithm learns (computes) the weights for a multilayer network. It employs a gradient descent to attempt to minimize the squared error between the network output values and the target values for these outputs. The backpropagation tries to minimize sum of the errors over all of the network output units E(w) = ½ (t kd o kd ) 2 d D and k outputs where outputs is the set of output units in the network, and t kd and o kd are the target and output values associated with the kth output unit and training example d. MINE 432 - Industrial Automation and Robotics 22
Applications of ANNs
Problem Types Solved by ANNs when we cannot formulate an algorithmic solution. when we can get lots of examples of the behavior we require. when we need to pick out the structure from existing data. Problem types suitable for ANN application: Storing and recalling patterns Classifying patterns Mapping inputs onto outputs Grouping similar patterns Finding solutions to constrained optimization problems MINE 432 - Industrial Automation and Robotics 24
Application of ANN in Mining As a new approach to model nonlinear relationships, artificial neural networks have found many applications in mining engineering such as blasting, drilling, rock mechanics and mineral processing, etc. In mining engineering, mostly, a MLP feedforward artificial neural network trained based on error back-propagation algorithm has been used to build a black-box model of relationship between one or several output variables and several inputs. MINE 432 - Industrial Automation and Robotics 25
ANN to Predict Flotation Efficiency Labidi et al., 2007 MINE 432 - Industrial Automation and Robotics 26
Defining Inputs and Outputs The first step in applying ANN approach is problem definition in terms of input and output variables and their ranges. Then, example data sets must be prepared for ANN training. The number of example data sets depends on input vector dimension. MINE 432 - Industrial Automation and Robotics 27
Optimizing ANN Architecture The optimal topologies of the network used to model the flotation stage was determined by a number of trial by varying the number of hidden layers, the number of neurons in each hidden layer, the type of function of neuron activation, the learning rate and momentum. The best neural network formed by two hidden layers each of which is constituted by 100 neurons. The activation function for both layers that gives the best result is the sigmoidal function MINE 432 - Industrial Automation and Robotics 28
ANN-Based Simulation of Flotation Process MINE 432 - Industrial Automation and Robotics 29
Simulated vs. Experimental Results MINE 432 - Industrial Automation and Robotics 30
Applying ANN in Blasting Optimization (Open Pit Mines) Monjezi et al., 2010 MINE 432 - Industrial Automation and Robotics 31
Optimized ANN Architecture 8-3-3-2 MINE 432 - Industrial Automation and Robotics 32
ANN Predictions MINE 432 - Industrial Automation and Robotics 33
Blasting Optimization Based on ANN Simulation Fragmentation improvement: D 80 from 63 cm to 37 cm Flyrock reduction: from 110 m to 30 m MINE 432 - Industrial Automation and Robotics 34
Questions?