Genetic Robots Play Football. William Jeggo BSc Computing

Transcription

1 Genetic Robots Play Football William Jeggo BSc Computing The candidate confirms that the work submitted is their own and the appropriate credit has been given where reference has been made to the work of others. I understand that failure to attribute material which is obtained from another source may be considered as plagiarism. (Signature of student) i

2 Summary This project is an investigation into the possibilities of neural networks to be used as autonomous robot controllers, to be used on the LEGO Mindstorms platform. The aim of this project is to use neural networks and genetic algorithms to train a robot to locate a football. The minimum requirements for this project were: Build a robot for the purpose of playing football To extend the genetic algorithm and neural network built by Marston (2002) To show that a combination of a neural network and genetic algorithm can be used to train a robot to play football Demonstrate and evaluate the learnt behaviour in the robot. During the course of the project the following achievements were made A genetic algorithm was modified to be capable of evolving neural networks A neural network was developed for the LEGO Mindstorms RCX A successful neural network was evolved that could locate a football on the football pitch ii

3 Acknowledgements I would like to thank Tony Cohn for his guidance and support on this project. Thanks Netta Cohen, Jason Noble and Seth Bullock for inspiring to do this in project after their fascinating bio-inspired lectures. Thanks to Derek Magee for his useful car/blob tracker and patience helping me out when I broke the system again. iii

4 Table of contents Summary... ii Acknowledgements...iii Table of contents... iv Table of Figures... vi Chapter Introduction Chapter summary The problem LEGO Mindstorms and Robocup Junior Why not use traditional programming techniques? Artificial Neural Networks Genetic Algorithms Previous work done using bio inspired techniques for robot control Chapter The neural network Summary Types of neural network Feed forward neural networks Kohonen Networks Recurrent networks Gas Nets Choice of Neural network Chapter The genetic algorithm Summary The fitness function Choice of population size Choice of selection method Elitism Choice of reproduction Chapter iv

5 Design and Implementation Summary Design of the robot Design of the software system Design of the robot control system Design of the Genetic Algorithm Design of the vision system Design of the script Implementation of the robot control software Implementation of the genetic algorithm Implementation of the vision system Chapter Results Summary Evaluation of the neural network Emergent behaviour Evaluation of the genetic algorithm Chapter Conclusions Summary Neural nets Genetic Algorithms Onboard learning Chapter Evaluation Summary Success of the project Further requirements Further work References Appendix A Personal reflection Appendix B Code for the neural network Appendix C Code for the genetic Algorithm v

6 Table of Figures Figure 1 Differences between chess and football, source (The Robocup Federation, 1998)... 2 Figure 2 McCulloch-Pitts Neuron modified from Pudi (2003)... 5 Figure 3 - Threshold and Linear activation functions - excitation against output... 5 Figure 4 Sigmoid activation function - excitation measured against input... 7 Figure 5 - Neural network for an exclusive OR... 8 Figure 6 - An example of a feed forward network Figure 7 Example of a recurrent network Figure 8 - Topology of the neural network Figure 9 - Demonstration of single point crossover Figure 10 Position of the robots sensors Figure 11 - Waterfall development cycle, from Wideman (2004) Figure 12 - Addition of card Figure 13 First trial of the vision out put system Figure 14 Topology of the test network Figure 15 Topology of the less successful neural networks weights Figure 16 Path taken by a less successful neural network Figure 17 Path of the robot controlled by a successful neural network Figure 18 Topology of the successful neural network vi

7 Chapter 1 Introduction 1.1 Chapter summary This chapter aims to introduce the project and gives an introduction to some of the key concepts. Most importantly this chapter demonstrates the problems of programming robots using conventional techniques and introduces the basics of neural networks for robot control and genetic algorithms as a training method for neural networks. 1.2 The problem Autonomous robotics is an area of computer science that has been explored for many years. One of the key issues is that it is very difficult to program a robot using traditional programming techniques as input from the sensors on the robot have a tendency to be noisy, which makes it difficult to apply set rules. Artificial intelligence is one of the most popular options when it comes to programming autonomous robots, with techniques such as neural networks and genetic algorithms often being used to design the controls for these robots. As a competitive challenge to the research community, Robocup was founded, to come up with novel approaches to solving many of the engineering and programming difficulties in this area. Robocup is a football tournament that is based on autonomous robotics with the aim of fielding a team of humanoid robots that can beat the human soccer world cup champions by the year 2050 (The Robocup Federation, 1998). Robocup has spawned Robocup Junior(, a league that is aimed at young people who are interested in robotics and based on the low cost introduction kit to robotics made by LEGO. The aim of this project is to design use a neural network and genetic algorithm that can locate a football on a football pitch as this is part of the challenge of creating an autonomous football player. 1

8 1.3 LEGO Mindstorms and Robocup Junior Since computers were invented research has been carried out in to how to make them behave in a more human manner. This has lead to computers being built to try and beat humans at games such as chess. This attempt was successful, but the method employed by the computer involved looking at billions of possible combinations of move per second (IBM). This does not teach us much about they way that human minds work. The Robocup Federation was founded as it was thought that the game of football would be more suitable for this type of research. The differences between chess and football are summarised by the following table: Chess Robocup Environment Static Dynamic State change Turn taking Real time Information accessibility Complete Incomplete Sensor readings Symbolic Non-symbolic Control Central Distributed Figure 1 Differences between chess and football, source (The Robocup Federation, 1998) This shows that the challenges posed by Robocup are more similar to those that are encountered by humans, as these generally do not involve such static rules as chess. Robocup has different categories of competition, there are four different robot leagues and there is also a simulation league. One other development of Robocup is the competition is Robocup Junior. Robocup Junior is a league of the Robocup competition aimed at getting young people involved in designing and programming robots (The official Robocup Junior web page). In the Robocup junior competition the robots are all built using the LEGO Mindstorms platform. The LEGO Mindstorms platform is a set of components based on normal LEGO that include a microprocessor block called the RCX, a number of different sensors and motors. These allow a low cost introduction to robotics and are ideal for Robocup Junior. 2

9 The RCX comes with a graphical programming environment based on LEGO building blocks (LEGO Mindstorms homepage, 2004). However this approach is limited as it only allows certain operations and the flexibility of a true programming language is needed to implement more advanced techniques on the robot. There are several different approaches and programming that can be used to program the robot. All of these approaches are reverse engineered firmware for the RCX and allow other programming languages to be run. BrickOS This is an alternative firmware for the RCX that allows code to be written for robot control in C or C++ (sourceforge.net, 2004) LeJOS This is an alternative firmware for the RCX that allows Java to run on the RCX (sourceforge.net, 2004) pbforth Runs the language Forth on the RCX. Forth is an interactive language allowing command line control of the RCX. However this functionality is not needed for this project (Baum, 2000) NQC This is the original hack of the RCX and is a programming language specifically for Mindstorms. NQC was chosen for this project as it has enough flexibility and the most extensive documentation (Baum, 2003 & sourceforge.net) 1.4 Why not use traditional programming techniques? Computers are very limited in some ways. They can be programmed to perform many tasks which follow a set of instructions quickly and efficiently, and this computational power is doubling approximately 18 months. The tasks that computers are not good at, such as facial recognition, are problems where it is difficult to explicitly program rules to cover the many possibilities presented, in this example the effects of different angles, distorted pictures, lighting and so forth. Humans however are good at these tasks and perform them easily and repeatedly. Brooks (1991) classifies this as a loose interpretation of intelligence. Artificial intelligence techniques have been used to control robots with some success. 3

10 Artificial intelligence is a field that has been researched almost since computers were invented. Early artificial intelligence techniques were design to model the entire brain at abstract tasks. Brooks (1991b) argues that these artificial intelligence tasks can solve simple problems, but only when they have been abstracted by the computer operators and that embodiment is a crucial factor in creating intelligent programs. Abstractions are not useful for robot control, as robots have sensors that require interpretation of real world data. More recently research has been carried out into techniques that model smaller parts of the brains circuitry. Techniques that use artificial neural networks have been successful at controlling robots with many studies based on this approach. 1.5 Artificial Neural Networks The idea of the artificial neural network is based on biological neural networks. Biological neural networks are made up of elements called neurons. For the sake of simplicity throughout this report any reference to neural networks or neurons is, unless explicitly stated, intended to refer to artificial neurons or artificial neural networks. The artificial neuron was formulated by McCulloch and Pitts (1943), and is based on biological neurons. A biological neuron has inputs and outputs that travel from the input to the output by electro-chemical signals. These biological neurons are linked to each other and pass messages between connections called synapses. If a biological neuron receives enough input then this will cause it to fire, send an electro-chemical signal to, its outputs. The stronger the connection between neurons at their synapses, the more likely the neuron is to fire. McCulloch and Pitts took this idea and modelled it. 4

11 Figure 2 McCulloch-Pitts Neuron modified from Pudi (2003) The artificial neuron receives a set of inputs, x i, which represent inputs to the biological neuron, and are represented by a number. Each input has an associated weight, w i, that represents the strength of the connection between the neuron and its input. The neuron then converts the inputs into a total input, which is usually the sum of the weighted inputs. This is compared against an activation function. Whether the neuron will fire, and the strength of the output is dependant on the result of the activation function. There are three different types of activation function; the simplest of these is the threshold activation function. Here if the total input exceeds a threshold then the neuron will fire and hence is a binary output. The second activation function is the linear activation function. With this function the output is proportional to the sum of the weighted inputs Threshold activation function Linear activation function Figure 3 - Threshold and Linear activation functions - excitation against output 5

12 6

13 The final type of activation function is the sigmoid function where the output is mapped onto the sigmoid function that maps it to a value between 0 and Sigmoid activation function Figure 4 Sigmoid activation function - excitation measured against input The benefit of using the sigmoid or linear activation functions is that it gives the outputs more information. The sigmoid function is essential for some training methods, such as back-propagation. Individual neurons can be used to solve certain linear problems such as a boolean AND. This would done with a simple threshold where w 1, w 2... w n = 1 and using a threshold activation function where the threshold n. The problem with single neurons is their inability to solving problems that are not linearly separable. This means that for problems such as the exclusive or cannot be determined by a single neuron (Bharath et al, 1994). To make more complex decisions it is necessary to include more neurons to make up a network. A small network of 3 neurons can solve the exclusive or problem where one would fail (see figure 5). 7

14 Figure 5 - Neural network for an exclusive OR With this example it is easily possible to mentally calculate the required weights required for the correct neural network, but what happens when there are many more inputs and outputs? For neural networks to run effectively there needs to be a method of determining the weights and required thresholds. There are several methods of training a network, the first of these is a biological paradigm based on the work of Donald Hebb (1949). The consequent learning rule states that if one neuron repeatedly fires to a connecting neuron then the efficiency of the connection will increase. Hebbian learning therefore concludes that by increasing the weight of the input if at the same time the output is active (Picton, 1994). This can be extended so that if the neuron does not fire then the weight of the connection will be inhibited, which leads to more accurate results. However, Hebbian learning takes no account of the actual value of the output, only the desired value. If the weights are adjusted by an amount that is dependant on the difference between the actual and desired amounts, the limitation of Hebbian learning can be overcome. This is known as the delta rule. Another method that has been used to train neural networks, and for other function, are genetic algorithms. 8

15 1.6 Genetic Algorithms Genetic Algorithms are another biologically inspired method of computation. The idea of the genetic algorithm was inspired by Darwin s (1906) theory of evolution. This famous theory declares that different species of animals adapt to their habitats over time by means of natural selection and mutation. This process can be broken down into four essential processes, self-reproduction, mutation, competition and natural selection. Self reproduction is obvious among any living organism and leads to heredity, parents giving their offspring certain characteristics. All living organisms are also affected by mutation, the random change of some of their genes over time. Natural selection is where the members of a population most suited to their environment are likely to survive and hence reproduce. Finally competition is guaranteed when a population is constrained by factors such as limited resources or predators, again ensuring the fittest survive. Evolution is the result of these processes on a population over time (Huxley, 1963). This idea was used by Holland (1975) to create an algorithm to explore search spaces based on this architecture. The search space is modelled using a chromosome which can be a string of bits, numbers or almost any quantitative set of values. Then a fitness function is chosen, which is used to evaluate the genes so that the chromosome that scores highest is the most likely to reproduce. Then a selection procedure takes place, there are several methods of selecting which chromosomes will go through to the next iteration. The selection procedure will be discussed in more depth in chapter 3. Then the successful members reproduce to give children this can be either sexual or asexual. The other factor that genetic algorithms use is mutation. Without mutation the genetic algorithm would be able to shuffle around the original chromosomes, but not produce any new ones. Mutation is a random even and is usually implemented as a random chance that at any point one of the genes in a chromosome may change. In a bit string this would be implemented as a bit flipping. The major advantage of a genetic algorithm is the way that it can search though a large search space and return good results. This is a key problem for the control 9

16 system such as a neural network, for an autonomous robot. There is typically a large search space for such problems ideally suited for genetic algorithms. 1.7 Previous work done using bio inspired techniques for robot control There has been a great deal of research into using bio inspired techniques for robot control. What follows is a short selection of some of the more relevant pieces. Randall Beer is one of the leading figures in the field of evolutionary agent control. He has produced a dynamical systems perspective for the explanation of and design of autonomous systems (1995). In this paper, Beer used a neural network and genetic algorithm to train a hexapod robot to move. He also showed that if sensory data was lost then the neural network degraded gracefully and could still function. Floreano and Mondada (1996) designed two experiments for mobile robots with onboard neural networks and trained by genetic algorithms. The reason for choosing to have on board training was their belief that, we think that computer simulations of robots can hardly capture the complexity of the interaction between a real robot and a physical environment where mechanical and physical laws, (such as wearing of the components, changing light conditions, friction, etc.), non-white noise at all levels and various types of malfunctioning play a major role). They followed up this work with another experiment designed to see if onboard learning was valuable (1996b), which they found to be true but time consuming. Lastly work was carried out on evolving behaviours for the LEGO Mindstorms platform by Richard Marston (2002) for his undergraduate dissertation. Marston designed a neural network that was trained to follow a black line. This report is based quite strongly on this project as there are many similarities between the two projects. 10

17 Chapter 2 The neural network 2.1 Summary This chapter introduces the reader to some of the different types of neural network and discusses their advantages and disadvantages. Then the choice of the neural network is described along with its topology. 2.2 Types of neural network Feed forward neural networks Feed forward neural networks have a network where the information flow is unidirectional. The neurons are arranged in layers with a series of input nodes, and then subsequent layers of neurons. Figure 6 - An example of a feed forward network One of the methods for training a feed forward neural network is back propagation. Back propagation is based on the delta rule. The weights of the last layer of the 11

18 network are modified by the error between the desired output and the actual output. For a multi-layer network though, the weights must be modified for the output at the last layer to the second from last layer, then to the previous layer until the front is reached. This is a very useful training set for problems such as image recognition, however when it comes to robot control it is not applicable. For robot control there is no guaranteed training set as robots exist in the real world, where input tends to be noisy Kohonen Networks A Kohonen network usually consists of a layer of input nodes that are fully connected to a two dimensional array of neurons. The neurons in this fully connected array will have similar weights to the neurons that are surrounding them. For each element in the training set one of the neurons in the fully connected array will have the closest set of weights to the desired weights. This neuron is the winner for that element and as a result will have its weight strengthened so that next time it sees that pattern it will react even more strongly. This process is repeated through the training set until some acceptable level of error is reached. Kohonen networks are largely used for pattern classification, although they have been used in robots as a means of route planning (Nehmzow et al, 1991) but not for robot control. As such we will not discuss them further here Recurrent networks The novel idea of recurrent neural networks is that unlike Kohonen or Feed forward networks the information does not flow from one end of the network to the other. Nodes in the network may be connected to other nodes anywhere in the network, and fully connected architectures, where every node is connected to every other node, including itself (see figure 7). This is closer to the biological paradigm of biological neurons, which are vastly interconnected. Recurrent networks are therefore more natural in the way they record data. 12

19 Figure 7 Example of a recurrent network From this figure it is obvious that time is going to be an additional factor, as otherwise the neurons could continuously cause themselves to fire, leading to an infinite loop. Hopfield (1982) postulated that each input should evaluate itself randomly in time, but with an average of how often the inputs should be evaluated. One of the advantages of using recurrent networks is their ability to store associative memories. Hopfield (1982) was designing the system to retrieve patterns, even if the pattern was corrupted or only given part of the pattern. He theorised that this is analogous to the way that organisms recall information by what he coined as content addressable memory. When a Hebbian network is presented with an input pattern the network does not immediately produce an output but after repeated time steps will converge on a stable pattern. This is where the neural network settles to a state where no further changes are taking place. Recurrent neural networks have been used for autonomous robot control by several researchers. Beer (1995) used these networks to evolve a hexapod robot; they were well designed for this task due to the content addressable memory that allows them to store the previous state of the network. 13

20 2.2.4 Gas Nets A new development in the field of neural networks was pioneered by Phil Husbands (1998) et al in the paper, Better living through Chemistry. Gas nets are a further step towards biological realism. McCulloch-Pitts neurons model the connections of biological neuron s synapses but do not model any other aspects of the brains chemistry. Husbands et al modelled the movement of gas through the brain, particularly Nitrous Oxide. Nitrous oxide diffuses through the brain and communicates between biological neurons that are physically close to each other instead of through synaptic connections. In biological systems the plasticity of the neurons is affected by the gas, in the presence of nitrous oxide neurons are more likely to form new connections with other neurons, and less likely if there is little nitrous oxide present. Husbands et al modelled this by mapping the neural network on a two dimensional plane and some of the nodes of this network can emit gases which modulate the behaviour of neurons that gain a large enough concentration of the gas. The interest in this paper is that the task that Husbands et al evaluated their neural network by measuring how quickly an evolved gas net compared to similar non gas nets. The results were that the gas nets performed significantly better in all of these tests. 2.3 Choice of Neural network Looking at the choices of neural networks it is obvious that there are several possibilities to consider. First it is important to note that using neural networks for robot control is a well established field and the successes of this approach have been used by many researchers. When considering the type of architecture for the neural network the most important factor to consider was number of connections that will be required for the network. Genetic algorithm will be training the neural network on board the robot and this process is time consuming (see following chapter), therefore it is important to keep the neural network as small as possible. 14

21 Out of the four types of neural network described above the one with the possibility of the smallest is a single layer feed forward network. The architecture is shown in figure 8. The second reason for this choice is that it is based on the neural network of Marston (2002). As one of the stated minimum requirements for this project was to extend the neural network and genetic algorithm designed by Marston, it was felt the neural network that was designed for tracking a dark line could be modified to get the robot to locate a ball instead. Again looking at the work of Marston it was decided to implement the activation function as a threshold as his experience showed that this was the only realistic way of activating the robots motors. Figure 8 - Topology of the neural network 15

22 Chapter 3 The genetic algorithm 3.1 Summary This chapter covers the choices made for the implementation of the genetic algorithm in this project and some of the difficulties that were faced. 3.2 The fitness function When designing a genetic algorithm, one of the most difficult aspects to program correctly is the fitness function. A poorly chosen fitness function will lead to learned behaviour in a robot controller, but it may not be the learned behaviour that is aimed for. Marston (2002) had many problems with his fitness function; the first fitness function that he used included a measure to reward the robot for the distance that it travelled as well as proximity to the line that the robot is designed to track. The movement factor in the genetic algorithm was included to ensure that the robot would not find the line and remain stationary until the trial period ended. This caused problems as the most successful robots were the ones that travelled fastest over the black lines in their environment. Marston then modified the genetic algorithm so that they would only be rewarded if they were in close proximity to the line but this also was not successful. When the reward for movement was removed the genetic algorithm trained the neural network correctly. With this in mind the fitness function for this projects genetic algorithm was chosen. The original fitness function was chosen as the amount of time in each trial the robot would be facing the ball. To implement this some method of locating the football and robot was needed so that a quantitative measure could be used in the genetic algorithm. A vision system was used using the car/blob tracker developed by Derek Magee (2001). This system works through detecting the changes in the background of an image to discover the new location of a blob. Unfortunately it was not possible to modify this software so that neither the ball nor robot could be tracked by this 16

23 system. After making the robot identifiable to the system with coloured markers it was possible for the vision system to identify the robot and direction that the robot was facing. This lead to a fitness function for the robot being chosen as 180 minus the angle between the robot and the football at the end of each trial. To overcome problems of the robot facing the ball at the start of each trial the robot started facing away from the ball at a randomly chosen angle greater than 90 degrees and less than Choice of population size For this experiment a small population was used at just 20 members. This decision was based on Marston s (2002) decision to use a small population as this seemed stable in his experiments. 3.4 Choice of selection method There are many types of selection method for genetic algorithms. These are normally based on the following three types: Rank based selection The fittest members of the population are selected to reproduce. Tournament selection Groups of randomly chosen members of the population are compared and the member with the highest fitness is chosen to reproduce. Roulette wheel selection Each member of the population has its fitness calculated as a proportion of group fitness. This is used as a probability that a member will be selected for reproduction. Again looking at the work of Marston (2002), even though it seems that rank based selection is the optimal method for putting evolutionary on the population this can lead to problems of epistasis. This is where the genetic algorithm converges on a 17

24 solution which is not the optimal solution. To counter this roulette wheel selection was used, based on Floreano and Mondada (1996b). A small population was decided for this network as it was felt that the amount of diversity present in the population would be fairly small, at just 20 members. The problem with having a small population is that it is possible for one member to dominate the scheme. To counter this it was decided that there would be a high rate of mutation, with 0.5 chance of any one member of the population having part of its chromosome mutated, by a random value of up to 10%. This concurs with Marston (2002), who also implemented a small population and high rate of mutation. 3.5 Elitism One other method that can choose members to go through to the next generation is elitism. This is where the fittest member of the population is automatically copied to the next round. This is useful for ensuring that the fittest member of the population automatically gets through to the next generation. It was decided that as the selection method was based on roulette selection that there was a possibility that a successful neural network would be evolved and this would be missed in the selection procedure. Therefore based on this it was decided to implement elitism for this population. 18

25 3.6 Choice of reproduction There are two possible methods of reproduction either sexual or asexual. Asexual reproduction ensures that an exact copy of the winners of the selection method get mutated and go through to the next round of the genetic algorithm. The disadvantage of this is that mutations are not shared through the population. To overcome this it was decided to use sexual reproduction. With sexual reproduction there is a possibility that the parent will have an exchange of part of its genome with another successful member. Figure 9 - Demonstration of single point crossover This was implemented as a crossover rate of 0.5, meaning that there is an even chance of any of the population being crossed with another member of the population. 19

26 Chapter 4 Design and Implementation 4.1 Summary This chapter covers the design and implementation aspects of this project. Many of the design and implementation issues are covered by the design of the neural network and genetic algorithm which have been covered previously (see chapters 3 and 4). There is also a short section discussing the design of the robot to allow the reader some reference when it comes to the design of the control architecture for the robot. 4.2 Design of the robot The RCX is the largest brick that has to be accommodated in the building of the robot, and has to be fairly central as the range of the cables to connect the sensors is limited. It was decided to give the robot differential steering (each of the powered wheels is connected to a different motor) as the methods for non-differential steering would still require two motors and be beyond the level of mechanical engineering standards required for this project. The powered wheels are set at the rear of the robot. 20

27 Figure 10 Position of the robots sensors Also included on the robot were two light sensors, one on either side of the robot, to allow it to have a measure of stereo vision. A rotation sensor was included so that the robot would be able to determine how far it had rotated; enabling the robot to start each rotated by a random amount. Finally, a touch sensor was included at the front of the robot to allow it to recognise if it ran into a wall during a trial. 4.3 Design of the software system One of the key problems for this system is that several different elements need to communicate. To overcome this it was decided to use the waterfall development cycle. This approach to software engineering is iterative, so the software works from the top to the bottom of figure 10 with feedback as need be. 21

28 Figure 11 - Waterfall development cycle, from Wideman (2004) Using this structure it is obvious that the first thing that needs to be defined are the system requirements. In this case it is defined by the minimum requirements of this project, of extending the work of Marston (2002) and training a robot to locate a football. Looking at what is needed for this project the software requirements for the final system are: The robots control system, including the neural network The genetic algorithm A vision system for the genetic algorithms fitness function. A system to connect these three systems so that learning can take place automatically. Looking at each of these in turn, the initial design and following implementation will be considered. 22

29 4.4 Design of the robot control system As covered in chapter 3 the initial design of the neural network is covered by the number of neurons that are required for the network to run. These were designed as software on the RCX with the neurons accepting input from the light sensors and were based on the McCulloch-Pitts neuron. Other parts of the control system that are needed for this system are summarised here: A spin function to ensure that the robot is facing away from the ball when a trial is started. Some process so that the trial will automatically end if the robot hits something, preventing damage to the robot. A restart function that will receive messages so that the robot can be returned to the start position. A method that will allow the neuron weights automatically so that the robot requires no human intervention. 4.5 Design of the Genetic Algorithm In the minimum requirements for this project one of the requirements is to extend the genetic algorithm and neural network designed by Richard Marston (2002). After consideration of his design it was decided that this would not be carried out. Marston s design is based on the Bio-inspired Evolutionary Agent Simulation Toolkit (BEAST, 2004) developed by the Informatics Research unit at Leeds university. The reason for this is that an alternative was found that was believed to be easier to implement. The code is based on the work of Ladd (1996) in his book, Genetic algorithms in C++ and only minor modification of the algorithm was required. The modification that was made was to change the mutation function from acting on an integer to an integer array. 23

30 4.6 Design of the vision system For the robot to function autonomously it was decided that a vision system should be implemented. The vision system was an extension of the Magee s (2001) car/blob tracker. This works by identifying the background of a video stream and then tracking the changes to the background image. This is stored as an RGB array of pixels. The car/blob tracker records this and outputs the width, height and origin of the blob as well as an identifying label. For this project what was needed was to identify the robot and the direction that it is facing. It was decided to extend the vision system by identifying the robot as a blob and then analyse the rectangle based on the width and height of the blob to determine the direction of the robot. For this to be possible the robot needed identifying features that would allow the vision system to identify its direction. To allow this a piece of red card was attached to the top of the robot at the front and a piece of green card attached at the back. Figure 12 - Addition of card This would allow the vision system to perform a raster scan on an image of dimensions width x height and identify areas of solid red or green. The design for identifying the ball was also based on this idea. As the ball is clear plastic with a green circuit board inside, the vision system could also use the same technique to identify the ball. The vision system will then output the co-ordinates of the ball and the co-ordinates and direction vector of the robot. 24

31 4.7 Design of the script Each of the above systems is to be implemented as separate files. Due to this a small script is needed to co-ordinate the above files so that they can interact. To send messages to the RCX the only method is by using the NQC command msg, which is only accessible from the command line. The decision was made to implement this in python as this is the scripting language that I have most familiarity with. The python script will handle: Recording the fitness from the vision system Sending messages to the RCX Activating the genetic algorithm Changing the output file for each generation 4.8 Implementation of the robot control software The neural network was implemented as software on the RCX. This is not as physically robust as a hardware implementation would be, but as the robot will not be in any hazardous situations this is not expected to be critical. The neurons were implemented as an array of integers, as the RCX cannot perform floating point operations. The array represents the weights for the neurons and the threshold for the activation function. These were to be sent to the RCX as infra red messages and due to the limitations of NQC, only values between 1 and 255 can be transmitted. Therefore the decision was made to include a constant factor to be multiplied against the threshold because otherwise the products of the weighted inputs had the potential to be an order of magnitude greater than the threshold. This would mean that if the sum of the weights for the inputs were greater than 5 (approximately), given light conditions, then the neuron would always fire. This was considered to unacceptably limit the amount of variation in the chromosome. A factor of 58 was chosen as after a little experimentation this gave a reasonable set of results. 25

32 The spin function was implemented using the robots rotation sensor. By testing the rotation sensor it was calculated that for the robot to complete a 360º rotation, the sensor would increase by 120. Therefore a random number was picked using a random function in the range 0 to 60 to determine the random angle and the robot was programmed will turn by 90º automatically. Also a simple NQC event was added to interrupt the program loop if the bump sensor was triggered. The restart function was not implemented due to problems with the vision system, see section Implementation of the genetic algorithm With the genetic algorithm there was not a great deal of additional coding needed. The changes made to the original file were to add some file input and output so that the process could be automated as fully as possible and the addition of a new mutation function as the original function was designed to work on a bit string, not an array of integers. For the initial population of the genetic algorithm a random set was created using a small python script to generate random as the settings for the neural network. Each of the neural networks was tested for 5 seconds as this was enough time for the robot to turn around and then move to the other end of the pitch Implementation of the vision system The vision system has two main tasks identifying the robot and identifying the ball. For the vision system to identify the robot the vision system the labels of new results were read into an array of values that contained the origin of the blob, its width and height. The idea for this is that this should contain the entire robot and be able to obtain the cluster of red pixels and the cluster of green pixels. Some trials were performed whilst the robot was stationary on the pitch and the RGB values observed. The RGB values were moderated with boolean logic as working on the red values alone was not sufficient due to the white pitch. Therefore only red values were picked if they were significantly higher than the green and blue values. 26

33 This approach was unsuccessful however as the RCX has a red button located on it, and some seemingly random pixels on the pitch were recognised by the vision system as being red (see figure 13). This had the effect of making the origin coordinates for the red card on the robot completely unreliable. Figure 13 First trial of the vision out put system To overcome this if a pixel was almost or completely surrounded by red pixels and was red itself then it would be counted towards the calculations for the centre of the red card. This process was applied to identification of the green card as well and after adjusting the logic to identify the green was successful. However these tests were performed when the robot was stationary. When the robot was in motion a new set of problems occurred. When the robot moved the tracking system did not update so that the entire robot was till captured by the surrounding frame. This lead to problems as it meant that for the majority of the trial, when the robot was in motion one of the pieces of coloured card was not recognised and so the direction of the robot could not be ascertained. In an attempt to overcome this problem the size of the frame was increased, first by forty pixels in both dimensions, then by eighty. However this did not succeed in tracking the robot and the computer upon which the vision system was running appeared to slow. As an assumption I would assume that this is due to a number of factors, mostly due to hardware limitations of the system on which the vision software ran. Manipulating visual data is notoriously expensive in computer time so iterating through the frame for each object to try and identify the robot uses yet more resources. Occasionally the vision system would pick up a new object when there was no new object or because a shadow passed over 27

34 the pitch. Each one of these objects would then have a frame around it and be tested to see if it was the robot. Tracking the ball proved to be equally problematic. Based on the same idea of using a frame surrounding the object it was attempted to identify the characteristics of the ball. Unfortunately the ball could not be classified in the same way. The colour properties that the ball showed vary depending on which way up the ball was. The ball has two circuit boards in the shape of a cross internally and a clear plastic cover meaning that the ball can either be a dark green or almost entirely transparent. The plastic coating for the ball is also highly reflective, making classification harder. To overcome the problem of tracking the ball it was decided that the ball should be fixed on the pitch so that an angle could be determined by the vision system at the end of each trial. 28

35 Chapter 5 Results 5.1 Summary This chapter gives the results of the neural network and how effective they have been at training the robot to locate the football. The genetic algorithm is then evaluated with respect to its rate of convergence to the stated goal. 5.2 Evaluation of the neural network To evaluate the success of the neural network some objective measure is needed. To test this, a simple neural network was designed by hand that turned to face the ball. Figure 14 Topology of the test network This will be tested using the robot control system so bear in mind that a constant factor of 58 will be multiplied to the thresholds before they are compared to the weighted sum. The logic behind this network is simple, as there are four possible states that the network can be in (approximate values for the light sensor readings are given): 29

36 Left Right Sum of Sum of Left neuron Right light sensor light sensor weighted inputs for left weighted inputs for right activated? neuron activated? neuron neuron No Yes No Yes No Yes No No It should be noted that this is a linearly separable task and can be accomplished by the right neuron alone. When this system was tested in the same way as the trials had been conducted for the population in the genetic algorithm the system was very successful, and located the ball every time. If the evolved neural network is as effective as the neural network above in locating the football then it will be considered a success. 5.3 Emergent behaviour The genetic algorithm produced a few notable behaviours in the neural networks. Initially as the weights set were random the behaviour exhibited by the robots was not very refined. All of the networks followed one of three distinct patterns, they either remained stationary, moved in a circle or moved in a straight line into the nearest wall. This behaviour can be accounted for as the different types of output from the neural network. In the sixteenth generation however a new behaviour emerged for the neural network. The robot headed straight for the wall but when it was reached the robot turned to face away from it (see figure 15). This was an encouraging sign as the walls surrounding the pitch are a matt black showing that the genetic algorithm was producing new output that showed sensitivity to the light sensors. 30

37 Figure 15 Topology of the less successful neural networks weights With this network the normal range of values is different from the trial neural network. Instead of the values being the ambient light when the robot gets close to the wall the values for the light sensors drop dramatically, from the normal average of around 40, to the upper 20s. This means that this network favours the left neuron (the bottom node in figure 16). The left neuron is firing continuously setting off the left motor. The right neuron is firing until the level of light reaching the light sensors drops to the level at which the cut off point is reached when the neuron stops powering the motor and the robot turns to follow the wall. This behaviour was is very similar to the behaviour that is desired in the successful neural network. The neural network moves the robot towards a light, but unfortunately it is not the higher level of light that is required to reach the football. The fitness function does not explicitly reward this behaviour. Although it is desirable behaviour these robots did not finish the trial facing the football. Although they normally followed the path shown in figure 16, which normally led them to a point where the angle between the robot and the ball was around 45º, some of the other robots had fortunate results and this behaviour was never in evidence throughout the population. Even if the original fitness function had been implemented then these neural networks would not have been rewarded as they did not spend time continuously facing the ball. 31

38 Figure 16 Path taken by a less successful neural network No other new behaviours emerged from the genetic algorithm until generation 55 when a robot almost successfully tracked the ball. The robot performed a turn and when it faced the ball it moved forwards. However this movement was not a straight line but an arc. After the robot had progressed along this arc until the both light sensors were no longer picking up input from the football, the robot stopped the arc and turned again until it was around 25º from the ball. Unfortunately a robot that was moving in circles finished its test facing the ball and so this behaviour was not preserved by the elitism function of the genetic algorithm. The robot that performed this test was one of the more successful amongst its generation and just 5 generations later this behaviour appeared again. With this trial the robot was successful and finished the trial touching the ball (see figure 17). 32

39 Figure 17 Path of the robot controlled by a successful neural network Figure 18 Topology of the successful neural network This neural network architecture is effective in locating the football as it strongly favours the right neuron (the top neuron in figure 18). This means that for this neuron no matter what the input the right neuron will fire. The left neuron is much weaker here and will only fire if both the light sensors are facing the ball. This means that the robot turns to face the ball and then heads towards it. 33

40 The arc is the only reason why this is not the optimal neural network. The reason for the arc was believed to be the rotation sensor. This sensor runs off gearing that is connected to the left wheel and due to the nature of the sensor, it takes away a little power from the wheel. This means that if the robot has equal power going to both wheels then the motion is still in a slight arc. To test this, the motion sensor was removed and the spin function commented out. The robot was then placed in the start position and a trial was run. In this trial run the robot did not arc as much as in its previous trial. When comparing the behaviour of the two robots, there is one obvious difference. The trial robot was designed so that its function was to turn towards the football, which it succeeded at. The behaviour of the evolved robot was different however. The control structure that it used included motion towards the football as well as identifying which direction the ball was in. 5.4 Evaluation of the genetic algorithm As this much of this project has been based on the work of Richard Marston (2002) it was felt to be appropriate to contrast the results of the genetic algorithm with Marston s. There were a number of differences between the tests and some differences in the implementation of the genetic algorithms. The selection method used was different as Richard Marston (2002) used tournament selection and the implementation for this project was roulette wheel. Also employed was elitism which combined with the roulette wheel selection seems to have lead to a degree of epistasis, the favouring of certain chromosomes to the detriment of the amount of genetic variation within the population. This is a possible reason why it took around twice as many generations for the genetic algorithm to converge. However the most likely reason for faster convergence in Marston s (2002) project is the implementation of the fitness function. Marston s fitness function was based on each chromosome having ten runs whereas for this project it was only possible to perform a single run for each chromosome. 34