Application of Neural Q-Learning Controllers on the Khepera II via Webots Software

Transcription

1 Inernaional Conference on Fascinaing Advancemen in Mechanical Engineering (FAME2008), 11-13, December 2008 Applicaion of Neural Q-Learning s on he Khepera II via Webos Sofware Velappa Ganapahy and Wen Lik Dennis Lui Absrac In recen years, here has been an increasing amoun of research performed in he area of mobile roboics. As such, numerous sraegies had been proposed o incorporae various fundamenal navigaion behaviors such as obsacle avoidance, wall following and pah planning o mobile robos. These conrollers were developed using differen mehods and echniques which range from he radiional logic conrollers o neural conrollers. Logic conrollers require he programmer o specify he required acion for all saes of he mobile robo and neural neworks, such as he popular backpropagaion feed-forward neural nework, will require he presence of a eacher during learning. To achieve a fully auonomous mobile robo, he robo should be incorporaed wih he abiliy o learn from is own experience. Supervised learning is an imporan kind of learning. However, i alone would no be sufficien for learning from ineracion. To enable, he mobile robo o learn hrough ineracion wih he environmen, he reinforcemen learning algorihms are invesigaed. In his paper, he Neural Q-Learning algorihm was implemened on he Khepera II via he Webos sofware. The designed conrollers include boh sensor and vision based conrollers. These conrollers are capable of exhibiing obsacle avoidance and wall following behaviors. In addiion, an obsacle avoidance conroller which is based on a combinaion of sensor and visual inpus via fuzzy logic was proposed. Keywords Reinforcemen Learning, Neural Q- Learning, Fuzzy Logic, Obsacle Avoidance, Wall Following, Khepera II, Webos. A I. INTRODUCTION S defined by Suon, R. S. and Baro, A. G. [1], reinforcemen learning is learning wha o do - so as o maximize he numerical signal reward. The learner is no old which acions o ake, as in mos forms of machine learning, bu Velappa Ganapahy is wih he School of Engineering, Monash Universiy Sunway Campus, Jalan Lagoon Selaon, Banday Sunway (phone: ; fax: ; velappa.ganapahy@eng.monash.edu.my). Wen Lik Dennis Lui., is wih he School of Engineering, Monash Universiy Clayon Campus, 3168 VIC Ausralia. ( dwllui@gmail.com). insead mus discover which acions yields he mos reward by rying hem. Figure 1 shows he agen-environmen ineracion in reinforcemen learning [1]. The Q-Learning algorihm proposed by Wakins, C. and Dayan, P. [2] is a widely used implemenaion of reinforcemen learning based on dynamic programming echnique and emporal difference mehods [1]. The algorihm esimaes he expeced discouned numerical signal reward Q(s,a) by aking acion a a sae s. This will bring he robo o he nex sae, s. The algorihm furher esimaes he numerical signal for he nex sae Q(s,a ) assuming acion a is aken in sae s. Then, using he resuls of each acion, i updaes he Q-values of he Q-able according o he following equaion, ( s, a) Q( s, a) + [ r + γ max ' Q( s', a' ) Q( s a) ] Q α a, (1) where α is he learning rae, γ is he discoun rae and r is he immediae reinforcemen. The naure of he Q-learning algorihm owards problems wih a discree se of saes and acions make i very suiable for he developmen of mobile robo navigaion behavior such as obsacle avoidance and wall following. reward r sae s r +1 s +1 Agen Environmen acion a Fig. 1 The Agen-Environmen Ineracion in Reinforcemen Learning However, is original sandard abular formaion used o hold Q-values would no yield an efficien sysem. For insance, a robo wih eigh sensors which has an inpu range of for each sensor wih five acions o choose from will require a ( x ) x 5 Q-able. To

2 Deparmen of Mechanical Engineering, Mepco Schlenk Engineering College, Sivakasi, India efficienly updae and read he Q-values from such a large able would impose a serious problem. The only was o learn anyhing a all on hese asks is o generalize from previously experienced saes o ones ha have never been seen. As such, he sandard abular formaion had been replaced by funcion approximaors such as neural neworks. II. RELATED WORKS Apar from applying neural neworks o Q- learning, i has also been applied ono acor-criic archiecures. The ypes of neural nework uilized for reinforcemen learning algorihms are he backpropagaion feed-forward neural nework, recurren neural nework and he self organizing maps. Jakša, R. e al [3], Yang, G.-S. e al [4] and Huang, B.-Q. e al [5] had similarly approached he problem of mobile robo navigaion hrough he combinaion of he mulilayer feed-forward neural nework wih reinforcemen learning algorihms. Jakša, R. e al [3] had uilized an acor criic archiecure whereas he laer wo had uilized he Q-Learning algorihm. All resuls obained are verified via simulaion resuls only. In addiion, he developed reinforcemen learning conrollers are enirely based on he mobile robos sensors. The main difference beween he recurren neural neworks and he backpropagaion feedforward neural neworks is heir inernal srucure. The inpu layer of he recurren neural nework is divided ino wo pars; he rue inpu unis and he conex unis. The conex unis simply hold a copy of he acivaions of he hidden unis from he previous ime seps. In 1998, Ona, A. e al [6] had horoughly discussed he archiecure, learning algorihms and inernal represenaion of recurren neural neworks for reinforcemen learning and had performed comparisons across he differen ypes of nework archiecures and learning algorihms hrough a simple problem. A he same ime, Cervera, E. and del Pobil, A.P. [7] had no only applied he recurren neural nework for a sensor-based goal finding ask, bu he duo exended i by proposing a new mehod for sae idenificaion ha eliminaes sensor ambiguiies. The implemenaion of self organizing maps for Q-learning was furher illusraed by Sehad, S. and Touze, C. [8]. This nework learns wihou he requiremens of supervision and i could be able o deec irregulariies and correlaions in he inpu, and adap o ha accordingly. The pair had used he self organizing maps ogeher wih Q- learning o develop an obsacle avoidance behavior for he robo and is inpus are made up of he 8 proximiy sensors. The oher caegory of reinforcemen learning conrollers is he vision-based conrollers. The previously discussed works are all sensor-based conrollers. The mos basic vision based conrollers are hose which direcly inpu he capured image o he neural nework. This is illusraed in he work of Iida, M. e al [9] and Shibaa, K. and Iida, M. [10]. The former uilized a linear grayscale camera (1x64 pixels) and he laer uilized a CCD camera (320x240 pixels). The acor-criic archiecure was uilized o enable he mobile robo o orienae iself owards an objec and pushes i owards he wall. To furher improve he behavior of vision based conrollers, Gaske, C. e al [11] had uilized a coninuous sae, coninuous acion reinforcemen learning algorihm based on a mulilayered feed-forward neural nework combined wih an inerpolaor. This inerpolaion scheme is known as wire-fiing. The wirefiing funcion is a moving leas squares inerpolaor which is used o increase he speed experienced during he Q-value updaing process. I allows he updaing process o be conduced whenever i is convenien. The simulaion resuls show ha he robo is capable of demonsraing wandering and servoing behaviors hrough rial and error using reinforcemen learning. III. SYSTEM OVERVIEW Webos [12] is a commercial mobile robo simulaion sofware used by over 250 universiies and research ceners worldwide o model, program and simulae mobile roboics. The main reason for is increasing populariy is due o is abiliy o reduce he overall developmen ime. Using his plaform, a flexible simulaion for he Khepera II was developed. Some of he noable feaures in he simulaion are he cusom maze design feaure, reposiioning and reorienaion of he sensors, changing wall and floor exures, ligh inensiies, ec. In addiion, Webos was inerfaced wih he Microsof Visual C++.Ne 2002 inegraed developmen environmen by using a combinaion of MC++, C++ and C programs. Furhermore, he C programs were inerfaced o Malab 7.1 hrough he Malab engine. Malab was furher used o communicae o he serial por such ha conrol commands could be sen o he real robo and vice-versa via he radio base and radio urre. The ineracion of he various modules of sysem is shown in Fig. 2. 2

3 Inernaional Conference on Fascinaing Advancemen in Mechanical Engineering (FAME2008), 11-13, December 2008 Auodesk 3ds Max MC++ & C++ Supervisor MC++, C++ & C Neural Nework, Image Processing Tool Box Malab Session (v) (vi) Deermine an acion, a according o he Bolzmann Probabiliy Disribuion (during learning) or he equaion a = max(q(s,a)) (afer learning). Robo akes acion, a and reach a new posiion. Ge curren sae. (vii) If a collision occurred, a negaive numerical reward signal will be graned and he robo is rese back o is iniial posiion. (viii) Then, generae Q arge equaion: according o he DBos Q arge ( s, a ) r( s, a, s ) + maxq( s, a ) γ (2) = a + 1εA Fig. 2 Sysem Overview The Neural Q-Learning algorihms are mosly wrien in C. I could be easily inerchanged from one algorihm o anoher algorihm by using Graphical User Inerface (GUI) developed for he robo conroller. By reaing Malab as a background compuaion engine, he C programs are able o make use of he neural nework oolbox in Malab. Thus, daa will be ransmied back and forh from he simulaion o Malab and vice-versa. IV. NEURAL Q-LEARNING In his work, a oal of four conrollers were developed. These conrollers are: (i) Sensor-based Obsacle Avoidance (ii) Sensor-based Wall Following (iii) Vision-based Obsacle Avoidance (iv) Obsacle Avoidance based on a Combinaion of Sensor and Visual Inpus A common learning algorihm applies o all hese conrollers. The Neural Q-Learning algorihm implemened is as follows, (i) (ii) (iii) Cross-Compilaion or Remoe Connecion Simulaion Iniialize he neural nework in Malab and randomly assign he weighs of he neural nework. Define he iniial posiion of he Khepera II in he simulaion. Obain he sensor readings from he infrared sensors/ visual inpu from he camera/combinaion of boh. (iv) Obain Q(s,a) for each acion by subsiuing he curren sae and acion ino he neural nework. (ix) (x) where γ is he discoun rae (0 γ 1), r s a, s is he reward signal ( ), + 1 assigned o acion a for bringing he robo from sae s o sae s +1. Consruc he error vecor by using Q arge for he oupu uni corresponding o he acion aken and 0 for oher oupu unis. Repea (iii)-(ix) unil he robo is able o demonsrae he expeced behavior. To allow he mobile robo o explore he environmen firs and slowly converge o exploiing he learn policy, he Bolzmann Probabiliy Disribuion was uilized. The Bolzmann probabiliy could be denoed by he following equaion, 1 prob ( ak ) = exp( Q( s, ak )/ T ) (3) f where = a ( Q( s, a) ) f exp / (4) T and is he curren ieraion and k is he index of he acion seleced. The Bolzmann Probabiliy Disribuion was originally derived for physics and chemisry applicaions. Neverheless, i was adoped for he use in reinforcemen learning algorihms o define a policy which becomes greedier over ime. The key parameer o ensure his policy is he T parameer, which is known as he emperaure. I conrols he randomness of he acion selecion by seing i high a he beginning of he learning phase and slowly decreasing on each ieraion by he following equaion, ( T ) T + β (5) 1 = Tmin + Tmin

4 Deparmen of Mechanical Engineering, Mepco Schlenk Engineering College, Sivakasi, India where T min and β (0 < β 1) are consans. Thus, as T approaches T min, he robo will change from exploraion o exploiaion of he learn policy. The robo will have a oal of 5 acions o selec a any sae. These acions are illusraed in Fig. 3. Fig. 3 The Five Acions of he Khepera II A. A Sensor-based Obsacle Avoidance This conroller is very much similar o he works of Yang, G.-S. e al [4] and Huang, B.-Q. e al [5]. However, hey had only validaed his in simulaion only. In his work, i has been furher exended o he validaion of he conroller on he acual robo. The inpu saes are he 8 sensor readings. The neural nework design which has successfully demonsraed he desired resul is a 3 layer feed-forward backpropagaion neural nework. Fig. 4 shows he neural nework archiecure for his conroller. I has 3 layers wih 8 neurons on he inpu layer (pure linear acivaion funcion), 16 neurons on he hidden layer (angen sigmoid acivaion funcion) and 5 neurons on he oupu layer (pure linear acivaion funcion). The Variable Learning Rae Backpropagaion raining algorihm is used o rain he neural nework and i has eigh inpu which ranges from 0 o 1022 for each infra-red disance sensors. The reward funcion is designed as, Forward Moion: Turn Lef: Turn Righ: Roae Lef: Roae Righ: B. Sensor-based Wall Following This conroller is basically an exension over he sensor-based obsacle avoidance RL conroller. The nework configuraions and parameers used are he same for boh conrollers. The only difference beween his and he previous conroller is is reward funcion. By alering he reward funcion, he robo learns o refine is behavior from an obsacle avoidance behavior o a much refined wall following behavior. The reward funcion is designed as, Forward Moion: Turn Lef: Turn Righ: Roae Lef: Roae Righ: A collision is defined o ake place when any one of he five fron sensors reads a value exceeding 600. If so, he reward for ha ieraion is However if he reading is in beween , hen he final reward obained is C. Vision-based Obsacle Avoidance This conroller represens is saes in a oally differen way if compared o he sensor-based conrollers. The visual inpu is acquired from he K213 Linear Grayscale Camera. I provides an image wih an array size of 1x64 pixels. To allow he robo o acquire an obsacle avoidance behavior hrough he linear grayscale inpu, some modificaions are performed o he original environmen. The surrounding wall exures will be required o be changed o black and whie sripes. The number and widh of hese sripes changes as he robo moves oward or away from he walls. As such, his will serve as a paern for he robo o deermine wheher i is close o a wall, based on he visual inpu. Fig. 5 shows he Khepera II wih he K213 Linear Grayscale Camera Module. Fig. 4 Neural Nework Archiecure A collision is defined o ake place when any one of he five fron sensors reads in a value exceeding 600. If so, he reward for ha ieraion is Fig. 5 Khepera II wih he K213 Linear Grayscale Camera Looking a he size of he inpu image, i suggess ha if he image is o be applied direcly o he neural nework, he neural nework will require 64 neurons on is inpu layer. However, via several experimens, i was found ha he neural nework was no able o generalize. Thus, 4

5 Inernaional Conference on Fascinaing Advancemen in Mechanical Engineering (FAME2008), 11-13, December 2008 he 64 grayscale values are divided ino 8 segmens, each conaining 8 pixels. For each segmen, he average grayscale value is calculaed and fed o he neural nework on each ieraion. This conversion resuls in 8 average grayscale values for he 8 segmens. Similarly, he neural nework archiecure shown in Fig. 4 could be applied o his conroller. The reward funcion was designed similar o he sensor-based obsacle avoidance conroller. If (ds0 is medium) and (a0 is medium) hen (oupu1 is medium) (1) If (ds0 is medium) and (a0 is low) hen (oupu1 is medium) (1) If (ds0 is low) and (a0 is high) hen (oupu1 is high) (1) If (ds0 is low) and (a0 is medium) hen (oupu1 is medium) (1) If (ds0 is low) and (a0 is low) hen (oupu1 is low) (1) D. Obsacle Avoidance based on a Combinaion of Sensor and Visual Inpus This conroller was proposed in order o overcome he weakness of he vision-based obsacle avoidance conroller. Is weakness will be illusraed in he following secion. This conroller does no only avoid obsacles bu i also says away from black objecs. This behavior is creaed by implemening a Fuzzy Logic (FLC). The FLC fuzzifies he 8 sensor and 8 average grayscale values ino 8 oupus. I is designed in such a way ha he sensor readings ake more prioriy over he visual inpu. For each inpu, hree membership funcions are specified i.e. low, medium and high. The membership funcions for he sensor inpus are as illusraed in Fig. 6 and he membership funcions for he averaged grayscale values of he linear grayscale image are as illusraed in Fig. 6. Fig. 7 FLC Oupu Membership Funcion The neural nework archiecure implemened for his conroller is a 3 layer feed-forward backpropagaion neural nework. Likewise, i has 8 neurons on he inpu layer (pure linear acivaion funcion), bu 32 neurons on he hidden layer (angen sigmoid acivaion funcion) and 5 neurons on he oupu layer (pure linear acivaion funcion). The Variable Learning Rae Backpropagaion raining algorihm was uilized and is reward funcion is designed as, Forward Moion: Turn Lef: Turn Righ: Roae Lef: Roae Righ: Fig. 6 Membership Funcion for Sensor Readings and Averaged Grayscale Values As here are 8 sensors and 8 averaged grayscale values, he rules are relaively easy o define. The combinaion of he sensor inpu and image segmen will oally depend on is posiion. For example, sensor ds0 which is locaed on he lef side of he Khepera will be combined wih he averaged value of he lef mos segmen of he grayscale image. For he las wo averaged values, here are no oher opions bu o pair i wih he wo rear sensors. However, his is no o worry as he wo readings from he rear sensors are normally no aken ino consideraion as here are no backward moions. Fig. 7 shows he membership funcion of he FLC oupus and he rules for each pair of inpus are as follows, If (ds0 is high) hen (oupu1 is high) (1) A collision is defined o ake place when any one of he five fron sensors reads a value exceeding 600. If so, he reward for ha ieraion is Then if he oal number of inpus in he curren sae exceeding 200 is less han he oal number of inpus in he nex sae exceeding 200, hen he original numerical reward assigned for aking ha acion ges an addiional V. RESULTS AND DISCUSSION Before presening he resuls of he conrollers, here is a need o develop cerain measures o evaluae he funcion approximaor, which is he neural nework in his case. Mos supervised learning seeks o minimize he mean-squared error (MSE) over some disribuion, P, of he inpus. Anoher measure is he number of epochs i akes he neural nework o acquire a behavior. However, wih his algorihm, i is very hard o ensure ha a beer approximaion a some sae can be gained wihou he expense of worse approximaion a oher saes. This widely known

6 Deparmen of Mechanical Engineering, Mepco Schlenk Engineering College, Sivakasi, India issue is ermed as he inerference problem. As such, he mos effecive performance measure of he neural nework performance is hrough observaion of is acual behavior. Alhough his does no provide an accurae measure of is performance, however, i is he bes way o ensure he funcion approximaor had successfully acquired he desired behavior. Video clips of he robo were recorded for observaion purposes. However, o illusrae he resuls in his paper, he rajecory aken by he robo in he simulaed and acual environmen will be drawn. C. Vision-based Obsacle Avoidance The resuls shown in Fig. 10 sugges ha he neural nework is able o generalize when he average grayscale values are fed ino he neural nework. Alhough he robo learns how o avoid colliding ono walls hrough he visual inpu, however, is limied field of view has resuled in side collisions. Due o his, he conroller was no esed on he real robo o avoid unnecessary damage. A. Sensor-based Obsacle Avoidance I could be seen in Fig. 8 ha he robo has successfully demonsraed an obsacle avoidance behavior. In he simulaion, more obsacles are presen. This is due o he convenience of he cusom maze building feaure in he supervisor conroller program developed in Webos. However, for he real robo, he environmen is made much simpler due o mobiliy issues. The learning ime for each neural nework which has is weighs randomly assigned differs significanly. Thus, he number of epochs during each learning phase does no indicae anyhing a all. Fig. 8 Sensor-based Obsacle Avoidance Simulaion Resuls and Real Robo Resuls Fig. 10 Vision-based Obsacle Avoidance Simulaion 1 and Simulaion 2 D. Obsacle Avoidance based on a Combinaion of Sensor and Visual Inpus This conroller incorporaes wo behaviors wih he same goal; he obsacle avoidance behavior ogeher wih he avoidance of dark objec by implemening a FLC. To es his conroller, black objecs are placed on sraegic locaions on he walls which ac as a secondary guide for he robo o reach is final posiion. The sensor readings keep i safe from wall collisions on he side. This makes his conroller more superior o he visionbased conroller. Fig. 11 shows he rajecory aken by he robo in boh simulaed and acual environmen. B. Sensor-based Wall Following The sensor-based wall following behavior was successfully acquired by he robo and he resuls are illusraed in Fig. 9. As compared o he previous conroller, he movemen of he robo is much more refined. This is because he robo ravels following he posiion of he walls. To achieve his behavior will only require sligh modificaions in he reward funcion. (c) Fig. 9 Sensor-based Wall Following Simulaion Resuls and Real Robo Resuls Fig. 11 Obsacle Avoidance based on Combinaion of Sensor and Vision Inpus Simulaion Resuls, Real Robo Resuls 1, (c) Real Robo Resuls 2 6

7 Inernaional Conference on Fascinaing Advancemen in Mechanical Engineering (FAME2008), 11-13, December 2008 E. Discussion The main advanage of reinforcemen learning is is abiliy o allow he robo o learn hrough ineracion. The environmen can be oally unknown. I learns hrough he rewards i obains for each acion aken under differen saes. This concep is similar o how humans learn. Humans learn naurally hrough experience. For example, we learn no o ouch a cacus afer we ge our fingers pricked. The pain has resuled in a negaive reward. As such, hrough an accumulaion of differen experiences, humans acquire new skills and behavior. Of course, for he case of reinforcemen learning conroller, i will require furher advances in is heory before i could reach such levels. The drawback abou he learning algorihm is he large amoun of unknowns. All he parameers such as he discoun rae, iniial emperaure parameer, reward funcion and neural nework parameers are unknowns. There are no formulas or guidelines o selec an opimum se of parameers for he problem a hand. This has resuled in a major drawback when differen configuraions are being experimened. Performing analysis on he neural nework is already complex enough bu he addiional parameers inroduced by he learning algorihm makes he analysis even ougher. Due o large number of parameers which could be alered, i is ofen quie hard o idenify he acual reason for he failure of he robo o learn a desired behavior. The only way o idenify how hese parameers influence he sysem is hrough experience. Experimenal resuls reveal ha he discoun rae should always sar from a lower value such ha he neural nework, which has is weighs randomly assigned, could sele down o an equal sae before he acual learning phase sars. Then, he discoun rae is slowly increased as he number of ieraion increases. Furhermore, i was found ha he Variable Rae Learning Backpropagaion raining algorihm works well for he value approximaion problem. Anoher major drawback wih his algorihm is he presence of he inerference problem. One approach o his problem is o adop he Semi- Online Neural Q-Learning algorihm [13]. This nework acs locally and assures ha learning in one zone does no affec he learning in oher zones. I uses a daabase of learning samples. The main goal of his daabase is o include a represenaive se of visied learning samples, which is repeaedly used o updae he Neural Q- Learning algorihm. The immediae advanage of his is he sabiliy of he learning process and is convergence even in difficul problems. All he samples in he daabase will be compared o he new ones. If he old ones are found similar o he new ones, hey will be replaced. This ensures ha he daabase is mainained a he opimum size and no exra unnecessary raining ime is aken p due o duplicae samples. However, here are sill issues regarding he raining required for each raining cycle when he daabase scales up. Furhermore, an efficien updaing procedure is required for he daabase such ha minimum ime is required for he updaing process. VI. CONCLUSION In his paper, four reinforcemen learning conrollers based on he Neural Q-Learning algorihm were designed and esed on he acual and simulaed robo using he Webos Commercial Robo Simulaion sofware. In addiion o building a flexible simulaion environmen for he Khepera II, furher exension was made o he work by Yang, G.-S. e al [4] and Huang, B.-Q. e al [5] by validaing he sensorbased obsacle avoidance conroller on he acual robo. This evenually leads o he invesigaion of he wall following behavior and vision based conrollers wih he pros and cons of he learning algorihm observed during experimens being highlighed. In conclusion, he robo was able o acquire is desired behavior hrough is ineracion wih he environmen. ACKNOWLEDGMENT The auhors hank Monash Universiy Malaysia for he suppor of his work. REFERENCES [1] Suon, R.S. and Baro, A.G., Reinforcemen Learning: An Inroducion, MIT Press, [2] Wakins, C. and Dayan, P., Q-Learning, Machine Learning, vol.8, pp , [3] Jakša, R., Sinčák, P. and Majerník, P., Backpropagaion in Supervised and Reinforcemen Learning for Mobile Robo Conrol, Available: hp://neuron-ai.uke.sk/~jaksa/publicaion s/jaksa- Sincak-Majernik-ELCAS99.pdf (Accessed: 2006, April 24). [4] Yang, G.-S., Chen, E.-K. and An, C.-W., Mobile Robo Navigaion Using Neural Q-Learning, Proceedings of 2004 Inernaional Conference on Machine Learning and Cyberneics, vol. 1, pp , [5] Huang, B.-Q., Cao, G.-Y. and Guo, M., Reinforcemen Learning Neural Nework o he Problem of Auonomous Mobile Robo Obsacle Avoidance, Proceedings of 2005 Inernaional Conference on Machine Learning and Cyberneics, vol.1, pp , [6] Ona, A., Kia, H. and Nishikawa, Y., Recurren Neural Neworks for Reinforcemen Learning: Archiecure, Learning Algorihms and Inernal Represenaion, 1998 IEEE Inernaional Join Conference on Neural Neworks Proceedings, vol. 3, pp , [7] Cervera, E. and del Pobil, A.P., Eliminaing Sensor Ambiguiies Via Recurren Neural Neworks in Sensor- Based Learning, 1998 IEEE Inernaional Conference

8 Deparmen of Mechanical Engineering, Mepco Schlenk Engineering College, Sivakasi, India on Roboics and Auomaion, vol. 3, pp , [8] Sehad, S. and Touze, C., Self-Organizing Map for Reinforcemen Learning: Obsacle-Avoidance wih Khepera, Proceedings from Percepion o Acion Conference, pp , [9] Iida, M., Sugisaka, M. and Shibaam K., Applicaion of Direc-Vision-Based Reinforcemen Learning o a Real Mobile Robo, Available: hp://shws.cc.oiau.ac.jp/~shibaa/pub/iconip02-iida.pdf (Accessed: 2006, June 24). [10] Shibaa, K. and Iida, M., Acquisiion of Box Pushing by Direc-Vision-Based Reinforcemen Learning, Available: hp://shws.cc.oia-u.ac.jp/~shibaa/pub/sic E03.pdf (Accessed: 2006, June 24). [11] Gaske, C., Flecher, L. and Zelinsky, A., Reinforcemen Learning for a Vision Based Mobile Robo, Available: hp://users.rsise.anu.edu.au/~rsl/r sl_papers/2000iros-nomad.pdf (Accessed: 2006, May 31). [12] Webos. hp:// Commercial Mobile Robo Simulaion Sofware. [13] Semi-Online Neural-Q Learning, Available: hp:// sca.es/tesis_udg/available/tdx //m cp3de3.pdf (2006, June 30). 8