Robot Docking Based on Omnidirectional Vision and Reinforcement Learning

Transcription

1 Robot Dockng Based on Omndrectonal Vson and Renforcement Learnng Davd Muse, Cornelus Weber and Stefan Wermter Hybrd Intellgent Systems, School of Computng and Technology Unversty of Sunderland, UK. Web: Abstract We present a system for vsual robotc dockng usng an omndrectonal camera coupled wth the actor crtc renforcement learnng algorthm. The system enables a PeopleBot robot to locate and approach a table so that t can pck an object from t usng the pan-tlt camera mounted on the robot. We use a staged approach to solve ths problem as there are dstnct sub tasks and dfferent sensors used. Startng wth random wanderng of the robot untl the table s located va a landmark, and then a network traned va renforcement allows the robot to turn to and approach the table. Once at the table the robot s to pck the object from t. We argue that our approach has a lot of potental allowng the learnng of robot control for navgaton removng the need for nternal maps of the envronment. Ths s acheved by allowng the robot to learn couplngs between motor actons and the poston of a landmark. 1 Introducton Navgaton s one of the most complex tasks currently under development n moble robotcs. There are several dfferent components to navgaton and many dfferent sensors that can be used to complete the task, from range fndng sensors to graphcal nformaton from a camera. The man functon of robot navgaton s to enable a robot to move around ts envronment, whether that s followng a calculated or predefned path to reach a specfc locaton or just random wanderng around the envronment. Some of the components nvolved n robotc navgaton are () localsaton, () path plannng and () obstacle avodance. For an overvew of localsaton and map-based navgaton see [1 & 2]. When dscussng robot navgaton, smultaneous localsaton and map buldng should be ncluded (see [3, 4 & 5] for some examples). There has been a lot of research and systems developed for robot navgaton usng range fndng sensors (sonar, laser range fnders etc) [6, 7 & 8] but there has been less research nto vsual robotc navgaton. There are recent developments n the feld of vsual navgaton manly concentratng on omndrectonal vson (see [9, 10 & 11] for examples). Many of the navgaton systems mplemented for robot navgaton stll use hard codng whch causes a problem wth the lack of adaptablty of the system. However, some systems have ncluded learnng (see [12 & 13] for examples). A common tranng method used for the learnng systems are varous forms of renforcement learnng, [14] provdes a good overvew. These learnng algorthms overcome the problem of supervsed learnng algorthms as nput output pars are

2 not requred for each stage of tranng. The only thng that s requred s the assgnment of the reward whch can be a problem for complex systems as dscussed n [15]. However, for systems where there s just one goal ths does not pose a problem as the reward wll be admnstered only when the agent reaches the goal. The focus on ths paper s to extend the system developed n [16] where renforcement learnng s used to allow a PeopleBot to dock to and pck an object (an orange) from a table. In ths system neural vson s used to locate the object n the mage, then usng traned motor actons (va the actor crtc learnng algorthm [17]) the am s to get the object to the bottom centre of the mage resultng n the object beng between the grppers of the robot. There are some lmtatons to the system whch need to be overcome to mprove ts usefulness. For example, the dockng can only work f the object s n sght from the begnnng whch results n the system beng confned to a very small area. Also the system fals f the object s lost from the mage. Fnally, the angle of the robot wth respect to the table s nferred from the odometry, whch makes t necessary to start at a gven angle. None of these are desrable and t s the am of ths work to address some of the lmtatons and extend the range that the robot can dock from. The system proposed n ths paper wll make use of an omndrectonal camera to locate and approach a table n an offce envronment. The use of an omndrectonal camera allows the robot to contnuously search the surroundng envronment for the table rather than just ahead of the robot. Here the extended system wll use the omndrectonal camera to locate the table va a landmark placed beneath t. Once located the robot s to turn and approach the table usng a network traned by renforcement. The remander of the paper s structured as follows; Secton 2 dscusses the task, the overall control of the system and what trggers the shfts between the dfferent phases. The frst phase uses an omndrectonal camera to detect any obstacles and take the necessary acton to avod them and s dscussed n Secton 2.1. The second phase uses the omndrectonal camera to locate the poston of the landmark n relaton to the robot, whch t then passes to a neural network to produce the requred motor acton on the robot and s dscussed n Secton 2.2. The fnal phase uses a neural system wth the pan tlt camera mounted on the robot to allow the robot to dock wth the object on the table and pck t up; ths s dscussed n Secton 2.3. Secton 3 covers the algorthm used for the table approachng phase of the extended scenaro. The expermentaton of the extended scenaro s then descrbed n secton 4. Fnally, Sectons 5 and 6 cover the dscusson and summary respectvely. 2 The Scenaro The overall scenaro s llustrated n Fgure 1. It starts wth the robot beng placed n the envronment at a random poston away from the table. The robot s then to wander around the envronment untl t locates the table (Phase I). Ths phase uses conventonal mage processng to detect and avod any obstacles. Once the table s located va a landmark placed beneath t the robot s to turn and approach the table

3 Fgure 1 - Scenaro (Phase II). Then once at the table the robot s to pck the object from the table (Phase III), ths system s dscussed n [16]. Both Phase II & III use neural networks traned wth the Actor Crtc learnng algorthm. The frst two phases of the system use an omndrectonal camera llustrated n Fgure 2 and the fnal phase uses the pan tlt camera mounted on the robot. Concal Mrror Camera Fgure 2 (Left) PeopleBot Robot wth mounted omndrectonal camera, (Rght) Close up of the Omndrectonal Camera

4 To enable the ntegraton of the three phases an overall control functon was needed to execute the relevant phases of the system dependng on the envronmental condtons. Fgure 3 shows the control algorthm. Whle the robot s not at the table Take a pcture (omndrectonal) Check f the landmark s n sght If the landmark s not n sght Wander Else the landmark s n sght Pass control to the actor crtc and get ext status If exted because landmark s lost Go back to Wanderng Else exted because robot s at the table Pass control to the object dockng End f End f End Whle Fgure 3 - System Algorthm When the robot s not at the table, or the landmark s not n sght, the robot checks for the landmark at each teraton through the system. The landmark that the robot looks for s produced by a board of red LED s whch s located drectly beneath the table as llustrated n Fgure 4. Whle the robot has not located the landmark the random wanderng system s executed. If the landmark has been located then control s passed to the table approachng behavour whch runs to completon. There are two possble outcomes for the table approachng whch are; () Lost sght of the landmark and () Reached the table. If the landmark has been lost then the robot starts to search for t agan, otherwse t has reached the table and control s passed to the object dockng whch completes the task. Landmark Fgure 4 - Setup of the Envronment: The landmark s used to dentfy the poston of the table

5 2.1 Phase I The Random Wanderng Ths behavour allows the robot to move around the envronment whle avodng obstacles. The system uses an omndrectonal camera (Fgure 2 rght) to get a vew of the envronment surroundng the robot. From ths mage the robot s able to detect obstacles and produce the requred motor acton to avod them. To perform ths detecton the behavour uses classcal mage processng to remove the background from the mage and leave only perceved obstacles, as seen n Fgure 5. Here the orgnal mage taken by the omndrectonal camera s n the left of the fgure, wth dfferent stages of the mage processng shown n the centre and rght. Fgure 5 Obstacle Detecton The centre mage s the ntermedate stage where just the background of the mage has been removed; ths s acheved by colour segmentaton of the most common colour from the mage. To fnd the most common colour n the mage a hstogram s produced for the RGB values of each pxel. Then the value wth the largest densty s found and any colour wthn the range of +/- 25 of the most common colour s removed. Ths removes the carpet from the mage (assumng that the carpet s present n the majorty of the mage) whch leaves the obstacles and some nose. Also at ths stage the range of the obstacle detecton s set removng any nose from the perphery of the mage. Then the nose s removed by mage eroson followed by dlaton. The eroson strps pxels away from the edges of all objects left n the mage. Ths removes the nose but t also reduces the sze of any obstacles present. To combat ths once the eroson has been performed, dlaton s performed to restore the obstacles to ther orgnal sze, the shape of the obstacles are slghtly dstorted by ths process. However, the obstacles left n the fnal mage are stll sutable to produce the requred motor acton to avod them. The last stage of the mage processng s to use edge detecton to leave only the outlnes of the obstacles (Fgure 5 rght). The robot always tres to move straght ahead unless an obstacle s detected n the robot s path. When ths happens the robot turns the mnmum safe amount allowed to avod the obstacles. In the example provded n Fgure 5, the robot cannot move straght ahead so the robot would turn to the left untl t can avod the obstacle on the rght of the mage. As the mage s a mrrored mage of the envronment the objects whch appear on one sde of the mage are physcally to the other sde of the robot. Once the robot has turned the requred amount t would start to move straght and the obstacle detecton would then be performed agan. 2.2 Phase II The Table Approachng Behavour Ths phase of the system allows the robot to approach the table (landmark) once detected. Ths has two ext statuses whch are () the robot lost sght of the

6 landmark or () the robot has reached the table. If the robot looses sght of the table t goes back to the wanderng phase untl t locates the landmark agan. Ths can happen f the landmark moves behnd one of the supportng pllars of the concal mrror. If the robot reaches the table, control wll be passed to the fnal stage of the system whch s to dock to and pck up the object. To allow the robot to move to the table a network was traned usng the Actor Crtc renforcement learnng rule [17]. The state space was the mage wth the goal set to where the landmark s perceved to be n front of the robot. The motor acton that the network performs s to rotate the robot to the left or to the rght dependng on where the landmark s perceved n relaton to the robot. The nput to the network s the x y coordnates of the closest pont of the perceved landmark. Once the landmark appears to be ahead of the robot, the robot then moves forward, checkng that the landmark s stll ahead of t. Once the landmark s ahead of the robot and less than the threshold dstance of 1 meter the robot then moves drectly forward untl the table sensors located on the robot s base are broken. When ths happens the robot s at the table and control s gven to Phase III. The robot only looks for the landmark n the range that the robot can detect drectly ahead (as the webcam produces a rectangular mage, more can be seen to the sdes of the robot. The range s set to the maxmum dstance the mage can detect ahead of the robot; ths s roughly 2m). If the landmark s detected outsde ths range when the robot turned t would lose sght of the landmark, therefore anythng outsde ths regon s gnored. If the landmark appears n the rght sde of the detectable range then the robot should rotate to the left as the mage s mrrored, f t appears n the left the robot should rotate to the rght and f t s straght ahead of the robot then t should move forward. LANDMARK DETECTED LANDMARK Fgure 6 - Landmark Detecton To detect the landmark classcal mage processng s once agan employed to detect the landmark as shown n Fgure 6. The orgnal mage s n the left of Fgure 6 wth the landmark hghlghted and the detected landmark s hghlghted n the rght of Fgure 6. The frst stage to the mage processng s to perform colour segmentaton where t segments any colour that s the desgnated colour of the landmark. Once ths process s complete edge detecton s used to leave just the edges of the remanng objects. Then t s assumed that the largest object left n the mage s the landmark. The last stage of the mage processng s to locate the closest pont of the landmark to the robot. Ths pont s then fed nto the network to produce the requred acton by the robot.

7 2.3 Phase III Dockng Ths phase allows the robot to dock to and pck an orange from the table. The functonalty of the system s descrbed n [16]. However, there s a problem wth ths system for the ntegraton nto the extended scenaro; the odometry of the robot s set to 0 and the robot must start parallel to the table to allow the robot to dock to the orange. Wth the table approachng system t cannot be guaranteed that the robot wll be parallel to the table and hence the robot wll not know the relatonshp between the odometry and the angle of the table. Before ths system s ntegrated t s requred that the angle of the table to the robot s calculated. To solve ths t s planned to use mage processng to detect and calculate the angel of the table n relaton to the robot. Once the robot reaches the table a pcture wll be taken usng the conventonal pan tlt camera mounted on the robot. The edge of the table wll then be detected usng colour thresholdng and edge detecton. a) b) c) d) α Fgure 7 - Edge Detecton of the Table The thresholdng wll be performed n the same way as n Phase I wth the most common colour beng removed. It s assumed that the most common colour wll ether be () the colour of the table tself or () the colour of the carpet beneath the table. In both cases the edge between the removed colour and the remanng colour wll be the edge of the table. Usng edge detecton the coordnates of the two end ponts of ths lne can be found and from ths the angle of the table calculated and used wth the odometry to get the robot to dock to the orange. Fgure 7 demonstrates ths mage processng usng the artfcal mage (a), here the whte s thought to be the most common colour so wll be removed and the remanng components of the mage are changed to whte (b). The next stage s to perform the edge detecton (c). Wth ths done the angle can be calculated (d) and used to alter the odometry of the robot. Ths s to remove the constrant that the robot must arrve parallel to the table.

8 3 Actor Crtc Algorthm The developed network s an extenson of the actor crtc model used n [17]. Here the system has been adapted to work wth contnuous real-world envronments. We have used ths algorthm n two phases of the scenaro: frst, the approach to the table (Phase II), and then to perform the dockng at the object. In Phase II, the nput to the network s the poston n the omndrectonal mage where the landmark appears as opposed to the locaton of the agent n the envronment. In Phase III, the nput s the perceved locaton of the object of nterest from the standard robot camera. For the archtecture of the network developed for Phase II, t was decded that there would be two nput neurons; one for the x and y coordnates respectvely, 50 hdden unts to cover the state space of the mage and two output neurons one for each of the actons to be performed and one neuron for the crtc. The archtecture s llustrated n Fgure 8. The hdden area covers only the detectable regon of the mage wth each neuron coverng roughly 40mm 2 of actual space. Ths results from the fact that the detectable range of the envronment s roughly a radus of 2m from the robot. All unts are fully connected to the hdden layer. Intally the crtcs weghts are set to 0 and are updated by Equaton 4. The Actor weghts (Motor Acton unts) are ntalsed randomly n the range of 0 1 and are updated va Equaton 7. Fnally, the weghts connectng the nput unts to the network (Hgh level vson) are set to 1 and these weghts are not updated. Fgure 8 - Archtecture of the Network. The nodes are fully connected, the nput for the x, y coordnates are normalsed nto the range 0-50 and the output of the network generates the motor acton to rotate the robot Equaton 1 descrbes the frng rate of the place cells (here the term place cell s used loosely as they encode a perceved poston of a landmark n the mage) to be calculated. The frng rate s defned as: 2 p s f = ( p) exp 2 2σ (1)

9 where p s the perceved poston of the landmark, s s the locaton n the mage where neuron has maxmal frng rate and σ s the radus of the Gaussan of the frng rates coverng the mage space of each neuron. Ths was set to 0.75 durng the experments. The frng rate C of the Crtc s calculated usng Equaton 2 and has only one output neuron as seen n Fgure 8. The frng rate of the crtc s thus a weghted sum of all of the frng rates of the place cells. C ( p) w f ( p) = To enable tranng of the weghts of the crtc some method s needed to calculate the error generated by the possble moves to be made by the robot. Ths s made possble by Equaton 3 and the dervaton of ths equaton can be found n [17]. t = Rt + γc t+1 ( p ) C( p ) t (2) δ (3) However as R t only equals 1 when the robot s at the goal locaton and C(p t+1 ) s 0 when ths occurs and vce versa they are never ncluded n the calculaton at the same tme. γ s the constant dscountng factor and was set to 0.7 for the experments. Wth the predcted error, the weghts of the crtc are updated proportonally to the product of the frng rate of the actve place cell and the error (Equaton 4). t ( p ) w δ f (4) Ths concludes the equatons that were used for the place cells and the crtc, fnally there are the equatons used for the actor. There were two output neurons used n ths experment, one to make the robot rotate to the left and the other to make the robot rotate to the rght. The actvaton of these neurons s acheved by takng the weghted sum of the actvatons of the surroundng place cell to the current locaton as llustrated n Equaton 5. a j ( p) z f ( p) = = A probablty s used to judge the drecton that the robot should move n, ths s llustrated n Equaton 6. Here the probablty that the robot wll move n one drecton s equal to the frng rate of that actor neuron dvded by the sum of the frng rate of all the actor neurons. To enable random exploraton when the system s tranng, a random number s generated between 0 and 1. Then the probablty of each neuron s ncrementally summed; when the result crosses the generated value that acton s executed. As the system s traned the lkelhood that the acton chosen s not the traned acton decreases. Ths s because as the network s traned the probablty that the traned acton wll occur wll approach 1. exp ( 2a ) (6) P j k exp j j t ( 2a ) Ultmately, the actor weghts are traned usng Equaton 7 n a modfed form of Hebban learnng where the weght s updated f the acton s chosen and not updated f the acton s not performed. Ths s acheved by settng g j (t) to 1 f the acton s chosen or to 0 f the acton s not performed. Wth ths form of tranng both the actor and the crtcs weghts can be bootstrapped and traned together. k ( p ) g ( t) (5) z j δ t f (7) t j

10 4 Expermentaton and Results To tran and test the network separate tranng and test data sets were produced. The tranng set contaned 1000 randomly generated samples and the test set contaned 500 randomly generated samples. These samples were stored n separate vectors and contaned the followng nformaton () the normalsed x coordnate of the landmark, () the normalsed y coordnate of the landmark, () the angle of the landmark n relaton to the robot and (v) the dstance of the landmark from the robot. Durng tranng each sample was fed nto the network and t ran untl the goal was acheved. Therefore, after each epoch there would be 1000 successful samples and the testng data was fed nto the network wthout any tranng takng place. The traned weghts of the crtc are shown n Fgure 9 (d), whch took 50 epochs to get the tranng to the level shown. It would have been mpractcal to tran the network on the robot due to the tme t would requre, so a smple smulator was employed whch used the tranng set to perform the acton recommended by the network (ths used the same data that would be generated from the mage processng). Ths was acheved by calculatng the next perceved poston of the landmark. Ths greatly reduced the tme needed to tran the network, for the 50 epochs t took roughly 5 hours to tran (ncludng the testng after each epoch) on a Lnux computer wth a 2GHz processor and 1 Ggabyte of ram. Fgure 9 also shows the untraned weghts (a), the weghts after the presentaton of 1 tranng sample (b) and the weghts after the presentaton of 500 tranng samples (c). Here t can be seen that the weghts spread from the goal locaton around the network durng the tranng. There s a V secton of the weghts that reman untraned, ths relates to the goal locaton (see Fgure 8) so no tranng s needed n ths secton of the network as the requred state s reached. a) b) c) d) Fgure 9 - Strength of Crtc Weghts Durng Tranng. (a) untraned weghts, (b) weghts after presentaton of 1 sample, (c) weghts after presentaton of 500 samples and (d) weghts after 50 epochs of tranng

11 Moves Tranng Run No of moves No of epochs Testng Set Tranng Set % of Correct Moves - Run % No of epochs Testng Set Tranng Set Fgure 10 - Tranng Stats (Top) average number of steps requred to reach the goal locaton durng the testng of the network. (Bottom) percentage of correct moves made durng the testng of the network. Fgure 10 shows the statstcs gathered durng the tranng of the network. After each epoch of tranng the network s tested both wth the tranng data and the testng data. Here the samples are presented to the network and data gathered about () number of steps needed to reach the goal and () the percentage of correct moves made by the network at each step. Durng ths testng of the network tranng was prohbted and the relevant statstcs gathered. Ths was done three tmes wth all results beng smlar. Fgure 10 (top) shows the average number of moves needed after each epoch for the goal to be reached. An average s taken for both the test and tranng set so the test value s averaged over the 500 test samples

12 and the tranng over the 1000 tranng samples. Intally, wth no tranng, t takes on average approxmately 650 steps for the agent to reach the goal locaton. Ths rapdly decreases and settles to about 10 steps after roughly 30 epochs, the number of steps requred for the testng and tranng sets are very smlar and the performance s as good on the testng set as the tranng set. Fgure 10 (bottom) llustrates the percentage of correct moves made at each step durng the testng of the network. As expected ntally, as the agent moves randomly the number of correct moves s roughly 50%, as there are two actons to be performed. Ths steadly rses durng tranng, however, ths doesn t stablse after 30 epochs lke the number of moves does. The performance keeps mprovng although the rate of mprovement does decrease after approxmately 60 epochs. In addton, the testng set doesn t perform as well as the tranng set does durng the testng; ths doesn t affect the average number of moves requred to reach the goal. 5 Dscusson The developed system has been successful n allowng the robot to approach the table from random places n the envronment. Once the orange dockng s lnked, the scenaro wll be complete. Renforcement learnng has been successfully used n two of the phases of ths applcaton. Ths llustrates that renforcement learnng s a vable opton for use n robot navgaton tasks. Ths poses qute an nterestng queston; humans can easly see dstnctve dfferences n tasks; would we be able to tran a computer to do a smlar thng? Instead of the programmer splttng the state space, could the computer automatcally partton the state space? Ths has been approached n [18 & 19]. In these papers dfferent technques are adopted to partton the state space. Smple portonng of the state space would not have been a vable opton n our approach as one network would be needed for the entre system. However, ths would result n a large state space coverng n our applcaton the vsual nputs of the omndrectonal camera as well as the pan-tlt camera. Therefore we have addressed ths curse of dmensonalty problem by segmentng the task nto phases resultng n two smaller manageable state spaces. Investgaton could be made nto mprovements n the network to enhance the percentage of correct moves made. Some possbltes could nclude ncreasng the number of samples n the tranng set to ncrease the coverage of the tranng, allowng more startng locatons to be traned. Another possblty could be to adjust the tranng algorthm to allow a smoother degrade n the strength of the crtcs weghts. As the agent move away from the goal locaton there are large decreases n the strength of the weghts to the extent that when the landmark appears behnd the agent the crtc s weghts are very weak so the agent may stll be movng randomly n ths secton. A smoother decrease n the crtc s weghts would allow ths secton of the network to have stronger weght connectons and thus mprove the performance of the network. There s one method that could be used to mprove the network nstantly whch would be to swtch from exploraton of the envronment to explotaton. Here the actor unt would be chosen whch would gve maxmum reward. Ths however could lead to suboptmal solutons f used too early n tranng. An alternatve to the developed system could be to pan and tlt the camera that s suppled wth the robot to fnd the target from a large dstance and perform the

13 whole acton based on ths vsual nformaton. So nstead of keepng the camera n a fxed poston the camera could be moved to locate the table and object. Ths requres a coordnate transform to allow the calculaton of the angle to the object gven the odometry of the robot, the perceved poston of the orange on the camera mage and the pan of the camera. Ths s also an approach whch we are currently pursung [20] Whle such an approach enhances the range of an acton strategy that reles on a sngle state space, there wll reman stuatons n whch a mult-step strategy has to be employed, such as f the target object s not vsble from the startng pont. Wthout the object vsble, agan one strategy s needed to get the robot close to the table and another for the dockng to the object. 6 Summary Ths paper has dscussed the navgaton system developed to allow the robot to frstly locate and dock to a table va a landmark. Ths greatly extended the range of dockng of the system developed n [16]. Both systems (the orgnal dockng and the extended navgaton) used the actor crtc renforcement technque to tran the networks they used to acheve ther goals. The extended navgaton system traned ts own network to allow the robot to move to the table, whch has been demonstrated to work effectvely. Once at the table the dockng phase s able to complete the task. The navgaton system developed has shown that renforcement learnng can successfully be appled to a real world robot navgaton task. Ths system shows great potental for the development of a more advanced navgaton system. Acknowledgements Ths s part of the MrrorBot project supported by the EU, FET-IST programme, grant IST , coordnated by Prof. Wermter. References 1. Fllat, D. & Meyer, J.A. Map-based navgaton n moble robots. I. A revew of localzaton strateges. J. of Cogntve Systems Research 2003, 4(4): Fllat, D. & Meyer, J.A. Map-based navgaton n moble robots. II. A revew of map-learnng and path-plannng strateges. J. of Cogntve Systems Research 2003, 4(4): Dssanayake, M.W.M.G. Newman, P. Clark, S. Durrant-Whte, H.F. & Csorba, M. A soluton to the smultaneous loacalzaton and map buldng (SLAM) problem. IEEE Transactons on Robotcs and Automaton 2001, 17(3): Tomats, N. Nourbakhsh, I. & Segwart, R. Hybrd smultaneous localzaton and map buldng; a natural ntegraton of topologcal and metrc. Robotcs and Autonomous Systems 2003, 44: Guvant, J.E. Masson, F.R. & Nebot E.M. Smultaneous localzaton and map buldng usng natural features of absolute nformaton. Robotcs and Autonomous Systems 2002, 40: Carell, R. & Frere, E.O. Corrdor navgaton and wall-followng stable control for sonar-based moble robots. Robotcs and Autonomous Systems 2003, 45:

14 7. Maaref, H. & Barret, C. Sensor-based navgaton of a moble robot n an ndoor envronment. Robotcs and Autonomous systems 2002, 38: Delgado, E. & Barrero, A. Sonar-based robot navgaton usng nonlnear robust observers. Automatca 2003, 39: Menegatt, E. Zoccarato, M. Pagello, E. & Ishguro, H. Image-based Monte Carlo localsaton wth omndrectonal mages. Robotcs and Autonomous Systems 2004, 48: Fala, M. & Basu, A. Robot navgaton usng panoramc trackng. Pattern Recognton 2004, 37: Jogan, M. & Leonards, A. Robust localsaton usng an omndrectonal appearance-based subspace model of envronment. Robotcs and Autonomous Systems 2003, 45: Gausser, P. Joulan, C. Banquet, J.P. Lepretre, S. & Revel, A. The Vsual Homng Problem: An Example of Robotcs/Bology Cross Fertlzaton. Robotcs and Autonomous Systems 2000, 30: Gausser, P. Revel, A. Joulan, C. & Zrehen, S. Lvng n a Partally Structured Envronment: How to Bypass the Lmtatons of Classcal Renforcement Technques. Robotcs and Autonomous Systems 1997, 20: Sutton, R.S. & Barto, A.G. Renforcement Learnng An Introducton. MIT Press Wörgötter, F. Actor-Crtc models of anmal control a crtque of renforcement learnng. Proceedng of Fourth Internatonal ICSC Symposum on Engneerng of Intellgent Systems Weber, C. Wermter, S. & Zochos, A. Robot dockng wth neural vson and renforcement. Knowledge Based Systems 2004, 12: Foster, D.J. Morrs, R.G.N. & Dayan, P. A model of hppocampally dependent navgaton, usng the temporal learnng rule. Hppocampus 2000, 10: Kondo, T. & Ito, K. A renforcement learnng wth evolutonary state recrutment strategy for autonomous moble robot control. Robotcs and Autonomous Systems 2004, 46: Lee, I.S.K. & Lau, A.Y.K. Adaptve state space parttonng for renforcement learnng. Engneerng Applcatons of Artfcal Intellgence 2004, 17: Weber, C. Muse, D. Elshaw, M. & Wermter, S. Neural robot dockng nvolvng a camera-drecton dependent vsual-motor coordnate transformaton. AI 2005 (submtted)