Passive Distance Learning for Robot Navigation

Transcription

1 Passive Distance Learning or Robot Navigation Sven Koenig Reid G. Simmons Schoo o Computer Science Carnegie Meon University Pittsburgh, PA skoenig@cs.cmu.edu reids@cs.cmu.edu Abstract Autonomous mobie robots need good modes o their environment, sensors and actuators to navigate reiaby and eicienty. Whie this inormation can be suppied by humans, or earned rom scratch through active exporation, such approaches are tedious and time-consuming. Our approach is to provide the robot with the topoogica and geometrica constraints that are easiy obtainabe by humans, and have the robot earn the rest whie in the course o perorming its tasks. We present GROW-BW, an unsupervised and passive distance earning agorithm that overcomes the probem that the robot can never be sure about its ocation i it is not aowed to reduce its uncertainty by asking a teacher or executing ocaization actions. Advantages o GROW-BW incude that the robot can be used immediatey to perorm navigation tasks and improves its perormance over time, ocusing its attention to routes that are more reevant or its tasks. We demonstrate that GROW-BW can earn good distance, sensor, and actuator modes with ony a sma amount o experience. 1 Introduction We are interested in providing the technoogy or oice or hospita deivery robots that are autonomous. Assume that you have just purchased such a deivery robot. Beore it can be used, it must gain some knowedge o its new environment. This can be achieved by either providing the robot with the necessary inormation or etting it expore its environment autonomousy. Both methods have disadvantages. Providing the robot with the necessary inormation suers rom the probem that some inormation is diicut or impossibe to provide by humans. The sensor and actuator modes o the robot, or exampe, depend not ony on its environment, but aso on characteristics o the robot itse, and one cannot expect consumers to be amiiar with detais o their newy purchased deivery robots. Other data coud be provided by the consumers, but might be cumbersome to obtain. I they do not know the exact engths o their corridors, or exampe, they have to measure them a task that the robot coud do itse. Letting the robot expore its environment autonomousy, a method that many researchers have investigated [Kuipers and Byun, 1988] [Basye et a., 1989] [Mataric, 1990] [Dean et a., 1992], suers rom the probem that the robot cannot be used immediatey and, during exporation, is ikey to get into situations o conusion or danger that require human intervention, since it has no initia knowedge o its environment. We thereore suggest combining both methods: the robot is provided with some inormation that is easiy avaiabe to humans, and it then autonomousy earns the rest o the inormation needed or reiabe navigation whie in the process o perorming its deivery tasks. We start by suppying the robot with a topoogica map o its environment. A topoogica map speciies andmarks (such as corridor junctions) and how they connect. Such a map can easiy be obtained rom a sketch drawn by peope amiiar with the environment. Figure 2 (center and right), or exampe, shows a sketch o a corridor environment and the corresponding topoogica map. Once equipped with a topoogica map, the robot coud use andmark-based navigation to perorm deivery tasks. However, andmark-based navigation techniques suer rom the probem that imperect sensors occasionay miss andmarks and even perect sensors are not abe to distinguish between a andmarks, such as corridor junctions o the same type (perceptua aiasing probem). The reiabiity and eiciency o the robot can be improved by adapting its sensor and actuator modes to its environment and, a simper task or peope, by providing it with distance inormation. However, peope oten err even with respect to distances uness they measure them. Athough the sketch o Figure 2 (center), or exampe, correcty speciies the topoogy, some o the arc engths are incorrect. It is thereore much more reiabe and convenient to et the robot earn the distance, sensor, and actuator modes itse.

2 Figure 1: Xavier and two screen shots o its user interace Figure 2: Corridor environment, sketch, and corresponding topoogica map We want the earning to be unsupervised (not to require a teacher during earning, ater it has been suppied with the topoogica map) and passive (not to expicity contro the robot s actions). Unsupervised, passive distance earning is not a trivia task, because the robot can never be sure about its ocation: it has no distance inormation avaiabe initiay, its sensors and actuators are noisy, and it cannot reduce the uncertainty about its ocation by asking a teacher or executing ocaization actions. In act, its positiona uncertainty may be quite signiicant. For exampe, Figures 1 (right) and 2 (et) show that ater traveing some distance, the robot is unsure about its ocation (the sizes o the circes are proportiona to the probabiity mass at each ocation). On the other hand, unsupervised, passive earning has the advantages that the robot can be used immediatey to perorm deivery tasks (since it has a topoogica map avaiabe) and it does not require a separate training phase or (ideay) any externa hep. In addition, the robot never stops earning: whenever it moves, it gains more and more experience with its environment which it continuay uses to improve its distance, sensor, and actuator modes and, as a consequence, aso its navigation perormance. Since it gains more inormation about routes that the robot traverses more oten, earning ocuses its attention to routes that are more reevant or the deivery tasks. In the next severa sections, we describe our agorithm or earning distances, sensor modes, and actuator modes in an indoor oice environment. We concude by presenting experimenta resuts showing that the agorithm can earn good modes with ony a sma amount o experience. Our research is carried out on Xavier and its simuator (Figure 1). Xavier is buit on an RWI B base and incudes bump sensors, sonars, a aser range sensor, and a coor camera on a pan-tit head. Contro, perception, and panning are a carried out on two on-board, muti-processing 486-based machines. Xavier roams the corridors o our buiding and can be controed by users wordwide via its experimenta Word Wide Web interace, that aows them to speciy goa ocations and tasks that Xavier has to perorm there. The interace can be reached via Xavier s homepage at Xavier. Eventuay, Xavier wi be used to deiver memos, etters, and printouts between the oices in our buiding. 2 Our Distance Learning Approach We have deveoped GROW-BW, an unsupervised, passive distance earning agorithm that uses an extension o the Baum-Wech (BW) agorithm [Rabiner, 1986]. GROW- BW is an eicient agorithm that does not aect the other components o the robot system (except by making them operate more reiaby) and can tune the initia ( actory programmed ) sensor and actuator modes to better match the environment o the robot whie it earns the distances

3 r r r r Figure 4: Four states representing one ocation 4 improve the structure Figure 3: Overview o the GROW-BW agorithm (despite the act that GROW-BW never knows the ground truth about what the sensors were actuay observing). Fur- thermore, it can take additiona knowedge into account, i avaiabe, such as equaity constraints between the engths o corridors, bounds on the possibe corridor engths, or subjective probabiity distributions over them. corresponds to a corridor junction Topoogica Map Distance Mode Sensor Modes Actuator Modes Instead o earning an exact corridor ength, GROW-BW earns a probabiity distribution over the possibe engths, which is more robust to sensor and actuator noise. Formay, or each corridor segment c, GROW-BW earns a probabiity distribution p c over the possibe engths o the corridor [ min (c) max (c)], where min (c) and max (c) are the minima and maxima bounds on the ength o the corridor segment and where ength reers to the perceived ength o the corridor, which 1 incudes the dead-reckoning error o the robot. Then, p c () is the probabiity with which GROW-BW beieves that the perceived ength o corridor c is. POMDP Mode Figure 3 iustrates the GROW-BW agorithm. First, a topoogica map, augmented 2 with sensor and actuator modes and an initia distance mode (e.g., a uniorm distribution 2 over the possibe corridor engths), is automaticay compied intoextended a Partiay Observabe Baum-Wech MarkovAgorithm Decision Process (POMDP) mode ( 1 ). This mode is used directy by our probabiistic panning [Koenig et a., improve 199] andthe navigation methods [Simmons and Koenig, 2 199] to direct the robot probabiities to a given goa ocation. Better distance, sensor, and actuator modes improve the navigation perormance o the robot. The robot Improved thereore improves POMDP itsmode modes rom experience using an extension o the Baum-Wech agorithm ( 2 ). The experience is given in orm o sequences o action and sensor reports (execution 3 traces) that are generated automaticay whenever the robot moves. The resuting POMDP has ess distance uncertainty and improved sensor and actuator modes Improved ( 3 ). Finay, Distance it may Mode be the case that max (c) min (c). Improved To avoid having Sensor to consider Modesa possibe engths initiay Improved (or in casesactuator where the given Modes bounds do not actuay incude the rea ength), we use a hi-cimbingtech- Figure : Corridor o ength 2 to 4 meters nique that iterativey changes the structure o the POMDP based on the resuts o the extended Baum-Wech agorithm ( 4 ). It starts with a sma bound max (c) and grows it i necessary unti there corresponds is a high probabiity to a that the rea corridor ength is contained corridor within junction the bounds. 3 The POMDP Mode POMDPs are popuar modes or optima decision making in uncertain conditions [Cassandra et a., 1994] [Parr and Russe, 199]. Our POMDP incorporates the distance uncertainty and the sensor and actuator modes o the robot. It is speciied as a inite set o states S, a set o actions VA(s) VA, or each state s S, that can be executed in Panning and Navigation Components that state, transition probabiities p(s s va) or a s s S and va VA(s) (the probabiity that the successor state is s i the robot executes action va in state s), and sensor probabiities p vs ( s) or a vs VS, F(vs), and s S (the Execution probabiitytraces that sensor vs reports eature when the robot is in state s). Each state encodes both the ocation and orientation o the robot. We discretize ocations with a resoution o one meter and orientations into the our compass directions (this assumes that corridors are straight and perpendicuar to each other). Right and et turn actions are deined or every state (Figure 4). Forward actions transition rom ocation to ocation, but are not deined or states that ace was. A actions are neary deterministic, but there is a sma chance that the robot ends up in any o the three unintended orientations (not shown in the igures). The POMDP is compied automaticay rom a topoogica map. The corridor part between two adjacent junctions in the topoogica map is modeed as sets o parae chains that share their irst and ast states (Figure ). Each chain cor-

4 responds to one o the possibe engths [ min (c) max (c)] or that stretch o corridor c. From each junction, orward actions have probabiistic outcomes according to the probabiities p c (). Each orward transition ater that is (neary) deterministic. Thus, our POMDP mode expicity modes distance uncertainty and diers in this respect rom a simiar mode by [Nourbakhsh et a., 199], that does not mode distances at a. It can thereore be quite arge; the sizes o our POMDPs are typicay on the order o thousands o states. It is possibe, however, to reduce the number o states required to mode a corridor c rom being quadratic in max (c) min (c) to being inear in max (c), at the cost o a oss in mode accuracy [Simmons and Koenig, 199]. 4 The Baum-Wech Agorithm The Baum-Wech agorithm [Rabiner, 1986] is a simpe expectation maximization (EM) agorithm or earning POMDPs rom observations. It is best known or its appication to speech recognition and handwriting recognition, but it has aso been appied in robotics, or exampe to interpret tee-operation commands [Hannaord and Lee, 1991; Yang et a., 1993]. In the oowing, we describe how we use the Baum-Wech agorithm to improve the initia POMDP. Whenever the robot moves, a sensor interpretation modue converts its continuous motion into discrete action reports and produces reports o high-eve eatures rom the raw sensor data. In the case o Xavier, or exampe, the sensor interpretation modue integrates data rom the whee encoders over time to produce a stream o discrete action reports (going orward one meter, turning et ninety degrees, and turning right ninety degrees). Simiary, sonar readings are bunded into three virtua sensors that report observations o was and openings o various sizes (sma, medium, and arge) in ront o Xavier and to its immediate et and right. An execution trace contains these action and sensor reports in chronoogica order. We use the Baum-Wech agorithm to estimate a POMDP that better its the given execution traces, in the sense that the probabiity with which the POMDP expains the sensor reports (given the action reports) is increased. The Baum- Wech agorithm operates as oows: It irst uses the given POMDP and a inormation contained in the execution traces to cacuate, or every point in time, a probabiity distribution over a states that represents the beie that the robot was in a certain state at a certain point in time. It then estimates an improved POMDP rom these probabiity distributions, using a maximum ikeihood approach. This estimation process is then repeated with the same execution traces and the improved POMDP unti some termination criterion is satisied. The run time o each iteration o the Baum-Wech agorithm is inear in the product o the tota ength o the given execution trace and the size o the POMDP, typicay being on the order o seconds to minutes or our appication. We have extended the Baum-Wech agorithm to address memory constraints and the probem that coecting training data is time consuming: The Baum-Wech agorithm has to run on-board the robot and shares its memory with many other processes that run concurrenty. To decrease the amount o memory that it requires, we use a siding time window on the execution trace. Time windows add a sma overhead to the run time and cause a sma oss in precision o the improved POMDP, but aow the memory requirements to be dynamicay scaed to the avaiabe memory. Given the reativey sow speed with which mobie robots can move, we aso want the Baum-Wech agorithm to earn good modes with as ew corridor traversas as possibe. To reduce the amount o training data that it needs to estimate good modes, we extended the earning agorithm to take advantage o avaiabe prior knowedge, such as geometrica constraints that can be deduced rom the topoogica map. One might know, or exampe, that two corridors are the same ength, because both are intersected orthogonay by the same pair o corridors. This decreases the number o parameters that have to be earned and thereore the amount o training data needed to prevent overitting. The origina Baum-Wech agorithm uses requency-based estimates, but these are not very reiabe when the execution traces are short. To understand why, consider the oowing anaogy: I a air coin is ipped once and comes up head, the requency-based estimate is that it aways comes up head. I this mode were used to predict uture coin ips, one woud be very surprised i the coin came up tais next time this woud be inconsistent with the earned mode. Our extended Baum-Wech agorithm soves this probem by using Bayes rue (Dirichet distributions) instead o requencies. For more detais and an empirica evauation o the extended Baum-Wech agorithm, see [Koenig and Simmons, 1996]. The GROW-BW Agorithm The Baum-Wech agorithm improves the probabiities o a POMDP, but never changes its structure (the number o states and their connectivity). This poses a probem, because the distance mode is party encoded in the structure o the POMDP: the possibe engths o a corridor are determined by the structure, whie the probabiity distribution over the possibe engths is determined by the probabiities. Consequenty, the Baum-Wech agorithm cannot assign a positive probabiity p c () to corridor engths [ min (c) max (c)] nor can it change the bounds. Thus, it cannot earn the rea corridor ength i the bounds are o but they might not be known. Guessing min (c) is easy:

5 The GROW-BW agorithm usesthe oowing parameters: X = ; Y = X; Z = , and P (0 1). In its simpest orm, it uses X = Y = Z = 0 and a sma positive vaue or P. 1. For each corridor c: set max(c) := min(c) + X + 1. (I a ower bound min(c) on the rea corridor ength is not known, use min(c) = 1.) 2. Compie a POMDP (see Section 3). 3. Use the extendedbaum-wech agorithm on the POMDP and the given execution traces to determine improved p c() or a corridors c and corridor engths with min(c) max(c) (see Section 4). 4. For each corridor c: i max (c) pc() Y P, then set max(c) := max(c) + Z I any max(c) was changed in Step 4, then go to Step 2, ese stop. Figure 6: The GROW-BW agorithm we can use the smaest positive ength according to our discretization granuarity. Guessing max (c) is harder: we coud, o course, guess a ridicuousy arge vaue, but this has the drawback that the POMDPs become very arge and the memory requirements o distance earning agorithms determine their tractabiity. Instead, we investigate earning agorithms that are abe to change the structure o the POMDP. Aternatives to the Baum-Wech agorithm or earning POMDPs are described by [Chrisman, 1992], [Stocke and Omohundro, 1993], and [McCaum, 199], among others. These agorithms are abe to change the structure o a POMDP, but have the disadvantage that they either require a arge amount o training data, earn task-speciic representations ony, or cannot utiize prior knowedge. Consequenty, we have designed a nove POMDP earning agorithm that we ca GROW-BW. GROW-BW achieves its power by utiizing the reguarities in the structure o our POMDP modes o the corridors. It takes advantage o the act that the Baum-Wech agorithm earns a good POMDP or the given structure, even i the structure is incorrect. This aows it to start with a sma POMDP, earn the best mode or that structure, see i the mode is good enough, and grow the mode i not. Initiay, GROW-BW guesses a sma upper bound max (c) on the rea corridor ength (Figure 6). It then compies a POMDP and uses the extended Baum-Wech agorithm to improve it. I the Baum-Wech agorithm indicates that it is ikey that the rea corridor ength is cose to the upper bound, GROW-BW increases the upper bound, adds a new parae chain to the corridor segment (Figure ), and repeats the procedure. In this way, i the initiay chosen upper bound was too sma, it can be increased to it the rea Figure 7: Exampe o myopic eects ength o the corridor. GROW-BW is a hi-cimbing agorithm and, thus, can suer rom myopic eects. Consider the most myopic version o GROW-BW, that uses the parameter vaues X = Y = Z = 0 (X is reated to the initia dierence between min (c) and max (c), Y is reated to the ranges o engths to consider when determining whether the mode is good enough, and Z is reated to how much to grow the mode at each step). To simpiy our argument, assume that a robot with (amost) perect sensors and actuators moves back and orth in the environment shown in Figure 7(A). I max (c) = 4 or a corridor pieces, then the best itting mode is the one where a traversed corridor pieces are our meters ong. (The robot expects to see a corridor opening every our meters, but sees them ony every eight meters. Thus, it cannot expain our observations on each round-trip, and no distance mode whose corridors are at most our meters ong can do better.) This eads GROW-BW to increase max (c) to ive or a traversed corridor segments. However, at this point the mode where a corridor segments are our meters ong is sti among the modes that, o a modes considered, expain the observations best (another such mode is the one where adjacent corridor pieces o the main corridor aternate between engths three and ive). I the Baum- Wech agorithm earns this mode, then GROW-BW stops without having earned the rea corridor engths. Note that we have constructed this exampe artiiciay the probem does not show up i the robot encounters both ends o the main corridor whie it moves orward and backward. Despite the theoretica imitations o hi-cimbing, our experience with GROW-BW shows that it appears to work we in practice. We attribute this to architectura eatures o buidings they are usuay constructed in a way that prevents peope rom getting ost, which appears to dampen myopic eects. However, it is possibe that a probem simiar to the one described coud show up in conjunction with oice doors aong a corridor. We thereore recommend using a ess myopic version o GROW-BW by setting the parameters X, Y, and possiby 8 meters Z to vaues that are arger than the typica distance between adjacent oice doors. Simiary, P (the probabiity threshod that triggers growing the mode) has to be chosen sma enough to prevent GROW-BW rom terminating prematurey. Other probems arise when the rea corridor ength is greater than max (c), since then the execution traces can be inconsistent with the POMDP, in the sense that the mode cannot ex-

6 Figure 8: Corridor with se-transitions pain the experience. One probem this might ead to is that the position estimation component o the navigation system may rue out a possibe ocations, eading the robot to become totay uncertain as to where it is. The robot then has to expicity reocaize itse, which may take a air amount o time. Another probem is that earning can no onger take pace. As an exampe, again consider the environment shown in Figure 7 and assume that the robot traversesthe main corridor rom beginning to end or a tota distance o 40 corresponds meters. This, to ahowever, is impossibe according to a mode corridor that assumes junction max (c) = 4 or a corridor pieces. We avoid both these probems by having the POMDP compier add se-transitions (with a sma probabiity Q) in both directions o the ongest chain in the POMDP representation o each corridor segment (Figure 8). In this way, a corridor engths with min (c) have positive probabiity. This does not mean, o course, that the GROW-BW agorithm is no onger needed. Using such a POMDP directy with the Baum-Wech agorithm woud not work very we i rea (c) max (c), because ony the probabiities p c () or min (c) max (c) can be speciied individuay. The probabiities or max (c) are exponentiay decreasing according to the oowing ormua: p c() = 1 min (c) 6 Experiments max (c) p c( ) (1 Q) Q max(c) We use the prototypica corridor environment shown in Figure 9(A) to iustrate the power o our earning agorithms. Remember that they discretize the possibe corridor engths with a precision o one meter. To match this assumption, a corridor engths in this environment are mutipes o one meter. In many ways, the environment is more compicated than what we have avaiabe in our buiding. It has many parae corridors and indistinguishabe junctions, which ampiies the perceptua aiasing probem. The experiment uses the rea-time Xavier simuator, a highy reaistic simuation o Xavier incuding noisy sensors and actuators, that has the exact same interace as Xavier itse, but aows us to make the experiments repeatabe. It is not based on the POMDP mode used or navigation and consequenty vioates the independence assumptions made by POMDP modes (just ike reaity). The earning agorithms can be used unchanged on Xavier itse. In this case, the execution traces are provided by Xavier instead o the simuator. We do not inorm the robot about its start ocation or orientation, its route, or its destination. Instead, we et it gain experience with the environment by guiding it through every corridor once, using two execution traces with dierent start ocations. The ony inormation that it has avaiabe is the topoogica map, the data rom its sensors, and the oowing obvious equaity constraints between corridor engths: (These constraints are not necessary or the earning agorithms, but they increase the quaity o the earned modes i the number o corridor traversas is sma [Koenig and Simmons, 1996].) rea(c 1) = rea(c 3) = rea(c 6) = rea(c 10) rea(c 2) = rea(c ) = rea(c 9) rea(c 4) = rea(c 7) = rea(c 8) rea(c 11) = rea(c 13) corresponds to a corridor junction rea(c 12) = rea(c 14) = rea(c 18) = rea(c 21) rea(c 1) = rea(c 19) rea(c 17) = rea(c 20) Given this inormation, the task o the robot is to annotate the topoogica map with distance inormation and to adapt its initia sensor and actuator modes to its environment. This earning task is particuary hard, since we assume that the robot does not even know its approximate start ocation or orientation. As a consequence, severa dierent routes can be consistent with the sensor data, especiay since the robot has noisy sensors and actuators and has no initia estimates o the corridor engths avaiabe. For exampe, the probabiity that the et and right virtua sensors overook a corridor junction is about ity percent. This reativey high probabiity is due to the sensors being quite conservative: they don t report eatures unti they have coected suicient evidence. Aso, since the virtua sensors are impemented as asynchronous processes, they sometimes do not report eatures in time. Our irst experiment uses the extended Baum-Wech agorithm directy. To make sure that it is abe to earn the rea corridor engths, we estimate the minima and maxima corridor engths cautiousy to guarantee that rea (c) [ min (c) max (c)]: we use min (c) = 2 meters and max (c) = 14 meters or every corridor piece c. The resuting POMDP has 6672 states and state transitions. Figure 9(B) depicts the corridor engths with the argest probabiity p c () in the earned mode: a 21 predicted corridor engths correspond to the rea corridor engths.

7 c 1 c 1 c 19 c 2 c 3 c 4 c 12 c 14 c 18 c 21 c c 6 c 7 c 17 c 20 Figure 9: Experimenta resuts Our second experiment uses GROW-BW with the parameters X = 0, Y = 0, Z = 1, P = Q = 0 0, and min (c) = 2 8 generate (or, synonymousy, expain) ong simuator exe- c they increased the probabiitywith which the POMDP coud meters or a corridor segments. That is, the initia estimate cution traces. Furthermore, both agorithms earned good or every corridor segment is max (c) = 3 meters and, (athough not perect) distance modes with ony one travercution i GROW-BW extends a corridor ength, it increases it by sa o each corridor: in a cases, they erred by ony one two meters. GROW-BW assumes a uniorm probabiity meter when they made a mistake. In genera, they can earn distribution over the possibe corridor engths. Given this - good distance modes with one to three corridor traversas, inormation, GROW-BWc needs 9 ony ourciterations 10 to converge. Figure 9(C) shows min (c) and max (c) or the ina Note that, athough the dead-reckoning error o our robot is depending on how conusing the corridor environment is. mode. The corresponding POMDP has ony 1176 states not overy arge, we cannot expect the earning agorithms and state transitions, and is thus much smaer than to earn a corridor engths perecty, because or exampe the POMDP rom our irst experiment (the POMDPs used the robot sometimes takes sharp and sometimes wide in the irst three iterations o (A) GROW-BW corridor are, environment o course, turns around corners which aects the distances traveed even smaer). This is the2 4case, because GROW-BW stops aong the corridors. to expand the upper bound o a corridor ength when it is suicienty sure that it is arger than the rea corridor ength (where the required amount o certainty is determined by c c c (B) actua (D) min distances: and max rea distances (c), and (GROW-BW): earned distances: min (c) argmax - (c) p c () The proba- the parameters Y and P). Because o the sma sizes o the POMDPs, GROW-BW is 1.84 times aster than the extended Baum-Wech agorithm, athough it has to ca the extended Baum-Wech agorithm repeatedy. biities p c () that GROW-BW earns are simiar to those earned in the irst experiment, and the corridor engths with the argest probabiity p c () are even identica: again, a corridor engths are earned correcty. 4 We repeated both experiments eight more times with dierent robot routes. The resuts are summarized in Tabe 1. Each corridor ength that, ater earning, does not have the argest probabiity among a possibe engths counts as one mistake in the coumn corridors. I severa corridor engths were constrained to be identica, we count ony one mistake per corridor group in the coumn groups. The POMDPs earned by GROW-BW were o the same quaity as the ones o the extended Baum-Wech agorithm: The earned sensors and actuator modes were simiar when we evauated them according to a) how much they reduced the positiona uncertainty o the robot and b) how much The experiments show that the sizes o the POMDPs produced by GROW-BW are roughy between our and six times smaer than the size o the POMDP that we used in conjunction with the extended Baum-Wech agorithm. As a resut, GROW-BW is amost two times aster than the extended Baum-Wech agorithm. 1 Thus, GROW-BW produces resuts simiar to those o the extended Baum-Wech agorithm, but works on much smaer POMDPs and thereore needs ess memory and oten ess run time. The eect is even more pronounced when the modes o our buiding are used, since they are much arger than the mode used here. We coud augment GROW-BW with a post-processing step that prunes the ina POMDP, thus making it even smaer. The corridor environment used in this exampe was extremey sma and thus one coud have used distance earning methods with a runtime that is exponentia in the tota ength o the execution traces, such as methods that 1 The seventh experiment in Tabe 1 is an exception. It contained a highy ambiguous execution trace and GROW-BW expanded the upper bound o one corridor up to a ength o 21(!) meters, which required 10 iterations. We coud not repicate this phenomenon when we used execution traces that traversed each corridor more than once.

8 Tabe 1: Comparison o GROW-BW with the extended Baum-Wech agorithm mistakes o mistakes o improvement improvement ext. Baum-Wech GROW-BW in the number in run time corridors groups corridors groups o states (out o 21) (out o 8) (out o 21) (out o 8) match the routes probabiisticay against the topoogica map (possiby combined with branch-and-bound methods to prune the search space). GROW-BW has two advantages over such methods: First, the mode that it earns (a POMDP) can directy be used by our probabiistic panning and navigation methods. Thus, there is no need or a mode transormation that might degrade the quaity o the mode. Second (and more importanty), the run-time o GROW-BW is ony inear in the ength o the execution trace. We have aso used GROW-BW to earn environments in which the successu execution o actions does not provide any inormation about the position o the robot, namey or earning the distances between adjacent oice doors and corridors in a ong haway that is traversed by a robot that does not know its starting position. 7 Extensions We have assumed that GROW-BW can be provided with a correct topoogica map. Athough this is a reaistic assumption or many robot earning scenarios, weakening it broadens the appication area o our agorithm. Consequenty, we are working on extending GROW-BW to be abe to correct sighty inaccurate topoogica maps. We are aso investigating whether it can be combined with the passive topoogica map earning approach by [Engeson and McDermott, 1992] to extend its appicabiity to scenarios where a quaitative map is not avaiabe at a. 8 Concusion In this paper, we have described GROW-BW, a distance earning agorithm that annotates a given topoogica map with distance inormation. GROW-BW uses an extension o the Baum-Wech agorithm as a subroutine. It is an unsupervised (does not require a teacher during earning) and passive (does not need to contro the robot at any time) earning method. GROW-BW overcomes the probem that the robot can never be sure about its ocation i it is not aowed to reduce its uncertainty by asking a teacher or executing ocaization actions. It has the advantage that the robot can be used immediatey to perorm navigation tasks, and autonomousy improves its perormance over time as it gains more experience with its environment, ocusing its attention to routes that are more reevant or its tasks. It works transparenty with the other components o the robot system, can adapt the actory programmed sensor and actuator modes to the environment o the robot whie it earns the distances, and is eicient. It uses siding time windows to minimize the amount o memory required, and as much or as itte additiona knowedge as is avaiabe to minimize the amount o experience required to earn good modes. It can utiize, or exampe, equaity constraints on the engths o two corridors, bounds on the possibe corridor engths, or subjective probabiity distributions over them. We demonstrated that GROW-BW can earn good distance modes with ony a sma amount o experience, oten with consideraby ess space and time than can the extended Baum-Wech agorithm, by itse. In concusion, GROW-BW earns quantitative inormation that is diicut to obtain rom humans (distances as we as sensor and actuator modes), but is abe to utiize a arge variety o quaitative (and quantitative) inormation that humans can easiy provide. In contrast, many other map earning approaches in the iterature attempt to earn maps rom scratch, not utiizing prior knowedge that is easiy avaiabe. Acknowedgements Thanks to Lonnie Chrisman, Richard Goodwin, Joseph O Suivan, and the rest o the Xavier group or hepu discussions on a variety o topics. This research was sponsored by the Wright Laboratory, Aeronautica Systems Center, Air Force Materie Command, USAF, and the Advanced Research Projects Agency (ARPA) under grant number F The views and concusions contained in this document are those o the authors and shoud not be interpreted as representing the oicia poicies, either expressed or impied, o the sponsoring organizations or the U.S. government.

9 Reerences (Basye et a., 1989) Basye, K.; Dean, T.; and Vitter, J.S Coping with uncertainty in map earning. In Proceedings o the Internationa Joint Conerence on Artiicia Inteigence (IJCAI) (Cassandra et a., 1994) Cassandra, A.R.; Kaebing, L.P.; and Littman, M.L Acting optimay in partiay observabe stochastic domains. In Proceedings o the Nationa Conerence on Artiicia Inteigence (AAAI) (Chrisman, 1992) Chrisman, L Reinorcement earning with perceptua aiasing: The perceptua distinctions approach. In Proceedings o the Nationa Conerence on Artiicia Inteigence (AAAI) (Dean et a., 1992) Dean, T.; Anguin, D.; Basye, K.; Engeson, S.; Kaebing, L.; Kokkevis, E.; and Maron, O Inerring inite automata with stochastic output unctions and an appication to map earning. In Proceedings o the Nationa Conerence on Artiicia Inteigence (AAAI) (Engeson and McDermott, 1992) Engeson, S.P. and Mc- Dermott, D.V Error correction in mobie robot map earning. In Proceedings o the IEEE Internationa Conerence on Robotics and Automation (Hannaord and Lee, 1991) Hannaord, B. and Lee, P Hidden Markov mode anaysis o orce/torque inormation in teemanipuation. The InternationaJourna o Robotics Research 10(): (Koenig and Simmons, 1996) Koenig, S. and Simmons, R.G Unsupervised earning o probabiistic modes or robot navigation. In Proceedings o the Internationa Conerence on Robotics and Automation. (Koenig et a., 199) Koenig, S.; Goodwin, R.; and Simmons, R.G Robot navigation with Markov modes: A ramework or path panning and earning with imited computationa resources. In Internationa Workshop on Reasoning with Uncertainty in Robotics. (Kuipers and Byun, 1988) Kuipers, B.J. and Byun, Y.-T A robust, quaitative method or robot spatia earning. In Proceedings o the Nationa Conerence on Artiicia Inteigence (AAAI) (Mataric, 1990) Mataric, M.J Environment earning using a distributed representation. In Proceedings o the IEEE Internationa Conerence on Robotics and Automation (McCaum, 199) McCaum, R.A Instance-based state identiication or reinorcement earning. In Advances in Neura Inormation Processing Systems 7. (Nourbakhsh et a., 199) Nourbakhsh, I.; Powers, R.; and Birchied, S Dervish: An oice-navigating robot. AI Magazine 16(2):3 60. (Parr and Russe, 199) Parr, R. and Russe, S Approximating optima poicies or partiay observabe stochastic domains. In Proceedings o the Internationa Joint Conerence on Artiicia Inteigence (IJ- CAI) (Rabiner, 1986) Rabiner, L.R An introduction to hidden Markov modes. IEEE ASSP Magazine (Simmons and Koenig, 199) Simmons, R. and Koenig, S Probabiistic robot navigation in partiay observabe environments. In Proceedings o the Internationa Joint Conerence on Artiicia Inteigence (IJ- CAI) (Stocke and Omohundro, 1993) Stocke, A. and Omohundro, S Hidden Markov mode induction by Bayesian mode merging. In Advances in Neura Inormation Processing Systems (Yang et a., 1993) Yang, J.; Xu, Y.; and Chen, C.S Hidden Markov mode approach to ski earning and its appication to teerobotics. In Proceedings o the IEEE Internationa Conerence on Robotics and Automation