Inferring Maps and Behaviors from Natural Language Instructions

Transcription

1 Inferring Maps and Behaviors from Naural Language Insrucions Felix Duvalle 1, Mahew R. Waler 2, Thomas Howard 2, Sachihra Hemachandra 2, Jean Oh 1, Seh Teller 2, Nicholas Roy 2, and Anhony Senz 1 1 Roboics Insiue, Carnegie Mellon Universiy, Pisburgh, PA, USA, {felixd,jeanoh,ony}@cmu.edu 2 CS & AI Lab, Massachuses Insiue of Technology, Cambridge, MA, USA, {mwaler,mhoward,sachih,eller,nickroy}@csail.mi.edu Absrac. Naural language provides a flexible, inuiive way for people o command robos, which is becoming increasingly imporan as robos ransiion o working alongside people in our homes and workplaces. To follow insrucions in unknown environmens, robos will be expeced o reason abou pars of he environmens ha were described in he insrucion, bu ha he robo has no direc knowledge abou. However, mos exising approaches o naural language undersanding require ha he robo s environmen be known a priori. This paper proposes a probabilisic framework ha enables robos o follow commands given in naural language, wihou any prior knowledge of he environmen. The novely lies in exploiing environmen informaion implici in he insrucion, hereby reaing language as a ype of sensor ha is used o formulae a prior disribuion over he unknown pars of he environmen. The algorihm hen uses his learned disribuion o infer a sequence of acions ha are mos consisen wih he command, updaing our belief as we gaher more meric informaion. We evaluae our approach hrough simulaion as well as experimens on wo mobile robos; our resuls demonsrae he algorihm s abiliy o follow navigaion commands wih performance comparable o ha of a fully-known environmen. 1 Inroducion Robos are increasingly performing collaboraive asks wih people a home, in he workplace, and oudoors, and wih his comes a need for efficien communicaion beween human and robo eammaes. Naural language offers an effecive means for unrained users o conrol complex robos, wihou requiring specialized inerfaces or exensive user raining. Enabling robos o undersand naural language insrucions would faciliae seamless coordinaion in humanrobo eams. However, inerpreing insrucions is a challenge, paricularly when he robo has lile or no prior knowledge of is environmen. In such cases, he The firs four auhors conribued equally o his paper.

2 2 F. Duvalle e al. annoaion o cone O o hydran O r back (o cone, o hydran ) R Uerance: cone go o he hydran behind he cone (a) Firs, we receive a verbal insrucion from he operaor. hydran samples (b) Nex, we infer he map disribuion from he uerance and prior observaions. acion acual hydran pose (c) We hen ake an acion (green), using he map and behavior disribuions. (d) This process repeas as he robo acquires new observaions, refining is belief. Fig. 1. Visualizaion of one run for he command go o he hydran behind he cone, showing he evoluion of our beliefs (he possible locaions of he hydran). The robo begins wih he cone in is field of view, bu does no know he hydran s locaion. robo should be capable of reasoning over he pars of he environmen ha are relevan o undersanding he insrucion, bu may no ye have been observed. Ofenimes, he command iself provides informaion abou he environmen ha can be used o hypohesize suiable world models, which can hen be used o generae he correc robo acions. For example, suppose a firs responder insrucs a robo o navigae o he car behind he building, where he car and building are ouside he robo s field-of-view and heir locaions are no known. While he robo has no a priori informaion abou he environmen, he insrucion conveys he knowledge ha here is likely one or more buildings and cars in he environmen, wih a leas one car being behind one of he buildings. The robo should be able o reason abou he car s possible locaion, and refine is prior as i carries ou he command (e.g., updae he car s possible locaion when i observes a building). This paper proposes a mehod ha enables robos o inerpre and execue naural language commands ha refer o unknown regions and objecs in he robo s environmen. We exploi he informaion implici in he user s command o learn an environmen model from he naural language insrucion, and hen solve for he policy ha is consisen wih he command under his world model. The robo updaes is inernal represenaion of he world as i makes new meric observaions (such as he locaion of perceived landmarks) and updaes is policy appropriaely. By reasoning and planning in he space of beliefs over objec

3 Inferring Maps and Behaviors from Naural Language Insrucions 3 locaions and groundings, we are able o reason abou elemens ha are no iniially observed, and robusly follow naural language insrucions given by a human operaor. More specifically, we describe in our approach (Secion 3) a probabilisic framework ha firs exracs annoaions from a naural language insrucion, consising of he objecs and regions described in he command and he given relaions beween hem (Fig. 1(a)). We hen rea hese annoaions as noisy sensor observaions in a mapping framework, and use hem o generae a disribuion over a semanic model of he environmen ha also incorporaes observaions from he robo s sensor sreams (Fig. 1(b)). This prior is used o ground he acions and goals from he command, resuling in a disribuion over desired behaviors. This is hen used o solve for a policy ha yields an acion ha is mos consisen wih he command, under he map disribuion so far (Fig. 1(c)). As he robo ravels and senses new meric informaion, i updaes is map prior and inferred behavior disribuion, and coninues o plan unil i reaches is desinaion (Fig. 1(d)). This framework in oulined in Figure 2. We evaluae our algorihm (Secion 4) hrough a series of simulaion-based and physical experimens on wo mobile robos ha demonsrae is effeciveness a carrying ou navigaion commands, as well as highligh he condiions under which i fails. Our resuls indicae ha exploiing he environmen knowledge implici in he insrucion enables us o predic a world model upon which we can successfully esimae he acion sequence mos consisen wih he command, approaching performance levels of complee a priori environmen knowledge. These resuls sugges ha uilizing informaion implicily conained in naural language insrucions can improve collaboraion in human-robo eams. 2 Relaed Work Naural language has proven o be effecive for commanding robos o follow roue direcions [1 5] and manipulae objecs [6]. The majoriy of prior approaches require a complee semanically-labeled environmen model ha capures he geomery, locaion, ype, and label of objecs and regions in he environmen [2, 5, 6]. Undersanding insrucions in unknown environmens is ofen more challenging. Previous approaches have eiher used a parser ha maps language direcly o plans [1, 3, 4], or rained a policy ha reasons abou uncerainy and can backrack when needed [7]. However, none of hese approaches direcly use he informaion conained in he insrucion o inform heir environmen represenaion or reason abou is uncerainy. We insead rea language as a sensor ha can be used o generae a prior over he possible locaions of landmarks by exploiing he informaion implicily conained in a given insrucion. Sae-of-he-ar semanic mapping frameworks focus on using he robo s sensor observaions o updae is represenaion of he world [8 10]. Some approaches [10] inegrae language descripions o improve he represenaion bu do no exend he maps based on naural language. Our approach reas naural language as anoher sensor and uses i o exend he spaial represenaion by

4 4 F. Duvalle e al. Annoaion Inference Annoaions Semanic Mapping Observaions Disribuion over maps "Go o he hydran behind he cone" Behavior Inference Policy Planner Acions Behavior Grounding Fig. 2. Framework ouline. adding boh opological and meric informaion, which is hen used for planning. Williams e al. [11] use a cogniive archiecure o add unvisied locaions o a parial map. However, hey only reason abou opological relaionships o unknown places, do no mainain muliple hypohesis, and make srong assumpions abou he environmen limiing he applicabiliy o real robo sysems. In conras, our approach reasons boh opologically and merically abou objecs and regions, and can deal wih ambiguiy, which allows us o operae in challenging environmens. As we reason in he space of disribuions over possible environmens, we draw from sraegies in he belief-space planning lieraure. Mos imporanly, we represen our belief using samples from he disribuion, similar o work by Pla e al. [12]. Insead of solving he complee Parially-Observable Markov Decision Process (POMDP), we insead seek efficien approximae soluions [13, 14]. 3 Technical Approach Our goal is o infer he mos likely fuure robo rajecory x +1:T up o ime horizon T, given he hisory of naural language uerances Λ, sensor observaions z, and odomery u (we denoe he hisory of a variable up o ime wih a superscrip): arg max p ( x +1:T Λ, z, u ). (1) x +1:T R n Inferring he maximum a poseriori rajecory (1) for a given naural language uerance is challenging wihou knowledge of he environmen for all bu rivial applicaions. To overcome his challenge, we inroduce a laen random variable S ha represens he world model as a semanic map ha encodes he locaion, geomery, and ype of he objecs wihin he environmen. This allows us o facor he disribuion as: arg max p(x +1:T S, Λ, z, u ) p(s Λ, z, u ) ds. (2) x +1:T R n S

5 Inferring Maps and Behaviors from Naural Language Insrucions 5 As we mainain he disribuion in he form of samples S (i), his simplifies o: arg max x +1:T R n i p(x +1:T S (i), Λ, z, u ) p(s (i) Λ, z, u ). (3) Based upon he robo s sensor and odomery sreams and he user s naural language inpu, our algorihm learns his disribuion online. We accomplish his hrough a filering process whereby we firs infer he disribuion over he world model S based upon annoaions idenified from he uerance Λ (second erm in he inegral in (2)), upon which we hen infer he consrains on he robo s acion ha are mos consisen wih he command given he curren map disribuion. A his poin, he algorihm solves for he mos likely policy under he learned disribuion over rajecories (firs erm in he inegral in (2)). During execuion, we coninuously updae he semanic map S as sensor daa arrives and refine he policy according o he re-grounded language. To efficienly conver unsrucured naural language o symbols ha represen he spaces of annoaions and behaviors, we use he Disribued Correspondence Graph (DCG) model [5]. The DCG model is a probabilisic graphical model composed of random variables ha represen language λ, groundings γ, and correspondences beween language and groundings φ and facors f. Each facor f ij in he DCG model is influenced by he curren phrase λ i, correspondence variable φ ij, grounding γ ij, and child phrase groundings γ cij. The parameers in each log-linear model υ are rained from a parallel corpus of labeled examples for annoaions and behaviors in he conex of a world model Υ. In each, we search for he unknown correspondence variables ha maximize he produc of facors: arg max φ Φ ) f ij (φ ij, γ ij, γ cij, λ i, Υ, υ. (4) i j An illusraion of he graphical model used o represen Equaion 4 is shown in Figure 3. In his figure, he black squares, whie circles, and gray circles represen facors, unknown random variables, and known random variables respecively. I is imporan o noe ha each phrase can have a differen number of verically aligned facors if he symbols used o ground paricular phrases differ. In his paper we use a binary correspondence variable o indicae he expression or rejecion of a paricular grounding for a phrase. We consruc he symbols used o represen each phrase using only he groundings wih a rue correspondence and ake he meaning of a uerance as he symbol inferred a he roo of parse ree. Figure 2 illusraes he archiecure of he inegraed sysem ha we consider for evaluaion. Firs, he naural language undersanding module infers a disribuion over annoaions conveyed by he uerance (Annoaion Inference). The semanic map learning mehod hen uses his informaion in conjuncion wih he prior annoaions and sensor measuremens o build a probabilisic model of objecs and heir relaionships in he environmen (Semanic Mapping). We hen formulae a disribuion over robo behaviors using he uerance and he semanic map disribuion (Behavior Inference). Nex, he planner compues a

6 6 F. Duvalle e al. γ 1j γ 2j γ 3j γ 4j γ 5j γ 6j φ 1j φ 2j φ 3j φ 4j φ 5j φ 6j γ 12 γ 22 γ 32 γ 42 γ 52 γ 62 φ 12 φ 22 φ 32 φ 42 φ 52 φ 62 γ 11 γ 21 γ 31 γ 41 γ 51 γ 61 φ 11 φ 21 φ 31 φ 41 φ 51 φ 61 λ 1 λ 2 λ 3 λ 4 λ 5 λ 6 go o he hydran behind he cone Fig. 3. A DCG used o infer annoaions or behaviors from he uerance go o he hydran behind he cone. The facors f ij, groundings γ ij, and correspondence variables φ ij are funcions of he symbols used o represen annoaions and behaviors. policy from his disribuion over behaviors and maps (Policy Planner). As he robo makes more observaions or receives addiional human inpu, we repea he las hree seps o coninuously updae our undersanding of he mos recen uerance. We now describe in more deail each of hese componens. 3.1 Annoaion Inference The space of symbols used o represen he meaning of phrases in map inference is composed of objecs, regions, and relaions. Since no world model is assumed when inferring linguisic annoaions from he uerance, he space of objecs is equal o he number of possible objec ypes ha could exis in he scene. Regions are some porion of sae-space ha is ypically associaed wih a relaionship o some objec. Relaions are a paricular ype of associaion beween a pair of objecs or regions (e.g., fron, back, near, far). Since any se of objecs, regions, and relaions may be inferred as par of he symbol grounding, he size of he space of groundings for map inference grows as he power se of he sum of hese symbols. We use he rained DCG model o infer a se of annoaions α from he posiively expressed groundings a he roo of he parse ree. 3.2 Semanic Mapping We rea he annoaions inferred from he uerance as noisy observaions α ha specify he exisence and spaial relaions beween labeled objecs in he environmen. We use hese observaions along wih hose from he robo s sensors o learn he disribuion over he semanic map S = {G, X }: p(s Λ, z, u ) p(s α, z, u ) (5a) = p(g, X α, z, u ) (5b) = p(x G, α, z, u )p(g α, z, u ), (5c)

7 Inferring Maps and Behaviors from Naural Language Insrucions 7 where he las line expresses he facorizaion ino a disribuion over he environmen opology (graph G ) and a condiional disribuion over he meric map (X ). Owing o he combinaorial number of candidae opologies [10], we employ a sample-based approximaion o he laer disribuion and model he condiional poserior over poses wih a Gaussian, paramerized in he canonical form. In his manner, each paricle S (i) opology G (i) = {G (i), X (i), w (i), a Gaussian disribuion over he poses X (i) } consiss of a sampled, and a weigh w (i). We noe ha his model is similar o ha of Waler e al. [10], hough in his work we don rea he labels as being uncerain. To efficienly mainain he semanic map disribuion over ime as he robo receives new annoaions and observaions during execuion, we use a Rao- Blackwellized paricle filer [15]. This filering process has wo key seps: Firs, he algorihm proposes updaes o each sampled opology ha express objec observaions and annoaions inferred from he uerance. Nex, he algorihm uses he proposed opology o perform a Bayesian updae o he Gaussian disribuion over he node (objec) poses, and updaes he paricle weighs so as o approximae he arge disribuion. We perform his process for each paricle and repea hese seps a each ime insance. The following paragraphs describe each operaion in more deail. During he proposal sep, we firs augmen each sample opology wih an addiional node and edge ha model he robo s moion u, resuling in a new opology S (i). We hen sample modificaions o he graph (i) based upon he mos recen annoaions (α ) and sensor observaions (z ): p(s (i) S (i) 1, α, z, u ) = p( (i) α S (i), α ) p( (i) z S (i), z ) p(s (i) = { (i) α, (i) S (i) z } 1, u ). (6) This updaes he proposed graph opology S (i) wih he graph modificaions (i) o yield he new semanic map S (i). The updaes can include he addiion of nodes o he graph represening newly hypohesized or observed objecs. They also may include he addiion of edges beween nodes o express spaial relaions inferred from observaions or annoaions. The graph modificaions are sampled from wo similar bu independen proposals for annoaions and observaions in a muli-sage process: p( (i) α S (i) p( (i) z S (i), α ) = j, z ) = j p( (i) α,j S (i), α,j ) (7a) p( (i) z,j S (i), z,j ). (7b) For each language annoaion componen α,j, we use a likelihood model over he spaial relaion o sample landmark and figure pairs for he grounding (7a). This model employs a Dirichle process prior ha accouns for he fac ha he annoaion may refer o exising or new objecs. If he landmark and/or he figure are sampled as new objecs, we add hese objecs o he paricle, and creae an edge beween hem. We also sample he meric consrain associaed wih his

8 8 F. Duvalle e al. edge, based on he spaial relaion. Similarly, for each objec z,j observed by he robo, we sample a grounding from he exising model of he world (7b). We add a new consrain o he objec when he grounding is valid, and creae a new objec and consrain when i is no. Afer proposing modificaions o each paricle, we perform a Bayesian updae o heir Gaussian disribuion. We hen re-weigh each paricle by aking ino accoun he likelihood of generaing language annoaions, as well as posiive and negaive observaions of objecs: w (i) = p(z, α S 1 ) w (i) 1 = p(α S 1 ) p(z S 1 ) w (i) 1. (8) For annoaions, we use he naural language grounding likelihood under he map a he previous ime sep. For objec observaions, we use he likelihood ha he observaions were (or were no) generaed based upon he previous map. This has he effec of down-weighing paricles for which he observaions are unexpeced. We normalize he weighs and re-sample if heir enropy exceeds a hreshold [15]. 3.3 Behavior Inference Given he uerance and he semanic map disribuion, we now infer a disribuion over robo behaviors. The space of symbols used o represen he meaning of phrases in behavior inference is composed of objecs, regions, acions, and goals. Objecs and regions are defined in he same manner as in map inference, hough he presence of objecs is a funcion of he inferred map. Acions and goals specify how he robo should perform a behavior o he planner. Since any se of acions and goals can be expressed o he planner, he space of groundings also grows as he power se of he sum of hese symbols. For he experimens discussed laer in Secion 4 we assume a number of objecs, regions, acions, and goals ha are proporional o he number of objecs in he hypohesized world model. We use he rained DCG model o infer a disribuion of behaviors β from he posiively expressed groundings a he roo of he parse ree. 3.4 Policy Planner Since i is difficul o boh represen and search he coninuum for a rajecory ha bes reflecs he enire insrucion in he conex of he semanic map, we insead learn a policy ha predics a single acion ha maximizes he one-sep expeced value of aking he acion a from he robo s curren pose x. This process is repeaed unil he policy declares i is done following he command using a separae acion a sop. As he robo moves in he environmen, i builds and updaes a graph of locaions i has previously visied, as well as froniers ha lie a he edge of explored space. This graph is used o generae a candidae se of acions ha consiss of all fronier nodes F as well as previously-visied nodes V ha he robo can ravel o nex: A = F V {a sop }. (9)

9 Inferring Maps and Behaviors from Naural Language Insrucions 9 (a) = 0 (b) = 4 (c) = 8 Fig. 4. Visualizaion of he value funcion over ime for he command go o he hydran behind he cone, where he riangle denoes he robo, squares denoe observed cones, and circles denoe hydrans ha are sampled (empy) and observed (filled). The robo sars off having observed he wo cones, and hypohesizes possible hydrans ha are consisen wih he command (a). The robo firs moves owards he lef cluser, bu afer no observing he hydran, he map disribuion peaks a he righ cluser (b). The robo hen moves righ and observes he acual hydran (c). The policy selecs he acion wih he maximum value under our value funcion: π(x ) = arg max V (x, a ). (10) a A The value of a paricular acion is a funcion of he behavior and he semanic map, which are no observable. Insead, we solve his using he QMDP algorihm [13] by aking he expeced value under he disribuions of he semanic map S and inferred behavior β j : V (x, a ) S (i) V β j ( x, a ; S (i), β j ) p ( βj S (i) ) ( (i)) p S. (11) There are many choices for he paricular value funcion o use, in his work we define he value for a semanic map paricle and behavior as an analogue of he MDP cos-o-go: V ( x, a ; S (i), β j ) = γ d(a,g s), (12) where γ is he MDP discoun facor and d is he Euclidean disance beween he acion node and he behavior s goal posiion g s. Our belief space policy π hen picks he maximum value acion. We re-evaluae his value funcion as he semanic map and behavior disribuions improve wih new observaions. Figure 4 demonsraes he evoluion of he value funcion over ime. 4 Resuls To analyze our approach, we firs evaluae he abiliy of our naural language undersanding module o independenly infer he correc annoaions and behav-

10 10 F. Duvalle e al. Table 1. Naural language undersanding resuls wih 95% confidence inervals. Model Accuracy (%) Training Time (sec) Inference Time (sec) Annoaion (10.83) (7.55) 0.44 (0.03) Behavior (6.83) (1.02) 0.05 (0.00) iors for given uerances. Nex, we analyze he effeciveness of our end-o-end framework hrough simulaions ha consider environmens and commands of varying complexiy, and differen amouns of prior knowledge. We hen demonsrae he uiliy of our approach in pracice using experimens run on wo mobile robo plaforms. These experimens provide insighs ino our algorihm s abiliy o infer he correc behavior in he presence of unknown and ambiguous environmens. 4.1 Naural Language Undersanding We evaluae he performance of our naural language undersanding componen in erms of he accuracy and compuaional complexiy of inference using holdou validaion. In each experimen, he corpus was randomly divided ino separae raining and es ses o evaluae wheher he model can recover he correc groundings from he uerance and he world model. Each model used 13,716 feaures ha checked for he presence of words, properies of groundings and correspondence variables, and relaionships beween curren and child groundings and searched he model wih a beam widh of 4. We conduced 8 experimens for each model ype using a corpus of 39 labeled examples of insrucions and groundings. For annoaion inference we assumed ha he space of groundings for every phrase is represened by 8 objec ypes, 54 regions, and 432 relaions. For behavior inference we assumed ha noun and preposiions ground o hypohesized objecs or regions while verbs ground o 2 possible acions, 3 possible modes, goal regions, and consrain regions. In he example illusraed in Fig. 3 wih a world model composed of seven hypohesized objecs he annoaion inference DCG model conained 5,934 random variables and 2,964 facors while he behavior inference DCG model conained 772 random variables and 383 facors. In each experimen 33% of he labeled examples in he corpus were randomly seleced for he holdou. The mean number of log-linear model raining examples exraced from he 26 randomly seleced labeled examples for annoaion and behavior inference was 83,547 and 9,224 respecively. Table 1 illusraes he saisics for he annoaion and behavior models. This experimen demonsraes ha we are able o learn many of he relaionships beween phrases, groundings, and correspondences wih a limied number of labeled insrucions, and infer a disribuion of symbols quickly enough for he proposed archiecure. As expeced he raining and inference ime for he annoaion model is much higher because of he difference in he complexiy of symbols. This is accepable for our framework since he annoaion model is only

11 Inferring Maps and Behaviors from Naural Language Insrucions 11 Table 2. Mone Carlo simulaion resuls wih 1σ confidence inervals (Hydran, Cone). Success Rae (%) Disance (m) World Range (m) Relaion Known Ours Known Ours 1H, 1C 3.0 null (1.69) (7.90) 1H, 1C 3.0 behind (1.69) (7.02) 1H, 2C 3.0 null (1.38) (18.50) 1H, 2C 3.0 behind (1.38) (29.66) 2H, 1C 3.0 null (1.81) (10.32) 2H, 1C 3.0 behind (1.86) (10.23) 2H, 1C 5.0 neares (1.54) (5.76) used once o infer a se of observaions, while he behavior model is used coninuously o process he map disribuions as new observaions are inegraed. 4.2 Mone Carlo Simulaions Nex, we evaluae he enire framework hrough an exended se of simulaions in order o undersand how he performance varies wih he environmen configuraion and he command. We consider four environmen emplaes, wih differen numbers of figures (hydrans) and landmarks (cones). For each configuraion, we sample en environmens, each wih differen objec poses. For hese environmens, we issued hree naural language insrucions go o he hydran, go o he hydran behind he cone, and go o he hydran neares o he cone. We noe ha hese commands were no par of he corpus ha we used o rain he DCG model. Addiionally, we considered six differen seings for he robo s sensing range (2 m, 3 m, 5 m, 10 m, 15 m, and 20 m) and performed approximaely 100 simulaions for each combinaion of environmen, command, and range. As a ground-ruh baseline, we performed en runs of each configuraion wih a compleely known world model. Table 2 presens he success rae and disance raveled by he robo for hese 100 simulaion configuraions. We considered a run o be successful if he planner sops wihin 1.5 m of he inended goal. Comparing agains commands ha do no provide a relaion (i.e., go o he hydran ), he resuls demonsrae ha our algorihm achieves greaer success and yields more efficien pahs by aking advanage of relaions in he command (i.e., go o he hydran behind he cone ). This is apparen in environmens consising of a single figure (hydran) as well as more ambiguous environmens ha consis of wo figures. Paricularly elling is he variaion in performance as a resul of differen sensing range. Figure 5 shows how success rae increases and disance raveled decreases as he robo s sensing range increases, quickly approaching he performance of he sysem when i begins wih a compleely known map of he environmen. One ineresing failure case is when he robo is insruced o go o he hydran neares o he cone in an environmen wih wo hydrans. In insances where he robo sees a hydran firs, i hypohesizes he locaion of he cone, and

12 12 F. Duvalle e al. Disance (m) Sensor Range (m) 100 Disance (m) Uknown Map Known Map Sensor Range (m) 100 Success (%) Sensor Range (m) Success (%) 50 0 Unknown Map Sensor Range (m) Fig. 5. Disance raveled (op) and success rae (boom) as a funcion of he sensor range for he commands go o he hydran behind he cone (lef) and go o he hydran neares o he cone (righ) in simulaion. hen idenifies he observed hydrans and hypohesized cones as being consisen wih he command. Since he robo never acually confirms he exisence of he cone in he real world, his resuls in he incorrec hydran being labeled as he goal. 4.3 Physical Experimens We applied our approach o wo mobile robos, a Husky A200 mobile robo (Fig. 6(a)) and an auonomous roboic wheelchair [16] (Fig. 6(b)). The use of boh plaforms demonsraes he applicaion of our algorihm o mobile robos wih differen vehicle configuraions, underlying moion planners, and sensor configuraions. The acions deermined by he planner are ranslaed ino liss of waypoins ha are handled by each robo s moion planner. We used AprilTag fiducials [17] o deec and esimae he relaive pose of objecs in he environmen, subjec o self-imposed angular and range resricions. In each experimen, a human operaor issues naural language commands in he form of ex ha involve (possibly null) spaial relaions beween one or wo objecs. The resuls ha follow involve he commands go o he hydran, go o he hydran behind he cone, and go o he hydran neares o he cone. As wih he simulaion-based experimens, hese insrucions did no mach hose from our raining se. For each of hese commands, we consider differen environmens by varying he number and posiion of he cones and hydrans and by changing he robo s sensing range. For each configuraion of he environmen, command, and sensing range, we perform en rials wih our algorihm. For a ground-ruh baseline, we perform an addiional run wih a compleely known world model. We consider a run o be a success when he robo s final desinaion is wihin 1.5 m of he inended goal. Table 3 presens he success rae and disance raveled by he wheelchair for hese experimens. Compared o he scenario in which he command does no

13 Inferring Maps and Behaviors from Naural Language Insrucions (a) Husky 13 (b) Wheelchair Fig. 6. The seup for he experimens wih he (a) Husky and (b) wheelchair plaforms. Table 3. Experimenal resuls wih 1σ confidence inervals (Hydran, Cone). Success Rae (%) World 1H, 1H, 1H, 2H, 2H, 2H, 1C 1C 2C 1C 1C 1C Disance (m) Range (m) Relaion Known Ours Known null behind behind behind neares neares Ours (7.20) (3.41) (2.08) (1.38) (0.39) (0.58) provide a relaion (i.e., go o he hydran ), we find ha our algorihm is able o ake advanage of available relaions ( go o he hydran behind he cone ) o yield behaviors closer o ha of ground ruh. The resuls are similar for he Husky plaform, which resuled in an 83.3% success rae when commanded o go o he hydran behind he cone in an environmen wih one cone and one hydran. These resuls demonsrae he usefulness of uilizing all of he informaion conained in he insrucion, such as he relaion beween various landmarks in he environmen ha can be helpful during navigaion. The robo rials exhibied a similar failure mode as he simulaed experimens: if he environmen conains wo figures (hydrans) and he robo only deecs one, he semanic map disribuion hen hypohesizes he exisence of cones in fron of he hydran, which leads o a behavior disribuion peaked around his goal and plans ha do no look for he possibiliy of anoher hydran in he environmen. As expeced, his effec is mos pronounced wih shorer sensing ranges (e.g., a 3 m sensing range for he command go o he hydran neares o he cone resuled in he robo reaching he goal in only half of he rials compared o a 4 m sensing range).

14 14 F. Duvalle e al. 5 Conclusions Enabling robos o reason abou pars of he environmen ha have no ye been visied solely from a naural language descripion serves as one sep owards effecive and naural collaboraion in human-robo eams. By reaing language as a sensor, we are able o pain a rough picure of wha he unvisied pars of he environmen could look like. We uilize his informaion during planning, and updae our belief wih acual sensor informaion during ask execuion. Our approach explois he informaion implicily conained in he language o infer he relaionship beween objecs ha may no be iniially observable, wihou having o consider hose annoaions as a separae uerance. By learning a disribuion over he map, we generae a useful prior ha enables he robo o sample possible hypoheses, represening differen environmen possibiliies ha are consisen wih boh he language and he available sensor daa. Learning a policy ha reasons in he belief space of hese samples achieves a level of performance ha approaches full knowledge of he world ahead of ime. We have evaluaed our approach in simulaion and on wo robo plaforms. These evaluaions provide a preliminary validaion of our framework. Fuure work will es he algorihm s abiliy o scale o larger environmens (e.g., rooms and hallways), and handle uerances ha presen complex relaions and more deailed behaviors han hose considered so far. Addiionally, we will focus on handling sreams of commands, including hose ha are given during execuion (e.g., go o he oher cone uered as he robo is moving owards he wrong cone). An addiional direcion for following work is o explicily reason over exploraory behaviors ha ake informaion gahering acions o resolve uncerainy in he map. Currenly, any exploraion on he par of he algorihm is opporunisic, which migh no be sufficien in more challenging scenarios. Acknowledgmens The auhors would like o hank Bob Dean for his help wih he Husky plaform. This work was suppored in par by he Roboics Consorium of he U.S. Army Research Laboraory under he Collaboraive Technology Alliance Program, Cooperaive Agreemen W911NF Bibliography [1] MacMahon, M., Sankiewicz, B., Kuipers, B.: Walk he alk: Connecing language, knowledge, and acion in roue insrucions. In: Proc. Na l Conf. on Arificial Inelligence (AAAI). (2006) [2] Kollar, T., Tellex, S., Roy, D., Roy, N.: Toward undersanding naural language direcions. In: Proc. In l. Conf. on Human-Robo Ineracion. (2010) [3] Chen, D.L., Mooney, R.J.: Learning o inerpre naural language navigaion insrucions from observaions. In: Proc. Na l Conf. on Arificial Inelligence (AAAI). (2011)

15 Inferring Maps and Behaviors from Naural Language Insrucions 15 [4] Mauszek, C., Herbs, E., Zelemoyer, L., Fox, D.: Learning o parse naural language commands o a robo conrol sysem. In: Proc. In l. Symp. on Experimenal Roboics (ISER). (2012) [5] Howard, T., Tellex, S., Roy, N.: A naural language planner inerface for mobile manipulaors. In: Proc. IEEE In l Conf. on Roboics and Auomaion (ICRA). (2014) [6] Tellex, S., Kollar, T., Dickerson, S., Waler, M.R., Banerjee, A.G., Teller, S., Roy, N.: Undersanding naural language commands for roboic navigaion and mobile manipulaion. In: Proc. Na l Conf. on Arificial Inelligence (AAAI). (2011) [7] Duvalle, F., Kollar, T., Senz, A.: Imiaion learning for naural language direcion following hrough unknown environmens. In: Proc. IEEE In l Conf. on Roboics and Auomaion (ICRA). (2013) [8] Zender, H., Marínez Mozos, O., Jensfel, P., Kruijff, G., Burgard, W.: Concepual spaial represenaions for indoor mobile robos. Roboics and Auonomous Sysems (2008) [9] Pronobis, A., Marínez Mozos, O., Capuo, B., Jensfel, P.: Muli-modal semanic place classificaion. In l J. of Roboics Research (2010) [10] Waler, M.R., Hemachandra, S., Homberg, B., Tellex, S., Teller, S.: Learning semanic maps from naural language descripions. In: Proc. Roboics: Science and Sysems (RSS). (2013) [11] Williams, T., Canrell, R., Briggs, G., Schermerhorn, P., Scheuz, M.: Grounding naural language references o unvisied and hypoheical locaions. In: Proc. Na l Conf. on Arificial Inelligence (AAAI). (2013) [12] Pla, R., Kaelbling, L., Lozano-Perez, T., Tedrake, R.: Simulaneous localizaion and grasping as a belief space conrol problem. In: Proc. In l. Symp. of Roboics Research (ISRR). (2011) [13] Liman, M.L., Cassandra, A.R., Kaelbling, L.P.: Learning policies for parially observable environmens: Scaling up. In: Proc. In l Conf. on Machine Learning (ICML). (1995) [14] Roy, N., Burgard, W., Fox, D., Thrun, S.: Coasal navigaion-mobile robo navigaion wih uncerainy in dynamic environmens. In: Proc. IEEE In l Conf. on Roboics and Auomaion (ICRA). (1999) [15] Douce, A., de Freias, N., Murphy, K., Russell, S.: Rao-Blackwellised paricle filering for dynamic Bayesian neworks. In: Proceedings of he Conference on Uncerainy in Arificial Inelligence (UAI). (2000) [16] Hemachandra, S., Kollar, T., Roy, N., Teller, S.: Following and inerpreing narraed guided ours. In: Proc. IEEE In l Conf. on Roboics and Auomaion (ICRA). (2011) [17] Olson, E.: AprilTag: A robus and flexible visual fiducial sysem. In: Proc. IEEE In l Conf. on Roboics and Auomaion (ICRA). (May 2011)