Learning Semantic Maps from Natural Language Descriptions

Transcription

1 Roboics: Science and Sysems 2013 Berlin, Germany, June 24-28, 2013 Learning Semanic Maps from Naural Language Descripions Mahew R. Waler, 1 Sachihra Hemachandra, 1 Bianca Homberg, Sefanie Tellex, and Seh Teller Compuer Science and Arificial Inelligence Laboraory Massachuses Insiue of Technology Cambridge, MA USA {mwaler, sachih, bhomberg, sefie10, eller}@csail.mi.edu Absrac This paper proposes an algorihm ha enables robos o efficienly learn human-cenric models of heir environmen from naural language descripions. Typical semanic mapping approaches augmen meric maps wih higher-level properies of he robo s surroundings (e.g., place ype, objec locaions), bu do no use his informaion o improve he meric map. The novely of our algorihm lies in fusing high-level knowledge, conveyed by speech, wih meric informaion from he robo s low-level sensor sreams. Our mehod joinly esimaes a hybrid meric, opological, and semanic represenaion of he environmen. This semanic graph provides a common framework in which we inegrae conceps from naural language descripions (e.g., labels and spaial relaions) wih meric observaions from low-level sensors. Our algorihm efficienly mainains a facored disribuion over semanic graphs based upon he sream of naural language and low-level sensor informaion. We evaluae he algorihm s performance and demonsrae ha he incorporaion of informaion from naural language increases he meric, opological and semanic accuracy of he recovered environmen model. The kichen is down he hall Fig. 1. A user giving a our o a roboic wheelchair designed o assis residens in a long-erm care faciliy. I. INTRODUCTION Unil recenly, robos ha operaed ouside he laboraory were limied o conrolled, prepared environmens ha explicily preven ineracion wih humans. There is an increasing demand, however, for robos ha operae no as machines used in isolaion, bu as co-inhabians ha assis people in a range of differen aciviies. If robos are o work effecively as our eammaes, hey mus become able o efficienly and flexibly inerpre and carry ou our requess. Recognizing his need, here has been increased focus on enabling robos o inerpre naural language commands [1, 2, 3, 4, 5]. This capabiliy would, for example, enable a firs responder o direc a microaerial vehicle by speaking fly up he sairs, proceed down he hall, and inspec he second room on he righ pas he kichen. A fundamenal challenge is o correcly associae linguisic elemens from he command o a robo s undersanding of he exernal world. We can alleviae his challenge by developing robos ha formulae knowledge represenaions ha model he higher-level semanic properies of heir environmen. We propose an approach ha enables robos o efficienly learn human-cenric models of he observed environmen from a narraed, guided our (Fig. 1) by fusing knowledge inferred from naural language descripions wih convenional low-level 1 The firs wo auhors conribued equally o his paper. sensor daa. Our mehod allows people o convey meaningful conceps, including semanic labels and relaions for boh local and disan regions of he environmen, simply by speaking o he robo. The challenge lies in effecively combining hese noisy, disparae sources of informaion. Spoken descripions convey conceps (e.g., he second room on he righ ) ha are ambiguous wih regard o heir meric associaions: hey may refer o he region ha he robo currenly occupies, o more disan pars of he environmen, or even o aspecs of he environmen ha he robo will never observe. In conras, he sensors ha robos commonly employ for mapping, such as cameras and LIDARs, yield meric observaions arising only from he robo s immediae surroundings. To handle ambiguiy, we propose o combine meric, opological, and semanic environmen represenaions ino a semanic graph. The meric layer akes he form of an occupancy-grid ha models local perceived srucure. The opological layer consiss of a graph in which nodes correspond o reachable regions of he environmen, and edges denoe pairwise spaial relaions. The semanic layer conains he labels wih which people refer o regions. This knowledge represenaion is well-suied o fusing conceps from spoken descripions wih he robo s meric observaions of is surroundings.

2 We esimae a join disribuion over he semanic, opological and meric maps, condiioned on he language and he meric observaions from he robo s propriocepive and exerocepive sensors. The space of semanic graphs, however, increases combinaorially wih he size of he environmen. We efficienly mainain he disribuion using a Rao-Blackwellized paricle filer [6] o rack a facored form of he join disribuion over semanic graphs. Specifically, we approximae he marginal over he space of opologies wih a se of paricles, and analyically model condiional disribuions over meric and semanic maps as Gaussian and Dirichle, respecively. The algorihm updaes hese disribuions ieraively over ime using spoken descripions and sensor measuremens. We model he likelihood of naural language uerances wih he Generalized Grounding Graph (G 3 ) framework [2]. Given a descripion, he G 3 model induces a disribuion over semanic labels for he nodes in he semanic graph ha we hen use o updae he Dirichle disribuion. The algorihm uses he resuling semanic disribuion o propose modificaions o he graph, allowing semanic informaion o influence he meric and opological layers. We demonsrae our algorihm hrough hree guided our experimens wihin mixed indoor-oudoor environmens. The resuls demonsrae ha by effecively inegraing knowledge from naural language descripions, he algorihm efficienly learns semanic environmen models and achieves higher accuracy han exising mehods. II. RELATED WORK Several researchers have augmened lower-level meric maps wih higher-level opological and/or semanic informaion [7, 8, 9, 10, 11]. Zender e al. [9] describe a framework for office environmens in which he semanic layer models room caegories and heir relaionship wih he labels of objecs wihin rooms. The sysem can hen classify room ypes based upon user-assered objec labels. Pronobis and Jensfel [8] describe a muli-modal probabilisic framework incorporaing semanic informaion from a wide variey of modaliies including deeced objecs, place appearance, and human-provided informaion. These approaches focus on augmening a meric map wih semanic informaion, raher han joinly esimaing he wo represenaions. They do no demonsrae improvemen of meric accuracy using semanic informaion. The problem of mapping linguisic elemens o heir corresponding manifesaion in he exernal world is referred o as he symbol grounding problem [12]. In he roboics domain, he grounding problem has been mainly addressed in he conex of following naural language commands [1, 2, 3, 13, 14, 15, 16, 17]. Canrell e al. [18] described an approach ha updaes he symbolic sae, bu no he meric sae, of he environmen. A conribuion of he proposed algorihm is a probabilisic framework ha uses learned semanic properies of he environmen o efficienly idenify loop closures, a fundamenal problem in simulaneous localizaion and mapping (SLAM). Semanic observaions, however, are no he only informaion Fig. 2. An example of a semanic graph. source useful for place recogniion. A number of soluions exis ha idenify loop closures based upon visual appearance [19, 20] and local meric srucure [21], among ohers. III. BUILDING SEMANTIC MAPS WITH LANGUAGE This secion presens our approach o mainaining a disribuion over semanic graphs, our environmen represenaion ha consiss joinly of meric, opological, and semanic maps. A. Semanic Graphs We model he environmen as a se of places, regions in he environmen a fixed disance apar ha he robo has visied. We represen each place by is pose x i in a global reference frame, and a label l i (e.g., gym, hallway ). More formally, we represen he environmen by he uple {G, X, L} ha consiues he semanic graph. The graph G = (V, E) denoes he environmen opology wih a verex V = {v 1, v 2,..., v } for each place ha he robo has visied, and undireced edges E ha signify observed relaions beween verices, based on meric or semanic informaion. The vecor X = [x 1, x 2,..., x ] encodes he pose associaed wih each verex. The se L = {l 1, l 2,..., l } includes he semanic label l i associaed wih each verex. The semanic graph (Fig. 2) grows as he robo moves hrough he environmen. Our mehod adds a new verex v +1 o he opology afer he robo ravels a specified disance, and augmens he vecor of poses and collecion of labels wih he corresponding pose x +1 and labels l +1, respecively. This model resembles he pose graph represenaion commonly employed by SLAM soluions [22]. B. Disribuion Over Semanic Graphs We esimae a join disribuion over he opology G, he vecor of locaions X, and he se of labels L. Formally, we mainain his disribuion over semanic graphs {G, X, L } a ime condiioned upon he hisory of meric exerocepive sensor daa z = {z 1, z 2,..., z }, odomery u = {u 1, u 2,..., u }, and naural language descripions λ = {λ 1, λ 2,..., λ }: p(g, X, L z, u, λ ). (1) Each λ i denoes a (possibly null) uerance, such as This is he kichen, or The gym is down he hall. We facor

3 he join poserior ino a disribuion over he graphs and a condiional disribuion over he node poses and labels: p(g, X, L z, u, λ ) = p(l X, G, z, u, λ ) p(x G, z, u, λ ) p(g z, u, λ ) (2) This facorizaion explicily models he dependence of he labels on he opology and place locaions, as well as he meric map s dependence on he consrains induced by he opology. The space of possible graphs for a paricular environmen is spanned by he allocaion of edges beween nodes. The number of edges, however, can be exponenial in he number of nodes. Hence, mainaining he full disribuion over graphs is inracable for all bu rivially small environmens. To overcome his complexiy, we assume as in Ranganahan and Dellaer [23] ha he disribuion over graphs is dominaed by a small subse of opologies while he likelihood associaed wih he majoriy of opologies is nearly zero. In general, his assumpion holds when he environmen srucure (e.g., indoor, man-made) or he robo moion (e.g., exploraion) limis conneciviy. In addiion, condiioning he graph on he spoken descripions furher increases he peakedness of he disribuion because i decreases he probabiliy of edges when he labels and semanic relaions are inconsisen wih he language. The assumpion ha he disribuion is concenraed around a limied se of opologies suggess he use of pariclebased mehods o represen he poserior over graphs, p(g z, u, λ ). Inspired by he derivaion of Ranganahan and Dellaer [23] for opological SLAM, we employ Rao- Blackwellizaion o model he facored formulaion (2), whereby we accompany he sample-based disribuion over graphs wih analyic represenaions for he condiional poseriors over he node locaions and labels. Specifically, we represen he poserior over he node poses p(x G, z, u, λ ) by a Gaussian, which we paramerize in he canonical form. We mainain a Dirichle disribuion ha models he poserior disribuion over he se of node labels p(l X, G, z, u, λ ). We represen he join disribuion over he opology, node locaions, and labels as a se of paricles: Each paricle P (i) where G (i) P = {P (1), P (2),..., P (n) }. (3) P consiss of he se { G (i), X (i), L (i), w (i) P (i) = }, (4) denoes a sample from he space of graphs; X (i) is he analyic is he analyic disribuion over locaions; L (i) disribuion over labels; and w (i) is he weigh of paricle i. Algorihm 1 oulines he process by which we recursively updae he disribuion over semanic graphs (2) o reflec he laes robo moion, meric sensor daa, and uerances. The following secions explain each sep in deail. Algorihm 1: Semanic Mapping Algorihm { } Inpu: P 1 = P (i) 1, and (u, z, λ ), where { } P (i) 1 = G (i) 1, X(i) 1, L(i) 1, w(i) 1 { } Oupu: P = P (i) for i = 1 o n do 1) Employ proposal disribuion p(g G (i) 1, z 1, u, λ ) o propagae he graph sample G (i) 1 according o odomery u and curren disribuions over labels L (i) 1 and poses X(i) 1. 2) Updae he Gaussian disribuion over he node poses X (i) according o he consrains induced by he newly-added graph edges. 3) Updae he Dirichle disribuion over he curren and adjacen nodes L (i) according o he language λ. 4) Compue he new paricle weigh w (i) based upon he previous weigh w (i) 1 and he meric daa z. end Normalize weighs and resample if needed. C. Augmening he Graph using he Proposal Disribuion Given he poserior disribuion over he semanic graph a ime 1, we firs compue he prior disribuion over he graph G. We do so by sampling from a proposal disribuion ha is he predicive prior of he curren graph given he previous graph and sensor daa, and he recen odomery and language: p(g G 1, z 1, u, λ ) (5) We formulae he proposal disribuion by firs augmening he graph o reflec he robo s moion. Specifically, we add a node v o he graph ha corresponds o he robo s curren pose wih an edge o he previous node v 1 ha represens he emporal consrain beween he wo poses. We denoe his inermediae graph as G. Similarly, we add he new pose as prediced by he robo s moion model o he vecor of poses X and he node s label o he label vecor L according o he process described in Subsecion III-E. 2 We formulae he proposal disribuion (5) in erms of he likelihood of adding edges beween nodes in his modified graph G. The sysem considers wo forms of addiional edges: firs, hose suggesed by he spaial disribuion of nodes and second, by he semanic disribuion for each node. 1) Spaial Disribuion-based Consrains: We firs propose connecions beween he robo s curren node v and ohers in he graph based upon heir meric locaion. We do so by sampling from a disance-based proposal disribuion biased owards nodes ha are spaially close. Doing so requires marginalizaion over he disances d beween node pairs, as shown in equaion (6) (we have omied he hisory of language 2 The label updae explains he presence of he laes language λ.

4 observaions λ, meric measuremens z 1, and odomery u for breviy). Equaion (6a) reflecs he assumpion ha addiional edges expressing consrains involving he curren node e j / E are condiionally independen. Equaion (6c) approximaes he marginal in erms of he disance beween he wo nodes associaed wih he addiional edge. p a (G G, z 1, u, λ ) = p(g j G ) (6a) j:e j / E = p(g j X, G, u )p(x G ) (6b) j:e j / E j:e j / E X d j p(g j d j, G )p(d j G ), (6c) The condiional disribuion p(g j d j, G 1, z 1, u ) expresses he likelihood of adding an edge beween nodes v and v j based upon heir spaial locaion. We represen he disribuion for a paricular edge beween verices v i and v j a disance d ij = x i x j 2 apar as p(g ij d ij, G, z 1, u ) γd 2, (7) ij where γ specifies disance bias. For he evaluaions in his paper, we use γ = 0.2. We approximae he disance prior p(d j G, z 1, u ) wih a folded Gaussian disribuion. The algorihm samples from he proposal disribuion (6) and adds he resuling edges o he graph. In pracice, we use laser scan measuremens o esimae he corresponding ransformaion. 2) Semanic Map-based Consrains: A fundamenal conribuion of our mehod is he abiliy for he semanic map o influence he meric and opological maps. This capabiliy resuls from he use of he label disribuions o perform place recogniion. The algorihm idenifies loop closures by sampling from a proposal disribuion ha expresses he semanic similariy beween nodes. In similar fashion o he spaial disance-based proposal, compuing he proposal requires marginalizing over he space of labels: p a (G G, z 1, u, λ ) = G, λ ) (8a) = j:e j / E L j:e j / E l,l j j:e j / E p(g j p(g j L, G, λ )p(l G ) p(g j l, l j, G )p(l, l j G ), (8b) (8c) where we have omied he meric, odomery, and language inpus for clariy. The firs line follows from he assumpion ha addiional edges ha express consrains o he curren node e j / E are condiionally independen. The second line represens he marginalizaion over he space of labels, while he las line resuls from he assumpion ha he semanic edge likelihoods depend only on he labels for he verex pair. We model he likelihood of edges beween wo nodes as non-zero for he same label: Fig. 3. Depiced as pie chars, he nodes label disribuions are used o propose new graph edges. The algorihm rejecs invalid edges ha resul from ambiguous labels (black) and adds he valid edge (green) o he graph. p(g j l, l j ) = { θ l if l = l j (9) 0 if l l j where θ l denoes he label-dependen likelihood ha edges exis beween nodes wih he same label. In pracice, we assume a uniform saliency prior for each label. Equaion (8c) hen measures he cosine similariy beween he label disribuions. We sample from he proposal disribuion (8a) o hypohesize new semanic map-based edges. As wih disance-based edges, we esimae he ransformaion associaed wih each edge based upon local meric observaions. Figure 3 shows several differen edges sampled from he proposal disribuion a one sage of a our. Here, he algorihm idenifies candidae loop closures beween differen enrances in he environmen and acceps hose (shown in green) whose local laser scans are consisen. Noe ha some paricles may add invalid edges (e.g., due o percepual aliasing), bu heir weighs will decrease as subsequen measuremens become inconsisen wih he hypohesis. D. Updaing he Meric Map Based on New Edges The proposal sep resuls in he addiion, o each paricle, of a new node a he curren robo pose, along wih an edge represening is emporal relaionship o he previous node. The proposal sep also hypohesizes addiional loop-closure edges. Nex, he algorihm incorporaes hese relaive pose consrains ino he Gaussian represenaion for he marginal disribuion over he map p(x G, z, u, λ ) = N 1 (X ; Σ 1, η ), (10) where Σ 1 and η are he informaion (inverse covariance) marix and informaion vecor ha paramerize he canonical form of he Gaussian. We uilize he isam algorihm [22] o updae he canonical form by ieraively solving for he QR facorizaion of he informaion marix.

5 E. Updaing he Semanic Map Based on Naural Language Nex, he algorihm updaes each paricle s analyic disribuion over he curren se of labels L = {l,1, l,2,..., l, }. This updae reflecs label informaion conveyed by spoken descripions as well as ha suggesed by he addiion of edges o he graph. In mainaining he disribuion, we assume ha he node labels are condiionally independen: p(l X, G, z, u, λ ) = p(l,i X, G, z, u, λ ). (11) i=1 This assumpion ignores dependencies beween labels associaed wih nearby nodes, bu simplifies he form for he disribuion over labels associaed wih a single node. We model each node s label disribuion as a Dirichle disribuion of he form p(l,i λ 1... λ ) = Dir(l,i ; α 1... α K ) = Γ( K 1 α i) Γ(α 1 )... Γ(α K ) K k=1 l α k 1,i,k. (12) We iniialize parameers α 1... α K o 0.2, corresponding o a uniform prior over he labels. Given subsequen language, his favors disribuions ha are peaked around a single label. We consider wo forms of naural language inpus. The firs are simple uerances ha refer only o he robo s curren posiion, such as This is he gym. The second are expressions ha convey semanic informaion and spaial relaions associaed wih possibly disan regions in he environmen, such as The kichen is down he hall, which include a figure ( he kichen ) and landmark ( he hall ). We have implemened our complex language sysem wih he words hrough, down, away from, and near. To undersand he expression The kichen is down he hall, he sysem mus firs ground he landmark phrase he hall o a specific objec in he environmen. I mus hen infer an objec in he environmen ha corresponds o he word he kichen. One can no longer assume ha he user is referring o he curren locaion as he kichen (referen) or ha he hall s (landmark) locaion is known. We use he label disribuion o reason over he possible nodes ha denoe he landmark. We accoun for he uncerainy in he figure by formulaing a disribuion over he nodes in he opology ha expresses heir likelihood of being he referen. We arrive a his disribuion using he G 3 framework [2] o infer groundings for he differen pars of he naural language descripion. In he case of his example, he framework uses he mulinomial disribuions over labels o find a node corresponding o he hall and induces a probabiliy disribuion over kichens based on he nodes ha are down he hall from he idenified landmark nodes. For boh ypes of expressions, he algorihm updaes he semanic disribuion according o he rule p(l,i λ = (k, i), l 1,i ) = K Γ( K 1 α 1 i + α) Γ(α 1 1 )... Γ(α 1 k + α)... Γ(α K ) k=1 l α k 1,i,k, (13) where α is se o he likelihood of he grounding. In he case of simple language, he grounding is rivial, and we use α = 1 for he curren node in he graph. For complex expressions, we use he likelihood from G 3 for α. G 3 creaes a vecor of grounding variables Γ for each linguisic consiuen in he naural language inpu λ. The oplevel consiuen γ a corresponds o he graph node o which he naural language inpu refers. Our aim is o find: α = p(γ a = x i λ) (14) We compue his probabiliy by marginalizing over groundings for oher variables in he language: α = Γ/γ a p(γ λ). (15) G 3 compues his disribuion by facoring according o he linguisic srucure of he naural language command: α = Γ/γ a 1 Z f(γ m λ m ) (16) Tellex e al. [2] describe he facorizaion process in deail. In addiion o inpu language, we also updae he label disribuion for a node when he proposal sep adds an edge o anoher node in he graph. These edges may correspond o emporal consrains ha exis beween consecuive nodes, or hey may denoe loop closures based upon he spaial disance beween nodes ha we infer from he meric map. Upon adding an edge o a node for which we have previously incorporaed a direc language observaion, we propagae he observed label o he newly conneced node using a value of α = 0.5. F. Updaing he Paricle Weighs Having proposed a new se of graphs {G (i) } and updaed he analyic disribuions over he meric and semanic maps for each paricle, we updae heir weighs. The updae follows from he raio beween he arge disribuion over he graph and he proposal disribuion, and can be shown o be where w (i) 1 w (i) m = p(z G (i), z 1, u, λ ) w (i) 1, (17) is he weigh of paricle i a ime 1 and w(i) denoes he weigh a ime. We evaluae he measuremen likelihood (e.g., of LIDAR) by marginalizing over he node poses p(z G (i), z 1, u, λ ) = p(z X (i) X, G (i), z 1, u, λ ) p(x (i) G (i), z 1, u, λ )dx, (18) which allows us o uilize he condiional measuremen model. In he experimens presened nex, we compue he condiional likelihood by maching he scans beween poses. Afer calculaing he new imporance weighs, we periodically perform resampling in which we replace poorly-weighed paricles wih hose wih higher weighs according o he algorihm of Douce e al. [6].

6 (a) No language consrains (b) Simple language (c) Complex language Fig. 4. Maximum likelihood semanic graphs for he small our. In conras o (a) he baseline algorihm, our mehod incorporaes key loop closures based upon (b) simple and (c) complex descripions ha resul in meric, opological, and semanic maps ha are noiceably more accurae. The dashed line denoes he approximae ground ruh rajecory. The inse presens a view of he semanic and opological maps near he gym region. IV. RESULTS We evaluae our algorihm hrough hree experimens in which a human gives a roboic wheelchair (Fig. 1) [11] a narraed our of buildings on he MIT campus. The robo was equipped wih a forward-facing LIDAR, wheel encoders, and a microphone. In he firs wo experimens, he robo was manually driven while he user inerjeced exual descripions of he environmen. In he hird experimen, he robo auonomously followed he human who provided spoken descripions. Speech recogniion was performed manually. A. Small Tour In he firs experimen (Fig. 4), he user sared a he elevaor lobby, visied he gym, exied he building, and laer reurned o he gym and elevaor lobby. The user provided exual descripions of he environmen, wice each for he elevaor lobby and gym regions. We compare our mehod wih differen ypes of language inpu agains a baseline algorihm. 1) No Language: We consider a baseline approach ha direcly labels nodes based upon simple language, bu does no propose edges based upon label disribuions. The baseline emulaes ypical soluions by augmening a sae-of-he-ar isam meric map wih a semanic layer wihou allowing semanic informaion o influence lower layers. Figure 4(a) presens he resuling meric, opological, and semanic maps ha consiue he semanic graph for he highes-weighed paricle. The accumulaion of odomery drif resuls in significan errors in he esimae for he robo s pose when revisiing he gym and elevaor lobby. Wihou reasoning over he semanic map, he algorihm is unable o deec loop closures. This resuls in significan errors in he meric map as well as he semanic map, which hallucinaes wo separae elevaor lobbies (purple) and gyms (orange). 2) Simple Language: We evaluae our algorihm in he case of simple language wih which he human references he robo s curren posiion when describing he environmen. Figure 4(b) presens he semanic graph corresponding o he highes-weighed paricle esimaed by our algorihm. By considering he semanic map when proposing loop closures, he algorihm recognizes ha he second region ha he user labeled as he gym is he same place ha was labeled earlier in he our. A he ime of receiving he second label, drif in he odomery led o significan error in he gym s locaion much like he baseline resul (Fig. 4(a)). The algorihm immediaely correcs his error in he semanic graph by using he label disribuion o propose loop closures a he gym and elevaor lobby, which would oherwise require searching a combinaorially large space. The resuling maximum likelihood map is opologically and semanically consisen hroughou and merically consisen for mos of he environmen. The excepion is he couryard, where only odomery measuremens were available, causing drif in he pose esimae. Aesing o he model s validiy, he ground ruh opology receives 92.7% of he probabiliy mass and, furhermore, he op four paricles are each consisen wih he ground ruh. 3) Complex Language: Nex, we consider algorihm s performance when naural language descripions reference locaions ha can no longer be assumed o be he robo s curren posiion. Specifically, we replaced he iniial labeling of he gym wih an indirec reference of he form he gym is down he hallway, wih he hallway labeled hrough simple language. The language inpus are oherwise idenical o hose employed for he simple language scenario and he baseline evaluaion. The algorihm incorporaes complex language ino he semanic map using he G 3 framework o infer he nodes in he graph ha consiue he referen (i.e., he gym ) and he landmark (i.e., he hallway ). This grounding aribues a non-zero likelihood o all nodes ha exhibi he relaion of being down from he nodes idenified as being he hallway. The inse view in Fig. 4(c) depics he label disribuions ha resul from his grounding. The algorihm aribues he gym label o muliple nodes in he semanic graph as a resul of he ambiguiy in he referen as well as he G 3 model for he near relaion. When he user laer labels he region afer reurning from he couryard, he algorihm proposes a loop closure despie significan drif in he esimae for he robo s pose. As wih he simple language scenario, his resuls in a semanic graph for he environmen ha is accurae opologically, semanically, and merically (Fig. 4(c)).

7 (a) Ground Truh (b) No language consrains (c) Complex language Fig. 5. Maximum likelihood semanic graphs for (a) he large our experimen. (b) The resul of he baseline algorihm wih leer pairs ha indicae map componens ha correspond o he same environmen region. (c) Our mehod wih inse views ha indicae he inclusion of wo complex language descripions. B. Large Tour C. Auonomous Tour The second experimen (Fig. 5) considers an exended our of MIT s Saa Cener, wo neighboring buildings, and heir shared couryard. The robo visied several places wih he same semanic aribues (e.g., elevaor lobbies, enrances, and cafeerias) and visied some places more han once (e.g., one cafeeria and he amphiheaer). We accompanied he our wih 20 descripions of he environmen ha included boh simple and complex language. As wih he shorer our, we compare our mehod agains he baseline semanic mapping algorihm. Figure 5(b) presens he baseline esimae for he environmen s semanic graph. Wihou incorporaing complex language or allowing semanic informaion o influence he opological and meric layers, he resuling semanic graph exhibis significan errors in he meric map, an incorrec opology, and aliasing of he labeled places ha he robo revisied. In conras, Fig. 5(c) demonsraes ha, by using semanic informaion o propose consrains in he opology, our algorihm yields correc opological and semanic maps, and meric maps wih noably less error. The resuling model assigns 93.5% of he probabiliy mass o he ground ruh opology, wih each of he op five paricles being consisen wih ground ruh. The resuls highligh he abiliy of our mehod o olerae ambiguiies in he labels assigned o differen regions of he environmen. This is a direc consequence of he use of semanic informaion, which allows he algorihm o significanly reduce he number of candidae loop closures ha is oherwise combinaorial in he size of he map. This enables he paricle filer o efficienly model he disribuion over graphs. While some paricles may propose invalid loop closures due o ambiguiy in he labels, he algorihm is able o recover wih a manageable number of paricles. For uerances wih complex language, he G3 framework was able o generae reasonable groundings for he referen locaions. However, due o he simplisic way in which we define regions, groundings for he lobby were no enirely accurae (Fig. 5(c), inse) as grounding valid pahs ha go hrough he enrance is sensiive o he local meric srucure of he landmark (enrance). In he hird experimen, he robo auonomously followed a user during a narraed our along a roue similar o ha of he firs experimen [24]. Using a headse microphone, he user provided spoken descripions of he environmen ha included ambiguous references o regions wih he same label (e.g., elevaor lobbies, enrances). The descripions included boh simple and complex uerances ha were manually annoaed. Figure 6 presens he maximum likelihood semanic graph ha our algorihm esimaes. By incorporaing informaion ha he naural language descripions convey, he algorihm recognizes key loop closures ha resul in accurae semanic maps. The resuling model assigns 82.9% of he probabiliy mass o he ground ruh opology, wih each of he op nine paricles being consisen wih ground ruh. V. C ONCLUSION We described a semanic mapping algorihm enabling robos o efficienly learn merically accurae semanic maps from naural language descripions. The algorihm infers rich models Fig. 6. Maximum likelihood map for he auonomous our.

8 of an environmen from complex expressions uered during a narraed our. Currenly, we assume ha he robo has previously visied boh he landmark and he referen locaions, and ha he user has already labeled he landmark. As such, he algorihm can incorrecly aribue labels in siuaions where he user refers o regions ha, while hey may be visible, he robo has no ye visied. This problem resuls from he algorihm needing o inegrae he spoken informaion in siu. We are currenly working on modifying our approach o allow he user o provide a sream of spoken descripions, and for he robo o laer ground he descripion wih sensor observaions as needed during environmen exploraion. This descripion need no be siuaed; such an approach offers he benefi ha he robo can learn semanic properies of he environmen wihou requiring ha he user provide a guided our. A presen, our mehod uses radiional sensors o observe only geomeric properies of he environmen. We are building upon echniques in scene classificaion, appearance modeling, and objec deecion o learn more complee maps by inferring higher-level semanic informaion from LIDAR and camera daa. We are also working oward auomaic region segmenaion in order o creae more meaningful opological eniies. Spoken descripions can convey informaion abou space ha includes he ypes of places, heir colloquial names, heir locaions wihin he environmen, and he ypes of objecs hey conain. Our curren approach suppors assigning labels and spaial relaionships o he environmen. A direcion for fuure work is o exend he scope of admissible descripions o include hose ha convey general properies of he environmen. For example, he robo should be able o infer knowledge from saemens such as you can find compuers in offices, or nurses saions end o be locaed near elevaor lobbies. Such an exension may build upon exising daa-driven effors oward learning onologies ha describe properies of space. In summary, we proposed an approach o learning humancenric maps of an environmen from user-provided naural language descripions. The novely lies in fusing high-level informaion conveyed by a user s speech wih low-level observaions from radiional sensors. By joinly esimaing he environmen s meric, opological, and semanic srucure, we demonsraed ha he algorihm yields accurae represenaions of is environmen. VI. ACKNOWLEDGMENTS We hank Nick Roy, Josh Joseph, and Javier Velez for heir helpful feedback. We graefully acknowledge Quana Compuer, which suppored his work. REFERENCES [1] C. Mauszek, D. Fox, and K. Koscher, Following direcions using saisical machine ranslaion, in Proc. ACM/IEEE In l. Conf. on Human-Robo Ineracion (HRI), 2010, pp [2] S. Tellex, T. Kollar, S. Dickerson, M. R. Waler, A. G. Banerjee, S. Teller, and N. Roy, Undersanding naural language commands for roboic navigaion and mobile manipulaion, in Proc. Na l Conf. on Arificial Inelligence (AAAI), 2011, pp [3] J. Dzifcak, M. Scheuz, C. Baral, and P. Schermerhorn, Wha o do and how o do i: Translaing naural language direcives ino emporal and dynamic logic represenaion for goal managemen and acion execuion, in Proc. IEEE In l Conf. on Roboics and Auomaion (ICRA), 2009, pp [4] G. Bugmann, E. Klein, S. Lauria, and T. Kyriacou, Corpus-based roboics: A roue insrucion example, Proc. Inelligen Auonomous Sysems, pp , [5] D. L. Chen and R. J. Mooney, Learning o inerpre naural language navigaion insrucions from observaions, in Proc. Na l Conf. on Arificial Inelligence (AAAI), 2011, pp [6] A. Douce, N. de Freias, K. Murphy, and S. Russell, Rao-Blackwellised paricle filering for dynamic bayesian neworks, in Proc. Conf. on Uncerainy in Arificial Inelligence (UAI), 2000, pp [7] B. Kuipers, The spaial semanic hierarchy, Arificial Inelligence, vol. 119, no. 1, pp , [8] A. Pronobis and P. Jensfel, Large-scale semanic mapping and reasoning wih heerogeneous modaliies, in Proc. IEEE In l Conf. on Roboics and Auomaion (ICRA), 2012, pp [9] H. Zender, O. Marínez Mozos, P. Jensfel, G. Kruijff, and W. Burgard, Concepual spaial represenaions for indoor mobile robos, Roboics and Auonomous Sysems, vol. 56, no. 6, pp , [10] T. Kollar and N. Roy, Uilizing objec-objec and objec-scene conex when planning o find hings, in Proc. IEEE In l Conf. on Roboics and Auomaion (ICRA), 2009, pp [11] S. Hemachandra, T. Kollar, N. Roy, and S. Teller, Following and inerpreing narraed guided ours, in Proc. IEEE In l Conf. on Roboics and Auomaion (ICRA), 2011, pp [12] S. Harnad, The symbol grounding problem, Physica D, vol. 42, pp , [13] M. MacMahon, B. Sankiewicz, and B. Kuipers, Walk he alk: Connecing language, knowledge, and acion in roue insrucions, in Proc. Na l Conf. on Arificial Inelligence (AAAI), 2006, pp [14] M. Skubic, D. Perzanowski, S. Blisard, A. Schulz, W. Adams, M. Bugajska, and D. Brock, Spaial language for human-robo dialogs, IEEE Trans. on Sysems, Man, and Cyberneics, Par C: Applicaions and Reviews, vol. 34, no. 2, pp , [15] T. Kollar, S. Tellex, D. Roy, and N. Roy, Toward undersanding naural language direcions, in Proc. ACM/IEEE In l. Conf. on Human-Robo Ineracion (HRI), 2010, pp [16] S. Tellex, P. Thaker, R. Deis, T. Kollar, and N. Roy, Toward informaion heoreic human-robo dialog, in Proc. Roboics: Science and Sysems (RSS), [17] C. Mauszek, N. FizGerald, L. Zelemoyer, L. Bo, and D. Fox, A join model of language and percepion for grounded aribue learning, in Proc. In l Conf. on Machine Learning (ICML), [18] R. Canrell, K. Talamadupula, P. Schermerhorn, J. Benon, S. Kambhampai, and M. Scheuz, Tell me when and why o do i!: Run-ime planner model updaes via naural language insrucion, in Proc. ACM/IEEE In l. Conf. on Human-Robo Ineracion (HRI), 2012, pp [19] S. Se, D. G. Lowe, and J. J. Lile, Vision-based global localizaion and mapping for mobile robos, Trans. on Roboics, vol. 21, no. 3, pp , [20] M. Cummins and P. Newman, FAB-MAP: Probabilisic localizaion and mapping in he space of appearance, In l J. of Roboics Research, vol. 27, no. 6, pp , [21] J.-S. Gumann and K. Konolige, Incremenal mapping of large cyclic environmens, in Proc. IEEE In l. Symp. on Compuaional Inelligence in Roboics and Auomaion, [22] M. Kaess, A. Ranganahan, and F. Dellaer, isam: Incremenal smoohing and mapping, Trans. on Roboics, vol. 24, no. 6, pp , [23] A. Ranganahan and F. Dellaer, Online probabilisic opological mapping, In l J. of Roboics Research, vol. 30, no. 6, pp , [24] S. Hemachandra, M. R. Waler, S. Tellex, and S. Teller, Learning semanic maps from naural language descripions, [Online]. Available: hp://vimeo.com/