Learning Spatial-Semantic Representations from Natural Language Descriptions and Scene Classifications

Transcription

1 Learning Spaial-Semanic Represenaions from Naural Language Descripions and Scene Classificaions Sachihra Hemachandra, Mahew R. Waler, Sefanie Tellex, and Seh Teller Absrac We describe a semanic mapping algorihm ha learns human-cenric environmen models from by inerpreing naural language uerances. Underlying he approach is a coupled meric, opological, and semanic represenaion of he environmen ha enables he mehod o infer and fuse informaion from naural language descripions wih low-level meric and appearance daa. We exend earlier work wih a novel formulaion incorporaes spaial layou ino a opological represenaion of he environmen. We also describe a facor graph formulaion of he semanic properies ha encodes human-cenric conceps such as ype and colloquial name for each mapped region. The algorihm infers hese properies by combining he user s naural language descripions wih image- and laser-based scene classificaion. We also propose a mechanism o more effecively ground naural language descripions of spaially non-local regions using semanic cues from oher modaliies. We describe how he algorihm employs his learned semanic informaion o propose valid opological hypoheses, leading o more accurae opological and meric maps. We demonsrae ha inegraing language wih oher sensor daa increases he accuracy of he achieved spaialsemanic represenaion of he environmen. I. INTRODUCTION A challenge in realizing robos capable of working producively alongside human parners is he developmen of efficien command and conrol mechanisms. Researchers have recenly sough o endow robos wih he abiliy o inerac more effecively wih people hrough naural language speech [1, 2, 3, 4, 5] and gesure undersanding [6]. Efficien ineracion can be faciliaed when robos reason over models ha encode high-level semanic properies of he environmen. For example, such models could help a microaerial vehicle inerpre a firs responder s command o fly up he sairway on he righ, go down he hall, and observe he kichen. Semanic mapping [7, 8, 9, 10] mehods exend he meric environmen models radiionally employed in roboics o include higher-level conceps, including ypes and colloquial names for regions, and he presence and use of objecs in he environmen. Such mehods ypically operae by augmening a sandard SLAM meric map wih a represenaion of he environmen s opology, and a disinc represenaion of is semanic properies, he laer of which is populaed by inerpreing he robo s sensor sream, hrough (e.g.) scene S. Hemachandra, M.R. Waler, and S. Teller are wih he Compuer Science and Arificial Inelligence Laboraory a he Massachuses Insiue of Technology, Cambridge, MA USA {sachih, mwaler, eller}@csail.mi.edu S. Tellex is wih he Compuer Science Deparmen a Brown Universiy, Providence, RI USA sefie10@cs.brown.edu Fig. 1. Maximum likelihood semanic map of he 6h floor of Saa building (pie chars denoe he likelihood of differen region caegories). classificaion. In his layered approach, he underlying meric map induces and embeds he opological and semanic aribues. However, while refinemens o he meric map improve he opological and semanic maps, mos echniques do no allow knowledge inferred a he semanic and opological levels o influence one anoher or he meric map. Raher, semanic informaion has been inferred from LIDAR and camera daa, coupled wih pre-rained scene appearance models. Some effors are capable of inegraing egocenric descripions abou he robo s curren locaion (e.g., we are in he kichen ), bu canno handle allocenric spaial language involving spaial relaions beween and labels for poenially disan regions in he environmen (e.g., he exi is nex o he cafeeria ). We addressed some of hese limiaions in previous work [11] wih an algorihm ha mainains a join disribuion over a Semanic Graph, a coupled meric, opological, and semanic environmen represenaion learned from user uerances and he robo s low-level sensor daa, during a guided our. Our framework was able o o learn properies of he environmen ha could no be perceived wih ypical sensors (e.g. colloquial names for regions, properies of areas ouside he robo s sensor range) and use semanic knowledge o influence he res of he semanic graph, allowing robos o efficienly learn environmen models from users. However, since ha algorihm assumed ha he environmen is a collecion of regions ha ake he form of fixed, uniformly-sized collecions of sequenial nodes, i can resul in a opology ha is inconsisen wih human conceps of space. Consequenly, he represenaion may no model he spaial exen o which he user s descripions refer, resuling in incorrec language groundings. The semanic informaion in he framework was limied o user-provided colloquial names and did no provide a means o reason over properies

2 such as region ype ha can be inferred from LIDARs, cameras, or oher onboard sensors. Addiionally, due o he absence of oher semanic informaion, he framework required ha a landmark locaion be labeled by he user before he uerance can be grounded (e.g., processing he phrase he kichen is down he hallway, requires he hallway o have already been labeled). This paper describes an exension of our earlier approach o learn richer and more meaningful semanic models of he environmen. Whereas our earlier framework reasoned only abou he conneciviy of deerminisically creaed regions (a fixed inervals), he curren approach reasons over he environmen s region segmenaion as well as is iner-region conneciviy. Addiionally, we propose a facor graph represenaion for he semanic model ha reasons no only over each region s labels, bu also is canonical ype. As before, we infer region labels from user-provided descripions, bu we also incorporae scene classificaion using he robo s onboard sensors, noably camera and laser range-finders, o esimae region ypes. By modeling he relaion beween an area s ype and is colloquial name, he algorihm can reason over boh region ype and region label, even in he absence of speech. This enables he mehod o more effecively ground allocenric user uerances (e.g., when grounding he phrase he kichen is down he hallway, we no longer require he user o explicily label he hallway beforehand). We also describe a mechanism by which he algorihm derives a semanically meaningful opology of he environmen based upon he richer facor graph model, where edges are proposed using a spaial-semanic prior disribuion. We show ha he improved opology model hen allows he mehod o beer handle ambiguiies common in naural language descripions. II. RELATED WORK A number of researchers have focused on he problem of consrucing semanic represenaions [7, 10, 8, 9]. Mos approaches have augmened lower-level meric maps wih higher-level opological and/or semanic informaion. However, hese ypically follow a boom-up approach in which higher-level conceps are consruced from lower-level informaion, wihou any informaion flow back down o lowerlevel represenaions. In Waler e al. [11] we addressed his by inroducing a framework ha uses semanic informaion derived from naural language descripions uered by humans o improve he opological and meric represenaions. Our proposed approach uses addiional semanic cues o evaluae semanic similariy of regions o updae he opology. Several exising approaches [8, 10] have incorporaed higher-level semanic conceps such as room ype and presence of objecs wih he use of appearance models. Pronobis and Jensfel [10] describe a muli-modal probabilisic framework incorporaing semanic informaion from a wide variey of modaliies including deeced objecs, place appearance, and human-provided informaion. However, heir approach is limied o handling egocenric descripions (e.g., we are in he living room ). Addiionally, hey infer opology based on Fig. 2. Example of a semanic graph: Two regions R 1 and R2 and heir consiuen nodes n i s; disribuions over node poses x i ; and he corresponding facor graph. door deecions, a heurisic which works well only in cerain kinds of environmens; hey do no mainain a disribuion over likely opologies. In [11], we mainained a hypohesis over he disribuion of opologies, bu he opology was consruced from segmens creaed a fixed spaial inervals, which can be inconsisen wih human conceps. In he presen work, we address his limiaion by proposing a mehod ha mainains muliple hypoheses abou region segmenaions and abou connecions among regions. Mapping linguisic elemens o heir corresponding physical eniies have been sudied by several researchers [3, 4, 2, 12, 13, 14, 15, 16] in he roboics domain. However, hese effors have no focused on consrucing semanic represenaions. In [11] we augmened our semanic graph wih complex language descripions. However, his could be accomplished only when he user had explicily labeled a locaion before describing anoher in reference o i (e.g., he gym is down he hallway requires prior labeling of he hallway). The resuling represenaion conained only labels obained hrough language descripions; i did no inegrae semanic cues from oher sources (such as appearance models). The presen mehod improves upon his by inegraing informaion from muliple semanic sources and mainaining a disribuion over a larger se of semanic properies, rendering i capable of grounding language even in he absence of pre-labeled landmark locaions. III. SEMANTIC GRAPH REPRESENTATION This secion presens our approach o mainaining a disribuion over semanic graphs, an environmen represenaion ha consiss joinly of meric, opological, and semanic maps. A. Semanic Graphs We define he semanic graph as a uple conaining opological, meric and semanic represenaions of he environmen. Figure 2 shows an example semanic graph for a rivial environmen. The opology G is composed of nodes n i ha denoe he robo s rajecory hrough he environmen (sampled a 1 m

3 disances), node conneciviy, and node region assignmens. We associae wih each node a se of observaions ha include laser scans z i, semanic appearance observaions a i based on laser l i and camera i i models, and available language observaions λ i. We assign nodes o regions R α = {n 1,.., n m } ha represen spaially coheren areas in he environmen (such as rooms or hallways) compaible wih human conceps. Undireced edges exis beween nodes in his graph, denoing raversabiliy beween wo nodes. Edges (denoing conneciviy) beween regions are inferred based on he edges beween nodes in he graph. A region edge exiss beween wo regions if a leas one graph edge connecs a node from one region o a node in he oher. The opological layer consiss of he nodes, edges and he region assignmens for he nodes. The pose of each node n i is represened by x i in a global reference frame. The meric layer is condiioned on he opology, where edges in he opology also include meric consrains beween he poses of he corresponding nodes. Meric consrains are calculaed by scan-maching he corresponding laser observaions of each region. A pose graph represenaion is employed o mainain he disribuion over he pose of each node, condiioned on hese consrains. Occupancy maps can be consruced based on he disribuion of hese node poses and heir corresponding laser observaions. Semanic informaion is also condiioned on he opology as shown in he Fig. 2. The semanic layer consiss of a facor graph, where variables represen properies in each region (i.e., he ype C r and labels Λ r ), properies ha can be observed a each node (in each region), and facors ha denoe he join likelihood of hese variables (e.g., he likelihood of observing a label given a paricular room ype). Observaions of hese region properies are made using laserand image-based scene classifiers and by grounding human descripions of he environmen. B. Disribuion Over Semanic Graphs We esimae a join disribuion over he opology G (which includes he nodes, heir conneciviy, and heir region assignmens), he vecor of locaions X, and he se of semanic properies S. Formally, we mainain his disribuion over semanic graphs {G, X, S } a ime condiioned upon he hisory of meric exerocepive sensor daa z = {z 1, z 2,..., z }, odomery u = {u 1, u 2,..., u }, scene appearance observaions a = {a 1, a 2,..., a } (where in our implemenaion a = {l, i }), and naural language descripions λ = {λ 1, λ 2,..., λ }: p(g, X, S z, u, a, λ ). (1) Each variable λ i denoes a (possibly null) uerance, such as This is he kichen, or The gym is down he hall. We facor he join poserior ino a disribuion over he graphs and a condiional disribuion over he node poses and labels: p(g, X, S z, a, u, λ ) = p(s X, G, z, a, u, λ ) p(x G, z, a, u, λ ) p(g z, a, u, λ ) (2) Algorihm 1: Semanic Mapping Algorihm { } Inpu: P 1 = P (i) 1, and (u, z, a, λ ), where { } P (i) 1 = G (i) 1, X(i) 1, S(i) 1, w(i) 1 { } Oupu: P = P (i) for i = 1 o n do 1) Employ proposal disribuion o propagae he graph sample based on u, λ and a. a) Sample region allocaion b) Sample region edges c) Merge newly conneced regions 2) Updae he Gaussian disribuion over he node end poses X (i) condiioned on opology. 3) Updae he facor graph represening semanic properies for he opology based on appearance observaions (l and i ) and language λ. 4) Compue he new paricle weigh w (i) upon he previous weigh w (i) 1 daa z. Normalize weighs and resample if needed. based and he meric As in Waler e al. [11], we mainain his facored disribuion using a Rao-Blackwellized paricle filer, miigaing he hyper-exponenial hypohesis space of he opology [11]. We represen he join disribuion over he opology, node locaions, and labels as a se of paricles: Each paricle P (i) P = {P (1), P (2),..., P (n) }. (3) P consiss of he se { G (i), X (i), S (i), w (i) P (i) = }, (4) where G (i) denoes a sample from he space of graphs; X (i) is he analyic disribuion over locaions; S (i) is he disribuion over semanic properies; and w (i) is he weigh of paricle i. IV. SEMANTIC MAPPING ALGORITHM Algorihm 1 oulines he process by which he mehod recursively updaes he disribuion over semanic graphs (2) o reflec he laes robo moion, meric sensor daa, laser- and image-based scene classificaions, and he naural language uerances. The following secions explain each sep in deail. A. The Proposal Disribuion We compue he prior disribuion over he semanic graph G, given he poserior from he las ime sep G 1, by sampling from a proposal disribuion. This proposal disribuion is he predicive prior of he curren graph given he previous graph, sensor daa (excluding he curren ime sep), appearance daa, odomery, and language: p(g G 1, z 1, a, u, λ ) (5)

4 We augmen G 1 o reflec he robo s moion by adding a node n o o he opology and an edge o he previous node n 1, resuling in an inermediae graph G. This represens he robo s curren pose and he conneciviy o is previous pose. This yields an updaed vecor of poses X and semanic properies S. The new node is assigned o he curren region. 1) Creaion of New Regions: We hen probabilisically bisec he curren region R c using he specral clusering mehod oulined in Blanco e al. [17]. We consruc he similariy marix using he laser poin overlap beween each pair of nodes in he region. Equaion 6 defines he likelihood of bisecing he region, which is based on he normalized cu value (N c ) of he graph involving he proposed segmens. The likelihood of acceping a proposed segmenaion rises as he N c value decreases, i.e. as he separaion of he wo segmens improves (minimizing he iner-region similariy): P (s/n cu ) = 1 (1 + αn 3 c ). This resuls in more spaially disinc areas in he world having a higher likelihood of being separae regions (i.e., more paricles will model hese areas as separae regions). If a paricle segmens he curren region, a new region R i is creaed which does no include he newly added node. 2) Edge Proposals: When a new region R i is creaed, he algorihm proposes edges beween his region, and oher regions in he opology (excluding he curren region R c ). Fig. 3. Example of region edges being proposed (black lines represens rejeced edge proposals; he red line represens an acceped edge) The algorihm samples iner-region edges from a spaialsemanic proposal disribuion ha incorporaes he semanic similariy of regions, as well as he spaial disribuions of is consiuen nodes. This reflecs he facs ha close regions and semanically similar regions are more likely o be conneced. We measure semanic similariy based upon he label disribuion associaed wih each region. The resuling likelihood has he form: p a (G G, z 1, u, a, λ ) = p(g ij G ) (7a) j:e ij / E p x (G ij G )p s (G ij G ) (7b) j:e ij / E where we have omied he hisory of language observaions λ, meric measuremens z 1, appearance measuremens a, and odomery u for breviy. Equaion (7a) reflecs he assumpion ha addiional edges expressing consrains (6) involving he curren node e ij / E are condiionally independen. While p x (G ij G ) encodes he likelihood of he edge based on he spaial properies of he wo regions, p s (G ij G ) describes he edge likelihood based on he regions semanic similariy. Equaion (7b) reflecs he condiional independence assumpion beween he spaial and he semanic likelihood of each edge. For he spaial disribuion prior, we define a variable d ij o be he disance beween he mean nodes of he wo regions, where he mean node is he node wih is pose closes o he region s average pose: p x (G ij G ) = p(g ij X, G, u )p(x G ) (8a) X = p(g ij d ij, G )p(d ij G ). (8b) d ij The condiional disribuion p(g ij d ij, G 1, z 1, u ) expresses he likelihood of adding an edge beween regions R and R j based upon he locaion of heir mean nodes. We represen he disribuion for a paricular edge beween regions R i and R j a disance d ij = X Ri X Rj 2 apar as: p(g ij d ij, G, z 1, u ) γd 2, (9) ij where γ specifies disance bias. For he evaluaions in his paper, we use γ = 0.3. We approximae he disance prior p(d ij G, z 1, u ) wih a folded Gaussian disribuion. For he semanic prior, regions wih similar label Λ disribuions will have a higher likelihood of having an edge beween hem. The label disribuions for he regions are par of he semanic properies (of each region) modeled in he semanic layer: p s (G ij G ) = p(g ij S, G )p(s G ) (10a) S = p(g ij Λ i, Λ j, G )p(λ i, Λ j G ). Λ i,λ j (10b) Equaion (11) expresses he likelihood of an edge exising beween wo regions, given he value of he regions respecive label values: p(g ij Λ i, Λ j ) = { θ Λi if Λ i = Λ j 0 if Λ i Λ j, (11) where θ Λi denoes he label-dependen likelihood ha edges exis beween nodes wih he same label. In pracice, we assume a uniform saliency prior for each label. Equaion (10b) hen measures he cosine similariy beween he label disribuions. Afer a region edge is sampled from he spaial-semanic prior, a scan-mach procedure aemps o find he bes alignmen beween he wo regions. Upon convergence of he scan-mach rouine, he edge is acceped and is used o updae he opology.

5 3) Region Merges: Afer a new region R i has been creaed and edges o oher regions have been checked and added, he algorihm deermines wheher i is possible o merge wih each region o which i is conneced. The newlycreaed region is merged wih an exising (conneced) region if he observaions associaed wih he smaller of he wo regions can be adequaely explained by he larger region. This resuls in regions being merged when he robo revisis already explored spaces (i.e., already-represened regions). This merge process is designed o ensure ha he complexiy of he opology increases only when he robo explores new areas, leading o more efficien region edge proposals as well as more compac language groundings. B. Updaing he Meric Map Based on New Edges The algorihm hen updaes he spaial disribuion of he node poses X condiioned on he proposed opology, p(x G, z, u, λ ) = N 1 (X ; Σ 1, η ), (12) where Σ 1 and η are he informaion marix and informaion vecor ha paramerize he canonical form of he Gaussian. We use he isam algorihm [18], which ieraively solves for he QR facorizaion of he informaion marix. C. Updaing he Semanic Layer Compared wih Waler e al. [11], our updaed represenaion mainains a disribuion abou a larger se of semanic properies associaed wih he environmen. The disribuion over he semanic layer is mainained using a facor graph [19] ha is condiioned on he opology for each paricle. Fig. 4. Semanic Layer (plae represenaion) As Fig. 4 shows, he semanic layer mainains wo variables associaed wih each region r, namely he region caegory C r (e.g., hallway, conference room, ec.) and he labels associaed wih he region Λ r. For example, a conference room can have muliple labels, such as meeing room, conference room. The facor ha joins hese wo variables represens he likelihood of each room caegory generaing a paricular label. A each node n wihin a region, he robo can observe one or more of hese semanic properies. In our curren implemenaion, hese are he region appearance observed from laser scanners l n or cameras i n, and he region labels λ k (and he associaed correspondence variables Φ k ). We run belief propagaion for each paricle a each ime sep as new variables and facors are added. We subsequenly updae he caegory and label disribuions for each region. The following subsecions ouline how we inegrae observaions of hese node properies o he semanic layer. 1) Inegraing Semanic Classificaion Resuls: Each node has an appearance variable a n which is relaed o is region caegory. We ouline several general appearance classes ( room, hallway and open area ) ha are hen observed using robo sensors. The facor ha connecs a region caegory variable C r o an appearance variable a n encodes he likelihood of a region caegory generaing an appearance class (e.g., how ofen does a conference room appear as a room). The caegory o appearance facor was rained using annoaed daa from several oher floors of he Saa building. We make wo observaions of a n using he laser and camera observaions a node n. These are represened in he facor graph as he laser appearance l n and he image appearance i n. We use wo pre-rained appearance models for laser observaions and camera images. The laser appearance classificaion model has been rained using laser feaures similar o hose oulined in Mozos e al. [20], while he image appearance model has been rained using CRFH [21]. Laser and camera appearance variables l n and i n are conneced o he node s appearance a n using facors buil from he confusion marix for he wo rained models. The classificaion resuls for he wo sensors provide a disribuion represening he likelihood of he observaions being generaed from each appearance caegory. The classifier oupus are inegraed o he facor graph as facors aached o variables l n and i n. 2) Inegraing Language Observaions: The robo can also receive eiher ego-cenric (e.g., I am a he kichen ) or allocenric (e.g., The kichen is down he hall ) descripions of he environmen from he user. We use hese observaions o updae he likelihood of observing labels in each region. We mainain he label for each region as anoher variable (Λ r ) in our facor graph. The region label is relaed o he region caegory, as each caegory is more likely o generae some labels han ohers. For example, while a person migh describe a cafeeria as a dinning hall, i is unlikely ha he will describe i as an office. For our curren experimens we have idenified a limied subse of labels associaed wih each region caegory (e.g., he hallway caegory can generae hall, hallway, corridor, or walkway ) in our represenaion. When building hese facors (beween labels and room caegories), for each caegory we assign higher likelihoods for is associaed labels and smaller likelihoods for he oher labels (capuring he likelihood of generaing hese labels given a paricular room caegory). A node can have none, one, or muliple label observaions, depending on he way he person describes a locaion. We represen each label observaion wih a variable λ k and a correspondence variable Φ k, which denoes he likelihood ha he label was inended o describe he curren locaion. The correspondence variable Φ is a binary-valued variable specifying wheher or no he label describes he region. If he label doesn correspond o ha region (i.e., if Φ k = 0), he observaion λ k is uninformaive abou he region s label, and will have equal likelihood for each label value. However, when he correspondence holds (Φ = 1), he facor

6 Fig. 5. Maximum likelihood semanic map of a muli-building environmen on he MIT campus. encodes he likely co-occurrences beween differen labels. For example, if he robo heard he label conference room, wih a high likelihood of Φ = 1, i will resul in oher likely labels (ha ofen co-occur wih conference room ) having high likelihoods (e.g., meeing room ) as well. Currenly high co-occurrence is added for words ha are synonyms (e.g., hallway and corridor ). In his way, we use he correspondence variable o handle he ambiguiy inheren in grounding naural language descripions. When a label is grounded o a region, we creae a label observaion λ k and correspondence variable Φ k, and connec i o he associaed region s label variable Λ r using a co-occurrence facor. We inegrae he correspondence observaion Φ by aaching a facor encoding his likelihood. We rea he observed label as having no uncerainy, as our curren model does no incorporae uncerainy arising from speech recogniion. We derive he facor denoing he observaion of Φ based on he ype of descripion given by he user. If he user describes he curren locaion (e.g., I am a he living room ), we have higher correspondence wih spaially local nodes. For such descripions, we allocae a high likelihood (0.8 probabiliy of Φ = 1) of correspondence wih he curren region. For allocenric descripions (e.g., he kichen is down he hallway, where he user describes he locaion of he referen kichen wih relaion o he landmark hallway ), we use he G 3 framework [4] o calculae he correspondence likelihood given he poenial landmarks. We marginalize he landmarks o arrive a correspondence likelihood of each referen region in a manner similar o our previous approach. In handling allocenric descripions he curren mehod improves upon our earlier approach in wo ways. Firsly, we no longer require common landmarks o be explicily described before being able o ground he language correcly. We do his by leveraging he richer semanic represenaion made possible by inegraing addiional semanic informaion o arrive a likely landmarks. Secondly, while some expressions can be ambiguous (e.g., here could be muliple regions down he hall), he presence of oher semanic cues allows he final label disribuion o be more accurae because incorrec groundings will have less impac if he region s appearance is differen from he label observaion. D. Updaing he Paricle Weighs and Resampling Paricle weighs are updaed and resampling carried ou in he same manner as Waler e al. [11]. V. RESULTS We evaluae our algorihm hrough four experimens in which a human gives a roboic wheelchair [9] a narraed guided our of he Saa building (S3, S4, S6) as well as a muli-building indoor our (MF). The robo was equipped wih a forward-facing LIDAR, a camera, wheel encoders, and a microphone. In hese experimens we drove he robo using a joysick, and provided i wih naural language descripions a specific salien locaions by yping. We evaluae he resuling semanic maps wih regards o heir opological accuracy, compacness, segmenaion accuracy and semanic accuracy. All experimens were run wih 10 paricles. The resuls show ha our framework produces more compac and more accurae semanic graphs han our previous approach. They also demonsrae he improvemen in semanic accuracy due o language descripions. We also show he abiliy of our framework o ground complex language even in he absence of previous labels for he referen (e.g. i handles he expression he lobby is down he hall even when he hall has no been labeled). A. Topological accuracy We compare he opological accuracy, condiioned upon he resuling segmenaion, by comparing he maximum likelihood map wih he ground ruh opology. We define a opology as maching ground ruh if node pairs ha are spaially close (1 m) in a merically accurae alignmen are a mos one region hop away. This avoids penalizing occasional regions ha do no conain valid edges as long as a nearby region was accuraely conneced (and possibly merged wih he nodes from a prior visi). This can happen when an edge was no sampled or when scan-maching failed o converge. The percenage of close node pairs ha were more han one region hop away from each oher for he hird, fourh and sixh floor were 2.8%, 3.7% and 3.8%, respecively. Mos region-maching errors occurred in areas wih significan cluer, causing scan-maching failures. Meric maps derived from he maximum likelihood paricles were accurae for all hree floors.

7 (a) Appearance only (b) Wih language Fig. 7. Region caegory disribuion (a) for a region wih only appearance informaion and (b) and wih boh appearance and language he lounge is behind us. (caegory lounge : yellow). TABLE II Fig. 6. Maximum likelihood semanic map of he 3rd floor of he Saa building (pie chars denoe he likelihood of differen region caegories). R EGION S EGMENTATION AND S EMANTIC ACCURACY Region Type TABLE I R EGION ALLOCATION EFFICIENCY (Sc ) Floor Saa Floor 3 (S3) Saa Floor 4 (S4) Saa Floor 6 (S6) Proposed Framework Conference room Elevaor lobby Hallway Lab Lounge Office Old Framework B. Topological Compacness We compare he allocaion of nodes o regions in he curren framework o he previous mehod. In he previous approach, he opology updae did no merge regions even when he robo revisied a region; i simply creaed an edge beween he regions. The redundancy of he regions has several negaive implicaions. Firsly, i unnecessarily increases he hypohesis space of possible region edges, reducing he likelihood of a sample proposing valid region edges. Secondly, i increases he hypohesis space for grounding language, forcing he framework o consider more region pairs as possible groundings for user descripions. We measure he dupliciy of he region allocaion as Sc = Ns /N, (13) where Ns is he number of close node pairs (< 1 m) assigned o he same region and N is he oal number of close node pairs. If he opology is efficien a allocaing regions, his raio should be high, as only nodes near region boundaries should belong o differen regions. Table I shows hese scores evaluaed for each approach, on hree floors. The new mehod scores significanly higher in all hree experimens. The difference of his score becomes more pronounced as he robo revisis more regions. Since he sixh floor daase did no have oo many revisied regions, he scores for he wo approaches are closer. C. Segmenaion Accuracy Table II oulines he segmenaion accuracy (of he maximum likelihood paricle) for wo daases, oulined according o region ype. We picked he bes maches based on he Jaccard index (number of inersecing nodes divided by he number of union nodes) for each ground ruh annoaed region and he resuling segmened region. Since our segmenaion mehod depends on he similariy of laser observaions, large cluered region ypes such as lab spaces and lounges Segmenaion Accuracy S3 MF Semanic Accuracy Wihou Lang Wih Lang S3 MF S3 MF end o be over-segmened. Addiionally long hallways end o be over-segmened by our mehod, which is refleced in he lower scores for hallways. D. Inference of Semanic Properies Table II also oulines he semanic accuracy (of he maximum likelihood paricle) for wo daases. Semanic accuracy was calculaed for each ground ruh region by assigning each consiuen node wih is paren region s caegory disribuion and aking he cosine similariy. We observe ha he semanic accuracy wih language is higher for mos region ypes (hallways were rarely described and as such show minimal improvemen). Some regions such as labs, which were labeled wih egocenric descripions, have low scores because he regions are over segmened and he language is aribued only o he curren region. Figure 7 compares he region caegory properies wih and wihou language. In he absence of language (Fig. 7(a)), he appearance of he region gives equal likelihood for boh elevaor lobby and lounge. In Fig. 7(b), he region was grounded wih he label lounge and he framework inferred a higher likelihood of he region caegory being a lounge. E. Grounding Allocenric Language Descripions We also esed our framework wih allocenric language descripions. When handling phrases ha include a landmark and a referen (e.g., he gym is down he hall ), our earlier framework required he landmark o have already been labeled before describing he referen locaion. Wih our new framework, he robo is able o ground language when he landmark corresponds o locaions ha may no have been labeled, bu can be inferred from oher semanic cues (e.g., via appearance classificaion). We esed his siuaion using several insances in our daase. Figure 8 shows insances in which allocenric language uerances were grounded ino he semanic graph. As he label disribuions for he surrounding regions demonsrae,

8 par by he Roboics Consorium of he U.S Army Research Laboraory under he Collaboraive Technology Alliance Program, Cooperaive Agreemen W911NF (a) Fig. 8. Resuling caegory disribuion for complex language phrases (a) he elevaor lobby is down he hall and (b) he lounge is down he hall. he framework is able o ground he referen wih a high degree of accuracy, even hough he landmark was never explicily labeled. However, since here is more uncerainy abou he landmark region, he informaion derived from he allocenric language has less influence on he semanic properies on he region (since we marginalize he landmark likelihood when calculaing he grounding likelihood Φ). VI. CONCLUSION We described a framework enabling robos o consruc spaial-semanic represenaions of oured environmens from naural language descripions and scene classificaions. Compared o earlier mehods, our approach handles complex naural language descripions more efficienly, and produces semanic represenaions ha are richer and more compac. Currenly, our mehod grounds descripive language under he assumpion ha i refers o conceps ha exis in he robo s represenaion, eiher due o previously inerpreed descripions or o appearance classificaion. We are exploring a mechanism ha reasons abou he presence of he referred eniies and grounds he uerance only when i is confiden abou he validiy of he grounding. We also plan o inegrae addiional semanic cues ha inform region labels and caegories, such as he presence of common objecs found indoors. In our curren implemenaion, he facors beween region caegory and labels were consruced using a se of predefined labels and relaionships idenified by us (using synonyms for region ypes). In he fuure we plan o learn hese relaionships using real-world daases. Our approach mainains muliple hypoheses regarding he region assignmens of he opology by using paricles o sample region assignmens based on a prior ha considers he spaial similariy of nodes. However, paricle weighs are calculaed based only on laser daa. This can lead o valid segmenaion hypoheses being discarded during resampling if he laser observaion likelihood is low. We plan o incorporae region hypohesis scores based on appearance. In he curren approach, he robo passively acceps human asserions abou he environmen. We also plan o explore more ineracive scenarios where he robo reasons over is hypohesis space and asks quesions of he human o resolve ambiguiies. (b) VII. ACKNOWLEDGMENTS We would like o hank S. Sheme and F. Paerhai for heir help collecing and annoaing daases as well as in raining he appearance models. This work was suppored in REFERENCES [1] G. Bugmann, E. Klein, S. Lauria, and T. Kyriacou, Corpus-based roboics: A roue insrucion example, Proc. Inelligen Auonomous Sysems, pp , [2] J. Dzifcak, M. Scheuz, C. Baral, and P. Schermerhorn, Wha o do and how o do i: Translaing naural language direcives ino emporal and dynamic logic represenaion for goal managemen and acion execuion, in Proc. IEEE In l Conf. on Roboics and Auomaion (ICRA), 2009, pp [3] C. Mauszek, D. Fox, and K. Koscher, Following direcions using saisical machine ranslaion, in Proc. ACM/IEEE In l. Conf. on Human-Robo Ineracion (HRI), 2010, pp [4] S. Tellex, T. Kollar, S. Dickerson, M. R. Waler, A. G. Banerjee, S. Teller, and N. Roy, Undersanding naural language commands for roboic navigaion and mobile manipulaion, in Proc. Na l Conf. on Arificial Inelligence (AAAI), 2011, pp [5] D. L. Chen and R. J. Mooney, Learning o inerpre naural language navigaion insrucions from observaions, in Proc. Na l Conf. on Arificial Inelligence (AAAI), 2011, pp [6] K. Nickel and R. Siefelhagen, Visual recogniion of poining gesures for human-robo ineracion, Image and Vision Compuing, vol. 25, no. 12, pp , Dec [7] B. Kuipers, The spaial semanic hierarchy, Arificial Inelligence, vol. 119, no. 1, pp , [8] H. Zender, O. Marínez Mozos, P. Jensfel, G. Kruijff, and W. Burgard, Concepual spaial represenaions for indoor mobile robos, Roboics and Auonomous Sysems, vol. 56, no. 6, pp , [9] S. Hemachandra, T. Kollar, N. Roy, and S. Teller, Following and inerpreing narraed guided ours, in Proc. IEEE In l Conf. on Roboics and Auomaion (ICRA), 2011, pp [10] A. Pronobis and P. Jensfel, Large-scale semanic mapping and reasoning wih heerogeneous modaliies, in Proc. IEEE In l Conf. on Roboics and Auomaion (ICRA), 2012, pp [11] M. R. Waler, S. Hemachandra, B. Homberg, S. Tellex, and S. Teller, Learning semanic maps from naural language descripions, in Proc. Roboics: Science and Sysems (RSS), [12] M. MacMahon, B. Sankiewicz, and B. Kuipers, Walk he alk: Connecing language, knowledge, and acion in roue insrucions, in Proc. Na l Conf. on Arificial Inelligence (AAAI), 2006, pp [13] M. Skubic, D. Perzanowski, S. Blisard, A. Schulz, W. Adams, M. Bugajska, and D. Brock, Spaial language for human-robo dialogs, IEEE Trans. on Sysems, Man, and Cyberneics, Par C: Applicaions and Reviews, vol. 34, no. 2, pp , [14] T. Kollar, S. Tellex, D. Roy, and N. Roy, Toward undersanding naural language direcions, in Proc. ACM/IEEE In l. Conf. on Human-Robo Ineracion (HRI), 2010, pp [15] S. Tellex, P. Thaker, R. Deis, T. Kollar, and N. Roy, Toward informaion heoreic human-robo dialog, in Proc. Roboics: Science and Sysems (RSS), [16] C. Mauszek, N. FizGerald, L. Zelemoyer, L. Bo, and D. Fox, A join model of language and percepion for grounded aribue learning, in Proc. In l Conf. on Machine Learning (ICML), [17] J.-L. Blanco, J. Gonzalez, and J. Fernandez-Madrigal, Consisen observaion grouping for generaing meric-opological maps ha improves robo localizaion, in Proc. IEEE In l Conf. on Roboics and Auomaion (ICRA), [18] M. Kaess, A. Ranganahan, and F. Dellaer, isam: Incremenal smoohing and mapping, Trans. on Roboics, vol. 24, no. 6, pp , [19] J. M. Mooij, libdai: A free and open source C++ library for discree approximae inference in graphical models, J. Machine Learning Research, vol. 11, pp , [20] O. Mozos, C. Sachniss, and W. Burgard, Supervised learning of places from range daa using adaboos, in Proc. IEEE In l Conf. on Roboics and Auomaion (ICRA), [21] A. Pronobis, O. M. Mozos, B. Capuo, and P. Jensfel, Muli-modal semanic place classificaion, In l J. of Roboics Research, vol. 29, no. 2 3, pp , [Online]. Available: hp://