OPEN-EASE A Knowledge Processing Service for Robots and Robotics/AI Researchers

Transcription

1 OPEN-EASE A Knowldg Procssing Srvic for Robots and Robotics/AI Rsarchrs Michal Btz 1, Moritz Tnorth 1 and Jan Winklr { btz, tnorth, winklr }@cs.uni-brmn.d Abstract Making futur autonomous robots capabl of accomplishing human-scal manipulation tasks rquirs us to quip thm with knowldg and rasoning mchanisms. W propos OPEN-EASE, a rmot knowldg rprsntation and procssing srvic that aims at facilitating ths capabilitis. OPEN-EASE givs its usrs unprcdntd accss to th knowldg of lading-dg autonomous robotic agnts. It also provids th rprsntational infrastructur to mak inhomognous xprinc data from robots and human manipulation pisods smantically accssibl, and is complmntd by a suit of softwar tools that nabl rsarchrs and robots to intrprt, analyz, visualiz, and larn from th xprinc data. Using OPEN-EASE usrs can rtriv th mmorizd xprincs of manipulation pisods and ask quris rgarding to what th robot saw, rasond, and did as wll as how th robot did it, why, and what ffcts it causd. I. INTRODUCTION Within th nxt yars autonomous mobil manipulation robots will b incrasingly rqustd to accomplish humanscal manipulation tasks: autonomous warhous robots will b askd to ftch th itms on a givn ordr list and pack thm into a box, and houshold robots will b taskd to clan th tabl and unload th dishwashr [1], [2]. It is th natur of such human-scal manipulation tasks that thy ar incompltly spcifid and that thy rquir rasoning from background knowldg to b accomplishd succssfully. Warhous robots hav to infr how to grasp, handl, and plac th objcts thy ar to collct and pack. Thy also hav to rason about whthr to pad th itms, wrap thm, and possibly vn about how th cntr of mass of th box might chang by packing it in diffrnt ways. Whn claning a tabl, th robot has to rason about th stat of objcts, whthr thy ar clan or dirty, filld or mpty, and handl thm accordingly. Diffrnt xprimntal autonomous robot control systms hav bn proposd that nabl robots to accomplish such tasks by mploying knowldg- and rasoning-nabld control,.g. [3], [4], [5], [6]. Howvr, quipping autonomous robots with comprhnsiv knowldg and th corrsponding rasoning capabilitis is a difficult and tdious programming task that might rquir proficincy in AI rasoning mthods and non-standard AI programming languags. For tams not having a background in this fild, th barrirs for quipping thir robots with intllignt problm-solving capabilitis ar oftn vry high. 1 Author nams in alphabtical ordr. All authors ar with th Institut for Artificial Intllignc and th TZI (Cntr for Computing Tchnologis), Univrsity of Brmn, Grmany. W propos OPEN-EASE 1, a rmot knowldg rprsntation and procssing srvic that aims at facilitating th us of Artificial Intllignc tchnology for quipping robots with knowldg and rasoning capabilitis. OPEN-EASE provids its usrs with unprcdntd accss to th knowldg of lading-dg autonomous robotic agnts prforming human-scal manipulation tasks. It includs 1) knowldg about th robot s hardwar, its capabilitis, its nvironmnt and th objcts it manipulats; 2) mmorizd xprincs of manipulation pisods that allow to rason about what th robot saw, rasond, and did, how it did that, why, and what ffcts it causd; and 3) knowldg obtaind from training pisods in which humans dmonstrat skills that th robot can larn from. This information can b rtrivd by quris formulatd in PROLOG, a gnral-purpos logic programming languag. Ths quris can b snt ithr by humans via a wb-basd graphical intrfac, or by robots that accss OPEN-EASE via a wbsrvic API. This way, thy can qury and us OPEN-EASE s background knowldg to provid smantic maning to thir snsor data and to th data structurs thy us for control purposs. W plan to xtnd th systm to lt rsarchrs and robots upload thir own data structurs and xcution log fils, to dclar thir lmnts as PROLOG ruls, and thrby to convrt thir data fils into virtual OPEN-EASE knowldg bass. OPEN-EASE can also b viwd as a mans for promoting opn rsarch in th domain of AI-nabld autonomous robot manipulation. Aftr ntring th wbsit rsarchrs hav complt accss to comprhnsiv data sts of robots prforming ftch-and-carry tasks and to human dmonstrations of som of ths tasks. A standardizd smantic rtrival languag provids full accss to all data and nabls rsarchrs to combin individual sourcs of information. Sophisticatd softwar tools nabl rsarchrs to visualiz and analyz data through th wb-basd intrfac. This way, rsarchrs in machin larning will b abl to crat ralistic and highly rlvant robot larning problms. Rsarchrs in computr vision will b abl to turn ral prcption tasks and th corrsponding snsor data into bnchmark problms. Our fforts in dvloping OPEN-EASE and making it publicly availabl can b considrd to b in th spirit of Nilsn s vision of Rinvnting Discovry [7], which promots nw ways of conducting rsarch mor ffctivly 1 EASE is th abbrviation of Evryday Activity Scinc and Enginring.

2 UIMAPrcption_oS8i 1 dtctdobj PancakMix_okhs 3 Tim start 6 Fig. 1. t t im ndt ArmMovmnt_nJwX taskcontxt GRASP objactdon PancakMix_okhs ttim ndtim star t t BasMovmnt_HfzM taskcontxt NAVIGATE star ttim Tim nd T im ArmMovmnt_iNZC taskcontxt PUTDOWN objactdon PancakMix_okhs ttim star T nd im t t t t Wb intrfac of O PEN -EASE. Fig. 2. through th coopration facilitis providd by modrn Intrnt tchnology. Inspiring bluprints for such wb srvics that promot opn rsarch in othr domains includ th Alln Human Brain Atlas [8] and th HapMap projct [9] which nabls ntworkd scinc in human gnom rsarch. Th scintific and tchnical contributions of th papr ar th (1) comprhnsivnss with which ral xcution data of modrn autonomous manipulation robots is loggd, stord and mad opnly accssibl to th rsarch community; (2) th rprsntational infrastructur that is providd to mak vry inhomognous xprinc data from diffrnt robots and vn human manipulation pisods smantically accssibl in a uniform and standardizd concpt vocabulary; and (3) a suit of softwar tools that nabl rsarchrs and robots to intrprt, analyz, visualiz, and larn from th xprinc data. Th rmaindr of th papr is organizd as follows. W start with a dscription of O PEN -EASE from a usr prspctiv. Thn, w giv an ovrviw of its functional componnts, xplain its implmntation in dtail, and prsnt som xmplary us cass. W finish with outlining som projctd applications, with a discussion of rlatd work, and our conclusions. II. A G LIMPSE AT O PEN -EASE To th human usr, O PEN -EASE prsnts itslf through th wb-basd intrfac dpictd in Figur 1. Th wb intrfac includs pans with diffrnt purposs. Th Prolog intraction pan (1) allows th usr to typ Prolog quris and commands and to s th answrs to ths quris. A list of prpard quris with English translation is providd in th qury list pan (2). Th 3D display pan (3) can visualiz th robot and its nvironmnt. Othr information such as trajctoris, robot and objct poss can b addd and highlightd. Th blif pan (3) nabls th usr to inspct th intrnal data structurs of th robot s blifs including objct, action, and location dscriptions usd by th robot. Finally, thr is th imag pan (4) for displaying imags capturd by th robot s camra, and th visual analytics pan (5) which can visualiz statistical data as bar charts and pi charts. Th concptual viw that O PEN -EASE imposs on th log data of manipulation activity pisods is that of a first-ordr Tim intrval viw of th log data of a grasping action. tim intrval logic [10], [11], [12] as shown in Figur 2, somtims also rfrrd to as a chronicl rprsntation [13]. Tim intrvals ar spcifid through th tim points at thir start and nd. Ths ar linkd to th corrsponding Unix tim stamps, which allows synchronization with othr rcordd data such as capturd imags and robot poss. Evnts, such as a raching motion, occur ovr tim intrvals. Actions ar considrd as vnts that ar causd by an agnt to achiv som goal. Occasions, rprsnting for instanc th stat of an objct bing at som location, hold ovr tim intrvals. Instantanous vnts and continuous stats occur at a tim instant ti, which is a tim intrval with th duration 0. As an xampl, considr th tim chronicl rprsntation of a ftch-and-plac task dpictd in Figur 2. Th figur includs th vnts that occur during th pisod that ar assrtd in th rprsntation languag through a st of facts including th following ons: o c c u r s ( ArmMovmnt njwx, [ t 6, t 8 ] ). t a s k t y p ( ArmMovmnt njwx, G r a s p i n g ). o b j c t a c t d o n ( ArmMovmnt njwx, P a n c a k M i x o k h s ). c a t g o r y ( P a n c a k M i x o k h s, PancakMix ). b l i f a t ( robot ( pr2 bas, Pos 423 ), t7 ). occurs ( UIMAPrcption os8i, t2 ). c a t g o r y ( UIMAPrcption os8i, ObjctDtctd ).... This rprsntation allows us to ask sophisticatd quris that combin information from ths logical facts with continuous and gomtric aspcts such as robot poss in thrdimnsional spac. W can for instanc qury for a task Tsk with th goal of grasping an objct of typ cup, and rtriv th Pos of th robot in trms of global map coordinats /map at th nd tim point End of th grasping action. This qury is answrd basd on th loggd xprinc data; its rsult is dpictd in Figur 3 (lft).? - t a s k g o a l ( Tsk, [ an, a c t i o n, [ typ, grasp ], [ o b j c t a c t d o n, [ an, o b j c t, [ t y p, cup ] ] ] ] ), t a s k n d ( Tsk, End ), r o b o t p o s a t t i m ( PR2, / map, End, P o s ). In addition to quris for individual tim points, w can also rtriv trajctoris of arbitrary parts of th robot whil prforming an action. That is, w first rtriv th tim intrval btwn St and End, which dnot th bginning

3 Fig. 3. Visual rsults of quris on loggd robot xprincs. and th nd of th grasping action, rad which grippr was usd for that action from th log data, and add th trajctory btwn ths tims to th visualization:? - t a s k g o a l ( Tsk, [ an, a c t i o n [ typ, g r a s p ] ] ), t a s k o u t c o m ( Tsk, s u c c s s ), t a s k s t a r t ( Act, St ), t a s k n d ( Act, End ), t a s k u s d g r i p p r ( Act, Grp ), a d d t r a j c t o r y ( Grp, St, End ). Quris ar not limitd to th loggd xprinc data, but may also includ gnral background knowldg. Th following two quris us th robot modl to highlight componnts connctd to th lft arm (in rd) and all camras of th robot (in blu), rspctivly. Th rsults ar also shown in th figurs abov.? - sub componnt ( pr2 : p r 2 l f t a r m, Sub ), h i g h l i g h t o b j c t ( Sub ).? - o w l i n d i v i d u a l o f (Cam, s r d l 2 : Camra ), h i g h l i g h t o b j c t (Cam ). III. OVERVIEW OF OPEN-EASE OPEN-EASE can b considrd as a hug, rmotly accssibl knowldg srvic that consists of 1) a big-data databas storing comprhnsiv data about pisods in which humans and robots prform complx manipulation tasks; 2) an ontology, i.. an ncyclopdic knowldg bas, that provids a concptual modl of manipulation activitis; 3) and softwar tools for qurying, visualizing, and analyzing th manipulation task pisods. A. Databass of Manipulation Episods Th data providd by OPEN-EASE compriss ( ) raw snsor data and th rsults of thir intrprtation by th robot, ( ) loggd robot bhavior including pos data, ( ) th robots plans and thir intrprtation, ( ) a structurd, smantically labld nvironmnt modl, and ( ) objcts and thir poss in th scn. Loggd plan intrprtation data, th nvironmnt modl and objct dtctions ar rprsntd in th Wb Ontology Languag OWL [14]. This rprsntation can b loadd into th knowldg bas and is availabl for rasoning using tmporal logics as dscribd in th nxt sction. Snsor data and robot pos data, howvr, ar usually of much highr volum, and storing thm in OWL would lad to significant ovrhad. Ths kinds of data ar thrfor stord in MongoDB [15], an fficint schma-lss, high-volum Fig. 5. Uppr concpt taxonomy usd in OPEN-EASE. databas. Th data can b accssd from th knowldg bas by spcial prdicats that, using procdural attachmnts, transparntly load th rquird information and rlat it to th smantic modl. This approach nabls OPEN-EASE to rconstruct th stat of th robot including its pos and th computational stat of th plan intrprtation at any tim. In addition, all imags that hav bn usd for prcption tasks ar stord, togthr with th rsults that th prcption algorithms computd and th poss of all objcts that ar rlvant for th manipulation tasks. B. OPEN-EASE Concpt Vocabulary OPEN-EASE provids standardizd smantic accss to th databas dscribd in th prvious subsction. Th approach combins an ontology, which dfins a concptual modl of manipulation activitis, with a st of qury prdicats for rasoning about this concptual modl. Th KNOWROB ontology [16] that is usd by OPEN-EASE and dpictd in Figur 5 dfins vnts and tmporal things, actions, spatial things including objcts, stuff, and agnts, as wll as mathmatical concpts as its main concpts. Th complt taxonomy counts about classs, including about 130 action classs, 7000 objct typs and 150 robot-spcific concpts, that can b dscribd by ovr 300 kinds of proprtis. Ths classs and thir instancs ar dfind through a st of assrtions and can b linkd to subsymbolic data that dscribs thm in mor dtail. For xampl, objct classs can rfr to 3D modls of thir gomtry and thir composition from functional componnts, as shown in Figur 6 for a bottl of pancak mix. Ths combind smantic-gomtric modls can b gnratd automatically from common 3D modls as thy can b found in public databas in th Intrnt [17]. Ths concpts provid th vocabulary that is usd by a st of prdicats for rprsnting and rasoning about robot activity pisods, th most important of which ar listd in Figur 4. Thy ar thmatically groupd into prdicats about

4 Prdicats on occasions, blifs, and vnts Prdicats on plans and plan intrprtation holds(occ,t i ) Th occasion (flunt) Occ holds task(tsk) Tasks on th intrprtation stack in th tim intrval T i task typ(tsk, Typ) Typ of this task lmnt blif at(occ,t i ) Th robot blivs at T i that task goal(tsk, G) Goal of task th occasion Occ holds at T i task start(tsk,t i ) Start tim of task occurs(ev, T i ) Evnt Ev occurs in tim intrval T i task nd(tsk,t i ) End tim of task Occasion typs task usd grippr(tsk, Grp) Grippr that has bn usd for a Tsk loc(obj, Loc) Location of an objct subtask(tsk,sub) Task is a parnt of Subtask objct visibl(obj) Objct is visibl to th robot subtask + (Tsk,Sub) Task is an ancstor of Subtask robot(part,loc) Location of th robot part Part task outcom(tsk,rs) Rsult of task (succss or fail) Evnt typs task failur(task,failur) Failur of a task objct prcivd(obj) Objct has bn prcivd failur typ(failur,typ) Typ of failur imag capturd(img) Imag has bn capturd failur attribut(failur, Nam, Val) Failur attribut (.g. rror mssag) Objct dscriptions dsig typ(dsig,tp) Typ of dsignator dsig prop(dsig,prop,val) Proprty valus of dsignator dsig pos(dsig,pos) Pos of prcivd objct dsignator matchs(dsig,dscr) Match dsignator to dscription Fig. 4. Prdicats for rasoning about th mmorizd xprincs. Fig. 6. PancakMix_okhs typ: knowrob:'pancakmix' linktocadmodl: pancak-mix.da proprphysicalparts: 'BottlCap463' proprphysicalparts: 'Containr461' BottlCap463 typ: knowrob:'cylindr' typ: knowrob:'handl' radius: 1.96, 'cm' lngth: 0.8, 'cm' Containr461 typ: knowrob:'cylindr' typ: knowrob:'handl' typ: knowrob:'containr' volum: 500, 'ml' Concptual knowldg about a bottl of pancak mix. occasions, blifs, and vnts; prdicats about plans and plan intrprtation; vnt typs; occasion typs; and objct dscriptions. Whil ths prdicats provid th usr with a uniform, logical viw on th data, thy may b computd at qury tim from diffrnt information sourcs. W call this concpt a virtual knowldg bas that is cratd on top of th smi-structurd and oftn high-volum log data [18]. Th contnt of th symbolic knowldg bas is computd on dmand as an abstractd viw on th subsymbolic data and is thrfor fully groundd in th robot s intrnal data structurs. Having a common symbolic rprsntation allows to asily combin data from diffrnt sourcs and to answr th sam quris on data sts of diffrnt structur. For xampl, som robots may not b controlld by a sophisticatd plan-basd controllr and hav lss information in th log fils. Howvr, as long as th following prdicats can b implmntd on top of th loggd data, th sam quris can b answrd. Th prdicats holds, blivs at, and occurs rprsnt diffrnt aspcts of a changing world stat: holds(occ,t i ) is tru iff th occasion typ Occ is tru throughout th tim intrval T i. Occasions (also calld flunts [19], [20]) rprsnt tim-varying stats of th world such as th locations of objcts. Formally, an occasion is a functional trm in logic that maps an occasion typ such as mpty(cup-23) into th tim intrvals in which cup-23 was mpty. Thus, th assrtion holds(mpty(cup-23),t-46) is tru if t-46 is a sub-intrval of a tim intrval in which th cup was mpty. Th prdicat blivs at(occ,t i ) is similar to holds(occ,t i ), with th diffrnc that blivs at rprsnts a blif of th robot rathr than th tru world stat. Typically, w distinguish btwn holds and blivs at only if w hav xtrnal dvics for obsrving manipulation pisods that can provid us with ground truth data, and rturn idntical valus othrwis. Th prdicat occurs(ev,t) stats that vnt Ev occurs throughout th tim intrval T. For xampl, th assrtion occurs(contact-13,t-31) stats that th contact vnt contact-13 has occurrd in th tim intrval t-31. In addition to changs of th outsid world, th pisods also dscrib th procss of plan intrprtation. In this contxt, w man by a task tsk (assrtion: task(tsk) th intntion of th robot to xcut a pic of th control program Exp. Th tasks that a robot gnrats whn xcuting its plan ar organizd in a hirarchical task ntwork (bcaus plan stps can b xcutd concurrntly). Th task hirarchis ar rprsntd through th prdicat subtask(tsk 1,tsk 0 ) stating that tsk 1 is a subtask of task tsk 0. Thus, if w ar intrstd how a task was xcutd, w hav to xplor th subtask rlations. If w want to infr why a task was xcutd, w hav to analyz its supr tasks. Th rprsntation furthr includs snsory vnts: Whnvr th camra drivr gts th command to captur an imag, an vnt of th form occurs(imag capturd(i), T) is automatically assrtd, and th corrsponding imag is stord as an ffct of th vnt. Rlatd to this ar th objct dscriptions th robot uss to manipulat objcts which ar calld dsignators in our systm. In th initial plan, th dscriptions might b abstract such as th grn cup on th kitchn countr. Whil th robot sarchs for and finds th cup, this abstract dscription will b updatd with th data xtractd from capturd imags: th xact pos of th cup, its siz, a bounding box, and so on. Th paramtrization of raching and grasping actions havily dpnds on th dscriptions of th objcts to b manipulatd, so thir valus and th volution ovr tim ar containd in th pisods. Information about dsignators can b rtrivd using th occasions dsig pos and dsig attribut. A snapshot of ths dscriptions is stord whnvr thy ar updatd, allowing

5 Prolog shll json_prolog ROS nod KnowRob Local visualization canvas Visualization markr publishr /visualization_markr topic Intrnt Wbsockt intrfac rosbridg Prolog shll json_prolog ROS nod KnowRob Local visualization canvas Visualization markr publishr /visualization_markr topic Intrnt Wbsockt intrfac rosbridg th systm to rason about th volution of th robot s blif ovr tim by comparing th dscriptions bfor and aftr an action. Th diffrncs ar usually thos attributs that can b infrrd from prcption, such as th pos and siz of th objct. OPEN-EASE provids th occasion matchs(dsig,dscr) to rason about objct dscriptions Dscr and whthr or not th robot blivs that thy ar satisfid by objcts in th world (Dsig). Th logical xprssion blivs at(matchs(dsig, [an, objct, [typ, cup], [color, rd]]), t) is tru for vry objct dscription Dsig that th robot blivs to rprsnt a rd cup. This logical qury languag is complmntd by a st of prdicats for loading and rasoning about lowr-lvl data and for visualizing th rsults of quris. Whn daling with gomtric information that changs ovr tim, sophisticatd mthods for transforming poss btwn coordinat frams ar rquird. W xtndd th widly-usd tf library in ROS to oprat on th databas of loggd data, offring th sam intrfac that is usd for runtim opration. C. Softwar Tools for Rcording, Qurying, Visualizing, and Analyzing Episods Th OPEN-EASE systm coms with a suit of softwar tools for logging data from robot manipulation pisods, for rasoning about thm, and for visualizing th rsults. Our robots prform thir tasks undr th suprvision of th CRAM xcutiv [21] that automatically rcords comprhnsiv log data as dscribd in [18]. Th approach is not limitd to robots running CRAM, and th sam analyss can b prformd on vry diffrnt log data as long as prdicats in th prvious sction can b computd from it. Howvr, th data producd by CRAM is much mor comprhnsiv and smantically rich than logs of othr xcutivs such as SMACH [22]. Th systm, including th qury-answring moduls and th wb intrfac, can ithr b usd as a hostd cloud srvic, or b downloadd as opn-sourc softwar 2 and installd locally. IV. IMPLEMENTATION KNOWROBS Th OPEN-EASE systm has bn implmntd in a cloud-basd vrsion of th KNOWROB robot knowldg bas [16]. KNOWROB provids xprssiv rprsntations and sophisticatd rasoning mthods that ar tailord to th nds of autonomous robots. Low-lvl data from robot and human activitis ar loggd into a big data databas using an xtnsion of th mongodb log tool [23]. Th mthods for rcording and rasoning about highr-lvl xprinc data ar basd on our own prior work [18]. Th communication btwn th browsr and th ROS systm in th cloud, as wll as many of th graphical lmnts in th qury frontnd, hav bn built using th robotwbtools framwork [24]. OPEN-EASE has to fulfill an important tchnical rquirmnt, namly to quip ach usr with hr own individual knowldg bas. This is ncssary bcaus usrs hav to load and unload knowldg bass from thir own 2 Installation instructions: and from common rpositoris to prform thir xprimnts, and will also assrt additional facts and ruls to work with th knowldg bas. This capability is providd by KNOWROBS, a Softwar-as-a-Srvic cloud application which offrs KNOWROB functionality to rmot usrs that can connct to a WbSockt [25] using th rosbridg [26] protocol. Wb sockts ar supportd by most modrn browsrs, but can also asily b implmntd as part of a clint application on a robot. KNOWROBS uss th highly fficint virtualization tchniqus of th Dockr framwork 3 to crat sparat virtual knowldg bass for ach usr. Instad of mulating a computr s hardwar, Dockr isolats procsss which still run on th sam Linux krnl w.r.t. procss IDs, mmory and storag rsourcs, computing tim, ntwork intrfacs, usr rights tc. Ths capabilitis allow us to provid individual knowldg bass to diffrnt usrs without prohibitiv usag of mmory and computing rsourcs. wbrob Port usr1 usr1_data /hom/ros/sandbox Privat containrs pr usr usr2 Port usr2_data /hom/ros/sandbox Common, shard containrs mongo_db MongoDB mongo_data /data/db knowrob_data /hom/ros/knowrob_data Fig. 7. Structur of th proposd systm. Each usr has a privat knowldg bas containr, but can transparntly accss shard datasts of robot data. Figur 7 visualizs th architctur of th KNOWROBS systm. Th wb-basd frontnd manags th diffrnt containrs and assigns thm to usrs onc thy log into th systm. Each usr has on containr with a complt KNOWROB systm, plus a containr for prsistnt data storag. In addition, w hav containrs that ar shard among all usrs, such as a common databas with loggd high-volum robot data and a common rpository of mmory pisods. V. USE CASES All of th following us cass hav bn ralizd using th sam approach. 1) A computr systm is gnrating or obsrving manipulation activitis. 2) Th systm is xtndd through a big-data logging mchanism that logs th highvolum and in particular subsymbolic systm stat data as comprhnsivly as possibl without slowing down th systm opration. 3) Th loggd data ar symbolically annotatd and intrprtd as instancs of th symbolic concpts in th KNOWROB taxonomy such that thy can b smantically indxd. 4) W can thn us th Prolog languag togthr with 3

6 th prdicats listd in Figur 4 to rason about manipulation pisods and answr quris about thm. A. Working with Robot Manipulation Episods Th combination of a powrful rprsntation and logicbasd qury languag with comprhnsiv gomtric information nabls robots to rconstruct th stat of th world as th robot blivd it to b at a smantically dscribd point in tim, for xampl at th momnt whn grasping a cup (Figur 8 lft). This allows th a-postriori analysis of failur situations, which can b vry hlpful in cas of incidntal problms that ar vry difficult to trac othrwis. By tsting nw algorithms on th loggd snsor data, on could tst whthr thy would hav prformd bttr in that rspctiv situation. By stting up a simulator with th loggd world stat, on could vn combin ths prcption rsults with (simulatd) robot actions. Fig. 8. Exampl us cas of rading subsymbolic information on poss and trajctoris basd on logical quris. Both robot log data and obsrvations of human actions can b qurid in th sam way. B. Working with Human Dmonstration Episods Th sam quris that can b answrd on loggd robot xprincs can also b answrd basd on obsrvations of human activitis (Figur 8 right). Th obvious rstriction is that information about th plans and intntions of th human is not availabl, but only th xtrnal obsrvation of th rsulting bhavior. If w annotat th obsrvations, ithr manually or using automatd activity rcognition mthods, w can us th sam mthods for smantic rtrival of obsrvd data that could.g. hlp with slcting data for analysis or larning purposs. C. Working with Collctions of Episods Having not only on, but a collction of pisods allows to comput statistical information that can b usd for valuation of th robot s prformanc and for larning prdiction modls. This hlps to answr qustions such as how long actions tak on avrag, how rliabl thy ar, and which failurs occur most frquntly. Figur 9 shows two xampls: Th majority of th rrors is of typ ManipulationPosUnrachabl, as can b sn in th chart on th lft, which suggsts that improvmnts of this componnt can hav a strong impact on th ovrall prformanc. Th avrag duration of tasks, shown in th chart on th right, givs robots and humans information on how long actions typically tak,.g. for schduling purposs. Fig. 9. Statistics computd from 25 manipulation pisods. Lft: Distribution of failur typs. Right: Avrag duration of common subtasks. D. Robots Using OPEN-EASE Whil w prsnt th rsults of quris as graphical visualizations in th wb-basd qury frontnd, th sam intrfac can b usd by robots to snd quris to a KNOWROB instanc in th cloud. In a local ROS stup, robots snd Prolog quris to KNOWROB ncodd in th JSON format via th json prolog srvic. KNOWROBS provids th sam intrfac, tunnld via a WbSockt connction. VI. PROJECTED APPLICATIONS Bsids bing a potntially powrful rmot knowldg procssing srvic for AI / Robotics rsarchrs and autonomous robots, w currntly invstigat a numbr of possibly high-impact applications of KNOWROBS, namly using KNOWROBS for ralizing 1) an Larning tool in AI-basd robotics. W ar using OPEN-EASE as a wb tool for taching a cours in intllignt robotics to lt studnts xplor th hardwar of robots, thir snsors and ffctors, and to gt bttr intuitions about th data that snsors gnrat ( Can you dtct th handls of cups using imags whr th camra is positiond at last 1,5m away from th cup?, or Which objcts or objct parts in th kitchn nvironmnt cannot b dtctd with th Kinct snsor of th robot? ). In addition, w lt th studnts do studnts xrciss with ral robot data, such as larning objct classifirs for th objcts that stand on th kitchn countr during a st of manipulation pisods. 2) a tool for rproducibl xprimntal data. OPEN- EASE improvs th rproducibility of xprimntal data. Considr th cas whr an xprimntal valuation of a scintific publication has to b xtndd aftr som tim. Rrunning th xprimnts is tdious and tim consuming and rquirs a similar hardwar stup. Th comprhnsiv log data collctd and th smantic rtrival facilitis supports rsarchrs to add missing valuations on th xisting xprimntal data. Rsarchrs can vn giv rviwrs and radrs accss to th data through OPEN-EASE, which allows thm to clarify qustions rgarding th xprimntal stting (.g. whr th robot stood whn th objct rcognition mchanisms succdd, or in which scns an objct could not b rcognizd). 3) a tool for opn robotics rsarch. Rcntly, progrss in many rsarch filds has bn fuld by making larg volums of data availabl and by running data and information analytics tools on thm, or by simply visually browsing and sarching through ths data. Exampls of such

7 initiativs ar th Alln Brain Atlas, gographic data, tc. OPEN-EASE currntly provids data from robotic agnts prforming ftch and plac tasks in a kitchn nvironmnt, usrs dmonstrating pancak making in a virtual rality gam, popl stting th tabl and claning up (from th TUM kitchn datast [27]). W plan to includ xprinc data from robotic agnts prforming chmical xprimnts, human-robot coopration, and othrs. This maks OPEN- EASE on of th most comprhnsiv and dtaild activity knowldg bass rlvant for autonomous robotics rsarch. 3) a tool for crating ralistic bnchmark problms for machin larning and robot prcption. Anothr possibl usag of OPEN-EASE is th cration of ralistic bnchmark datasts. Considr th cas whr you want to crat a ralistic st of robot prcption tasks in ordr to tst som nwly dvlopd robot prcption mthod. To do so, you could tak charactristic vryday activitis such as stting th tabl and first qury OPEN-EASE for th st of prcption tasks that a robot issus in ordr to prform such an activity. This analysis will rsult in som undrstanding of which typs of prcption tasks ar important and which ons ar not. W can thn gnrat ground-truth data by finding th tim instant whr th robot has th most complt and dtaild information about a spcifid scn, such as th objcts on a tabl. If ndd, th usr can assrt additional knowldg or corrct information by rtracting and assrting KNOWROB facts. Finally, th usr can mak up situations in which th prcption tasks ar to b prformd. 4) a tool for grounding and assssing th assumptions and infrnc mchanisms of action formalizations in knowldg rprsntation. Most knowldg rprsntation languags and mthods for symbolically rasoning about actions and chang ar basd on modling assumptions. OPEN-EASE givs rsarchrs in ths filds th opportunity to tst to what xtnt such assumptions ar valid for autonomous manipulation robots, and to what xtnt th infrncs prformd by ths formalisms ar valid with rspct to th bhavior and th physical ffcts that robotic agnts gnrat. VII. RELATED WORK OPEN-EASE is positiond in th intrsction of intllignt information systms and rmot softwar srvics for robots. In robotics, th Robo Brain 4 projct ld by Saxna and his collagus is most closly rlatd to OPEN-EASE. Robo Brain is a larg-scal computational systm that larns from publicly availabl Intrnt rsourcs, computr simulations, and ral-lif robot trials. It accumulats vrything into a comprhnsiv and intrconnctd knowldg bas. OPEN-EASE diffrs from Robo Brain in svral aspcts. OPEN-EASE incorporats data from diffrnt sourcs into a common, formal knowldg rprsntation languag with powrful infrnc mchanisms. Th data ar automatically gnratd through robots and obsrvation systms rathr than human computation mthods. Unlik in th Robo Brain 4 projct, howvr, which alrady broadly applis larning to th data, th larning fforts in OPEN-EASE hav not startd yt. OPEN-EASE follows up on rsarch that aims at providing knowldg bass for robots in th world-wid wb. Thos rsarch fforts includ RoboEarth [28] which invstigats how robots can shar thir knowldg by providing a mta rprsntation languag for robot plan schmata and knowldg bass that allows robots to upload, find, and download availabl knowldg and for chcking whthr thy can mploy th rspctiv knowldg [28], [29], [30]. OPEN-EASE is a cloud robotics application [31], [32], [33] that is spcializd for providing robots with knowldg. Othr xampls of cloud srvics ar Mujin 5, providing clint robots with motion planning capabilitis, and Googl goggls [34] that can rtriv wb pags matching capturd imags. Wb srvics ar mor common in th ara of intllignt information systms. Hr, WordNt 6, a dictionary knowldg bas, OpnCyc 7, an ncyclopdic knowldg bas, and th OpnMind and th OpnMind Indoor Common Sns [35] common-sns knowldg bass, th Googl Knowldg Graph [36] and DBpdia [37] ar popular xampls. Howvr, ths knowldg srvics focus primarily on txt and symbolic data, whil OPEN-EASE contains larg amounts of snsory data of many modalitis. Finally, thr ar also a numbr of data rpositoris for human manipulation activity data offring rlvant data, though not as formally rprsntd knowldg nor as a rady-to-us wb srvic. Such data rpositoris includ th TUM kitchn datast [27], th MPII Cooking Activitis datast![38], and th CMU MMAC datast [39]. VIII. CONCLUSIONS In this papr w hav dscribd and discussd OPEN- EASE, a rmot knowldg rprsntation and procssing srvic for human rsarchrs and robots. OPEN-EASE nabls its usrs to intrprt, analyz, visualiz, and larn from th xprinc data from robots and human manipulation pisods. Using OPEN-EASE usrs can rtriv th mmorizd xprincs of manipulation pisods and ask quris rgarding what th robot saw, rasond, and did as wll as how th robot did it, why, and what ffcts it causd. OPEN-EASE is uniqu bcaus of (1) th comprhnsivnss with which ral xcution data of modrn autonomous manipulation robots is loggd, stord and mad opnly accssibl to th rsarch community; (2) th rprsntational infrastructur that is providd to mak vry inhomognous xprinc data from diffrnt robots and vn human manipulation pisods smantically accssibl in a uniform and standardizd concpt vocabulary; and (3) a suit of softwar tools that nabl rsarchrs and robots to intrprt, analyz, visualiz, and larn from th xprinc data. Projctd applications of OPEN-EASE includ its us an Larning tool in AI-basd robotics, a tool for making

8 rproducibl xprimntal data accssibl and for nabling smantic information rtrival, and a tool for opn robotics rsarch. OPEN-EASE can b accssd through th wb pag ACKNOWLEDGMENTS This work has bn supportd in part by th EU FP7 Projct RoboHow (grant numbr ). W would lik to thank Arn Stfs, Asil Kaan Bozcuoglu and Danil Bßlr for thir hlp with th visualization tools and th qury library. REFERENCES [1] SPARC Th Partnrship for Robotics in Europ, Stratgic Rsarch Agnda for Robotics in Europ , urobotics aisbl, Tch. Rp., 2014, SPARC.pdf. [2] J. Hollrbach, M. Mason, and H. Christnsn, A Roadmap for US Robotics From Intrnt to Robotics, Computing Community Consortium (CCC), Tch. Rp., [3] M. Btz, D. Jain, L. Mösnlchnr, M. Tnorth, L. Kunz, N. Blodow, and D. Pangrcic, Cognition-nabld autonomous robot control for th ralization of hom chor task intllignc, Procdings of th IEEE, vol. 100, no. 8, pp , [4] B. C. Williams, M. Ingham, S. H. Chung, and P. H. Elliott, Modlbasd Programming of Intllignt Embddd Systms and Robotic Spac Explorrs, Procdings of th IEEE: Spcial Issu on Modling and Dsign of Embddd Softwar, vol. 9, no. 1, pp , January [5] R. Alami, R. Chatila, A. Clodic, S. Flury, M. Hrrb, V. Montruil, and E. A. Sisbot, Towards human-awar cognitiv robots, in AAAI-06, Stanford Spring Symposium, [6] K. Okada, M. Kojima, S. Tokutsu, Y. Mori, T. Maki, and M. Inaba, Intgrating rcognition and action through task-rlvant knowldg for daily assistiv humanoids, Advancd Robotics, vol. 23, pp (22), March [7] M. Nilsn, Rinvnting discovry: Th Nw Era of Ntworkd Scinc. Princton Univrsity Prss, [8] M. J. Hawrylycz, E. S. Lin, A. L. Guillozt-Bongaarts, E. H. Shn, L. Ng, J. A. Millr, L. N. van d Lagmaat, K. A. Smith, A. Ebbrt, Z. L. Rily, t al., An anatomically comprhnsiv atlas of th adult human brain transcriptom, Natur, vol. 489, no. 7416, pp , [9] R. A. Gibbs, J. W. Blmont, P. Hardnbol, T. D. Willis, F. Yu, H. Yang, L.-Y. Ch ang, W. Huang, B. Liu, Y. Shn, t al., Th intrnational hapmap projct, Natur, vol. 426, no. 6968, pp , [10] D. McDrmott, A Tmporal Logic for Rasoning About Procsss and Plans, Cognitiv Scinc, vol. 6, no. 2, pp , [11], Rasoning about plans, in Formal Thoris of th Commonsns World, J. R. Hobbs and R. C. Moor, Eds. Norwood, NJ: Ablx, 1985, pp [12] J. Alln, Maintaining knowldg about tmporal intrvals, Communications of th ACM, vol. 26, no. 11, pp , [13] M. Ghallab, On Chronicls: Rprsntation, On-lin Rcognition and Larning, in Intrnational Confrnc on th Principls of Knowldg Rprsntation and Rasoning. Morgan Kaufmann Publishrs, 1996, pp [14] W3C, OWL 2 Wb Ontology Languag: Structural Spcification and Functional-Styl Syntax. World Wid Wb Consortium, 2009, [15] K. Chodorow and M. Dirolf, MongoDB: Th Dfinitiv Guid. O Rilly & Associats, [16] M. Tnorth and M. Btz, KnowRob A Knowldg Procssing Infrastructur for Cognition-nabld Robots, Intrnational Journal of Robotics Rsarch (IJRR), vol. 32, no. 5, pp , April [17] M. Tnorth, S. Profantr, F. Balint-Bnczdi, and M. Btz, Dcomposing CAD Modls of Objcts of Daily Us and Rasoning about thir Functional Parts, in IEEE/RSJ Intrnational Confrnc on Intllignt Robots and Systms (IROS), Tokyo Big Sight, Japan, Novmbr , pp [18] J. Winklr, M. Tnorth, A. K. Bozcuoglu, and M. Btz, CRAMm mmoris for robots prforming vryday manipulation activitis, Advancs in Cognitiv Systms, vol. 3, pp , [19] J. McCarthy and P. J. Hays, Som philosophical problms from th standpoint of artificial intllignc, in Machin Intllignc 4, B. Mltzr and D. Michi, Eds. Edinburgh Univrsity Prss, 1969, pp [20] R. Ritr, Knowldg in Action: Logical Foundations for Spcifying and Implmnting Dynamical Systms. MIT prss, [21] M. Btz, L. Mösnlchnr, and M. Tnorth, CRAM A Cognitiv Robot Abstract Machin for Evryday Manipulation in Human Environmnts, in Procdings of th IEEE/RSJ Intrnational Confrnc on Intllignt Robots and Systms, Taipi, Taiwan, Octobr , pp [22] J. Bohrn and S. Cousins, Th SMACH High-Lvl Excutiv, IEEE Robotics and Automation Magazin, vol. 17, pp , [23] T. Nimullr, G. Lakmyr, and S. S. Srinivasa, A Gnric Robot Databas and its Application in Fault Analysis and Prformanc Evaluation, in Procdings of th IEEE/RSJ Intrnational Confrnc on Intllignt Robots and Systms Vilamoura, Algarv, Portugal: IEEE/RAS, [24] B. Alxandr, K. Hsiao, C. Jnkins, B. Suay, and R. Toris, Robot wb tools [ros topics], Robotics & Automation Magazin, IEEE, vol. 19, no. 4, pp , [25] I. Ftt and A. Mlnikov, Th WbSockt Protocol, RFC 6455 (Proposd Standard), Intrnt Enginring Task Forc, Dc [Onlin]. Availabl: [26] C. Crick, G. Jay, S. Osntoski, B. Pitzr, and O. C. Jnkins, Rosbridg: Ros for non-ros usrs, in Procdings of th 15th Intrnational Symposium on Robotics Rsarch, [27] M. Tnorth, J. Bandouch, and M. Btz, Th TUM Kitchn Data St of Evryday Manipulation Activitis for Motion Tracking and Action Rcognition, in IEEE Intrnational Workshop on Tracking Humans for th Evaluation of thir Motion in Imag Squncs (THEMIS), in conjunction with ICCV2009, [28] M. Waibl, M. Btz, R. D Andra, R. Janssn, M. Tnorth, J. Civra, J. Elfring, D. Gálvz-Lópz, K. Häussrmann, J. Montil, A. Przylo, B. Schißl, O. Zwigl, and R. van d Molngraft, RoboEarth - A World Wid Wb for Robots, Robotics & Automation Magazin, vol. 18, no. 2, pp , [29] M. Tnorth, A. C. Przylo, R. Lafrnz, and M. Btz, Th RoboEarth languag: Rprsnting and Exchanging Knowldg about Actions, Objcts, and Environmnts, in IEEE Intrnational Confrnc on Robotics and Automation (ICRA), St. Paul, MN, USA, May , bst Cognitiv Robotics Papr Award. [30] M. Tnorth, U. Klank, D. Pangrcic, and M. Btz, Wb-nabld Robots Robots that Us th Wb as an Information Rsourc, Robotics & Automation Magazin, vol. 18, no. 2, pp , [31] E. Guizzo, Robots with thir hads in th clouds, Spctrum, IEEE, vol. 48, no. 3, pp , [32] K. Goldbrg and B. Kho, Cloud robotics and automation: A survy of rlatd work, EECS Dpartmnt, Univrsity of California, Brkly, Tch. Rp. UCB/EECS , Jan [Onlin]. Availabl: [33] D. Hunzikr, M. Gajamohan, M. Waibl, and R. D Andra, Rapyuta: Th roboarth cloud ngin, in IEEE Intrnational Confrnc on Robotics and Automation (ICRA), [34] B. Kho, A. Matsukawa, S. Candido, J. Kuffnr, and K. Goldbrg, Cloud-basd robot grasping with th googl objct rcognition ngin, in IEEE Intrnational Confrnc on Robotics and Automation (ICRA), [35] R. Gupta and M. J. Kochndrfr, Common sns data acquisition for indoor mobil robots, in Nintnth National Confrnc on Artificial Intllignc (AAAI-04, 2004, pp [36] J. Edr, Knowldg graph basd sarch systm, 2012, us Patnt App. 13/404,109. [Onlin]. Availabl: US [37] S. Aur, C. Bizr, G. Kobilarov, J. Lhmann, R. Cyganiak, and Z. Ivs, Dbpdia: A nuclus for a wb of opn data, Th Smantic Wb, pp , [38] M. Rohrbach, S. Amin, M. Andriluka, and B. Schil, A databas for fin graind activity dtction of cooking activitis, in 2012 IEEE Confrnc on Computr Vision and Pattrn Rcognition (CVPR), Providnc, Unitd Stats, Jun [39] F. D la Torr, J. Hodgins, J. Montano, S. Valcarcl, and J. Macy, Guid to th Carngi Mllon Univrsity Multimodal Activity (CMU- MMAC) Databas, CMU-RI-TR-08-22, Robotics Institut, Carngi Mllon Univrsity, Tch. Rp., 2009.