ARobotLearningfromDemonstrationFrameworktoPerform Force-based Manipulation Tasks

Transcription

1 Noname manuscrip No. (will be insered by he edior) ARoboLearningfromDemonsraionFrameworkoPerform Force-based Manipulaion Tasks Received: dae / Acceped: dae Absrac This paper proposes an end-o-end learning from demonsraion framework for eaching forcebased manipulaion asks o robos. The srenghs of his work are many-fold: firs, we deal wih he problem of learning hrough force percepions exclusively. Second, we propose o exploi hapic feedback boh as a means for improving eacher demonsraions and as a human-robo ineracion ool, esablishing a bidirecional communicaion channel beween he eacher and he robo, in conras o works using kinesheic eaching. Third, we address he well-known wha o imiae? problem from a differen poin of view, based on he muual informaion beween percepions and acions. Lasly, he eacher s demonsraions are encoded using a Hidden Markov Model, and he robo execuion phase is developed by implemening a modified version of Gaussian Mixure Regression ha uses implici emporal informaion from he probabilisic model, needed when ackling asks wih ambiguous percepions. Experimenal resuls show ha he robo is able o learn and reproduce wo differen manipulaion asks, wih a performance comparable o he eacher s one. Keywords Programming by demonsraion Imiaion learning Hapic percepion Muual Informaion HMM GMR Roboic Manipulaion 1BackgroundandRelaedWork One of he main challenges in Roboics is o develop robos ha can inerac wih humans in a naural way, sharing he same dynamic and unsrucured environmens. Learning from demonsraion (LfD) 1 is a ype of human-robo ineracion (HRI) whose purpose is o ransfer knowledge or skills o he robo. Here, a human user carries ou examples of a given ask while a robo observes hese execuions and exracs relevan informaion for learning, represening and reproducing he augh ask under unknown condiions [1, 2]. HRI requires suiable communicaion channels for conveying informaion beween a human and a robo [3,4]. In LfD, mos exising works rely on vision or on moion sensors as inpu channels o he roboic sysem. As for vision-based inpu, posiional informaion abou he objecs in he scene is capured wih a se of cameras, which are also used o locae and follow markers placed on he eacher s body [5, 6]. Mos sae-of-hear approaches consider vision as he bes choice for exracing informaion from eacher examples, as human beings do in everyday asks [7, 8]. However, visionbased sysems mus deal wih ypical problems as occlusion, appearance changes and complex human-robo kinemaics mapping, which can be parly solved by using moion sensors insead. Such ype of sensors allows o rack he eacher s moion more precisely and o esablish a simpler mapping, which make hem appropriae o each asks o humanoid robos [5,9 11]. In conras o hese works, we are concerned wih learning from force-based percepions. Force conveys relevan informaion for several asks where vision or moion sensors can no provide sufficien daa o learn a moion or a se of primiives. In many daily asks, people use force-based percepions o perform successfully. Examples include assembly processes, opening doors, pulling drawers, cuing slices of bread, ec. Robos 1 Also known as programming by demonsraion or imiaion learning.

2 2 Fig. 1 Enire learning framework. (Top)Task learning sage.(boom) Robo execuion sage.the filering module consiss of he signal processing o achieve a high fideliy bidirecional communicaion channel. The feaure selecion block corresponds o he proposed soluion for he Wha o Imiae? problem hrough muual informaion analysis. may also ake advanage of force-orque informaion for learning his kind of asks. Evrard e al. [12] proposed a learning srucure similar o ours, where a humanoid robo learns o carry ou collaboraive manipulaion asks (objec-lifing along a verical axis) using force daa. An exension of his research [13], combines LfD and adapive conrol for eaching he ask, endowing he robo wih variable impedance and an adapive algorihm o generae differen reference kinemaic profiles depending on he perceived force. Kormushev e al. [14] proposed o use a hapic device for defining he force profile of conac-based asks (ironing and door opening) while he robo follows a previously learned rajecory. This cied work uses kinesheic guidance and does no exploi hapic feedback as a ool for improving eacher demonsraions, hus avoiding several challenging problems arising when clean and realisic signals are o be displayed o he human during he demonsraions. We conribue a complee force daabased learning framework ha includes filering processes and high-fideliy hapic feedback. This esablishes a force-based bidirecional communicaion channel, which has seldom been exploied as a human-robo ineracion ool in LfD in conras o kinesheic-based eaching and vision-based sysems. Anoher poin o be addressed in LfD is relaed o he learning level of he ask. Teacher demonsraions may be encoded a a symbolic level, where he ask is ofen represened as sae-acion pairs in a graph-like srucure [15, 16], or as moion primiives following hierarchical approaches [17, 18]. A rajecory level, on he oher hand, he aim is ha he robo exracs a generalized rajecory (movemen) from slighly differen eacher execuions [19 21]. Unlike he cied works, our conribuion mixes conceps from hese wo levels as he same goal may be reached from differen rajecories or iniial saes of he ask. We propose o encode he demonsraions hrough a se of saes using a Hidden Markov Model (HMM) and o execue he skill using a modified version of Gaussian Mixure Regression (GMRa) ha explois he emporal coherence of he ask a hand. Regardless of he paricular approach, researchers have o deal wih hree main problems: wha o imiae?, how o imiae? and when o imiae? [22]. The firs quesion refers o exracing he mos relevan informaion of he ask necessary o learn and reproduce i successfully. The second key quesion addresses he problem of how he robo can reproduce he ask based on he eacher execuions. The hird problem is relaed o he iming of he learning phase based on he robo percepions (observaions). In his paper, we propose o solve he firs issue hrough Muual Informaion () analysis, and o ackle he second problem hrough an HMM/GMRa-based framework, as shown in Figure 1. This paper is organized as follows: Secion 2 describes our experimenal seup and he wo manipulaion asks augh o he robo. Secion 3 explains how we ackle he wha o imiae? problem by exracing he mos relevan feaures of he asks via. Afer, in Secions 4 and 5, he learning and reproducion phases are described respecively, firs, illusraing he saisical encoding of he demonsraions by using an HMM and hen, showing how he implici emporal informaion in HMM is exploied a he execuion sage (he enire process is shown in Figure 1). Secion 6 shows compuaional and robo execuion resuls. Finally, he conclusions of his paper and fuure work are presened in Secion 7.

3 A Robo Learning from Demonsraion Framework o Perform Force-based Manipulaion Tasks 3 2ExperimenalSeups To es our learning framework based on force percepions, we consruced an experimenal seup o each a roboic manipulaor o carry ou wo differen manipulaion asks using exclusively hapic daa. In boh scenarios a human user holding he end-effecor of a 6-DoF hapic inerface (Dela device from Force Dimension) eleoperaes a roboic arm (RX6 from Säubli) which has a force-orque sensor (Shunk FTC-5) placed on is wris. Force-based percepions are feedback o he eacher in order o esablish a bidirecional communicaion channel during he demonsraion sage. This implies o work a a minimum frequency of 1 Hz o have a high fideliy force reflecion and a sable eleoperaion sysem, which grealy depend on he execued processes beween he posiion sensing of he hapic device and when he sensed force is refleced on i. Our experimenal seup akes such requiremen ino accoun and guaranees a high bandwidh communicaion in he hapic loop. Two differen asks are proposed o es and analyze he performance of he proposed framework. Their paricular feaures are described below. 2.1 Ball-in-box Task In his ask he robo holds a plasic conainer wih a seel sphere inside i, as shown in Figure 2. A he demonsraion phase, he eacher repeaedly carries ou he ask o be learned, which consiss of aking he ball ou of he box hrough he hole, following a specific moion sraegy: Saring a some predefined iniial posiions, he ball is driven owards he wall adjacen o he hole, and hen forced o roll along his wall o he hole (see Figure 2). During he demonsraions, he eacher feels a he end-effecor of he hapic device he forceorque sensed a he roboic wris. Noe ha he eacher has an addiional informaion source by waching he scene direcly. No visual daa are provided o he robo. We like o emphasize ha his paricular manipulaion ask has been chosen because i is well-defined and simple enough o permi analyzing each sage of he proposed LfD framework separaely and in deph Filering Processes A firs experimenal finding derived from he use of hapic feedback in his bidirecional learning framework is he need for filering. Several people esed he experimenal seing, by eleoperaing he roboic Fig. 2 Experimenal scenario of he ball-in-box ask. (A he boom righ corner )Iniialposiionsofheballforheraining phase and moion sraegy followed by he eacher. Numbering is counerclockwise. arm hrough he hapic inerface while receiving forceorque feedback from he sensor mouned on he roboic wris. Iniially, hey eleoperaed he robo while feeling boh he conainer s mass and he ball s dynamics. Then, hey carried ou he same ask jus feeling he ball s dynamics. All he paricipans argued ha he presence of he conainer s mass was a very disracing facor making he ask more difficul o each. Thus, filering and dynamic compensaion are necessary o obain beer demonsraions and o improve he bidirecional communicaion channel, as explained below. Formally, he force-orque signals from he sensor can be expressed as: F /T s = F /T b + F /T m + ε (1) where F /T b corresponds o he ball dynamics, F /T m represens he conainer mass and ε is he noise (where we include he conainer vibraions). For achieving a clean and sable communicaion channel beween he human and he robo, i is necessary o display only relevan force-orque signals o he eacher, ha is, hose corresponding o he dynamics of he ball inside he conainer. Therefore force-orque produced by noise and he conainer s mass mus be removed before sending force informaion o he hapic conroller. Since he box is no a perfecly rigid srucure, i vibraes as he robo moves. These unwaned vibraions inroduce noise in he eleoperaion sysem, leading o unsable behavior. To avoid his, we implemened a low-pass digial filer ha cus ou all vibraion signals on he force-orque sensor in such a way ha he

4 4 ε effecs in Equaion 1 are grealy reduced, in a similar way as done in [23] for suppressing residual vibraions in flexible payloads carried by robo manipulaors. We compued he signals fundamenal frequency by subjecing he conainer wih he ball inside o vibraions hrough a force applied perpendicularly o he conainer s base, a he fron edge of i. Then, he frequency specrum of he generaed daa was analyzed, from which we obained he fundamenal frequency (7.5Hz) as he cuoff frequency of our low-pass filer. Using MATLAB R s FDAool, we designed he filer by implemening he Consrained Leas Squares echnique whose order was 75 [24] Dynamic Compensaion During he demonsraion sage, mos of he people acing as eachers declared ha having o compensae he conainer s mass while execuing he demonsraions makes i considerably harder o focus on he ask s goal. Therefore, we model he effecs of he conainer s mass dynamics on he force-orque signals wih he aim of removing hem and jus sending percepions conveying he ball s moion (similarly as in [25, 26]). Considering he sysem shown in Figure 3, le p denoe he posiion of he cener of graviy of he conainer, ω is angular velociy, m is mass, I is momen of ineria, F /T s and F /T e he sensor and exernal forces/orques respecively, and r s and r e he vecors from he cener of graviy of he conainer o he sensor and exernal forces frames. Then, using he Newon-Euler equaions, we obain: F = m p = mg + F e + F s T = I ω + ω Iω = T s + r s F s + T e + r e F e If we assume very low linear and angular acceleraions as well as very low angular velociies, we obain he following approximaion: F s = mg F e (2) T s + r s F s = T e r e F e (3) Deermining he force and orque values generaed by he conainer s mass via he sensor measuremens wihou he ball inside, and using he former equaions, i is possible o remove hese undesirable force-orque signals from sensor readings and o reurn jus he ball dynamics effecs. The users agree ha he fel forceorque values a he end-effecor afer compensaion are realisic enough as o provide a clear undersanding of how heir acions ranslae ino moions of he ball. Fig. 3 Dynamic compensaion model for removing conainer s mass effecs from he force-orque percepions sen o he hapic conroller. 2.2 Pouring ask The second ask consiss of pouring drinks. Here, he roboic arm holds a 1 lier plasic bole full of iny meallic spheres, which play he role of a fluid (his soluion was adoped o avoid spilling liquid during ess). The eacher eleoperaes he robo in order o demonsrae how o pour 1 ml drinks ino a plasic glass, as shown in Figure 4. Every sample of he ask sars from an unique predefined iniial pose of he bole, which is also he sop configuraion once he robo has poured a drink. Iniially, he bole is compleely full, and he eacher carries ou several demonsraions unil he bole is empy. Thus, he iniial force-orque values for each example vary according o how much fluid has been poured previously. I is worh o highligh ha such changes in he inpu variables a he beginning of he demonsraions are similar o hose observed in he ball-in-box ask for each iniial posiion of he sphere inside he conainer. Again, i was necessary o implemen a smoohing filer o reduce he noise from he sensor readings, his ime generaed by he iny meallic spheres. On he oher hand, he dynamic compensaion model presened previously was also used here for removing he bole mass effecs from he sensor readings, in order o feedback he eacher wih only he exernal forcesorques generaed by he fluid a he demonsraion phase. Noe ha in his ask, he eacher is also able o wach he scene direcly, hus he/she can know he locaion of he glass in he robo workspace. Such informaion is no provided o he robo during he execuion phase because he glass posiion is predefined in advance and fixed across he examples. 2 2 Noe ha a camera sysem may also be used o know he locaion of he glass in he robo frame, so ha he demonsraions would also be dependen on his parameer.

5 A Robo Learning from Demonsraion Framework o Perform Force-based Manipulaion Tasks 5 Fig. 4 Experimenal scenario of he pouring ask. The eacher demonsraes he robo how o pour 1 ml drinks ino a plasic glass by eleoperaion. Tamosiunaie e al. [27] ackled he same ask using reinforcemen learning, which was applied o improve he iniial encoding obained from human demonsraions modeled hrough dynamic moion primiives. Thus, his ask allows o show ha he proposed framework can be used for learning more realisic force-based skills like he ones ha an inexperienced human wish o each o a home service robo [28]. 3FeaureSelecionhroughMuual Informaion The wha o Imiae? problem means o deermine which informaion of he demonsraions is relevan for learning he ask a hand successfully [29]. Mos works ackle his problem by analyzing he variabiliy across demonsraions of he ask a rajecory level. Those pars wih large variances do no have o be learned precisely, whereas low variance suggess ha he corresponding moion segmen is significan and deserves o be learned [14, 3]. This approach explois variance for consrucing ask consrains [31] as well as for deermining secure ineracion zones in a robo coaching framework [32]. However, he cied works do no focus on he relaive relevance of each individual inpu dimension for he ask o be learned. Bu irrelevan or redundan informaion may acually be presen across inpu dimensions, which can increase he compuaional cos of he learning sage and make he ask harder o learn. The poin is o selec he mos relevan subse of inpu variables. The benefis in compuaional cos and noise reducion during he learning sage do ouperform a hypoheical and marginal loss of informaion. Furhermore, his approach is compaible wih he previously described variance-based analysis crierion. In lieraure, wo ypes of approaches ackle he problem of selecing a subse of variables from he original daa. Feaure selecion mehods keep only useful feaures and discard ohers, while feaure ransform echniques consruc new variables ou of he original ones [33]. In LfD, feaure selecion may be preferable o ransforms because i could be essenial o reain some of he original feaures. For insance, in acive learning, he robo may le he eacher know which percepions i has seleced, in order o ge feedback abou how well or how convenien is selecion was according o he human knowledge of he ask. Such human assisance will no be available if he robo carries ou he selecion from inpu ransforms. This fac may occur in [3], where he auhors propose o projec he human samples ono a laen space obained from a principal componen analysis o diminish redundancies, where he ransformed variables do no have a clear inerpreaion for a human eacher anymore. Also, his analysis is applied o he inpu variables of he problem, wihou aking ino accoun how hese influence he oupu variables. In addiion, when he number of irrelevan percepions exceeds he number of relevan inpus by orders of magniude, learning a ransform reliably may require excessive amouns of raining daa. Since our framework may be used as he basis of LfD srucures, i is more generic and suiable if i may provide clear informaion abou wha he robo considers ha should be imiaed. Thus, feaure selecion mehods are preferred in his conex. Here we use he Muual Informaion () crierion, which allows o esablish which inpu variables give more informaion wih respec o heir effecs on he oupus (i.e., how force-orque percepions affec he eacher acions). In conras o oher echniques (e.g., correlaion crierion), deecs non-linear dependencies beween inpus and oupus [34]. The purpose of his mehod in feaure selecion [35] is he reducion of he oupu daa uncerainy provided by each inpu variable. In our conex, depending on how he uncerainy of he oupu daa is reduced, a robo percepion gives more or less informaion abou he desired acions. Noe ha his approach has shown saisfacory resuls in sensor fusion [36] and vision-based posiioning of a roboic arm [37]. In order o apply analysis o our resuling raining daa (afer filering and dynamic compensaion), le us define he value beween wo coninuous variables x and y as follows (more deails in [38]): 3 I(x; y) = x y p(x, y) log p(x, y) p(x)p(y) (4) 3 The basic division and produc rules of log can be applied for numerical sabiliy.

6 6.7.5 ω 1 ω 2 ω ω 4 ω 5 ω ω 1 ω 2 ω ω 4 ω 5 ω ω 1 ω 2 ω ω 4 ω 5 ω Fx Fy Fz Ty Tz Fx Fy Fz Ty Tz Inpus Inpus (a) for all he inpu-oupu pairs..2.2 Fx Fy Fz Ty Tz Fx Fy Fz Ty Tz Inpus Inpus (b) Condiional given T x..2.2 Fx Fy Fz Tz Fx Fy Fz Tz Inpus Inpus (c) Condiional given T x and T y. Fig. 5 values a each variable selecion phase for he ball-in-box ask..6.4 q 1 q 2 q q 4 q 5 q q 1 q 2 q q 4 q 5 q q 1 q 2 q q 4 q 5 q Fx Fy Fz Ty Tz Fx Fy Fz Ty Tz Inpus Inpus (a) for all he inpu-oupu pairs..1.1 Fx Fy Fz Ty Tz Fx Fy Fz Ty Tz Inpus Inpus (b) Condiional given T x..1.1 Fx Fy Ty Tz Fx Fy Ty Tz Inpus Inpus (c) Condiional given T x and F z. Fig. 6 values a each variable selecion phase for he pouring ask. Ty (Nm) T x (Nm) Pos 1 Pos 2 Pos 3 Pos 4 Pos 5 Pos 6 Pos 7 Pos 8 Pos 9 Pos 1 Pos 11 Fig. 7 Torques map represening clusers for each iniial posiion of he ball inside he conainer. Ploing he firs samples of he variables mos relevan o he curren ask, T x vs. T y,iisobservedheydodescribewhereheballisinhe box. 3.1 Ball-in-box Task Le us define he inpus of his manipulaion ask as he wrench ϑ = {F x, F y, F z, T x, T y, T z }, i.e., he sensed forces and orques in he robo s frame, and he oupus as he join velociies of he robo defined by ω = {ω 1,...,ω Nq }, where N q is he number of joins of he robo. Using equaion 4, he value is compued for each inpu-oupu pair using enire daa sreams obained a he demonsraion phase. Boh he marginal and join probabiliies are approximaed using hisogram-based densiies, which are compued from discree pariions of he daaspace. 4 The quanizaion error in he conversion from coninuous variables o discree ones is bounded by some consan value which depends only on he number of pariions ha divide he coninuous space [39]. Figure 5(a) shows he differen values for all he inpu-oupu pairs in he ask. In general erms, he inpu variables F y and T z show less relevance whereas T x and T y are he mos correlaed variables wih he 4 Oher ype of non-parameric densiy may also be used, such as Parzen windows. oupus. This does make sense as hey are he variables ha give he mos useful informaion for knowing where he ball is inside he box (see Figure 7). These resuls confirm wha we inuiively expeced abou which inpu variables were he mos relevan for his ask. Noe ha his echnique can be also applied in more complex problems where he imporan percepions can no be easily deeced. Noneheless, once values have been compued, he problem is o selec a subse Ω of k percepions from he original se Ψ of n inpus, ha is maximally informaive abou he robo acions. In his conex, he compued values provide a ranking ha can be used for selecing he mos relevan inpu. However, o choose he remaining k 1 percepions, he redundancy among inpus mus be aken ino consideraion. To achieve his, we resor o a greedy selecion algorihm known as muual informaion-based feaure selecion deduced from uniform disribuions (FS-U) [39], which was adaped o our learning framework as described in Algorihm 1. The core of his echnique is o selec he res of variables by maximizing I(y; x i x s ), his means o choose he inpu x i ha provides mos informaion abou y given x s. In his approach, he compuaion of he condiional muual informaion is approximaed as follows: I(y; x i x s )=I(y; x i ) I(y; x s) H(x s ) I(x s; x i ), (5) where H(x s )represensheenropyofx s (deails in [39]). Noe ha he algorihm can be exended o a mulidimensional oupu case assuming a se of N m inpu variables X = {x 1,...,x Nm } and oupu daaspace Y = {y 1,...,y Nn } of dimensionaliy N n. For his ask, we se k = 3 and carried ou he FS-U o choose he percepions o be used in nex sages of he learning framework. The seleced variables were he subse Ω = {T x, T y, F x }, as shown in Figures 5(b) and 5(c). I should be noed ha he val-

7 A Robo Learning from Demonsraion Framework o Perform Force-based Manipulaion Tasks 7 Algorihm 1 FS-U 1: Iniializaion: Se Ω {},andψ X 2: Compue : Obain I(y j, x i ), x i X,and y j Y 3: Mean : I(Y, x i )= 1 Nn N n j=1 I(y j, x i), x i X 4: Selec he mos relevan inpu: Find he inpu x s =argmax xi X I(Y, x i), and se Ω {x s }, Ψ \{x s } 5: Greedy selecion: for =1 k 1 do 5.1: Compue he condiional I(Y, x i x s ), x i X 5.2: Find x s =argmax xi X I(Y, x i x s ), and se Ω {x s }, Ψ \{x s } end for 6: Oupu he se Ω ues for F x and F z are very similar for mos oupus iniially (see Figure 5(a)), however, FS-U auomaically chooses F x and discards F z. This is in accordance o inuiion, since F z is he force along he verical axis in he robo frame, which represens he graviaional force of he ball. Such force generaes he orques abou he axes x and y, hus here is a high correlaion beween F z and {T x,t y }. due o he fluid dynamics ino he bole. Noe ha such dynamics may be hardly modeled as repored in [27], bu he algorihm was able o deec ha T y was non-linearly correlaed o he robo moion encapsulaing par of he fluid dynamics (also confirmed afer a deailed analysis of he daa sreams). In sum, is shown o be a good and advisable ool for exracing he percepions ha are relevan in a LfD framework. An ineresing aspec o highligh is ha FS-U ofen gives more preference o redundan variables over irrelevan ones during he selecion process, as also noed in [4]. In his case he variables T x and F z provide nearly he same informaion abou he ask, bu boh are chosen even being redundan, because heir relevance wih respec o he oupus keeps high despie one of hem has been seleced previously. We consider ha such behavior is no a drawback in he LfD conex, because our aim is o exrac he relevan percepions of he ask (even if several of hem provide similar informaion). Noneheless, his fac opens an aracive issue of research o be ackled in near fuure works. 3.2 Pouring Task In his ask he inpu variables also are he wrench ϑ, bu he oupu variables are he join robo posiion defined by q = {q 1,...,q Nq } a insan + 1 for he given ϑ a. Such change of oupu variables for his ask is aimed a showing he generic significance of he proposed learning framework for differen represenaions of he ask sae. Again, he value was compued for all he inpu-oupu pairs (as shown in Figure 6(a)) in order o selec he mos imporan inpu variable, T x in his case. Noe ha T x and F z display nearly he same value for all he robo joins. This is an expeced resul because F z is he verical force in he robo frame represening he graviaional componen of he load (i.e., he bole and fluid masses), while T x is approximaely he orque generaed by such a load. This means ha boh variables are providing similar informaion wih respec o he robo movemens, because hey significanly vary as he fluid comes ou of he bole. Subsequenly, Algorihm 1 was applied o selec he remaining k 1 variables (wih k = 3), from which he resuling mos informaive se of inpus was Ω = {T x, F z, T y }. The selecion process and he values of he condiional muual informaion are shown in Figures 6(b) and 6(c). There is an ineresing aspec o highligh from his resul: he hird seleced variable T y shows sligh variaions when he robo roaes he bole o pour a drink, which are likely produced by he locaion change of he cener of mass of he load 3.3 Auomaic selecion of inpu variables As shown previously, he number of inpus o be seleced was predefined in advance, however i would be desirable o have a measure o decide on he opimal number of componens seleced by FS-U. In his direcion, le us define a new variable ζ ha compues he raio a ieraion beween he informaion a candidae inpu variable x c provides and he one already given by he curren subse of seleced percepions Ω as follows, ζ = I(Y, x c x s ) I(Y, Ω ), (6) where x c =argmax x I(Y, x i X i x s )andi(y, Ω )= j=1 I(Y, xj s x j 1 s ). Such condiional muual informaion raio shows how much informaion he nex inpu o be seleced provides aking ino consideraion he accumulaed condiional given by he curren seleced variables. In his sense, i is desired ha ζ is greaer han a predefined hreshold <φ 1, which conrols wha is he minimum informaion raio ha allows o selec one more inpu variable (i.e., he minimum muual informaion ha a variable should provide). I is worh menioning ha his new selecion crierion would modify sep 5 in Algorihm 1, where he greedy selecion is now conrolled by ζ, which is evaluaed a each ieraion before selecing he nex inpu. Thus, he algorihm keeps selecing variables while

8 8 he condiion ζ φ is saisfied. Noe ha he higher φ, he more selecive he algorihm. We again subjeced he raining daa of boh manipulaion asks o analysis, bu his ime using he modified version of Algorihm 1 wih φ =.3. Regarding he ball-in-box ask, he resuling subse of seleced inpus was again Ω = {T x, T y, F x }, supporing he analysis explained in Secion 3.1. In conras, for he pouring ask, he resuling seleced percepions were Ω = {T x, F z }. This ells us ha T y migh no provide enough informaion abou he robo acions when T x and F z have been already seleced. We will assess he framework performance using hese las subses of inpu variables in Secion 6. 4LearningheTask Previous research in LfD [41], has proposed o use Gaussian Mixure Models (GMM) for encoding manipulaion asks. However, his algorihm does no exrac emporal informaion from daa, and ime mus explicily be considered as an inpu variable if required by he ype of ask o be learned (as in Calinon e al. [3]). Force-orque signals end o show very large ime discrepancies, which may be ackled using echniques as dynamic ime warping, bu increasing he complexiy of he learning framework. To avoid including his explici emporal dependency in he model we resor o HMM, which reas and explois he sequenial paerns in he daa and i is herefore more appropriae o encode he feaures of our asks wihou using ime as an addiional inpu variable, which would significanly consrain he generalizaion capabiliies. HMM can be inerpreed as an exension of GMM in which he choice of he mixure componen for each observaion depends also on he choice of he componen for he previous observaion. This echnique has been widely used in several compuer science areas as speech recogniion [42], human moion paerns encoding [43] and LfD applicaions [32, 44], among ohers. Mos of LfD works using HMM address he problem by learning rajecories from human demonsraions [12] or by encoding a ask wih predefined saes as in assembly processes ha can be represened a a symbolic level [45]. However, our problem differs from hese and oher works in he following poins: The ask goal may be achieved by execuing differen rajecories depending on he iniial condiions of he ask (e.g., iniial posiion of he ball inside he conainer for he ball-in-box ask or he fluid quaniy inside he bole for he pouring skill). We do no use ime as an addiional inpu variable. Formally, given our experimenal seing described in Secion 2 and following he noaion of [46], le us denoe a raining daapoin as d m p RD, wih m = 1, 2,...,M and p =1, 2,...,P,whereM is he number of demonsraions, P is he number of daapoins colleced along demonsraion m, and D is he oal number of inpu and oupu variables. We used an N - saes ergodic HMM defined as λ =(A, B, π) where: A = {a ij } is he sae ransiion probabiliy marix, wih 1 i, j N. B = {b j (k)} is he observaion symbol probabiliy marix, wih 1 k (M P ) and assuming coninuous observaion densiies defined as normal disribuions N (O; µ j, Σ j ). π = {π i } is he iniial sae probabiliy vecor, wih 1 i N. N is he number of model saes. The main idea is o adjus he model o maximize P (O λ), where O = {O 1, O 2,..., O T } is an observaion sequence wih each O corresponding o a raining daapoin d m p.thebaum-welch mehod is used o achieve such an objecive (more deails in [46]). In order o describe he procedure for re-esimaion of HMM parameers, i is necessary o define he following variables: ξ (i, j) = γ (i) = N i=1 α (i)a ij b j (O +1 )β +1 (j) N j=1 α (i)a ij b j (O +1 )β +1 (j) (7) N ξ (i, j) (8) j=1 where α and β are called forward and backward variables, respecively, and are defined as: α 1 (i) =π i b i (O 1 ) [ N ] α +1 (j) = α (i)a ij b j (O +1 ) i=1 β T (i) =1 N β (i) = a ij b j (O +1 )β +1 (j) j=1 From equaions 7 and 8, he HMM parameers are ieraively esimaed as follows: π i = γ 1 (i) T 1 =1 a ij = ξ (i, j) T 1 =1 γ (i) T =1 µ jk = γ (j, k)o T =1 γ (j, k) T = =1 γ (j, k)(o µ jk )(O µ jk ) jk T =1 γ (j, k)

9 A Robo Learning from Demonsraion Framework o Perform Force-based Manipulaion Tasks T y ω ω 6 (a) 3-saes HMM rained wih posiions {1,2,3,4} ω T y (b) 3-saes HMM rained wih posiions {7,8,9,1} ω 6 Fig. 8 Resuling HMM for wo differen raining daases of he ball-in-box ask. Top: Inpu space composed of he mos relevan inpus {T x,t y }. Boom: Oupuspacecomposedofrobojoinvelociiesplayinghemosimporanroleforhe given ask. For boh cases, he hidden lef-o-righ srucure is obained afer convergence (having an ergodic HMM a he beginning). These equaions permi obaining a suiable rained HMM ha represens he eacher demonsraions saisically hrough a saes model capuring he robo moion for given force-based percepions and aking emporal coherence ino accoun from he resuling marix A. 4.1 Ball-in-box Task For encoding his ask, inpus are he force-orque sensed a he roboic wris and oupus are he velociy commands ω l a each robo join q l wih l =1,...,N q.noe ha join velociies were chosen as oupus because hey do represen he robo acions o be performed according o he force-orque percepions. As explained in Secion 3, we found he subse Ω of seleced inpus as hose needed o learn he ask successfully, because hey showed o conain he mos relevan informaion abou he ask oupus. Thus, each raining daapoin is defined as d m p = {T x,t y,f x,ω 1,...,ω Nq }. In oher words, λ is encoding he join disribuion P (Ω, ω). To undersand beer his idea, Figure 8 shows he HMM convergence for wo differen daases: Figure 8(a) displays a 3-saes HMM rained wih similar demonsraions saring from posiions {1,2,3,4}, while Figure 8(b) shows anoher 3-saes model rained wih samples saring from posiions {7,8,9,1}. Observe how he hidden lef-o-righ srucure is obained afer convergence (having an ergodic HMM a he beginning), which is he appropriae opology for learning hese daases separaely. For boh cases, he resuling vecor π gives as iniial sae he blue Gaussian, ha corresponds o he firs movemen carried ou by he eacher (i.e., when he user oriens he robo in such a way ha he ball rolls owards he wall adjacen o he hole). In Figure 8(a), blue and red saes inersec each oher in inpu space, covering he same segmens of rajecories. In his case, he emporal informaion is essenial o deermine wha velociy command has o be provided, which is no clear using a GMM-based approach. 4.2 Pouring Task In order o show he flexibiliy of he proposed framework, we use a differen represenaion of he ask sae for learning he pouring skill. Specifically, inpus are he seleced subse of variables Ω previously obained in Secion 3 and he curren join posiions q a ime sep. 5 Oupus are he desired robo sae o be achieved a + 1. Thus, each raining daapoin is defined as d m p = {T x, F z, q1,..., q N q, q1 +1,..., q +1 N q }.Then, in his case he model is encoding he join disribuion P (Ω, q, q +1 ). Figure 9 shows wo single HMM encoding differen ses of demonsraions of he pouring ask. The displayed projecion of he models corresponds o he mos relevan inpu T x and he robo join q 6 ha is roaed o pour he drinks. Noe ha here is one sae encapsulaing he sar and end of he skill a he same ime 5 I should be noed ha q was no considered in he based analysis because i is known ha q +1 is highly correlaed o is values a ime sep because he dynamics of he ask.

10 T x (a) 3-saes HMM encoding he 2 nd se of pouring samples q T x (b) 3-saes HMM encoding he 4 h se of pouring samples Fig. 9 Resuling HMMs for wo differen se of demonsraions of he pouring ask. (i.e., he green Gaussian), while he red ellipse is encoding when he fluid comes ou from he bole while he robo slighly roaes q 6. The complee model of his skill and he reproducion resuls are shown and analyzed laer on. 5TaskReproducion Since he asks are neiher sricly learned as a sequence of discree acions nor as simple rajecories, i is necessary o find a suiable way o reconsruc he oupu commands, given a percepion and he resuling rained HMM. To achieve his goal, a modified version of GMR (here named GMRa) is used for compuing he robo acions o be sen o he conroller as he desired robo sae o be achieved, as described nex. Recen works [11, 41] proposed o use GMM/GMR for learning asks a rajecory level, where he main idea is o model daa from a mixure of Gaussians and o compue predicions for a given se of queries hrough regression by applying he original version of GMR. In his approach, sandard GMR averages he differen observaions, even if hey have been observed a differen pars of he skill. Formally, for each Gaussian componen i, boh inpu x and oupu y are separaed by expressing he mean and covariance marix as: [ ] µ x µ i = i, Σ i = µ y i [ Σ xx i Σ yx i Σ xy i Σ yy i Then, he condiional expecaion ŷ given he inpu vecor x, for a mixure of N Gaussians is: ŷ = N [ β i µ y i=1 i + Σyx i ] (Σ xx i ) 1 (x µ x i ) ] (9) p(i)p(x i) where β i = N is a weigh exclusively based j=1 p(j)p(x j) on he inpu variables (mainly force-orque daa in our asks). We aim a predicing he necessary robo commands as a funcion of is force-based percepions in order o follow he augh sraegy for each ask as close as possible. We adop he approach proposed by Calinon e al. in [47], where he robo s acions are compued from a modified version of he well-known regression echnique GMR (which we name GMRa). This version compues he predicions from a mixure of Gaussians (e.g., he HMM saes) aking he encapsulaed emporal informaion by he HMM (i.e., he variable α) ino accoun along wih he given inpus (i.e., he robo percepions). In his way, our learning framework is able o handle percepual aliases, his means ha he robo may be able o carry ou he correc acion if more han one oupu exis for he same percepion paern, by aking advanage of he sequenial informaion of he ask. This makes he proposed srucure more generic and versaile, hus being useful for a wider se of manipulaion skills. In GMRa, he weighs are esimaed using he acual values of he inpus, and also implicily heir previous values, hrough he ransiion probabiliies relaed o he forward variable α. Formally, he definiion of he new GMRa based on emporal informaion is given by: ŷ = N i=1 α(i) [ µ y i + Σyx i (Σ xx i ) 1 (x µ x i )] (1) where α(i) is he forward variable for he i-h Gaussian in he HMM. This variable expresses he probabiliy of observing he parial sequence, O = {O 1,O 2,...,O } and of being in sae S i a ime. Now, for a given forceorque percepion, he prediced command is based on curren and pas observaions, which makes sense for hose asks where more han one oupu exiss for a given inpu paern. Noe ha Lee and O s work [32] proposes a similar framework ha encodes he demonsraions hrough

11 A Robo Learning from Demonsraion Framework o Perform Force-based Manipulaion Tasks q5 ω Ty ω Robo execuion Teacher demonsraion Fig. 1 Top:InpuspaernforT x and T y when he ball sars a posiion number 3. Middle: Trajecories of joins q 5 and q 6 corresponding o boh eacher demonsraion and robo s execuion obained from he velociy profiles ω 5 and ω 6 shown a boom. an HMM and rerieves he robo acions using a imedriven version of he classical Gaussian Mixure Regression (GMR). In conras o he forward variable-based weighs, he weighing mechanism used by GMR exclusively depends on ime, and neiher previous observaions nor sequenial informaion are aken ino accoun in his approach. Such approach migh show an unsaisfacory performance when force daa presen large ime discrepancies, because he explici use of ime a he reproducion phase. 5.1 Ball-in-box Task Figure 13 shows he robo join rajecories and velociies obained while he eacher demonsraes how o ake he ball ou of he box, when saring a posiion 7. The rajecories and velociy profiles of he robo in he execuion phase are also displayed. These predicions have been compued via GMRa for he inpus displayed in he firs row of he figure and using he HMM displayed in Figure 8(b). I can be observed ha he learning framework is able o compue he correc velociy commands o follow he eacher s sraegy as well as o accomplish he ask s goal. In addiion, every join rajecory is very similar o he desired one, even for hose robo joins ha are no playing a relevan role in he ask (e.g., q 1 or q 3 ). The predicions in Figure 1 were obained from he HMM shown in Figure 8(a). The mos relevan feaure of his example is how he learning framework performs successfully even when he inpu daa lie simulaneously on wo HMM saes. Figure 8(a) shows wo overlapping saes, where GMR may likely rerieve a wrong velociy command if a given inpu daapoin lies in his zone. Insead, GMRa performs correcly as i akes no only he given percepion ino accoun, bu also he sequence of saes hrough α (a) Reproducion of he pouring ask using he model rained wih he 2 nd se of samples (b) Reproducion of he pouring ask using he model rained wih he 4 h se of samples Fig. 11 Lef :Torquepaernaroundheaxisx during he reproducion. Middle: Trajecoryofherobojoinq 6 ha roaes he bole o pour he drinks. Righ: T x vs. q 6 plo showing a reproducion paern quie similar o he ones observed in he demonsraions se. 5.2 Pouring Task Figures 11(a) and 11(b) show he obained reproducions using he models previously displayed in Figures 9(a) and 9(b), respecively. For boh execuions, he iniial force-orque percepion was slighly differen from he ones sensed during he demonsraion phase, for which he robo performed successfully, as evidenced from he rajecories followed by q 6.Noehasuchrajecories show ha he robo comes back o he saring configuraion afer having poured a drink, as expeced. These resuls provided a good saring poin o evaluae he encoding and reproducion capabiliies of he proposed framework in more real force-based asks. In Secion 6 resuls of he complee model encoding all he provided demonsraions of he pouring skill are analyzed. The Malab source codes of he proposed learning framework will be made publicly available a he ime of publicaion. 6Experimenaionwihhecompleesysems In his secion we show how FS-based inpus selecion influence he robo performance, and also we analyze he encoding and reproducion resuls of he proposed framework for boh manipulaion asks previously described in Secion Assessing he muual informaion crierion In order o assess he ineres of using FS wihin our learning framework, we evaluae he robo performance in erms of he roo mean squared error (RMSE) of he join rajecories. The objecive is o observe how

12 12 RMSE Complee se of inpus FS Mos relevan inpu RMSE Complee se of inpus FS Mos relevan inpu q i (a) Ball-in-box ask q i (b) Pouring ask Fig. 12 Roo mean squared error of robo reproducions for a given se of query vecors.differen ses of inpu variables are used in order o es he robo performance for hree differen cases:(a) when he original inpu se is used (blue bar), (b) applying FS (green bar) and (c) using only he mos relevan inpu (red bar). he robo performance varies for hree differen cases, namely: (a) when all he inpu variables are used, (b) when only he percepions seleced by he modified FS-U compose he inpu space (see Secion 3), and (c) when he mos relevan inpu is solely used. To achieve his, hree differen HMMs were rained using he aforemenioned daases, and a query vecor for every iniial ball posiion was exraced from he demonsraions. The mean RMSE for each robo join was compued across all he iniial posiions. Ball-in-box ask Figure 12(a) shows he RMSE values (given in degrees) for he hree differen cases. On he one hand, noe ha he RMSE across all he robo joins shows nearly he same values for cases (a) and (b), which proves ha he FS-based dimensionaliy reducion does no affec he robo performance because he unseleced inpu variables do no influence he robo behavior (e.g., F y and T z are weakly correlaed o he robo acions). The RMSE even slighly decreases in case (b) for some joins, which migh mean ha he removed percepions were inroducing noise (or a leas no useful informaion) ino he sysem, making a bi harder o reproduce he ask saisfacorily. On he oher hand, looking a he RMSE of he robo joins q 5 and q 6 (hose playing he mos relevan role o carry ou his ask), i is observed ha in case (c), i.e., when he learning framework uses exclusively he mos relevan percepion T x,heroboperformance deerioraes, which migh indicae ha he robo does no have enough informaion o perform successfully. Here, i should be menioned ha he robo is no able o carry ou he ask when perceiving only T x,because his variable does no describe enirely he locaion of he ball inside he box (see Figure 7). Pouring ask The same se of experimens carried ou o assess robo performance for he ball-in-box ask was also carried ou for his ask. Again, hree HMMs were rained and a query vecor of every pouring demonsraion (four in his ask, as described in Secion 2.2) was used o compue he RMSE of he resuling robo join rajecories. The hree same cases (a), (b) and (c) above were used o analyze how FS-U may influence he robo performance in his ask. Figure 12(b) shows he mean RMSE obained for all he robo joins across he four reproducions. Again, i is observed ha he robo performance is almos he same for cases (a) and (b), hus FS-U does no deeriorae robo execuion while i reduces daa dimensionaliy and saves compuaional resources. For case (c), RMSE values are a bi greaer han he ones observed for cases (a) and (b), however, he robo was also able o carry ou he ask successfully. This may be explained by he fac ha F z nearly provides he same informaion given by T x.theyprovided redundan informaion abou he ask as shown in Figure 6, bu boh of hem are relevan in he sense of heir correlaion wih he robo oupu commands (see Secion 3.2). 6.2 Encoding and reproducion resuls Compuaional and experimenal resuls of he wo manipulaion ask are explained and analyzed in he nex paragraphs, where he models were rained using he inpu daaspace reduced hrough FS. Ball-in-box Task To evaluae he performance of he proposed learning framework in his scenario, he eacher carried ou four demonsraions for en differen iniial ball posiions

13 A Robo Learning from Demonsraion Framework o Perform Force-based Manipulaion Tasks Ty q q q ω1 ω2 ω3 ω q q5 ω ω Fig. 13 Top:ThefirsrowshowshepaernofinpusT x and T y when he ball sars a posiion number 7, and he remaining rows display he robo join rajecories and velociy profiles when he eacher demonsraed he ask (solid blue line) and when he robo execued is moions based on predicions given by GMRa (dashed red line). Boom: Lef image shows a snapsho of he beginning of he learned ask. The cener image displays he momen where he robo has compleed he firs sage of he sraegy and sars o orien he box for aking he ball owards he hole. The righ image shows he successful compleion of he ask. placed along he box edges. Every demonsraion was execued by eleoperaing he roboic arm hrough he 6-DoF hapic device (as shown in Figure 2) and following he moion sraegy explained in Secion 2. The resuling raining daase consised of all daapoins d m p, which were used o rain several HMMs by applying he Baum-Welch mehod unil convergence. To find he bes model, we resor o he Bayesian Informaion Crierion (BIC), which allows o find a rade-off beween opimizing he model s fiing and he number of saes [48]. Noe ha he seleced HMM will be a model ha can fi he daa well, wih no overfiing in BIC sense. Figure 14 displays he differen BIC values for he se of models esed, and Figure 15 shows he seleced 5-saes HMM. The execuion and generalizaion capabiliies were esed for some of hese models using query daa exraced from he demonsraions and real experimens. The 2-saes HMM showed he wors performance, his model was no able o carry ou he ask saring a any place, even if i did i from a pre-rained iniial posiion. The HMMs wih 4, 8 and 9 saes could achieve he goal from every pre-rained posiions bu someimes failed saring a non-rained iniial configuraions, showing poor generalizaion capabiliies. Finally, he models wih 5, 6 and 7 saes showed very similar performances wih no clear differences, and all of hem performed he ask successfully. Observing he seleced 5-saes HMM, i is ineresing o highligh how he proposed framework is able o learn a muliple soluion ask by aking advanage

14 14 BIC 6.8 x Number of saes Fig. 14 BIC values for models wih differen number of saes. of he HMM properies. The model is shown in Figure 15, where he blue sae in he inpu space covers he beginning of all demonsraions whose iniial posiions are placed on he wall opposie o where he hole is. A hese saring posiions, a larger velociy command is required o draw he ball ou of is resing configuraion by moving he robo join q 6 (Figure 15, oupu space projecion). Afer, he green and ligh-blue saes represen he movemens o force he ball o role o he hole, hrough q 5 and depending on wheher he ball is up or down wih respec o he hole (i.e., posiive or negaive velociy commands, respecively). The yellow Gaussian can be considered as an inermediae sae he sysem goes hrough o reach he final sae (red ellipse) a which he velociy commands are zero (i.e., when he ball is geing ou of he box) in inpu space. As for he predicion phase, one eacher s demonsraion for each iniial posiion was removed from he raining examples and used as query daa for evaluaing he learning framework performance by comparing is resuls wih he eacher execuions. All robo join rajecories obained from velociy commands synhesized by our HMM/GMRa approach are smooher han he eacher s demonsraions (as shown for iniial posiions 3 and 7 in Figures 1 and 13, respecively). By observing he obained velociy profiles for each robo join, one sees ha hey are also smooher han eacher ones, because human user execuions show several abrup changes, which are no over-fied by our learning framework. This can be aribued o he fac of using GMRa o rerieve he velociy command, because his ype of regression akes he covariance informaion ino accoun for compuing he esimaion of he oupu, ouperforming echniques ha only use he mean of he Gaussians. Thus, we can conclude in his conex ha he robo performs beer han he eacher. In addiion, all synhesized rajecories follow he same moion paern as ha of he eacher s execuions, which indicaes ha he sraegy applied by he human user was learned successfully. Once compuaional resuls were saisfacory, we validaed our framework on he experimenal seup. Firs, he robo had o perform he ask wih he ball saring a he already rained iniial posiions (see Figure 2). For all experimens, he robo was able o carry ou he ask effecively. Afer his, a second se of ess was execued, where he ball was locaed a random posiions inside he conainer. For hese ess, he robo was also able o achieve he ask s goal, execuing he moions learned for he closes iniial posiion, by idenifying he corresponding HMM sae. I was observed ha in some execuions he ball reached and surpassed he hole, wihou falling hrough i. This behavior may be jusified by he fac ha we are assuming a quasisaic case in our ask. 6 However, he robo was always able o ake he ball ou of he box afer some more execuions, as i correcly idenified he HMM sae corresponding o he curren and pas inpu paerns (aking ino accoun he emporal informaion). This means ha he robo generaes is acions as a funcion of is curren and pas percepions, following he augh moion sraegy. If he robo fails o reach he goal, he ball goes o anoher posiion inside he box, providing new percepions from which he robo can compue new movemens. Videos showing execuions of learned rajecories are available online a hp://dl.dropbox.com/u/ /jisr/jisr.hml. As he robo was able o accomplish he desired goal in every es, even when he ball reached and surpassed he hole, we evaluaed he performance of he robo execuions using a ime-based crierion [49]. Here, he idea is o deermine how much ime he robo akes o complee he ask successfully by execuing he commands obained from he proposed framework compared wih he hree following cases: (i) he robo execues hand-coded acions according o pre-programmed if-hen rules, (ii) he eacher carries ou he ask by eleoperaion following he menioned sraegy, (iii) he robo performs random movemens ha may ake he ball owards he hole. Figure 16 shows execuion imes for he aforemenioned cases. As expeced, he eacher s execuions show o he lowes imes, excep for posiion number 2 where he robo was faser han he human. A relevan aspec o discuss is he fac ha he robo execuion imes are much larger han he eacher s ones for posiions 3 o 8. Regarding posiions 3 o 5, higher imes are due o he fac ha he robo sars he ask by moving he join q 6 as expeced, however i also 6 On he one hand, he model variables are force-orque and join velociies a he given ime sep, hus no informaion abou he pas is explicily provided. On he oher hand, he robo conroller only allows posiion-based conrol, hus i is no possible o send he desired velociy commands direcly.

15 A Robo Learning from Demonsraion Framework o Perform Force-based Manipulaion Tasks 15 ω5.5.5 Projecion of he 5-saes HMM T y ω 6 Represenaion of he HMM ransiions Fig. 15 Resuling 5-saes HMM rained wih demonsraions saring a every posiion inside he box. Top: Inpu space composed of he mos relevan inpus {T x T y }. Middle: Oupuspacecomposedofrobojoinvelociiesplaying he mos imporan role for he given ask. Boom: Represenaion of he resuling ransiion probabiliies marix. As expeced, he mos likely ransiions from he blue sae ake he robo o he ligh blue or green ellipses. Moreover, he ransiions from hese Gaussians and he yellow one ake he sysem owards he final sae. moves q 5 slighly which someimes causes he ball o go o he boom of he box, jusifying higher sandard deviaions for posiions 3 and 4. This is a normal effec because he firs sae of he learned HMM covers non-zero angular velociies for he variable ω 5. Thus, in hese cases, he robo idenifies he new sae where he ball is and changes is moion sraegy according o he given inpu daa for reaching he arge. In he case of posiions 6 o 8, he robo does also move he join q 5, however i is because he eacher demonsraions showed ha he human ries o guaranee a sable moion by aking he ball owards he wall adjacen o he hole along he wall a he boom of he box. This causes ha, when he ball reaches he wall adjacen o he hole, he robo has o carry ou more movemens in order o ake he mealic sphere owards he hole, since he robo mus compensae he iniial inclinaion of he box given by he wrong moion of q 5. Thus, he high robo execuion imes are mainly a consequence of wo facors: firs, here is a delay beween he sensing and execuion phases ha increases he ime measures as he ball is farher from he arge, and second, he join velociy profiles of he robo execuion show lesser magniudes han he eacher ones (as observed in Figures 13 and 1), implying ha when hese velociy commands are ranslaed ino desired posiional configuraions of he robo, he joins roaion is lower and more velociy commands are needed o orien he box. Regarding he imes shown for he hand-coded acions, several robo learned execuions ouperformed he hand-coded ones (e.g., saring a posiions 1, 2, 4, 7, 9 and 1). This mainly happened because he handcoded acions also suffered he surpassing effec, ha is, he ball did no go ou hrough he hole a he firs aemp. Moreover, i is imporan o emphasize ha he if-hen rules programming was edious and imeconsuming, even for his simple ask. On he one hand, i was essenial o deermine how he inpu space could be ransformed o discree regions o se he if condiions. On he oher hand, a uning process was needed o specify he velociy commands ha he robo execued. One may hink ha he higher he velociy, he less ime he robo migh ake o accomplish he ask, however he surpassing effec may occur more ofen, increasing he ime execuion significanly. Thus, he learning-based approach is preferred because being similarly efficien, i is friendlier and can be applied by non-exper users. Finally, execuion imes for a random sraegy show ha rying o accomplish he goal by chance is possible, neverheless his implies much higher imes and variances in comparison wih when he robo carried ou he ask by using he augh sraegy. These high values occur because he random sraegy does no impose movemen consrains o he robo and, herefore, a huge se of available moions can be execued, leading o very varied and long rials. This consiues a reference (lower bound) for comparison purposes, agains which he improvemen aained by differen learning echniques and eaching sraegies can be evaluaed. Pouring Task In order o each he robo o pour drinks, hree complee execuions of he ask are provided o he robo by eleoperaion as described in Secion 2. Such execuions consis of saring wih he bole full of fluid and pouring four 1ml drinks. Noe ha afer each drink is poured, he iniial force-orque value changes for he nex demonsraion, which condiions he robo movemens as shown in Figure 17 where he gray lines