arxiv: v1 [cs.ro] 13 Oct 2016

Transcription

1 Sim-to-Rel Robot Lerning from Pixels with Progressive Nets Andrei A. Rusu, Mtej Vecerik, Thoms Rothörl, Nicols Heess, Rzvn Pscnu, Ri Hdsell rxiv: v1 [cs.ro] 13 Oct 216 Google DeepMind London, UK {ndreirusu, mtejvecerik, tcr, heess, rzp, Abstrct Applying end-to-end lerning to solve complex, interctive, pixel-driven control tsks on robot is n unsolved problem. Deep Reinforcement Lerning lgorithms re too slow to chieve performnce on rel robot, but their potentil hs been demonstrted in simulted environments. We propose using progressive networks to bridge the relity gp nd trnsfer lerned policies from simultion to the rel world. The progressive net pproch is generl frmework tht enbles reuse of everything from low-level visul fetures to high-level policies for trnsfer to new tsks, enbling compositionl, yet simple, pproch to building complex skills. We present n erly demonstrtion of this pproch with number of experiments in the domin of robot mnipultion tht focus on bridging the relity gp. Unlike other proposed pproches, our rel-world experiments demonstrte successful tsk lerning from rw visul input on fully ctuted robot mnipultor. Moreover, rther thn relying on model-bsed trjectory optimistion, the tsk lerning is ccomplished using only deep reinforcement lerning nd sprse rewrds. 1 Introduction Deep Reinforcement Lerning offers new promise for chieving humn-level control in robotics domins, especilly for pixel-to-ction scenrios where stte estimtion is from high dimensionl sensors nd environment interction nd feedbck re criticl. With deep RL, new set of lgorithms hs emerged tht cn ttin sophisticted, precise control on chllenging tsks, but these ccomplishments hve been demonstrted primrily in simultion, rther thn on ctul robot pltforms. While recent dvnces in simultion-driven deep RL re impressive [8, 19, 6, 12, 2, 1, ], demonstrting lerning cpbilities on rel robots remins the br by which we must mesure the prcticl pplicbility of these methods. However, this poses significnt chllenge, given the long, dthungry trining prdigm of pixel-bsed deep RL methods nd the reltive frilty of reserch robots nd their humn hndlers. One solution is to use trnsfer lerning methods to bridge the relity gp tht seprtes simultion from rel world domins. In this pper, we use progressive networks, deep lerning rchitecture tht hs recently been proposed for trnsfer lerning, to demonstrte such n pproch, thus providing proof-of-concept pthwy by which deep RL cn be used to effect fst policy lerning on rel robot. Progressive nets hve been shown to produce positive trnsfer between disprte tsks such s Atri gmes by utilizing lterl connections to previously lernt models [18]. The ddition of new cpcity for ech new tsk llows specilized input fetures to be lerned, n importnt dvntge for deep RL lgorithms which re improved by shrply-tuned perceptul fetures. An dvntge of progressive

2 nets compred with other methods for trnsfer lerning or domin dpttion is tht multiple tsks my be lerned sequentilly, without needing to specify source nd trget tsks. This pper presents n pproch for trnsfer from simultion to the rel robot tht is proven using rel-world, sprse-rewrd tsks. The tsks re lerned using end-to-end deep RL, with RGB inputs nd joint velocity output ctions. First, n ctor-critic network is trined in simultion using multiple synchronous workers [1]. The network hs convolutionl encoder followed by n LSTM. From the LSTM stte, using liner lyer, we compute set of discrete ction outputs tht control the different degrees of freedom of the simulted robot s well s the vlue function. After trining, new network is initilized with lterl, nonliner connections to ech convolutionl nd recurrent lyer of the simultion-trined network. The new network is trined on similr tsk on the rel robot. Our initil findings show tht the inductive bis imprted by the fetures nd encoded policy of the simultion net is enough to give drmtic lerning speed-up on the rel robot. 2 Trnsfer Lerning from Simultion to Rel Our pproch relies on the progressive nets rchitecture, which enbles trnsfer lerning through lterl connections which connect ech lyer of previously lernt network columns to ech new column, thus supporting rich compositionlity of fetures. We first summrize progressive nets nd then discuss its ppliction for trnsfer in robot domins. 2.1 Progressive Networks Progressive networks re idel for simultion-to-rel trnsfer of policies in robot control domins, for multiple resons. First, fetures lernt for one tsk my be trnsferred to mny new tsks without destruction from fine-tuning. Second, the columns my be heterogeneous, which my be importnt for solving different tsks, including different input modlities, or simply to improve lerning speed when trnsferring to the rel robot. Third, progressive nets dd new cpcity, including new input connections, when trnsferring to new tsks. This is dvntgeous for bridging the relity gp, to ccommodte dissimilr inputs between simultion nd rel sensors. A progressive network strts with single column: deep neurl network hving L lyers with hidden ctivtions h i R ni, with n i the number of units t lyer i L, nd prmeters Θ trined to convergence. When switching to second tsk, the prmeters Θ re frozen nd new column with prmeters Θ is instntited (with rndom initiliztion), where lyer h i receives input from both h i 1 nd h i 1 vi lterl connections. Progressive networks cn be generlized in strightforwrd mnner to hve rbitrry network width per column/lyer, to ccommodte vrying degrees of tsk difficulty, or to compile lterl connections from multiple, independent networks in n ensemble setting. h (k) i = f W (k) i h (k) i 1 + U (k:j) i h (j) i 1, j<k where W (k) i R ni ni 1 is the weight mtrix of lyer i of column k, U (k:j) i R ni nj re the lterl connections from lyer i 1 of column j, to lyer i of column k nd h is the network input. f is n element-wise non-linerity: we use f(x) = mx(, x) for ll intermedite lyers. In the stndrd pretrin-nd-finetune prdigm, there is often n implicit ssumption of overlp between the tsks. Finetuning is efficient in this setting, s prmeters need only be djusted slightly to the trget domin, nd often only the top lyer is retrined. In contrst, we mke no ssumptions bout the reltionship between tsks, which my in prctice be orthogonl or even dversril. Progressive networks side-step this issue by llocting new column, potentilly with different structure or inputs, for ech new tsk. Columns in progressive networks re free to reuse, modify or ignore previously lerned fetures vi the lterl connections. Appliction to Reinforcement Lerning. Although progressive networks re widely pplicble, this pper focuses on their ppliction to deep reinforcement lerning. In this cse, ech column is trined to solve prticulr Mrkov Decision Process (MDP): the k-th column thus defines policy 2

3 π (k) ( s) tking s input stte s given by the environment, nd generting probbilities over ctions π (k) ( s) := h (k) L (s). At ech time-step, n ction is smpled from this distribution nd tken in the environment, yielding the subsequent stte. This policy implicitly defines sttionry distribution ρ π (k)(s, ) over sttes nd ctions. 2.2 Approch The proposed pproch for trnsfer from simulted to rel robot domins is bsed on progressive network with some specific chnges. First, the columns of progressive net do not need to hve identicl cpcity or structure, nd this cn be n dvntge in sim-to-rel situtions. Thus, the simultion-trined column is designed to hve sufficient cpcity nd depth to lern the tsk from scrtch, but the robot-trined columns hve miniml cpcity, to encourge fst lerning nd limit totl prmeter growth. Secondly, the lyer-wise dpters proposed for progressive nets re unnecessry for the output lyers of complementry sequences of tsks, so they re not used. Third, the output lyer of the robot-trined column is initilised from the simultion-trined column in order to improve explortion. These rchitecturl fetures re shown in Fig. 1. simultion relity output 1 output 2 output 3 output 1 output 2 output 3 (3) (3) (3) (3) input input Figure 1: Depiction of progressive network, left, nd modified progressive rchitecture used for robot trnsfer lerning, right. The first column is trined on Tsk 1, in simultion, the second column is trined on Tsk 1 on the robot, nd the third column is trined on Tsk 2 on the robot. Columns my differ in cpcity, nd the dpter functions (mrked ) re not used for the output lyers of this non-dversril sequence of tsks. The gretest risk in this pproch to trnsfer lerning is tht the rewrds will be so sprse or nonexistent in the rel domin tht the reinforcement lerning will never tke off. Thus, in order to mximise the likelihood of rewrd during explortion in the rel domin, the new column is initilised such tht the initil policy of the gent will be identicl to the previous column. This is ccomplished by initilising the weights coming from the lst lyer of the previous column to the output lyer of the new column with the output weights of the previous column, nd the connections incoming from the lst hidden lyer of the current column re initilised with zero-vlued weights. Thus, using the exmple network in Fig. 1 (right), when prmeters Θ re instntited, lyer output 2 will hve input connections from h 2 nd h 2. However, unlike the other prmeters in Θ, which will be rndomly initilised, the weights W o will be zeros nd the weights U o (1:2) will be copied from. Note tht this only ffects the initil policy of the gent nd does not prevent the new column W o from trining. 3 Relted Literture There exist mny different prdigms for domin trnsfer nd mny pproches designed specificlly for deep neurl models, but substntilly fewer pproches for trnsfer from simultion to relity for robot domins. Even more rre re methods tht cn be used for trnsfer in interctive, rich sensor domins using end-to-end (pixel-to-ction) lerning. A growing body of work hs been investigting the bility of deep networks to trnsfer between domins. Some reserch [16, 21] considers simply ugmenting the trget domin dt with dt from the source domin where n lignment exists. Building on this work, [13] strts from the observtion tht s one looks t higher lyers in the model, the trnsferbility of the fetures decreses quickly. To correct this effect, soft constrint is dded tht enforces the distribution of the fetures to be 3

4 more similr. In [13], confusion loss is proposed which forces the model to ignore vritions in the dt tht seprte the two domins [24, 2]. Bsed on [24], [23] ttempts to ddress the simultion to relity gp by using ligned dt. The work is focused on pose estimtion of the robotic rm, where trining hppens on triple loss tht looks t ligned simultion to rel dt, including the domin confusion loss. The pper does not show the efficiency of the method on lerning novel complex policies. Severl recent works from the supervised lerning literture, e.g. [4, 1, 3], demonstrte how ides from the dversril trining of neurl networks cn be used to reduce the sensitivity of trined network to inter-domin vritions, without requiring ligned trining dt. Intuitively these pproches trin representtion tht mkes it hrd to distinguish between dt points drwn from the different domins. These ides hve, however, not yet been tested in the context of control. Demonstrting the difficulty of the problem, [21] provides evidence tht simple ppliction of model trined on synthetic dt on the rel robot fils. The pper lso shows tht the min filure point is the discrepncy in visul cues between simultion nd relity. Prtil success on trnsferring from simultion to rel robot hs been reported [2, 7, 26]. They focus primrily on the problem of trnsfer from more restricted simpler version of tsk to the full, more difficult version. While trnsfer from simultion to relity remins difficult, progress hs been mde with directly lerning neurl network control policies on rel robot, both from low-dimensionl representtions of the stte nd from visul input (e.g. [9],[11]). While the results re impressive, to chieve sufficient dt efficiency these works currently rely on reltively restrictive tsk setups, specilized visul rchitectures, nd crefully designed trining regimes. Alterntive pproches embrce big dt ides for robotics ([17, 1]). 4 Experiments For trining in simultion, we use the Async Advntge Actor-Critic (A3C) frmework introduced in [1]. Compred to DQN [14], the model simultneously lerns policy nd vlue function for predicting expected future rewrds, nd cn be trined with CPUs, using multiple threds. A3C hs been shown to converge fster thn DQN, which mkes it dvntgeous for reserch experimenttion. For the mnipultion domin of the Jco rm, the gent policy controls nine degrees of freedom using velocity commnds. This includes six joints on the rm plus three ctuted fingers. The full policy Π(A s, θ) comprises nine joint policies lernt by the gent, ech one softmx connected to the inputs from the previous lyer nd ny lterl connections. Ech joint policy i hs three ctions ( fixed positive velocity, fixed negtive velocity, nd zero velocity): π i ( i s; θ i ). This discrete ction set, while potentilly lcking the precision of continuous control policy, hs worked well in prctice. There is lso single vlue function tht is linerly connected to the previous lyer nd lterl lyers: V (s, θ v ). We evlute both feedforwrd nd recurrent neurl networks. Both hve convolutionl input lyers followed by either fully connected lyer or n LSTM. A stndrd-sized network is used for the simultion-trined column nd reduced-cpcity network is used for the robot-trined columns, chosen becuse we found empiriclly tht more cpcity does not ccelerte lerning (see Section4.2), presumbly becuse of the fetures reused from the previous column. Detils of the rchitecture re given in Figure 2 nd Tble 1. In ll vrints, the input is 3x64x64 pixels nd the output is 28 (9 discrete joint policies plus one vlue function). The MuJoCo physics simultor [22] is used to trin the first column for our experiments, with rendered cmer view to provide observtions. In the rel domin, similrly positioned RGB cmer provides the input. While the modeled Jco nd its dynmics re quite ccurte, the visul discrepncies re obvious, s shown in Figure 3. The experiments re ll focused round the tsk of reching to visul trget, with only pure rewrd (no shping) provided s feedbck. Though simple, this tsk requires tht the stte of the rm, nd the position of the trget, be correctly inferred from the visul observtion, nd tht the gent lern robust control over high-dimensionl stte spce. The rm is set to rndom strt position t the beginning of every episode, nd the trget is plced t rndom position in 4cm by 3cm re. The gent receives rewrd of +1 if its plm is within 1cm of the trget, nd episodes lst for t most steps. Though there is some vrince due to the rndomized strt stte, well-performing gent cn rech 4

5 LSTM fc conv 2 conv v... h 4 h v... h 4 h 3 feedforwrd recurrent wide nrrow wide nrrow fc (output) LSTM fc conv conv prms 621K 39K 299K 37K x t Figure 2: Detiled schemtic of progressive recurrent network rchitecture. The ctivtions of the LSTM re connected s inputs to the progressive column. The fctored policy nd single vlue function re shown. Tble 1: Network sizes for wide columns (simultion-trined) nd nrrow columns (robottrined). For ll networks, the first convolutionl lyer uses 8x8, stride 4 kernels nd the second uses x, stride 2 kernels. The totl prmeters include the lterl connections. Figure 3: Smple imges from the rel cmer input imge nd the MuJoCo-rendered imge. Though more relistic model ppernce could hve been used, the blocky Jco model ws used to ccelerte MuJoCo rendering, which ws done on CPUs. The imges show the diversity of Jco strt positions nd trget positions. n verge score of t lest 3 points. The episode is terminted if the gent cuses sfety violtion through self-intersection or by touching the tble top or by exceeding set joint limits. 4.1 Trining in simultion The first column is trined in simultion using A3C, s previously mentioned, using wide feedforwrd or recurrent network. Intuitively, it mkes sense to use lrger cpcity network for trining in simultion, to rech mximum performnce. We verified this intuition by compring wide nd nrrow network rchitectures, nd found tht the nrrow network hd slower lerning nd worse performnce (see Figure 4). We lso see tht the LSTM model out-performs the feedforwrd model by n verge Rewrds Simultion-trined first column Wide network Nrrow network Steps 1e7 Rewrds Simultion-trined first column (LSTM) Nrrow network Wide network Steps 1e7 Figure 4: Lerning curves re shown for wide nd nrrow versions of the feedforwrd (left) nd recurrent (right) models, which re trined with the MuJoCo simultor. The plots show men nd vrince over trining runs with different seeds nd hyperprmeters. Stble performnce is reched fter pproximtely million steps, which is more thn one million episodes. While both the feedforwrd nd the recurrent models lern the tsk, the recurrent network reches higher finl men score.

6 Rel-robot-trined progressive nets vs. bselines 3 2 Rewrds 1 Wide column (progressive) Nrrow column (progressive) Wide column (finetuned) Nrrow column (from scrtch) Wide column (from scrtch) Steps Figure : Rel robot trining: We compre progressive, finetuning, nd from scrtch lerning curves. All experiments use recurrent rchitecture, trined on the robot, from RGB inputs. We compre wide nd nrrow columns for both the progressive experiments nd the rndomly initilized bseline. For ll results, medinfiltered solid curve is shown overlid on the rw rewrds (dotted line). The from scrtch bseline ws rndomly initilized nrrow or wide column, both of which fil to get ny rewrd during trining. of 3 points per episode. Even on this reltively simple tsk, full performnce is only chieved fter substntil interction with the environment, on the order of million steps - number which is infesible with rel robot. The simultion trining, compred with the rel robot, is ccelerted becuse of fst rendering, multithreded lerning lgorithms, nd the bility to continuously trin without humn involvement. We clculte tht lerning this tsk, which trins to convergence in 24 hours using CPU compute cluster, would tke 3 dys on the rel robot even with continuous trining for 24 hours dy. Moreover, multiple experiments in prllel were used to explore hyperprmeters in simultion; this sort of serch would multiply the hypotheticl rel robot trining time. In simultion, we explore lerning rtes nd entropy costs, which re smpled uniformly t rndom on log scle. Lerning rtes re smpled between e- nd e-3 nd entropy costs between 1e- nd 1e-2. The configurtion with the best finl performnce from grid of 3 is chosen s first column. For rel Jco experiments, both lerning rtes nd entropy costs were optimized seprtely using simulted trnsfer experiment with single-threded gent (A2C). 4.2 Trnsfer to the robot To trin on the rel Jco, flt trget is mnully repositioned within 4cm by 3cm re on every third episode. Rewrds re given utomticlly by trcking the colored trget nd giving rewrd bsed on the position of the Jco gripper with respect to it. We trin bseline from scrtch, finetuned first column, nd progressive second column. Ech experiment is run for pproximtely 6 steps (bout four hours). The bseline is trined by rndomly initilizing nrrow network nd then trining. We lso try rndomly initilized wide network. As seen in Figure (green curve), the rndomly initilized column fils to lern nd the gent gets zero rewrd throughout trining. The progressive second column gets to 34 points, while the experiment with finetuning, which strts with the simultion-trined column nd continues trining on the robot, does not rech the sme score s the progressive network. Finetuning vs. progressive pproches. The progressive pproch is clerly well-suited for continul lerning scenrios, where it is importnt to mitigte forgetting of previous tsks while supporting trnsfer to new tsks, but the dvntge is less intuitive for curricul of tsks where the focus is on mximising trnsfer lerning. To ssess this empiriclly, we strt with simultor-trined first column, s described bove, nd then either finetune tht column or dd nrrow progressive column nd 6

7 Finl rewrds Subtle perspective chnges 3 2 finetuned progressive Trils sorted by decresing finl rewrds Finl rewrds Significnt perspective chnges 2 finetuned 2 progressive Trils sorted by decresing finl rewrds Finl rewrds Subtle color chnges finetuned progressive Trils sorted by decresing finl rewrds Finl rewrds Significnt color chnges finetuned progressive Trils sorted by decresing finl rewrds Figure 6: To nlyse the reltive stbility nd performnce of finetuning vs. progressive pproches, we dd color or perspective chnges to the environment in simultion nd then trin 3 networks with different rndom seeds, lerning rtes, nd entropy costs. The progressive networks hve significntly higher performnce nd less sensitivity to hyperprmeter selection for ll four experiments. retrin for the recher tsk under vriety of conditions, including smll or lrge color chnges nd smll or lrge perspective chnges. For ech of these environment perturbtions, we trin 3 times with different seeds, lerning rtes, nd entropy costs, which re the most sensitive hyperprmeters. As shown in Figure 6, we find tht progressive networks re more stble nd rech higher finl performnce thn finetuning. 4.3 Trnsfer to dynmic robot tsk with proprioception Unlike the finetuning prdigm, which is unble to ccommodte chnging network morphology or new input modlities, progressive nets offer flexibility tht is dvntgeous for trnsferring to new dt sources while still leverging previous knowledge. To demonstrte this, we trin second column on the recher tsk but dd proprioceptive fetures s n dditionl input, longside the RGB imges. The proprioceptive fetures re joint ngles nd velocities for ech of the 9 joints of the rm nd fingers, 18 in totl, input to MLP ( single liner lyer plus ReLu) nd joined with the outputs of the convolutionl stck. Then, third progressive column is dded tht only lerns from the proprioceptive fetures, while the visul input is forwrded through the previous columns nd the fetures re used vi the lterl connections. A digrm of this rchitecture is shown in Figure 7 (left). To evlute this rchitecture, we trin on dynmic trget tsk. By employing smll motorized pulley, the red trget is smoothly trnslted cross the tble with rndom reversls in the motion, creting trcking tsk tht requires different control policy while mintining similr visul presenttion. Other spects of the tsk, including rewrds nd episode lengths, were kept the sme. If the second column is trined on this conveyor tsk, the lerning is reltively slow, nd full performnce is reched fter steps (bout 4 hours). If the second column is insted trined on the sttic recher tsk, nd the third column is then trined on the conveyor tsk, we observe immedite trnsfer, nd full performnce is reched lmost immeditely (Figure 7, right). This demonstrtes both the utility of progressive nets for curriculum tsks, s well s the cpbility of the rchitecture to immeditely reuse previously lernt fetures. Discussion Trnsfer lerning, the bility to ccumulte nd trnsfer knowledge to new domins, is core chrcteristic of intelligent beings. Progressive neurl networks offer frmework tht cn be used for continul lerning of mny tsks nd which fcilittes trnsfer lerning, even cross the divide which seprtes simultion nd robot. We took full dvntge of the flexibility nd computtionl scling fforded by simultion nd compred mny hyperprmeters nd rchitectures for rndom strt, rndom trget control tsk with visul input, then successfully trnsferred the skill to n gent trining on the rel robot. In order to fulfill the potentil of deep reinforcement lerning pplied in rel-world robotic domins, lerning needs to become mny times more efficient. One route to chieving this is vi trnsfer lerning from simultion-trined gents. We hve described n initil set of experiments tht prove tht progressive nets cn be used to chieve relible, fst trnsfer for pixel-to-ction RL policies. 7

8 simultion relity Rel-robot-trined progressive nets (conveyor tsk) 3 2 Rewrds enc 1 enc 2 enc 3 1 x sttic tsk x x sttic tsk dynmic tsk Progressive 3 column Progressive 2 column Steps Figure 7: Rel robot trining results re shown for the dynmic conveyor tsk. A three-column rchitecture is depicted (left), in which vision (x) is used to trin column one, vision nd proprioception (φ) re used in column two, nd only proprioception is used to trin column three. Encoder 1 is convolutionl net, encoder 2 is convolutionl net with proprioceptive fetures dded before the LSTM, nd encoder 3 is n MLP. The lerning curves (right) show the results of trining on conveyor (dynmic trget) tsk. If the conveyor tsk is lerned s the third column, rther thn the second, then the lerning is significntly fster. References [1] Hn Ajkn, Pscl Germin, Hugo Lrochelle, Frnçois Lviolette, nd Mrio Mrchnd. Domindversril neurl networks. CoRR, bs/ , 214. [2] Smuel Brrett, Mtthew E. Tylor, nd Peter Stone. Trnsfer lerning for reinforcement lerning on physicl robot. In Ninth Interntionl Conference on Autonomous Agents nd Multigent Systems - Adptive Lerning Agents Workshop (AAMAS - ALA), 21. [3] Konstntinos Bousmlis, George Trigeorgis, Nthn Silbermn, Dilip Krishnn, nd Dumitru Erhn. Domin seprtion networks. CoRR, bs/ , 216. [4] Yroslv Gnin, Evgeniy Ustinov, Hn Ajkn, Pscl Germin, Hugo Lrochelle, Frnçois Lviolette, Mrio Mrchnd, nd Victor S. Lempitsky. Domin-dversril trining of neurl networks. CoRR, bs/1.7818, 21. [] Shixing Gu, Timothy P. Lillicrp, Ily Sutskever, nd Sergey Levine. Continuous deep q-lerning with model-bsed ccelertion. In ICML 216, 216. [6] Nicols Heess, Gregory Wyne, Dvid Silver, Timothy P. Lillicrp, Tom Erez, nd Yuvl Tss. Lerning continuous control policies by stochstic vlue grdients. In Advnces in Neurl Informtion Processing Systems 28: Annul Conference on Neurl Informtion Processing Systems 21, December 7-12, 21, Montrel, Quebec, Cnd, pges , 21. [7] S. Jmes nd E. Johns. 3D Simultion for Robot Arm Control with Deep Q-Lerning. ArXiv e-prints, 216. [8] Sergey Levine nd Pieter Abbeel. Lerning neurl network policies with guided policy serch under unknown dynmics. In Z. Ghhrmni, M. Welling, C. Cortes, N. D. Lwrence, nd K. Q. Weinberger, editors, Advnces in Neurl Informtion Processing Systems 27, pges Currn Assocites, Inc., 214. [9] Sergey Levine, Chelse Finn, Trevor Drrell, nd Pieter Abbeel. End-to-end trining of deep visuomotor policies. CoRR, bs/14.72, 21. [1] Sergey Levine, Peter Pstor, Alex Krizhevsky, nd Deirdre Quillen. Lerning hnd-eye coordintion for robotic grsping with deep lerning nd lrge-scle dt collection. CoRR, bs/ , 216. [11] Sergey Levine, Noln Wgener, nd Pieter Abbeel. Lerning contct-rich mnipultion skills with guided policy serch. In IEEE Interntionl Conference on Robotics nd Automtion, ICRA 21, Settle, WA, USA, 26-3 My, 21, pges , 21. [12] Timothy P. Lillicrp, Jonthn J. Hunt, Alexnder Pritzel, Nicols Heess, Tom Erez, Yuvl Tss, Dvid Silver, nd Dn Wierstr. Continuous control with deep reinforcement lerning. CoRR, bs/ , 21. 8

9 [13] Mingsheng Long, Yue Co, Jinmin Wng, nd Michel I. Jordn. Lerning trnsferble fetures with deep dpttion networks. In Proceedings of the 32nd Interntionl Conference on Mchine Lerning, ICML 21, Lille, Frnce, 6-11 July 21, pges 97 1, 21. [14] V. Mnih, Kk Kvukcuoglu, D. Silver, A. Rusu, J. Veness, M. Bellemre, A. Grves, M. Riedmiller, A. Fidjelnd, G. Ostrovski, S. Petersen, C. Bettie, A. Sdik, I. Antonoglou, H. King, D. Kumrn, D. Wierstr, S. Legg, nd D. Hssbis. Humn-level control through deep reinforcement lerning. Nture, 18(74):29 33, 21. [1] Volodymyr Mnih, Adrià Puigdomènech Bdi, Mehdi Mirz, Alex Grves, Timothy P. Lillicrp, Tim Hrley, Dvid Silver, nd Kory Kvukcuoglu. Asynchronous methods for deep reinforcement lerning. In Int l Conf. on Mchine Lerning (ICML), 216. [16] Xingcho Peng, Bochen Sun, Krim Ali, nd Kte Senko. Lerning deep object detectors from 3d models. In 21 IEEE Interntionl Conference on Computer Vision, ICCV 21, Sntigo, Chile, December 7-13, 21, pges , 21. [17] Lerrel Pinto nd Abhinv Gupt. Supersizing self-supervision: Lerning to grsp from k tries nd 7 robot hours. In ICRA 216, 216. [18] Andrei Rusu, Neil Rbinowitz, Guillume Desjrdins, Hubert Soyer, Jmes Kirkptrick, Kory Kvukcuoglu, Rzvn Pscnu, nd Ri Hdsell. Progressive neurl networks. rxiv preprint rxiv: , 216. [19] John Schulmn, Sergey Levine, Philipp Moritz, Michel I. Jordn, nd Pieter Abbeel. Trust region policy optimiztion. In Proceedings of the 32nd Interntionl Conference on Mchine Lerning (ICML), 21. [2] John Schulmn, Philipp Moritz, Sergey Levine, Michel Jordn, nd Pieter Abbeel. High-dimensionl continuous control using generlized dvntge estimtion. In Proceedings of the Interntionl Conference on Lerning Representtions (ICLR), 216. [21] Ho Su, Chrles Ruizhongti Qi, Yngyn Li, nd Leonids J. Guibs. Render for CNN: viewpoint estimtion in imges using cnns trined with rendered 3d model views. In 21 IEEE Interntionl Conference on Computer Vision, ICCV 21, Sntigo, Chile, December 7-13, 21, pges , 21. [22] E. Todorov, T. Erez, nd Y. Tss. Mujoco: A physics engine for model-bsed control. In Interntionl Conference on Intelligent Robots nd Systems IROS, 212. [23] Eric Tzeng, Coline Devin, Judy Hoffmn, Chelse Finn, Xingcho Peng, Sergey Levine, Kte Senko, nd Trevor Drrell. Towrds dpting deep visuomotor representtions from simulted to rel environments. CoRR, bs/ , 21. [24] Eric Tzeng, Judy Hoffmn, Trevor Drrell, nd Kte Senko. Simultneous deep trnsfer cross domins nd tsks. In 21 IEEE Interntionl Conference on Computer Vision, ICCV 21, Sntigo, Chile, December 7-13, 21, pges , 21. [2] Eric Tzeng, Judy Hoffmn, Ning Zhng, Kte Senko, nd Trevor Drrell. Deep domin confusion: Mximizing for domin invrince. CoRR, bs/ , 214. [26] Yuke Zhu, Roozbeh Mottghi, Eric Kolve, Joseph J. Lim, Abhinv Gupt, Li Fei-Fei, nd Ali Frhdi. Trget-driven visul nvigtion in indoor scenes using deep reinforcement lerning. CoRR, bs/ ,