Combining Fitness-based Search and User Modeling in Evolutionary Robotics

Transcription

1 Combnng Ftness-based Search and User Modelng n Evolutonary Robotcs Josh C. Bongard Dept. of Computer Scence Unversty of Vermont josh.bongard@uvm.edu Gregory S. Hornby Unv. of Calforna Santa Cruz NASA Ames Research Center gregory.s.hornby@nasa.gov ABSTRACT Methodologes are emergng n many branches of computer scence that demonstrate how human users and automated algorthms can collaborate on a problem such that ther combned solutons outperform those produced by ether humans or algorthms alone. The problem of behavor optmzaton n robotcs seems partcularly well-suted for ths approach because humans have ntutons about how anmals and thus robots should and should not behave, and can vsually detect non-optmal behavors that are trapped n local optma. Here we ntroduce a multobjectve approach n whch a surrogate user (whch stands n for a human user) deflects search away from local optma and a tradtonal ftness functon eventually leads search toward the global optmum. We show that ths approach produces superor solutons for a deceptve robotcs problem compared to a smlar search method that s guded by just a surrogate user or just a ftness functon. Categores and Subject Descrptors I.2.9[Computng Methodologes]: Artfcal Intellgence Robotcs General Terms Expermentaton, Algorthms Keywords Evolutonary Robotcs, Interactve Evolutonary Algorthms, Evolutonary Algorthms 1. INTRODUCTION One of the orgnal goals of computer scence n general, and Artfcal Intellgence n partcular, was complete automaton: once a problem s formulated n enough detal an AI algorthm should be able to automatcally generate a soluton. Recently however there has been a turn toward Copyrght Assocaton for Computng Machnery. ACM acknowledges that ths contrbuton was authored or co-authored by an employee, contractor or afflate of the natonal government of Unted States. As such, the government of Unted States retans a nonexclusve, royalty-free rght to publsh or reproduce ths artcle, or to allow others to do so, for Government purposes only. GECCO 13, July 6 10, 2013, Amsterdam, The Netherlands. ACM /13/07...$ algorthms that combne human users and automated algorthms such that the two complement one another. Humans supply ntuton, provde outsde of the box deas or recrut addtonal humans through socal networks; computers provde brute force search wthn the space delneated by the human team partcpants. One of the frst examples of so-called human-based computaton nclude the ESP game n whch mages are tagged wth approprate text descrptons generated through ndrect human corroboraton [24]. In another example, vdeo game players fold smulated protens nto approprate confguratons and often solve computatonally-ntractable proten foldng problems [25]. Many such human-based computaton algorthms recrut, flter and combne contrbutons from large numbers of users; thus crowdsourcng [9] can be seen as one form of human-based computaton. Arguably, the frst class of human-based computer algorthms s the Interactve Evolutonary Algorthm (IEA). Ths technque was poneered by Rchard Dawkns: a set of computer-generated mages resemblng nsects are shown to a user; the user selects a subset that they prefer; and the non-selected mages are replaced by mutated and/or crossed versons of the preferred mages [7]. Several examples of IEAs followed (e.g. [22], [8]). An nteractve evolutonary robotcs method was also proposed [10] n whch the ftness of a gven robot control polcy was determned by hand. However, all nteractve evolutonary algorthms that employ a human nstead of a ftness functon suffer from one crtcal drawback: user fatgue. In practce, such methods requre the user to supply hundreds or thousands of preferences to produce truly novel solutons, or solutons that compete wth those produced by automated, ftness-functon drve evolutonary algorthms. To address ths drawback, a new class of nteractve evolutonary algorthms have appeared whch buld a user model. Such algorthms follow a four-step procedure: (1) the user s presented wth several canddate solutons; (2) the user s preferences or rankngs are collected; (3) a model s traned such that t takes as nput features of a canddate soluton and produces as output a predcton for how much the user wll lke that soluton. Once traned, (4) the user model can then stand n for the human user. In [2], user feedback was nput to a parameter estmaton system. Although ths leverages statstcal machne learnng methods, t remans lmted to parameterzed desgn spaces. An alternatve method was not constraned to parameterzed desgn spaces: user feedback was used to learn weghts on grammatcal rules for constructng desgns [5]. However f desgns are

2 descrbed by a large number of rules, an ntractable amount of user feedback s requred to learn the requste number of weghts. In ths work we search a parameterzed desgn space, butlttle change to the proposed method would be requred to expand t for searchng open-ended desgn spaces. Schmdt and Lpson [18] employed neural networks to model a user who ndcated preferences for certan lne drawngs over others. Hornby and Bongard [12] extended ths method and used t to show that such a user model could successfully stand n for a human user: employng a user model mproved an IEA such that t was 2.5 tmes faster and 15 tmes more relable. Elsewhere, Hornby and Bongard [11] compared twowaysof modelngusers. Inbothcases theuser s shown a grd of canddate solutons and forced to choose ther favortes. In the frst, the user model s suppled wth pars of canddate solutons and s traned to reproduce the user s preferences. In the second, scores are computed for all of the canddate solutons shown to the user: f the user preferred soluton A over soluton B and soluton B over soluton C, then score(a) > score(b) > score(c). Then, the user model s suppled wth only one canddate soluton and the model s traned to reproduce the score of that soluton. Ths latter approach s employed n ths paper. Outsde of evolutonary computaton, learnng user models from preferences has also been explored n the doman of renforcement learnng [6]. Recently, user modelng has been appled to robotcs [1]. The method, Preference-based Polcy Learnng, trans a user model to reproduce a user-generated rankng of robot control polces. The model takes as nput a compresson of the sensor/motor tme seres generated by the robot usng a gven control polcy and must produce as output a successful predcton of that polcy s user-generated rank. However, n all of these methods that construct and employ a user model, the model replaces the ftness functon, much lke a human replaces the ftness functon n Interactve Evolutonary Algorthms. Ths beles an assumpton underlyng both user-model-based methods and Interactve Evolutonary Algorthms: ftness functons are deceptve 1. The ftness functon s also dspensed wth n novelty search [14] n whch the qualty of a canddate soluton s determned to be ts dstance from prevous solutons. However, Akrour et al [1] showed that a user-model-based method can outperform novelty search because the latter algorthm spends much effort n explorng very poor solutons. Here we hypothesze that a user model and ftness functon together can gude search better than ether on ts own. If only a user model s employed, the user must teach t to gude search away from local optma and toward the global optmum. If both are employed, the user model can gude search away from local optma but the ftness functon can then automatcally take over and gude search toward the global optmum wthout further nput from the user. The results we present here support our hypothess: If a user model complements rather than replaces a ftness functon for an evolutonary robotcs task, better solutons can be found than ether the ftness functon gudng search alone or the user model gudng search alone. The next secton descrbes the robot task and these three algorthm varants. Secton 3 presents the results from these 1 If the problem s not deceptve, then user nfluence durng search s not requred. a b e a Fgure 1: The robot, ts task envronment and two evolved behavors. When only a ftness functon s employed (a), control polces evolve that walk the robot toward the barrer and become stuck there (a.a) rather than movng around the barrer and reachng the goal poston (a.b). If a user model and a ftness functon are used (b), control polces evolve that successfully gude the robot around the barrer (b.a-e). Vdeos of the robots can be vewed at bt.ly/127jnlm. varants and sectons 4 and 5 provde some analyss, dscusson and concludng remarks. 2. METHODS Here we descrbe three algorthm varants that generate control polces for a quadrupedal robot such that the robot travels around a rectangular barrer and reaches a target object located on the other sde (Fg. 1). The frst varant uses a ftness functon that rewards control polces that mnmze the dstance from the robot to the target object (Sect. 2.1). The second varant learns a model of a human user who prefers control polces that enable the robot to reach the rght edge of the barrer and then approach the target object (Sect. 2.2). The thrd varant uses both the ftness functon and a user model to gude the robot to the target object (2.3). b d c

3 2.1 Ftness-based Search (FS) In the tradtonal evolutonary robotcs paradgm, a ftness functon s desgned and used to perform dfferental survval and reproducton wthn a populaton of smulated or physcal robots [16]. To nvestgate the relatonshp between user modelng and ftness-based search we evolve a populaton of control polces that are rewarded for gudng a smulated quadrupedal robot around a barrer to reach a target object on the far sde The robot The robot s composed of four, one degree-of-freedom rotatonal jonts that attach each leg to the man body 2. The axes of the four jonts are set such that the leg sweeps through the plane dagonal to the man body (llustrated by the arrow-and-lne n Fg. 1.b.a). Ths provdes the robot wth a holonomc drve system: the robot can move n any drecton regardless of ts orentaton. More mportantly, the robot can descrbe a curved trajectory around the barrer (Fg. 1b) wthout havng to turn n that drecton ntally. Each jont s actuated wth a motor that can rotate the jont through [ 45 o,+45 o ]. Each foot contans a bnary tactle sensor. The robot s also equpped wth fve photosensors, one n each component of ts body. The target object on the far sde of the barrer emts a lght; occluson s not modeled n these experments, so t s assumed that the photosensors can detect the lght source even f the barrer les between the object and the sensor. Photosensors return a value between zero for complete darkness and one for maxmal brghtness. The robot also has a compass sensor that returns a value also n [0,1]: zero when facng forward (ndcated by the dotted arrow n Fg. 1a.a); 0.5 when facng backward; and 0.75 when facng to the rght The control polcy A feedforward artfcal neural network wth no hdden layer s used to control the robot: each of the four tactle sensors, fve photosensors and one compass sensor connect to each of the four motors, yeldng 40 synaptc connectons to optmze The evolutonary algorthm The lght green robot n Fg. 1 ndcates the desred fnal state of the robot: ths robot s standng over the target object (unseen n the fgure). Ths generates strong sgnals n the fve photosensors embedded n ths fnal-state robot, denoted as s (r) 1,..., s(r) 5. The ftness of a control polcy (whch s to be maxmzed) s calculated as f = 1 1+( 5 T =1 t=1 s(t) s (r) )/5T where T = 1000 s the total number of tme steps that each control polcy s evaluated for; and s (t) denotes the value of the th photosensor at tme step t. We have shown n past work [3] that ths characterzaton allows for the evoluton of dfferent behavors wthout havng to rewrte the ftness functon for each new behavor. For example f brachaton s desred, the robot can be started at one end of a set of suspended rungs and the target 2 The robots were evolvedwthn theopendynamcs Engne physcs smulator. (1) object (along wth the end-state robot) placed at the other end. If star clmbng s desred, the robot can be placed at the base of the stars and the target object and end-state robot placed at the top. By creatng an nteractve 3D envronment n whch a human user can nteractvely construct the task envronment of the robot, another form of nteractve evolutonary robotcs was acheved: users can select for dfferent behavors by alterng the task envronment rather than the ftness functon. In the present work, the ftness functon was then ncorporated as one objectve n a b-objectve optmzaton method, Age-Ftness Pareto Optmzaton (AFPO) [20]: obj 1 = f (2) obj 2 = age (3) where age s defned as the age snce the control polcy or ts oldest ancestor was ntroduced nto the evolvng populaton [13]. In the frst generaton each polcy has an age of 1. Any polces that survve nto the next generaton, or ther offsprng, have an age of 2, and so on. Polces that le on the Pareto front produce offsprng that replace domnated polces. At each generaton a new, random polcy wth an age of 1 s njected nto the populaton. AFPO protects young yet promsng solutons long enough for some of them to evolve offsprng that push older solutons out of the populaton. Fg. 2 llustrates how the three algorthm varants dffer from one another. In Ftness-based Search (FS), an ntal populaton of randomly-generated control polces are created (Fg. 2a). Snce no user preferences are ever requested from the user (Fg. 2b) and thus no preferences are ever receved from the user (Fg. 2c), the objectves from Eqns. 2 and 3 are employed to evolve the control polces (Fg. 2d). 2.2 Preference-based Polcy Learnng (PPL) Preference-based Polcy Learnng (PPL) [1] nvolves four steps: the robot demonstrates a number of behavors; the user ranks the control polces that generated those behavors; amodels learned untlttakesas nputthecompressed sensor/motor tme seres of each polcy and returns the rank ofeach polcy; andfnallytherobot usesths model tosearch for polces that obtan ever hgher rankng scores from the model. PPL s here adapted to the robot and task at hand. Frst, a populaton of randomly-generated polces are created(fg. 2a). Two polces are chosen at random (Fg. 2b) and shown to the user (Fg. 2e). Once the user ndcates a preference (Fg. 2f), the preference s collected (Fg. 2h) and a rankng of the polces s computed. Ths s accomplshed by creatng an n n matrx P, where n s the number of polces that the user has suppled a preference for. Element p j = +1 f the user preferred polcy over polcy j and p j = 1 f she preferred j over. Element p = 0 for each polcy. The score of each polcy s then stored n the vector c. Each element n c s computed usng c = rowsum(p, ) mnrowsum(p) maxrowsum(p) mnrowsum(p) where rowsum(p,) s the sum of the th row n P, mn rowsum(p)sthemnmumrowsumnpandmaxrowsum(p) s the maxmum row sum n P. Ths formulaton ensures (4)

4 obj1: ft*score obj2: age obj1: score obj2: age obj1: ft obj2: age Evolve Collect pref Y k N j N d a h Y FS-PPL? Y Prefs receved? c e N b Y Pref requred? f Pref avalable? N Dsplay robots g Tran model sensor data predcted score Fgure 2: Program flow for the three algorthm varants. For Ftness-based Search, evoluton uses only the ftness functon and age (a-d). For Preferencebased Polcy Search, evoluton (a) perodcally requests (b) and receves (e,f,h) a preference from the user. Whle the user observes a robot par (e), the user model s mproved (f,g). When several preferences have been collected (c) user model-generated scores and age are used (j) durng evoluton (a). Ftness-based Search and Preference-based Polcy Search (FS-PPS) follows the same flow as PPS but uses the ftness functon, the user model and age (k) to evaluate control polces. that the lowest-rankng polcy has a score of 0 and the hghest-rankng polcy a score of 1. A sngle polcy s now drawn from the polcy populaton (Fg. 2a) and shown alongsde one of the orgnal two polces shown to the user (Fg. 2e). Whle the algorthm wats for the user to ndcate a preference, tranng of the user model can now commence User Model Tranng In the PPL varant mplemented here, the user model takes the form of an artfcal neural network. Sensor data generated by a control polcy s suppled at the nput layer and the value arrvng at the sngle output neuron s treated as the model s predcton of that polcy s rank. The nput layer s composed of 12 neurons. The frst sx neurons report the values of the fve photosensors (s 1-s 5) and one compass sensor (s 6) halfway through the evaluaton perod (s (T/2) 1... s (T/2) 6 ). The second sx neurons report the values of the fve photosensors and one compass sensor at the end of the evaluaton perod (s (T) 1... s (T) 6 ). The orgnal PPL formulaton reported n [1] suppled a compresson of the entre sensor-motor tme seres to the user model. Here, only a subset of the sensor data s suppled to the user model. Admttedly ths subset was chosen because tsclear thatforthstasktherobotshouldmovetotherght of the barrer about halfway through the evaluaton perod (Fg. 1b.c) and then to another poston at the end of the evaluaton perod (Fg. 1b.e). Thus, ths partcular subset of sensor data may not allow the user model to learn a user s preferences f they prefer some other behavor. However, ths ssue of what aspect of the robot s behavor to present to the user model has been dscussed elsewhere. Although mportant and n need of nvestgaton, ths ssue s outsde the scope of the present work. Whle the user s consderng whch of two robots to prefer (Fg. 2e), the user model s traned usng backpropagaton [17]. For each polcy out of the n polces that the user has ndcated a preference for so far (at the outset ths s n = 2), the sensor data from polcy s presented at the nput layer and the predcted score s collected from the output neuron. The error between ths predcted score and the score stored at c s computed and back-propagated. Backpropagaton contnues to terate across the n polces untl the user supples a new preference (Fg. 2f). (Ths allows the model to capture a user s strategy regardless of whether her preferences are transtve.) Once the user does supply a preference (Fg. 2f,h), element p k and p k can be flled n the preference matrx P, where s one of the orgnal two control polces and k s the polcy newly-drawn from the populaton. So the user s now shown two robots controlled by polces j and k, where polcy j was the second of the orgnal two polces. Once the user supples ths preference, P s completely flled n and c can be updated. Thus whenever a new polcy s drawn from the populaton (Fg. 2a,b), the user must supply m new preferences, where m s the number of polces that have already been shown to the user. Once c s recomputed, evoluton can commence(fg. 2a): the polces n the current populaton are evolved usng ther ages and ther scores as predcted by the user model (Fg. 2b,c,,j) Drawng a Polcy for Scorng Ten generatons of evoluton elapse before the algorthm sends a new polcy to the user for scorng. For each new polcy drawn from the evolvng populaton, an addtonal row and column are added to P and an addtonal element s added to c to accommodate the addtonal preferences and score t generates, respectvely. When the score for the new polcy has been determned, another 10 generatons evolve before a new polcy s sent to the user, and so on untl the tral termnates. There are varous methods that could be employed to decde whch polcy, when scored by the user, wll elct the most nformaton about what the user prefers. One method would be to employ query by commttee [23] n whch multple models are traned, and the polcy sent to the user s the only that maxmzes the score predctons across the models. Another would be to employ the Estmaton-Exploraton Algorthm [4], whch uses model dsagreement as n query by commttee, but evolves a separate populaton of polces such that the ftness of a polcy s the amount of score predcton dsagreement t nduces n the user models. Query by commttee was attempted n the present work (data not shown) but t dd not yeld a better result than a smpler method: Wthn the current populaton, the polcy wth the hghest predcted score s sent to the user, as long as that polcy has not prevously been sent The Surrogate User In order to compare dfferent user modelng methods t

5 s necessary to collect a large number of preferences from a human user. Followng [12, 11] we here create a surrogate user. The surrogate user s a separate algorthm that supples preferences nstead of a human user. The surrogate user can be programmed to mmc any detal of a human user, such as nose (mstakenly preferrng the less satsfactory polcy) or fatgue (the tme to receve a preference from the user ncreases as evoluton proceeds). Because n the PPL varant the ftness functon s not used, the user must gude the robot to the rght edge of the barrer (Fg. 1b.c) and then to the end poston (Fg. 1b.e) ndrectly usng preferences. Thus, for any par of polces and j shown to the surrogate user, t wll prefer polcy over j f dedge() + dend() < dedge(j)+dend(j) (5) 2 2 dedge() = (x (T 2 ) 4.5) 2 +(y (T 2 ) 3) 2 (6) dend() = (x (T) 0) 2 +(y (T) 6) 2, (7) where dedge() reports how close control polcy gets the robot to therght edge of the barrer (located at x = 4.5,y = 3.0; see Fg. 1a) halfway through the evaluaton perod(t/2), and dend() reports how close control polcy gets the robot to the end poston (located at x = 0.0,y = 6.0) by the end of the evaluaton perod (T). Otherwse, t wll prefer polcy j over polcy. 2.3 Combnng Ftness-based Search and Preference-based Polcy Learnng (FS-PPL) The proposed method combnes ftness-based search wth preference-based polcy learnng (FS-PPL): both the ftness functon and the user model gude search. The FS-PPL algorthm follows exactly the same program flow as the PPL varant descrbed above, wth four modfcatons. Frst, the two objectves used durng evoluton combne age, ftness, and scores produced by the user model(fg. 2k). It would have been possble to conduct evoluton usng three objectves rather than two, but there are known challenges wth ncreasng the number of objectves [21] and ths would have made a far comparson to the frst two algorthm varants (both of whch employ two objectves) more dffcult. The second modfcaton nvolves the surrogate user. The surrogate user n PPL prefers control polces that get the robot close to the barrer s edge halfway through evaluaton and close to the target poston at the end of evaluaton (Eqn. 5). The surrogate user employed n FS-PPL prefers polcy over polcy j f dedge() < dedge(j). (8) That s, the surrogate user prefers polces that get the robot as close to the barrer s rght edge as possble halfway through evaluaton. It s assumed that once the user model learns ths preference and gudes the robot to ths pont, the ftness functon wll gude the robot toward the target object n the latter half of the evaluaton perod. The thrd modfcaton nvolved a reducton n the sze of the user model. For PPL, the user model s a 12-nput, 3-hdden, 1-output neuron feedforward neural network. For FS-PPL, the user model s a 6-nput, 3-hdden, 1-output network. Snce the user s preferences are only senstve to the poston of the robot halfway through the evaluaton perod, durng tranng of the user model n FS-PPL, only the values of the fve photosensors and one compass sensor halfway through the evaluaton perod are fed as nput (s (T/2) 1,..., s (T/2) 5, s (T/2) 6 ). Ths does reduce the number of free parameters that must be learned through backpropagaton n the FS-PPL surrogate user compared to the PPL surrogate user, but was consdered a far comparson. If a ftness functon s not employed, for ths task, the user model must make predctons based on sensor data from halfway through and at the end of the evaluaton perod. The FS-PPL user model only needs to learn based on sensor data from halfway through the evaluaton perod. The fourth dfference between FS-PPL and PPL s that the FS-PPL surrogate user wll stop supplyng preferences once any control polcy enables the robot to move above the barrer at any pont durng ts trajectory (ts poston reaches y > 3; Fg. 1a). It s assumed that at ths pont, the user model s suffcently well traned to gude robots to the barrer s rght-hand edge. Henceforth, among polces that get the robot beyond the barrer, the ftness functon wll favor those that get closer to the target object n the latter half of the evaluaton perod. 3. RESULTS For each of thethree algorthm varants (FS, PPL andfs- PPL), 100 ndependent trals were conducted. Each tral was conducted usng a populaton sze of 30. Each tral contnued untl 9 hours of CPU tme elapsed 3. Fg. 3a reports how close the best control polcy n the populaton got to the target object at the end of the tral. As can be seen, combnng a ftness functon and user model (dotted lne) led to polces that got sgnfcantly closer to the target object than those polces evolved usng only a ftness functon (sold lne) or only a user model (dashed lne). The preferences of the surrogate models are predcated on the absolute poston of the robot (Eqns. 5-8), yet the user model only has access to the robot s photo- and compass sensor values. Thus, the user model must learn some functon of these sensor values that approxmates Eqns The success of FS-PPL demonstrates that ths s possble. However, t s possble that n some crcumstances the user model may not be capable of learnng such a mappng. To test ths we formulated two addtonal algorthm varants: we agan ran PPL and FS-PPL, but employed neural network-based user models that had no hdden layer. We ran 100 trals of each, and report the result n Fg. 3b. Now, both PPL and FS-PPL perform worse than the algorthm that only employs the ftness functon. Ths ndcates that there s a mappng between sensor values and the absolute poston of the robot, but that that mappng s nonlnear: the user models that can only learn lnear functons faled to learn such a mappng. Fnally, t s possble for a user model to fal to learn a mappng from sensor values to absolute poston because the robot s equpped wth an nsuffcent number or type of 3 AlthoughCPUtmedoesnotreporttheabsoluteamountof computatonal effort requred to reach a gven dstance from the target object, t does provde nsght nto the relatve performance of the three algorthm varants gven the same computatonal budget.

6 Dstance from target object Dstance from target object Dstance from target object CPU Hours (a) CPU Hours (b) CPU Hours (c) Fgure 3: Performances of ftness-based search (dashed lne), preference-based polcy learnng (sold lne) and ftness-based search combned wth preference-based polcy learnng (dotted lne). Thck lnes present the mean; pars of thn lnes bracket ±1 standard error of the mean. Relatve performances employng neural network-based user models wth one hdden layer of three neurons (a); no hdden layer (b); and one hdden layer wth three neurons but dened nput from the compass sensor (c). sensors. In a fnal par of varants we ran PPL and FS-PPL n whch the fve photo sensor values were fed to the user model, but the compass sensor value was wthheld (a constant value of 1 was suppled nstead). As shown n Fg. 3c, as for the case of the lnear user models, PPL (dashed lne) and FS-PPL (dotted lne) agan faled to learn a predctve user model and both varants faled n comparson to the ftness functon-only varant (sold lne). 4. DISCUSSION The falure of the ftness-based search varant s obvous: there s a local optmum n whch the robot runs forward, colldes wth the barrer and stays trapped there. The successful behavor of the FS-PPL varant usng nonlnear models and both photo- and compass sensor data (Fg. 3a, dotted lne) however s nstructve. The ntal spke early n evoluton ndcates the perod durng whch the user model learns the surrogate user s strategy: the user model learns to reward control polces that gude the robot to the rght-hand edge of the barrer by the halfway pont of the evaluaton perod. Such polces are correctly gven a hgh score by the user model, and control polces that drve the robot nto the barrer (Fg. 1a) are correctly gven low scores. However, most of these strateges actually gude the robot further from the target object than those that drve the robot nto the barrer: the robot may walk to the barrer s rght-hand edge but then walk further to the rght, or may collde wth the barrer s edge and come to a stop. Ths explans the spke n dstance durng the early evoluton of FS-PPL. At ths pont however the surrogate user stops provdng preferences because at least one of these robots gets beyond the barrer (y > 3). Now, search gradually evolves polces that reach the barrer s rght-hand edge but also manages to move beyond t and slghtly toward the target object: such polces receve a hgher ftness value than those that that get stuck at the barrer even though both polces may obtan about the same user model score. Thus, even though the user has stopped nteractng wth the system and the user model ceases to learn or dscover polces wth ever hgher scores, the control polces contnue to mprove. In contrast, the PPL varant fals because ether t never learns the surrogate user s strategy or t requres many more user preferences: the user model must frst learn to gude robots to the barrer s edge, and then collect addtonal preferences to learn to gude t to the target object. 4.1 User model analyss A challenge for any user modelng system however s ensurng that the user model can learn the user s strategy even fshemakesdecsonsbasedondatathatsnotdrectlyavalable to the model. In the example gven here, the surrogate user provdes preference based on the absolute poston of the robot, yet the robot cannot sense ts own poston. The success of the FS-PPL varant ndcates that such a mappng can be learned, but only f a nonlnear model s employed, and only f both photo- and compass sensor data s made avalable to t. The Eureqa symbolc regresson tool [19] was used to dscover what form ths mappng from sensor data to absolute poston s. Ths was accomplshed by takng the traned user model from one of the successful FS-PPL trals and 200 control polces at random from later generatons of

7 Mean Absolute Error [MAE] a 0.2 b Error Tme [seconds] Complexty Fronter Solutons sze Fgure 4: a: Evolutonary mprovement n dscoverng a relatonshp between a successful user model s output and the sensor data suppled to t. b: The fnal Pareto front of the evolutonary search. that run. Each of the 200 polces was re-run and the sensor values arsng half way through the evaluaton perod were recorded. Each of these 200 sensor value sets was suppled to the user model and the score returned by the model was recorded. Ths provded us wth 200 tuples of the form (s (T/2) 1,...,s (T/2) 6,o) where o s the score output by the user model and les n the range of [0,1]. Ths data was provded to Eureqa and allowed to evolve for 42 mnutes on a standard PC 4. Fg. 4a reports the evolutonary mprovement n search and Fg. 4b reports the fnal Pareto front returned by Eureqa, whch evolves low error and low complexty solutons of the form o = f(s (T/2) 1,...,s (T/2) 6 ). Only the four algebrac operators +,,, and / were allowed. Table 1 reports a subset of the equatons from the front shown n Fg. 4b. The most accurate model of sze 3 ndcates that a hgh score s output f s 5 takes as low a value as possble. Ths ffth photosensor s located n the robot s back left leg (ths leg s darkened and crcled n Fg. 1b). As can be seen n that fnal successful trajectory, the back left leg ponts away from the target object at the half way pont (Fg. 1b.c), lowerng ths sensor s value relatve to the other four photosensors. However, the s 1 terms n the denomnators of the 20- and 22-node solutons ndcate that hgher scores are gven for hgher values of s 1, whch s the front left leg. Agan, n Fg. 1b.c t can be seen that the front left s closer to the target object than the back left leg. Thus, part of the user model rewards for the front leg beng closer to the target object than the back left leg. But 4 Do not panc: ths amount of computaton tme was chosen arbtrarly. Table 1: Approxmatons of a successful user model. Sze=number of operators and operands comprsng the model; r 2 =the amount of varaton explaned by that model. Sze r 2 User model o = o = 0.83 s o = s o = s o = s s o = s 3 +s 6 s o = o = s s 2 6 s s 3 s s s 3 6 s s 2 6 these relatve sensor values can also be acheved by the robot that drves nto the barrer (Fg. 1a.a). So, the user model also rewards for partcular values of the compass sensor (s 6 s prevalent n Table 1), whch ndcates that the user model rewards for partcular orentatons of the robot. By combnng the photo- and compass sensor values, the user model s guardng aganst sensor alasng. Fnally, t s clear that the lnear models (frst three rows n Table 1) are poor approxmatons of the user model and only the nonlnear models (lower rows) approach good approxmatons of the user model. Ths corroborates our fndng that only nonlnear user models (coupled wth the ftness functon) successfully dscovered the optmal soluton for ths problem. 5. CONCLUSIONS Here we have demonstrated an evolutonary robotcs method n whch a human user teaches a user model to gude search away from local optma, but can then cease nteractng wth the system whle the ftness functon automatcally gudes search toward the global optmum. Ths s demonstrated here as follows: the user gudes the robot toward the rght edge of a barrer ndrectly by supplyng preferences for rghtward tendng trajectores. Once the robot clears the barrer the user ceases nteracton, and the ftness functon selects for control polces that brng the robot from the barrer s edge to the target poston. Ths stands n contrast to Preference-based Polcy Learnng [1], whch also employs a learned model of the user but, snce there s no ftness functon, the user must gude the robot to the barrer s edge and then to the target poston. Ths approach was found to perform poorly on ths task. Our fndng resembles the fndng n [15] where t was shown that novelty search whch, lke PPL, attempts to guard search from becomng mred n local optma can beneft from the focusng effect of a ftness functon. In the work presented here, the user s preference could easly be modeled because preferences were generated by a fxed strategy. In future work we wll nvestgate user preferences that are more nosy, can only be descrbed by complex functons derved from sensor data, and non-statonary.

8 Besdes provdng preferences, there are other ways that human users mght postvely nfluence an evolutonary algorthm wthout havng to wrte code. Gven the rght ftness functon, a user can create vrtual task envronments nteractvely that select for dfferent knds of behavors [3]. Or, the user may manpulate a vrtual robot to demonstrate an approxmaton of the desred behavor and multobjectve optmzaton may then balance retenton of the demonstrated behavor wth satsfacton of the ftness functon. In general, user modelng methods show promse because they allow a casual user to ncorporate ther ntutons about a robot task nto search wthout havng to wrte computer code: they do so smply by ndcatng whch behavors they lke more than others. In future work we plan to nvestgate multobjectve systems that combne dversty-generatng methods such as novelty search, ftness functons, user modelng, dfferent optons for user nteracton and combnng nteractons from multple users. Acknowledgements Ths work was supported by Natonal Scence Foundaton Grant PECASE and DARPA M3 grant W911NF The authors also acknowledge the Vermont Advanced Computng Core whch s supported by NASA(NNX 06AC88G), at the Unversty of Vermont for provdng Hgh Performance Computng resources that have contrbuted to the research results reported wthn ths paper. 6. REFERENCES [1] R. Akrour, M. Schoenauer, and M. Sebag. Preference-based polcy learnng. Machne Learnng and Knowledge Dscovery n Databases, pages 12 27, [2] G. Barnum and C. Mattson. A computatonally asssted methodology for preference-guded conceptual desgn. Journal of mechancal desgn, 132(12), [3] J. Bongard, P. Belveau, and G. Hornby. Avodng local optma wth nteractve evolutonary robotcs. In Proceedngs of the fourteenth nternatonal conference on Genetc and evolutonary computaton conference companon, pages ACM, [4] J. Bongard and H. Lpson. Automated reverse engneerng of nonlnear dynamcal systems. Proceedngs of the Natonal Academy of Scence, 104(24): , [5] M. Campbell, R. Ra, and T. Kurtoglu. A stochastc graph grammar algorthm for nteractve search. In 14th Desgn for Manufacturng and the Lfe Cycle Conference, pages , [6] W. Cheng, J. Fürnkranz, E. Hüllermeer, and S. Park. Preference-based polcy teraton: Leveragng preference learnng for renforcement learnng. Machne Learnng and Knowledge Dscovery n Databases, pages , [7] R. Dawkns. The blnd watchmaker: Why the evdence of evoluton reveals a unverse wthout desgn. WW Norton & Company, [8] K. Deb, A. Snha, P. Korhonen, and J. Wallenus. An nteractve evolutonary multobjectve optmzaton method based on progressvely approxmated value functons. Evolutonary Computaton, IEEE Transactons on, 14(5): , [9] A. Doan, R. Ramakrshnan, and A. Halevy. Crowdsourcng systems on the world-wde web. Communcatons of the ACM, 54(4):86 96, [10] F. Gruau and K. Quatramaran. Cellular encodng for nteractve evolutonary robotcs. In Fourth European Conference on Artfcal Lfe, pages , [11] G. Hornby and J. Bongard. Acceleratng human-computer collaboratve search through learnng comparatve and predctve user models. In Procs. of the fourteenth Intl. Conf. on Genetc and evolutonary computaton conference, pages , [12] G. Hornby and J. Bongard. Learnng comparatve user models for acceleratng human-computer collaboratve search. Evolutonary and Bologcally Inspred Musc, Sound, Art and Desgn, pages , [13] G. S. Hornby. ALPS: The age-layered populaton structure for reducng the problem of premature convergence. In Proc. of the Genetc and Evolutonary Computaton Conference, GECCO-2006, pages , [14] J. Lehman and K. Stanley. Explotng open-endedness to solve problems through the search for novelty. Artfcal Lfe, 11:329, [15] J. Mouret. Novelty-based multobjectvzaton. New Horzons n Evolutonary Robotcs, pages , [16] S. Nolf and D. Floreano. Evolutonary Robotcs. MIT Press, Boston, MA, [17] D. Rumelhart, G. Hnton, and R. Wllams. Learnng representatons by back-propagatng errors. Cogntve modelng, 1:213, [18] M. Schmdt and H. Lpson. Actvely probng and modelng users n nteractve coevoluton. In Procs. of the Eghth Annual Conf. on Genetc and Evolutonary Computaton, pages , [19] M. Schmdt and H. Lpson. Dstllng free-form natural laws from expermental data. Scence, 324(5923):81, [20] M. Schmdt and H. Lpson. Age-ftness pareto optmzaton. Genetc Programmng Theory and Practce VIII, pages , [21] O. Schutze, A. Lara, and C. Coello. On the nfluence of the number of objectves on the hardness of a multobjectve optmzaton problem. Evolutonary Computaton, IEEE Transactons on, 15(4): , [22] J. Secretan, N. Beato, D. D Ambroso, A. Rodrguez, A. Campbell, J. Folsom-Kovark, and K. Stanley. Pcbreeder: A case study n collaboratve evolutonary exploraton of desgn space. Evolutonary Computaton, 19(3): , [23] H. S. Seung, M. Opper, and H. Sompolnsky. Query by commttee. In Proceedngs of the Ffth Workshop on Computatonal Learnng Theory, pages , New York: ACM Press, [24] L. Von Ahn. Games wth a purpose. Computer, 39(6):92 94, [25] P. Wolynes. Latest foldng game results: Proten A barely frustrates computatonalsts. Proceedngs of the Natonal Academy of Scences, 101(18): , 2004.