ECE-517: Reinforcement Learning in Artificial Intelligence

Transcription

1 ECE-517: Reinforcemen Learning in Arificial Inelligence Lecure 12: Generalizaion and Funcion Approximaion Ocober 13, 2015 Dr. Iamar Arel College of Engineering Deparmen of Elecrical Engineering and Compuer Science The Univeriy of Tenneee Fall

2 Ouline Inroducion Value Predicion wih funcion approximaion Gradien Decen framework On-Line Gradien-Decen TD(l) Linear mehod Conrol wih Funcion Approximaion ECE 517: Reinforcemen Learning in AI 2

3 Inroducion We have o far aumed a abular view of value or aevalue funcion Inherenly limi our problem-pace o mall ae/acion e Space requiremen orage of value Compuaion complexiy weeping/updaing he value Communicaion conrain geing he daa where i need o go Realiy i very differen high-dimenional ae repreenaion are common We will nex look a generalizaion an aemp by he agen o learn abou a large ae e while viiing/ experiencing only a mall ube of i People do i how can machine achieve he ame goal? ECE 517: Reinforcemen Learning in AI 3

4 General Approach Luckily, many approximaion echnique have been developed e.g. mulivariae funcion approximaion cheme We will uilize uch echnique in a RL conex ECE 517: Reinforcemen Learning in AI 4

5 Value Predicion wih FA A uual, le ar wih predicion of V p Inead of uing a able for V, he laer will be repreened in a parameerized funcional form T ( 1), (2),, ( n) ranpoe We ll aume ha V i a ufficienly mooh differeniable funcion of, for all. For example, a neural nework can be rained o predic V where are he connecion weigh V ( ) f We will require ha i much maller han he ae e, or ha our funcion i regularized When a ingle ae i backed up, he change generalize o affec he value of many oher ae ECE 517: Reinforcemen Learning in AI 5,

6 Adap Supervied Learning Algorihm Training Info = deired (arge) oupu Inpu Supervied Learning Syem Oupu Training example = {inpu, arge oupu} Error = (arge oupu acual oupu) ECE 517: Reinforcemen Learning in AI 6

7 Performance Meaure Le u aume ha raining example all ake he form decripion of p, V ( ) A common performance meric i he mean-quared error (MSE) over a diribuion P : MSE( ) P( ) S Q: Why ue P? I MSE he be meric? Le u aume ha P i alway he diribuion of ae a which backup are done On-policy diribuion: he diribuion creaed while following he policy being evaluaed Sronger reul are available for hi diribuion. p V ( ) V ( ) 2 ECE 517: Reinforcemen Learning in AI 7

8 Gradien Decen Le f be any funcion of he parameer pace. I gradien a any poin in hi pace i : T f ( ) f ( ) f ( ) f ( ),,,. (1) (2) ( n) (2) (1), (2) T (1) (1) We ieraively move down he gradien: 1 f ( ) ECE 517: Reinforcemen Learning in AI 8

9 ECE 517: Reinforcemen Learning in AI 9 Gradien Decen in RL Le now conider he cae where he arge oupu, v, for ample i no he rue value (unavailable) In uch cae we perform an approximae updae, uch ha where v i an unbiaed eimae of he arge oupu. Example of v are: Mone Carlo mehod: v = R TD(l): R l The general gradien-decen i guaraneed o converge o a local minimum ) ( ) ( ) ( 2 ) ( ) ( V V v V v V V p

10 On-Line Gradien-Decen TD(l) ECE 517: Reinforcemen Learning in AI 10

11 ECE 517: Reinforcemen Learning in AI 11 Reidual Gradien Decen The following aemen i no compleely accurae: ince i ugge ha which i no rue, e.g. o, we hould be wriing (reidual GD): Commen: he whole cheme i no longer upervied learning baed! ) ( ) ( ) ( V V v V v 0 v ) ( ) ( V v V r v ) ( ) ( ) ( V V V v

12 Linear Mehod One of he mo imporan pecial cae of GD FA V become a linear funcion of he parameer vecor For every ae, here i a (real valued) column vecor of feaure T ( 1), (2),, ( n) The feaure can be conruced from he ae in many way The linear approximae ae-value funcion i given by V n T ( ) ( i) ( i) V ( )? i1 ECE 517: Reinforcemen Learning in AI 12

13 Nice Properie of Linear FA Mehod The gradien i very imple: For MSE, he error urface i imple: quadraic urface wih a ingle (global) minimum Linear gradien decen TD(l) converge: Sep ize decreae appropriaely V () On-line ampling (ae ampled from he on-policy diribuion) Converge o parameer vecor wih propery: MSE( ) 1 l 1 MSE( ) (Tiikli & Van Roy, 1997) be parameer vecor ECE 517: Reinforcemen Learning in AI 13

14 Limiaion of Pure Linear Mehod Many applicaion require a mixure (e.g. produc) of he differen feaure componen Linear form prohibi direc repreenaion of he ineracion beween feaure Inuiion: feaure i i good only in he abence of feaure j Example: Pole Balancing ak High angular velociy can be good or bad If he angle i high imminen danger of falling (bad ae) If he angle i low he pole i righing ielf (good ae) In uch cae we need o inroduce feaure ha expre a mixure of oher feaure ECE 517: Reinforcemen Learning in AI 14

15 Coare Coding Feaure Compoiion/Exracion 0 ECE 517: Reinforcemen Learning in AI 15

16 Shaping Generalizaion in Coare Coding If we rain a one poin (ae), X, he parameer of all circle inerecing X will be affeced Conequence: he value funcion of all poin wihin he union of he circle will be affeced Greaer affec for poin ha have more circle in common wih X ECE 517: Reinforcemen Learning in AI 16

17 Learning and Coare Coding All hree cae have he ame number of feaure (50), learning rae i 0.2/m (m he number of feaure preen in each example) ECE 517: Reinforcemen Learning in AI 17

18 Tile Coding 0 Binary feaure for each ile Number of feaure preen a any one ime i conan Binary feaure mean weighed um eay o compue Eay o compue indice of he feaure preen ECE 517: Reinforcemen Learning in AI 18

19 Tile Coding Con. Irregular iling Hahing 0 ECE 517: Reinforcemen Learning in AI 19

20 Radial Bai Funcion (RBF) Naural exenion of coare coding o coninuou-valued feaure e.g., Gauian (i) exp c i 2 2 i 0 2 ECE 517: Reinforcemen Learning in AI 20

21 Conrol wih Funcion Approximaion Learning ae-acion value Training example of he form: decripion of The general gradien-decen rule: 1 v Q (,a ) Q(,a ) Gradien-decen Sara(l) (backward view):, a and v 1 e where r 1 Q ( 1,a 1 ) Q (,a ) e le 1 Q (,a ) ECE 517: Reinforcemen Learning in AI 21

22 GPI wih Linear Gradien Decen Sara(l) ECE 517: Reinforcemen Learning in AI 22

23 GPI Linear Gradien Decen Wakin Q(l) ECE 517: Reinforcemen Learning in AI 23

24 Unique Challenge Poed by Reinforcemen Learning In pracice, i i difficul o apply off-he-helf upervied learning echnique Reinforcemen learning violae many aumpion commonly made for funcion approximaion Nonaionary arge Nonaionary inpu diribuion Non-uniform diribuion over raining example Correlaion beween inpu/oupu pair "Cla" (acion) imbalance Can reul in unable, low, or eniive learning, or may preven ucceful learning ourigh Noable wih larger or more complicaed model Soluion? ECE 517: Reinforcemen Learning in AI 24

25 Addreing he Challenge Reidual gradien help o mooh he effec of moving arge Can preven your model from chaing arge Uing a "frozen" copy of your model o generae arge Targe will be ou of dae in a ene, bu fixed Sore previou raniion in a buffer Allow (more) uniform ampling of raining example Adap eligibiliy race o your model Can help wih correlaed example Ue enemble of model or pariion model parameer Can preven "unlearning" or forgeing pa experience Very acive area of reearch ECE 517: Reinforcemen Learning in AI 25

26 Mounain-Car Tak Example Challenge: driving an underpowered car up a eep mounain road Graviy i ronger han i engine Soluion approach: build enough ineria from oher lope o carry i up he oppoie lope Example of a ak where hing can ge wore in a ene (farher from he goal) before hey ge beer Hard o olve uing claic conrol cheme Reward i -1 for all ep unil he epiode erminae Acion full hrole forward (+1), full hrole revere (-1) and zero hrole (0) Two 9x9 overlapping ile were ued o repreen he coninuou ae pace ECE 517: Reinforcemen Learning in AI 26

27 Mounain-Car Tak ECE 517: Reinforcemen Learning in AI 27

28 Mounain-Car Reul (five 9 by 9 iling were ued) ECE 517: Reinforcemen Learning in AI 28

29 Mounain Car wih Radial Bai Funcion ECE 517: Reinforcemen Learning in AI 29

30 Summary Generalizaion i an imporan RL aribue Adaping upervied-learning funcion approximaion mehod Each backup i reaed a a learning example Gradien-decen mehod Linear gradien-decen mehod Radial bai funcion Tile coding Nonlinear gradien-decen mehod? NN Backpropagaion? Subleie involving funcion approximaion, boorapping and he on-policy/off-policy diincion ECE 517: Reinforcemen Learning in AI 30