ECE-517 Reinforcement Learning in Artificial Intelligence

Transcription

1 ECE-517 Reinforcemen Learning in Arificial Inelligence Lecure 11: Temporal Difference Learning (con.), Eligibiliy Traces Ocober 8, 2015 Dr. Iamar Arel College of Engineering Deparmen of Elecrical Engineering and Compuer Science The Universiy of Tennessee Fall

2 Ouline Acor-Criic Model (TD) Eligibiliy Traces ECE 517: Reinforcemen Learning in AI 2

3 Acor-Criic Mehods Explici (and independen) represenaion of policy and value funcion A criique (scalar) signal drives all learning in boh acor and criic These mehods received much aenion early on, and are being revisied now! Appealing in conex of psychological and neural models Dopamine Neurons (W. Schulz e al., Cambridge, 2003) ECE 517: Reinforcemen Learning in AI 3

4 Acor-Criic Deails Typically, he criic is a sae-value funcion Afer each acion selecion, an evaluaion error is obained in he form V ( s ) 1 V ( s 1 where V is he criic s curren value esimae Posiive error acion a should be srenghened for he fuure Typical acor is a parameerized mapping of saes o acions Suppose acions are generaed by sofmax hen he agen can updae he preferences as r p( s, a) e ( s, a) Pr p( s, b) e p( s a a s s, a ) p( s, a ) ECE 517: Reinforcemen Learning in AI 4 b )

5 Acor Criic Models (con.) Acor-Criic mehods offer a powerful framework for scalable RL sysems (as will be shown laer) They are paricular ineresing since hey Operae inherenly online Require minimal compuaion in order o selec acions e.g. Draw a number from a given disribuion Using neural neworks i will be equivalen o a single feed-forward pass ECE 517: Reinforcemen Learning in AI 5

6 Summary of TD TD is based on predicion (and associaed error) Inroduced one-sep abular model-free TD mehods Exended predicion o conrol by employing some form of GPI On-policy conrol: SARSA Off-policy conrol: Q-learning These mehods boosrap and sample, combining aspecs of DP and MC mehods Have shown o have some correspondence wih biological cogniive processes ECE 517: Reinforcemen Learning in AI 6

7 Unified View of RL mehods (so far) ECE 517: Reinforcemen Learning in AI 7

8 Eligibiliy Traces ET are one of he basic pracical mechanisms in RL Almos any TD mehods can be combined wih ET o obain a more efficien learning engine Combine TD conceps wih Mone Carlo ideas Addresses he gap beween evens and raining daa Temporary record of occurrence of an even Trace marks memory parameers associaed wih he even as eligible for undergoing learning changes When TD error is recorded eligible saes or acions are assigned credi or blame for he error There will be wo views of ET Forward view more heoreic Backward view more mechanisic ECE 517: Reinforcemen Learning in AI 8

9 n-sep TD Predicion Idea: Look farher ino he fuure when you do TD backup (1, 2, 3,, n seps) ECE 517: Reinforcemen Learning in AI 9

10 Mahemaics of n-sep TD Predicion Mone Carlo: R r 1 r 2 2 r 3 T 1 r T TD(0): 1-sep esimae of remaining reurn: R (1) r 1 V (s 1 ) muli-sep TD: 2-sep reurn: R (2) r 1 r 2 2 V (s 2 ) n-sep reurn a ime : (n R ) r 1 r 2 2 r 3 n1 r n n V (s n ) ECE 517: Reinforcemen Learning in AI 10

11 Learning wih n-sep Backups Backup (on-line or off-line): V (s ) (n R ) V (s ) Error reducion propery of n-sep reurns (n max E s {R ) s s} V (s) n max s V (s) V (s) Maximum error using n-sep reurn Maximum error using V(s) Using his, one can show ha n-sep mehods converge Yields a family of mehods, of which TD and MC are members ECE 517: Reinforcemen Learning in AI 11

12 On-line vs. Off-line Updaing In on-line updaing updaes are done during he episode, as soon as he incremen is compued In ha case we have V 1( s) V ( s) V ( s) In off-line updaing we updae he value of each sae a he end of he episode Incremens are accumulaed and calculaed on he side Values are consan hroughou he episode Given a value V(s), he new value (in he nex episode) will be V ( s) T 1 0 V ( s) ECE 517: Reinforcemen Learning in AI 12

13 Random Walk Revisied: e.g. for 19-Sep Random Walk ECE 517: Reinforcemen Learning in AI 13

14 Averaging n-sep Reurns n-sep mehods were inroduced o help wih TD(l) undersanding Idea: backup an average of several reurns e.g. backup half of 2-sep and half of 4-sep R avg The above is called a complex backup Draw each componen (2) (4) Label wih he weighs for ha componen TD(l) can be viewed as one way of averaging n-sep backups 1 2 R 1 2 R One backup ECE 517: Reinforcemen Learning in AI 14

15 Forward View of TD(l) TD(l) is a mehod for averaging all n-sep backups Weigh by l n-1 (ime since visiaion) l-reurn: R l (1 l) l n1 (n ) R n1 Backup using l-reurn: V (s ) R l V (s ) ECE 517: Reinforcemen Learning in AI 15

16 l-reurn Weighing Funcion for episodic asks R l (1 l) T 1 l n1 n1 Unil erminaion R (n ) l T 1 R Afer erminaion ECE 517: Reinforcemen Learning in AI 16

17 Relaion of l-reurn o TD(0) and Mone Carlo l-reurn can be rewrien as: R l (1 l) T 1 l n1 n1 If l = 1, you ge Mone Carlo: R (n ) l T 1 R R l (11) T 1 1 n1 n1 R (n ) 1 T 1 R R If l = 0, you ge TD(0) R l (1 0) T 1 n1 0 n1 R ( n) 0 T 1 R R (1) reminder : R (1) r 1 V ( s 1 ) ECE 517: Reinforcemen Learning in AI 17

18 Forward View of TD(l) Look forward from each sae o deermine updae from fuure saes and rewards Q: Can his be pracically implemened? ECE 517: Reinforcemen Learning in AI 18

19 l-reurn on he Random Walk Same 19 sae random walk as before Q: Why do you hink inermediae values of l are bes? ECE 517: Reinforcemen Learning in AI 19

20 Backward View The forward view was heoreical The backward view is for pracical mechanism r 1 V (s 1 ) V (s ) Shou backwards over ime The srengh of your voice decreases wih emporal disance by l ECE 517: Reinforcemen Learning in AI 20

21 Backward View of TD(l) TD(l) paramerically shifs from TD o MC New variable called eligibiliy race On each sep, decay all races by l is he discoun rae and lis he Reurn weighing coefficien Incremen he race for he curren sae by 1 Accumulaing race is hus e (s) e (s) le 1 (s) le 1 (s) 1 if s s if s s ECE 517: Reinforcemen Learning in AI 21

22 On-line Tabular TD(l) Iniialize V ( s) arbirarily Repea (for each episode) : Iniialize s e( s) 0, for all s S Repea (for each sep of episode) : a acion given by for s Take acion a, observe reward, r, and 0 r V ( s) V ( s) e( s) e( s) 1 For all s : V ( s) V ( s) e( s) e( s) le( s) s s Unil s is erminal nex sae s ECE 517: Reinforcemen Learning in AI 22

23 Relaion of Backwards View o MC & TD(0) Using he updae rule: As before, if you se l o 0, you ge o TD(0) If you se l1 (no decay), you ge MC bu in a beer way V (s) e (s) Can apply TD(1) o coninuing asks Works incremenally and on-line (insead of waiing o he end of he episode) In beween earlier saes are given less credi for he TD Error ECE 517: Reinforcemen Learning in AI 23

24 On-line versus Off-line on Random Walk Same 19 sae random walk On-line performs beer over a broader range of parameers ECE 517: Reinforcemen Learning in AI 24

25 ECE 517: Reinforcemen Learning in AI 25 Conrol: Sarsa(l) Nex we wan o use ET for conrol, no jus predicion (i.e. esimaion of value funcions) Idea: we save eligibiliy for sae-acion pairs insead of jus saes oherwise a s e a a s s a s e a s e ), ( & if 1 ), ( ), ( 1 1 l l ), ( ), ( ), ( ), ( ), ( a s Q a s Q r a s e a s Q a s Q

26 Sarsa(l) Algorihm Iniialize Q(s,a) arbirarily Repea (for each episode) : e(s,a) 0, for all s,a Iniialize s,a Repea (for each sep of episode): Take acion a, observe r, s Choose a from s using policy derived from Q (e.g. - greedy) r Q( s, a ) Q(s,a) e(s,a) e(s,a) 1 For all s,a : 0 Q(s,a) Q(s,a) e(s,a) e(s,a) le(s,a) s s ;a a Unil s is erminal ECE 517: Reinforcemen Learning in AI 26

27 Implemening off-policy mehods wih ET {Q(l)} Two mehods have been proposed ha combine ET and Q-Learning: Wakins s Q(l) and Peng s Q(l) Recall ha Q-learning is an off-policy mehod Learns abou greedy policy while follows exploraory acions Suppose he agen follows he greedy policy for he firs wo seps, bu no on he hird Wakins: Zero ou eligibiliy race afer a non-greedy acion. Do max when backing up a firs non-greedy choice. r 2 n1 n 1 r 2 r 3 r n max Q ( sn, a) a ECE 517: Reinforcemen Learning in AI 27

28 Wakins s Q(l) Iniialize Q(s,a) arbirarily Repea (for each episode) : e(s,a) 0, for all s,a Iniialize s,a Repea (for each sep of episode): Take acion a, observe r, s Choose a from s using policy derived from Q (e.g. - greedy) a * argmax b Q( s,b) (if a ies for he max, hen a * r Q( s, a ) Q(s,a * ) e(s,a) e(s,a) 1 For all s,a : 0 Q(s,a) Q(s,a) e(s,a) If a a *, hen e(s,a) le(s,a) s s ;a a Unil s is erminal else e(s,a) 0 a ) ECE 517: Reinforcemen Learning in AI 28

29 Peng s Q(l) Disadvanage o Wakins s mehod: Early in learning, he eligibiliy race will be cu (zeroed ou), frequenly resuling in lile advanage o races Peng: Backup max acion excep a he end Never cu races Disadvanage: Complicaed o implemen ECE 517: Reinforcemen Learning in AI 29

30 Variable l ET mehods can improve by allowing l o change over ime Can generalize o variable l e (s) l e 1 (s) l e 1 (s) 1 if s s if s s l can be defined, for example, (as a funcion of ime) as l l(s ) or l l Saes visied wih high cerainy values l 0 Use ha value esimae fully and ignore subsequen saes Saes visied wih uncerainy of values l 1 Causes heir esimaed values o have lile effec on any updaes ECE 517: Reinforcemen Learning in AI 30

31 Conclusions Eligibiliy Traces offer an efficien, incremenal way o combine MC and TD Includes advanages of MC Can deal wih lack of Markov propery Consider an n-sep inerval for improved performance Includes advanages of TD Using TD error Boosrapping Can significanly speed learning Commonly used in pracice! Does have a cos in compuaion ECE 517: Reinforcemen Learning in AI 31

32 Forward View = Backward View The forward (heoreical) view of TD(l) is equivalen o he backward (mechanisic) view for off-line updaing The book shows (pp ): T 1 V TD (s) V l (s ) 0 T 1 0 I ss Backward updaes Forward updaes algebra shown in book T 1 V TD (s) I ss 0 T 1 0 T 1 k (l) k k T 1 V l (s )I ss I ss 0 T 1 0 T 1 k (l) k k On-line updaing wih small is similar ECE 517: Reinforcemen Learning in AI 32