Solving Continuous Action/State Problem in Q-Learning Using Extended Rule Based Fuzzy Inference Systems

Transcription

1 7 ICASE: The Insttute o Control, Automaton and Systems Engneers, KOREA Vol., No., September, Solvng Contnuous Acton/State Problem n Q-Learnng Usng Extended Rule Based Fuzzy Inerence Systems Mn-Soeng Km and Ju-Jang Lee Abstract: Q-learnng s a knd o renorcement learnng where the agent solves the gven task based on rewards receved rom the envronment. Most research done n the eld o Q-learnng has ocused on dscrete domans, although the envronment wth whch the agent must nteract s generally contnuous. Thus we need to devse some methods that enable Q-learnng to be applcable to the contnuous problem doman. In ths paper, an extended uzzy rule s proposed so that t can ncorporate Q-learnng. The nterpolaton technque, whch s wdely used n memory -based learnng, s adopted to represent the approprate Q value o r current state and acton par n each extended uzzy rule. The resultng structure based on the uzzy nerence system has the capablty o solvng the contnuous state and acton problem n Q-learnng. Also t can generate uzzy rules va nteractng wth the envronment wthout a pror knowledge about the envronment. The eectveness o the proposed structure s shown through smulaton on the cart-pole system. Keywords:Q-Learnng, renorcement learnng, uzzy logc controller I. Introducton Q-learnng s a knd o renorcement learnng where the agent solves the gven task based on rewards receved rom the envronment. In Q-learnng[], Q value whch can be seen as a qualty value or a certan acton n a certan state s generated and an agent can learn to select ts proper acton n each state based on these Q values. However, most research done n the eld o renorcement learnng has ocused on dscrete domans, although the envronment wth whch the agent must nteract s usually contnuous. Thus we need to devse some methods that enable Q-learnng to be applcable to the contnuous problem doman. Moreover, rom the control perspectve, t s natural that the agent wth contnuous acton can perorm better than the one wth lmted dscrete actons. There has been much research to make Q-learnng deal wth the contnuous state/acton spaces.[]-[] In these methods, however, the acton selecton method s based on a step search n acton space. That s, the Q value or all actons n the possble acton range s calculated and the acton o whch the Q value s the maxmum s selected to be perormed. The contnuous property o acton s dependent on the step sze. Moreover, the sequence o generated actons does not change contnuously, or smoothly. The contnuty o acton s mportant rom the practcal vew because the control nput cannot change rapdly by an actuator. Meanwhle, n the uzzy ne r- ence system (FIS), both the contnuous state and contnuous acton can be descrbed by a nte set o uzzy varables. Thus, the FIS s promsng as an eectve model to overcome the lmtatons o conventonal Q-learnng.[5]-[7] In ths paper, the FIS and Q-learnng are combned to resolve the contnuous state and acton problem n Q-learnng. A basc uzzy rule s extended so that Q-learnng can be ncorporated and the nterpolaton technque, whch s wdely used n memory -based learnng, s adopted to represent the approprate Q value or current state and current acton. The remander o ths paper s organzed as ollows. In Sec- Manuscrpt receved: Nov.,., Accepted: May. 5,. Mn-Soeng Km : Department o Electrcal Engneerng and Computer Scence, KAIST (mbella@cas.kast.ac.kr) Ju-Jang Lee: Department o Computer Scence and Electrcal Engneerng, KAIST (lee@ee.kast.ac.kr) ton, the basc uzzy rule and the conventonal Q-learnng algorthm are brely revewed. In Secton, the concept o extended rule s addressed. Also Q-value representaton scheme usng kernel uncton based nterpolaton s developed, whch enable the ntroducton o Q-learnng n FIS. The proposed structure and the learnng algorthm or ths structure are shown n Secton. In Secton 5, the smulaton results are presented to show the eectveness o the proposed methods. The concluson about the characterstcs o proposed methods s presented n Secton 6. II. Prelmnares. Fuzzy nerence system Fuzzy Inerence Systems (FIS) are expert systems based on -then rules, n whch p remses and conclusons are expressed by means o lngustc terms. The most mportant eatures o FIS are that they can ncorporate experts' pror knowledge nto ther parameter and that these parameters have clear physcal meanng. The FIS rule base s made o N derent rules o the ollowng general orm: R : s s A ands s A andsk s AM, then y s B k where s = [ s s sk] R s nput state varables, A ( where m =, M ) and B are uzzy membershps or each -th rule and characterzed by lngustc label (e.g., small, medum, etc ). These membershps are represented by a uncton µ A : s [,] and µ B : s [,], respectvely. A FIS are based on the uzzy rule base that conssts o several uzzy rules. A FIS usually has a uzzer that translates real-valued nput nto uzzy values and a deuzzer that translates uzzy output values nto real-valued output. Thus contnuous states or actons can be descrbed by a nte set o uzzy varables. Thus, a FIS seems to be an eectve model to overcome the lmtatons o conventonal Q-learnng, that s, contnuous state/acton problem. By combnng a FIS and Q- learnng, we can make Q-learnng able to deal wth contnuous states whle generatng contnuous actons. () m

2 Transacton on Control, Automaton, and Systems Engneerng Vol., No., September, 7. Q-learnng In Q-learnng, the agent selects an approprate acton or the current state usng the correspondng Q value o the current state and acton. The Q value s the mmedate reward by executng acton rom state plus the value o sum o expected uture reward ollowng the optmal polcy thereter. The basc Q-learnng algorthm s as ollows: Intalze all Q ( s, values to and observe the current state s Do orever Select an acton a and execute t Receve mmedate reward r Observe new state s Update Q ( s, value usng the ollowng equaton Q( s, Q( s, + α [ r + γ maxq( s, b) Q( s, ] () b A Replace s by s A detaled explanaton can be ound n [8]. III. Extended rule. Extended rule To ncorporate Q-learnng n FIS, an extended rule s proposed n ths paper. The proposed extended rule s derent rom the basc uzzy rule n that t has many canddates that can be selected to be an actual consequent part o the rule and has several characterstcs as ollows: - For one antecedentpart o each rule, several canddate consequent parts(actons) exst. - Each canddate acton has ts own Q value. - A rule acton s selected among canddate actons by usng polcy and each canddate's Q-value. Ths rule acton becomes an actual consequent part o the rule and nal acton s obtaned by deuzzcaton. Ths extended rule can be descrbed as ollows: R : s ands s L k s L ands M, then a k s L s π (u, Q ) where s = [ s s sk] R s nput state varables, L ( where m =, M ) s uzzy membershps related to each nput state. a s a -th rule acton whch corresponds to the actual consequent part o -th rule, R. π (, ) s a polcy that selects a rule acton rom dscrete acton set A. A s a collecton o an acton, u (where =, p ), whch s a canddate acton or the consequent part o -th rule. Every u has ts own Q value, Q. The superscrpt represents rule number and =, N. In each rule, one o the canddate actons s selected based on ther Q value and the polcy. Thereore the Q-learnng process must be ncorporated. A Rule's canddate acton s currently selected acton, among canddate consequent parts, to be an actual consequent part. These selected rule actons generate nal acton, a, by deuzzcaton. The nal acton s, however, generally derent rom each rule's rule acton. Because Q value s the unctonal value o state and acton, derent value o acton cannot be used n obtanng Q-value () m or current state and the nal acton. That s, we must have Q- value or the nal acton n each rule to obtan Q ( s, ) by deu zzcaton. Thus some knds o technque to obtan the Q value correspondng to nal acton n each rule have to be devsed, the nterpolaton technque s used to obtan a Q value or the nal acton n each rule n ths paper.. Q value representaton n the extended rule Once the nal acton, a, s determned rom the uzzy rule base, the Q(s, value or a certan acton a, whch s not ncluded n the dscrete acton set, can be calculated usng nterpolaton technque. In other words, Q value or the nal acton n R can be obtaned usng the ollowng equaton: p Q( R, ) = K ( u p, a ) Q () = K( u, a ) h where K (, ) s a kernel uncton whch determnes the degree o how each ( u, Q ) par n R contrbutes to calcu- latng Q R, a ) and has the ollowng orm: ( K( u, a ( a ) ) exp( u = ) (5) σ IV. Sel-organzng uzzy nerence system. The structure o SOFIS-Q In ths secton, the sel-organzng uzzy nerence system based on Q-learnng(SOFIS-Q) s ntroduced. The term, selorganzng, means that the consequent part o the uzzy rule s automatcally selected rom the dscrete acton set based on the polcy and on the Q value or each canddate acton. The whole structure s shown n Fg.. Ths structure perorms the uzzy nerence based on the uzzy rule base that conssts o extended rules n (). The nputs or the network are the states o the envronment or a plant. Outputs are the nal acton and the Q value or the nal acton and the current state (.e., Q ( s, ) ). The SOFIS-Q network conssts o total 5 layers. Layer to layer acheves uzzcaton n the nput state that resolves contnuous state representaton problem n Q-learnng. Layer to layer 5 generates contnuous acton by uzzy ne r- ence and also generates the assocated Q value or that nerred acton by uzzy nerence mxed wth the nterpolaton technque. Layer 5 Layer Layer Layer Layer u u u Q( s, u a R R ) u u u R u nal acton a u s k n u n u Fg.. The structure o SOFIS-Q. R n n u

3 7 ICASE: The Insttute o Control, Automaton and Systems Engneers, KOREA Vol., No., September, Layer The uncton o the rst layer s to receve the value o the state and transmt ths value to the next layer. Each node n ths layer corresponds to one nput state varable s ( =, k) Layer Ths layer conssts o several lngustc labels or each nput varable. Lngustc label L m n () has ts own membershp uncton that s denoted by µ L m ( ). Trangular and trapezo - dal membershp unctons are used n the paper. The output value rom layer ndcates a membershp degree o the current state. Layer One lnk rom a node n layer to the node n layer corresponds to one antecedent part o FIS. That s, t represents the part o each rule R n (). The value o each node n layer smply represents how much current state s belongs to the rule R and can be obtaned by uzzy and operator. Ths value represents the rng strength o R or the state s and denoted as α R. α R s obtaned usng the ollowng equaton. M α ( s) = µ ( s) (6) R m= where the uzzy and operator s mplemented by usng product operaton. Layer Layer corresponds to the then part o each uzzy rule. Based on Q values and polcy, one canddate acton rom the dscrete acton set s selected as a rule acton a n () and these rule actons rom each rule are combned to generate the nal acton. Lm a = π ( u, Q ) (7) where s the number o canddate actons and π (, ) represents the polcy used to select the acton. In ths paper, ε -greedy polcy [8] s used. Ths polcy selects an acton o whch Q value s the maxmum wth the probablty o ( ε ). Besdes the selecton o the rule acton, the Q value or the nal acton s calculated by () ter the nal acton s calculated n layer 5. Layer 5 In layer 5, the nal acton s obtaned by usng a weghted sum o each rule acton wth ther rng strength as ollows. N a = α a N = αrh Once a s obtaned, value o Q ( R, a ) s calculated or each rule usng (). Then, Q ( s, ) value can be obtaned usng the ollowng equaton. R (8). Learnng algorthm or SOFIS-Q The update o Q-value or each rule n SOFIS-Q s perormed through the change n the Q value o each canddate acton or each rule. To speed up the learnng, a replacng elgblty [9] s adopted. The elgblty s obtaned usng the ollowng equaton. e e αr = N α Rk k= e K ( u, a ) p K( u, a ) h R / u par s selected. = λγ otherwse () where λ s an elgblty rate whch s used to weght each (rule,rule acton)-par accordng to ther proxmty to the occurrng tme step rom the current state. The learnng procedure s shown below:. Intalze all Q values o all canddate actons n each rule to zero. Perceve the current state s curr and generate nal acton a curr by (8). Calculate Q ( s curr, a ) by (9) and calculate elg blty curr traces by (). Perorm a curr, receve reward r next and transton to the next state s next. 5. Obtan Q ( s next, a next ) by (9) and calculate temporal error ε. ε = r + γq( s, a ) Q( s, a ) next next next curr curr 6. For each canddate acton o every rule, update the Q value as Q curr 7. I tral s not ended, set = Q curr +αεe a = a and goto step. curr next V. Smulaton. Cart-pole balancng problem The cart -pole balancng problem s concerned wth how to balance an uprght pole. The pole has only one degree o reedom and the prmary control task s to keep the pole vertcally balanced whle keepng the cart wthn the ral track boundares. Four state varables are used n descrbng ths system and one varable represents the orce appled to the cart. These are: θ : the angle o the pole rom uprght poston(radan); θ : the angular velocty o the pole(radan/seconds); x : the poston o the cart's center (meters); ẋ : the velocty o the cart (meter/seconds); : orce appled to the cart (Newton) The model dynamcs and correspondng parameters can be ound n []. Euler method s used or smulaton usng a tme step o.s. The constrants on the varables are θ, x.m, and N. N Q( s, ) = α N = α Rh Q( R, a ) R (9) It s assumed that the dynamcs o the system s unknown to the controller. The controller can be normed o only the values o the state varables and the reward sgnal at each tme step. The only avalable eedback or SOFIS-Q s alure sgnal when θ > or x >. m. The reward sgnal gven to

4 Transacton on Control, Automaton, and Systems Engneerng Vol., No., September, 7 the controller s as ollows: θ > or x = >.m r otherwse The smulatons were done wth varous numbers o canddate actons. The parameter values used or Q-learnng are as ollows: γ s.9, α s. and λ s.7.. Smulaton results Fgure and Fg. show learnng curves or the case where two canddate actons and ve canddate actons, wth zero ntal condton, are used, respectvely. The curve conssts o ten consecutve runs where a run conssts o several trals. Each tral ends n two cases. Frst, the pole has allen or the cart poston exceeds the lmts o the ral track. Second, the tme step exceeds. -tme step corresponds to seconds n smulaton tme. Fnally, rom Fg. to Fg. 6, shown are the resultng traectores o θ, x and or learned controller wth 5 dscrete canddate actons. These traectores are selected as a sample rom many results. The proposed SOFIS-Q can learn to acheve the gven task only wth the bnary reward sgnal.. Comparson study There were ew approaches that can resolve contnuous state and acton problem smultaneously wth Q-learnng. Snce the ocus o ths paper s on whether the agent can generate contnuous acton or not. Thus, we compare the results n the vew o exerted control eort. Fg. 8 shows the results rom the obtaned data by the one-step search based approach, n whch the control s selected by ncreasng control value wth a nte step sze and comparng the resultng Q value or each control value. Although the approaches based on the one Fg.. Learnng curve wth canddate actons trals trals Fg.. Learnng curve wth 5 canddate actons. trals Table. Average learnng speed. Method Average tral Conventonal Q-learnng 5 Fuzzy nterpolaton based Q-learnng 96 Sel-organzng modular neural network 5 The proposed method wth canddate actons 7 The proposed method wth 5 canddate actons 9 è θ (deg) 5 (deg) X x (meter) (meter) control - - Fg.. Traectory o pole angle Fg. 5. Traectory o cart poston Fg. 6. Generated control orce.

5 7 ICASE: The Insttute o Control, Automaton and Systems Engneers, KOREA Vol., No., September, step search mght be derent accordng to each method, t can be sad that, because selecton o control acton s based upon the one-step search, the resultng control eort has smlar characterstcs or the most algorthm. Thus we here adopt the one-step search algorthm whch uses the uzzy network as an approxmator o Q(s, values and used the same learnng parameters and nput space partton. From the Fgures, we can drectly see the advantage o ablty that can generate contnuous acton. The control traectory o Fg. 8 s very derent rom that o Fg. 7. For the one-step search method, we can make a smoother control by reducng the step sze used n searchng. Ths reducng n step sze, however, means more computaton tme. Thus we must trade o between the contnuty and the computaton tme wth one-step search based algorthm. More mportantly, the resultng control by the onestep search method s not truly contnuous n that ts value can ump rom one acton value to another value dscontnuously. However, n SOFIS-Q, the generated control s nherently contnuous. control control Fg. 7. Generated control orce wth the proposed method. control Fg. 8. Generated control orce wth one-step search method. VI. Concluson In ths paper, the sel-organzng uzzy nerence system by Q-learnng s proposed. The purpose o ths SOFIS-Q s to cope wth contnuous states whle generatng contnuous actons n Q-learnng. By methods based on uzzcaton, the contnuous nput state problem s easly resolved. By Usng deuzzcaton, the contnuous acton problem s also resolved. By the extended rule and the nterpolaton technque, the SOFIS-Q behaves as a uzzy nerence system whle approxmatng the Q(s, values. Several smulatons or a pole - balancng problem were conducted to show the eectveness o the proposed structure. To show the advantage o the contnuous acton, the comparson wth the one step search based method was shown and the learnng speed was compared. By ncorporatng the uzzy nerence system wth Q-learnng, wthout the model and knowledge about the plant, the SOFIS- Q can adust descendant part o a FIS automatcally. In SOFIS-Q, there are several canddate actons n each rule, but what number s the optmal one or the problem can not be determned. There should be some prunng or growng methods that automatcally ncrease or decrease the number o canddate actons. Smlarly where to locate the canddate acton n acton space can be a problem accordng to the gven task. Thus research on sel-locatng the canddate acton should be made. Reerences [] C. J. C. H. Watkns and P. Dayan, Techncal note: Q- learnng, Machne Learnng, vol. 8, pp. 79-9, 99. [] T. Horuch, A. Funo, O. Kata, and T. Sawarag, Fuzzy nterpolaton based Q-learnng wth contnuous states and acton, Proc. FUZZ-IEEE'96, 5th Int. Con. Fuzzy Systems., New Orleans, pp. 59-6, 996. [] J. H. Km, I. H. Suh, S. R. Oh, Y. J. Cho, and Y. K. Chung, Regon-based Q-learnng usng convex clusterng approach, Proc. IROS 97, pp. 6-67, 997. [] J. C. Santamara, R. S. Sutton, and A. Ram, Experments wth renorcement learnng n problems wth contnuous state and acton spaces, CONIS Techncal Report 96-88, Unversty o Massachusetts, December, 996. [5] A. Bonarn, Delayed renorcement, uzzy {Q}-learnng and uzzy logc controllers, Proc. Herrera, pp. 7-66, 996. [6] P. Y. Glorennec, Fuzzy Q-learnng and dynamcal uzzy Q-learnng, Proc. FUZZ-IEEE'9, rd Int. Con. Fuzzy Systems., Orlando, FL, vol., pp. 7-79, 99. [7] K.B. Sm, Fuzzy nerence-based renorcement learnng o dynamc recurrent neural networks, SICE Annual Conerence o Japan(Internatonal sesson), Tokushma, Japan, pp. 8-88, 997. [8] Rchard S. Sutton and Andrew G. Barto, Renorcement Learnng: An Introducton, MIT Press, Camb rdge, MA, 998. [9] S. P. Sngh, R. S. Sutton, Renorcement learnng wth replacng elgblty traces, Machne Learnng, vol., pp. -58, 996. [] C. T. Ln and C. S. G. Lee, Renorcement structure / parameter learnng or neural-network-based uzzy logc control systems, IEEE Transactons on Fuzzy Systems, vol., no., pp. 6-6, February, 99. [] S. G. Hong, M. S. Km, W. Km, and J. J. Lee, Selorganzng modular neural network or renorcement learnng problem, Proceedngs o the ICMT' 99, Oct. -, pp , Pusan, Korea.

6 Transacton on Control, Automaton, and Systems Engneerng Vol., No., September, 75 Mn-Soeng Km He receved the B.S. degree rom Hanyang unversty n 997, and M. S. degree rom KAIST n 999. He s currently workng toward the Ph.D. degree n electrcal engneerng at KAIST. Hs research nterests are artcal ntellgence, renorcement learnng and moble robots. Ju-Jang Lee He was born n Seoul, Korea, on November, 98. He receved the B.S. and M.S. degrees n electrcal engneerng rom Seoul Natonal Unversty n 97 and 977, respectvely, and the Ph.D. degree n electrcal engneerng rom the unversty o Wsconsn n 98. He s a proessor n the department o electrcal engneerng, KAIST, whch he oned 98. Hs research nterests are n the areas o space robotcs, varable structure systems, ntellgent moble robot control, ITS and robust control theory.