EFFECT OF REWARD PREDICTION ERRORS ON THE EMOTIONAL STATE OF A MOBILE ROBOT

Transcription

1 In D. Reier & F. E. Rier (Eds.), Proceedings of he 14h Inernaional Conference on Cogniive Modeling (ICCM 16). Universiy Park, PA: Penn Sae. EFFECT OF REWARD PREDICTION ERRORS ON THE EMOTIONAL STATE OF A MOBILE ROBOT Vidullan Surendran (vus133@psu.edu) The Pennsylvania Sae Universiy, 234 Hammond Bldg. Universiy Park, PA USA Lyle N. Long (lnl@psu.edu) The Pennsylvania Sae Universiy, 233 Hammond Bldg. Universiy Park, PA USA Absrac A goal-based dynamic acion selecion mechanism incorporaing a model for emoions and emperamen was developed for use wih small and inexpensive mobile robos. A mobile robo was developed o es he acion selecion mechanism by recreaing he scenario of an animal foraging for food while avoiding predaors. Four emoions of anger, fear, happiness, and surprise were modelled which were affeced by evens such as finding food, encounering a predaor, encounering a boundary wall, finding a safe area, and being in a sae of low healh. The model incorporaed a reward predicion module ha alered he effec of an even based on he error beween when an even occurred and when i was prediced o occur. The model also included a decay erm ha resuled in he emoions reurning o heir seady sae values unless here was coninual reinforcemen hrough he occurrence of evens. The effec of differing emperamens on he emoions was sudied by defining an irae emperamen. Keywords: Emoions; emperamen; affecive compuing; roboics; acion selecion. Inroducion Affecive compuing seeks o undersand and develop sysems ha can recognize and simulae human emoions. R.W. Picard (00) highlighed he imporance of he field by exploring neurological sudies ha indicaed human cogniion was inrinsically linked wih emoions. She also argues ha ha he developmen of affecive compuing is criical o advancing emoion and cogniion heory. Breazeal and Brooks (05) considered cogniion and emoions o be wo disinc sysems ha evolved in inelligen creaures under social and environmenal processes o aid in opimal funcioning. Cogniion is deemed o be responsible for inerpreing he world whereas emoions are deemed o be responsible for evaluaing he value of evens. Emoions hus help prioriize concerns while minimizing disracions. The simulaion of human emoions is grealy complicaed by he fac ha here is no acceped model ha explains and predics he wide range of emoions we experience. There sill is no consensus in he lieraure on he number of base emoions. Paul Ekman (1999) proposes a lis of 15 emoions, each represening a family of emoions. A sudy on dynamic facial expressions of emoion by Jack e al. (15) challenges his noion by suggesing ha basic emoion communicaion comprises fewer caegories. I is clear ha our undersanding of he subjec is sill in is infancy. A sudy on momenary subjecive well-being by Ruledge e al. (14) resuled in a model of happiness referred o as he Happiness Equaion in popular culure. They showed ha momenary happiness in response o a probabilisic reward is explained by he combined influence of he reward expecaions and he predicion errors from he expecaions; no by curren ask earnings as one would naively conclude. Long e al. (15) and Long (15) adaped his model o simulae he eigh universal emoions of fear, anger, sadness, happiness, disgus, surprise, rus, and ineres in cogniive mobile robos. The emoion and emperamen engine hey developed was incorporaed ino SS-RICS which is a cogniive archiecure developed a he Army Research Laboraory (Troy D. Kelley, 06). Surendran (15) buil upon his emoional model by incorporaing a reward predicion error componen wih a focus on developing a model ha could be execued wih limied compuaional resources. This model did no use SS-RICS and implemened a new acion selecion mechanism (ASM) capable of running on a small mobile robo wih a Raspberry Pi processor. Tes Seup A clien-server archiecure was used wih an auonomous agen acing as he clien, and a compuer running he ASM and affecive model acing as he server. A roboic plaform was developed specifically for his sudy wih an emphasis on a form facor under 0 cm 2 and reduced cos. I was based on he Raspberry Pi microprocessor and is capable of image processing a an average of 5 frames a second, orienaion sensing, collision deecion hrough infrared sensors, and baery operaion for an hour. The behavior and emoional sae changes of a small roden foraging for food while avoiding predaors was chosen o be simulaed. Differenly colored balls having a diameer of 10 cenimeers were used o represen food sources (green), predaors (red), and safe areas (purple). The agen was placed inside an enclosed space conaining he hree ypes of balls which were randomly placed. One of he purple objecs represening a safe area was chosen as he saring posiion of he robo. The behavior ha was simulaed can be described as follows. The mobile robo ress a a safe area unil is healh 151

2 decays below a hreshold level riggering a search for food; his simulaes hunger. The robo hen searches he surrounding area unil i idenifies a green ball which i racks owards. I hen consumes he food by being in close proximiy (wihin 5 cenimeers) of he ball. The saiaion of hunger is simulaed as a ime dependen increase in he robo s healh for as long as i says in close proximiy o he objec. When is healh has been compleely recharged i hen seeks ou a safe area o res a, unil is healh drops below he hreshold riggering a repea of he cycle. Throughou he simulaion, he mobile robo consanly avoids red danger balls and collisions wih he boundary walls. Whenever a danger is idenified, he robo abandons is curren ask and insead akes evasive manoeuvers o avoid he danger. Only saionary predaors were considered in his sudy. A linear decrease in healh wih respec o ime was assumed in his sudy. I is o be noed ha he framework allows one o implemen more complex models of healh and consumpion. Acion Selecion Mechanism (ASM) In order for he sysem o be auonomous i had o decide wha o do. In order for i o be successful i had o inelligenly selec appropriae acions based on exernal and inernal simuli. Dynamic planning mehods compue he nex acion o be aken based on he curren inernal and exernal sae. This ype of ASM is ideal when limied compuaional resources are available. On he oher hand, o replicae he behavior required a goal driven archiecure was used (Brom & Bryson, 15). To his end a new hybrid archiecure was implemened in he final sysem. The behavior implemened was decomposed ino goals and subgoals. Goals were namely finding food, finding a safe area, avoiding danger and resing. These goals were hen decomposed ino subgoals such as finding a ball, racking a ball, eaing a ball, avoiding a ball, ec. Each of hese subgoals was associaed wih a dynamic plan ha he ASM selecs in order o accomplish he goal. This modular srucure permied reuse of plans as he goals had similar subgoals. Condiion-acion rules similar o hose used in exper sysems were used o implemen he dynamic plans. Rules defined he knowledge he sysem possessed and in his sysem are eiher facual or procedural. Facual knowledge as he name suggess consiss of facs such as he color of a ball defining if i relaes o food, danger, or a safe area. Procedural rules govern how acions he robo ake will be carried ou. Conflic resoluion in case of compeing goals is handled using prioriies wih cerain goals having a higher prioriy. For example, if a danger ball is seen in fron of an objecive ball, he robo would avoid he danger insead of moving owards he objecive. An even handling funcion and an inerrup mechanism implemened in sofware handle flow conrol. The even handler execues he appropriae plan based on he curren goal, subgoal, and sysem sae. The inerrup handler allows he currenly execuing goal o be paused when a goal wih a higher prioriy is o be execued. Once complee, i reloads he previous goal and subgoal. Time Sep Each ime sep is defined o be one cycle of he mobile robo sending daa o he command cener and receiving conrol commands. This was chosen over ime as hardware limiaions prevened real ime image processing and consequenly governed he rae of daa ransmission. This ensured consisency as oherwise a slower sysem would experience quicker decays in healh and emoions due o processing ime. If enough processing power is available, he framework allows clock ime o be used insead. Emoions Engine The compuaional model obained by Ruledge e al. (14) is shown in Equaion (1). In he model CR represens cerain rewards, EV heir expeced value and RPE he reward predicion error. Happiness() = w 0 + w 1 γ j CR j + w 2 γ j EV j + w 3 γ j RPE j Long e al. (15) modified his model as shown in Equaion (2) o incorporae emoions and emperamens ino cogniive mobile robos. The winner-ake-all approach hey implemened mean ha he emoion wih he highes value was considered as he robo s emoional sae. In his model R ij + represens posiive reinforcemens while R ij represens negaive reinforcemens. Emoions() i = w 0i + γ i j (w 1 i R ij + + w 2 i R ij ) The emoional model used in his research sudy is shown in Equaion (3) and incorporaes he RPE erm from equaion (1) ino equaion (2). Emoion() i = w 0i + γ i ( f j ) w1 i R j n n + γ i ( f j ) w2 i RPE j The posiive or negaive effec of an even on he emoion is represened by he erm R j. The reward predicion error is represened by he erm RPE. w 0, w 1, and w 2 are weighing facors. γ is a decay facor ha governs he impac of pas evens on he curren emoional sae. Togeher hese values define he emperamen of an agen. (1) (2) (3) 152

3 Emoions and Temperamen Consans A disincion has o be made beween emperamens and emoional saes. Temperamens are considered o be biologically based and derived from geneic predisposiions, mauraion, and experience. They are expeced o be relaively sable over ime. Emoions from a funcionalis approach are defined as a person s readiness o esablish, mainain or change one s relaionship o his or her changing circumsances. In conras o emperamenal variabiliy, emoional reacions can be enduring or brief (Thompson & Winer, 15). Four emoional saes of anger, fear, happiness, and surprise deemed o mos likely be affeced by he es scenario were modelled in his sudy. Each emoion is assigned weighing facors and a decay facor ha correspond o Equaion (3) as shown in Table 1. To beer illusrae he sysem, he weighing facors were experimenally seleced o resul in significan changes in emoional values wihin a range of ±50. The seady sae value (w 0 ) of he emoions was se as zero for he same reason. Fuure work will address how o opimize hese consans o make he robo as effecive as possible. Table 1: Emoion consans Emoion w 0 w 1 w 2 γ Anger Fear Happiness Surprise Table 2: Emoion modifiers Anger Fear Happiness Surprise Danger encounered Found Food Reurned o safe area Wall encounered Healh oo low Componens The emoions engine is made up of hree major componens; shor erm memory, he RPE module, and he command modifier. Shor Term Memory The emoional sae of an agen is deemed o depend on is inernal sae and exernal simuli such as evens. Evens are defined as ineracions wih he environmen ha resul in a change of he ASM goal or subgoal. Table 2 liss all defined evens along wih heir reward values. The emoions engine records regisered evens, he reward value for he even, and he ime seps aken o complee he ASM goal ha led o i. Due o he decay componen (γ) of he emoional model, i was found ha a memory lengh of six evens was opimal as he exponenial decay mean ha he value of any previous evens was negligibly small. Reward Predicion Error (RPE) Module Momenary subjecive well-being was found o depend no only on an even bu errors in predicing he occurrence of said even. Unexpeced evens have a higher impac on he value of an emoion (Ruledge e al., 14). Due o a lack of long erm memory and learning capabiliies, he average number of ime seps aken for an even o occur in he pas is used o predic fuure expecaions. The difference beween his predicion and he acual ime aken for he even o occur is considered o be he RPE. A he sar of an agen s life each ype of evens is prediced o ake 25 ime seps o occur. An excepion is he Healh oo low even which is considered o ake 300 ime seps o occur. These values were chosen as hey were found o be he average number of ime seps aken in he majoriy of ess. As evens of a ype occur during operaion, he predicion for ha even is updaed. Command Modifier Daa sored by he shor erm memory module is sen o he RPE module o calculae he RPE. Using Equaion (3) he insananeous value of he four emoions are hen calculaed and in combinaion define he emoional sae of he robo. These calculaions are carried ou every ime sep regardless of an even being regisered. The command modifier allows he emoions engine o modify he ASM commands generaed and he inernal sae of he agen, based on he curren emoional sae. Condiional logic and saemens are used for his purpose. In his sudy we have chosen o apply he following modifiers, Speed = curren speed + value a + value f value h Where value a, value f, and value h denoe he numeric values of he emoions anger, fear, and happiness respecively. This modifier increases he speed of he robo when he value of anger or fear increases and decreases i when happiness increases. This shows how all emoions can compee agains each oher o affec behavior. if speed < 90 hen speed = 90 This modifier ses a lower bound on he speed of he robo. if value f > hen randomly sop he robo s moion This simulaes a imid robo by randomly sending i sop commands if he value of fear becomes greaer han. The is similar o he endency of an animal o momenarily freeze when frighened. if surprise > hen make a 530 urn This simulaes a surprised robo by making i urn around if he value of surprise ges larger han. This is similar o an animal being sarled and a robo spinning around was no only amusing, bu easily observable. 153

4 Resuls The effecs of he differen ype of evens, he RPE module, and varying emperamens were considered. The es scenario picured in Figure 1 consised of wo safe balls, hree food balls, and hree danger balls. I was run for 300 ime seps. Evens are signified on he plos using he leers shown in Table 3. A negaive RPE denoes an unexpeced even while a posiive RPE denoes an even ha was prediced o occur earlier han observed. Table 4 liss all evens ha were regisered along wih heir ime seps and he calculaed RPE. Table 3: Evens and heir signifiers Even Danger encounered Found Food Reurned o safe area Wall encounered Healh oo low Figure 1: Tes Scenario Denoed by a f b w h Table 4 Time seps, Even ype, and RPE value of all evens Sep Even f f f w w w w RPE Sep Even a w w a w w w RPE Sep Even b w w w f w f RPE Sep Even w f w f f w RPE Effec of Evens on he Emoional Sae Figure 1 shows he variaion in he individual emoions whereas Figure 2 shows he variaion in he robo s speed and is healh. From he daa we can infer ha, once he healh dropped below he hreshold of 75 a ime sep 10, he agen began searching for food and found i a ime sep 22, ha is in 12 seps. The RPE led o a larger spike in happiness and a large reducion in robo speed. During is search for a safe area i had o avoid he boundary walls, food balls, and he danger balls each of which affeced he emoions as seen in he previous scenarios. The robo finally found a safe area a ime sep 214. The iniial predicion by he RPE module was ha a safe area would be reached in 25 ime seps. In his case since ha led o an RPE of 164, insead of an increase in happiness and reducion in anger, surprise, and fear, we see he opposie effec. Since he robo ook a large number of ime seps o reurn o a safe area, when i did is healh had dropped below he hreshold of 75. This caused i o immediaely sar searching for a food ball. A approximaely ime sep 241 here was an anomaly where he robo incorrecly idenified a food ball as a boundary wall momenarily before correcly recognizing i as a food ball. Effec of he RPE Module Figure 3 shows he effec of he RPE on he emoional sae value of anger. Using Table 4 we sudy some key evens ha illusrae he effec of he RPE module: Food even a ime sep 22 (RPE of -13) This even happened sooner han prediced by 13 ime seps. The effec of he RPE module was a greaer reducion in anger han would have been wihou i. Wall even a ime sep 35 (RPE of +10) This even happened laer han prediced by 10 ime seps and hus is impac was reduced. This is seen as anger values no being as large as hey would have wihou he RPE module. Wall even a ime sep 43, 51, 59 (RPE of -22, -11, -5) We observe ha he impac of he wall a sep 43 is greaer han ha a sep 51 and so on. When he RPE module was disabled, he impac of all 3 wall evens was exacly he same. Safe area ball a ime sep 214 (RPE of 164) The exremely large predicion error mean ha insead of a decrease in anger ha would have been expeced when he robo reurned o a safe area, here was an increase in anger. Basically he opposie of he effec he even would have had on he emoions had he RPE module been disabled. We can hus conclude ha he RPE module is crucial in modelling he psychological effec of expecaions governing he effec of an even. Wihou i, an even would affec he emoional sae exacly he same way regardless of when i occurs. Effec of Varying Temperamens Temperamens are defined by he weighing facors shown in Table

5 Value Variaion in Emoions Due o Evens Time Seps Figure 1: Shows he variaion in he values of he four emoions Evens Anger Fear Happiness Value Robo Sae Variables wih Evens Time Seps Figure 2: Shows he variaion in he speed and healh of he robo for a es scenario Evens Speed Healh Value Variaion of Anger Time Seps Figure 3: Compares he variaion of anger wih and wihou he RPE module Evens Wih RPE Wihou RPE Value Variaion wih Temperamens Time Seps Figure 4: Compares he value of anger for an irae and a neural emperamen Evens Neural Angry Irae 155

6 Figure 4 shows he change in he value of he emoion anger for an irae emperamen which was defined by modifying he weighing facors affecing he emoion anger. Irae: The weighing facors used for anger were: Emoion w 0 w 1 w 2 γ Anger We observe ha he seady sae value of he affeced emoion is offse from 0 by he value of he weighing facor w 0. The facor w 1 conrols he insananeous effec of an even on he emoions. As γ is increased we see ha an emoion akes longer o decay o is seady sae value. Thus an irae emperamen causes he robo o have a higher seady sae value of anger and remain angry for longer once an even angers i. Conclusions By adaping a model for momenary well-being (Ruledge e al., 14) as done by Long e al. (15), and incorporaing a reward predicion error, an acion selecion mechanism wih an inegraed emoions engine has been implemened in addiion o he developmen of a low cos robo o es he mechanism (Surendan, 15). The ASM developed was a hybrid of a goal-based and a dynamic planning acion selecion mechanism. Each goal was associaed wih subgoals, and each subgoal wih a dynamic plan. Differen evens were defined ha affeced he individual emoions and he ASM provided means by which he emoional sae could be used o modify he inernal sae of he robo hrough means of command modifiers. Command modifiers also allowed he modificaion of dynamic plans and he value of emoions o be coupled hrough suiable models. Temperamens and emoional variabiliy were defined using a marix of consans. Varying he emperamens was observed o resul in a differen emoional sae over he ime period of he experimen even when he scenario was kep consan. Finally, he imporance of he reward predicion error was highlighed by showing ha wihou i evens affeced he emoions by he same amoun regardless of when hey occurred. Wih he RPE, i was possible o mimic he emoional response observed in humans by Ruledge e al (14). Addiional ess should be conduced by specifying a more comprehensive lis of command modifiers and fine uning he emperamen values based on he observaion of an animal foraging for food in he wild. Since he emperamen is specified by means of consans shown in Table 1, his leads o he possibiliy of an agen wih a ime-varian emperamen ha can aler is emoional sensiiviy during run ime. The emoional model could also be modified o inroduce a ime lag beween he occurrence of an even and i affecing he emoional sae. This would allow he simulaion of he four classic emperamens, namely, melancholic, phlegmaic, choleric, and sanguine heorized by Greek philosophers ( Four Humors - And here s he humor of i: Shakespeare and he four humors, 12). Anoher approach would be he specificaion of he emperamen in erms of he Big 5 rais of exraversion, agreeableness, openness, conscieniousness, and neuroicism ofen described in he lieraure (Digman, 1990). The low cos of he robo also permis he acquisiion of a non-homogeneous robo swarm wih varying emperamens o explore if emoions increase he effeciveness of he swarm. References Brom, C., & Bryson, J. (06). Acion selecion for Inelligen Sysems. European Nework for he Advancemen of Arificial Cogniive Sysems. Breazeal, C. & Brooks, R. (04). Robo Emoion: A Funcional Perspecive. In Who Needs Emoions, J. Fellous (Ed.), Oxford Universiy Press. Digman, J. M. (1990). Personaliy Srucure: Emergence of he Five-Facor Model. Annual Review of Psychology, 41(1), Ekman, P. (1999). Basic emoions. In Handbook of cogniion and emoion. Wiley & Sons Ld. Four Humors - And here's he humor of i: Shakespeare and he four humors. (12, January 30). Rerieved from hps:// s.hml. Jack, R. E., Garrod, O. G. B., & Schyns, P. G. (15). Dynamic Facial Expressions of Emoion Transmi an Evolving Hierarchy of Signals over Time. Curren Biology, 24(2), Long, L. N. (15). Modeling Emoion and Temperamen on Cogniive Mobile Robos. Presened a he 22nd Annual ACT-R Workshop. Pisburgh PA: Carnegie Mellon Universiy. Long, L. N., Kelley, T. D., & Avery, E. S. (15). An Emoion and Temperamen Model for Cogniive Mobile Robos. Presened a he 24h Conference on Behavior Represenaion in Modeling and Simulaion (BRIMS), Washingon, DC. Thompson, R. A., & Winer, A. C. (15). The individual child: emperamen, emoion, self, and personaliy. In Developmenal science: An advanced exbook, Ediion: 6. Psychology Press / Taylor & Francis. Ruledge, R. B., Skandali, N., Dayan, P., & Dolan, R. J. (14). A compuaional and neural model of momenary subjecive well-being. Proceedings of he Naional Academy of Sciences, 111(33), Picard, R. W. (00). Affecive Compuing. Cambridge, MA: The MIT Press. Kelley, T. D. (06). Developing a psychologically inspired cogniive archiecure for roboic conrol: The symbolic and subsymbolic roboic inelligence conrol sysem (SS-RICS). Advanced Roboic Sysems, 3(3), Surendran, V. (15). A dynamic acion selecion mechanism for an auonomous agen wih a model for is emoional sae. M.S. hesis, Aerospace Engineering, The Pennsylvania Sae Universiy, Sae College, PA. 156