Effective Emotional Model of Pet-type Robot in Interactive Emotion Communication

Transcription

1 SCIS & ISIS 200, Dec. 8-2, 200, Okayama Convention Center, Okayama, Japan Effective Emotional Model of Pet-type Robot in Interactive Emotion Communication Ryohei Taki, Yoichiro Maeda and Yasutake Takahashi Dept. of Human and Artificial Intelligent Systems Graduate School of Engineering, University of Fukui 3-9- Bunkyo, Fukui, Japan (rtaki, maeda, Abstract In order for the robot and human to live together, the flexible understanding ability of human intention is required for the robot. In this research, we aim to realize Interactive Emotion Communication (IEC) which is a bidirectional communication based on emotional behaviors between human and robot. The purpose of IEC is to raise the personal affinity which the robot gives to the human by interactive emotional behaviors between both of them. IEC consists of three processes: () recognizing human emotion, (2) generating robot emotion, and (3) expressing robot emotion. In this research, we mainly developed the process of generating robot emotion. We consider that patterns of emotional behaviors desired to the robot vary from person to person in IEC. In this research, we perform the individual preference analysis for emotional behaviors. I. INTRODUCTION Recently, opportunities when a robot contacts human are increasing [], [2], therefore the technology for interactive communication with human is gradually needed. In addition, the flexible understanding ability of human intension and the expressing ability of robot intension are required for the robot to live together. Technology for realizing the interactive communication between human and robot has not been established yet, therefore there are few robots which communicate to human smoothly. In order to understand the human intention and express the robot intention, some researches used the nonverbal information have been proposed [3 7]. If there are difference between the verbal communication and the nonverbal communication when we convey our emotion and attitude, nonverbal communication includes over 90% information for the emotion of interlocutor [8]. There are various kinds of nonverbal communication that is eye sign, voice, expression, gesture, and so on. On the other hand, we feel the unpleasantness for the robot expression like human [9]. We have been performed the research on the nonverbal communication based on human and robot behaviors. In this research, a method of emotion inference from the behavior of human is proposed [0]. At first, the body feature of a subject is extracted based on the Laban s theory []. Next, we obtain the basic emotional degree [0] by fuzzy inference using extracted human body feature. Finally, the emotion value of human behavior is evaluated based on the Russell s Circumplex Model [2]. We describe the outline of these theories in section III. In this research, we aim to realize Interactive Emotion Communication (we call IEC) which is a bidirectional communication based on emotional behavior between human and robot. As a result of the analysis we aim to give high interpersonal affinity of robot to human. Moreover, we report on the image of communication between human and robot. We can conjecture that patterns of emotional behaviors demanded from robot are different by person in IEC. This research experiments individual preference analysis for emotional behavior each of subject. II. INTERACTIVE EMOTION COMMUNICATION (IEC) This research assumes the bidirectional communication model through emotional behaviors between human and robot as one example of several nonverbal communications. The emotional behavior means the gesture or dance [3] to represent emotion to an opposite person. We assume that there are a human A and a robot B here. First of all, human A generates something an emotion and expresses the emotional behavior to robot B by his gesture (Step). Next, B recognizes A s emotion through his vision ability (Step2). Then B generates his emotion and expresses emotional behaviors to A (Step3). Moreover A is cheer up by B (Step4). In this way, the interactive communication by emotional behaviors is constructed between A and B. To realize IEC, the following three processes become necessary. ) Recognizing human s emotion 2) Generating robot s emotion 3) Expressing robot s emotion We call this bidirectional communication Interactive Emotion Communication (IEC). The human communicates through languages, but it is very hard for the robot to communicate with the human by recent technology. Therefore we consider that the human and robot enable to communicate smoothly through emotions. In the process of generating robot s emotion in this paper, the robot emotional behavior is decided based on the evaluation of the personal preference analysis after section IV. Fig. shows the conceptual drawing of IEC. In this research we proposed a new trial of the communication between an ability of the proposed method. Our final goal is 99

2 A Step I'm sad. I became happy. Step4 Fig.. [Recognizing Emotion] Interactive Emotion Communication Step2 He seems to be sad. B [Generating Emotion] I will make him happy. [Expressing Emotion] Step3 Interactive Emotion Communication (IEC) to build the robot which is able to recognize human s emotion and express its emotion by bidirectional communication based on IEC model with high interpersonal affinity. III. FUZZY EMOTION INFERENCE SYSTEM (FEIS) Mainly in this chapter, we explain the first one of abovementioned IEC recognizing human s emotion process which inferences human s emotion from emotional behaviors of human. We used the fuzzy inference to infer the human emotion from the human body feature. We adapted Laban s theory and Russell s Circumplex Model to decrease the input and output dimension in the fuzzy inference and make the fuzzy rules simply. A. Laban s Theory Laban s theory [] proposed by R. Laban is a method to extract the macro features from human body motions. This method has three types of description about motion features, that is, Effort-Shape Description, Motif Description and Structural Description. Above all Effort-Shape Description describes the quality of motions and meaning of the expression. Because it is useful when human body motion is classified according to the visual function, we use this type of description to analyze human body motions. R. Laban proposed the theory that there are bipolar systems based on Fighting Form and Indulging Form in the expression of human body motion. Fighting Form means active and vivid body motion and Indulging Form means slow and gentle body motion. The concept that human motions are subdivided by these forms is Effort-Shape Description. Effort is effective to classify the body motion based on Kansei information. Shape shows the feature of overall static shape of the body motion, moreover Shape do not include considering local motion feature. In this research, we suppose that Time Effort is the speed of the center of gravity of body, Flow Effort is the acceleration of hands, Table-Plane Shape is the area of body and Door-Plane Shape is the height of the center of gravity of body. These features could be measured by a camera of robot. We excluded Weight Effort, Space Effort, Wheel Plane from motion features of Laban s theory because these features are hard to define and measure by a camera. B. Fuzzy Emotion Inference Figure 2, Table I shows membership functions, singletons, and fuzzy rules used in FEIS of this research. Tanabe et al. AS AM AL 0 PL PM PH VS VM VF HS HM HL s s2 s3 s4 0 p p2 p3 p4 0 v v2 v3 v4 0 h h2 h3 h4 (a) Area : La (b) Position : Lp (c) Velocity : Lv (d) Hand Acceleraion : Lh NUL NUM NUS NEU PPS PPM PPL Rx nu3 nu2 nu 0 pp pp2 pp3 (e) Pleasure and Unpleasure : Rx VS VM VF Fig. 2. HS HM HL HS HM HL HS HM HL NSL NSM NSS NEU PAS PAM PAL ns3 ns2 ns Ry 0 pa pa2 pa3 (f) Arousing and Sleepy : Ry Values of Membership Functions and Singltons TABLE I FUZZY EMOTION INFERENCE RULE AS AM AL PL PM PH PL PM PH PL PM PH NUS NEU NEU NEU NEU PPL NEU PPL PPL NSL NSM NEU NSL NSL NSM NSL NSL NSL NUM NUS PPM NEU NEU PPL NEU NEU PPL NSM NEU PAS NSL NSM NEU NSL NSL NSM NUM NUM NUS NEU NEU PPM NEU PPM PPL NSS PAS PAM NSM NEU PAS NSL NSM NSM NUM NUS PPS NEU PPM PPM PPM PPL PPL NSS NSS PAS NSM NSM PAS NSL NSM NEU NUL NEU NUS NUM NEU PPM PPM PPL PPL NSS NEU PAM NEU NEU NEU NSM NEU PAS NUL NUL NUM NUL NUM NEU NUS PPS PPM PAS PAM PAL NSS PAM PAM NSS PAS PAS NUL NUM NUM NUM NEU NEU NUS PPM PPL PAM PAM PAL NSS NEU PAM NSM NSL PAS NUL NUL NUM NUL NEU NUS NUM PPS PPM PAM PAL PAL NEU PAM PAL NSL PAS PAM NUL NUL NUL NUL NUL NUM NUL NUM PPS PAL PAL PAL PAM PAL PAL PAS PAM PAL (Upper Label: Rx, LowerLabel:Ry) proposed the basic theory of this system. The basic emotional degrees extracted from the motion analysis based on Laban s Theory as input values of fuzzy inference are defined in this system. Values of Pleasure and Unpleasure and Arousing and Sleepy axis are decided based on the rule of Table I so that the system obtains an emotion value on Russell s Circumplex Model. C. Russell s Circumplex Model J. A. Russell in 980 proposed the Circumplex Model [2] that all emotions are expressed by the circumplex arrangement on the plane defined by two dimensions: Pleasure and Unpleasure and Arousing and Sleepy. Additional proposal by Witvliet and Vrana [4], [5] which four basic emotions apply to each quadrant of this model is proposed. Therefore we also defined the human emotion by using these four basic emotions in each quadrant as JOY, ANG, SAD, and REL (See Fig.3). In this research, the human emotion is inferred from R x (Pleasure and Unpleasure) and R y (Arousing and Sleepy) obtained by FEIS. We decide the human emotion based on the quadrant which the inference result (R x, R y ) is belonging. The emotion value (E i : i = JOY, ANG, SAD, REL) means an emotional strength in this method. E i is calculated from Eqs.() and (2). 200

3 ANGER Unpleasure TENSE DISTRESSED ANNOYED FRUSTRATED SADNESS ALARMED AROUSED AFRAID ANGRY Arousing Sleepy EXCITED ASTONISHED JOY DELIGHTED GLAD HAPPY PLEASED SARENE MISERABLE CALM DEPRESSED AT EASE SAD RELAXED GLOOMY BORED SLEEPY DROOPY TIRED Ry O (Rx,Ry) Rx Pleasure SATISFIED CONTENT RELAXATION Fig. 3. Basic Emotions on Russell s Circumplex Model (quoted from reference [2]) E i = D. Algorithm Flow R 2 x + R 2 y sin(π 2θ) () θ = arctan R y R x (2) JOY 0 θ< 2 π ANG i = 2 π θ<π SAD π θ< 3 2 π 3 REL 2 π θ<2π Figure 4 shows the procedure of Fuzzy Emotion Inference System (we call FEIS) proposed in this research. FEIS is constructed with the following algorithm. () Measuring human emotional behaviors with CCD camera. (2) Extracting body features from motion analysis based on Laban s Theory. (3) Calculating the basic emotional degree by fuzzy inference using body features. (4) Obtaining the emotion value used Russell s Circumplex Model from the basic emotional degree. (5) Expressing robot emotional behaviors based on the emotion value. In this research four basic emotions (Joy: JOY, Anger: ANG, Sadness: SAD, Relaxation: REL) are used as known as the basic emotion in Japan. In the following sections, we discuss human s and robot s emotions only about this four types of emotions. IV. EXPERIMENT OF IEC We were able to infer human emotions by above-mentioned FEIS, additionally attempt the interactive experiment between human and pet-type robot. In this experiment, we define each basic emotional behavior expressed by human as JOY-H, ANG-H, SAD-H, and REL-H, each basic emotional behavior expressed by robot as JOY-R, ANG-R, SAD-R, and REL- R, and emotion estimated with the inference from human emotional behaviors by FEIS as JOY-F, ANG-F, SAD-F, and REL-F. A. Experimental Environment We used AIBO (SONY ERS-7) as a pet-type robot in this experiment because we think that a pet-type robot possesses high interpersonal affinity. The environment of this experiment is shown in Fig.5 which shows an example of the communication between a subject and a robot through the computer in case of JOY-H. Table II shows values of membership function (See Fig.2) used in this experiment. We made subjects express various behaviors and executed questionnaires for them to decide these membership functions and singleton values. FEIS was constructed on the computer in this experiment so that we gather huge image data (3 to 5 frames/sec) of human s expression ways. Moreover the outputs of FEIS are sent to the robot through wireless LAN. In addition we asked an observer person to evaluate the accuracy of FEIS. B. Experimental Precondition This experiment was contributed to subjects two university students of 20 generations. Subjects attached five markers that each color was different to his head, both hands and both foots in order to extract body features by the computer. We made subjects perform emotional behaviors freely without (2) Body Feature (3) Basic Emotional Degree (4) Russell's Laban's Motion Analysis Fuzzy Inference Circumplex Model Input Output Motion Measurement Emotion Value Human () Emotional Behavior Robot (5) Fig. 4. Fuzzy Emotion Inference System (FEIS) Fig. 5. Experimental Environment (in case of JOY-H) 20

4 TABLE II MEMBERSHIP FUNCTION AND SINGLETON VALUES La Lp Lv Lh s = 50 p =85 v =5 h =20 s2 = 300 p2 = 70 v2 =0 h2 =45 s3 = 450 p3 = 200 v3 =25 h3 =90 s4 = 700 p4 = 300 v4 = 50 h4 = 200 Rx Ry pp = 00 nu = 00 pa = 00 ns = 00 pp2 = 200 nu2 = 200 pa2 = 200 ns2 = 200 pp3 = 300 nu3 = 300 pa3 = 300 ns3 = 300 restrictions of the way to express each basic emotion. Subject expresses emotional behaviors while he images various situations as shown in Table III. In order to realize a kind of natural living situation, we ask two subjects to think about the situations which were imaged easily by them in advance. Table III shows the emotional situations imaged by two subjects. In this research, at first, emotional behaviors of subject are measured by the web camera connected with the personal computer (See Fig.5). Next, the human emotions from emotional behaviors are recognized by FEIS. The observer plays a role of checking the output of FEIS at real time. Furthermore, the output of FEIS is sent to the robot through the wireless LAN. Finally, the robot expresses emotional behaviors according to the output of FEIS. Emotional behaviors expressed by the robot were restricted to one motion of 4 patterns (JOY-R, ANG-R, SAD-R, REL- R), and each behavior was performed within 3 to 6 seconds. In order to simplify the experiment, we assumed that robot emotional behaviors are the following actions. JOY-R : Robot holds up both hands cheerfully. ANG-R : Robot flings both hands to the ground. SAD-R : Robot droops one s head. REL-R : Robot stretches its arms and legs. Subject person observes a robot in front of him as well as expressing emotional behaviors, and the questionnaire on his impression hold in each robot reaction was investigated after the experiment. Beforehand the subject has known which emotion each emotional behavior expressed, so that the subject is able to understand robot s emotion. The experimental flow is shown as follows. Step : We constructed 6 emotional models (See Table IV) of generating robot s emotion. TABLE III EMOTIONAL SITUATIONS IMAGED BY TWO SUBJECTS Subjects Emotions Situations JOY-H Trial thing was successful. A ANG-H Trial thing was unsuccessful. SAD-H Precious item was broken. REL-H He takes some hot drink. JOY-H His desire was satisfied. B ANG-H He felt insulted. SAD-H He was betrayed. REL-H He absorbed himself in hobby. Model Model Model 2 Model 3 Model 4 Model 5 Model 6 TABLE IV 6EMOTIONAL MODELS IN EXPERIMENT Contents Robot expresses each emotional behavior with same relationship to the result of FEIS, that is, robot expresses as same emotion as human expressed. Robot expresses random emotional behaviors without the relationship to the result of FEIS. Robot expresses emotional behaviors with positive evaluation value. Robot expresses top two emotional behaviors with positive evaluation value. Robot expresses only emotional behavior with the best evaluation. Robot expresses only emotional behavior of the best model of another subject. Step-a : We have already performed the experiment of personal preference analysis [7] and selected the generating robot s emotion from 6 patterns (See Table V and Table VI). Step-b : We constructed effective models (model 4 to 6) of generating robot s emotion as subject A s and B s preference. Step2 : We performed the experiment of Interactive Emotion Communication between human and robot. Step2-a : Subject expresses emotional behaviors wearing with 5 color markers for one minute per each emotion while reminding an emotional situation (See Table III). Step2-b : Web camera obtains images of subject s emotional behavior and the human emotion from emotional behaviors is recognized by FEIS in computer. Step2-c : Computer generates robot s emotion according to emotional model (See Table V and Table VI) Step2-d : Computer sends robot s emotions to the robot. Step2-e : Robot expresses emotional behaviors. Step2-f : Subject observes the robot while he expresses emotional behaviors. Step2-g : This experiment is repeated from Step2-a to Step2-f. Step3 : The questionnaire on his impression hold in each emotional behavior of robot reactions was investigated after the experiment for the subject. C. Impression of Robot Emotional Behavior In this experiment we evaluate each emotional model constructed in the advance experiment [7]. We prepared 6 adjective pairs, Animal-like Mechanical (S ), Interesting Boring (S 2 ), Complex Simple (S 3 ), Familiar Unfamiliar (S 4 ), Natural Unnatural (S 5 ), and Likable Dislikable (S 6 ) for the evaluation of questionnaire, and subjects evaluate robot reactions with 7 grades score ( 3 to 3). We calculated the evaluation value σ because we recognize the likability degree of both subjects. σ which is the weighted 202

5 TABLE V EMOTIONAL MODEL (SUBJECT A) TABLE VI EMOTIONAL MODEL (SUBJECT B) Model Model Model 2 Model 3 Model 4 Model 5 Model 6 Subject Emotion Generate Percentage JOY-R ANG-R SAD-R REL-R JOY-H 00% ANG-H - 00% - - SAD-H % - REL-H % JOY-H 25% 25% 25% 25% ANG-H 25% 25% 25% 25% SAD-H 25% 25% 25% 25% REL-H 25% 25% 25% 25% JOY-H 43% 20% 27% 0% ANG-H - 80% 20% - SAD-H 45% 3% 29% 3% REL-H - 2% 2% 77% JOY-H 62% - 38% - ANG-H - 80% 20% - SAD-H 6% - 39% - REL-H - - 2% 79% JOY-H 00% ANG-H - 00% - - SAD-H 00% REL-H % JOY-H 50% 50% - - ANG-H % - SAD-H 60% - 40% - REL-H - 54% - 46% Model Model Model 2 Model 3 Model 4 Model 5 Model 6 Subject Emotion Generate Percentage JOY-R ANG-R SAD-R REL-R JOY-H 00% ANG-H - 00% - - SAD-H % - REL-H % JOY-H 25% 25% 25% 25% ANG-H 25% 25% 25% 25% SAD-H 25% 25% 25% 25% REL-H 25% 25% 25% 25% JOY-H 50% 50% - - ANG-H % - SAD-H 34% 2% 23% 22% REL-H 24% 4% - 35% JOY-H 50% 50% - - ANG-H % - SAD-H 60% - 40% - REL-H - 54% - 46% JOY-H 00% ANG-H % - SAD-H 00% REL-H - 00% - - JOY-H 00% ANG-H - 00% - - SAD-H 00% REL-H % sum of subject s evaluation score S i (i =,,6) in questionnaire is calculated as shown in Eq.(3). σ = 6 α i S i (3) i= Moreover we asked both subjects the significance weight α i (i =,,6) for 6 adjective pairs in questionnaire in order to calculate σ. In advance, this significance weight is calculated by the percentage of significant factor for each adjective pair obtained by the questionnaire to subjects. The weights α i are expressed with the percentage for Animal-like (α ), Interesting (α 2 ), Complex (α 3 ), Familiar (α 4 ), Natural (α 5 ), and Likable (α 6 ). After the advance experiment [7], we obtained (α, α 2, α 3, α 4, α 5, α 6 )=(0.5, 0.05, 0.5, 0.5, 0.0, 0.5) for subject A, (0.5, 0., 0.025, 0., 0.25, 0.5) for subject B as the weight of score S i. Subject A s and subject B s evaluation values (σ A, σ B ), which are calculated with Eq.(3), are useful when we compare the personal preference. D. Experimental Results Figure 6 shows subject A s (σ A ) and subject B s (σ B ) evaluation. Subject A s evaluation is shown by the solid line and subject B s evaluation by the broken line in this figure. Moreover Table VII shows each evaluation values for 6 emotional models. In addition, light-gray cells show the worst impression model and dark-gray cells show the best impression model in this table as compared with each evaluation. In these results, model 4 was the best impression and model was the worst impression in subject A, and model 5 was the best impression and model was the worst impression in subject B. Because the effective model (model 4 to 6) were well impression, generating robot s emotion from all combination between human and robot emotion was effective in IEC. Subject A s and B s best model were different, but both subject s worst model were consistent. From these results, we confirmed selecting the robot emotional model by the subject s evaluation value is very useful. The best emotional model was different between two subjects, but the both subject s evaluation values of model 4 to 6 were higher than model to 3. These results were very important because the impression includes the variety of human preference. Hereafter we will inspect the tendency for the preference of the robot emotion by repeating these experiments. V. CONCLUSION In this paper, we inspected the human impression for the robot which performed emotional behaviors in actual as emotional model. Moreover we confirmed that the robot gives different impressions to each subject in various situations. The experimental results indicate some guidelines in order to raise the interpersonal affinity between human and robot. In this research, we have executed the experiment with only two subjects. However, we must perform the next experiment by cooperating with more subjects. In the future, we must construct the system which all processes are performed in the robot. Furthermore, the parameter tuning of fuzzy rules takes a lot of time to adapt for each subject. Therefore, we must develop the system which is easy to construct fuzzy rules even in case of the experiment with many subjects. ACKNOWLEDGMENT This research was partially supported by the Ministry of Education, Science, Sports and Culture, Grant-in-Aid for Scientific Research (C), 200,

6 Fig. 6. Impressions of Two Subjects TABLE VII EVALUATION VALUES OF TWO SUBJECTS Model σ A σ B Model Model Model Model Model Model REFERENCES [] Special Issue of Commercialization of Robotic Research Achievements, Journal of the Robotics Society of Japan, Vol.23, No.2, 2005 (in Japanese) [2] Special Issue of Human Symbiotic System, Journal of Japan Society for Fuzzy Theory and Intelligent Informatics, Vol.2, No.5, 2009 (in Japanese) [3] K. Itoh, H. Miwa, M. Matsumoto, M. Zecca, H. Takanobu, S. Rocdella, M. C. Carrozza, P. Dario and A. Takanishi, Various Emotion Expressions with Emotion Expression Humanoid Robot WE-4RII, Proceeding of the st IEEE Technical Exhibition Based Conference on Robotics and Automation, pp.35-36, 2004 [4] A. Bruce, I. Nourbakhsh and R. Simmons, The Role of Expressiveness and Attention in Human-Robot Interaction, Proceedings of 2002 IEEE International Conference on Robotics and Automation (ICRA 02), Vol.4, pp , 2002 [5] P. Y. Oudeyer, The production and recognition of emotions in speech: features and algorithms, International Journal of Human-Computer Studies, Vol.62, pp.57-83, 2003 [6] M. Kanoh, S. Iwata, S. Kato and H. Itoh, Emotive Facial Expressions of Sensitivity Communication Robot Ifbot, Kansei Engineering International, Vol.5, No.3, pp.35-42, 2005 [7] S. Mitshuyoshi, F. Ren, Y. Tanaka and S. Kuroiwa, Non-verbal Voice Emotion Analysis System, Journal of Innovative Computing, Information and Control, Vol.2, No.4, pp , 2006 [8] A.Mehrabian, Nonverbal Communication, Aldine De Gruyter, 2007 [9] K. F. MackDorman, Androids as an Experimental Apparatus: Why Is There an Uncanny Valley and Can We Exploit It?, Toward Social Mechanisms of Android Science, pp.06-8, 2005 [0] N. Tanabe, Y. Maeda, Emotional Behavior Evaluation Method Used Fuzzy Reasoning for Pet-type Robot, Human and Artificial Intelligence Systems From Control to Autonomy(HART), pp , 2004 [] R. Laban, The Mastery of Movement, Plays, Inc., 97 [2] J. A. Russell, A circumplex model of affect, Journal of Personality and Social Psychology, Vol.39, pp.6-78, 980 [3] T. Nakata, T. Mori and T. Sato, Analysis of impression of Robot Bodily Expression, Journal of Robotics and Mechatronics, Vol.5, No., pp.27-36, 2002 [4] C. V. O. Witvliet, S. R. Vrana, Psychophysiological responses as indices of affective dimensions, Psychophysiology, 32, pp , 995 [5] C. V. O. Witvliet, S. R. Vrana, Emotional imagery, the visual startle, and covariation bias: An affective matching account, Biological Psychology, Vol.52, pp , 2000 [6] E. L. Lehmann, NONPARAMETRICS Statistical Methods Based on Ranks, Holden-Days, Inc., 975 [7] R. Taki, Y. Maeda and Y. Takahashi, Personal Preference Analysis for Emotional Behavior Response of Autonomous Robot in Interactive Emotion Communication, Journal of Advanced Computational Intelligence and Intelligent Informatics, Vol.4, No.7, 200 (in Print) 204