INTERACTIVE EMOTION COMMUNICATION BETWEEN HUMAN AND ROBOT. Received February 2010; revised August 2010

Transcription

1 International Journal of Innovative Computing, Information and Control ICIC International c 2 ISSN Volume 7, Number 5(B), May 2 pp INTERACTIVE EMOTION COMMUNICATION BETWEEN HUMAN AND ROBOT Yoichiro Maeda and Ryohei Taki 2 Graduate School of Engineering 2 Faculty of Engineering University of Fukui 3-9-, Bunkyo, Fukui City, Fukui Prefecture, 9-857, Japan { maeda; rtaki }@ir.his.u-fukui.ac.jp Received February 2; revised August 2 Abstract. In this paper, we aim to realize the bidirectional communication that a human and a robot perform the face to face interaction based on the behavior with emotion. A model of Interactive Emotion Communication (IEC) is proposed in this paper. This model is a kind of interactive process that a robot infers human emotions from human behavior and generates robot emotions. In addition, we evaluated the impression for emotional behavior of pet-type robot in the experiment of human-robot interaction. Keywords: Emotion, Communication, Interaction, Fuzzy inference. Introduction. Recently, opportunities when a robot contacts human are increasing, therefore, the technology for the interactive communication with human is gradually needed. In addition, the flexible understanding ability of human intension and the expressing ability of robot intention are required for the robot to live together. Technology for realizing the interactive communication between human and robot has not been established yet, therefore, there are few robots which communicate to human smoothly. In order to understand the human intension and express the robot intention, some researches used the nonverbal information have been proposed [, 2, 3, 4]. If there are difference between the verbal and nonverbal communication when we convey our emotion and attitude, nonverbal communication includes over 9% information for the emotion of interlocutor. There are various kinds of nonverbal communication that is eye sign, voice, expression, gesture, and so on. On the other hand, we feel the unpleasantness for the robot expression like human. We have been performed the research on the nonverbal communication based on human and robot behavior. Because a robot does not have various abilities to express own intention and can perform only the restrictive way of motions, sounds and so on, the communication between human and robot is generally very difficult. For this reason, we proposed a method Interactive Emotion Communication to communicate through emotional behavior. By this method, we try to realize the interaction that the human and robot enable to communicate smoothly through emotions. In this research, the method of emotion inference from the human behavior is used [5]. At first, the body feature of a subject is extracted based on the Laban s theory. Next, we obtain the basic emotional degree by fuzzy inference using extracted human body feature. Finally, the emotion value of human behavior is evaluated based on the Russell s Circumplex Model. In this research, we aim to realize Interactive Emotion Communication (we call IEC) which is a bidirectional communication based on the emotional behavior between human and robot. 296

2 2962 Y. MAEDA AND R. TAKI Step I'm sad. [Recognize Emotion] Step2 He seems to be sad. Interactive Emotion Communication [Generate Emotion] Step4 I'm happy. Step3 I will make him happy. [Express Emotion] Figure. Interactive emotion communication (IEC) () Input (2) (3) (4) Output (5) Human Russell's Laban's Motion Analysis Fuzzy Inference Robot Motion Circumplex Model Motion Measurement Body Feature Basic Emotional Degree Emotion Behavior Figure 2. Fuzzy emotion inference system (FEIS) 2. Interactive Emotion Communication. This research assumes the bidirectional communication model through emotional behavior between human and robot as one example of several nonverbal communication. The emotional behavior means the gesture or dance to represent emotion to an opposite person. We call this bidirectional communication Interactive Emotion Communication (IEC). Figure shows the conceptual drawing of IEC. We assume that there are two persons A and B here. First of all, A generates something emotion and expresses emotional behavior to B by his gesture (Step ). Next, B recognizes A s emotion by his vision ability (Step 2). B expresses emotional behavior to A with B s emotion (Step 3). In this way, the interactive communication by emotional behavior is constructed between A and B. Assuming that B is a robot, the following three processes become necessary.. Recognizing human s emotion. 2. Generating robot s emotion. 3. Expressing robot s emotion. There are many recent researches to understand human emotion [6, 7] or express robot emotion [8], however, we could find very few references regarding the interaction between the human with various ways of expression and the robot with only restrictive way of expression. In this research, we proposed a new trial of the communication between a human and a robot with their behavior through emotions. We try to investigate the ability of the proposed method. Our final goal is to build the robot which is able to recognize human s emotion and express its emotion by the bidirectional communication based on IEC model with high interpersonal affinity. 3. Fuzzy Emotion Inference System based on Laban s Theory. Mainly in this chapter, we explain the first one of above-mentioned IEC recognizing human s emotion process which inferences human s emotion from emotional behavior of human.

3 IEC BETWEEN HUMAN AND ROBOT 2963 Table. Classification of effort (quoted from [9]) Motion Element Effort Measurable Aspects Classifiable Aspects Weight Firm Gentle Resistance Levity Time Sudden Sustained Speed Duration Space Direct Flexible Direction Expansion Flow Bound Free Control Fluency Table 2. Classification of Shape (quoted from [9]) Plane of Form Fighting Form Indulging Form Table Plane Enclosing Spreading Door Plane Ascending Descending Wheel Plane Retreating Advancing 3.. Fuzzy emotion inference system algorithm. Figure 2 shows the procedure of Fuzzy Emotion Inference System (we call FEIS) proposed in this research. FEIS is proposed with the following algorithm. () Measuring human emotional behavior using a CCD camera. (2) Extracting body features from the motion analysis based on Laban s Theory. (3) Calculating the basic emotional degree by the fuzzy inference using body features. (4) Obtaining the emotion value using Russell s Circumplex Model based on the basic emotional degree. (5) Expressing robot emotional behavior based on the emotion value. The emotion used in this research into four basic emotions (Joy: JOY, Anger: ANG, Sadness: SAD, Relaxation: REL). In the following sections, we discuss human s and robot s emotions only about these four types of emotions Motion analysis based on Laban s theory. Laban s theory [9] proposed by R. Laban is a method to extract the macro features from human body motions. This method has three types of description about motion features, that is, Effort-Shape Description, Motif Description and Structural Description. Above all Effort-Shape Description describes the quality of motions and meaning of the expression. Because it is useful when the human body motion is classified according to the visual function, we use this type of description to analyze human body motions. R. Laban proposed the theory that there are a bipolar system based on Fighting Form and Indulging Form in the expression of human body motion. Fighting From means active and vivid body motion and Indulging Form means slow and gentle body motion. The concept that human motions are subdivided by these forms is Effort-Shape Description. Tables and 2 show the classification of Effort and Shape. Effort is effective to classify the body motion based on Kansei information. Shape shows the feature of overall static shape of the body motion, moreover, Shape do not include considering local motion feature. In this research, we suppose that Time Effort is the speed of the body s center of gravity, Flow Effort is the hand acceleration, Table-Plane

4 2964 Y. MAEDA AND R. TAKI AS AM AL PL PM PH VS VM VF HS HM HL s s2 s3 s4 p p2 p3 p4 v v2 v3 v4 h h2 h3 h4 (a) Area : La (b) Position : Lp (c) Velocity : Lv (d) Hand Acceleraion : Lh NUL NUM NUS NEU PPS PPM PPL NSL NSM NSS NEU PAS PAM PAL Rx nu3 nu2 nu pp pp2 pp3 (e) Pleasure and Unpleasure : Rx ns3 ns2 ns pa pa2 pa3 (f) Arousal and Sleep : Ry Ry Figure 3. Membership functions and singletons Table 3. Values of membership functions and singltons La Lp Lv Lh s=5 p=85 v=5 h=2 s2=3 p2=7 v2= h2=45 s3=45 p3=2 v3=25 h3=9 s4=7 p4=3 v4=5 h4=2 Rx Ry pp= nu= pa= ns= pp2=2 nu2= 2 pa2=2 ns2= 2 pp3=3 nu3= 3 pa3=3 ns3= 3 Shape is the body area and Door-Plane Shape is the height of the body s center of gravity Fuzzy emotion inference rule. Figure 3, Tables 3 and 4 show membership functions, singletons and fuzzy rules used in FEIS of this research. Tanabe et al. proposed the basic theory of this system [5]. The basic emotional degrees extracted from the motion analysis based on Laban s Theory as input values of fuzzy inference are defined in this system. Values of Pleasure and Unpleasure and Arousal and Sleep axis are decided based on the rule of Table 4 so that the system obtains an emotion value on Russell s Circumplex Model Russell s Circumplex model. J. A. Russell in 98 proposed the Circumplex Model [] that all emotions are expressed by the circumplex model on the plane defined by two dimensions: Pleasure and Unpleasure and Arousal and Sleep. Additional proposal by Witvliet and Vrana [] which four basic emotions apply to each quadrant of this model is proposed. Therefore, we also defined the human emotion by using these four basic emotions in each quadrant as JOY, ANG, SAD and REL (See Figure 4). In this research, the human emotion is inferred from R x (Pleasure and Unpleasure) and R y (Arousal and Sleep) obtained by FEIS. We decide the human emotion based on the quadrant which the inference results (R x, R y ) are belonging. The emotion value (E i : i = JOY, ANG, SAD, REL) means an emotional strength in this method. E i is calculated from Equations () and (2). E i = Rx 2 + Ry 2 sin(π 2θ) () θ = arctan R y R x (2)

5 VS VM VF HS HM HL HS HM HL HS HM HL IEC BETWEEN HUMAN AND ROBOT 2965 Table 4. Fuzzy emotion inference rule AS AM AL PL PM PH PL PM PH PL PM PH NUS NEU NEU NEU NEU PPL NEU PPL PPL NSL NSM NEU NSL NSL NSM NSL NSL NSL NUM NUS PPM NEU NEU PPL NEU NEU PPL NSM NEU PAS NSL NSM NEU NSL NSL NSM NUM NUM NUS NEU NEU PPM NEU PPM PPL NSS PAS PAM NSM NEU PAS NSL NSM NSM NUM NUS PPS NEU PPM PPM PPM PPL PPL NSS NSS PAS NSM NSM PAS NSL NSM NEU NUL NEU NUS NUM NEU PPM PPM PPL PPL NSS NEU PAM NEU NEU NEU NSM NEU PAS NUL NUL NUM NUL NUM NEU NUS PPS PPM PAS PAM PAL NSS PAM PAM NSS PAS PAS NUL NUM NUM NUM NEU NEU NUS PPM PPL PAM PAM PAL NSS NEU PAM NSM NSL PAS NUL NUL NUM NUL NEU NUS NUM PPS PPM PAM PAL PAL NEU PAM PAL NSL PAS PAM NUL NUL NUL NUL NUL NUM NUL NUM PPS PAL PAL PAL PAM PAL PAL PAS PAM PAL (Upper Label: Rx, Lower Label: Ry) ANGER Unpleasure TENSE DISTRESSED ANNOYED FRUSTRATED SADNESS ALARMED AROUSED AFRAID ANGRY Arousal Sleep EXCITED ASTONISHED JOY DELIGHTED GLAD HAPPY PLEASED SARENE MISERABLE CALM DEPRESSED AT EASE SAD RELAXED GLOOMY BORED SLEEPY DROOPY TIRED Ry O (Rx, Ry) Rx Pleasure SATISFIED CONTENT RELAXATION Figure 4. Basic emotions on Russell s circumplex model (quoted from []) JOY ANG i = SAD REL θ < π π θ < π 2 2 π θ < 3π 3π θ < 2π 2 2

6 2966 Y. MAEDA AND R. TAKI Figure 5. Experimental environment 4. Experiment. We were able to infer rough human emotions by the above-mentioned FEIS, additionally attempt the interactive experiment between human and pet-type robot. In this experiment, we define each basic emotional behavior expressed by human as JOY- H, ANG-H, SAD-H and REL-H, and each basic emotional behavior expressed by robot as JOY-R, ANG-R, SAD-R and REL-R. 4.. Experimental environment. We used AIBO (SONY ERS-7) as a pet-type robot in this experiment. The program of AIBO is freely read and written by the personal computer through the exclusive memory stick. The joint is 2 degrees-of-freedom in total. The environment of this experiment is shown in Figure 5. FEIS was constructed on the another computer with the web camera in this experiment because huge image data (3 to 5 frames/sec) of human s expression was processed. However, in the next stage, we have a plan to realize the face to face communication between human and robot Precondition of experiment. This experiment was performed by cooperating of 22 years old student in our laboratory as a subject. Subjects were attached five markers with different colors on his head, both hands and both feet to extract body features. We made a subject express emotional behavior freely without time limit. In this research, at first, emotional behavior of subject is measured by the web camera connected with the personal computer (See Figure 5). Next, the human emotions from emotional behavior are recognized by FEIS. Furthermore, the result of FEIS is sent to the robot through the wireless LAN. Finally, the robot expresses emotional behavior according to the result of FEIS. Emotional behavior expressed by the robot were restricted to one motion of 4 patterns (JOY-R, ANG-R, SAD-R, REL-R), and the expression time of each behavior was spent within 3 to 6 seconds. We tried to perform the following four kinds of experiment. () Random Reaction: Robot expresses random emotional behavior without the relationship to the result of FEIS. (2) Echo Reaction: Robot expresses each emotional behavior with same relationship to the result of FEIS, that is, robot expresses as same emotion as human expressed. (3) Contrary Reaction: Although robot expresses each emotional behavior with opposite relationship to the result of FEIS, that is, robot expresses the inverse emotion set on the origin symmetry in Figure 4. (4) Variable Reaction: Robot expresses emotional behavior taught by observer without relationship to the result of FEIS. Robot performs emotional behavior by observer. However, robot expresses an inverse emotion at the probability of 5%. For example, the inverse emotion means that robot expresses SAD-R if the result of FEIS is JOY-H, and robot expresses REL-R if the result of FEIS is ANG-H.

7 IEC BETWEEN HUMAN AND ROBOT 2967 Table 5. Communication time and order of likability Reaction Communication Order of Time (sec) Likability Random Reaction 6 2 Echo Reaction 9 Contrary Reaction 36 4 Variable Reaction 7 3 Subject person observes the robot in front of him as well as expressing emotional behavior, and the questionnaire on his impression hold in each robot reaction was investigated after the experiment. We performed the experiment in order of () Random Reaction to (4) Variable Reaction. Beforehand the subject has known which emotion each emotional behavior expressed, so as to the subject is able to understand robot s emotion Experimental results. Table 5 shows the result of communication between human and robot. The communication time shows the time interval when the subject was communicating with the robot, in other words, the time interval until getting tired. And the order of likability means the order which the subject prefers each reaction. We think that long communication time shows good communication comparatively, because we communicate for a long time with favorite friends. () Random Reaction experiment shows an incoherent type of emotional behavior and timing, whereas the evaluation of Random Reaction reached high value contrary to our expectations. Random Reaction s experimental result is shown in Figure 6 as solid lines. This graph shows outputs of FEIS and motion timings to express emotional behavior by the robot. X axis on these graphs means time of the experiment and y axis means emotion values (E i ). In this paper, we divide the experimental result to four graphs ((a) JOY, (b) ANG, (c) SAD and (d) REL). Gray area shows the time period when the subject performed his emotion behavior in this experiment, moreover, the timing when the robot generated its emotion is shown with broken lines. Because AIBO expressed sometimes same emotion that a subject expresses, we think the subject believed that AIBO expressed emotion according with the result of FEIS. The impression that a robot was acting by himself independently was given to the subject according to the questionnaire. Robot expressed JOY-R frequently when the subject expressed SAD-H. We think that these robot reactions connect with good impression. (2) Echo Reaction s experimental result is shown in Figure 7. In this experiment, because FEIS was able to infer the original human emotion accurately, the robot expressed its emotion according as human s behavior. Echo Reaction experiment gave the impression that this robot was considerably more intellectual and most familiar to the human, nevertheless, the communication time about 9 seconds was shortest than any other reactions. This is because the subject was bored with simple patterns that robot repeated the same emotion of human s. (3) Contrary Reaction experiment gave an impression estranged from the robot, therefore, the impression of this experiment was the worst. Contrary Reaction s experimental result is shown in Figure 8. The reason of this result is that only one emotional behavior of the robot was selected for an emotional behavior of the human in case of reactions of (2) and (3). For example, the robot expressed only SAD-R when the human expressed JOY-H in reaction (3). However, the two reactions were significantly different in the order of likability. This reason is because negative robot emotion (SAD-R and ANG-R) in case of positive human emotion (JOY-H and REL-H) was worse impression than positive robot

8 2968 Y. MAEDA AND R. TAKI (a)joy (b)ang (c)sad (d)rel (a)joy (b)ang (c)sad (d)rel Figure 6. FEIS result (Case (): Random reaction) Figure 7. FEIS result (Case (2): Echo reaction) emotion (JOY-R and REL-R) in case of negative human emotion (SAD-H and ANG-H). The communication time was comparatively short 36 seconds. This reason is because the subject noticed that robot intentionally expressed the inverse emotion. (4) Variable Reaction experiment gave an impression that it was the most complicated emotion expressed to the subject and secondly bad impression. Variable Reaction s experimental result is shown in Figure 9. The subject was going to convey his emotion desperately, therefore the communication time was the longest in all other experiment.

9 IEC BETWEEN HUMAN AND ROBOT (a)joy (a)joy (b)ang (b)ang (c)sad (c)sad (d)rel (d)rel Figure 8. FEIS result (Case (3): Contrary reaction) Figure 9. FEIS result (Case (4): Variable reaction) The subject did not feel close friendship, but he also was given the impression that this reaction was the most interesting Remarks. The subject said that the communication experiment with the robot was interesting, so we were able to confirm that the communication with the robot is useful and attracts human s interaction. Echo Reaction experiment had a high evaluation value, however, it was early to be tired of him so communication time was very short. This reaction is still far from practicality and necessary to be improved.

10 297 Y. MAEDA AND R. TAKI By contrast the subject tried to communicate in the experiment of Contrary Reaction and Variable Reaction although they were gotten the impression which are not familiar. He answered that he wanted the robot to express the same emotion in the questionnaire, therefore, we could confirm the effect of communicating actively. This result is useful when we consider about robot suitable reactions. 5. Conclusion. In this paper, we constructed a basic system that a robot communicates with a human based on IEC, moreover inspected the impression for the robot which performed emotional behavior in actual. In consequence, the difference of robot reaction influenced to human s impression and speed to be tired. We must consider what emotion the most natural reaction to generate for human s emotion is, and what reaction the highest interpersonal affinity gives. In this research, the system inferred the human emotion from emotional behavior by the computer to gather many images of human behavior. However, in the future, we must construct the system which all processes are performed in the robot. Furthermore, the parameter tuning of fuzzy rules takes a lot of time to adapt for each subject. Therefore, we must develop the system which is easy to construct fuzzy rules even in case of the experiment with many subjects. Acknowledgment. This research was partially supported by the Ministry of Education, Science, Sports and Culture, Grant-in-Aid for Scientific Research (C), 29-2, REFERENCES [] K. Itoh, H. Miwa, M. Matsumoto, M. Zecca et al., Various emotion expressions with emotion expression humanoid robot WE-4RII, Proc. of the st IEEE Technical Exhibition Based Conference on Robotics and Automation, pp.35-36, 24. [2] A. Bruce, I. Nourbakhsh and R. Simmons, The role of expressiveness and attention in humanrobot interaction, Proc. of 22 IEEE International Conference on Robotics and Automation, vol.4, pp , 22. [3] P. Y. Oudeyer, The production and recognition of emotions in speech: Features and algorithms, International Journal of Human-Computer Studies, vol.62, pp.57-83, 23. [4] M. Kanoh, S. Iwata, S. Kato and H. Itoh, Emotive facial expressions of sensitivity communication robot Ifbot, Kansei Engineering International, vol.5, no.3, pp.35-42, 25. [5] N. Tanabe and Y. Maeda, Emotional behavior evaluation method used fuzzy reasoning for pet-type robot, Human and Artificial Intelligence Systems From Control to Autonomy (HART), pp , 24. [6] J. Minato, K. Matsumoto, F. Ren, S. Tsuchiya and S. Kuroiwa, Evaluation of emotion estimation methods based on statistic features of emotion tagged corpus, International Journal of Innovative Computing, Information and Control, vol.4, no.8, pp.93-94, 28. [7] Y. Oyama and Y. Narita, A proposal for automatic analysis of emotions using facial charts, International Journal of Innovative Computing, Information and Control, vol.5, no.3, pp , 29. [8] T. Nomura and A. Nakao, Comparison on identification of affective body motions by robots between elder people and university students: A case study in Japan, International Journal of Social Robotics, vol.2, pp.47-57, 2. [9] R. Laban, The Mastery of Movement, Plays, Inc., 97. [] J. A. Russell, A Circumplex model of affect, Journal of Personality and Social Psychology, vol.39, pp.6-78, 98. [] C. V. O. Witvliet and S. R. Vrana, Psychophysiological responses as indices of affective dimensions, Psychophysiology, vol.32, pp , 995.