Imitation based Human-Robot Interaction -Roles of Joint Attention and Motion Prediction-

Transcription

1 Proceedings of the 2004 IEEE International Workshop on Robot and Human Interactive Communication Kurashiki, Okayama Japan September 20-22,2004 Imitation based Human-Robot Interaction -Roles of Joint Attention and Motion Prediction- Yusuke AKIWA,* Yuki SUGA,* Tetsuya OGATA** and Shigeki SUGANO* * Humanoid Robotics Instrument (HRI), Waseda University Ookubo, Shinjyuku, Tokyo, , Japan ** Graduate School of Informatics, Kyoto University Yoshida-honmachi, Sakyo-ku, Kyoto, , Japan y. akiwa.623qruri.waseda. jp, ysuga@suou. waseda. jp, ogata@i.kyoto-u. ac. jp, sugano@paradise.mech. waseda. ac. j p Abstract Behavior imitation is crucial for the acquisition of intelligence as well as in communication. This paper describes two kinds of experiments of human-robot communication based on behavior imitation. One compared results obtained when the robot did and did not predict the experimental subject s behaviors by using past datasets, and the other compared results obtained with and without target objects in the simulator environment. The result of former experiment showed that the prediction of thle subject s behaviors increase the subject s interest. The result of the latter experiment confirmed that the presence of objects facilitates joint attention and make human-robot communication possible even when the robot uses a simple imitation mechanism. This result shows that in human-robot communication, human not only recognizes the behaviors of the robot passively but also adapts to the situation actively. In conclusion, it is confirmed that motion prediction an,d the presence of objects for joint attention are important for human-robot communication. 1 Introduction The serious problem posed by Japan s decreasing birthrate and aging population is expected to be solved by using robot technology in the care of the elderly. Since this will require robots to interact with human beings, they should be able to recognize expressions of emotion and be able to communicate nonverbally its well as verbally. Many researchers have been investigating these kinds of abilities (e.g.., multimodal conversation [I][2] [3], nodding robots [4] [5] [6] [7], and physical interaction with human beings [8] [9]). We focus on the face-to-face communication between a human being and a robot. We pay particular attention to imitation, which is thought to be one of the most crucial functions for the capability of communication. For example, it is well known that young children imitate their mother s behaviors, and that this helps them to learn to control their motions. M. Kawato has worked on easier ways to program the behavior of humanoid robots, and one of those ways is by having watching robots learn human behaviors [12]. Y. Kuniyosh has tried to reveal the structure of imitation and the mechanism of imitation development [13] [14]. And H. Kozima has investigated the mechanism and development of communication ability.they were developed a robot Infanoid that has pre-verbal communication devices. They have tried to give it attention-sharing ability, in the sense of monitoring a human being s attention [15] [16] [17]. These researchers have explored the nature and development of imitation and shared attention. They have made robots that can imitate and can discern where a person s attention is directed. However they have not directly addressed the problem of communication. We therefore investigated the communicative aspect of the imitation and tried to develop a human-robot communication system. We applied the concepts of shared attention and motion pre /04/$ IEEE

2 diction into our system. To consider the function of imitation in communication, these are thought to be key concepts as mentioned in next chapter. This paper is organized as follows. Section 2 describes the key concepts of our approach to imitation based communication, Section 3 describes our experimental system and methods, Section 4 describes and discusses our experimental results, and Section 5 concludes the paper with a brief summary and our plans for future work. 2 Our Approach 2.1 Motion prediction It is easily guessed that the robot which just imitates human motion makes human bored. Because it cannot prevent the subject from predicting the robot s behavior. In human-human communication, on the other hand, people infer each other s internal state. This difficult makes the communication interesting. This inferential aspect must be applied for humanrobot communication. We implemented a simple prediction function to the imitation system based on difference approximation. We accumulated two subject s hands positions. The imitation system calculates the next coordinates of the experimental subject s hands by using the differences between their previous positions. This function not only makes the robot s behavior smooth and natural, but also makes causes observers surprise, because a wrong prediction makes the robot s motion very different from human motion. This makes it hard for the person to predict the robot s behavior, and the communication will be interesting. 2.2 Joint Attention Joint attention is an early developing socialcommunicative skill in which two people (usually a young child and an adult) use gestures and gaze to share attention with respect to interesting objects or events. The infant turns their eyes to the object that the parents turn their eyes to and the infant pays attention to that object. After a while, the infant starts to pay attention to something actively. The infant thereby learns how to communicate with human beings through the recognition shared with his/her mother. Preverbal communication-that is, an infant s communication using nonverbal channels (e.g., joint attention and facial expressions)-thus plays an indispensable role in the development of human communication. We therefore implemented the concept of joint attention in our imitation system so that we could explore its role in communication. In this system, when the experimental subject interacts with the virtual robot on a simulator, we place the target objects in the real world and the virtual space. These target objects make it easy for the experimental subject to understand the robot s behavior. It is said that inferring partner s internal state is crucial for communication in theory of mind. 3 Experimental System and Method 3.1 Platform System Our platform system uses visual-audio information. An experimental subject is equipped with three markers on his/her head and hands. Using two images captured by two cameras, the system calculates the three-dimensional coordinates of the markers by using an image processor based on the trigonometric algorithms. The audio information used in the system includes the pace, volume, and speed data. The system also recognizes words spoken by an experimental subject. The system output is the robot s voice and behavior in a virtual space. The virtual robot has two arms (three degrees of freedom (DOF) in each shoulder and one DOF in each elbow) and a head (with two DOF), and it has enough joints to make simple gestures. The experimental subjects sat in front of the display and interacted with the virtual robot shown on the display. The experimental environment is shown in Figure 1. Figure 1: experimental environment Part (a) shows the distance between the subject and the display, part (b) shows a camera view, and

3 part (c) shows the displayed robot. 3.2 Motion prediction We evaluated the roles of motion prediction in communication by comparing the results of an experiments in which the experimental subjects ((fifteen students) tried, using only gestures, to teach a behavior to a robot with and without the motion-prediction function. The motion prediction function diagram is shown in Figure 2. In this experiment, we didn t use the voice information. In every processing step, with the image processor (IP7000, HICOS Inc.), the system calculated the 3-D positions based on the trigonometric algorithms, and also calculated the attention direction of subject s head based on the difference between the sizes of two markers on subject s head. Then the system ordered the virtual robot in the simulator to reach its arm to the calculated positions, and ordered it to move its head. In the frame interval (50 ms), the prediction function calculates the next position of the subject s hands based on the difference approximation. -- Head direction Was the robot friendly? Do you feel sense of unity? Do you want to interact with the robot actively? Was your behavior and robot s behavior similar? Are you interested in interaction with robot? Could you interact with robot? Was the robot actively? Did the robot behave variously? Did the robot behave rapidly (with your behavior)? Did you feel that the robot s behavior was flabbergasting? Did you feel that the robot interact with you? Was the robot s behavior speedy? Did you feel that the robot was living being or machinery? Did you feel boring? Is the experiment simple or complexity? Figure 3: Questionnaire items change of it. The delimited behaviors are named Behavior unit. The behavior units include movement data, voice data and the three-dimensional positions of the subject s hands. From the accumulated behavior units, the system adopts one which has the closest position data from current positions of the subject s hands, and outputs it. input output Figure 2: Motion Prediction Function Diagram After the experiment, we used a questionnaire survey to evaluate the impressions of the experimental subjects. The questionnaire asked twenty questions such as Figure Joint attention The system algorithm is shown in Figure 4. The system recognizes the word, and calculates the joint angles of the subject s arms and neck, using the algorithm described above. Next, the system integrates the voice information (the recognized word, volume and pace) and the visual information (the joint angles). In the integration section, the system delimits the subject s behavior based on the magnitude of the We evaluated the roles of joint attention to communication by having the experimental subjects (fifteen students) try, using both voice and gestures, to teach a robot with a joint-attention system, and comparing the results obtained with and without target objects

4 (a soccer ball and a die (see Figure 5)) in the experimental space. We simplified the problem by not the taking effects of the robot s eyes in the joint-attention system I I Figure 5: Target objects After the experiment, a questiomaire was taken survey to evaluate the experimental subject s impressions of the robot s behaviors. Tlhe questionnaire asked ten questions such as Figure 3. 4 Experimental Result and Discussion. 4.1 Experiment 1 (Prediction Function) Graphs comparing the predicted coordinates of the hands with the actual coordinates of the experimental subject s hands (acquired from the cameras) are shown in Figure 6. We defined the prediction rate P its follows: p 2 N N is the total number of predictbons and S is the number of correct predictions. When we defined the threshold range as of 50 mm (half the width of a human palm) because such differences ;are enough small not to influence people s impression,s, the prediction rates of the X, Y, and Z coordinates were respectively 60.4%, 75.6%, and 39.6%. In the result of the principal component analysis of the questionnaire, Interest in the robot was obtained in the first principal component. And Mind like intention was obtained in the second principal component. And in the principal component, we examined whether we use the motion prediction or not. For each principal component, we detected a significant difference between the cases in which motion prediction was used and the cases in which it was not used (Figure 7). When we did not use the motion-prediction function, the robot on the simulator simply imitated the Time[sl - Actual coordinate * Prediction coordinate Figure 6: Prediction Coordinate and Actual Coordinate 1 1,- \, I \ Interest Mind ;keh$ntion in the robot P=O.O15 :Simple imitation :Imitation with 0- motion prediction P< U Figure 7: Prediction Function Result experimental subject s behavior. That is, it simply imitated passively. As time went by, the experimental subject became bored and eventually lost interest in the robot. When we used the motion-prediction function and the function predicted the correct motion, the robot predicted the experimental subject s behavior and the robot s behavior preceded the subject s behavior. The experimental subjects then had the impression that the robot was imitating them just as another person would. The motion prediction was sometimes wrong, and such failures made the robot s behavior different from the subject s behavior and sometimes surprised the subject. This made the robot s behavior interesting to the experimental subjects. Consequently with the

5 prediction function, we could improve our imitation system. In addition, our experimental results suggested that the motion prediction plays a big part in communication. 4.2 Experiment 2 (Joint Attention) In a result of the principal component analysis of the questionnaire, the communication ability was obtained in the first principal component and the sense of unity with the robot was obtained as the second principal component. For each principal component we detected a significant difference depending on -- the presence or absence of the target objects (Figure 8). Communication ability I P= Ii Sense of unity I with robot P=O.O21 I-I] mss :With target objects :Without target object P<0.05 Figure 8: Joint Attention Result If the robot points of the target object, the experimental subject can guess that the robot recognized it. Since the experimental subject can regard the robot s behavior as a response to his/her teaching, the existence of a target object makes it easy to make sense. So the experimental subject can guess the robot s behavior. When there are no target objects, for the experimental subject robot s behavior is only information to guess what the robot thought. So the experimental subject was bored with the robot s behavior. Consequently the existence of the target objects suggests,joint attention, and helps the experimental subject make better guesses about the robot s behavior. The existence of the target objects helps give a sense of unity and a feeling of empathy. The sense of unity can be considered to enhance of the communication ability of the robot. The existence of target objects makes the experimental subject to feel as if the robot has mind like a person s mind. We confirmed that joint attention is necessary for guessing a partner s internal state from his/her behavior. This fact also can be explained from the aspect of theory of mind. Joint attention can play a role in thinking about a partner s position and in thinking from the partner s point of view. 5 Conclusion and Future Works. In this work we paid particular attention to imitation because it is important for the acquisition of intelligence and in communication. We showed the results of two experiments using an imitation communication system. In one experiment, the robot predicted the experimental subject s behaviors by using past datasets. In the other, focusing on the joint attention that is an aspect of communication, we compared between what happened when there were target objects in the simulator environment and what happened when there were not. The results of the former experiment showed that the prediction of personfs behaviors evoked that person s interest. This result shows that in human-robot communication, the human being not only recognizes the behaviors of robots passively but also adapts to the situation actively. This also implies that the human-robot communication is similar to human-human communication by introducing the prediction system. The latter experiment confirmed that the presence of objects facilitates joint attention and can make human-robot communication possible even when the robot uses a simple imitation mechanism. In conclusion, it is confirmed that motion prediction and the presence of object for joint attention are important for human-robot communication. In the future, we will experiment with the system that combines motion prediction and the jointattention system. We will also investigate the relation between the prediction rate and the robot s impression, how much the prediction rate makes an impression on the experimental subject. In addition, we will investigate various ways of motion prediction, the relation between the way of the motion prediction and the prediction rate, the influence of the experimental subject s gender, and the influence of the experimental subject s age. Acknowledgment This research was supported in part by a Grant-in-Aid for the WABOT-HOUSE Project by Gifu Prefecture. References [l] Yosuke Matsusaka, Shinya Fujie; Tetsunori Kobayashi, Modeling of conversational strategy for the robot participating in the group conversation, ISCA Proc. Interspeech 2001, pp : Sept

6 [2] S.Hashimoto,et al. Humanoid Robot-Development of information Assistant Robot Hadaly-,, 6th IEEE International Workshop on Robot and Communication, [3] S.Hashimoto,et al. Humanoid Robots in Waseda University - Haddaly2 and WABIAN-, Autonomous Robots, Vol. 12, No. 1, pp , Jan [4] S.Kawato and J.Ohya, Real-time detection of nodding and head-shaking by directly detecting and tracking the between-eyes, in Proceeding of Forth IEEE international conference on automatic face and gesture recognition, pp , [5] T.Watanabe, M.Okubo, M.Inado.me, Respiratory Entrainment in Face-to-Face Communication, 13th Symposium on Human Interface, oct.21-23, 1997 Osaka. [6] T.Watanabe, E-C0SMIC:Emboclied Communication System for Mind Connection, The Institute oj Electronics, Information and Communication Begineem? Technical Report of IEICE, HCS ( ) [7] Y.Ishii, T.Watanabe Evaluation of the Arrangement of VirtualActors in Remote Communication by Using the Embodied Virtual Communication System, Human Interface Society, Vol. 4, No. 2, pp , [S] T.Tojo, Y.Matsusaka, T.Ishii,T.Kobayashi, A conversational robot utilizing facial and body expressions, in Proceddings of 2000 IEEE SMC2000, Vol. 2, pp , [9] Takayuki Kanda, Hirishi Ishiguro, Tetsuo Ono, Michita Imai, Ryohei Nakatsu, An evaluatuion on interaction between humans and an autonomous robot Robovie, The Robotics Society of.japan, Vol. 20 No. 3, pp , [ 101 Hiroshi Kobayashi, Fumio Hara,.Facial interaction between anmated 3d face robot and human beings, in Proceedings of 1997 IEEE SMC97, vo1.4, pp , [ll] H.Miwa, T.Okuchi, H.Takanobu, A.Takanishi, Development of a New Human-like Head Robot WE-4, IROS2002, Vol., pp , [12] Schaal S. Is imitation learning the route to humanoid robots?, Trends in Cognitive Science 3 (6): pp , [13] S.Kagami, J.J.Kuffner, K.Nishiwaki, Y. Kuniyoshi, K.Okada, Mhaba, H.Inoue, Hu:manoid Arm Motion Planning Using Stereo Vision and RRT Search, Journal of Robotics and Mechatronics, Vol. 15, No. 2, pp , [14] K.Okada, Y.Suzuki, Y.Kuniyoshi, IW.Inaba, H.Inoue, Behavior Acquisition of Humanoid Robot by Imitation, The Robotics Society of Japan, pp , sep.18-20: [15] H.Kozima, H.Yano, A Robot that Learns to Communicate with Human Caregivers, The First lnternational Workshop on Epigenetic Robotics, Lund, Sweden, [16] H.Kozima, Eric Vatikiotis-Bateson, Communicative Criteria for Processing Time/Space-Varying Information, International Workshop on Robot and Human Interactive Communication (ROMA N-2001, Paris, France), [17] M.Imai, T.Ono, H.Ishiguro, Physical Relation and Expression:Joint Attention for Human-Robot Interaction, RO-MAN2001, pp , [18] T.Nishimura, S.Nozaki, H.Yabe, R.Oka, Interactive Motion Dialogue between Human Gesture and Movement of Mobile Robot, The Japanese Society for Artificial Intelligence, SIG-Cll-2000-JULY-06: pp , [19] S.Takase, K.Hanahara, Y.Tada, A Study of Robot Arm Imitating Human Arm Postures Based on Visual Information ~ The Society of Instrument and Control Engineers,, pp , [20] Y.Miyake, Man-Machine Communication as Co- Generation Process,Human Interface Society ~ pp , [all T.Komatsu, K.Suzuki, K.Ueda, K.Hiraki, N.Oka Speech Meaning Acquisition Model by Interaction with its User,Human Interface Society, pp ,