Dynamic Emotion-Based Human-Robot Collaborative Assembly in Manufacturing:The Preliminary Concepts

Transcription

1 Dynamic Emotion-Based Human-Robot Collaborative Assembly in Manufacturing:The Preliminary Concepts Rahman S. M. Mizanoor, David Adam Spencer, Xiaotian Wang and Yue Wang Department of Mechanical Engineering Clemson University, Clemson, SC 29634, USA s: {rrahman, Abstract In this paper, we present the preliminary concepts regarding the dynamic emotion-based human-robot collaboration in an assembly task in manufacturing. We employ an anthropomorphic robot with emotion display abilities to collaborate with a human for the assembly task. The human serves as a supervisor and a co-worker when collaborating with the robot for the task. We consider situations when there is neither direct physical contact nor tele-operated interaction between the robot and the human, but they work side by side with each other, share the same objective and working environment. We first identify how the human may be influenced by the robot s emotions when the human collaborates with the robot during the task. We then propose intelligent control algorithms that may help the robot dynamically adjust its emotions with the task situations. We preliminarily investigate the effects of emotions on Human-Robot Interactions (HRI) and on the assembly performance. The results show that the static emotion produces better HRI and assembly performance than that the robot produces with no emotion. The dynamic emotions of the robot may improve HRI in manufacturing operations to enhance manufacturing performances in terms of productivity, efficiency, quality, safety, cost effectiveness etc. Keywords- Dynamic Emotion; Humanoid Manufacturing Robot; Intelligent Control; Human-Robot Collaboration; Intelligent Assembly; Smart Manufacturing I. INTRODUCTION The human shows various emotions through various facial expressions. The common emotions observed in humans are anger, frustration, disgust, fear, distress, calmness, joy, sorrow, surprise, interest, boredom, anxiousness, curiosity, attraction, desire, admiration, sadness, trust, embarrassment, helplessness, powerlessness, worry, doubt, anxiety, shock, stress etc. Each emotion has its antecedent conditions and affects the behaviors and functions of the human as well as that of other humans next to the human. Emotions are dynamic phenomena [1]. Emotions foster face to face human-human communications and help the human understand another human s mental state based on emotions and then may help decide appropriate responses, interactions and behaviors to another human. Some emotions may affect positively and enhance the effectiveness of the communications and interactions, while other emotions may affect the interactions negatively [1]. Social robotics researchers were inspired by the emotions of the humans to develop robots with anthropomorphic appearance and human-like emotions. It was believed that such anthropomorphic appearance and human-like emotional features would make the human-robot interactions and collaborations more effective and realistic [2]-[3]. MIT robot Kismet, Japanese robot Kobian, British robot ERWIN, NAO, icat, Flobi etc. are the early-stage anthropomorphic robots that could exhibit emotions [4]-[6]. Most recently, several anthropomorphic robots with indistinguishable human appearance have been introduced that can exhibit more realistic and human-like emotions. These robots include Hanson robokind robots with realistic human faces, Geminoid robots, Nadine etc. [7]-[8]. However, all of the above state-of-the-art humanoid robots with emotions have been proposed for human-robot social interactions [9]-[10]. So far, there are almost no practical applications of such emotional robots; instead their applications are limited to the laboratory experiments only. Furthermore, the emotions are mostly static and are not dynamically adjustable with that of their human counterparts and with the task situations [9]-[10], which is a hindrance towards the symbiosis in the human-robot collaborations [11]. It is true that some robots can recognize the human s emotions and based on it can change their own emotions dynamically [4]. But, these robots are for social services and their dynamic ability is still very limited. The emotions are also not associated with their functions. On the other hand, most of the robotic devices used in the manufacturing environments collaborating with humans have functional configuration and appearance that lack anthropomorphism and emotions [12]-[13]. We believe that humanoid robots with properly designed dynamic emotional abilities used in manufacturing environments may enhance the human-robot interactions (HRI) and thus may improve the manufacturing performances in various terms such as productivity, efficiency, costs, quality etc. Because, dynamic emotion capability in the robot may help display the robot s mental states based on the task situations that may make the communications between the robot and the human more effective that may in turn enhance the manufacturing performances. However, such anthropomorphic robots with dynamic emotional abilities have never been used in manufacturing environments, and the effects of the dynamic

2 emotions of robots on the manufacturing performances have never been investigated. Hence, we propose an anthropomorphic robot with emotional abilities for collaboration with human in an assembly task in manufacturing. We identify a specific task where the human and the robot collaborate to assemble parts/objects. The human acts as both a supervisor and a co-worker. We propose an algorithm that may be used to make the robot s emotions dynamic based on the task situations. We then explain how the robot s emotions help the human better adjust with the robot. We also propose the experiment methods that may help understand the effects of robot s dynamic emotions on the HRI and on manufacturing performances. As the proof of concepts, we present the preliminary results that compare the HRI between no emotion and static emotion of the robot for the human-robot cooperative assembly task. II. THE MANUFACTURING ROBOT We use a Rethink Robotics Baxter humanoid manufacturing research robot made by Rethink Robotics [14] as shown in Fig.1, which is the first collaborative anthropomorphic robot with emotions used in manufacturing activities. Its both hands can work simultaneously (multitasking arms). Each arm has 7 Degrees-of-Freedom (DoFs). The joints are compliant and enriched with back-driveable motors and force sensors. Through force detection, it can feel contact with objects and work surfaces. Figure 1. The Rethink Baxter robot (loaned from BMW) to be used for the proposed human-robot collaborative assembly in manufacturing. The left-hand end-effectors (grippers) are changeable and are suitable for pick and place (PnP) operations. The right hand has pneumatic end-effector (gripper) that can be used to pick and place very thin papers and plates. Baxter is suitable for material handling, machine tending, testing and sorting, light assembly, finishing operations etc. It has a moveable base and a rotary screen at its head where emotions can be displayed. The robot emotions may be dynamically changed in the screen based on task situations when the robot works with the human coworkers in the manufacturing environments. Base color of the screen can also be adjusted to reflect changes in its emotions. The human co-worker can teach the robot by demonstration, i.e. by moving the robot arms and joints and showing how to perform a manufacturing task. The robot is controlled through the Robot Operating System (ROS) software [15]. III. THE SELECTED HUMAN-ROBOT COLLABORATIVE ASSEMBLY TASK Figure 2. The selected human-robot collaborative assembly task. We select a representative assembly task that is to be performed by the Baxter robot and the human co-worker in collaboration. As shown in Fig.2, the robot at first picks a rectangular object (first object) and places it on a table surface. Then, the human picks another object (second object) and places it on the first object. Then, the robot again picks another object (third object) and places it on the second object. In this task, the human has roles as both a co-worker and a supervisor. The human co-works with the robot for the assembly task as mentioned above. In addition, the human has also supervisory roles such as- 1. The human teaches the robot how to pick and place the object, 2. The human ensures that the correct objects (correct size, shape etc.) are input to the robot timely and are placed at the right location so that the robot can pick it correctly, 3. The human dispatches the objects placed by the robot timely so that the locations where the robot needs to place the objects are always empty, 4. The human ensures that no unwanted persons and obstacles can come to the workspace, 5. The human ensures the robot s power supply, 6. The human also fixes jigs or fixtures if necessary, does other auxiliary works related to assembly, 7. The human changes the control commands through the software inputs, keyboard control or joystick control if necessary, 8. When the task is finished, the human either turns the robot off or transfers the robot to another task and teaches the new task to the robot again etc.

3 Thus, the human shares the assembly task and the manufacturing environments with the robot for its twofold roles (supervisor and co-worker). We select this assembly task because- 1. This type of assembly tasks is very common in most manufacturing environments and thus we believe that our model may be the most useful to many manufacturing companies e.g. automobile assembly. 2. Human-robot collaboration is very important and very clear for this type of assembly tasks. We believe that performances for human-robot collaboration for this task may be better than that for individual human or the robot because the manual task is burden to the human and the fully autonomous task is not flexible. Instead, human-robot collaboration reduces human s burdens and at the same time increases flexibility [21]-[23]. 3. We think that the emotions of the robot and the human may play significant roles for such assembly task etc. IV. EMOTION-BASED INTERACTIONS BETWEEN THE ROBOT AND THE HUMAN DURING THE TASK There are face-to-face communications when the human plays his/her supervisory and co-worker roles with the robot during the assembly task in the manufacturing environments as explained in section III. The face-to-face communications for the above task may produce three types of emotion-based interactions between the human and the robot as follows- 1. The robot s emotions are dynamically changed based on the task situations and the human can recognize the robot s emotions and is influenced by the robot s emotions, 2. The human s emotions are dynamically changed based on the task situations and the robot can recognize the human s emotions and is influenced by the human s emotions, 3. The human and the robot both can recognize the dynamically changed emotions of each other and are influenced by the emotions of each other (in this case, emotion-based bi-directional communications occur between the human and the robot, which is natural). This paper is confined to the above case#1 only where the robot s emotions are dynamically changed based on the task situations and the human can recognize the robot s emotions and is influenced by the robot s emotions. In this case, the robot evaluates the human s cooperation levels and displays its emotional reactions on its head screen. The emotional displays in the robot may be changed based on the task situations and on the human s cooperation levels. The human may be able to recognize the robot s emotions, understand and analyze the task situations as well as the robot s status and its understanding of the task, and adjust his /her cooperation levels so that he/she can provide the best cooperation to the robot counterpart. Different types of emotions may be displayed in the robot screen based on the task situations and on the human s Figure 3. Prospective emotions that the robot may display at various task situations and cooperation levels [14]. cooperation levels. The robot s neutral emotional state may be displayed when the robot is ready to receive the training from the human. The robot may show a concentrating emotion when it concentrates on the work with the human. The robot may be confused that may be reflected as emotion in the screen when the robot needs more inputs on the task. The robot may show surprised emotion when it is surprised e.g. when someone enters the work area unexpectedly. The robot may show sad emotion when it is on hold and awaits further instructions. The robot may show asleep when it is on standby. The robot may show a happy emotion when everything seems to be satisfactory to the robot. The robot may show an angry emotion when it becomes helpless, it sees no hope to proceed with the works, and so forth. Figure 3 shows the prospective emotions that the robot may display at various task situations and cooperation levels. V. INTELLIGENT CONTROL ALGORITHMS FOR THE DYNAMIC EMOTIONS Figure 4 shows the proposed intelligent control algorithms for dynamic emotions for the human-robot collaborative assembly. The algorithms associate dynamic emotions with the task described in Fig.2. At first, the robot shows the neutral emotion in its screen, which means that the robot is ready to receive training for the task. Then, the robot moves from the initial position to the location where the robot needs to pick the object (InitToPick function runs/playbacks that was previously taught to the robot by the human). The robot end effector includes an infrared rangefinder that is used to locate the edges of the object and determine height for picking the object from the surface (table).we assume that the correct object s height (h) is x with a tolerance of y. If the object (input by the human) is found within the correct height, the robot expresses happiness and then it shows the concentrating emotion and then picks the object and places at the specified location on the table (PnP function runs/playbacks that was

4 Start Neutral InitToPick Yes h No Happy Confused Concentrating Sad PnP Yes h No End Happy Concentrating Angry End PnP End Figure 4. The intelligent control algorithm for dynamic emotions for the human-robot collaborative assembly. previously taught to the robot by the human).then, the human places the second object on the first object that was picked and placed by the robot. However, if the object s height is not correct, then the robot becomes confused that makes it sad later. If the human can understand the problem and replace the wrong object by the correct object swiftly, then the robot becomes happy and concentrates on the work. Then, the robot picks the object and places it at the designated location on the table, and the human then puts the second object on the first object placed by the robot. However, if the correct object is still not found, the robot becomes angry. The robot picks the third object and places it on the second object in the similar way. As the next step, we will determine the dynamics of the robot emotions inspired by that of the human [16]-[20]. We will quantify the dynamic evolution of robot emotions through analytical mathematical equations. We then design suitable intelligent control algorithms based on the emotion dynamics to dynamically adjust the emotions of the robot with that of the human and with the task situations. VI. EXPERIMENTS We need to evaluate the effectiveness of the aforementioned control algorithms to produce dynamic emotions in the robot s head screen that are adjusted dynamically with that of the task situations. We also need to evaluate how the dynamic emotions of the robot make the human-robot interactions more effective that may be reflected through the enhancement in the assembly performances in terms of efficiency, productivity, costs, quality, safety etc. To understand the effects of the dynamic emotions of the robot on the manufacturing performances, we at first need to determine an objective evaluation scheme for each of the manufacturing performance criterion (efficiency, productivity, costs, quality and safety) for the assembly task. Then, we need to evaluate the manufacturing performance separately for the following three conditions: 1. Assembly task done in collaboration between the robot and the human with no emotion of the robot (in this condition, the emotion display screen of the robot will be either turned off or removed from the robot),

5 Mean evaluation score 2. Assembly task done in collaboration between the robot and the human with static emotion of the robot (in this condition, the emotion display screen of the robot will be turned on and the robot will be operated under its current static emotion abilities e.g., only the concentrating emotion will be displayed when the robot concentrates on the work), 3. Assembly task done in collaboration between the robot and the human with dynamic emotion of the robot (in this condition, the emotion display screen of the robot will be turned on and the robot will be operated with its dynamic emotion abilities based on the novel control algorithms depicted in Fig.4). If we compare the performances among the aforementioned three conditions, then we may be able to understand the effects of dynamic emotions on assembly performances. However, as an initial effort, we have conducted preliminary experiments for conditions #1 and 2 separately. For condition#1, eleven (11) subjects separately collaborated with the robot for the assembly task introduced in Fig.2, but there was no emotion of the robot. The experimenter subjectively evaluated the HRI for each subject separately based on the following two evaluation criteria using a rating scale between 1 (lowest) and 5 (highest): 1. Awareness: how aware the human was of the task situations (situational awareness) 2. Engagement: how engaged the human was with the robot during the task. The experimenter also subjectively evaluated the assembly performance of each subject using the same rating scale based on the following criterion: 1. Accuracy: how accurately (positional or placement accuracy) the human and the robot collaboratively assembled the objects. The similar procedures were followed for condition#2, but the robot had a single static emotion (concentration) displayed on the screen while working with the human. Figure 5 shows the experiment procedures for conditions#1 and 2. VII. EXPERIMENT RESULTS We determined the mean (n=11) values of the evaluation scores for each evaluation criterion separately for no emotion and static emotion condition as shown in Fig.6. The results show that the static emotion produces better HRI than the no emotion condition. The better HRI also produces better assembly performance for the static emotion condition. However, the reliability of the results will need to be increased by increasing the number of subjects, improving the experiment setup and procedures, improving the evaluation criteria and metrics etc. On the other hand, the static emotion may be proven worse than the no emotion condition if a wrong static emotion is displayed or if the static emotion does not always fit with the dynamic task situations, etc. (a) (b) Figure 5. (a) The human and robot collaboratively assemble three objects where the robot has no emotion, (b) the human and robot collaboratively assemble three objects where the robot has a static emotion (concentrating emotion) No emotion Static emotion Engagement Awareness Accuracy Evaluation criteria Figure 6. Mean (n=11) evaluation results with standard deviations for HRI and assembly performances for no emotion and a static emotion condition for the human-robot collaborative assembly task. VIII. CONCLUSIONS AND FUTURE WORKS We present the collaborative manufacturing robot Rethink Baxter. We also present an assembly task for manufacturing where the robot and the human collaboratively perform the assembly task. We have identified the situations where the emotions of the robot may affect the HRI and the manufacturing performances. We then proposed intelligent control algorithms to adjust the robot emotions dynamically with that of the task situations. As the proof of concepts, we also preliminarily investigated the effects of emotions on HRI and assembly performance through a no emotion versus static emotion experiment for the assembly task. The results

6 preliminarily show that the static emotion produces better HRI and assembly performance (on average) than that produced for the no emotion condition. In the near future, we will reinvestigate the effects of emotions on HRI and assembly performance through no emotion versus static emotion experiment using more subjects, better hypotheses and improved experiment methods and metrics. We will report the results of the effectiveness of the control algorithms in producing the dynamic emotions as well as of the effects of the dynamic emotions on the HRI and on the assembly performances in the manufacturing. We will determine the dynamics of the robot emotions inspired by that of the human and will quantify the dynamic evolution of robot emotions through analytical mathematical equations. We then further enrich the control algorithms based on the emotion dynamics and adjust the emotions of the robot with that of the human and with the task situations. ACKNOWLEDGEMENT The authors express thanks and gratitude to BMW Manufacturing Co., Spartanburg, SC , USA for supporting with the Baxter robot used for the research presented in this paper. The authors are also thankful to all the subjects who participated in the experiments. REFERENCES [1] D. Robinson, Brain function, mental experience and personality, The Netherlands Journal of Psychology, Vol.64, pp , [2] C. Breazeal, Emotion and sociable humanoid robots International Journal of Human-Computer Studies-Application of affective computing in human-computer interaction, Vol. 59, No. 1-2, 2003, pp [3] K. Malchus, P. Jaecks, O. Damm, P. Stenneken, C. Meyer, B. Wrede, The role of emotional congruence in human-robot interaction, in Proc. of th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pp , 3-6 March [4] C. Breazeal, Sociable machines: expressive social exchange between humans and robots, Sc.D. Dissertation, Department of Electrical Engineering and Computer Science, MIT, [5] A. Breemen, X. Yan, and B. Meerbeek, icat: an animated userinterface robot with personality, in Proceedings of the Fourth International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS '05), ACM, NY, USA, pp , [6] S. Wachsmuth, S. Schulz, F. Lier, F. Siepmann, and I. Lutkebohle The Robot Head Flobi : A Research Platform for Cognitive Interaction Technology, S. Wölfl (Ed.): Poster and Demo Track of the 35th German Conference on Artificial Intelligence (KI-2012), pp. 3-7, [7] S. Rahman, Generating human-like social motion in a human-looking humanoid robot: The biomimetic approach, in Proc. of 2013 IEEE International Conference on Robotics and Biomimetics (ROBIO), pp , Dec [8] S. Nishio, H. Ishiguro, N. Hagita, Geminoid: teleoperated android of an existing person, Chapter in Humanoid Robots: New Developments, I- Tech Education and Publishing, Vienna, Austria, pp , June, [9] F. Eyssel, D. Kuchenbrandt, F. Hegel, and L. Ruiter, Activating elicited agent knowledge: how robot and user features shape the perception of social robots, in Proc. of the st IEEE Int. Symp. on Robot and Human Interactive Communication, pp [10] F. Hegel, S. Krach, T. Kircher, B. Wrede, Understanding social robots: a user study on anthropomorphism, in Proc. of 2008 IEEE Int. Symp. on Robot and Human Interactive Communication, pp [11] K. Kawamura, T. Rogers, K. Hambuchen, D. Erol, Towards a human robot symbiotic system, Robotics and Computer Integrated Manufacturing, Vol.19, pp , [12] C. Heyer, Human-robot interaction and future industrial robotics applications, Intelligent Robots and Systems (IROS), 2010 IEEE/RSJ International Conference on, pp , Oct [13] S. Nikolaidis, P. Lasota, G. Rossano, C. Martinez, T. Fuhlbrigge, and J. Shah, Human-robot collaboration in manufacturing: quantitative evaluation of predictable, convergent joint action, in Proc. of the 44th International Symposium on Robotics, [14] [15] [16] M. Nasr, Modeling emotion dynamics in intelligent agents, Master Thesis, Texas A & M University, USA, [17] J. Steephen, HED: a computational model of affective adaptation and emotion dynamics, IEEE Transactions on Affective Computing, Vol. 4, No. 2, pp , April-June [18] E. Schmidt and Y. Kim, Modeling musical emotion dynamics with conditional random fields, in Proc. of th International Society for Music Information Retrieval Conference (ISMIR 2011). [19] M. Belhaj, F. Kebair, and L. Said, A computational model of emotions for the simulation of human emotional dynamics in emergency situations, International Journal of Computer Theory and Engineering, Vol. 6, No. 3, pp , [20] D. Kim, P. Baranyi, Novel emotion dynamic express for robot, Applied Machine Intelligence and Informatics (SAMI), 2011 IEEE 9th International Symposium on, pp , Jan [21] S.M.M.Rahman, R. Ikeura, Weight-perception-based novel control of a power-assist robot for the cooperative lifting of light-weight objects, International Journal of Advanced Robotic Systems, Vol.9, No.118, 13 pages, Oct [22] S.M.M.Rahman, R. Ikeura, Improving interactions between a power assist robot system and its human user in horizontal transfer of objects using a novel adaptive control method, Advances in Human-Computer Interaction, Vol. 2012, Article ID , 12 pages, December [23] S.M.M.Rahman, R. Ikeura, M. Nobe, S. Hayakawa, H. Sawai, Weightperception-based model of power assist system for lifting objects, International Journal of Automation Technology- Special Issue on Robotic Technology to Extend Workers Physical Abilities and Skills, Vol.3, No.6, pp ,Nov