Implementation of Face Detection and Recognition of Indonesian Language in Communication Between Humans and Robots

Transcription

1 2016 International Conference on Information, Communication Technology and System (ICTS) Implementation of Face Detection and Recognition of Indonesian Language in Communication Between Humans and Robots Muhtadin Department of Multimedia and Network Engineering Achmad Rizal Muttaqin Department of Electrical Engineering Surya Sumpeno Department of Multimedia and Network Engineering Abstract Communication between man and machine is a challenging subject. In general, the communication between man and machine is done with rigid language (inhuman). Many scientists are trying to create a way of communication between humans and machines charmer, more personal and interactive. This paper presents the results of research work that implements interactive communication between humans and machines (robots). Communication becomes more personal by using face detection to the speaker (human). Communication is also done with more interactive by adding the ability for robots to communicate using Indonesian language. We utilize the NAO robot as a machine that can interact with humans. We employ speech Engine System consisted of Speech to Text, dialog machine, and Text to Speech which make NAO Robot can speak and understand simple human speech in Indonesian Language. The resulted system was able to perform face detection continuously on angles of range between 0 and 30, detect human speech accurately by 80.45%, and give response to human speech appropriately by 47.58%. Keywords Human-Robot Interaction; Artificial Intelligence; Face Detection; Speech to Text; Text to Speech I. INTRODUCTION Research on Human-Robot Interaction is a relatively new topic and can still be developed to further direction. Robotbased services such as food serving for patients in hospitals, household work have brought autonomous robots into human life [1]. However, the aspect of human-robot interaction in doing these tasks is still at a level which is minimum. Humanrobot social interaction plays an important role in spreading the use of the robot in human daily life. Through effective social interaction, robots will be able to perform many tasks in the human society. These tasks may include, but not limited to, handling various house duties, providing medical care for elderly people, assisting people with motor or cognitive disabilities, educational entertainment (edutainment), personal assistance, giving directions at information points in public places, etc [2]. Research on Human-Robot Interaction is often implemented on a humanoid robot. A humanoid robot is a robot with its body shape built to resemble that of the human body. In general, humanoid robots have a torso, a head, two arms, and two legs, though some forms of humanoid robots may model only part of the body, for example, from the waist up. Some humanoid robots may also have heads designed to replicate human facial features such as eyes and mouths [3]. NAO Robot as a humanoid robot, still have developable features. One of these developable features is that how NAO can communicate with human interactively. Making interactively communicative NAO Robot can be achieved by detecting its surrounding object visually or generating speech using Speech Engine System. Object detection method is intended to make NAO Robot can detect its surrounding object, which in this case is human face. And about Speech Engine System mentioned before, it is a system consisted of Speech to Text, dialog machine, and Text to Speech which make NAO Robot can speak and understand simple human speech in Indonesian Language. II. DESIGN AND IMPLEMENTATION A. System Design In this research, the system is designed to implement interactive communication between human and Aldebaran NAO robot using face detection and Indonesian language. Overall system design is illustrated in Fig. 1. In this system, the voice from the operator serves as the system input. When the system detects a human face, then the system will begin to detect the sounds around. If the system detects that there are sounds around, then the system will perform tracking on detected human face. Detected sound will then be analyzed to obtain features that represent the information contained in the sounds. The information obtained will be used as input to the artificial intelligence component of the system. Artificial intelligence component in the system is a subsystem in the form of a combination of units of simple knowledge. Artificial intelligence component will obtain the output corresponding to the given input based on matching unit of the knowledge. The output of the artificial intelligence component will then be used as input command to the actuator component of the system. The actuator components of the system will then perform some actions in the form of movement or speech corresponding to the given input command /16/$ IEEE 53

2 Fig. 1. System Design B. Workflow Based on the concept of the system that has been described, the workflow of the system from beginning to end can be divided into two parts, that is: the part related to the interaction based on face detection and the part related to the interaction based on Indonesian language. By dividing the workflow of the system into several smaller parts, the overall system will be expected to be relatively easy to implement. And in general, the system is implemented as a Fig. 2. Template Matching Algorithm Flowchart desktop application written in the Python programming language and named as NANA (NANA Aldebaran NAO App). C. Face Detection based Interaction Implementation Face detection based interaction implementation is done by going through the steps illustrated in Fig. 1(a). The stages are carried out as follows, in the beginning the system obtain the image from the robot s camera sensor. The system will then perform face detection process on the obtained image. The face detection process on the system is implemented using Haar-Cascade Classifier method that has been implemented in OpenCV [4]. The result of the face detection process is the coordinate position of the detected face. The obtained coordinates will then be used as input to update the angle position of the robot s head. The system will then obtain the image from the robot s camera sensor again and does the same process continuously. In addition, the face detection process is also designed using Template Matching method, the method of matching a particular image (template) in the other image. The algorithm used by the system to perform Template Matching method is illustrated in Fig. 2. D. Indonesian language based Interaction Implementation Indonesian language based interaction implementation is done by going through the steps illustrated in Fig. 1(b). The stages are carried out as follows, in the beginning the system obtain human speech by means of recording human speech into wav file format. The file that contains the recorded human speech will then be analyzed to obtain information in 54

3 text form. Process of analyzing audio data to obtain information in text form is referred as Speech to Text. The Speech to Text process of the system is implemented using the help of Google Speech API [5]. In detail, the Speech to Text process of the system is implemented as a routine that sends request to Google Speech API's servers with audio file contained recorded human speech as data. The result of the Speech to Text process is text that contains information about the contents of the audio file that is sent to Google Speech API's servers. The obtained text will then be used as input of artificial intelligence components of the system. Artificial intelligence component of the system is implemented using AIML (Artificial Intelligence Markup Language), which was first implemented as a chatterbot named ALICE (Artificial Linguistic Internet Computer Entity) [6]. Simply put, artificial intelligence components are component that acts as dialog machine. The knowledge of the dialog machine defined as units of a pair input pattern and output templates that are related to specific topics of conversation. The input text data will be matched with the input pattern and the artificial intelligence components will then provide responses in text form. The resulted text will then be used as the input to the Text to Speech process, ie. process of synthetizing audio from text data. The Text to Speech process on the system is implemented using espeak [7], MBROLA [8], and Indonesian voice database that has been made in preexisting research on the Indonesian Text to Speech [9]. The results of the Text to Speech process is an audio file. In addition, the obtained text is also matched with BEAST (Bodily Expressive Action Stimulus Test) pose database [10]. The synthetized audio file will then be sent to the Aldebaran NAO robot and played through the robot s speakers. In addition, the robot Aldebaran NAO also perform movements that mimic the pose BEAST. By sounding an audio file on the robot s speakers and making robot mimicking BEAST pose, they will make Aldebaran NAO robot as if it is able to interact with humans. E. Robot Pose Design based on BEAST BEAST (Bodily Expressive Action Stimulus Test) [10] pose implemented in this project aims to improve interactivity of the robot. This is done because the robot does not have facial expressions so that it is able to express emotions during interaction. In the process of making the design of BEAST pose implementation on the robot, Aldebaran NAO robot has degree of freedom that can be seen in Table 1. The process of making robot s BEAST pose implementation is done by using software provided by Aldebaran, namely Choregraphe. BEAST poses used in this project at total of 9 poses, all of which are poses that express happy emotions. The sample of BEAST pose implementation at Aldebaran NAO robot can be seen in Table 2. TABLE 1. ALDEBARAN NAO H25 V33 RANGE OF JOINT VALUES IN DEGREE Joint Angle Name Min Value Max Value Head Pitch RShoulder Pitch Roll RElbow Roll RWrist Yaw RHand Open and Close LShoulder Pitch Roll LElbow Roll LWrist Yaw LHand Open and Close Hip YawPitch RHip Pitch Roll RKnee Pitch RAnkle Pitch Roll LHip Pitch Roll LKnee Pitch LAnkle Pitch TABLE 2. EXAMPLE OF BEAST IMPLEMENTATION ON NAO Pose Code BEAST Pose Implementation on NAO F01HA III. EXPERIMENT RESULTS AND ANALYSIS A. Speech to Text During test of Speech to Text, the response which the system gives to voice input provided by the user is observed. From the audio data, most of the words are able to be detected by the system well. Fig. 3 below shows sample of detected word of audio data received by the system. And success rate of Speech to Text process given by total 312 words which provided by 13 participants is provided by Fig. 4. Fig. 3. Sample of Speech to Text Result 55

4 Exactly detected Improperly detected Respond appropriately unable to respond Fig. 4. Speech to Text success rate The undetected or incorrectly detected words can be caused by some factor, such as noise around the voice source, volume of voice source which too low, or system which is not ready yet to process the user s speeches. B. Dialog Machine During the test of machine dialog performance, the degree to which the system is able to give response to user s speeches which have a lot of possibilities is observed. Some of the input and output which considered as matched pair by dialog machine is shown in Table 3. Input which have no matching pair will be matched to pattern * which means system is unable to give expected response, thus giving unexpected response. And ratio between expected and unexpected response given by dialog machine is provided by Fig. 5. TABLE 3. INPUT-OUTPUT PAIR EXPECTED BY MACHINE DIALOG Input Pattern Matched Response 1. Selamat Pagi 2. Selamat Siang 3. Selamat Sore 4. Selamat Malam 1. Apa kabar 2. Apa kabar NAO 3. Halo apa kabar 4. Hai apa kabar 1. Siapa namamu 2. Namamu siapa 3. Halo siapa namamu 4. Hai namamu siapa 1. Nama saya rizal 2. Saya rizal 1. Berapa jumlah roda mobil 1. SELAMAT * 1. APA KABAR 2. APA KABAR * 3. _ APA KABAR 1. SIAPA NAMAMU 2. NAMAMU SIAPA 3. _ SIAPA NAMAMU 4. _ NAMAMU SIAPA 1. NAMA SAYA * 2. SAYA * 1. * 1. Selamat Pagi 2. Selamat Siang 3. Selamat Sore 4. Selamat Malam 1. Saya baik-baik saja terima kasih 1. Saya robot NAO 2. Saya NAO 1. Senang bisa berkenalan dengan Anda rizal 2. Salam kenal rizal 1. Maaf saya tidak mengerti maksud Anda Fig. 5. Machine Dialog Performance C. Face Detection During the test of face detection performance, the range of angles that must be met for the system to be able to detect the face of the user is observed. Angles used on this test are 0, 30, and 45 clockwise and counter clockwise. The test for angles of 30 and 45 are done by doing head-turning movement from angle of 0 to 30 and 45 respectively, both clockwise and counter clockwise. Some angles tested in this project are shown in Table 4. TABLE 4. VALUE OF ANGLES USED FOR FACE DETECTION TEST Angle Face Detected Note 0 Yes - 30 (clockwise) Yes - 30 (counter clockwise) Yes - 45 (clockwise) Yes often difficult to detect 45 (counter clockwise) IV. Yes CONCLUSION faces often difficult to detect faces. This research has successfully implemented a new way of interaction between human and robot. Interaction with robots becomes more personal with the features face tracking to the speaker, how to communicate also become more interactive by using everyday language, using Indonesian. From this research work, we obtained the following conclusions: 1. Implementation of Face Detection and Face Tracking using OpenCV library can be quite good with the conditions detected faces you want to be facing angle between 0-30 degrees. 56

5 2. Implementation of Speech to Text using Google Speech API proved to be quite good with an accuracy of 80.45%. 3. Implementation of the dialog machine using AIML make the system capable of responding to 47.58% of the human speech appropriately. REFERENCES [1] D. Wilkes, A. Alford, R. Pack, R. Rogers, R. Peters, and K. Kawamura, Toward socially intelligent service robots, Appl. Artif. Intell. J., vol. 12, pp , [2] Salichs, M.A.; Barber, R.; Khamis, A.M.; Malfaz, M.; Gorostiza, J.F.; Pacheco, R.; Rivas, R.; Corrales, A.; Delgado, E.; Garcia, D.;, "Maggie: A Robotic Platform for Human-Robot Social Interaction," Robotics, Automation and Mechatronics, 2006 IEEE Conference on, vol., no., pp.1-7, 13 June [3] Humanoid robot - Wikipedia, the free encyclopedia. [Online]. Available at: [Accessed: 05-Jan-2015]. [4] OpenCV OpenCV. [Online]. Available at: [Accessed: 05-Jan-2015]. [5] Chrome Browser. [Online]. Available at: [Accessed: 05-Jan-2015]. [6] A. L. I. C. E. The Artificial Linguistic Internet Computer Entity. [Online]. Available at: [Accessed: 05-Jan- 2015]. [7] espeak: Speech Synthesizer. [Online]. Available at: [Accessed: 05-Jan-2015]. [8] The MBROLA PROJECT HOMEPAGE. [Online]. Available at: [Accessed: 05-Jan-2015]. [9] Indonesian Text to Speech. [Online]. Available at: [Accessed: 05-Jan-2015]. [10] B. de Gelder and J. Van den Stock, The Bodily Expressive Action Stimulus Test (BEAST). Construction and Validation of a Stimulus Basis for Measuring Perception of Whole Body Expression of Emotions, Frontiers in Psychology, vol. 2,