Effectiveness of Multi-modal Techniques in Human-Computer Interaction: Experimental Results with the Computer Chess Player Turk-2

Transcription

1 Effectiveness of Multi-modal Techniques in Human-Computer Interaction: Experimental Results with the Computer Chess Player Turk-2 Levente Sajó, Péter Váradi, Attila Fazekas University of Debrecen, Faculty of Informatics Image Processing Group of Debrecen H-4010 Debrecen, P.O.Box 12, Hungary inf.unideb.hu, inf.unideb.hu Abstract. Development of techniques based on multi-modal human-computer communication highly contributes to create user friendlier informational systems. An experimental system, Multi-modal Chess Player Turk-2, has been built which is able to play chess but also fulfills the requirements of multi-modality. The first report about this system was published in [1]. Now we describe the human experiments made with this system and present the results of the evaluation. These results proves us that the multi-modal interaction makes information systems to become more human-like and easier to use thus becoming more acceptable by everyday people. Keywords. Computer Chess Player, Turk-2, Multi-modal Human-computer Interaction, human experiments. 1. Introduction Development in computer industry step-bystep has involved computers to our everyday life. Computers become faster and cheaper, which lead to the spread of computer-based information systems in different areas. Information systems could mean the general purpose personal computers used at people's home but also those machines which were built for a special task, e.g. ATMs at banks or interactive schedules and ticket machines at train stations. In spite of these systems became increasingly efficient and easier to use, from the part of the users a kind of disfavor can be observed. This can be because of the lack of knowledge for controlling the new technologies. Another reason is that every system should be used differently, thus day by day a lot of new information needed to be learned to be able to use them. Furthermore, the time is getting shorter between the releases of new generation systems. The way of usage of the information systems is similar to the usage of communication languages. If we know more languages, it is easier to learn a new one. If we are familiar in using more information systems, then using a new system causes fewer problems for us. The difficulties in the communication with these systems are that the user is the one who should learn the language of a system, how to give commands to it and how to understand its answers. The research field of multi-modal human-computer interaction addresses the problem of developing information systems to be able to communicate with the users in a more human-like way. Since we only consider the communication between humans and computers, the terms of "Multi-modal Human-Computer Interaction" (MMHCI) and "Multi-modal Interaction" (MMI) will be used as synonyms. 2. Multi-modal Human-Computer Interaction It is difficult to define what exactly makes the communication human-like. Usually it means using input and output channels which are more natural for users. Appearing Graphical User Interfaces at the beginning of '80s was the first step in the direction of the user-friendliness. GUIs changed the previous text-based interfaces by providing a common look and feel. Icons, drawings and images were used to represent the data visually and provided direct control of the objects by using mouse and keyboard. Appearing graphical operating systems like Apple MacOS, Microsoft Windows or Linux X Window System helped a lot in the spreading of personal computers. Even GUIs became very popular among the users they still only uses a subset of human sensory systems. Multi-modal user interfaces are the candidates to continue the way of building better user interfaces started by the GUIs. A multi-modal application is different from GUI applications, beyond the traditional input devices, it is using alternative communication

2 channels. Multi-modal human computer interaction considers the relation of computers with the users from the human side focusing on perception and control. The term modality refers to human senses or input/output channels. Raisimo divided human senses in seven groups from neurobiological point of view: internal chemical, external chemical (taste, smell), somatic sense (touch, pressure, temperature, pain), muscle senses (stretch, tension, join position), sense of balance, hearing and vision. The goal of multi-modal human computer interaction is the synergetic use of the most of the abilities that human have [6]. Our vision is such future information systems which are able to use parallelly multiple input and output modalities to interact with the user in a more natural and efficient way. Users can communicate by using human language and gestures, and the systems can understand the all parameters of the conversation. The systems should be able also to reply in an user friendly way, by speaking and making different gestures. Of course, these requirements are quite general and we are still far from a complete information system based on multi-modal human-computer interaction techniques, but there are some results which point to the directions described above. Analyzing human body gestures has been a preferred topic by the researchers. For example gestures have been extracted in fields of martial arts games. [4] computer aided Tai Chi is presented. On human body inexpensive wearable sensors were installed for capturing Tai Chi movements. The captured movement later is analyzed by the computer increasing the effectiveness of human-made analyzation. Audio channels also have been used for input modalities. Speech recognition tools like Microsoft Speech API can be used for recognition of limited set of spoken commands. In [3] Igarashi and Hughes proposes the use of non-verbal features in speech, like pitch, volume, and continuation, to directly control interactive applications as an alternative of recognizing spoken commands. They give examples such as television controller, where the user says "Channel up, ta ta ta" the channel number increases by three. Some works address the combinations of different input channels. In [2] is presented a perceptual user interface for controlling a flying cartoon-animated dragon in QuiQui's Giant Bounce. This is an interactive computer game for 4 to 9 years old children. The dragon flies on the user's movements and breathes fire when the user shouts. Using multi-modal inputs and outputs in the same application resulted interesting solutions. In [7] an interactive poker game is presented in which one human user plays against two animated agents. The application combines several modalities like facial gesture and body animation, emotion modeling and speech synthesis for driving the behavior of virtual characters and this way enhancing the naturalness of interaction in the game. Building humanoid robots is a highly preferred topic in many studies. These "emotional" robots are able to move their body and their hands and on their faces they can imitate human emotions. Such a robot is presented in [5], called icat, which is the opponent of a human player in a chess match, and its emotive behavior is influenced by the game state. Our aim was to build a system which implements interfaces based on several modalities both on input and output channel and to study if the human-like interface provides an added value. We chose chess as a kernel game application because it is a popular and wildly known game and therefore learning the rules causing fewer problems for the users. We wanted to put our effort to the communication part of the system with the users and not on the artificial intelligence part. We implemented the system in performed user studies with two versions, with multi-modal communication module (reffered in the following: TH) and without (NoTH). By comparing reactions, we intended to get an insight about the role of the multi-modal interfaces. 3. The Turk-2's architecture In this chapter we describe briefly our multimodal chess player system. The central part of the system is the controller which consists of three modules: the turn manager, which is responsible for the game flow and orchestrates the communication between the user and the chess engine, the emotion monitor, which keeps track of the emotional state of the user and the Turk2, and the chess engine which acts as the "mind" of the Turk2, deciding on the chess moves. The different components can be grouped into two main modules. (See Fig. 1.)

3 Figure 1. The architecture of Turk-2. The first module (marked with dots in Fig. 1) is the human-computer communication module, providing multi-modal input and output facilities. This module has two kinds of input, one from the speech recognizer and one from the player's face analyzer, which localizes faces and detects the facial expressions on the human player's face. The inputs from these components are processed by the turn manager, which generates the output and forwards it to the talking head. The second module (marked with dashes in Fig. 1) is the chess game module. This module contains two sub-modules, the chess state detector and the robot-arm with its controller. The chess-engine is responsible for the chessgame. The second detects the changes on the chess-board, and if there was a move, sends it to the chess-engine. The chess-engine generates the next move, which is executed by the robot-arm on the board. The reader can find more details about the different components of the systems in [1]. In the following we focus on presenting the human tests and their evaluations. 4. Experimental study The goal of this study is to assess the importance of multi-modal interfaces in humancomputer interaction. We wanted to find out weather adding a multi-modal interface to a computer chess game results better game experience. The chess player Turk-2, sketched in the previous chapter, was used to make human tests and the results of these tests have been evaluated to confirm our observations The scenario In the test 16 people have been participated. The participants were between 18 and 25 years old, eight of the participants were males, and eight females. They were everyday computer users not having too much connection with computer science. They did not know anything about the Turk-2 system, they were only told that they need to play chess against the computer. The chess playing skills of the participants varied from basic level (knowing only the rules) to the level of an average chess player because we wanted to test our system as a game and not as an exercise. Before starting a session, each participant received instruction from an assistant. The assistant also described what the test is about and set up the system. During the game time the assistant remained "invisible". The participants had to play two games per test. One game was with a simplified version of Turk-2 - without multi-modal communication part (NoTH) - the other was with the complete system (TH). The orders of the games were decided randomly at the beginning. During the games a video was recorded of the participants and the important (unexpected) events were collected into a report by assistant. In 6 sessions, parallelly, a video was taken of the talking head, too. The videos were processed later. At the end of the test the participants were asked to fill a questionnaire. After the test, each video taken about human faces or about talking head's faces was was annotated with event and timestamp pairs. The questionnaire contains ten yes/no questions regarding the participants' impressions about Turk-2 system. The answers to these questions should be rated from 1 to 10. The questions oriented to three main topics: which game did the participants enjoy more, what were their opinions about the human-likeness of the system provided by the multi-modal techniques and did they consider the talking head as the part of the game. The questionnaire also contains a part where participants could write some sentences about their impressions.

4 Figure 2. The distribution of looking directions in percentage of the whole duration of the game: with TH when the player is on turn, with TH when the computer is on turn, with noth when the player is on turn, with noth when the computer is on turn. Figure 3. Average laughing, smiling and speaking time in percentage of the whole duration of the game. In games with TH the users are laughing, smiling and speaking more then in games with noth. This happens mainly when they are looking on it at the same time Evaluation of the tests Both objective and subjective evaluation of the tests were used. In case of objective evaluation the extracted datas from the annotation files and the questionnaires were submitted to statistical analyzation. As software SPSS v17.0 were used. First, the normality of the statistical variables were checked (it varied between p=0.4 to p=0.9) and then Student's t-test were applied. In some cases bivariate correlations and Friedman test for several related samples have been applied. In case of the videos recorded from the players' face, the total duration of the events are calculated and these summarized values are used to define the hypotheses. The videos of the talking head were analyzed in connection with the other video for determining the effects of the mult-modal components on the participants. The operator's notes and our impressions about the videos gave us a lot of interesting observation, too. In the following the research questions and their answers are discussed from five different aspects: How did the players react to the talking head? Because chess is a game played in turns, human players changed their behaviors depending on who is on turn. When it was human player's turn, most of the time they were looking at the chessboard and thinking on their next moves. During computer's turn, there was nothing to engross the players. They were usually looking outside of the game scene. If the robot arm started to move, they followed it with their eyes. The presence of talking head was important to fill the emptiness while the computer was "thinking". Applying t-test, it can be proved that the means of total durations of looking to the chessboard were statistically equal in both test cases (p=0.822). Futhermore, in case of TH looking out and looking on the robot arm decrased (p=0.006 and p=0.002) and instead participants were looking on the talking head. Fig. 2 shows the distribution of mean looking directions in both cases (TH and noth). Applying bivariate correlation, statistically can be showed that eye contacting with talking

5 Figure 4. Females were smiling, laughing and talking more then males. head happened when the players were in passive state (p=0.01, corr=0.836). It happened, mainly, when they completed their move and the computer was on turn. Mostly this was after the turn-taking. But many times they also glanced to the talking head while they were following with their eyes the robot-arm moves. It can be showed that the players were smiling, laughing and talking more in games with TH (p=0.001) which is illustrated on Fig. 3. Furthermore, smiling, laughing and talking in games with noth and without looking at the talking head in games with TH happend in the same amount (p=0.794). The presence of the talking head made the players to perform these utterances in increased amount while they were looking at the talking head at the same time Is the talking head treated as a person? The talking head's presence had effect on the players' reactions during the game. They treated the Turk-2 as a person. Usually human players requited talking head's manifestations with smiles and laughing. Many times the participants replied to the talking head's questions and sentences. For example when the computer finished the move and then said "Check this out" the answer was that "I'm not happy with this". Sometimes they praised the computer after a good move: "Clever!" or fretted oneself "You are glad to hit, aren't you?". They also made different comments about the talking head's face: "You are so sly!", "Evil!", "She's actually grinning at me!?", or japed with it: "You are so ugly", "He has sleepy dust in his eyes". Some of the participants were jesting with the talking head by repeating his sentences: "Take your time", "Tough situation". By applying Friedman test on the questionarie, statistically can be proved that the answers connected to the human-likeness of the Turk-2 are related to the engagement and that the participants considered the talking head as a human-like partner in the game (p=0.06). This shows us the importance of developing more human-like MMIs Effect of the talking head on game experience Overall, it can be assumed that participants have enjoyed more playing with TH. On the questionnaire 95% said that it was better to play with TH. They spoke highly about talking head, saying that talking head made the game more interesting and entertaining. Their concentration during the game also confirms the positive effects of the talking head. Usually participants played better with TH, 80% of the games were longer, only in one case happened that somebody got checkmate in less turns playing with TH. The engagement of the participants also can be confirmed, by analyzing the mean thinking time in the two test cases. Using t-test can be proved that the participants were thinking longer in games with TH (p=0.017). The importance of MMI could be observed particularly in those tests when participants played first with TH and then with noth. In these cases the absence of talking head was more conspicuous and the players gave voice to this that "The first game with talking head was much better" or "I'm missing the talking head". Generating the next move and preparing for executing it with the robot-arm, usually took for the computer a few seconds. During this time participants did not understand what's happening. The speed of the robot-arm was found to be also slow in these cases. Players could hardly wait until it finishes. In the games with TH, it was

6 easy to notify the players that it was their turn, by changing the talking head's gaze from the chessboard to the player's direction, or in some cases announcing the opponent in words. They liked when talking head announced his next move or if there was a hit, check or checkmate. In those cases when the talking head skipped to announce his next move or his hit the partcipants noticed it: "Oh, he forgot to tell his next step!" Gender differences Analyzing the test result separately in case of males and females, a few differences can be observed. In overall can be assumed that boys were interested more in playing chess, defeating the computer. Females enjoyed more playing with TH and gave higher ratings in the questionnaires and they also head more interaction with the talking head. By applying t- test, statistically can be proved that females were smiling, laughing and talking more then males (p=0.01) in spite of that there is no difference in mean thinking time between the two genders (p=0.161). (See Fig 4.) Further observations For the question, should the talking head talk more and be more emotional or should it be silent with "poker face", about 70 percent of the participants declared that it should be more emotional, because "playing with a silent and poker-faced talking head is like playing alone". Our talking head, prepared for this system, was somewhere in middle between the two extremes. Usually, participants opinion was that is better to play with a more talkative partner having a varying and large vocabulary. Interpretation of identical expressions: the same facial expression of the talking head was interpreted differently by the players. At the beginning of the game the smile of the talking head was only an "innocent" smile, but at the end of the game, when the computer was close to win, the same smile was interpreted as "malevolent". 5. Conclusion This work concludes the results of human tests in which the effect of multi-modal interfaces in human computer interaction has been studied. Despite the relatively small number of participants in these tests, the results show us that a system which implements MMI solutions appears to be more human-like and provides better game experience. Giving face for the computer makes users to express more likely their feelings, in some cases they even treated the computer as a person by talking to and arguing with it. In the future would be good to make more dynamic tests with a large number of participants in this topic. For this we are going to use another game, called rock-paper-scissors, to which will be attached different multi-modal interface. Having a larger number of tests statistical measures could give more precise results. But, by considering these results can be concluded that developing image processing techniques which leads to better multi-modal interfaces will have a bright future. 6. References [1] Fazekas A, Sajó L. Multi-modal Humancomputer Chess Player The Turk 2. in Proc. of ITI2007, June, 2007, Cavtat, Croatia. [2] Hämäläinen P, Höysniemi J. A Computer Vision and Hearing Based User Interface for a Computer Game for Children. 7th ERCIM Workshop "User Interfaces For All", [3] Igarashi T, Hughes J F. Voice as Sound: Using Non-verbal Voice Input for Interactive Control. 14th Annual Symposium on User Interface Software and Technology, p , [4] Kunze K, Barry M, Heinz E A, Lukowicz P, Majoe D, Gutknecht J. Towards Recognizing Tai Chi - An Initial Experiment Using Wearable Sensors. IFAWC2006, Mobile Research Center, TZI Universität Bremen, Germany, [5] Leite J, Pereira A. icat, the Affective Chess Player. 7th Int. Joint Conf. on Autonomous Agents and Multiagent Systems, vol. 3, p , Estoril, Portugal, [6] Raisamo R. Multimodal Human Computer Interaction: a constructive and empirical study. Academic dissertation, University of Helsinki, [7] Schroder M, Gebhard P, Charfuelan M, Endres C, Kipp M, Pammi S, Rumpler M, Turk O. Enhancing Animated Agents in an Instrumented Poker Game. KI2008. LNCS, vol. 5243, pp , Springer, 2008.