Quick Button Selection with Eye Gazing for General GUI Environment

Transcription

1 International Conference on Software: Theory and Practice (ICS2000) Quick Button Selection with Eye Gazing for General GUI Environment Masatake Yamato 1 Akito Monden 1 Ken-ichi Matsumoto 1 Katsuro Inoue 1,2 Koji Torii 1 1 Graduate School of Information Science, Nara Institute of Science and Technology Takayama, Ikoma, Nara , Japan 2 Graduate School of Engineering and Science, Osaka University 1-3 Machikaneyama-cho, Toyonaka, Osaka , Japan masata-y@is.aist-nara.ac.jp Fax: Abstract This paper proposes an efficient technique for eye gaze interface suitable for the general GUI environments such as Microsoft Windows. Our technique uses an eye and a hand together: the eye for moving cursors onto the GUI button (move operation), and the hand for pushing the GUI button (push operation). In order to make our technique more effective for the general GUI environments, we also propose the following two techniques to assist the move operation: (1) Automatic adjustment and (2) Manual adjustment. In the automatic adjustment, the cursor automatically moves to the closest GUI button when we push a mouse button. In the manual adjustment, we can move the cursor roughly by an eye, then move it a little more by the mouse onto the GUI button. In the experiment to evaluate our method, GUI button selection by manual adjustment showed better performance than the selection by a mouse even in the situation that has many small GUI buttons placed very closely each other on the GUI. 1. INTRODUCTION In recent years, user interfaces using humans eye movement as an input have become a popular topic among HCI researchers [9][11][13][16]. Such interfaces called eye gaze interfaces enable us to operate GUIs without using hands. The user of such interfaces only needs to look at the GUI for selecting buttons and/or menu items, zooming in/out a window, scrolling a window and so on [2][4][8][14]. Such an interface is useful not only for the person who has physically handicaps in his/her hand, but also for the person who do not have enough working space for operating mouse and keyboard to use computers. The increase of researches in eye gaze interface is due to the improvement of eye tracking devices for eye measurement [1][6][16]. Formerly, users of an eye tracking device had to fix their head and/or to wear a heavy helmet for measurement. Such a device was not considered as a daily-use input device because of its poor accuracy, difficulties in wearing for a long time, and restrictions on humans' actions. However, today' eye tracking device is getting to be more accurate, convenient and inexpensive. It is enough compact and does not restrict the users actions. Users do not need to wear a heavy helmet anymore; they only need to wear glasses instead [7][8] (see Figure 1). We believe that eye tracking device will become one of the popular input devices in general computer environments in the near future. However, conventional eye gaze interfaces are not supposed to use in general GUI environments such as Microsoft Windows, Mac OS, and X-Window, etc. They assume specialized GUI environments that users do not use their hands. In this environment, the eye tracking device is regarded as a substitute for a keyboard and a mouse. If we apply such interfaces to the general environments, in other word, assuming the situation that we can use our hands, we will have following two problems[15]: (p1) Selection error problem: This problem is known as Midas Touch Problem, which comes from the difficulty in judging whether the user is gazing on a GUI button to select it or just looking it [5]. If the interface simply assumes that a button at which the user looks is to be selected, every GUI button he/she looks at will be selected. In this case, we cannot look anywhere without issuing a button selection. Although such an interface is still very useful in the situation that we cannot use hands, most people who use general GUIs do not find any worth in using such an errorful interface. In addition, we do have some studies that solve this problem; so far, most of them use specialized GUIs that are not easily applicable to the present GUI situations [3][10]. (p2) Selection performance problem: We have no evidence that the conventional eye gaze

2 Move operation with eye tracking device Where is the GUI button Push operation with mouse I have found it. Clicking Figure 2. technique Figure 1. Eye tracking device interface is more efficient than the mouse interface when we apply it to general GUI environments. Generally, they were tested only in the specialized GUI situations that have a few (and also huge) GUI buttons [6] [16]. Actually, there are many tiny icons and buttons scattered on the present GUIs. We need an efficient eye gaze interface suitable to general GUI environments. The goal of this research is to realize an errorless and efficient eye gaze interface suitable to general GUI environments. As a first step in this goal, this paper proposes efficient techniques for selecting GUI buttons on general GUIs. In order to overcome above problem p1 and p2, we propose following two techniques: (1) technique In the proposed interface, we allow users to use their eye and hand together in doing operations in the general GUI environments. An eye is mainly used for moving a cursor; and, a hand is mainly used for identifying a GUI button to be operated. Hence, in order to select a GUI button, for example an icon, a user has simply to look it and push the mouse button. The user can move cursor very quickly and does not be annoyed by Midas Touch Problem, which points out that no one can decide perfectly whether he/she wants to select the GUI button which he/she is gazing on. (2) Adjustment technique To improve accuracy and efficiency of cursor movement by eyes, the proposed interface provides us with two types of cursor location adjustment: automatic adjustment and manual adjustment. Automatic adjustment the cursor automatically moves to the closest GUI button when a user pushes a mouse button. Manual adjustment a user can move the cursor roughly by an eye, and then move it onto the GUI button precisely by a mouse. A similar technique is also proposed by Zhai et al [16]. However, their technique does not consider the situation that has a lot of buttons placed on general GUIs. Size Distance Highlighted button Figure 3. Window for experiment (size: 5cm 2, distance: 5cm) The rest of this paper first describes characteristics of eye as an input to computers and our basic technique to combine an eye tracking device and a mouse (Section 2). Next, This paper describes a pilot experiment that probes and clarifies the actual problems, which may occur when we use our technique in the general GUI environments (Section 3). Then we propose a cursor adjustment technique based on the results of the pilot experiment (Section 4). Afterward, we describe an experiment to evaluate the adjustment technique in various GUI environments (Section 5); and in the end, conclusions and future topics will be shown (Section 6). 2. INTERFACE USING EYE TRACKING DEVICE AND MOUSE TOGETHER In this section, before proposing our eye gaze interface, we discuss the characteristics of an eye as an input to computers, on both strong points and weak points. Based on the discussion, we propose an eye gaze interface in the next. 2.1 Characteristics of an eye as an input to computers Characteristics an of eye as an input to computers are

3 Figure 4. The number of misses (pilot experiment) as follows: (c1) High speed movements Moving cursors by an eye is much faster than by a mouse because the eye can move very quickly in comparison to the hand. Since the range of the eye movement in the computer display is narrow, the speed of eye movements is about from 350 degree/sec. to 500 degree/sec [8]. It takes only about 150 ms to move the gazing point from a corner to the opposite corner of 21-inch size display. (c2) Midas Touch Problem If we consider of an eye gaze interface that can select a GUI button simply by looking it, every GUI button a user looks at will be selected. In this case, the user cannot look anywhere without issuing a button selection. Ideally, the interface should act on the user s eye input only when he/she wants it to and let him/her just look around when that s he/she wants, but two cases are impossible to distinguish in general. Eye gaze interfaces need to avoid this problem [5][12]. (c3) Jittery motions during a fixation Even if a user thinks that he/she is staring at a certain point on the display, actually it is not true because his/her eye makes jittery motions. This makes it difficult for users to point exactly at the very small GUI button by eye. (c4) Measurement accuracy The measurement accuracy of the present eye tracking device is not high enough to point a very small GUI button. In general, the accuracy is about from 0.5 degree to 1 degree as the angle of view. In case that Figure 5. Task completion time (pilot experiment) the user sits 50 centimeter away from a display, the error on the display is about 0.9 centimeter. Considering that the size of GUI button used in general GUI like MS-Windows is about one centimeter square, the accuracy of the eye tracking device is not high enough. 2.2 technique We allow users to use their eye and hand together in doing operations in the general GUI environment. We split the operation of selecting a GUI button into two operations: move operation and push operation. We apply an eye tracking device to the move operation, and a mouse to the push operation (see Figure 2). Under this policy, in order to select a GUI button, for example an icon, a user has simply to look it and click the mouse button. On account of characteristic c1 described in 2.1, eye tracking device is suitable to the move operation. On the other hand, on account of characteristic c2, eye tracking device is not suitable to the push operation; and, the mouse is more efficient for this purpose. 3. PILOT EXPERIMENT 3.1 Objective, materials and task In this experiment, we probe and clarify the actual problems that may occur when we use our combination technique in the general GUI environments. Objective

4 Large button Small button 2 2 Cannot move the cursor to Push operation The gazing point jitters in The gazing point jitters and the button area. moves out of the button area. Figure 6. Influence of the jitter motions Error The closest button is selected. Figure 8. Automatic adjustment Large button Small button Rough movement by look 2 2 error The error is existing but the cursor is in the button area. error The error is existing and the cursor is out of the button area. Figure 7. Influence of the measurement error The major objectives of this pilot experiment are as follows: We need to know how serious the characteristic c3 (jittery motions during a fixation) will be, when we use the combination technique in actual GUI environments. We also need to clarify the influence of characteristic c4 (measurement accuracy of an eye tracking device). Subject The subject is one graduate student at Nara Institute Science and Technology. The subject uses Microsoft Windows usually and is familiar with operations by mouse. Task We prepared many sizes of GUI buttons; and, we placed them in many layout patterns. As a single task of this experiment, the subject is asked to select one highlighted GUI button out of nine buttons on a window for 10 times (Figure 3). The highlighted point will change randomly after selecting a GUI button. Ten kinds of the GUI button layout patterns are prepared: 5 different sizes of GUI buttons (1 square centimeter, 2 square centimeters, 3 square centimeters, 4 square centimeters and, 5 square centimeters), and two different distances between GUI buttons (5 centimeters and 1 centimeters). The subject is asked to execute the task twice: once by using the combination technique and once by using a mouse. Before executing a task, the subject is asked to train each technique well. Therefore, the result of the experiment is supposed to have little influence of experiences in each technique. Collected data Figure 9. Manual adjustment The time spent for selecting GUI buttons 10 times is collected in each layout pattern. We also counted the number of selecting misses. Here, we regard it as a miss when the subject did the push operation on a button that is not highlighted or he/she did it in outside of the GUI button areas. If the subject made a mistake, the window beeps as an alert. In this case, the subject is asked to try to select the highlighted button till he/she selects a correct GUI button. 3.2 Results of pilot experiment Delicate adjustment by mouse The number of misses while executing the task is shown in figure 4. The task completion time is shown in figure 5. We describe each of them next. Misses As shown in figure 4, we find that only a few misses are found if the GUI buttons are larger than 3 square centimeters. In such cases, with the combination technique, the subject can select a GUI button without making a mistake. However, it is difficult to select a GUI button smaller than 3 square centimeters using the combination technique. If a GUI button gets smaller than 3 square centimeters, the number of misses increases more. (The measurement error of eye tracking device is about one centimeter.) Task Completion Time Figure 5 shows that the combination technique is more efficient than the mouse technique when the GUI button is larger than 3 square centimeters. This is observed in both 5cm distance case and 1cm distance case. However, in case the button is smaller than 3 square centimeters, the combination technique is less effective than the mouse technique. Especially, in case the button size is one square centimeter, the combination technique is extremely ineffective in both 5cm distance case and 1cm distance case.

5 Considering both the number of misses and the task completion time, we find it is almost impossible to select the small button by the combination technique. Since a small GUI button of one square centimeter is often used in general GUIs, we need to improve our technique. 3.3 Problems in combination technique As a result of the pilot experiment, we find it is difficult to select small GUI buttons by the combination technique. We consider reasons of the difficulty are: In case the GUI button is small, the user cannot keep the cursor pointing on a small GUI button because of the characteristic c3 (the jittery motions during a fixation). On the other hand, in case the GUI button is large, the path of jittering gazing point is inside the area of the GUI button (see Figure 6). In case the GUI button is small, because of the characteristic c4 (measurement accuracy is not high enough), the eye tracking device often cannot detect the correct GUI button on which the user is actually gazing. On the other hand, if the GUI button is large, the GUI button holds the cursor in the correct GUI button area even if the measurement error does exist (see Figure 7). 4. ADJUSTMENT TECHNIQUES In order to reduce the misses and to make our technique more efficient, we propose two adjustment techniques that assist the move operation: automatic adjustment and manual adjustment. In the automatic adjustment, the interface adjusts the cursor position automatically on to the nearest GUI button. In the manual adjustment, the user can move the cursor roughly by an eye, and then move it onto the GUI button precisely by a mouse. 4.1 Automatic adjustment technique If a user uses the automatic adjustment technique, the closest GUI button from the place where the user is looking (gazing point) is selected automatically when the user did the push operation. If the gazing point is already in the area of GUI button, the behavior of the cursor in the automatic adjustment technique is same as the combination technique with no adjustment. Even if the gazing point is out of an area of GUI button because of the measurement error and/or jittery motions, the closest button from the user s gazing point will be selected. This technique is useful especially in case a GUI button is very small (see Figure 8). Furthermore, the combination technique with automatic adjustment might be more effective than the combination technique without adjustment technique because if we use the automatic adjustment, we do not need to look at the GUI button precisely. We can look at the GUI button roughly to select it. 4.2 Manual adjustment technique If a user uses the manual adjustment technique, the input device for the move operation is switched from the eye tracking device to a mouse when the user moves the mouse by his/her hand. Position of the cursor is kept when we switch the input device. The user of the manual adjustment can use the eye tracking device for rough cursor movement and can use the mouse for the delicate adjustment. (see Figure 9). The behavior of the cursor in the manual adjustment technique is same as the combination technique without adjustment if the user did not move the mouse. If the user found that it is difficult to move the cursor onto an area of a certain GUI button because of the measurement error and/or jittery motions, the user can switch the input device into the mouse. If the user switches the input device properly, the misses may be reduced. 5. EXPERIMENT 5.1. Design of the experiment We have conducted an experiment to evaluate the two adjustment techniques. The design of the experiment (subject, task and collected data) is the same as the pilot experiment, except the techniques we used. The subject is asked to execute the task twice: once by the combination technique with automatic adjustment and once by the combination technique with manual adjustment. 5.2 Result of the experiment The numbers of misses observed in tasks are shown in figure 10. Task completion times are shown in figure 11. In order to make it easy to compare two adjustment techniques with the mouse technique, the results of the pilot experiments are also shown in figure 10 and 11.

6 Manual Automatic Manual Automatic 16 Automatic Manual Manual Automatic Misses Unlike the result of pilot experiment, the distance between GUI buttons has an influence on the number of misses and the task completion time. From figure 10, in the 5cm distance layout, it is clear that the number of misses is very few in both automatic adjustment case and manual adjustment case. If we use the adjustment techniques, we can decrease the number of misses as compared with the case where the combination technique without any adjustment technique is used. However, in the 1cm distance layout, many misses were observed when the user used the automatic adjustment technique. As the size of the GUI button gets smaller, the number of misses becomes larger. On the other hand, when the user uses the manual adjustment, almost no miss was observed. This means we can use the manual adjustment technique without any stress. Task Completion Time Figure 10. The number of misses In the 5cm distance layout, by using either automatic adjustment technique or manual adjustment technique, the user can select the GUI button faster than by using the mouse technique. Furthermore, automatic adjustment showed better performance than manual adjustment. However, in the 1cm distance layout, the automatic adjustment is less efficient than the mouse technique when the GUI button is smaller than 4 Figure 11. Task completion time square centimeters. If the size of a GUI button is 1 square centimeter, it is almost impossible to select with using automatic adjustment. On the other hand, manual adjustment technique showed good performance even if the size of the button is 1 square centimeter. Comparisons among the techniques So far, the manual adjustment showed best performance among four techniques (combination, automatic, manual, and mouse) in all the situations. Table 1 summarizes the result of the experiment. In the columns of task completion time, comparisons between proposed techniques and the mouse technique are shown. In the right column, sums of misses are shown. From table 1, we see that the manual adjustment showed very few misses; and also, it showed better performance than the mouse, in all the situations. Therefore, we believe that the manual adjustment is useful in general GUI environments that have many tiny GUI buttons placed closely each other. 6. CONCLUSIONS AND FUTURE TOPICS We have proposed a new approach of the eye gaze interface in which computer users use both their eye and hand in doing operations in general GUI environments. The basic idea of our approach is to use the eye for moving a cursor and to use the hand for identifying a GUI button to be selected. To improve the accuracy and efficiency of cursor movement by an

7 Table1. Comparison between adjustment techniques Task completion time Far Near Adjustment Large Small Large Small Sum of misses No Adjustment Faster Slower Faster Slower 49 Automatic Faster Faster Faster Slower 26 Manual Faster Faster Faster Faster 2 Far and Near stands for the distance between GUI buttons. Large and Small stands for the size of GUI buttons. eye, the proposed interface provides two types of cursor adjustment technique: an automatic adjustment technique and a manual adjustment technique. The proposed interface has been applied to an experiment in which various GUI environments were employed. The results showed that the proposed interface augmented by manual adjustment is superior to the mouse interface in terms of accuracy and efficiency of GUI button selection even if many small GUI buttons have been placed with closely. This result indicates that the manual adjustment technique is useful in general GUI environments that have many tiny GUI buttons placed closely each other. In the future, we are to extend our technique for the drag-and-drop operation; and also, we are planning to use a keyboard and an eye together to operate GUI objects concerning to the text input. We believe that incorporating an eye with the present GUI environments will lead us to the more useful, comfortable and efficient interfaces in the near future. Reference 1. Ebisawa, Y., Ohtani, M. and Sugioka, A. Proposal of a zoom and focus control method using an ultrasonic distance-meter for video-based eye-gaze detection under free-head conditions, Proc. the IEEE Engineering in Medicine and Biology Society (1996), CD ROM Edition. 2. Findlay, J.M., Walker, R. and Kentridge, R.W. EYE MOVEMENT RESEARCH Mechanisms, Processes and Applications, Vol. 6 of Studies in visual information processing (1995). ELSEVIER. 3. Hansen, J.P. et al. Eye-Gaze Control of Multimedia Systems, Symbosis of Human and Artifact, Anzai, Y., Ogawa, K. and Mori, H. (Eds.), vol. 20A (1995), 37-42, Elsevier Science. 4. Howarth, P. Keeping an eye on your interface: The potential for eye-based control of graphical user interfaces (G.U.I s), Proc. HCI 94 (1994). 5. Jacob, R.J.K. What You Look At is What You Get: Eye Movement-Based Interaction Techniques, Proc. CHI 90 (1990), Kuno, Y., Yagi, T., Fuji, H., Koga, K. and Uchikawa, Y. Development of Eye-gaze Input Interface using EOG, Transactions of Information Processing Society of Japan, vol. 39, no. 5 (1998), (in Japanese) 7. Ohno, T. Features of Eye Gaze Interface for Selection Tasks, 3 rd Asia Pacific Computer Human Interaction (1998), IEEE Computer Society, Ohno, T. Quick Menu Selection Task with Eye Mark, Transactions of Information Processing Society of Japan, vol. 40, no. 2 (1999), (in Japanese) 9. Salvucci, D.D. Inferring Intent in Eye-Based Interfaces: Tracing Eye Movements with Process Models, Proc. CHI 99 (1999), Shaw, R., Crisman, E., Loomis, A. and Laszewski, Z. The Eye Wink Control Interface: Using the Computer to Provide the Severely Disabled with Increased Flexibility and Comfort, 3rd Annual IEEE Symposium on Computer-Based Medical Systems (1990), Velichkovsky, B.M. and Hansen, J.P. New Technological Windows into Mind: There is More in Eyes and Brains for Human-Computer Interaction, Proc. CHI 96 (1996), ACM Press, Velichkovsky, B., Sprenger, A. and Unema, P. Towards gaze-mediated interaction: Collecting solutions of the Midas touch problems. < ml>. 13. Vertegaal, R. The GAZE Groupware System: Mediating Joint Attention in Multiparty Communication and Collaboration, Proc. CHI 99

8 (1999), Yamato, M., Monden, A., Takada, Y., Matsumoto, K. and Torii, K. Scrolling the Text Windows by Looking, Transactions of Information Processing Society of Japan, vol. 40, no. 2 (1999), (in Japanese) 15. Yamato, M., Monden, A., Matsumoto, K., Inoue, K., Torii, K. Button Selection for Common GUIs Using Eye and Hand Together, Proc. AVI 2000 (to be appeared). 16. Zhai, S., Morimoto, C. and Ihde, S. Manual And Gaze Input Cascaded (MAGIC) Pointing, Proc. CHI 99 (1999),