PupilMouse: Cursor Control by Head Rotation Using Pupil Detection Technique

Transcription

1 PupilMouse: Cursor Control by Head Rotation Using Pupil Detection Technique Yoshinobu Ebisawa, Daisuke Ishima, Shintaro Inoue, Yasuko Murayama Faculty of Engineering, Shizuoka University Hamamatsu, , Japan ABSTRACT We propose PupilMouse in order to control a cursor on a PC screen by movements of a pupil or two pupils in the camera image, caused by head movements. In the system, the states of the eyes (open or close) can be judged by detecting the existence of the pupils, they are reflected to the button push states of the ordinary two-button mouse. A combination of a couple of infrared illuminator made it easy to detect two pupils. The image processing was executed using only a general-purpose image input board and a PC. An advantage of this system is that it is unnecessary to attach marker or sensor to the head. Another advantage is that a click or drag is possible without any other device. This human interface would be useful especially for physically handicapped people, who can rotate their heads, to control a PC. Keywords: pupil detection, handicapped, human interface, cursor control, button control 1 INTRODUCTION Even the severely handicapped can control electric home appliances and communicate with the outside world via internet, if they can use a PC. In the severely handicapped, there are many people who can move the head. In many cases, in order to use a PC, they push keys with a stick mouthed. However, in the GUI environment, mouse operation plays an important role as well as key operation. On the contrary if they can use a certain mouse emulator, the repetition of the cursor movement and click enable also character inputs on all application software by using a software keyboard presented on a PC screen. Therefore, if a good device that can move the cursor with head movement is developed, it is expected that it enable the handicapped people to operate a PC easily. So far, the head movement-based cursor control devices using ultrasonic sensors [1], infrared LED [2], and tilt sensor [3], and the method using a magnetic sensor, have been proposed. In all cases, the sensor must be attached to the head; this becomes a burden for users. In such a device, HeadMouse [4] and NaturalPoint [5] have come out to the market. In these methods, a special marker (seal) is attached to the forehead usually. The position of the marker is detected by a camera; its motion is reflected to cursor motion. In this method, although attaching the marker has some problem, the biggest problem is that there is no way to click in the device itself. To solve this problem, when the cursor is kept for a short period of time, e.g. 1 sec, within an area where the user wants to click, a click is carried out. However, when the user carelessly stops the head elsewhere, this method results in making the user click a different area from that which the user wants to click. The system developed by Betke et al. is commercialized as CameraMouse [6]. It tracks the predetermined portion of the face by the template matching method. This system has an advantage that it is unnecessary to attach something to the head. However, the system itself has no way to click. Betke et al. tried to deal with the problem by preparing the area where a click is not carried out even if cursor has stopped [7]. However, since it is necessary to move the cursor to the area intentionally, it may not be so user-friendly. We are developing the non-contact video-based eye-gaze detection system [8]. For highly precise eye-gaze detection, it is necessary to detect 3D eye position precisely. So we focused on the fact that it is relatively easy to detect pupils from a face image if irradiating the near-infrared light for the face. And we tried to detect the 3D pupil positions of the two pupils by the stereo camera method, and got a good result. The good result came from the image processing method in one camera image and the near-infrared irradiation method, which made it easy to detect the pupil centers from the face image precisely and stably [9]. By using the pupil detection technique, basically the right and left pupils could be distinguished and their pupil center coordinates as well as their existences in the image could be detected accurately. We considered that since opening and closing of the eyes can be detected if using this technique, a click and drag are implemented with opening and closing of the eyes. In this paper, we propose PupilMouse, in which a motion of the pupil(s) in the image is reflected to a motion of the cursor of a mouse and the opening and closing of the eyes are reflected to the button states of the mouse. 2 METHODS Pupil Center Detection Technique If the near-infrared LEDs arranged around the aperture of a camera as shown in Fig. 1 are switched on, the image having the bright pupils (bright pupil image) can be obtained as shown in Fig. 2(a). If the same LEDS arranged apart from the aperture of the camera are

2 switched on, the image having the dark pupils (dark pupil image) can be obtained (Fig. 2(b)). A subtraction of the dark pupil image from the bright pupil image make it easy to detect the pupils because almost all background images disappear as shown in Fig. 2(c). Similar pupil detection techniques have been described elsewhere [8][10]-[12]. In the present system, the above-mentioned two sets of LEDs were alternately switched on, synchronously with the video odd/even signal. Using an image grabber board, the image from the camera is digitized and transferred into the memory on a PC. The PC made an image processing including image difference and detected the center coordinates and the existence of the two pupils. However, in the difference image, the face contour also appears. The contour and the glass reflection light which is the LEDs light reflected off corrective eyeglasses can be misidentified as pupils. To avoid this, (1) the face area was detected from the bright pupil image, utilizing the fact that the face area is brighter than the background. (2) Once the pupils were detected, small windows were given around the pupils (see Fig. 2(d)-(f)). Initial pupil detection procedure was repeated every frame when the pupil was not detected. In the procedure, by subtracting the four-frame former odd image from the current odd image, the areas of the eyes were extracted within the face area as a binary image. From the difference image (the subtraction image of the current dark pupil image from the current bright pupil image) within the eye areas, the small area including the pixel having the highest intensity was detected as primary pupil. After masking this area, the secondary pupil was detected likewise. As mentioned above, initial detection of the pupils was executed utilizing a blink. (3) In the subsequent frames, the pupil positions are predicted using the Kalman filter. The small windows were applied to the predicted positions. The pupil was searched within the windows. The pupil center was determined in high resolution using a pupil parameter obtained from the combination of the separability [13] and the image intensity. (4) Eye closing was judged by the existence of the pupil within the window, which was identified using the pupil parameter and a predetermined constant threshold. The detailed image processing method was described in [9]. The average calculation time for detecting the coordinates of two pupils was approximately 10 ms in a PC (Pentium 4, 2.5MHz). To diminish the influence of the ambient light, the band-pass filter (half intensity width: 17nm) having the same central wavelength as that of the LEDs was attached to the front of the camera lens. eye ends the drag (Fig. 3(b)). Cursor shift during dragging is reflected to the movement of the center of the pupil of the opened eye. When blinking, both pupils disappear. A blink is reflected to no cursor shift or no button push. Fig. 2. Fig. 1. Arrangement of LEDs. Images and detection of pupils. Scheme of Pupil Mouse As shown in Fig. 3(a), basically movements of the centers of the two pupils in the camera image, which are caused by shifting or rotating the head, shift the cursor on a screen of a PC. Whether the eyes are open and closed can be determined by the existences of the pupils. By a wink of the right or left eye, a corresponding right or left click can be executed. Closing one eye and moving the head starts the drag and subsequently opening the closed Fig. 3. Correspondence between pupil center shifts in camera image and cursor shift on PC screen.

3 Correspondence of Pupil Detection States to Button Push States As shown in Table 1, there are four combinations in the detection states of the right and left pupils: (1) both pupils detected, (2) right pupil detected and left pupil undetected, (3) left pupil detected and right pupil undetected, and (4) neither pupil detected. These four kinds of pupil detection states are reflected to the button push states of a general-purpose two-button mouse. (1) Neither button pushed (up, up), (2) only left button pushed (up, down), and (3) only right button pushed (down, up). (4) This condition occurs when a blink occurs. Fortunately, on the two-button mouse, the condition that both buttons are pushed (down, down) is not used. So, in the case of (4), suppose that neither of the buttons is pushed like the case of (1). Here, although there can be a cursor movement in the case of (1), no cursor movement in the case of (4). Furthermore, the following processing is carried out about the blink. When it had been judged that one of buttons was pushed in the previous image frame and neither pupil is detected in the current frame, a blink may have occurred. In this case, it is not judged immediately as the condition where neither button is being pushed. If the same pupil detection state has continued for a while, the button push state is judged as changed. Moreover, also in an ordinary blink, two pupils may not disappear or re-appear simultaneously, but a time difference of one or two frames may arise. If the pupil detection states are reflected to the button push states simply, it carries out different operation from the user's intention. Therefore, in any shift of the pupil detection states, it was shifted to the corresponding new button push state when a new different pupil detection state has continued for three frames (0.1 seconds). Table 1. Correspondence of pupil detection states to button push states. Correspondence of Pupil Movement to Cursor Movement When both eyes are open as shown in Fig. 1(a), the amount of change between frames of the average position of both pupil centers is reflected to the amount of cursor movement between frames. Here, the reason for using not one pupil center but two pupil centers is for cursor movement to be more stable. Since one eye is closed during a drag, the amount of movement of the pupil of an open eye is reflected to the amount of cursor movement. Here, to relate pupil center movement to cursor movement, it is possible to use the following equation (linear function). Cx = Kx Px (1) Cy = Ky Py where Px and Py indicate horizontal and vertical pupil center movement between frames and Cx and Cy are horizontal and vertical cursor movement between frames. Kx and Ky are constants. They are freely changed by a user so that it may be easy to use. Now, assume that a user moves the cursor to the target position and then closes one eye as it is. Although it is easy to make the cursor approach a target position, it is accompanied by some strain to make the cursor agree with the target position. In that case, a user's head may vibrate delicately. The head movement may lead to vibration of the cursor. So we tried also the following nonlinear equations. b Cx = Kx Px (2) b Cy = Ky Py In these equations, If b > 1, the cursor shows a tendency to move greatly by a quick pupil motion and shows almost no reaction by a slow pupil motion. Eq. (2) is written for its intelligibleness. Practically, the following equations were used. Cx = Kx Px Px Cy = Ky Py Py b 1 b 1 (3) In the experiments, they are used as b=2 (quadric function). Experiments All processing described below was executed using a PC (Pentium 4, 2.5GHz). A camera and light sources can be placed everywhere, e.g., above or side of the display. In this experiment, they were placed below the 17 inch PC display as shown in Fig. 4. Three normal students were served as subjects. They were seated approximately 70 cm from the display. On the display screen, the face images as shown in Fig. 5 were observable. By observing those images, the states of pupil detection were checked. However, during experiments, a window as shown in Figs. 6 and 7 was extended on the whole screen. The range to which the cursor moves in the window (white portion) was 330 x 240 mm and corresponded to 1276 x 935 pixels. By the image processing [9], the center coordinates of the two pupils in each frame were determined by sub-pixel within in horizontal and in vertical. The following two experiments were conducted to show the usefulness of the pupil mouse: Click and drag experiments. Click experiment: In every experimental trial, a cursor and Target 1 appeared at random positions in the window (Fig. 6). The subject was asked to move the cursor to the target by head movement and then to close his left eye (to click). The sizes of the target and the cursor were 12 x 12 mm (45 x 45 pixels) and 5 x 5 mm (18 x 18 pixels), respectively. When the upper left corner of the cursor was included in the area of the target, the color of the target was reversed. This reversal let the subject know

4 that it was in preparation for click. When it was judged that the click occurred when the target was in reverse, the click succeeded and the trial was completed, and then immediately the next trial was started. When the click failed, the trial was continued until the click was successful. Three experimental sessions were conducted for each subject. Each session consisted of 10 trials. Subjects A and B participated in this type of experiment for the first time. Subject C was one of the developers of this system and she was familiar to this experiment. Both the experiments using the linear function as Kx=12 and Ky=14 in Eq. (1) and the experiment using the quadric function as b=2, Kx=6 and Ky=8 in Eq. (2) were conducted. These values were chosen commonly for the three subjects so that they can move the cursor reasonably to the ends of the screen. For each of the experiments using the linear and the quadric functions, the first session was started after approximately two minutes training. The same click experiment was conducted by using an ordinary mouse. Drag experiment: In every trial, the cursor and Targets 1 and 2 appeared at random positions (Fig. 7). The cursor and Target 1 were the same as used in the click experiment. The size of Target 2 was 14 x 12 mm (52 x 45). In this experiment, the subject was asked to move the cursor to Target 1, and then to close the left eye. By this procedure (the same as the click experiment), Target 1 was attached to the cursor and the drag was started. Next, the subject was asked to move the cursor with the target to Target 2 by head movement, with the left eye closed. When the upper left corner of the Target 1 was included in the area of Target 1, the color of this target was reversed. The subject was asked to open the closed left eye. When it was judged that the left eye was opened when the target was in reverse, the drag succeeded and the trial was completed, and then immediately the next trial was started. When the drag failed, the trial was repeated until it was successful. Three sessions were conducted in each subject. Each experiment consisted of five trails. For the relationship between the pupil and cursor motions, the quadric function was used as Kx=6 and Ky=8 in Eq. (2). Fig. 5. Face images for checking pupil detection Fig. 6. PC screen in click experiment. Fig. 7. PC screen in drag experiment. Fig. 4. Pupil mouse system. 3 RESULTS Click Experiment Table 2 indicates the period of time from Target 1 appearance until click completion. In the averages of the three subjects, the pupil mouse showed approximately 3 sec while the ordinary mouse took 1.0 sec. However,

5 subject C showed 2.2 sec in the pupil mouse. It can be said that the speed was very quick. The required time was hardly different between for the linear function and quadric function. However, on the average of the three subjects, the misclick number for the quadric function showed half of that for the linear function. In actual PC operations, the misclick means that the user clicked a different position and needs a time to correct the error by the misclick. Accordingly, the decrease of the misclick number is important. Figs. 8 and 9 show samples of the cursor trajectories for the linear and quadric functions, respectively, on subject C. In these figures, the small dots depict the cursor positions in every frame, and the squares indicate the positions where Target 1 appeared. The numbers indicate the trial order. When the cursor moved largely, the distances between the adjacent dots for the quadric function (Fig. 9) were longer than for the linear function (Fig. 8). Within or around the target, the cursor positions disperse largely for the linear function but small for the quadric function. This difference is one of the reasons why the misclick number decreased for the quadric function for the linear function. Drag Experiment Table 3 shows the average of a required time from the start of a drag (capturing Target 1) until the end of the drag (releasing the target). It included the time in the case of releasing and recapturing the target on the half way of dragging (drag failure). Like the click experiment, subject C was faster than the other two subjects. For all subjects, the required time in the drag experiment was longer than that in the click experiment. Fig. 10 shows the cursor trajectories from the start until the end of the drags. The types of squares in this figure indicate the appearing positions of Targets 1 and 2. Fig. 8. Fig. 9. Sample of cursor trajectories in click experiment (linear function). Sample of cursor trajectories in click experiment (quadric function). Table 2. Results of click experiment (N=30). Time Required for One Number of Misclick Click Subejct Pupil Mouse Ordinary Pupil Mouse Ordinary Linear Quadric Mouse Linear Quadric Mouse A 3.8s 3.6s 0.9s B 2.8s 3.3s 1.0s C 2.2s 2.2s 1.1s Mean 2.9s 3.0s 1.0s Table 3. Results of drag experiment (N=15). Time Required for One Click Subject Pupil Mouse (Quadric) Ordinary Mouse A 4.8s 0.9s B 4.7s 1.3s C 2.4s 1.1s Mean 4.0s 1.1s Fig. 10. Sample of cursor trajectory in drag experiment (quadric function). 4. DISCUSSION The results for subject C showed good enough to use the pupil mouse in both the click and drag experiments. The other two subjects were naïve for these experiments;

6 it was difficult to rotate their head obliquely. Accordingly, they showed a tendency to rotate horizontally at first, and to rotate vertically, and vice versa. Such two-step head movements seemed to be a reason why the two subjects were slower than subject C. The cursor trajectory shown in Fig. 8 corresponds to the shift of two pupils between frames. This result indicates that the pupil centers could have been detected precisely and stably. It is most important for the pupil mouse to detect the pupil centers precisely and stably. Accordingly, appropriate face illumination and image processing methods were necessary. The detail of the pupil detection method used in the present study was described in [9]. Since the pupil center signal has not been moving-averaged, the subjects did perceive almost no delay of cursor movement against head movement. There was the case that the click failed even though the subjects closed or opened their eye after confirming that Target 1 or 2 reversed in color. One of the reasons results from the use of the cursor position in the moment the pupil just disappeared or reappeared. That is, there are several frames from the moment the users decided to close or open their one eye until it was judged that the corresponding pupil disappeared or reappeared. During these periods of time, the eye or the head may moves. These cause the cursor shift after the user intended to click. This problem was solved to some extent by using the quadric function for the relationship between the pupil and cursor motions. However, this problem would be almost completely solved by using the cursor position at the ascended time from the moment it was judged that the pupil disappeared or reappeared. 5. CONCLUSION At the present stage of this study, the pupil mouse has a problem that many misclicks are seen. This problem would be solved by the simple device. The pupil mouse has some advantages as a pointing device. First, it can be used with no sensor or no marker attached to the face. Second, no calibration is necessary unlike the line-of-sight pointing devices. Third, anyone can immediately use the system by placing his or her face in the camera frame because initial setting for image processing (e.g., threshold setting) is unnecessary. Fourth, due to the use of the infrared light sources, the pupil mouse can be used in complete darkness. In the present report, the gains (Kx, Ky) between the pupil motion and cursor motion did not changed for all subjects. However, it is possible to choose the better gains adjusted for each subject. Moreover, for the use of the severely handicapped, the use of the correspondence between the pupil position and the cursor position may be better than the use of the correspondence between the pupil movement and the cursor movement. We want to examine it. For the experiments in the present study, we developed the special programs. However, they can not operate the cursor on other application software. To make the pupil mouse emulate the ordinary mouse is one of the future subjects. Furthermore, we want to enable character inputs on application software by the combination of a software keyboard and the pupil mouse, to have the handicapped use it, and to confirm the usefulness of the pupil mouse. ACKNOWLEDGMENT The present research was supported by Cooperative Link of Unique Science and Technology for Economy Revitalization (CLUSTER) in Hamamatsu area (Hamamatsu Optronics Cluster). REFERECES [1] M. Nunoshita, Y. Ebisawa, Head Pointer Based on Ultrasonic Position Measurement, Proc of the Second Joint EMBS/BMES Conference, 2002, pp [2] D.G. Evans, R. Drew, and P. Blenkhorn, Controlling mouse pointer position using an infrared head-operated joystick, IEEE Trans. Rehab. Eng., Vol. 8, No.1, 2000, pp [3] Y.-L. Chen, Application of tilt sensors in human-computer mouse interface for people with disabilities, IEEE Trans. Neural Syst. & Rehab. Eng., Vol. 9, No. 3, 2001, pp [4] [5] [6] [7] M. Betke, J. Gips, and P. Fleming, The camera Mouse, Visual Tracking of Body Features to Provide Computer Access for People With Severe Disabilities, IEEE Trans on Neural Syst. and Rehab. Eng., Vol.10, No. 1, 2002, pp [8] Y. Ebisawa, Improved video-based eye-gaze detection method IEEE Trans. Instr. and Meas., Vol. 47, No. 4, 1998, pp [9] Y. Ebisawa, Realtime 3D Position Detection of Human Pupil, Proceedings of IEEE International Conference on Virtual Environments, Human-Computer Interfaces, and Measurement Systems (VECIMS 2004), Boston, MD, USA, July 2004, pp [10] A. Tomono, M. Iida, Y. Kobayashi, A TV camera system which extracts feature points for non-contact eye movement detection, Proceedings of the SPIE Optics, Illumination, and Image Sensing for Machine Vision IV, Vol. 1194, 1989, pp [11] C. H. Morimoto, D. Koons, A. Amir, M. Flickner, Pupil detection and tracking using multiple light sources, Image and Vision Computing, Vol.18, 2000, pp [12] Q. Ji, X. Yang, Real-time eye, gaze, and face pose tracking for monitoring driver vigilance, Real-Time Imaging, Vol.8, 2002, pp [13] K. Fukui, Edge extraction method based on separability of image features, IEICE Trans. Inf. & Syst., Vol. E78-D, No.12, 1995, pp